Nov 5, 16:24 UTC Resolved - This incident is now resolved Nov 5, 16:03 UTC Monitoring - We have resolved the API degradation issue, and are monitoring the services. Nov 5, 15:23 UTC Investigating - We are currently investigating an issue with degraded API response times on our platform. Dashboard services may have delayed response times, as well as some of our API endpoint responses. Active platform services are not affected by degradation.
Nov 4, 13:38 UTC Update - We can confirm that the revocation of all affected certificates is now complete. Let's Encrypt has published every revocation in its Certificate Revocation Lists (CRLs). Every Let's Encrypt certificate that was on the affected server has been revoked with the reason "key compromise". Browsers and clients that check revocation will now reject the old certificates. The replacement certificates now in use are not affected, and customers don't need to do anything. Our provider's investigation is still ongoing. We will continue to follow up and will share the final outcome once it is complete. Oct 31, 12:50 UTC Update - We've completed the reissuance of managed TLS certificates for custom domains. All managed certificates have been reissued with new private keys, and the previous certificates have been revoked. No action is required from customers using managed certificates. If you manage your own TLS certificates and haven't done so yet, we continue to recommend revoking and reissuing them with new private keys as a precaution. - What we're doing to prevent this in the future We have stopped using the affected provider while their investigation continues. We recognise that we can't rely on any single layer of security, so we're taking additional steps to reduce the potential impact of similar incidents that could occur in the future. We're reducing what each server holds: we're changing how configurations are distributed so that each PoP only receives the configuration and TLS certificates for the sites it is actively serving, limiting the data stored on any single server. Neither our investigation nor the provider's has found any evidence of unauthorized access to our server. Should anything change, we will provide an update via email and on our status page. Thank you for your patience while we completed this work. At bunny.net, we understand the trust you place in us to keep your content secure, and we take that responsibility seriously. If y
Oct 24, 20:43 UTC Resolved - This incident has been resolved, we're not observing performance issues anymore. Oct 24, 19:35 UTC Monitoring - We've found and resolved the issue and are actively monitoring performance. Oct 24, 18:45 UTC Investigating - We are currently investigating this issue affecting the CDN performance in DE.
Oct 16, 10:19 UTC Resolved - This issue has now been resolved. Oct 15, 21:20 UTC Monitoring - A fix has been implemented and we are monitoring the results. Oct 15, 21:09 UTC Investigating - We're currently investigating intermittent issues when listing/uploading to Storage zones in DE and failed uploads in Stream.
Oct 15, 13:15 UTC Resolved - Root Cause behind this incident was a core network outage on one of our major Frankfurt providers that made a large number of Bunny servers in Frankfurt unreachable between 02:36 and 02:44 UTC. We haven't observed any further issues since 02:44 UTC, nor expect any further issues due to this and will be resolving this incident at this time. Oct 15, 07:24 UTC Update - All affected services have been stable, we are continuing to monitor Oct 15, 02:44 UTC Monitoring - We are investigating a network issue that lasted approximately 10 minutes and affected the availability of the Frankfurt Storage, Magic Containers, Bunny Database and CDN regions
Oct 7, 22:36 UTC Resolved - This incident has now been resolved. Oct 7, 11:13 UTC Monitoring - Our Engineering team has identified a process load issue which has now been resolved. The team are now monitoring the situation Oct 7, 10:13 UTC Investigating - We are currently investigating Database performance degradation in some regions, our Engineering team are currently investigating root cause.
Oct 6, 08:58 UTC Resolved - This incident has been resolved Oct 6, 00:07 UTC Monitoring - A suboptimal log query had put a lot of load to an infrastructure component affecting performance of the whole system. The query has been isolated. We're observing better throughput, but will be monitoring the situation for the next few hours. Oct 5, 23:02 UTC Investigating - We're investigating performance degradation within our CDN Logging infrastructure, impacting our CDN Logging offering.
Sep 25, 17:15 UTC Resolved - A subset of optimizer workers exhibited disk performance issues, impacting Optimizer software's throughput. A fix has been deployed to all servers sharing this same hardware model and performance is back at normal levels again. Sep 25, 14:49 UTC Investigating - We are currently investigating degraded optimizer processing, causing increased latency in serving assets on CDN. This issue sits under image transformation and asset compression only.
Sep 22, 14:07 UTC Resolved - Network performance issues caused by an upstream provider have been mitigated via traffic rerouting. All services and connection timeouts have returned to normal. Sep 22, 13:37 UTC Investigating - We are currently investigating intermittent timeouts with Bunny dashboard access (Login) and API responses within Poland
Sep 21, 22:53 UTC Resolved - This incident has been resolved. Sep 21, 21:42 UTC Monitoring - The synchronization delays have been resolved, and normal operations have been restored. We will continue monitoring the service to ensure everything remains stable. Sep 21, 19:34 UTC Investigating - We are currently investigating an issue causing synchronization delays across our CDN. Our team is actively working to identify the cause and restore normal operations. We will provide further updates as more information becomes available. We apologize for any inconvenience this may cause.
Sep 21, 09:17 UTC Resolved - Video encoding delays have now been fully resolved. Sep 19, 12:00 UTC Monitoring - We have noted improvement in transcoding process delays, and are monitoring the situation. Sep 17, 20:31 UTC Identified - We have been made aware of increased processing time in video encoding queues. We're working on alleviating the underlying issue.
Sep 8, 18:24 UTC Resolved - The issue affecting payment attempts on the billing dashboard has been resolved. Our monitoring has confirmed that the service is operating normally. Sep 8, 17:07 UTC Monitoring - A fix has been deployed for the issue affecting payment attempts on the billing dashboard. The service has been restored, and we are currently monitoring the situation to ensure everything continues to operate as expected. Sep 8, 15:47 UTC Identified - We are currently aware of an issue of failed payment attempts on our billing dashboard on user accounts. Our team are working to restore its functionality.
Sep 3, 15:38 UTC Resolved - This issue is now fully resolved. Our Engineering team have identified additional guardrails to prevent this specific issue from happening in future. Sep 3, 14:25 UTC Update - We have confirmed that all services have fully resumed. Our Engineering team are currently monitoring the situation. Playback has now fully resumed without issue. New uploads during the outage have failed and thus would need re-uploading. Existing transcoding videos were not lost, however the status reported on some of these videos may be inaccurate as a result. Sep 3, 14:01 UTC Update - We are observing some isolated issues on the back of the fix. We are working to fully restore the process on our platform. Sep 3, 14:00 UTC Monitoring - We have identified a backend database issue which had caused a timeout across our video servers (to include our player and uploading process). The problem was rectified and we are now monitoring the situation. Sep 3, 13:47 UTC Investigating - Our Engineering team are currently investigating an issue on our Stream platform of playback failures on our embedded players, as well as upload failures to our platform.
Sep 3, 13:11 UTC Resolved - This incident has been resolved. Sep 3, 08:53 UTC Identified - We are currently observing performance degradation with a Tier 1 transit provider in the Los Angeles area due to multiple fiber cuts on their network. This has impacted connectivity between the Bunny Optimizer LA cluster and a small number of US CDN PoPs. Traffic has been rerouted to minimize the impact.
Sep 2, 15:51 UTC Resolved - A wider network incident in Los Angeles impacted Bunny Services hosted in the region. Affected network providers have mitigated the incident and all impacted services have remained stable since then. Sep 2, 14:38 UTC Investigating - We're investigating reports of widespread network issues in Los Angeles that seem to be impacting our services as well. We are working with our providers in the area to understand the issue at hand and mitigate.
Aug 25, 22:31 UTC Resolved - After monitoring the affected services, we have confirmed that Database management and idle Database activation are operating normally. Aug 25, 16:06 UTC Monitoring - The issue has been resolved and the affected services have been restored. Our Engineering team will continue monitoring the platform to ensure everything remains stable. Aug 25, 16:01 UTC Update - Our Engineering team are continuing to work on the issue. Additionally, we are aware of some working Databases failing during this outage. This is related and our team are also working to address this. Aug 25, 14:06 UTC Identified - We have identified an issue with our Database controller, that our Engineering team are working to restore. Databases already online are not affected and will continue operating normally, but idle Databases or Database management is currently unavailable.
Aug 18, 04:29 UTC Resolved - This incident has been resolved. Aug 18, 04:09 UTC Monitoring - A fix has been implemented and we are monitoring the results. Aug 18, 02:28 UTC Investigating - We are currently investigating elevated error rates on the Storage API in our London, UK region. Some API operations may be affected as a result.