Live outage feed

Outages and recoveries from the services you depend on.

Earlier incidents

40 incidents

Back to latest

Aug 4, 2023

Aug 3, 2023

Aug 2, 2023

Aug 1, 2023

Jul 31, 2023

Jul 29, 2023

  1. MapboxAbsent congestion coloring in China, Korea, Southwest Japan, and South East Asia

    We have identified an issue impacting traffic visualization and ETA's in the following countries: China, Korea, Southwest Japan (locations west of Tokushima), and much of South East Asia. There were three major impact areas, as described below. All have been resolved as of the timestamps listed for each. Customers using traffic-maps v1 & v2 in the affected region would have seen tiles with drastically underreported congestion coloring, from the period of 2023-07-17T04:04:08.148Z to 2023-07-22T01:30:12.079Z UTC (4 days 21.5 hr). Customers using the mapbox/driving-traffic profiles on the Directions, Map Matching, and Matrix APIs may have received inaccurate ETA's during the period of 2023-07-19 04:49 to 2023-07-22 01:53:48 UTC (2 days and 21 hours). Finally, customers of mapbox-speeds typical data received incomplete data from the period of 2023-07-12 05:03:52 to 2023-07-21 18:05:55 UTC (9 days 13 hours). The issue is now resolved and we sincerely apologize for any inconvenience.

    Resolved

Jul 28, 2023

  1. ApigeeEdgeCloud: Intermittent runtime impact for some customers in US-West

    The issue was identified to a single node issue in one of our Cassandra pods in the US-West region which was mitigated. We're not seeing any further errors post mitigation but will continue to monitor. We'll do a review of this incident and take the necessary actions to improve our systems to prevent a recurrence. Thanks again for your patience and partnership!

    Resolved

  2. MailChannelsOutage

    All the queued mail should be delivered now. If you are still seeing any issues, please contact support@mailchannels.com

    Resolved

  3. MailChannelsReseller API Degradation

    We have now identified and resolved the performance issues with the Reseller API. If you have any questions or concerns, please contact MailChannels Support (support@mailchannels.com)

    Resolved

Jul 27, 2023

Jul 26, 2023

Jul 25, 2023

  1. KeeperLogin Errors - US Production

    For 9 minutes from 2:46PM PST to 2:55 PM PST, a high number of login errors occurred as a result of a scheduled backend API software update. The issue resolved itself after the update was completed. The development team is researching the underlying cause.

    Resolved

Jul 24, 2023

  1. MailChannelsReseller API Degradation

    We have now identified and resolved the performance issues with the Reseller API. If you have any questions or concerns, please contact MailChannels Support (support@mailchannels.com)

    Resolved

Jul 21, 2023

  1. SvixUS region: delay in delivery of some messages + a small amount of API endpoints had higher latency

    There was delay in sending messages in the US region and higher latency in some API endpoints. As for the latency: around 0.003% of the requests during this period had a "client disconnect" (which means that the client was disconnected before the server responded which usually means a client timeout or a networking issue), and a total of 2 "create message" calls took more than 5 seconds (across all customers, so a very tiny fraction of the requests).

    Resolved

Jul 20, 2023

Jul 19, 2023

  1. MapboxElevated API Errors

    Due to an incident with our cloud service provider, multiple Mapbox services had elevated HTTP 421 errors. This issue lasted from 4:30 PM to 5:15 PM UTC. The issue was resolved and all services returned to normal operation afterwards.

    Resolved

Jul 18, 2023

  1. GOV.UKIntermittent failures of Whitehall scheduled publishing

    We have triaged this issue to the relevant team to fix. Only a small subset of publications are affected, and 2nd line are able to manually publish affected editions as and when the issue occurs, meaning that the only user-visible impact is a small delay to a small proportion of scheduled publications. We are marking this incident as resolved now.

    Resolved

Jul 17, 2023

Jul 15, 2023

Jul 14, 2023

  1. MapboxLatency and timeouts on forward geocoding

    On Thursday, 07/13/2023 between 19:20 UTC and 20:14 UTC (a period of 54 minutes) a subset of forward geocoding requests were impacted by high latency and some timeouts. The service is now operating normally.

    Resolved

  2. OpsgeniePerformance issues and outages with Cloud products

    ### **SUMMARY** We understand the importance of providing reliable and consistent service to our valued customers. On July 6, 2023, from 03:52 to 15:11 UTC, we experienced an issue with an upgraded version of a third-party tool that functions as our internal artifact management system. Despite our monitoring system identifying the incident within two minutes, this issue led to the degradation of the scaling capabilities of our internal hosting platform, resulting in service degradation or outages for customers of Atlassian cloud. In response to this situation, we are taking immediate measures to enhance the stability of our system and prevent similar issues from re-occurring. ### **IMPACT** This incident affected multiple regions and products due to the diminished scaling capabilities of our internal hosting platform. In most products and offerings, customers faced reduced functionality, slower response times, and limited access to specific features. ### **ROOT CAUSE** The root cause of the incident was the introduction of new functionality in a third-party tool that functions as our internal artifact management system. It led to an unexpected increase in the load on the primary database of the artifact system. Upon identifying and localizing the problem, we promptly adjusted the system configuration to regain stability. ### **REMEDIAL ACTIONS PLAN & NEXT STEPS** Over the next months, we will enact a temporary freeze on non-critical upgrades of the artifact management system, and we will focus our efforts on three high-priority initiatives: 1. **Enhancing system scaling:** We prioritized work ensuring that downtime in a critical infrastructure component does not affect the scaling of other components. We expect to complete this initiative within the next two months. 2. **Reducing interdependencies:** We are working to mitigate the risk of potential cascading failures by ensuring that significant system components are able to operate independently in the case of issues. Initiatives 1 and 2 are already in progress but have been given priority to be completed as soon as possible. 3. **Strengthening testing procedures:** Alongside these initiatives, we are addressing the need for even more stringent testing procedures than we already have in place to prevent potential issues in future updates. We are committed to collaborating closely with our technology partners to ensure the most optimal experience for our customers. We apologize for any inconvenience caused by this incident and appreciate your understanding. Our team is dedicated to continually improving our systems and processes to provide you with the exceptional service you deserve. Thank you for your continued support and trust in us. Sincerely, Atlassian Customer Support

    Resolved

Jul 13, 2023

Jul 11, 2023

Jul 10, 2023

Older incidents