DopplerUnable to fetch secrets from the Doppler dashboard and API
This incident was caused by a faulty Kubernetes NetworkPolicy change. We’ll be evaluating how we can adjust our deployment procedures to catch these changes in the future.
Resolved
Outages and recoveries from the services you depend on.
40 incidents
This incident was caused by a faulty Kubernetes NetworkPolicy change. We’ll be evaluating how we can adjust our deployment procedures to catch these changes in the future.
Resolved
Our engineers have deployed a fix and the issue is now resolved
Resolved
This issue has now been resolved. Users with clusters that were previously throttled will now see their changes to resource limits reflected.
Resolved
We mitigated the issue with Sign-ups, Product Activation, and Billing, and the systems are back to BAU, and all functionality is restored.
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
We have identified an issue impacting traffic visualization and ETA's in the following countries: China, Korea, Southwest Japan (locations west of Tokushima), and much of South East Asia. There were three major impact areas, as described below. All have been resolved as of the timestamps listed for each. Customers using traffic-maps v1 & v2 in the affected region would have seen tiles with drastically underreported congestion coloring, from the period of 2023-07-17T04:04:08.148Z to 2023-07-22T01:30:12.079Z UTC (4 days 21.5 hr). Customers using the mapbox/driving-traffic profiles on the Directions, Map Matching, and Matrix APIs may have received inaccurate ETA's during the period of 2023-07-19 04:49 to 2023-07-22 01:53:48 UTC (2 days and 21 hours). Finally, customers of mapbox-speeds typical data received incomplete data from the period of 2023-07-12 05:03:52 to 2023-07-21 18:05:55 UTC (9 days 13 hours). The issue is now resolved and we sincerely apologize for any inconvenience.
Resolved
The issue was identified to a single node issue in one of our Cassandra pods in the US-West region which was mitigated. We're not seeing any further errors post mitigation but will continue to monitor. We'll do a review of this incident and take the necessary actions to improve our systems to prevent a recurrence. Thanks again for your patience and partnership!
Resolved
All the queued mail should be delivered now. If you are still seeing any issues, please contact support@mailchannels.com
Resolved
We have now identified and resolved the performance issues with the Reseller API. If you have any questions or concerns, please contact MailChannels Support (support@mailchannels.com)
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
Issues impacting GCP Multi-Region Serverless Public Preview host cluster connectivity are now resolved.
Resolved
For 9 minutes from 2:46PM PST to 2:55 PM PST, a high number of login errors occurred as a result of a scheduled backend API software update. The issue resolved itself after the update was completed. The development team is researching the underlying cause.
Resolved
🛠 We've resolved all known issues of Vods not playing across Twitch.
Resolved
This incident has been resolved.
Resolved
We have now identified and resolved the performance issues with the Reseller API. If you have any questions or concerns, please contact MailChannels Support (support@mailchannels.com)
Resolved
This incident has been resolved
Resolved
This incident has been resolved.
Resolved
There was delay in sending messages in the US region and higher latency in some API endpoints. As for the latency: around 0.003% of the requests during this period had a "client disconnect" (which means that the client was disconnected before the server responded which usually means a client timeout or a networking issue), and a total of 2 "create message" calls took more than 5 seconds (across all customers, so a very tiny fraction of the requests).
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
Due to an incident with our cloud service provider, multiple Mapbox services had elevated HTTP 421 errors. This issue lasted from 4:30 PM to 5:15 PM UTC. The issue was resolved and all services returned to normal operation afterwards.
Resolved
We have triaged this issue to the relevant team to fix. Only a small subset of publications are affected, and 2nd line are able to manually publish affected editions as and when the issue occurs, meaning that the only user-visible impact is a small delay to a small proportion of scheduled publications. We are marking this incident as resolved now.
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
On Thursday, 07/13/2023 between 19:20 UTC and 20:14 UTC (a period of 54 minutes) a subset of forward geocoding requests were impacted by high latency and some timeouts. The service is now operating normally.
Resolved
### **SUMMARY** We understand the importance of providing reliable and consistent service to our valued customers. On July 6, 2023, from 03:52 to 15:11 UTC, we experienced an issue with an upgraded version of a third-party tool that functions as our internal artifact management system. Despite our monitoring system identifying the incident within two minutes, this issue led to the degradation of the scaling capabilities of our internal hosting platform, resulting in service degradation or outages for customers of Atlassian cloud. In response to this situation, we are taking immediate measures to enhance the stability of our system and prevent similar issues from re-occurring. ### **IMPACT** This incident affected multiple regions and products due to the diminished scaling capabilities of our internal hosting platform. In most products and offerings, customers faced reduced functionality, slower response times, and limited access to specific features. ### **ROOT CAUSE** The root cause of the incident was the introduction of new functionality in a third-party tool that functions as our internal artifact management system. It led to an unexpected increase in the load on the primary database of the artifact system. Upon identifying and localizing the problem, we promptly adjusted the system configuration to regain stability. ### **REMEDIAL ACTIONS PLAN & NEXT STEPS** Over the next months, we will enact a temporary freeze on non-critical upgrades of the artifact management system, and we will focus our efforts on three high-priority initiatives: 1. **Enhancing system scaling:** We prioritized work ensuring that downtime in a critical infrastructure component does not affect the scaling of other components. We expect to complete this initiative within the next two months. 2. **Reducing interdependencies:** We are working to mitigate the risk of potential cascading failures by ensuring that significant system components are able to operate independently in the case of issues. Initiatives 1 and 2 are already in progress but have been given priority to be completed as soon as possible. 3. **Strengthening testing procedures:** Alongside these initiatives, we are addressing the need for even more stringent testing procedures than we already have in place to prevent potential issues in future updates. We are committed to collaborating closely with our technology partners to ensure the most optimal experience for our customers. We apologize for any inconvenience caused by this incident and appreciate your understanding. Our team is dedicated to continually improving our systems and processes to provide you with the exceptional service you deserve. Thank you for your continued support and trust in us. Sincerely, Atlassian Customer Support
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
Between 13:53 UTC and 14:06 UTC users would have experienced elevated API errors for our Directions, Isochrone, and Map Matching APIs.
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved