Live outage feed

Outages and recoveries from the services you depend on.

Earlier incidents

40 incidents

Back to latest

Jul 17, 2026

  1. Fly.ioBrief Flycast / MPG interruption in YYZ

    A bad deployment momentarily caused issues with Flycast connectivity, and, by extension, MPG, in our YYZ region. The deployment was immediately rolled back and connectivity was restored shortly after.

    Resolved

  2. GitHubDisruption with some GitHub services

    On July 14, 2026, the GitHub Codespaces service was degraded during two periods — between 06:00 UTC and 09:56 UTC, and again between 10:54 UTC and 12:53 UTC — and some users experienced intermittent failures or delays when creating new codespaces. Impact was concentrated in a subset of geographic regions. During the first period, the error rate averaged 0.5% and peaked at 4.6% of codespace creation requests. The second period was more pronounced, peaking at approximately 30% of codespace creation requests in the most-affected region before recovery. Both periods were caused by an unexpected surge in codespace creation from an abusive actor that drained the available compute capacity in the affected regions faster than it could be replenished. <br /><br />We mitigated the impact by identifying and stopping the sources of the excess creation volume, reducing the resources that could be consumed in the affected regions, and rebalancing traffic across regions to restore capacity. Codespace creation success rates returned to normal after each period. <br /><br />We are working to add automated, low-latency controls to throttle abnormal codespace creation and to strengthen our detection and safeguards, so we can reduce our time to detection and mitigation of issues like this in the future.

    Resolved

  3. ConfluenceReliability issues with Jira and Confluence

    ## Summary On Jul 6, 2026, between 06:51 and 20:17 UTC, Atlassian customers using cloud products in European (EU) regions experienced service disruptions affecting automation rule execution, user search, user picker, and related workflows. The incident began when a core identity service experienced database saturation in EU regions, increasing latency and error rates. While responders worked to restore capacity, an emergency mitigation was applied which blocked automation rules from accessing the identity endpoint. In parallel, the elevated identity latency contributed to a cascading failure in a downstream user search service, degrading user search and user picker experiences across multiple products. Service was progressively restored after additional database read capacity was added, the emergency block was removed, and the user search service was manually scaled up. ## Impact The incident affected customers across multiple Atlassian products in EU regions on Jul 6, 2026 between 06:51 and 20:17 UTC. - **Core Identity service degradation:** elevated intermittent access denied rates were observed across products in EU regions. - **Automation rule execution failures:** automation rules using the default user (Automation for Jira) in Jira, Jira Service Management, and Jira Product Discovery in EU regions failed with a permissions error. - **User search and user picker failures:** user search success rates dropped significantly in EU regions, with user picker capability largely unavailable across products. ## Root Cause The incident originated from a recent configuration change that reduced the cache lifetime for a core identity service. As EU workday traffic ramped up on July 6, 2026, a larger share of requests began reaching the backing database directly rather than being served from cache. The EU databases did not have sufficient regional capacity to absorb the increased load, and database CPU reached saturation, causing elevated intermittent access denied rates. As part of the mitigation, an emergency block was applied to reduce load on the saturated identity service database and prioritize restoring the core identity service. This inadvertently prevented automation rules from passing their pre-execution permission checks, causing them to fail. The extended duration was primarily due to a secondary wave of automation rule failures caused by the processing limit throttling. In parallel, the sustained degradation caused increased latency and intermittent failures across products that depend on these checks, and contributed to a cascading failure in a downstream user search service. As a result, user search and user picker experiences became largely unavailable in EU regions until the permissions service recovered and the user search service was scaled up to restore capacity. ## Remedial Actions Plan & Next Steps We know outages impact customers' productivity. Atlassian is prioritizing the following actions to help prevent similar incidents in future: - **Improve safeguards for high-impact emergency mitigations:** Harden operational tooling so mitigation steps that could disable customer-facing workflows require additional review. - **Strengthen identity infrastructure capacity and change safety:** Review regional database sizing and capacity headroom to reduce the risk of saturation during peak traffic periods, and refine the assessment process for configuration changes so downstream load impact is evaluated before production rollout. - **Strengthen resilience against cascading failures from upstream degradation:** Audit backpressure handling, circuit breaker configuration, and autoscaling behavior in services, so that upstream degradation does not cause broader product impact across user search, user picker, and related experiences. - **Harden replay mechanisms for Automation rule failures:** Increase resilience of automation rules to reduce impact associated with incidents occurring in dependencies.. We recognize how critical reliable product workflows are for our customers, and we apologize to customers who were impacted by this incident. Thanks, Atlassian Customer Support

    Resolved

  4. JiraReliability issues with Jira and Confluence

    ## Summary On Jul 6, 2026, between 06:51 and 20:17 UTC, Atlassian customers using cloud products in European (EU) regions experienced service disruptions affecting automation rule execution, user search, user picker, and related workflows. The incident began when a core identity service experienced database saturation in EU regions, increasing latency and error rates. While responders worked to restore capacity, an emergency mitigation was applied which blocked automation rules from accessing the identity endpoint. In parallel, the elevated identity latency contributed to a cascading failure in a downstream user search service, degrading user search and user picker experiences across multiple products. Service was progressively restored after additional database read capacity was added, the emergency block was removed, and the user search service was manually scaled up. ## Impact The incident affected customers across multiple Atlassian products in EU regions on Jul 6, 2026 between 06:51 and 20:17 UTC. - **Core Identity service degradation:** elevated intermittent access denied rates were observed across products in EU regions. - **Automation rule execution failures:** automation rules using the default user (Automation for Jira) in Jira, Jira Service Management, and Jira Product Discovery in EU regions failed with a permissions error. - **User search and user picker failures:** user search success rates dropped significantly in EU regions, with user picker capability largely unavailable across products. ## Root Cause The incident originated from a recent configuration change that reduced the cache lifetime for a core identity service. As EU workday traffic ramped up on July 6, 2026, a larger share of requests began reaching the backing database directly rather than being served from cache. The EU databases did not have sufficient regional capacity to absorb the increased load, and database CPU reached saturation, causing elevated intermittent access denied rates. As part of the mitigation, an emergency block was applied to reduce load on the saturated identity service database and prioritize restoring the core identity service. This inadvertently prevented automation rules from passing their pre-execution permission checks, causing them to fail. The extended duration was primarily due to a secondary wave of automation rule failures caused by the processing limit throttling. In parallel, the sustained degradation caused increased latency and intermittent failures across products that depend on these checks, and contributed to a cascading failure in a downstream user search service. As a result, user search and user picker experiences became largely unavailable in EU regions until the permissions service recovered and the user search service was scaled up to restore capacity. ## Remedial Actions Plan & Next Steps We know outages impact customers' productivity. Atlassian is prioritizing the following actions to help prevent similar incidents in future: - **Improve safeguards for high-impact emergency mitigations:** Harden operational tooling so mitigation steps that could disable customer-facing workflows require additional review. - **Strengthen identity infrastructure capacity and change safety:** Review regional database sizing and capacity headroom to reduce the risk of saturation during peak traffic periods, and refine the assessment process for configuration changes so downstream load impact is evaluated before production rollout. - **Strengthen resilience against cascading failures from upstream degradation:** Audit backpressure handling, circuit breaker configuration, and autoscaling behavior in services, so that upstream degradation does not cause broader product impact across user search, user picker, and related experiences. - **Harden replay mechanisms for Automation rule failures:** Increase resilience of automation rules to reduce impact associated with incidents occurring in dependencies.. We recognize how critical reliable product workflows are for our customers, and we apologize to customers who were impacted by this incident. Thanks, Atlassian Customer Support

    Resolved

  5. TemporalCustomers in AWS eu-west-1 may experience elevated API latencies

    A spike in workload caused resource contention in the underlying infrastructure, resulting in elevated latency for critical APIs. We scaled up the underlying resources to handle the increased load, and latencies have returned to normal. We will continue to monitor the system, but the mitigation is complete and the incident has been resolved.

    Resolved

  6. VercelIncreased invocation failures for Hobby Team functions

    This incident has been resolved. A subset of deployments created under the Hobby plan teams during Jul 17, 2026 10:18 - Jul 17, 2026 14:10 UTC were impacted by this incident. New deployments after this window are unaffected.

    Resolved

  7. MixpanelTemporary Data Ingestion Delay for US Projects

    # Summary Between approximately **11:35 PM PT on July 10 and 7:19 AM PT on July 11, 2026**, data ingestion for Mixpanel projects in the US region ran behind by up to ~2 hours. During this window, reports and dashboards temporarily showed incomplete data — recent time ranges could appear as sharp, artificial drops in metrics such as active users or revenue. **No data was lost.** All events were queued durably and processed in full once the backlog cleared; metrics returned to accurate values on their own, with no customer action required. The cause originated in our ingestion controls: for a small number of very high-volume projects, the ingestion rates we allowed had drifted out of alignment with the capacity provisioned for those projects. Large historical imports — entirely legitimate use of the platform — were therefore admitted faster than their infrastructure could absorb, and two properties of our pipeline turned that localized overload into a region-wide delay. The fixes below realign those controls so that any import, of any size, is automatically kept within safe bounds. # What happened Mixpanel's ingestion pipeline is sharded and multi-tenant: each project's data is distributed across a set of partitions sized for its expected volume, and for throughput, events from many customers are processed together in batches. This design delivers high efficiency, but it depends on one invariant: the rate at which we admit a project's traffic must match the capacity provisioned for it. When that invariant holds, even very large imports are absorbed smoothly. Here, it did not hold. A small number of very large historical data imports ran on projects whose allowed ingestion rates had, over time, grown well beyond their provisioned partition capacity. The excess volume concentrated onto specific partitions as **hot spots**, saturating the portion of the streaming fleet serving them. Two factors then widened the impact: * **Batched, multi-tenant processing amplified the hot spots.** Because events from many customers travel together in batches, slowness and failures on the overloaded partitions delayed unrelated customers' events sharing those batches. * **Automatic scale-up was ineffective.** Adding capacity cannot dissolve a hot spot of this kind, because the overloaded partitions stay pinned to the same infrastructure, and a capacity limit in a component of our streaming infrastructure prevented the added capacity from taking effect. Together, these turned what should have been a brief, self-healing slowdown into a multi-hour delay requiring manual intervention. # Timeline \(Pacific Time, July 10–11\) * **11:35 PM** — Automated alerting detected the ingestion backlog; on-call engineer engaged immediately. * **12:49 AM** — Status page incident posted; impact scoped to the US region only. * **1:23 AM** — The largest contributing import was paused. * **1:55–6:48 AM** — Progressive mitigations: additional traffic sources throttled, certain backlogged data deferred with the owning customer's agreement, failure isolation enabled in the pipeline, and affected infrastructure nodes replaced. * **7:19 AM** — Backlog fully processed; all projects current. Status page moved to monitoring, then resolved after a stable observation period. # Root cause 1. **Ingestion rate limits misaligned with provisioned capacity.** For the driving projects, the rates our platform admitted had grown out of step with the infrastructure provisioned for them, so legitimate high-volume imports were let in faster than their partitions could absorb. This is the systemic root cause. The imports themselves were a legitimate use of the platform. 2. **Batched, multi-tenant processing amplified the overload.** Failures on the overloaded partitions delayed unrelated customers' events sharing the same processing batches, spreading a localized problem across the platform. 3. **Traffic to the overloaded partitions could not be redistributed.** Partition-to-server assignment in the streaming layer was not load-aware, so the hot spots stayed pinned to the same servers regardless of fleet size — total capacity was sufficient, but it could not be brought to bear. A scaling limit in a component of our streaming infrastructure compounded this by preventing scale-up from adding capacity, which delayed diagnosis until engineers intervened manually. # What we're changing The end state we are building toward: **every project's ingestion limits automatically match its provisioned capacity, so that imports of any size, including full historical backfills and data warehouse syncs, can run without advance coordination, and one project's volume is prevented from affecting another's data freshness.**. Already deployed — improving our ingestion handling: * **Improved hot-spot handling.** Traffic distribution in the streaming layer is now load-aware, spreading concentrated load more evenly across the fleet, and individual problem items are now retried separately instead of holding up the rest of their batch — reducing, though not eliminating, the impact a localized overload can have on unrelated traffic. * **Re-provisioned the highest-volume workloads** onto appropriately sized infrastructure, prioritized by risk. Already deployed — changing how we operate the third-party streaming service: * **Audited and re-tuned the fleet's capacity profile** so individual servers have substantially more headroom for concentrated load, and worked with the provider to resolve the scaling limitation encountered during the incident. In progress: * **Capacity-aware rate limiting** — closing the gaps between individually granted project rate limits and each project's real provisioned capacity, extending rate limiting to ingestion paths that previously lacked it, and coupling any future limit increase to a capacity increase, which is designed to prevent this class of misalignment from recurring.. * **Finer-grained volume monitoring and alerting** so capacity misalignment is detected and corrected before it can affect any customer. * Evaluating stronger workload isolation/backlog recovery prioritization for bulk/backfill traffic paths, so historical imports have reduced impact on live traffic # Common questions * **Was any data lost?** No. Events were durably queued throughout the incident and were fully processed once the backlog cleared. Any metric drops seen during the window were a display artifact of the delay and self-corrected. * **Do I need to coordinate large imports or backfills with Mixpanel?** No — Our goal is for the platform to keep any import within safe rates automatically, so that backfills and warehouse syncs can run without scheduling or notice. We are still rolling out the capacity-aware controls that deliver this. In the meantime, if you are planning an unusually large import or backfill, we recommend coordinating with your account team so we can confirm capacity in advance. * **How is this prevented going forward?** The systemic fix is tightening the gaps between individually granted project rate limits and each project's real provisioned capacity, and extending rate limiting to the ingestion paths that previously lacked it — so overload of this kind is stopped at admission. In addition, the scaling limitation that prolonged the incident is fixed, the pipeline now retries individual problem items separately so a localized overload has far less impact on unrelated traffic, and bulk traffic is being further isolated from live traffic. * **Can individual projects be prioritized during recovery?** This capability did not exist during the incident — all projects recovered at the same rate. We are evaluating prioritization mechanisms for backlog recovery as part of our follow-up work. We apologize for the disruption and for the concern the temporarily depressed metrics caused. Please reach out through your account team or support with any questions.

    Resolved

  8. New RelicE2M (Events-2-Metrics) authorization issues

    Between 15:00 UTC on July 14th and 10:00 UTC on July 17th, we experienced an issue where some customers experienced authorization issues when attempting to create, read or update E2M (Events-2-Metrics) rules in the US, EU and JP regions. This was due to a recent change we have made to our authorization endpoints. These endpoints now require S2S authentication which has caused requests to our E2M (Events-2-Metrics) to fail. Our teams have rolled back these changes until a permanent fix can be put in place. The issue is now resolved, and the impacted services have returned to normal service.

    Resolved

  9. HootsuiteInbox 2.0 - Conversation History Loading Issues

    We experienced an issue affecting Inbox 2.0 conversation history loading between 3:00am and 3:57am PST. This has since been fully resolved. If you continue to experience any problems, please reach out to our support team.

    Resolved

  10. GitHubDegraded REST API Availability

    From 22:21 UTC - 23:50 UTC on July 16, 2026, the REST API experienced significant degradation. During this period, about 39% of REST API requests failed with HTTP 500 level responses, with the errors peaking at 44.3%. <br /><br />We identified the issue as an infrastructure change that wrongly marked the majority of API backends in a single region as unhealthy. As a result, requests routed to those backends failed before reaching the application layer. <br /><br />To prevent this from happening again, we're improving our systems to catch this kind of invalid configuration before it reaches production. We'll also audit the related systems to make them more resilient to future changes, and we're increasing our monitoring sensitivity so we're alerted to problems like this sooner.

    Resolved

  11. CircleCIErrors accessing CircleCI and delays starting pipelines and scheduled workflows

    An incident affecting GitHub's API caused CircleCI customers to experience errors loading the web app and signing in, along with pipelines and scheduled workflows failing to start or running late. GitHub has resolved the underlying incident (https://www.githubstatus.com/) and all affected functionality has returned to normal. Customers whose jobs or pipelines failed may rerun them. Scheduled pipelines and workflows that were missed during the incident will not run automatically and will need to be re-triggered. We thank you for your patience.

    Resolved

  12. AssemblyAIDegraded streaming session creation for Universal Streaming English in us-west

    This incident affected newly created Universal Streaming English sessions in the us-west region on July 7 between 15:00 and 15:10 UTC. Affected sessions were cancelled with error code 3005 shortly after connecting and did not return transcripts. Sessions already in progress, other regions, and other streaming models were unaffected. Service returned to normal at 15:10 UTC. We apologize for the disruption to your workflows. If you have any questions, please reach out to support@assemblyai.com.

    Resolved

Jul 16, 2026

  1. MintlifyWe are seeing upstream GitHub API degradation

    This incident is resolved. GitHub-connected editor sessions, repository actions, and deployments are operating normally. If you continue to experience issues, please contact support@mintlify.com "support@mintlify.com": mailto:support@mintlify.com.

    Resolved

  2. HootsuiteReceiving Push Notifications and Composer loading difficulties

    Our push notification provider experienced a service disruption that impacted Hootsuite between approximately 2:30 PM PST and 3:05 PM PST on July 16, 2026. During this time, some push notifications may not have been delivered and Composer may have experienced slower load times. The issue has since been resolved. If you continue to experience issues with push notifications or Composer loading, please reach out to our customer support team.

    Resolved

  3. InngestMetrics Data Delays

    Metrics in the UI are now restored. Users may notice metrics data continue to be unavailable during the time of the incident, from the timeframe 7/16/26 15:45 UTC until approximately 7/16/26 21:07 UTC

    Resolved

Older incidents