Live outage feed

Outages and recoveries from the services you depend on.

Earlier incidents

40 incidents

Back to latest

Mar 13, 2024

  1. UpstashMaintenance - Global Ap-Southeast-1

    We have taken actions to increase the capacity of the region. During the operation, clients might have experienced higher than usual latencies for about 15 minutes.

    Resolved

Mar 12, 2024

Mar 11, 2024

Mar 8, 2024

  1. imgixIssues with imgix web tools

    # Postmortem # **What happened?** On Feb 20, 2024, at 19:00 UTC, Uploads using the \`/api/v1/sources/upload/\` API endpoint experienced slowness and some timeouts during this time.   The issue had been completely resolved by Feb 21, 2024, 20:00 UTC. # **How were customers impacted?** Between Feb 20, 2024, 19:00 UTC, and Feb 21, 2024, 20:00 UTC, some requests to our upload API experienced slow responses, and a subset of requests resulted in timeout errors. Uploads using the \`/api/v1/sources/<source\_id>/upload-sessions/\` endpoints were unaffected; uploads using the Asset Manager UI were unaffected. # **What went wrong during the incident?** Two compounding issues caused the slowdown. The first is a service update creating an issue with our /api/v1/sources/upload/\` API endpoint.  Secondly, a simultaneous and separate slowdown affected the cloud servers responsible for executing the upload actions.  The service update issue increased the load on our upload function, which was already under strain due to the cloud servers experiencing a slowdown. These factors combined to cause the slow and timed-out responses. # **What will imgix do to prevent this in the future?** We have streamlined the upload process so that this cannot happen again.

    Resolved

  2. AppsFlyerDelay in reports data availability

    Data recovery has been completed and the issue is now resolved. If you encounter any further related issues, please don't hesitate to contact us. Thank you for your patience and understanding.

    Resolved

  3. TypeformTypeform Contact Us Form - Login Redirect not working

    We are glad to inform you that the fix has been implemented and the incident resolved. We will run a problem investigation to understand what happened and how to prevent it from happening again. Thanks for your understanding & sorry for any inconvenience this has caused.

    Resolved

Mar 7, 2024

  1. AirtableIncreased error rates

    We have resolved this incident, and Airtable’s services are functioning normally. We’re sorry for the inconvenience to our customers’ operations.

    Resolved

Mar 6, 2024

  1. PandaDocIncident notification

    We're all set! PandaDoc is working properly and the incident has been fully resolved. If you continue to experience any issues with this, please reach out to us at support@pandadoc.com.

    Resolved

  2. AirtableIncident - 2024-03-05

    The major ISP's incident that was preventing some customers from reaching Airtable has been resolved. Thank you for your patience!

    Resolved

Mar 5, 2024

  1. Hotjar'Sign in with Google' is unavailable

    While Google is yet to share any public update on this incident, our team is now able to sign in to Hotjar using the 'Sign in with Google' option. Thanks for bearing with us while this was happening!

    Resolved

Mar 4, 2024

Mar 1, 2024

Feb 29, 2024

  1. OpsgenieInvestigating new product purchasing

    Between 28th Feb 2024 23:15 UTC to 29th Feb 2024 00:05 UTC, we experienced issue with new product purchasing for all products. All new sign up products have been successfully provision and confirmed issue has been resolved and the service is operating normally.

    Resolved

  2. TrelloInvestigating new product purchasing

    Between 28th Feb 2024 23:15 UTC to 29th Feb 2024 00:05 UTC, we experienced issue with new product purchasing for all products. All new sign up products have been successfully provision and confirmed issue has been resolved and the service is operating normally.

    Resolved

Feb 28, 2024

  1. FigmaService Disruption

    Figma users experienced intermittent connectivity issues for 12 minutes. We have since resolved the underlying issue and are monitoring closely.

    Resolved

Feb 27, 2024

  1. TrelloService Disruptions Affecting Atlassian Products

    ### **Summary** On February 14, 2024, between 20:05 UTC and 23:03 UTC, Atlassian customers on the following cloud products encountered a service disruption: Access, Atlas, Atlassian Analytics, Bitbucket, Compass, Confluence, Ecosystem apps, Jira Service Management, Jira Software, Jira Work Management, Jira Product Discovery, Opsgenie, StatusPage, and Trello. As part of a security and compliance uplift, we had scheduled the deletion of unused and legacy domain names used for internal service-to-service connections. Active domain names were incorrectly deleted during this event. This impacted all cloud customers across all regions. The issue was identified and resolved through the rollback of the faulty deployment to restore the domain names and Atlassian systems to a stable state. The time to resolution was two hours and 58 minutes. ### **IMPACT** External customers started reporting issues with Atlassian cloud products at 20:52 UTC. The impact of the failed change led to performance degradation or in some cases, complete service disruption. Symptoms experienced by end-users were unsuccessful page loads and/or failed interactions with our cloud products. ### **ROOT CAUSE** As part of a security and compliance uplift, we had scheduled the deletion of unused and legacy domain names that were being used for internal service-to-service connections. Active domain names were incorrectly deleted during this operation. ### **REMEDIAL ACTIONS PLAN & NEXT STEPS** We know that outages impact your productivity. The detection was delayed because existing testing & monitoring focused on service health rather than the entire system’s availability. To prevent a recurrence of this type of incident, we are implementing the following improvement measures: * Canary checks to monitor the entire system availability. * Faster rollback procedures for this type of service impact. * Stricter change control procedures for infrastructure modifications. * Migration of all DNS records to centralised management and stricter access controls on modification to DNS records. We apologize to customers whose services were impacted during this incident; we are taking immediate steps to improve the platform’s performance and availability. ‌ Thanks, Atlassian Customer Support

    Resolved

  2. OpsgenieService Disruptions Affecting Atlassian Products

    ### **Summary** On February 14, 2024, between 20:05 UTC and 23:03 UTC, Atlassian customers on the following cloud products encountered a service disruption: Access, Atlas, Atlassian Analytics, Bitbucket, Compass, Confluence, Ecosystem apps, Jira Service Management, Jira Software, Jira Work Management, Jira Product Discovery, Opsgenie, StatusPage, and Trello. As part of a security and compliance uplift, we had scheduled the deletion of unused and legacy domain names used for internal service-to-service connections. Active domain names were incorrectly deleted during this event. This impacted all cloud customers across all regions. The issue was identified and resolved through the rollback of the faulty deployment to restore the domain names and Atlassian systems to a stable state. The time to resolution was two hours and 58 minutes. ### **IMPACT** External customers started reporting issues with Atlassian cloud products at 20:52 UTC. The impact of the failed change led to performance degradation or in some cases, complete service disruption. Symptoms experienced by end-users were unsuccessful page loads and/or failed interactions with our cloud products. ### **ROOT CAUSE** As part of a security and compliance uplift, we had scheduled the deletion of unused and legacy domain names that were being used for internal service-to-service connections. Active domain names were incorrectly deleted during this operation. ### **REMEDIAL ACTIONS PLAN & NEXT STEPS** We know that outages impact your productivity. The detection was delayed because existing testing & monitoring focused on service health rather than the entire system’s availability. To prevent a recurrence of this type of incident, we are implementing the following improvement measures: * Canary checks to monitor the entire system availability. * Faster rollback procedures for this type of service impact. * Stricter change control procedures for infrastructure modifications. * Migration of all DNS records to centralised management and stricter access controls on modification to DNS records. We apologize to customers whose services were impacted during this incident; we are taking immediate steps to improve the platform’s performance and availability. ‌ Thanks, Atlassian Customer Support

    Resolved

Feb 26, 2024

  1. imgixEncoding Error for Web folder

    # Postmortem # **What happened?** On January 22, 2024, 07:38 UTC, some Web Folder requests to the imgix service began to return a `403` response.  By 11:56 UTC, the issue had been completely resolved. # **How were customers impacted?** Between 07:38 UTC and 11:36 UTC, several requests to web folder sources began to return a `403` error. This affected a small amount of assets \(<0.1%\).  # **What went wrong during the incident?** A service update caused an issue with web folders with directories using double slashes at the Origin URL \(`//`\). This update caused an encoding error, leading to `403` responses for Origins matching the double slash pattern. A fix was pushed to resolve this URL pattern, allowing us to fetch images from affected Origins. # **What will imgix do to prevent this in the future?** We will update our tests to catch encoding issues in the future.

    Resolved

Feb 25, 2024

Feb 24, 2024

Feb 23, 2024

  1. TrelloTrello is slow or unavailable

    ### Summary On Feb. 13, 2024, between 8:00 AM and 11:34 AM UTC, Trello experienced severely degraded performance appearing as a full or partial outage to Atlassian customers. The event was triggered by a buildup of long-running queries against our database, leading to slowed API response times and causing Trello to be degraded or unavailable for users. The root cause of the incident was identified as a compression change in our database deployed approximately 11 hours earlier during a low-traffic period. As European customers came online, traffic started increasing, resulting in a buildup of queries and the subsequent incident. The incident was detected by our monitoring system at 8:07 AM UTC and was mitigated by reverting the compression change and restarting components of our database system. The total time to resolution was 3 hours and 34 minutes. ### **IMPACT** The overall impact was between 8:00 AM and 11:34 AM UTC on Feb. 13, 2024. The incident caused Trello to be fully or partially unavailable for customers using or attempting to access the site during this period. ### **ROOT CAUSE** The issue was caused by a compression change in our database, which resulted in the build up of queries in the system. This build up then caused API response times to increase to critical levels. During the incident many users received HTTP 429 errors as the system began rate-limiting in an attempt to recover. Users that did not receive errors experienced API response times 10-100x slower than our standard response times. ### **REMEDIAL ACTIONS PLAN & NEXT STEPS** We know that outages impact your productivity. We are prioritizing the following actions to avoid repeating this incident and reduce time to resolution: * Improve our process for releasing incremental configuration changes which would have allowed the team to identify the root cause before a peak load period and prevent similar incidents. * Adjust priority level of alerts related to this class of incident to improve signal to noise ratio and drive faster time to resolution. We apologize to customers whose services were impacted during this incident; we are taking immediate steps to improve Trello’s performance and availability. Thanks, Atlassian Customer Support

    Resolved

  2. TypeformTypeform Builder - Degraded - Builder not loading

    We are glad to inform you that the fix has been implemented and the incident resolved. If users still experience any issues, they should try refreshing/clearing the cache in their browser. We will run a problem investigation to understand what happened and how to prevent it from happening again. Thanks for your understanding & sorry for any inconvenience this has caused.

    Resolved

Feb 22, 2024

Older incidents