Scout APMDatabase maintenance
Some metrics are still delayed, but backfill is proceeding.
Resolved
Outages and recoveries from the services you depend on.
40 incidents
Some metrics are still delayed, but backfill is proceeding.
Resolved
The incident is resolved.
Resolved
This incident has been resolved.
Resolved
We had a brief issue this morning (6/25/2025 from 7:25 AM to 9:40 AM PT) affecting the Qwen-3-32B shared endpoint. Some users may have seen 404 or 503 errors during that time.
Resolved
The issue has been fully resolved. All affected components are operating normally again.
Resolved
This incident has been resolved.
Resolved
Between 07:40 UTC and 10:31 UTC, we experienced issues affecting user syncing in Atassian Administration. This affected Confluence, Jira Work Management, Jira Service Management, Jira, Trello, and Guard. The issue has been resolved and the service is operating normally.
Resolved
Between 07:40 UTC and 10:31 UTC, we experienced issues affecting user syncing in Atassian Administration. This affected Confluence, Jira Work Management, Jira Service Management, Jira, Trello, and Guard. The issue has been resolved and the service is operating normally.
Resolved
Between 06:35 to 9:27 UTC, we experienced degraded performance for Atlassian Bitbucket. The issue has been resolved and the service is operating normally.
Resolved
The issue has been fully resolved. All affected components are operating normally again.
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
We've validated our infrastructure changes and are marking the incident as resolved.
Resolved
Between 16:31 UTC and 19:48 UTC, we experienced an issue due to which some POS transactions could not be completed as expected. A fix was implemented at 17:44 UTC which resolved the issue. The monitoring period for this issue has ended and no further impact has been observed. Merchants still experiencing issues can try disconnecting, forgetting, and reconnecting their card readers to resolve errors.
Resolved
At 8:15 AM PST, our DevOps team was alerted to elevated API error rates in production. Upon investigation, we identified the following: * The production database was experiencing high table lock contention. * This was triggered by several large-scale BreachWatch data updates. * A recent backend migration had unintentionally doubled the number of parallel update operations, which exacerbated the issue. * A database failover was performed to restore service stability. The issue was fully resolved by 8:40 AM PST. The engineering team has addressed the underlying problem in the update processes and is implementing safeguards to prevent recurrence, including query optimization and improved job orchestration. We apologize for the disruption.
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
We have resolved an issue that affected attachment loading during C2C and Sandbox data copy migrations from 19-06-2025 18:00 UTC to 24-06-2025 08:49 UTC. If you encountered this issue, please delete the affected projects from the destination site and re-run the migration. For ongoing migrations, allow them to complete and then follow the same steps. For Sandbox FDC migrations, complete them as usual and re-run if needed. We appreciate your patience and cooperation.
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
Refactored projects should be fully functional
Resolved
This incident has been resolved.
Resolved
The issue has been fixed, and systems are behaving normally.
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
We have resolved the stability issue on our mobile app, and performance is back to normal. We appreciate your patience.
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
The issue has been fully resolved. All affected components are operating normally again.
Resolved
# Incident Summary Between **17:55 and 20:22 UTC** on **June 12, 2025**, Imgix services experienced major service disruptions across several key interfaces: * **Dashboard and Asset Manager**: These interfaces were inaccessible, preventing users from managing their assets or viewing account information. * **Management API**: Requests to the Management API consistently returned errors, affecting workflows reliant on programmatic updates or asset administration. * **Rendering API:** Approximately 8% of all Imgix service requests failed due to a high error rate \(~80%\) for **uncached** assets via the Rendering API. Requests in the EU saw a lower failure rate \(~50%\) and a faster recovery time \(45 minutes\) for these uncached requests. # What caused it The incident was triggered by a global outage within Google Cloud, which serves as a core infrastructure provider for Imgix. The outage affected most services in all regions simultaneously. You can read more about the [Google Cloud outage here](https://status.cloud.google.com/incidents/ow5i3PPK96RduMcb1SsW#RP1d9aZLNFZEJmTBk8e1). # What happened * **17:55 UTC:** Internal alerts triggered due to a spike in rendering errors and service timeouts. * **18:01 UTC:** A short investigation uncovers several timeouts and increased error rates from Google Cloud. * **18:09 UTC:** Our status page is updated. * **18:53 UTC:** A major Google Cloud outage is confirmed, after which we update our status page. * **18:00–20:16 UTC:** Mitigation efforts were hampered by the far-reaching effects of the outage, preventing us from redirecting traffic or applying configuration changes. * **Throughout:** We confirm that cached images were not affected, though the downtime of several data sources prevents evaluating the full scope and effect of the outage. * **20:16 UTC:** Google reported recovery in all regions except `us-central1`. This allowed us to verify **significantly lower error rates for EU traffic** * **20:47 UTC:** Imgix systems achieved full recovery. * **20:52 UTC:** The incident was officially resolved on our status page. * **21:23 UTC:** Google confirms a full-service recovery at 21:23 UTC. # What went wrong * Google Cloud experienced an outage that simultaneously affected nearly every service in every region worldwide, negating our multi-region redundancy for the image rendering service. * 3rd party services \(such as our CDN\) were also affected by the outage, which removed some of our options for redirecting traffic across regions based on performance. * The outage included the control panes that Google provides to its customers, which removed additional options for redirecting traffic and implementing mitigations. # What we will do to prevent this in the future * Continuing our ongoing internal discussions and evaluations of a multi-cloud render stack to enable failover in the event of a provider-wide outages. * Continue evaluating and improving tools to automatically and manually shift traffic as necessary at each layer of the stack. * Review and enhance incident communication protocols, focusing on faster root cause disclosure and update frequency.
Resolved
On June 12, from approximately 2:00 PM to 3:15 PM EDT, a widespread outage at our cloud provider caused most Vimeo services to become temporarily unavailable. During this time, Vimeo users received generic error messages when attempting to access core aspects of the service, including upload, playback, transcoding, API, analytics, and live streaming. Vimeo service was restored when our cloud provider resolved the issue, although residual effects continued until 4:30 PM EDT while our systems worked through large backlogs in the transcoding pipeline. As a result of this outage, our engineering leadership is reviewing our plan for multilevel cloud failure to improve recovery time in the future.
Resolved
On June 7, between approximately 1:00 AM and 6:00 AM EDT, Vimeo was inaccessible to some logged-in users due to failing requests to a crucial service. API requests might have returned HTTP status code 500 during this time. We determined that the characteristics of a critical database significantly delayed requests to the service and ultimately overwhelmed the Vimeo application. Once we scaled up the affected database, logged-in access was restored.
Resolved
Some Llama 3.3-70B requests failed due to 5xx errors. We restored normal service performance at 3:54 PM EST on June 19th.
Resolved
Llama3.1-8B was unavailable. We restored normal service performance at 3:21 PM EST on June 19th.
Resolved
This incident has been resolved.
Resolved
This incident has been resolved. If you are still experiencing issues please contact our support at support@upcloud.com.
Resolved
This incident has been resolved.
Resolved
On June 12, 2025, between 10:51 AM and 1:36 PM PT, Pillar experienced a service outage due to a widespread disruption within Google Cloud Platform \(GCP\), our cloud infrastructure provider. During this time, customers were unable to access the Pillar web application \([app.pillar.hr](http://app.pillar.hr)\), and Pillar integrations within Jobvite and Lever were also unavailable. The issue was caused by a global outage affecting GCP services, which directly impacted the infrastructure Pillar relies on to remain accessible. While our team closely monitored the situation, resolution was dependent on GCP restoring its services. Once the platform-wide issue was resolved, Pillar services recovered automatically. Our team also proactively reviewed the impacted period and worked to recover and restore any affected interviews where possible. However, due to the nature of the outage, we were unable to capture data from some interviews that were scheduled during the downtime. To improve our response to future incidents, we are integrating Pillar monitoring and alerting into Employ’s centralized Incident Response system and evaluating additional strategies to improve platform resilience. Moving forward, customers can view live system availability and updates anytime at [https://status.pillar.hr/](https://status.pillar.hr/). We sincerely apologize for the disruption and appreciate your continued trust in us. If you have any questions, please don’t hesitate to reach out.
Resolved
Dear Valued Customer, We recently experienced inbound call failures affecting users in the India region during the time window specified below. The issue has since been resolved, and all services are currently operating normally. Incident Start Time: 19th June 2025 11:45 UTC Incident End Time: 19th June 2025 12:50 UTC If you have any questions or concerns regarding this incident, please contact Plivo Support. Support portal: https://support.plivo.com/hc/en-us
Resolved
Monitoring has shown that the implemented fix has sustained mitigation of the issue. Incident is now resolved.
Resolved