PyPIIssue with uploads
Uploads are now being accepted properly.
Resolved
Outages and recoveries from the services you depend on.
40 incidents
Uploads are now being accepted properly.
Resolved
9th August 12:57 UTC We have become aware of an issue with our Ably Reactor Queues in eu-west-1 and are investigating. 9th August 13:30 UTC We have identified the issue and rectified it. The coordinator node for our eu-west-1 Reactor Queue cluster became unresponsive. This meant that the eu-west-1 endpoint - https://eu-west-1-a-queue.ably.io:15671 - was also unresponsive. 9th August 13:36 UTC We are monitoring the cluster to ensure it is working as expected. 9th Aug 13:44 UTC We can confirm that Ably Reactor Queues in eu-west-1 are now operational.
Resolved
This incident has been resolved.
Resolved
We have mitigated this issue, and Airtable's service is now operating normally. We deeply apologize for the inconvenience.
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
Dashboard and documentation tools were affected by an upstream service provider outage. Rendering services were not impacted.
Resolved
Our engineers have rolled out a fix for the issue. This issue is now resolved.
Resolved
We have resolved the stability issues with the logging in with Facebook feature on our site and performance is back to normal. We appreciate your patience.
Resolved
Between 2024-08-07 5.20am UTC to 2024-08-07 5.40am UTC, we experienced degraded performance with Bitbucket Website. The issue has been resolved and the service is operating normally.
Resolved
The fix has rolled out completely, and things are back to normal.
Resolved
We are glad to inform you that the fix has been implemented and the incident resolved. We will run a problem investigation to understand what happened and how to prevent it from happening again. Thanks for your understanding & sorry for any inconvenience this has caused.
Resolved
This incident has been resolved.
Resolved
**Incident:** On 2024-08-01 at 14:09 UTC, an increased connection rate on a central database saturated the database’s ability to accept new connections. As a result, Asana became unavailable to users in all regions. In response, Asana engineers throttled traffic for automations, webhooks, and API, and were able to restore primary functionality of the Asana application at 15:01 UTC. The application continued to run with sporadic unavailability until 18:22 UTC, when the underlying cause of the increased connections was determined and disabled. At this point the application was fully operational, but asynchronous workloads such as automations and notifications were delayed due to the work enqueued while traffic was throttled. The incident was resolved when asynchronous work queues fully caught up at 19:39 UTC. **Impact:** From 14:09 UTC to 15:01 UTC, users were unable to access Asana through the web, mobile, or API. From 15:01 UTC to 18:22 UTC users could access Asana through the web and mobile, but experienced sporadic unavailability. Access to the API was restored gradually between 15:15 UTC and 15:58 UTC. From 15:01 UTC to 19:39 UTC users experienced delayed automations, background actions, webhooks, and event streams. No customer data was lost. **Moving forward:** We have reverted the change that caused the increased connections. We are adding monitoring to detect changes in database connection rates, so we can identify potential problems earlier. Beyond this, we will be replicating data from the central database to other databases to reduce the impact of any single database being overloaded.
Resolved
From 22:10Z through 22:32Z on 2024-07-25 Bitly experienced a brief outage. The root cause was a failure during a planned host migration which temporarily impacted internal service discovery.
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
The Vector Tiles API error rates have subsided and the service is fully operational.
Resolved
This incident has been resolved.
Resolved
The service is back to normal.
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
We identified an upstream problem with our email service provider, which has been resolved.
Resolved
We are glad to inform you that the fix has been implemented and the incident is resolved. Respondent and Self Follow-Up emails were delayed temporarily and these messages have now been delivered. New Follow-Up messages are being delivered as expected. We will run a problem investigation to understand what happened and how to prevent it from happening again. Thanks for your understanding & sorry for any inconvenience this has caused.
Resolved
We have identified an issue in both EU and USA RT SaaS which impacted new sessions which were passing `punctuation_overrides` in the transcription config. The issue is now resolved. We apologise for any inconvenience caused. If you are still facing issues with the service, please reach out to our support team here https://support.speechmatics.com/hc/en-gb.
Resolved
This incident has been resolved.
Resolved
Navigation api requests (Directions, Map-matching) in hitting our `ap-southeast-1` datacenter experienced stale or missing live-data and congestion for 90 minutes. Incident has been resolved
Resolved
This incident has been resolved.
Resolved
On 2024-07-26, a deployment of new code introduced a bug which caused delays for automations and background actions for a subset of our customers. Jobs were delayed between 00:55 and 12:30 UTC. We reverted this deployment, which allowed our system to catch up. Impact: During the incident, a subset of customers experienced delayed background actions, such as rules, email notifications, imports, and exports. For a small fraction of customers, during a 7-hour period, automations may not have executed as intended. Moving forward: We have identified the root cause of this particular bug and will improve our systems and processes to prevent it from happening in the future.
Resolved
The incident has been resolved and connections are working as expected.
Resolved
This incident has been resolved.
Resolved
We're investigating issues in the us-east-2 (Ohio) region. All other regions are operating normally, so service should be unaffected, other than slightly higher than normal latency if you would normally be routed to the Ohio region. 28th Jul 05:20 PM AWS are reporting connectivity issues in us-east-2, see https://health.aws.amazon.com/health/status We have disabled the region at the dns level in all cluster in which it operates, so traffic that would have been going to Ohio will now be going to the next closest region (such as us-east-1 East Virginia). The service should still be operating normally; please contact Ably support in the usual way if you do experience any problems. 28th Jul 05:56 PM We are continuing to monitor AWS reports on the incident and will re-enable us-east-2 when we are confident that that region is stable. Until then, the Ably service will continue to operate as normal with traffic routed through the five other regions in which our production cluster operates. 29th Jul 09:34 AM The us-east-2 region has now been re-enabled in all clusters which used in that region.
Resolved
We are investigating elevated error rates across multiple regions for a period of 8 minutes (10:36 - 10:44 UTC). Service appears to be operating normally now. 20th Oct 11:20 AM Investigations show there was a transient netsplit between us-west-1 and other regions, leading to increased error rates and high latencies for the duration of the netsplit for customers in the us-west-1 area. Customers may have been routed to other regions.
Resolved
Customers using our presence feature may have seen sporadic spurious leave + immediate re-enter events for presence members over the last two days, since Tuesday. We have rolled back to a version of the service which does not have the bug, and we apologise for any inconvenience.
Resolved
External monitors are reporting reachability issues for US-East-2 Ohio endpoints. All other data centers are operating normally, and service is unaffected. For users who would normally reach us-east-2, SDKs will automatically retry any timed out requests at a fallback data center, which may result in increased latency. 26th Jan 01:09 AM The external monitor ceased to report any issues with us-east-2 since 00:40 UTC. No other data source or monitor indicated any problem, so we are considering the possiblity that the problem was with the external monitoring system. However, we are still investigating and have not yet made a final determination.
Resolved