Project Actions Failing Across Multiple Regions

Updates

1. Resolved

   4 months ago

   Project actions have been sustained at normal performance. This incident is resolved.

2. Monitoring

   4 months ago

   The mitigation has returned project activity to normal performance. Project actions that failed during the outage are being requeued to run. We will continue to monitor to ensure project functionality.

3. Identified

   4 months ago

   We have identified a problem with a specific job in us-east-1, we have mitigated the job and are observing the results.

4. Investigating

   4 months ago

   We are testing mitigation options to attempt to restore project action performance.

5. Investigating

   4 months ago

   We have determined that there are four main project actions impacted in the identified regions: Pause/unpause failures, Restore/PITR failures, PITR configuration failures and Backup problems. We are pursuing root cause and mitigation options in parallel.

6. Investigating

   4 months ago

   Project actions in us-east-1, us-east-2, sa-east-1, ca-central-1 are experiencing significant increases in error rates. We are exploring the best options for restoring performance.

7. Investigating

   4 months ago

   Certain project actions are failing. We are actively investigating to determine the regions impacted and source of the issue.