Resolved incident

Jobs stuck in pending state

Anyscale · Major · Started

Duration
5h 56m
Severity
Major
Status
Resolved

Summary

Major · Resolved

This issue has now been resolved.

View official incident ↗

Event timeline

How this incident unfolded, from the official status page.

  1. Investigating

    Anyscale

    We are currently experiencing downtime on Anyscale Platform, UI and SDK/CLI. Submitted jobs are stuck in pending state. We are investing the issue.

  2. Monitoring

    Anyscale

    Anyscale Platform, UI and SDK/CLI have started to recover. At this point, the root cause appears to be related to an outage with AWS STS affecting the Anyscale Control Plane. The AWS outage appears to be mitigated at this time.

  3. Resolved

    Anyscale

    Downtime on Platform, UI and SDK/CLI is now resolved. Thank you for your patience as we worked to resolve the issue

  4. Investigating

    Anyscale

    We are currently investigating this issue impacting the Anyscale platform.

  5. Investigating

    Anyscale

    Update: Team is continuing to investigate. Anyscale Jobs continue to be in status "Pending" for 5-10 minutes before being reconciled.

  6. Monitoring

    Anyscale

    A fix has been identified and implemented. We are observing a reduction in the job reconciliation backlog to normal levels, and customer jobs are now being submitted as expected.

  7. Resolved

    Anyscale

    This issue has now been resolved.