Significant delays in asset generation

Resolved

Major impact

Started

Lasted 1h 15m

View official incident ↗

Updates

  1. Resolved

    The issue has been fully resolved and all systems are back to normal operation. We apologize for the disruption.

  2. Monitoring

    All systems have recovered and are processing normally. Gen4 Turbo is experiencing elevated queue wait times due to backlog, but this should clear soon.

  3. Monitoring

    We've identified the root cause of the generation delays and have rolled out a fix. Generations should now be processing normally. We'll continue monitoring to ensure stability.

  4. Investigating

    We're continuing to investigate the issue. A significant number of generations are currently failing to process.

  5. Investigating

    We're aware that some users are experiencing significant delays in generations and are actively investigating the issue. We'll provide updates as we learn more.

  6. Investigating

    Our engineering team is investigating a problem affecting users. We're aware of the problem and are working towards a solution. We will provide an update soon.