Resolved incident

Global Outage

Black Forest Labs · Critical · Started

Duration
42h 23m
Severity
Critical
Status
Resolved

Summary

Critical · Resolved

Our third party had resolved their issue be we kept monitoring closely, we trust the stability now.

View official incident ↗

Event timeline

How this incident unfolded, from the official status page.

  1. Resolved

    Black Forest Labs

    Our third party had resolved their issue be we kept monitoring closely, we trust the stability now.

  2. Identified

    Black Forest Labs

    Apologies, false status update

  3. Monitoring

    Black Forest Labs

    A fix has been implemented. Please use the global endpoint: https://api.bfl.ai. Regional endpoints are also now operational: `api.eu.bfl.ai`, and `api.us.bfl.ai` As part of our cloud-outage mitigation, we’ve temporarily moved delivery to the following Azure Blob endpoints. Please whitelist them: - *.blob.core.windows.net We intend to switch back to our standard domains (delivery.*.bfl.ai and delivery-*.bfl.ai) as soon as it’s viable. We recommend keeping those domains whitelisted as well, if they aren’t already. Note: there’s a difference between delivery.*.bfl.ai and delivery-*.bfl.ai. Please ensure both patterns are allowed.

  4. Monitoring

    Black Forest Labs

    A fix has been implemented. Please use the global endpoint: https://api.bfl.ai. Regional endpoints are also now operational: `api.eu.bfl.ai`, and `api.us.bfl.ai`

  5. Monitoring

    Black Forest Labs

    A fix has been implemented. Please use the global endpoint: https://api.bfl.ai. DNS cache needs to be flushed. Regional endpoints: api.us.bfl.ai and api.eu.bfl.ai are not functional at the moment.

  6. Identified

    Black Forest Labs

    We have implemented a mitigation to the problem. Please use regional endpoints for now: `https://api.eu2.bfl.ai` and `https://api.eu4.bfl.ai/`

  7. Identified

    Black Forest Labs

    Azure is currently experience issues resulting in a full outage in our image generation services. We are awaiting a status update from them.