Security Reports not loading for some usersDegraded Performance

Resolved

Started

Lasted 18h 11m

View official incident ↗

Updates

  1. Resolved

    We observed that the backlog of jobs have been processed and new reports should be coming in on the normal schedule. With this update the incident is resolved. More info: gitlab.com/gitlab-com/gl-infra/production/-/work_items/23024

  2. Monitoring

    No material updates to report. We are continuing to monitor the report-ingestion backlog and working on mitigation. More info: gitlab.com/gitlab-com/gl-infra/production/-/work_items/23024

  3. Monitoring

    No material updates to report. We are continuing to monitor the report-ingestion backlog and working on mitigation. More info: https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/23024

  4. Monitoring

    Monitoring continues. A backlog of ingestion events is currently being processed, which we expect will take some time. See: https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/23024

  5. Monitoring

    Monitoring continues. A backlog of ingestion events is currently being processed, which we expect will take some time. We will keep posting updates as we have them. See: https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/23024

  6. Monitoring

    We continue to see improvements across our graphs and reduced backlog, more info: gitlab.com/gitlab-com/gl-infra/production/-/work_items/23024

  7. Monitoring

    We continue to see improvements across our graphs, see also: gitlab.com/gitlab-com/gl-infra/production/-/work_items/23024

  8. Monitoring

    The mitigation we implemented is reducing the backlog of events. We are now monitoring the situation, see also: gitlab.com/gitlab-com/gl-infra/production/-/work_items/23024

  9. Identified

    We've identified the cause and are working on mitigation. More info: https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/23024

  10. Investigating

    Security and vulnerability reports are not loading or being ingested for some users. We are investigating now. For more info: https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/23024