HYOK, Private VCS, and Private Run Tasks returning 403 errors

Resolved

Started

Lasted 1h 10m

View official incident ↗

Updates

  1. Resolved

    Our Engineering team has resolved the issue. The previous fix was incomplete and rolled back by a legacy deployment process, but the fix is now permanently rolled out. If you continue to experience issues, please contact support.

  2. Investigating

    After a brief recovery the service has resumed emitting 403 errors at a high rate. Our engineering team is investigating. Next update time: 04:30 UTC

  3. Resolved

    Our Engineering team has resolved the issue. All systems should now be operating normally. If you continue to experience any problems, please open a ticket with our support team.

  4. Identified

    Our Engineering team has identified the cause and we are actively working on a fix. Action underway: Restoring service. Next update time: 04:10 UTC

  5. Investigating

    We are aware of elevated 403 errors being returned when requests are forwarded through the tfc-agent. This may impact HYOK, Private VCS, and Private Run Tasks. Action underway: Our engineers are investigating the cause now. Next update time: 03:55 UTC