BufferIssue with creating Pinterest posts
Pinterest post creation was blocked by an error saying a board must be selected, even when a board was selected.
Resolved
Outages and recoveries from the services you depend on.
40 incidents
Pinterest post creation was blocked by an error saying a board must be selected, even when a board was selected.
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
This incident has been resolved. Incident Start Time: 2026-09-22 03:05 UTC Incident End Time: 2026-09-22 05:31 UTC
Resolved
The elevated error rate has been resolved and we're following up internally to prevent future occurrences.
Resolved
This issue has been resolved. Impact occurred from 5:50pm PT / 00:50 UTC to 7:10pm PT / 02:10 UTC.
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
This incident has been resolved. All affected services, including Flink in AWS us-west-2, have fully recovered, and no further customer impact is expected. No action is required from customers.
Resolved
We've rolled out a fix for the degradation and availability has recovered
Resolved
# Elevated API Errors **Date:** 09/18/2026 **Duration:** 22:40–23:03 UTC \(23 minutes\) **Status:** Resolved ## Summary On September 18, 2026, from 22:40 to 23:03 UTC, WorkOS experienced an outage affecting authentication and other API requests. Customers encountered failed sign-ins, API errors, timeouts, and long delays. Authentication was most severely affected. An infrastructure failover in the cache supporting our feature-flag system exposed weaknesses in application connection recovery and fallback behavior. Some connections remained stalled after the cache was available again, delaying requests and exhausting database connection pools. Service recovered by 23:03 UTC. We subsequently deployed changes to improve connection recovery. We’re sorry for the disruption this caused you and your users. ## Background WorkOS uses feature flags to control application behavior. The affected backend services read flag data from a shared Redis datastore populated by LaunchDarkly Relay. Each application also keeps a local cache, with a 30-second expiration. While an entry is valid, the application can evaluate it locally; after it expires, the application reads Redis. The Redis cluster already had a primary and replica in separate AWS availability zones, with automatic failover enabled. This redundancy allowed AWS to promote a replacement primary. Applications still needed to detect failed connections and reconnect to that primary. ## What Happened ### Affected products Some requests experienced errors or delays across the following products: * **AuthKit:** Sign-in, token requests, user-management operations, and hosted authentication pages. * **Single Sign-On \(SSO\):** Authorization and SAML callback requests. * **WorkOS API:** Selected operations, including organization requests, API-key validation, and connection requests. * **Dashboard:** Some operations that relied on the affected APIs. * **Admin Portal:** Portal link generation and monitored setup requests. * **Directory Sync:** SCIM OAuth token requests, plus delays and client disconnects in Directory Sync API traffic. * **Audit Logs:** Management operations, particularly action-schema creation. ### Customer impact Customer impact lasted from **22:40 to 23:03 UTC**. Across our public edge, **24.1% of recorded requests returned server errors**, and **17.2% ended with the client disconnecting**. The table breaks down request groups by product. | Product | Server errors \(HTTP 5xx\) | Client disconnects \(HTTP 499\) | | --- | --- | --- | | AuthKit, User Management and organization APIs | 37.22% | 24.59% | | Hosted AuthKit \(default-domain proxy\) | 7.27% | 13.77% | | SSO and connection APIs | 2.22% | 2.31% | | Dashboard | 27.10% | 8.08% | | Admin Portal | 24.61% | 7.47% | | Directory Sync APIs | 0.08% | 2.80% | | Audit Logs management APIs | 22.82% | 3.55% | | Audit Logs event capture | 0.0006% | 0.11% | #### Latency Samples * **Authenticate POST requests:** At the peak, one-minute median latency reached approximately 32 seconds and p95 latency reached approximately 45 seconds, compared with roughly 65 ms and 140 ms before the incident. * **Directory Sync:** Peak one-minute p95 latency for selected SCIM user-creation and PATCH-update operations reached approximately 22 seconds, compared with roughly 65 ms before the incident. ### Root cause and contributing factors Around 22:40 UTC, the Redis primary became unavailable to application clients. AWS ElastiCache promoted its replica, completing the failover at 22:40:17 UTC. New connections could reach the promoted node, but some existing application connections remained stalled. Three application behaviors prolonged and amplified the interruption: 1. **Flag-call timeouts did not recover stalled connections.** An application-level deadline returned a fallback value after two seconds, but it did not cancel the underlying Redis command or replace the connection. Additional reads accumulated while clients waited to detect the connection failure. Some commands remained pending for approximately 16 minutes. 2. **External waits held database connections open.** Some flag checks occurred inside database transactions. Repeated waits of several seconds kept pooled database connections occupied and, in some cases, held row locks. Available connections were consumed, causing other requests to queue and fail. Database compute capacity was not the limiting factor. 3. **A fallback selected an incompatible decryption path.** A flag used during a decryption-service migration defaulted to the old behavior when flag evaluation failed. That sent some encrypted values to a legacy service that could not decrypt them, causing additional authentication failures. The local cache reduced normal Redis traffic, but expired entries still depended on Redis. Our two-second deadline limited how long a flag caller waited; it did not provide prompt recovery of a stale, established connection. This was a gap in our resiliency safeguards. ### Recovery and communication Responders restarted affected application instances, which triggered reconnection to the healthy Redis instance. As connections recovered and queued work drained, error rates and latency returned to normal. We also identified a notification gap: the initial status page update did not select the affected service components. Customers subscribed only to those components therefore did not receive the incident notification. ## Timeline All times are UTC. * **September 18, 22:40** — Customer-facing errors and latency increase. AWS completes promotion of the Redis replica at 22:40:17. * **22:44** — Engineers declare an incident following automated alerts. * **22:49** — The public status-page incident is opened. * **22:53** — Investigation identifies stalled feature-flag Redis clients and database connection starvation. * **22:55–23:03** — Responders restart affected application instances, existing clients begin reconnecting, and queued work drains. Recovery progresses across services. * **23:03** — Customer-facing error rates and latency return to normal. * **23:29** — The status-page incident is marked resolved after monitoring. * **September 19, 01:44** — A Redis-client recovery fix is deployed to production after validation with controlled failure and failover tests in staging. ## Remediation ### Immediate Fixes Completed: * **Restored service:** Restarted affected application instances and monitored recovery as clients reconnected and database queues drained. * **Bounded Redis failures:** Deployed a Redis-client change to detect stalled established connections, bound failed operations, and reconnect more promptly. * **Corrected the decryption fallback:** Changed the migration flag’s fallback to behavior compatible with both current and legacy ciphertext formats. * **Validated the recovery fix:** Tested stalled-client behavior and an ElastiCache failover in staging before deploying the connection changes to production. ### Near-term Improvements The following work remains open: * Extend bounded connection recovery to other Redis clients and verify their behavior during failover \(in progress\). * Remove feature-flag and decryption-service network waits from authentication database transactions \(in progress\). * Complete the legacy decryption migration and retire its obsolete flag and fallback paths. * Audit feature-flag defaults against stable production behavior and retire flags whose migrations are complete. * Evaluate and pilot streaming flag updates into application memory, reducing the need to contact Redis while serving requests. ### Preventive Measures Our follow-up work includes: * Improving visibility into flag-evaluation timeouts, fallback use, and connection recovery. * Adding checks for external network waits inside database transactions. * Requiring affected service components to be selected when opening a status-page incident, so component-specific subscribers receive notifications. ## Conclusion We recognize the impact this incident had on you and your customers. We’re sorry for the interruption and for the notification gap. We have deployed the immediate connection-recovery and fallback fixes. We will track the remaining corrective work through completion and update this analysis as unresolved findings are confirmed.
Resolved
We did not experience any customer impact during this AWS us-east-1 outage.
Resolved
This incident has been resolved.
Resolved
Vonage services have been fully restored following today’s service interruption, which began at 2026-09-21 18:40 UTC. Full resolution has been confirmed effective 2026-09-21 19:59 UTC. Thank you for your patience and understanding as we worked to restore services as quickly as possible. If you have any questions, please contact api.support.vonage.com.
Resolved
We've resolved the incident and the affected service is now operational.
Resolved
This incident has been resolved. Incident Start Time: 2026-09-21 22:29 UTC Incident End Time: 2026-09-21 23:05 UTC
Resolved
This incident has been resolved.
Resolved
# **Summary** To improve the security posture, On September 14, 2026, Harness performed a scheduled rotation of the TLS certificate used to secure Delegate connections for accounts using mutual TLS \(mTLS\),The replacement certificate passed Harness's existing pre-deployment validation and smoke testing. However, the rotation also introduced a change in certificate authority, which required additional certificate-chain and gateway configuration this prevented affected Delegates from connecting to Harness. Harness has identified the contributing factors and is implementing the corrective actions described below to prevent a recurrence. # **Impact** Delegates configured to use mutual TLS in Prod environment were unable to connect to Harness for until Harness applied a temporary mitigation \(disabling mTLS\) that restored connectivity while the underlying issue was corrected. This affected the ability to run pipelines and other operations dependent on the affected Delegates. Full, permanent connectivity — with mTLS restored to its normal, fully secure configuration — was confirmed for both environments by 7:20 PM PST. # **Root Cause** As part of routine certificate maintenance, Harness rotated the certificate used by the gateway component that secures mutual TLS connections for Delegates in prod environments. The rotation also introduced a change in certificate authority, which required additional certificate-chain and gateway configuration that was not fully covered by the existing validation process. As a result, the gateway was unable to load the new certificate correctly, and Delegates relying on mutual TLS could not establish a secure connection. # **Remediation** Once the certificate rotation was confirmed as the source of the issue, Harness engineering worked continuously to correct the certificate and validate the fix with Citi before fully restoring the secure configuration. Corrective and Preventive Actions During remediation, Harness also identified and corrected a secondary issue with the replacement certificate's chain, which required an additional deployment before connectivity was fully and permanently restored. # **Next Steps** Harness is taking the following steps to prevent a recurrence of this issue: 1. Extend pre-deployment validation and smoke testing to explicitly cover certificate authority changes and full certificate-chain verification, closing the specific gap that allowed this configuration through. 2. Enhance automation for managing mTLS-related configuration changes, 3. Improve monitoring and alerting for Delegate connectivity, including account-level visibility, so similar issues are detected automatically
Resolved
This incident has been resolved.
Resolved
### Summary On September 11, 2026, between 07:02 and 07:56 UTC, Atlassian customers primarily from the EU region and some from APAC region using Jira and Jira Service Management products were unable to view work items or access boards consistently. The event was triggered by a backward-incompatible internal configuration rollout in the underlying graph data platform. The incident was detected within 3 minutes by our automated monitoring system and mitigated by rolling forward the stable deployment to all regions. The total time to resolution was about 54 minutes. ### IMPACT The overall impact occurred between September 11, 2026, 07:02 UTC and September 11, 2026, 07:56 UTC on two products. During this period, some customers in the EU region using Jira and Jira Service Management experienced intermittent failures or degraded performance. We noted that a configuration failure prevented critical dependency services from scaling, which resulted in an outage for affected users. Jira and Jira Service Management users experienced timeout errors and failed or delayed page loads when attempting to access the work item view, issue navigator, and board view. ### ROOT CAUSE The incident was caused by the deployment of a change to the graph store service, which serves links to third-party entities such as pull requests. The change caused startup checks to fail, preventing new nodes from joining the cluster. During peak traffic, the service could not scale sufficiently, causing requests from Jira and Jira Service Management to the graph store service to fail. ### REMEDIAL ACTIONS PLAN & NEXT STEPS We know that outages impact your productivity. We are prioritizing the following improvement actions to avoid repeating this type of incident: - Implement Code-Based Configuration: We are moving configuration definition files to be part of the core codebase to ensure they are versioned and deployed synchronously with the application logic. - Graceful Degradation in Jira: We are implementing changes to the affected Jira views to segregate core queries from noncritical dependency queries, allowing the page to remain functional even if a downstream service fails. - Optimize Deployment Pipeline: Deployment pipelines are being optimized to enable accelerated, parallel updates during operational response scenarios. We apologize to customers whose services were impacted during this incident; we are taking immediate steps to improve the platform’s performance and availability. Thanks, Atlassian Customer Support
Resolved
The incident has been resolved.
Resolved
**Summary** On September 16, 2026, between 20:04 and 21:08 UTC, our US Async API returned elevated server errors. At peak, roughly 50% of async transcription requests on the US endpoint failed, and a portion of the requests that did succeed took longer than usual to complete. Our EU endpoint and real-time/streaming services were not affected. **Root cause** The failure originated in AWS SQS, the managed queuing service our transcription pipeline uses to distribute work. Specifically, the Fair Queues feature began rejecting valid messages with an `InvalidParameterValue` error on a parameter our services had been sending successfully for months. There was no deploy or infrastructure change on our side that triggered this; AWS confirmed it as a service-side issue and rolled back the change on their end later that evening. Because the affected queues sit in the path between our API and our transcription workers, requests that could not be queued failed outright, and a portion of the work that did get through was delayed behind reduced throughput. We mitigated by disabling Fair Queues across our services and reverting to standard SQS queues, which restored normal operation roughly an hour after the first error. **What we're doing about it** * **Faster configuration changes.** Much of our service configuration lives in environment variables, which require task restarts to propagate. We are moving this to a global feature flag system so mitigations like this one take seconds rather than minutes. * **Expedited emergency deploys.** We are adding a reviewed break-glass path so urgent fixes can bypass non-essential CI steps. * **Multi-region failover.** We are prioritizing work to fail the US pipeline over to a second region when a regional provider dependency degrades, rather than relying on feature-level mitigations alone. We're sorry for the disruption. No customer audio or transcript data was lost, and any requests that failed during this window can be safely resubmitted.
Resolved
This incident has been resolved.
Resolved
The systems are stable and all services are functioning normally again.
Resolved
At this time we are resolving this incident. If you continue to experience performance degradation in the Singapore area, please open a support ticket. More details in gitlab.com/gitlab-com/gl-infra/production/-/work_items/22981.
Resolved
This incident has been resolved.
Resolved
The issue intermittently affecting Agent runs and Zap runs using AI by Zapier has been resolved. The confirmed impact window goes from September 18, 2026 at 2:00am UTC to today, September 21, at 15:00 UTC. Our team continues to work with our service provider to prevent this issue in the future, implement better safeguards, and consolidate self-healing capabilities. Some Agent runs from previous days (September 18 to September 20) could not be replayed to avoid duplicate agent executions. For Agents, if you want to replay affected runs, go to your Agents dashboard: https://agents.zapier.com/bots, open the All Activity: https://agents.zapier.com/activity tab to view recent runs, and review the agent runs during the incident window. You can replay any runs that need to be reprocessed by prompting the agent in the chat to retry. For Zap runs using AI by Zapier, no action needed. Our team has successfully replayed all affected AI by Zapier runs. Many thanks for your patience while we worked through this. If you gave any questions or need help, reach out to our support team at: zapier.com/get-help.
Resolved
This incident is resolved, and magic link login is fully functional. Malicious activity caused our email provider to flag the Knock dashboard's transactional emails, disrupting magic link deliverability. We've blocked the activity and restored email delivery. Only Knock's emails from hello@notify.knock.app were impacted. Customer email channels were unaffected.
Resolved
There was an impact of increased 5xx on writes with metrics. The error rates have returned back to normal, and our team is continuing to investigate the cause.
Resolved
Resolved
This incident has been resolved.
Resolved
The incident has been resolved.
Resolved
# Root Cause Analysis Report # 1. Summary On Wednesday, August 26, 2026, some customers running Infrastructure as Code Management \(IaCM\) pipelines in Harness production saw those pipelines start, sit in a Running state with no init or plan steps, then abort or expire after about 10 minutes. OpenTofu / Terraform work did not start. The failure presented as a pipeline timeout; it was not an OpenTofu timeout and not a failure of the customer's infrastructure-as-code. Other product areas that do not go through this IaCM-to-Pipeline handoff were not affected by this failure mode. # 2. Impact * Some IaCM pipelines did not complete. Stages appeared stuck, then timed out or aborted. * Execution graphs showed the IaCM stage Running with no child steps. Init and plan never ran, so there was no plan output and no apply. * A reported affected execution ran for 10 minutes and 7 seconds before being aborted. That interval is a platform stuck-execution safeguard, not a customer-configured step-timeout setting. * Customers needed to re-run their pipeline after service was restored in their environment. # 3. Root Cause IaCM stages are driven by the Harness Pipeline service. After an IaCM stage starts, IaCM must tell Pipeline to create the child steps \(init, plan, and so on\). Those messages travel on Harness's internal event bus. An incompatible modification within a shared library \(unrelated to IaCM specifically\) was inherited by the IaCM Manager upon rebuilding with the updated version, without any direct changes to IaCM functionality. # 5. Remediation IaCM Manager was rolled back to an older version in each affected production environment, All affected environments were restored by 09:00:40 UTC on August 26. Customers do not need to change pipeline YAML, OpenTofu/Terraform configuration, or connectors. Re-running the same pipeline after service was restored is sufficient. # 6. Preventive Actions 1. **Backward-compatible or atomic SDK/event changes:** producer and consumer changes in the shared Pipeline SDK and Pipeline service must remain backward compatible, or be shipped and deployed together atomically, so that mixed versions can never silently drop messages. 2. **Detection:** add alerting for an IaCM stage that stays Running with zero children, or for child-step events that are produced but never consumed, rather than relying on the time based abort as the first signal.
Resolved
All services remain stable. The incident is now resolved.
Resolved
This incident has been resolved.
Resolved
Flow message sending has fully recovered and is processing at normal speeds. Delayed messages were not lost and have since been delivered. This incident has been resolved.
Resolved
We are aware of reports of developer API tokens getting unexpectedly invalidated. We are currently investigating this issue. Customers who are affected will need to generate new tokens in developer settings as the affected tokens cannot be re-used.
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved