SourcegraphAnalytics Maintenance
analytics.sourcegraph.com "analytics.sourcegraph.com": http://analytics.sourcegraph.com is back up and running.
Resolved
Outages and recoveries from the services you depend on.
40 incidents
analytics.sourcegraph.com "analytics.sourcegraph.com": http://analytics.sourcegraph.com is back up and running.
Resolved
All systems are operational.
Resolved
Access issues have been fully resolved, and this incident is now closed. Sorry for any inconvenience this caused, and thank you for your patience.
Resolved
A race condition with our deployment pipeline and ECS cluster caused our load balancer to point to inactive machines. While the current issue is resolved, the incident response team will work on mitigation of future issues of this type.
Resolved
This issue is now considered fully resolved. Thanks again for your patience as we worked on correcting this issue.
Resolved
We are continuing to monitor the system to ensure everything remains stable.
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
Issue has been resolved.
Resolved
All systems operational.
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
This advisory is now being closed. We encourage our customers to ensure that any affected WordPress plugins have been updated to the latest available versions and continue following WordPress security best practices. If you believe your website may have been impacted or require assistance, please contact our Support team.
Resolved
This incident has been resolved. Async transcription processing times on the US endpoint have returned to normal ranges and webhook deliveries are operating normally.
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
On July 7th at 2:10 PM UTC, some users experienced issues loading the Wix Editors. The issue also affected saving and publishing for some users during this period. All functionality was fully restored within approximately 20 minutes. The root cause was a deployment of a new version of one of our internal platform services that did not follow our standard gradual rollout process. Instead of being deployed incrementally across our infrastructure, as is our normal practice, the new version was released to all environments at once. This caused elevated latency in that service, which cascaded to the Editors. Our team identified the root cause quickly, rolled back the service to its previous stable version, and accelerated the recovery by restarting affected instances across our infrastructure. This fully resolved the issue. Wix continuously deploys updates across our microservice platform, the vast majority seamlessly and without any user impact. The safeguard that is specifically designed to prevent this type of issue was not applied in this case. Following this incident, we have introduced automated guardrails on the affected service to prevent new versions from causing latency spikes that propagate to dependent services. We have also strengthened our platform-wide rollout policies to enforce gradual deployment practices and make it harder to bypass monitoring and staged rollout checks in the future.
Resolved
On July 7th at 7:50 AM UTC, some users experienced issues loading the Wix Dashboard and Wix Editors. All functionality was fully restored within approximately 20 minutes. The root cause was an interaction between two events: a new automated database infrastructure optimization feature introduced on July 5th, and a routine planned traffic shift between data center regions carried out on the morning of July 7th. The new feature is designed to improve infrastructure efficiency by continuously monitoring database throughput usage and automatically adjusting capacity limits to reflect actual demand. On July 7th, before the day's planned traffic operations had taken place, the feature applied reduced capacity limits to several database clusters in one of our regions, based on the lower load observed earlier that morning. Shortly after, a routine planned traffic shift moved additional user traffic to that same region. The database clusters, now operating under reduced limits, received more load than those limits could accommodate, causing database response times to spike and errors to cascade across our dashboard and editors. Our team identified the root cause and resolved the incident by redirecting traffic to an unaffected region. We also provisioned new database nodes with higher capacity limits to restore full throughput in the impacted region. Following this incident, we have paused the automated optimization feature while we put additional safeguards in place. We have audited all database clusters where capacity adjustments were applied to confirm they are operating correctly. We are also implementing coordination checks to ensure that automated capacity adjustments and planned traffic operations are always validated against one another. We have additionally improved monitoring and alerting for automated infrastructure changes to enable faster detection and response if a similar situation were to arise in the future.
Resolved
Resolved
We have observed continued stability, and the cPanel licence updates have gone smoothly so far. If you need any further assistance, please do not hesitate to reach out to our Support team via chat or email.
Resolved
We're pleased to inform you that the issue affecting Bluesky functionality in Hootsuite and Talkwalker has been successfully resolved. Bluesky has confirmed the issue on their end is fixed, and our monitoring logs show that services are recovering - you should be able to publish and schedule to Bluesky in Hootsuite as usual, and Bluesky data is flowing into Talkwalker again. Thank you for your patience while we worked through this. If you continue to experience any issues, please don't hesitate to reach out to our support team.
Resolved
This incident has been resolved.
Resolved
This incident is resolved. Shared embeds and public links have been stable since 1:50 AM PT (08:50 UTC), and we have verified affected links render normally. A permanent fix ships later today. No data was lost and no customer action is needed.
Resolved
This is now resolved. If you have any questions or need further assistance, please reach out to us via email at support@upcloud.com or via Live Chat in our control panel: https://hub.upcloud.com
Resolved
This incident has been resolved.
Resolved
This incident has been resolved, and service is operating normally again. Thanks for your patience.
Resolved
Between approximately 20:56 UTC on July 8 and 01:40 UTC on July 9, customers using the V2 Config Delivery API may have experienced incorrect evaluation of launched Feature Gates. During this window, launched Gates could return false instead of their expected default. V1 consumers were not affected. We identified the root cause as a recent change to config-spec generation. We rolled back the affected services across all regions and regenerated config payloads. We have verified that launched Gates are again returning their expected defaults, and all Config Delivery API components are now operational. No action is required to restore correct evaluation going forward. However, if your systems persisted data or took actions based on evaluations returned by the V2 Config Delivery API during the impact window, we recommend reviewing and reconciling that downstream data, as affected launched Gates may have returned false during that time.
Resolved
The mitigation steps taken have resolved the issue. Latency has returned to normal levels, and we will continue monitoring for stability.
Resolved
This issue has been resolved.
Resolved
The issue affecting Fleet has been resolved, and users can interact with their agents normally again. Thanks for your patience.
Resolved
The issue affecting Fleet has been resolved, and users can interact with their agents normally again. Thanks for your patience.
Resolved
Snowflake Engineering has completed the postmortem of this service incident. A detailed Root Cause Analysis \(RCA\) is available on the Snowflake Community site: [https://community.snowflake.com/s/article/INC20000063](https://community.snowflake.com/s/article/INC20000063)
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
A set of fixes have been implemented and no further latency or failures have been observed in AWS us-east-1 since 17:00 UTC.
Resolved
This incident has been resolved.
Resolved
All services are back to operational.
Resolved
**Incident**: A configuration change inadvertently removed a critical resource required by one of our services. While existing infrastructure continued to operate normally, newly scaled infrastructure could not become ready. As morning traffic increased, there was a growing gap between incoming requests and our system's capacity to handle them, resulting in service degradation. We resolved the issue by manually restoring the missing resource and correcting the configuration change that caused its removal. **Impact**: For approximately 1.5 hours, a subset of customers experienced a full outage of our web application and desktop applications. During this time, affected users were unable to access Asana. No customer data was lost. **Moving forward**: We are implementing additional safeguards in our configuration management process to prevent critical resources from being inadvertently removed. We are also enhancing our monitoring systems to detect degradation earlier, before it impacts customer access. Our metric considers a weighted average of uptime experienced by users at each data center. The number of minutes of downtime shown reflects this weighted average.
Resolved