BitlyDelay in webhook deliverability
This incident has been resolved.
Resolved
Outages and recoveries from the services you depend on.
40 incidents
This incident has been resolved.
Resolved
Our engineering team has confirmed this issue has been mitigated. We apologise for any inconvenience this may have caused you.
Resolved
Forge Invocations had an 8 minute outage between 2023-11-29 03:05:13 UTC to 2023-11-29 03:13:27 UTC resulting in Smart Links failing. This service has recovered post this time period.
Resolved
This incident has been resolved.
Resolved
A high IO demand from a tenant caused around 200 projects on the same k8s node to get stuck, preventing new connections for a duration of about two hours. The k8s node itself became not manageable. We have isolated the tenant and we're working on a solution to prevent this from happening in the future.
Resolved
We're excited to inform you that we've shipped upgrades to our production environment, enabling scheduled reports once again. What's Changed: To continuously improve and ensure the security of our services, we've implemented additional controls including domain restrictions and a limitation on the number of recipients per email. This is specifically for mitigation purposes. From now on, users will start receiving emails for the reports they've scheduled for themselves, and they will also have the ability to create new tasks. Impact: Please note that any existing scheduled jobs with external recipients will no longer be editable. However, users can delete these and create new jobs using their email IDs. Thank you for your patience during these changes. We want to assure you that future updates and communications will be shared promptly to keep you informed. We appreciate your understanding and continued support.
Resolved
This incident has been resolved.
Resolved
Our engineering team has rolled out a fix and has confirmed webhooks delivery is fully operational on the affected EU cluster.
Resolved
Serverless clusters in GCP europe-west1 were inaccessible.
Resolved
Figma observed increased errors between 3:36pm PT and 3:39pm PT. The incident has been resolved.
Resolved
The FIDO2 / Yubikey login issues have been resolved. The engineering team is still investigating root cause and we'll update the case notes shortly.
Resolved
There was an intermittent error with status embed page where cached responses were incorrectly being shown to a few customers. This is resolved as of now.
Resolved
This incident has been resolved. If you're still seeing issues, please reach out at https://trello.com/contact/
Resolved
The original issue is identified and resolved.
Resolved
Incident resolved.
Resolved
The incident has been successfully resolved, and normal operations have been restored to all affected systems and services. Our team conducted a thorough investigation, implemented necessary actions, and analyzed the underlying cause to prevent similar incidents in the future.
Resolved
The incident has been successfully resolved, and normal operations have been restored to all affected systems and services.
Resolved
A solution has been put in place and the platform is functioning stably.
Resolved
The incident has been successfully resolved, and normal operations have been restored to all affected systems and services.
Resolved
The incident has been successfully resolved, and normal operations have been restored to all affected systems and services.
Resolved
This incident has been resolved
Resolved
Figma experienced an issue between 1:32am - 1:49am GMT that caused certain key operations to fail. The issue has been resolved.
Resolved
Cluster operations appear to once again be healthy. We will continue to monitor for subsequent events/impacts due to the ongoing Quay.io issue.
Resolved
The incident has been successfully resolved, and normal operations have been restored to all affected systems and services.
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
We discovered this morning that our Logplex pipeline, which handles Heroku platform errors, had an interruption in processing, which caused platform errors not to be recorded. This has been corrected, the backlog is being processed, and additional monitoring has been added to avoid this issue in the future.
Resolved
We want to let you know that the changes we made on our side fixed all the connection issues our users were experiencing with Trello, and all services are functional right now. We appreciate your patience and understanding.
Resolved
This incident has been resolved.
Resolved
All operations have returned to normal, and the backlog of jobs has been cleared.
Resolved
This disruption has been resolved.
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
** If you didn't receive an email directly, your rooms were not affected by this incident ** We wish to notify you of a system irregularity on November 6th at 6:34 AM EST, which impacted our database services. This anomaly could potentially affect storage for rooms active at that time, meaning that at least one user was connected to that room at precisely 6:34 AM EST. We sent an email to all our customers potentially impacted. This email includes the list of rooms that might have been impacted. A small number of rooms created shortly before the incident cannot be restored. The vast majority of rooms can be restored thanks to our backups. We're available to restore data if necessary upon request. We have conducted a full investigation and instituted measures to bolster our systems, safeguarding against future incidents. Ensuring the integrity of your data and the reliability of our service remains our utmost priority. Please do not hesitate to reach out to initiate any necessary restoration or to address your concerns.
Resolved
On `2023-10-28` we experienced a DDoS attack against our platform. Our oncall team responded promptly and worked to mitigate impacts to our services. From `04:56 UTC` to `06:26 UTC` we experienced intermittent but significant impact to the Bitly website, Link-in-Bio pages, and redirects for custom domains with DNS A records. Some custom domains continued to see impact until `19:51 UTC` when we released additional updates to our mitigation strategy. The Bitly API and links on the bit.ly domain and custom domains with DNS CNAME records continued to work throughout this period. Additionally, during this incident our status page provider was conducting a maintenance window and experienced an outage of its own which affected our ability to communicate our investigation and response to this incident via status.bitly.com in a timely manner. At `05:55 UTC` we were able to update bitly.com with some details of our incident response. We apologize for the inconvenience this outage may have caused and understand the impact to our customers. We place a high value on the reliability of our services and are evaluating ways to improve our ability to mitigate this type of issue. We also understand the impact and potential confusion caused by the lack of messaging during this outage. We value highly and will work to improve our ability to communicate about incidents in a timely fashion.
Resolved
The SCIM issue with Azure has been resolved.
Resolved
At 12:51AM PST, an AWS Aurora database failover occurred. The Keeper application servers properly recovered after 60 seconds. From 1:08AM PST to 1:16AM PST, a high number of login errors occurred from a subsequent and related AWS Elasticache failure. After further investigation, the team will update this event postmortem.
Resolved
### **SUMMARY** From 06:00 UTC to 07:45 UTC on October 28, 2023, Atlassian customers using Statuspage had intermittent issues with all Statuspage functionality. The event occurred due to a database performance issue during a [scheduled database maintenance](https://metastatuspage.com/incidents/s21b66328h9j). This impacted customers in all regions. The incident was detected within one minute by monitoring the upgrade process and mitigated by rolling back to a known good snapshot which put Statuspage systems into a known good state. The total time to resolution was about one hour and 45 minutes. ### **IMPACT** The overall impact was between 06:00 UTC and 07:45 UTC October 28, 2023. This incident affected Statuspage customers from all regions and caused intermittent backend errors on all Statuspage activity including viewing pages, adding subscribers, and creating/updating events. We performed a rollback operation during recovery to return to a known good state. ### **ROOT CAUSE** The issue was caused by database performance issues after a routine database maintenance and upgrade. As a result, our backends returned intermittent errors to several user requests. ### **REMEDIAL ACTIONS PLAN & NEXT STEPS** We take the utmost care to provide a highly reliable service. We will pursue several preventive measures to ensure that this situation does not occur in the future, including: * Fixing the cause of the performance issues before future upgrades; and * Improving our testing process for database upgrades to catch potential performance issues. We apologize to customers whose services were impacted during this incident; we are taking immediate steps to improve the platform’s performance and availability. Thanks, Atlassian Customer Support
Resolved
# What happened? On October 23, 2023, between 21:43 UTC and 23:14 UTC, imgix experienced a partial outage affecting images served from the Rendering API. During this time, a small percentage \(<0.45% on average\) of non-cached requests returned a server error. A fix was implemented at 23:02 UTC, which allowed the service to recover by 23:14 UTC fully. # How were customers impacted? Between 21:46 UTC and 23:14 UTC, requests to the Rendering API returned a server error, with 0.65% of all requests to our CDN returning an error at the height of the incident. Additionally, Sources returned an unknown status between 21:06 UTC to 21:09 UTC. During this period, customers reported being unable to create Sources. # What went wrong during the incident? Our Rendering API experienced an unexpected interaction that caused a dramatic increase in server load. This caused error rates to increase as the network became overloaded slowly. The errors fluctuated between 0.07% to 0.65% until we resolved the issue. To restore the service, our engineers re-configured our network traffic to handle the unexpected Rendering behavior. During the incident, a separate issue \(unrelated to rendering\) impacted our Source data. This led to a delay in investigating the cause of the rendering errors. # What will imgix do to prevent this in the future? We have taken the following steps to prevent this issue from recurring: * Fixed the misconfigured server interaction * We will put an alert system in place to notify us when traffic congestion happens from a misconfigured source interaction. We are in the process of implementing the following: * Conducting a review of our current tooling to increase our traffic and network configuration capabilities. * Reviewing our current configuration to limit the affected services should a similar incident happen.
Resolved