MailChannelsDelay in outbound email delivery
Queued emails have now been delivered. If you continue to see any issues, please reach out to us at support@mailchannels.com
Resolved
Outages and recoveries from the services you depend on.
40 incidents
Queued emails have now been delivered. If you continue to see any issues, please reach out to us at support@mailchannels.com
Resolved
From approximately 7:26 to 7:42 UTC, an internal change caused Ask product responses to be rejected instead of saved. Our monitoring system alerted us to this issue and we reverted the change immediately and everything is working as expected again.
Resolved
This incident has been resolved.
Resolved
A fix has been deployed and we are monitoring reconnections. We will backfill missed Activity Logs as needed after workplaces reconnect Splunk.
Resolved
This incident has been resolved.
Resolved
The issue is resolved and all dashboards are available. If you're still experiencing any related issues, please contact our customer support.
Resolved
The incident has been successfully resolved, and normal operations have been restored to all affected systems and services.
Resolved
This incident has been resolved.
Resolved
The incident has been successfully resolved, and normal operations have been restored to all affected systems and services. The root cause of the incident was a release of new cloud control plane version, which didn't have any issues itself, but triggered pulling new compute images from the registry on all k8s nodes. In some cases it took unexpectedly long -- up to 15 minutes.
Resolved
This incident has been resolved.
Resolved
The incident has been resolved, and traffic is now served without any issue.
Resolved
This incident has been resolved.
Resolved
The delayed invoices for upgrades from an Observe Business Plan to an Observe Scale Plan will be processed by the end of today, 28 May 2024
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
The incident has been successfully resolved, and normal operations have been restored to all affected systems and services. Our team conducted a thorough investigation, implemented necessary actions, and analyzed the underlying cause to prevent similar incidents in the future.
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
# **Postmortem** # **What happened?** On May 23, 2024, at 19:23 UTC, an increased load on the rendering infrastructure was detected. Actions were taken to scale out our system to handle the additional traffic. This incident was resolved at 19:36. # **How were customers impacted?** During the incident, customers experienced increased error rates for recent renders, intermediate errors increased in our system, and response times for requests increased. # **What went wrong during the incident?** During the incident, our team implemented a service change that led to assets being dropped. This led to an increase in requests to our system. The increased requests to our system led to `429` and `5XX` errors. # **What will imgix do to prevent this in the future?** To prevent similar incidents, we will: * Improve procedures for pre-scaling instances during critical updates. * Conduct impact assessments before issuing significant changes. * Enhance monitoring and alerting systems to predict and manage load increases better.
Resolved
The incident has been successfully resolved, and normal operations have been restored to all affected systems and services.
Resolved
Our engineering team has resolved this issue, and the Card Repeater Power-Up is working again. Cards that missed their scheduled repeat will run at their next scheduled time.
Resolved
Sessions and in-app events processing is back to normal. If you encounter any further related issues, please don't hesitate to contact us. Thank you for your patience and understanding.
Resolved
We have restored the full search functionality. The incident is now resolved.
Resolved
This incident has been resolved.
Resolved
On May 23, from 7:42 AM UTC to 8 AM, the Plugin Portal was unavailable following a failed maintenance operation. Service has been restored successfully.
Resolved
The issue with 3D Terrain distortion has been mitigated and fixes have been shipped to all Static Image and Tile endpoints.
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
The issue is now resolved, and our teams continue to monitor the dashboard's performance. If you encounter any further issues, please report them to our technical support team.
Resolved
The disturbance have been resolved.
Resolved
This incident has been resolved.
Resolved
This incident has been resolved.
Resolved
The issue has been resolved. The issue was caused by a backend software update which had the effect of throwing an error during some Okta SSO login transactions. Our DevOps team quickly reverted the change and our engineers will be addressing the bug in a follow-up release.
Resolved
This incident has been resolved.
Resolved
This issue has been resolved for all affected projects as of Thursday, 2024-05-16 18:10 US/Pacific. We thank you for your patience while we worked on resolving the issue.
Resolved
The Google Cloud Platform networking issue has been resolved. If you have any questions or concerns please submit a ticket to our support portal.
Resolved
From 2:45PM to 3:00PM PST, several of the Keeper application API instances in the US region were responding with HTTP 500 errors due to an internal application error. The Keeper DevOps team identified the error and restarted those affected instances. The software engineering team has been investigating the root cause and will be addressing in an upcoming backend release.
Resolved
We're all set! PandaDoc is working properly and the incident has been fully resolved. If you continue to experience any issues with this, please reach out to us at support@pandadoc.com.
Resolved
This incident has been resolved.
Resolved