(Automated) Elevated error rates detected.

Summary

On August 1, 2025, the DSDLink ECP website became unavailable for approximately 10 minutes. Our engineering team was immediately alerted, and the issue was promptly resolved.

Root Cause

The outage was triggered during a routine automated upgrade process for the system. A failure occurred while uploading a new software image to our container registry (AWS ECR), likely due to an internal networking issue at AWS. The deployment automation proceeded to update the live service with an incomplete test image.

Resolution

Our on-call engineer was alerted by our monitoring systems as soon as the issue occurred. They immediately began an investigation and were able to manually roll back the failed deployment and restore the previous, stable version of the website. Service was restored within 10 minutes of the initial alert.

Corrective and Preventative Actions

To prevent a recurrence of this issue, we have implemented the following improvements:

1. Enhanced Deployment Logic: We have updated our automated deployment scripts. The system will now halt the upgrade process immediately if it detects any failure during the image upload to our container registry. This ensures that an incomplete or incorrect image can never be deployed to the production environment. This action was completed on August 1, 2025.
2. Improved Monitoring: We are reviewing our monitoring and alerting to ensure even faster detection of similar deployment-related failures in the future.