On Tuesday, July 8, 2025, some of our customers experienced a service disruption that prevented them loading the static content from our Content Delivery Network (CDN). The incident caused pages to be un-usable for users who had not loaded the cached content yet today, for approximately 43 minutes, depending on DNS delays. We have identified the root cause and have implemented a fix.
The incident was caused by incorrect DNS caching within the Cloudflare network, which is not an Encompass vendor but is a common DNS provider outside of our AWS CloudFront CDN. CloudFront uses geographic routing, providing different server IPs based on the user's location. For an unknown reason, Cloudflare had cached and was serving incorrect IP addresses for our CDN endpoint, leading to connection failures for users whose DNS queries were routed through the affected Cloudflare locations.
To resolve the issue, our engineering team forced a refresh of the DNS cache. This was accomplished by toggling the DNS records in AWS Route53 for the affected domains from an A record to a CNAME record, and then immediately back to an A record. This action prompted a network-wide update, flushing the incorrect IPs from Cloudflare's cache and propagating the correct ones.