Issues Loading Pages and printing

Incident Report for Encompass Technologies

Postmortem

Summary

On Tuesday, July 8, 2025, some of our customers experienced a service disruption that prevented them loading the static content from our Content Delivery Network (CDN). The incident caused pages to be un-usable for users who had not loaded the cached content yet today, for approximately 43 minutes, depending on DNS delays. We have identified the root cause and have implemented a fix.

Timeline (All times MDT)

  • 9:03 AM: Our engineering team began investigating alarms from our monitoring services
  • 9:30 AM: The team identified and incorrect IP address for the cdn.e8.co in the CloudFlare network (other DNS providers were correct), and began implementing a fix by forcing an update to the DNS records to propagate the correct IP addresses.
  • 9:46 AM: The refreshed DNS IP addresses had propagated to all major DNS providers, though some smaller DNS providers may have held onto the old IP for a longer than expected.
  • 10:35 AM: Monitoring period complete and multiple reports confirmed issue was no longer present.

Root Cause

The incident was caused by incorrect DNS caching within the Cloudflare network, which is not an Encompass vendor but is a common DNS provider outside of our AWS CloudFront CDN. CloudFront uses geographic routing, providing different server IPs based on the user's location. For an unknown reason, Cloudflare had cached and was serving incorrect IP addresses for our CDN endpoint, leading to connection failures for users whose DNS queries were routed through the affected Cloudflare locations.

Resolution

To resolve the issue, our engineering team forced a refresh of the DNS cache. This was accomplished by toggling the DNS records in AWS Route53 for the affected domains from an A record to a CNAME record, and then immediately back to an A record. This action prompted a network-wide update, flushing the incorrect IPs from Cloudflare's cache and propagating the correct ones.

Posted Jul 14, 2025 - 11:27 MDT

Resolved

This incident has been resolved, a post mortem will be posted in 5-7 business days.
Posted Jul 08, 2025 - 11:05 MDT

Update

All major DNS providers have accurate information now from our monitoring.
We will continue to monitor for 30 minutes for any outstanding issues.

Local office networks may have lingering DNS caches that need to be cleared manually.
Posted Jul 08, 2025 - 10:35 MDT

Update

We are continuing to monitor for any further issues.
Posted Jul 08, 2025 - 10:26 MDT

Monitoring

We're experiencing DNS propagation issues affecting some users' ability to access our CDN. The issue will resolve automatically as DNS updates propagate globally over the next 5-60 minutes, depending on local caching settings.

Quick fixes for affected users, if you're unable to load images/files on our site:

A. Wait 5-60 minutes - the issue will resolve automatically as DNS updates spread
B. Restart your router - unplug for 30 seconds, plug back in, wait 2-3 minutes
C. Try a different network - switch to mobile data or different WiFi
D. Restart your device if the above doesn't work

For tech-savvy users:
Temporarily change DNS to 8.8.8.8 and 1.1.1.1 in network settings
Posted Jul 08, 2025 - 09:46 MDT

Identified

We're experiencing DNS propagation issues affecting some users' ability to access our CDN. This appears to be due to a DNS propagation issue between AWS Cloudfront and the Cloudflare networks in particular.
Posted Jul 08, 2025 - 09:38 MDT

Investigating

We are currently investigating the issue, there appears to be an issue with the AWS CDN
Posted Jul 08, 2025 - 09:03 MDT
This incident affected: Encompass Cloud Platform (ECP) and Trading Partner EDI Connections.