Posted on July 17, 2020
On Friday July 17, 2020, Zesty.io experienced 27 minutes of downtime across all services due to a network outage from our DNS provider CloudFlare. Here is the official statement from CloudFlare:
Today a configuration error in our backbone network caused an outage for Internet properties and Cloudflare services that lasted 27 minutes. We saw traffic drop by about 50% across our network. Because of the architecture of our backbone this outage didn’t affect the entire Cloudflare network and was localized to certain geographies. That change started the outage at 21:12. Once the outage was understood, the Atlanta router was disabled and traffic began flowing normally again at 21:39 – Official Blog Post from CloudFlare
WebEngine: Customers who have cached websites through Fastly or Akamai, and who do not use CloudFlare as their DNS provider, would still have webpages loading but without media (images or video) stored on Zesty.io.
APIs: All APIs were running, but were not accessible due to network outage.
UIs: All customers would not have been able to log into the Zesty.io content manager or accounts system due to the network outage.
This incident was not isolated to Zesty.io; a large number of widely consumed services were affected.
The Zesty.io engineering team has prepared a DNS disaster migration plan in an event that networking fails via the DNS provider in the future.