Posted on June 21, 2021
On Monday June 22nd 2021, a handful of customers who had domains configured prior to 2019 intermittently experienced outages to their production domains between 5:40pm and 7:27pm. Customers who configured domains prior to 2015 experienced downtime intermittently until 11:10pm PT. The issue was caused by credentials expiring on a legacy Kubernetes cluster.
-
Intermittent outages may have occurred as early as 5:40pm PT
-
The issue was mitigated for a majority of customers at approximately 7:27pm PT by promoting all production instances to the latest WebEngine service
-
Some instances that were configured over 6 years ago were offline until 11:10pm PT
-
Outages affected a small portion of our customer base
WebEngine: Page rendering to the registered production domain.
Manager UI: Manager UI Publishing.
Instances API: Instances API Create Item Publishing endpoint
The legacy service was removed from the serving pipeline for most customers who do not have a configuration prior to 2018. The legacy service that caused the issue is due to sunset on August 1st. All new services are auto managed and do not require manual credential rotations.