Martus Solutions - East US outage during deployment – Incident details

East US outage during deployment

Resolved
Major outage
Started about 1 month agoLasted 32 minutes

Affected

US

Major outage from 1:44 AM to 2:16 AM

Updates
  • Resolved
    UTC
    Resolved

    AGIC (App Gateway Ingress Controller) fired a reconciliation during the deployment window, capturing all 16 services mid-transition and pushing all the stale IPs into the App Gateway backend pools. Once the old pods stopped, the App Gateway began health-probing stale IPs, resulting in 502 errors for all sites. The AGIC was restarted to clear the stale IPs.

    This incident has been resolved.

  • Investigating
    UTC
    Investigating
    We are currently investigating this incident.