Resolved -
This incident has been resolved and services are back to normal. We'll be doing a post morterm with upstream infrastructure providers to better understand the root cause.
Oct 20, 15:15 PDT
Update -
We're investigating reports of unexpected product impact (e.g., copilot looking visually different). We've traced the issue to an outage with an upstream feature flag provider that's been impacted by the AWS outage. We expect the upstream service to be increasing operational availability shortly based on our communications with them. In the meantime, we're investigating interventions we can do on our end to reduce impact.
We'll share an update in the next 30 minutes.
Oct 20, 14:35 PDT
Update -
We've implemented a workaround to bypass the affected services. All systems are returning to operational.
Separately, we're still monitoring the root infrastructure AWS outage.
Oct 20, 14:18 PDT
Monitoring -
We're seeing service start to come back up as a result of AWS services slowly being brought back to an operational state. We're implementing an infrastructure workaround at the moment to bypass down services. Either way, we're expecting the Command AI service to be up in the next 30 minutes. We'll post an update in 15 minutes.
Oct 20, 14:02 PDT
Update -
We've confirmed a major outage and identified the problem as an upstream infrastructure issue due to a networking outage at AWS. This is impacting how traffic is routed to our systems. We're actively monitoring the situation.
Oct 20, 12:16 PDT
Investigating -
We're currently investigating internal alerts indicating an API outage
Oct 20, 11:56 PDT