Previous incidents
Scheduled Database Upgrade
Resolved Dec 19 at 03:00am UTC
We will be performing a scheduled upgrade of our Postgres datastores on December 19th 2023 at 3am UTC (i.e. the 18th @ 9pm CST). During the upgrade and changeover there may be up to 30 minutes of downtime.
Partial Outage
Resolved Dec 18 at 05:00pm UTC
Today at 7am CST, we experienced a partial outage due to elevated API volume. We scaled up our infrastructure to accommodate the unexpected volume, but due to other factors, we continued to see elevated error rates. This was due to our event backlog filling up faster than we could process (due to multiple factors including webhook delivery failures), eventually leading to an out-of-memory failure on our Redis datastores, resulting in further failed requests. We resolved the webhook-related fa...