Back to overview

Partial Outage

Dec 18 at 05:00pm UTC
Affected services
Licensing & Distribution API

Dec 18 at 05:00pm UTC

Today at 7am CST, we experienced a partial outage due to elevated API volume. We scaled up our infrastructure to accommodate the unexpected volume, but due to other factors, we continued to see elevated error rates. This was due to our event backlog filling up faster than we could process (due to multiple factors including webhook delivery failures), eventually leading to an out-of-memory failure on our Redis datastores, resulting in further failed requests. We resolved the webhook-related failures, which relieved pressure on our Redis datastores, and scaled up our worker formation to process the backlog as quick a possible. Full service was restored at 8am CST, but the event logs were still delayed. As of 10am CST, the backlog has been cleared and service is operating normally. We apologize for the inconvenience. As part of our database upgrade later tonight, we will be better prepared for unexpected spikes in volume like this moving forward.