Maintenance

Database Maintenance

We will be performing scheduled software upgrades to our production databases. Expected downtime is up to 30 minutes. Scheduled time is 03:30:00 UTC timezone.

Maintenance

Database maintenance

Routine database maintenance. The API may be unavailable for up to 5 minutes.

Resolved

Migrated to Comodo TLS Certificate

Due to the widespread issues caused by Lets Encrypt and lack of support for outdated OpenSSL clients, we have moved to a Comodo TLS certificate. Moving forward, this should resolve all TLS connection issues caused by the expired Lets Encrypt root certificate, DST Root CA X3.

We unfortunately experienced around 10–15 minutes of downtime during the migration to the new CA due to a misconfiguration on our end, but everything should be green now.

Please reach out if you continue to experience any issues.

Resolved

Expired TLS Certificate Errors

We are seeing reports of some customers receiving an expired TLS cerificate error when connecting to our APIs. Lets Encrypt's older root cert, DST Root CA X3, expired today, and we are working with customers to resolve this issue for affected devices. To resolve, Let's Encrypt's new root cert, ISRG Root X1, needs to be trusted. Unfortunately, this is currently out of our control and older clients must be updated to trust the new CA

Please upgrade your ca-certificates. For example — on Unix, you will need to upgrade the ca-certificates package; on macOS, you may need to reboot, reinstall OpenSSL >= 1.1 and then upgrade curl to the latest version via Homebrew.

Additional reading and steps to resolve the issues on devices that do not have an up-to-date certificate chain: https://letsencrypt.org/docs/dst-root-ca-x3-expiration-september-2021/

Other discussions and links that may help:

- https://twitter.com/Scott_Helme/status/1443293844292919304
- https://community.letsencrypt.org/t/help-thread-for-dst-root-ca-x3-expiration-september-2021/149190/424
- https://github.com/mperham/sidekiq/issues/5008#issuecomment-931543300
- https://www.kobelnet.ch/2021/09/30/old-lets-encrypt-root-certificate-expiration-workaround
- https://twitter.com/search?q=letsencrypt&src=typed_query&f=live

We have marked this incident as resolved because it is a client issue, not a server issue. Though, we will continue to monitor if there's anything we can do on our end to lessen its impact. Please reach out if you need assistance upgrading.