Incidents and maintenance reported on status page for Keygen https://status.keygen.sh/ https://d1lppblt9t2x15.cloudfront.net/logos/31861f55c16793ad623efe5319b9b6b9.png https://status.keygen.sh/ en https://status.keygen.sh/maintenance/989035 Fri, 31 Jul 2026 21:11:38 -0000 https://status.keygen.sh/incident/989035#5e1995e57c1654c22606e171b27d6b3f4526f0d515eaa4232ef53acc82904295 Maintenance We are planning to upgrade our Postgres databases on 2026-08-08 at 07:00:00 UTC. During the changeover, there may be up to 5 minutes of downtime, but it should largely go unnoticed. https://status.keygen.sh/maintenance/947194 Wed, 08 Jul 2026 13:27:09 -0000 https://status.keygen.sh/incident/947194#aaf8022319b23c74fd2e7d9c992229fbe3fd4edcdabbd62e6bf673b69a277181 Maintenance We will be performing scheduled maintenance on our Redis datastores on 2026-07-12 00:00:00 UTC. During the changeover there may be up to 5 minutes of downtime. https://status.keygen.sh/maintenance/898778 Mon, 18 May 2026 14:33:14 -0000 https://status.keygen.sh/incident/898778#0c1dbf31b3940bf14a3382899eea2719d672d40435d756eb71ab77a4c4144760 Maintenance We are planning to upgrade our Postgres databases on 2026-05-23 at 07:00:00 UTC. During the changeover, there may be up to 5 minutes of downtime, but it should largely go unnoticed. https://status.keygen.sh/incident/890678 Fri, 08 May 2026 04:44:00 -0000 https://status.keygen.sh/incident/890678#22b2f33af52e151736c1ec3d13a75ae32005d642eae6dd244743ceefb04d6101 Incident Our services no longer appear to be affected by the ongoing AWS outage following traffic reroutes on their end. We are continuing to monitor the situation and will post updates here if anything changes. https://status.keygen.sh/incident/890678 Fri, 08 May 2026 03:10:00 -0000 https://status.keygen.sh/incident/890678#d2dfae5c5082d6a89fe5bb5eef3f1425dc9478fc3b804d560a2e2160ebd732be Incident We experiencing degraded performance at this time due to an ongoing AWS outage. Our team is investigating the issue and we will provide an update when one becomes available. Sorry for the inconvenience. For more information, see: https://health.aws.amazon.com/health/status https://status.keygen.sh/maintenance/878955 Fri, 24 Apr 2026 14:18:31 -0000 https://status.keygen.sh/incident/878955#7e445d052a38488dac4798f114bf60caae3e77ebe703dea4d050b456ee3003b9 Maintenance We will be performing scheduled maintenance on our Redis datastores on 2026-04-26 00:00:00 UTC. During the changeover there may be up to 5 minutes of downtime. https://status.keygen.sh/maintenance/853839 Fri, 20 Mar 2026 20:02:04 -0000 https://status.keygen.sh/incident/853839#46a613c44168c0586675706f9afa19bc44dd4bfca0a9bac078662c13ccf438ec Maintenance We will be performing scheduled maintenance on our Postgres datastores on 2026-03-28 07:00:00 UTC. During the changeover there may be up to 5 minutes of downtime. https://status.keygen.sh/maintenance/817684 Tue, 03 Feb 2026 15:25:37 -0000 https://status.keygen.sh/incident/817684#7287235a3a7128eac2c5d0da35762f628dd12f7f7b5c180f05f65d31e9cdc8d0 Maintenance On 2026-02-08 at 00:00 UTC, we will be performing scheduled maintenance on our Redis cluster. There may be a brief period of downtime during the change over. https://status.keygen.sh/maintenance/797106 Thu, 01 Jan 2026 17:25:45 -0000 https://status.keygen.sh/incident/797106#748f0d7e784ce87e4d055178c9dedd6a5916f9b09703c2c988235e582a04e2cb Maintenance We will be upgrading our production Postgres databases on 2026-01-10 at 07:00:00 UTC. Please expect a brief period of downtime while we finalize the upgrade and perform a changeover. https://status.keygen.sh/maintenance/773269 Mon, 24 Nov 2025 16:06:37 -0000 https://status.keygen.sh/incident/773269#8e53f092502644b0452f90f604e8b10ab7c1a73d8b18f8d726c61db5fa6d10fa Maintenance We will be performing scheduled maintenance on our Postgres datastores on 2025-11-29 at 07:00:00 UTC. During the changeover there may be up to 5 minutes of downtime. https://status.keygen.sh/incident/747478 Mon, 20 Oct 2025 18:20:00 -0000 https://status.keygen.sh/incident/747478#54f98ed3dba08202c2fcaecea57277695e2bf6d379af3501f7cb15611361e07b Incident This issue has been resolved. https://status.keygen.sh/incident/747478 Mon, 20 Oct 2025 16:10:00 -0000 https://status.keygen.sh/incident/747478#27ba4648f914f37fa67ab3684b2307bbecf7015220b6e82a05f05df945b7a6a1 Incident We are currently experiencing an outage with an upstream infrastructure provider, Heroku. We are attempting to resolve the issue as quickly as possible. We will post updates here when we have them. Heroku's status is available here: https://status.salesforce.com/generalmessages/20000226 https://status.keygen.sh/maintenance/744797 Thu, 16 Oct 2025 13:49:43 -0000 https://status.keygen.sh/incident/744797#af2a229645c939c49dea6dd34042ef6349ba08a1e30f5dbe36a0aa23b387c959 Maintenance On 2025-10-26 at 00:00 UTC, we will be performing scheduled maintenance on our Redis cluster. There may be a brief period of downtime during the change over. https://status.keygen.sh/maintenance/721176 Tue, 09 Sep 2025 13:49:06 -0000 https://status.keygen.sh/incident/721176#8e5b9beb24cf1db7f3d3525f79b2aa4ba53b7af83a320c983aeb90a922711cb9 Maintenance We will be performing scheduled maintenance on our Postgres datastores on 2025-09-13 at 07:00:00 UTC. During the changeover there may be up to 5 minutes of downtime. https://status.keygen.sh/maintenance/700175 Mon, 04 Aug 2025 14:14:01 -0000 https://status.keygen.sh/incident/700175#9215aba98521065dd6cf46a0473accb9bc80ee64656866b32362b91b43ec5e41 Maintenance On 2025-08-10 at 00:00 UTC, we will be performing scheduled maintenance on our Redis cluster. There may be a brief period of downtime during the change over. https://status.keygen.sh/maintenance/608311 Tue, 24 Jun 2025 13:55:14 -0000 https://status.keygen.sh/incident/608311#75dfaf4873468afd8848f25506bf06b8a730f33102ca2f0d539a206404dee228 Maintenance On 2025-06-28 at 07:00:00 UTC, we'll be performing a scheduled upgrade of our Postgres databases. There may be a brief period of scheduled downtime lasting up to 5 minutes during the changeover process. https://status.keygen.sh/maintenance/534929 Wed, 26 Mar 2025 13:55:57 -0000 https://status.keygen.sh/incident/534929#a87af0345c993f397943ed8b8c3df3764ff7785a0636a796a3ab576653d484a3 Maintenance We will be performing scheduled maintenance on our Postgres datastores on 2025-03-29 at 07:00:00 UTC. During the changeover there may be up to 5 minutes of downtime. https://status.keygen.sh/maintenance/526421 Tue, 11 Mar 2025 12:09:57 -0000 https://status.keygen.sh/incident/526421#6bdd7f94601ed26f436e3257a5b45b59b7a27f1804c3fcb373bde1270d8f42c0 Maintenance We will be performing a scheduled upgrade of our Redis datastores on March 16th 2025 at 12am UTC. During the upgrade and changeover there may be up to 5 minutes of downtime. https://status.keygen.sh/maintenance/492144 Tue, 07 Jan 2025 18:56:09 -0000 https://status.keygen.sh/incident/492144#9fa094d8bd96f92f15c1960ad042d127e0b88d859f425c45e8b9625d6d655219 Maintenance We're seeing a volume increase this year across multiple large accounts, and we need to upgrade our database infrastructure to support this increase moving forward. We will be taking an outage tonight to facilitate the upgrade and changeover. We will attempt to minimize downtime, but expect at least 5 minutes of downtime during the changeover. Request timing may increase or become sporadic throughout the day as we prepare the database for the upgrade procedure. Sorry for the inconvenience. https://status.keygen.sh/maintenance/445793 Wed, 16 Oct 2024 19:49:04 -0000 https://status.keygen.sh/incident/445793#c710d2ffe273dae623fd2348ec513aa95c92bcfd5df9dd7980cb997fd8f5acc6 Maintenance We will be performing scheduled maintenance on our Postgres datastores on 2024-10-26 at 07:00:00 UTC. During the changeover there may be up to 5 minutes of downtime. https://status.keygen.sh/maintenance/440345 Mon, 07 Oct 2024 01:29:42 -0000 https://status.keygen.sh/incident/440345#e82dd5e082bb79a5d0325751a4e03097467b8cad0ca06c8efe0442f9b401d2d7 Maintenance We will be performing a scheduled upgrade of our Redis datastores on October 10th 2024 at 2am UTC (i.e. the 9th @ 9pm CST). During the upgrade and changeover there may be up to 5 minutes of downtime. https://status.keygen.sh/maintenance/422353 Sun, 01 Sep 2024 00:00:00 -0000 https://status.keygen.sh/incident/422353#f3d08247babb2a092a58f0117d36263f88d52803cb1c9459716eb6486bcbef9c Maintenance We started rotating the TLS certificates for our Redis databases. There may be up to 5 minutes of downtime while the services restart. https://status.keygen.sh/maintenance/407981 Fri, 02 Aug 2024 15:23:23 -0000 https://status.keygen.sh/incident/407981#5c68f06c30e7914aba1bfa58a674970729bd30001fd9f30a309ceebc0d07b479 Maintenance We will be performing scheduled maintenance on our Redis databases at 2024-08-11 00:00:00 UTC. There may be up to 5 minutes of downtime as we do a changeover, but it should largely go unnoticed. https://status.keygen.sh/maintenance/407561 Thu, 01 Aug 2024 21:18:38 -0000 https://status.keygen.sh/incident/407561#8c7aec7c5e458b4be98a67ca1da71dd5f283d06354f32d964703ff825c877dd9 Maintenance We will be performing scheduled maintenance on our Postgres databases at 2024-08-10 07:00:00 UTC. There may be up to 5 minutes of downtime as we do a changeover, but it should largely go unnoticed. https://status.keygen.sh/maintenance/371330 Sat, 18 May 2024 15:18:28 -0000 https://status.keygen.sh/incident/371330#29341780c4414419c2568bac3086ded8b3660d5f5959cb0448821fa96d444835 Maintenance We will be performing scheduled maintenance on our Redis datastores on 2024-05-26 00:00:00 UTC. During the changeover process, there may be up to 5 minutes of downtime. https://status.keygen.sh/maintenance/334436 Fri, 01 Mar 2024 21:59:05 -0000 https://status.keygen.sh/incident/334436#54927b5d2ae2801b4b53da5607c14d8214262900b9079256b05bdb3b57044096 Maintenance We will be performing scheduled maintenance on our Postgres datastores on 2024-03-09 07:00:00 UTC. During the changeover there may be up to 5 minutes of downtime. https://status.keygen.sh/incident/333245 Wed, 28 Feb 2024 16:24:00 -0000 https://status.keygen.sh/incident/333245#8510c05c85d0c08a735cb8f4b71edf6e724ab8f2aa9fb91db0ca66b4485153e1 Incident The deployment has been rolled back and API performance has returned to normal. https://status.keygen.sh/incident/333245 Wed, 28 Feb 2024 16:17:00 -0000 https://status.keygen.sh/incident/333245#7f083516c5ff43e91430abba33b8626077821db38928737732e75413639744f5 Incident We are aware of degraded API performance due to a bad deploy. We are working on rolling back the changes now. We apologize for the inconvenience. https://status.keygen.sh/incident/322421 Tue, 06 Feb 2024 20:04:00 -0000 https://status.keygen.sh/incident/322421#85b20b34c2eec955264963d89ab22f32a1404a90178d4cdbad07ab378e78f8e8 Incident We were able to locate the source of the problem on our end that was limiting our scaling under certain loads. We've deployed a fix and are now accepting all traffic. If there's a silver lining, it's that we fixed a long standing performance problem that reared itself sporadically, and now we know why. We're going to write up a complete postmortem soon. Thanks so much for your patience, and I sincerely apologize for the problems this outage has caused all your businesses over the last 36 hours. It's been a nightmare, but we came out stronger. https://status.keygen.sh/incident/322421 Tue, 06 Feb 2024 12:14:00 -0000 https://status.keygen.sh/incident/322421#4454d249bdf949013e486097ac04ac512c38a4d76df5e6bcb14f53807a533e64 Incident Services have stabilized, but we are continuing to load shed the DDoS. We are continuing to monitor the incident and scale up infrastructure to handle the next traffic spike without downtime. https://status.keygen.sh/incident/322421 Tue, 06 Feb 2024 10:06:00 -0000 https://status.keygen.sh/incident/322421#dce7783abab5cc2b66def11f610ddf6fb6521a09f7c1db9ae76549c971ac0ee9 Incident We're aware of the same issue happening again. We're working on restoring services. https://status.keygen.sh/incident/322421 Mon, 05 Feb 2024 21:01:00 -0000 https://status.keygen.sh/incident/322421#4b4bc2cfe4b9b8140616e8dc5237a8f9a7b7c9856ad1e7328f83039a6a46013b Incident Today, we experienced an unintentional DDoS. It was not an attack as we originally reported (we were unaware of its nature). It was largely due to an unexpected increase in API volume, which caused a cascading series of events that ultimately caused an outage that lasted over 5 hours. This is unacceptable, and I apologize. The timestamps in this root cause analysis will be in CST. At 12:30am CST, we experienced a large, sudden surge in legitimate traffic. This caused our autoscaler to kick in and scale up infrastructure. But because the source of the traffic surge was legitmate traffic, we were unable to rate limit effectively. By the time our infrastructure was scaled up enough to sustain the traffic, multiple systems had began failing due to a backup in job processessing. To make matters worse, the source of the traffic had an aggressive retry algorithm configured, where instead of backing off exponentially, it sped up, which effectively DDoS'd us with the volume of traffic. Three hours elapsed before we were made aware of the incident. At 4am CST, we were able to page an engineer to investigate. This was a terrible response time, and we'll cover what went wrong here at the end. But at 4am, we were aware of and unraveling the incident, and formulating a response. We noticed some outlier IPs i.r.t. volume, and we immediately began blacklisting the IP addresses that were sending higher-than-normal API volume. But that wasn't enough. And there were too many IPs. The volume was distributed, and as far as we could tell, it was legitimate traffic. This made things complicated. To prevent further strain on our systems, we enabled Cloudflare's "Under Attack" mode, which effectively blocked the majority of API requests (since an API client cannot solve a Cloudflare "challenge response"). We wished there was another way, but Cloudflare's WAF is not very friendly to API-only products. We ultimately did this so that our systems could recover, and to give us time to determine next steps. Our Redis instances were at capacity due to the number of jobs to process, and this "pause" gave our systems time to recover. Around 5am, once our systems were sufficiently recovered, we disabled Cloudflare's "Under Attack" mode. But the DDoS was still ongoing, and we determined the source of the traffic was originating from a single account. We began sending challenge responses to requests matching that account, effectively blocking traffic until we could deal with it in a more controlled fashion. After getting in contact with the account owner, we determined that the traffic was indeed legitimate. We began letting in a percentage of the traffic until their applications were able to stabilize and cease retries (which was effectively the source of the DDoS). We then began receiving reports that customers were experiencing intermittent signature verification failures. After investigation, this was due to Cloudflare modifying the Date header we send, which is a part of a response's cryptographic signature. It was intermittent because it was dependent upon the current time. For example, let's assume a request has a timestamp of: 2024-02-04T23:01:10.993239 This results in us sending the following Date header: Date: Sun, 04 Feb 2024 23:01:10 GMT Unfortunately, the Date header is rewritten by Cloudflare to the following: Date: Sun, 04 Feb 2024 23:01:11 GMT Note the difference in the second portion. By the time Cloudflare sent the response, the Date had changed, because we were at milisecond .993239. This change in Date is what was causing the signature verification failures, since the Date header is used in the response signature to prevent timing attacks. At 5am, service was restored for all accounts, and API volume was back to a normal level. At this point, we disabled Cloudflare's CDN to resolve the signature verification issue. Our request logs and event logs were delayed for another hour before normalizing. It's unacceptable that it took over 3 hours to acknowledge the incident, and ultimately 5 hours of downtime, and I apologize for that. I was the on-call engineer, and I failed that duty. A few things went wrong: 1. Our pager service was not configured to send a phone call due to an oversight in the on-call configuration. 2. The on-call engineer's phone was placed on DnD mode (do-not-disturb) at 12am CST. 3. The pager service's text notifications were not whitelisted from DnD mode. 4. The email alerts were not seen due to DnD mode being enabled (I was asleep). 5. There were no escalation policies for paging more aggresively. I sincerely apologize for this incident. I feel terrible. I feel defeated. I haven't slept. It's been a day of putting out fires in the aftermath, as I'm sure it has been for you. A lot of things went wrong, and I'm still trying to piece things together. But the above is what I do know now. I'm looking at ways to prevent this from happening again. From scaling up infrastructure, to increasing datastore capacity, to finding a new WAF provider that is compatible with our API. I'm in talks with multiple providers. I understand licensing is a vital part of a software business, and we regret this happened today. I'm sorry. https://status.keygen.sh/incident/322421 Mon, 05 Feb 2024 15:44:00 -0000 https://status.keygen.sh/incident/322421#2dabb311a4f57ebdafe71c409226dbf19d1fc36b918c2443b7a6929ebd4f125b Incident API availability has returned to normal, and response signatures should now be working as expected. The bulk of the traffic has been mitigated, but we are continuing to monitor the situation as overall request volume is still elevated. If another surge occurs, we will update this report. We have a lot of data in our backlog, so request/event logs and webhooks may be delayed. https://status.keygen.sh/incident/322421 Mon, 05 Feb 2024 13:50:00 -0000 https://status.keygen.sh/incident/322421#f7b460fff0426b49caa32e9abe986b65a3239d40beed91d00881cf4a74746848 Incident We've employed Cloudflare to mitigate the issue, but this has had an effect on signature verifications. Due to the way Cloudflare operates, the Date header Cloudflare overwrites may not match the Date header Keygen sends, resulting in signature verification errors. If possible, use the Keygen-Date header for signature verification purposes, instead of Date, to mitigate this issue until we're able to resolve this issue with Cloudflare. https://status.keygen.sh/incident/322421 Mon, 05 Feb 2024 11:27:00 -0000 https://status.keygen.sh/incident/322421#2e2f9dcbb677d189d76a2cb71e9204cdd2a060b9b4d8fb6af4975c5ee7b1c366 Incident We are still under an active DDoS attack, but we have put in place mitigations. Most API traffic has returned to normal, but we are still working with our infrastructure providers on a full resolution. Some accounts may still be inaccessible due to the nature of the attack. Thank you for your patience. https://status.keygen.sh/incident/322421 Mon, 05 Feb 2024 10:12:00 -0000 https://status.keygen.sh/incident/322421#f4ff61ad26915ed94cad879fc17f1223231cc932604ad37323b397b9604aaf9a Incident As of 23:48:02, we have been experiencing a major DDoS attack. We're working with our infrastructure provider on a resolution. Thank you for your patience. https://status.keygen.sh/incident/317300 Thu, 25 Jan 2024 02:39:00 -0000 https://status.keygen.sh/incident/317300#62ed0b907000dd6761504e7d3aa4952c0c75766cbb3956659f62d082f34a355c Incident Everything looks to be back to normal. Please reach out to support@keygen.sh if you encounter any issues, or if you have further questions about what occurred. https://status.keygen.sh/incident/317300 Wed, 24 Jan 2024 22:37:00 -0000 https://status.keygen.sh/incident/317300#ed269937fcfc2beed6a66e30cba357faa341735f6bb24e9ca7ab25b47ba5b5db Incident Today, we attempted to deploy an upgrade to our application that effected encrypted columns in our databases. Unfortunately, during the rollout, we discovered that lookups on certain encrypted attributes, such as license key, began to fail. We quickly found the cause of the problem, rolled back, and eventually deployed a fix. During this time, you may have received 401 or 404 responses indicating that a license key did not exist, when it actually did. We are continuing to monitor the situation and will be keeping this status update open for the next couple hours. No data was lost during this incident; only lookups were effected. https://status.keygen.sh/incident/302160 Mon, 18 Dec 2023 17:00:00 -0000 https://status.keygen.sh/incident/302160#da1f28934fe22c81df8ff3dacefdad60c3714bbfc624a6787fd7af18788fe93b Incident Today at 7am CST, we experienced a partial outage due to elevated API volume. We scaled up our infrastructure to accommodate the unexpected volume, but due to other factors, we continued to see elevated error rates. This was due to our event backlog filling up faster than we could process (due to multiple factors including webhook delivery failures), eventually leading to an out-of-memory failure on our Redis datastores, resulting in further failed requests. We resolved the webhook-related failures, which relieved pressure on our Redis datastores, and scaled up our worker formation to process the backlog as quick a possible. Full service was restored at 8am CST, but the event logs were still delayed. As of 10am CST, the backlog has been cleared and service is operating normally. We apologize for the inconvenience. As part of our database upgrade later tonight, we will be better prepared for unexpected spikes in volume like this moving forward. https://status.keygen.sh/maintenance/301275 Sat, 16 Dec 2023 06:13:12 -0000 https://status.keygen.sh/incident/301275#b4b14c2e971fd20186d08e115d212cbc022550cc9cb6e3273c4b69737c3dc582 Maintenance We will be performing a scheduled upgrade of our Postgres datastores on December 19th 2023 at 3am UTC (i.e. the 18th @ 9pm CST). During the upgrade and changeover there may be up to 30 minutes of downtime. https://status.keygen.sh/maintenance/282013 Fri, 03 Nov 2023 15:56:22 -0000 https://status.keygen.sh/incident/282013#4c19ec2ea18174eb6acf9c2c73a693f6c7c2585fe6dae307c88f928301963b4c Maintenance We will be performing scheduled maintenance on our Postgres datastores on November 11th 2023 at 7am UTC. During the changeover there may be up to 5 minutes of downtime. https://status.keygen.sh/maintenance/248726 Mon, 21 Aug 2023 15:59:43 -0000 https://status.keygen.sh/incident/248726#b654c4389cf262486356814edefe1968e04b3bde5565f96cad2f2069a686c4ca Maintenance We will be performing scheduled maintenance on our Postgres datastores on August 26th 2023 at 7am UTC. During the changeover there may be up to 5 minutes of downtime, but it should largely go unnoticed. https://status.keygen.sh/maintenance/239802 Mon, 31 Jul 2023 21:19:33 -0000 https://status.keygen.sh/incident/239802#c85025364b7e0ba8c39d76c12fa8fe012026b6fa0e81ca41926e1e145d25ec09 Maintenance We will be upgrading our Redis datastores on August 6th 2023 at 12am UTC. During the changeover process, there may be up to 5 minutes of downtime. https://status.keygen.sh/maintenance/216375 Mon, 05 Jun 2023 17:14:25 -0000 https://status.keygen.sh/incident/216375#c0dad15fb663fd3518fd18a90c0a74cfb5f86d487da77c6a978346f4fda9c951 Maintenance We will be performing scheduled maintenance on our Postgres datastores on June 10th 2023 at 12am UTC. During the changeover there may be up to 5 minutes of downtime, but it should largely go unnoticed. https://status.keygen.sh/maintenance/193603 Thu, 06 Apr 2023 15:44:59 -0000 https://status.keygen.sh/incident/193603#81a62c028c499c0013d5fe4a4d5fd0ac0c69bca20ea4f00ffba5fa0ff38744cb Maintenance We will be upgrading our Redis datastores on Tuesday, April 10th 2023 at 4am UTC (10pm CST, 8pm PST). During the changeover process, there may be up to 5 minutes of downtime. https://status.keygen.sh/maintenance/188017 Wed, 22 Mar 2023 18:53:18 -0000 https://status.keygen.sh/incident/188017#27db759f11d87c0b3535bf672086c8da026e03cec27ef30601d530cac22e2166 Maintenance Routine database maintenance. The API may be unavailable for up to 5 minutes while we do a changeover. Time is UTC. https://status.keygen.sh/maintenance/150963 Thu, 05 Jan 2023 17:11:31 -0000 https://status.keygen.sh/incident/150963#2d57454c24cf0664300bcc5f9682ed11b746386d8a532f70f56a1a4cbde7035b Maintenance Routine database maintenance. The API may be unavailable for up to 5 minutes as we do a changeover. Scheduled time is 2023-01-15 00:00:00 UTC. https://status.keygen.sh/maintenance/129277 Mon, 24 Oct 2022 15:34:28 -0000 https://status.keygen.sh/incident/129277#597f7f2179bc217c677da3394b24cdde05b80489fb155740771c5edda82d30ae Maintenance We'll be performing routine database maintenance on our Redis cluster. There may be a small amount of downtime while we do a change over, but this maintenance should largely go unnoticed. Scheduled time is 2022-10-30 00:00:00 UTC. https://status.keygen.sh/maintenance/118976 Sun, 18 Sep 2022 21:32:27 -0000 https://status.keygen.sh/incident/118976#997948d6ff539f420ab88e97930e6cec3bc6ac631ddf4f7827ea6b2f0f8cf075 Maintenance We will be performing maintenance on our Postgres cluster. No downtime is expected, but some requests may be delayed for a moment while we switch over to our HA standbys. Scheduled time is 2022-09-24 07:00:00 UTC. https://status.keygen.sh/incident/112994 Fri, 26 Aug 2022 15:27:00 -0000 https://status.keygen.sh/incident/112994#984244549636057fda7a36404c6ba69e0bc40722c85ace37372e93734402997b Incident Looks to have been short lived. We’ll continue to monitor the situation, but it looks to be resolved. https://status.keygen.sh/incident/112994 Fri, 26 Aug 2022 15:23:00 -0000 https://status.keygen.sh/incident/112994#9681d08b50fe4e4fb5816978a9f82ccee36c5dcae66bdc70d8c58c18a7d5e226 Incident We are experiencing DNS issues again due to an upstream provider. We are working to resolve the issue. https://status.keygen.sh/incident/112108 Wed, 24 Aug 2022 01:00:00 -0000 https://status.keygen.sh/incident/112108#0987fe937501f1644b6f5d662a416d0df400c3d7df077319669a45d2bc18aa0b Incident This is now resolved. https://status.keygen.sh/incident/112108 Tue, 23 Aug 2022 23:28:00 -0000 https://status.keygen.sh/incident/112108#fb7329dfde0a5d73e8fba52bf089f89dcc79d33f0a52432556688d1080b6658a Incident We are investigating issues with an upstream DNS provider. More information is available here: https://status.heroku.com/ We are working with Heroku to resolve the issue. https://status.keygen.sh/maintenance/98608 Wed, 29 Jun 2022 16:18:36 -0000 https://status.keygen.sh/incident/98608#97dee54f9a7c817a6d89393a8aa5dee4e7239705fd332070f8d4a948fd933730 Maintenance We will be performing scheduled software upgrades on our production databases. Potential downtime is up to 30 minutes but expected to be much shorter. Scheduled time is 02:00:00 UTC timezone. https://status.keygen.sh/maintenance/93468 Wed, 08 Jun 2022 15:30:00 -0000 https://status.keygen.sh/incident/93468#22f9f939efb71972c0a5e3817920696a92193905fe7310e8a4161e1d97820295 Maintenance We're currently performing maintenance on request logs and other types of metric logs. Dashboard metrics and logs may be intermittently unavailable until complete. We apologize for any inconvenience. https://status.keygen.sh/maintenance/82730 Tue, 19 Apr 2022 02:52:46 -0000 https://status.keygen.sh/incident/82730#e840f02bc343c2f11434e01f52c3c54b775ba8f10faa232a88d9268cfe9ee4a4 Maintenance Scheduled maintenance window for our databases. The service may be unavailable for a couple minutes as we do a changeover. https://status.keygen.sh/maintenance/79461 Fri, 01 Apr 2022 21:47:00 -0000 https://status.keygen.sh/incident/79461#d1b1b0eee25d1f4901be4735daad9e30c4754a822dac5c25be91d84c2eba8ae9 Maintenance We'll be performing maintenance on our Redis datastores. There may be a few minutes of downtime while we do a changeover. https://status.keygen.sh/incident/72656 Thu, 24 Feb 2022 17:35:00 -0000 https://status.keygen.sh/incident/72656#9141f4431593c30bf83d9ce834d8526b72c909be76f1e6ca5bfb691ea9bdfbbb Incident Webhooks have fully recovered. All systems are back to normal. https://status.keygen.sh/incident/72656 Thu, 24 Feb 2022 17:18:00 -0000 https://status.keygen.sh/incident/72656#b5240b7a7d0cbb22769bd6eb1fba2e818055e74603c51f97d7facac20ace7b2b Incident All services have recovered. It may take awhile to work through our webhook backlog. We will continue to monitor the situation. https://status.keygen.sh/incident/72656 Thu, 24 Feb 2022 17:16:00 -0000 https://status.keygen.sh/incident/72656#8b2f1a3203ff75d556036bb7e1ddcc179e3f0aabe256c461ef169d3c5e012d8a Incident Platform functionality is starting to recover. We will continue to monitor as things recover further. Thank you again for your patience. https://status.keygen.sh/incident/72656 Thu, 24 Feb 2022 17:10:00 -0000 https://status.keygen.sh/incident/72656#a8897dacc16714ae81ec7d0a5028ab7a68694964b45e07b0db8d0dcf0c275b69 Incident We are still waiting on Heroku to restore platform functionality. We have heard reports of some applications recovering, but Keygen is largely still down. Since we have been down for over 30 minutes, we are initiating disaster recovery procedures. We sincerely apologize for the trouble. https://status.keygen.sh/incident/72656 Thu, 24 Feb 2022 16:36:00 -0000 https://status.keygen.sh/incident/72656#f0bca151923350d96f50d6dc9fc6b5fd9970e037eb56c14e983fb548716aacf1 Incident We are currently experiencing an outage with an upstream infrastructure provider, Heroku. We are attempting to resolve the issue as quickly as possible. We will post updates here when we have them. Heroku's status is available here: https://status.heroku.com. Comments: https://news.ycombinator.com/item?id=30456688. https://status.keygen.sh/maintenance/69133 Thu, 03 Feb 2022 14:05:46 -0000 https://status.keygen.sh/incident/69133#1d01bc40dda232aaacbf5bb5b448ee966e6834b443ec29e638b0b29da220a04d Maintenance Scheduled maintenance window for our databases. The service may be unavailable for a couple minutes as we do a changeover. https://status.keygen.sh/maintenance/68781 Mon, 31 Jan 2022 18:00:00 -0000 https://status.keygen.sh/incident/68781#ed390fc70f5fcdf55d8d4de3760301c9c6b85bf674eb59116cfc8dcf18b198e1 Maintenance This migration should largely go unnoticed. But for clients running an outdated version of OpenSSL, clients may begin receiving CERT_HAS_EXPIRED errors. More info here: https://keygen.sh/docs/api/connectivity/ https://status.keygen.sh/maintenance/67359 Sun, 23 Jan 2022 00:00:00 -0000 https://status.keygen.sh/incident/67359#bbe26b16d0f794bddaa381637ad388dc62c91aff2191023e1118462634e39d3f Maintenance Scheduled maintenance window for our cache datastores. The service may be unavailable for a couple minutes as we do a changeover. https://status.keygen.sh/maintenance/58967 Thu, 11 Nov 2021 22:05:48 -0000 https://status.keygen.sh/incident/58967#8bbd511fe1acc5ecfcca2b47dba253b1efa19ddc70005f95e02a490be3bdc7b9 Maintenance We will be performing scheduled software upgrades to our production databases. Expected downtime is up to 30 minutes. Scheduled time is 03:30:00 UTC timezone. https://status.keygen.sh/maintenance/55704 Sun, 03 Oct 2021 00:00:00 -0000 https://status.keygen.sh/incident/55704#dab9a5eb25869156b821b73823c9ba8a653586d4d1598e6a1b845b1f4e78ef74 Maintenance Routine database maintenance. The API may be unavailable for up to 5 minutes. https://status.keygen.sh/incident/55557 Fri, 01 Oct 2021 04:27:27 -0000 https://status.keygen.sh/incident/55557#286c97e73d241af68617fe5b39679b01ff28e426148d05f39088294ee5d7ddb1 Incident Due to the widespread issues caused by Lets Encrypt and lack of support for outdated OpenSSL clients, we have moved to a Comodo TLS certificate. Moving forward, this should resolve all TLS connection issues caused by the expired Lets Encrypt root certificate, DST Root CA X3. We unfortunately experienced around 10–15 minutes of downtime during the migration to the new CA due to a misconfiguration on our end, but everything should be green now. Please reach out if you continue to experience any issues. https://status.keygen.sh/incident/55497 Thu, 30 Sep 2021 15:26:41 -0000 https://status.keygen.sh/incident/55497#c9638f6f221a6be6fe46745bfe9edb4436784c08647da00cf65a8951b4fc69e4 Incident We are seeing reports of some customers receiving an expired TLS cerificate error when connecting to our APIs. Lets Encrypt's older root cert, DST Root CA X3, expired today, and we are working with customers to resolve this issue for affected devices. To resolve, Let's Encrypt's new root cert, ISRG Root X1, needs to be trusted. Unfortunately, this is currently out of our control and older clients must be updated to trust the new CA Please upgrade your ca-certificates. For example — on Unix, you will need to upgrade the ca-certificates package; on macOS, you may need to reboot, reinstall OpenSSL >= 1.1 and then upgrade curl to the latest version via Homebrew. Additional reading and steps to resolve the issues on devices that do not have an up-to-date certificate chain: https://letsencrypt.org/docs/dst-root-ca-x3-expiration-september-2021/ Other discussions and links that may help: - https://twitter.com/Scott_Helme/status/1443293844292919304 - https://community.letsencrypt.org/t/help-thread-for-dst-root-ca-x3-expiration-september-2021/149190/424 - https://github.com/mperham/sidekiq/issues/5008#issuecomment-931543300 - https://www.kobelnet.ch/2021/09/30/old-lets-encrypt-root-certificate-expiration-workaround - https://twitter.com/search?q=letsencrypt&src=typed_query&f=live We have marked this incident as resolved because it is a client issue, not a server issue. Though, we will continue to monitor if there's anything we can do on our end to lessen its impact. Please reach out if you need assistance upgrading.