Incidents reported on status page for Keygen https://status.keygen.sh/ https://d1lppblt9t2x15.cloudfront.net/logos/31861f55c16793ad623efe5319b9b6b9.png https://status.keygen.sh/ en https://status.keygen.sh/ Fri, 11 Apr 2025 06:00:37 +0000 https://status.keygen.sh/#95d9a43b9b60fa0b9d05f526e534ab33f10d198c14699c4adc3d31d4117f6c57 Dashboard recovered https://status.keygen.sh/ Fri, 11 Apr 2025 05:59:27 +0000 https://status.keygen.sh/#95d9a43b9b60fa0b9d05f526e534ab33f10d198c14699c4adc3d31d4117f6c57 Dashboard went down https://status.keygen.sh/incident/534929 Sat, 29 Mar 2025 08:00:05 +0000 https://status.keygen.sh/incident/534929#6ff15a54acac60b389800431025334ab4f0f940b90a0a3e540c7178e83878b91 Maintenance completed https://status.keygen.sh/incident/534929 Sat, 29 Mar 2025 08:00:05 +0000 https://status.keygen.sh/incident/534929#6ff15a54acac60b389800431025334ab4f0f940b90a0a3e540c7178e83878b91 Maintenance completed https://status.keygen.sh/incident/534929 Sat, 29 Mar 2025 08:00:05 +0000 https://status.keygen.sh/incident/534929#6ff15a54acac60b389800431025334ab4f0f940b90a0a3e540c7178e83878b91 Maintenance completed https://status.keygen.sh/incident/534929 Sat, 29 Mar 2025 07:00:05 -0000 https://status.keygen.sh/incident/534929#a87af0345c993f397943ed8b8c3df3764ff7785a0636a796a3ab576653d484a3 We will be performing scheduled maintenance on our Postgres datastores on 2025-03-29 at 07:00:00 UTC. During the changeover there may be up to 5 minutes of downtime. https://status.keygen.sh/incident/534929 Sat, 29 Mar 2025 07:00:05 -0000 https://status.keygen.sh/incident/534929#a87af0345c993f397943ed8b8c3df3764ff7785a0636a796a3ab576653d484a3 We will be performing scheduled maintenance on our Postgres datastores on 2025-03-29 at 07:00:00 UTC. During the changeover there may be up to 5 minutes of downtime. https://status.keygen.sh/incident/534929 Sat, 29 Mar 2025 07:00:05 -0000 https://status.keygen.sh/incident/534929#a87af0345c993f397943ed8b8c3df3764ff7785a0636a796a3ab576653d484a3 We will be performing scheduled maintenance on our Postgres datastores on 2025-03-29 at 07:00:00 UTC. During the changeover there may be up to 5 minutes of downtime. https://status.keygen.sh/incident/526421 Sun, 16 Mar 2025 00:30:18 +0000 https://status.keygen.sh/incident/526421#4ff102d5e24f736356411d75dc6f1072d74613f63af27f7f663d89fd24ab3fff Maintenance completed https://status.keygen.sh/incident/526421 Sun, 16 Mar 2025 00:30:18 +0000 https://status.keygen.sh/incident/526421#4ff102d5e24f736356411d75dc6f1072d74613f63af27f7f663d89fd24ab3fff Maintenance completed https://status.keygen.sh/incident/526421 Sun, 16 Mar 2025 00:30:18 +0000 https://status.keygen.sh/incident/526421#4ff102d5e24f736356411d75dc6f1072d74613f63af27f7f663d89fd24ab3fff Maintenance completed https://status.keygen.sh/incident/526421 Sun, 16 Mar 2025 00:00:18 -0000 https://status.keygen.sh/incident/526421#6bdd7f94601ed26f436e3257a5b45b59b7a27f1804c3fcb373bde1270d8f42c0 We will be performing a scheduled upgrade of our Redis datastores on March 16th 2025 at 12am UTC. During the upgrade and changeover there may be up to 5 minutes of downtime. https://status.keygen.sh/incident/526421 Sun, 16 Mar 2025 00:00:18 -0000 https://status.keygen.sh/incident/526421#6bdd7f94601ed26f436e3257a5b45b59b7a27f1804c3fcb373bde1270d8f42c0 We will be performing a scheduled upgrade of our Redis datastores on March 16th 2025 at 12am UTC. During the upgrade and changeover there may be up to 5 minutes of downtime. https://status.keygen.sh/incident/526421 Sun, 16 Mar 2025 00:00:18 -0000 https://status.keygen.sh/incident/526421#6bdd7f94601ed26f436e3257a5b45b59b7a27f1804c3fcb373bde1270d8f42c0 We will be performing a scheduled upgrade of our Redis datastores on March 16th 2025 at 12am UTC. During the upgrade and changeover there may be up to 5 minutes of downtime. https://status.keygen.sh/incident/492144 Wed, 08 Jan 2025 08:00:00 +0000 https://status.keygen.sh/incident/492144#2af6a324c170537f69ae8ed67d389695450592b612552ab2097bb9fc5f631962 Maintenance completed https://status.keygen.sh/incident/492144 Wed, 08 Jan 2025 08:00:00 +0000 https://status.keygen.sh/incident/492144#2af6a324c170537f69ae8ed67d389695450592b612552ab2097bb9fc5f631962 Maintenance completed https://status.keygen.sh/incident/492144 Wed, 08 Jan 2025 08:00:00 +0000 https://status.keygen.sh/incident/492144#2af6a324c170537f69ae8ed67d389695450592b612552ab2097bb9fc5f631962 Maintenance completed https://status.keygen.sh/incident/492144 Wed, 08 Jan 2025 07:17:18 -0000 https://status.keygen.sh/incident/492144#9fa094d8bd96f92f15c1960ad042d127e0b88d859f425c45e8b9625d6d655219 We're seeing a volume increase this year across multiple large accounts, and we need to upgrade our database infrastructure to support this increase moving forward. We will be taking an outage tonight to facilitate the upgrade and changeover. We will attempt to minimize downtime, but expect at least 5 minutes of downtime during the changeover. Request timing may increase or become sporadic throughout the day as we prepare the database for the upgrade procedure. Sorry for the inconvenience. https://status.keygen.sh/incident/492144 Wed, 08 Jan 2025 07:17:18 -0000 https://status.keygen.sh/incident/492144#9fa094d8bd96f92f15c1960ad042d127e0b88d859f425c45e8b9625d6d655219 We're seeing a volume increase this year across multiple large accounts, and we need to upgrade our database infrastructure to support this increase moving forward. We will be taking an outage tonight to facilitate the upgrade and changeover. We will attempt to minimize downtime, but expect at least 5 minutes of downtime during the changeover. Request timing may increase or become sporadic throughout the day as we prepare the database for the upgrade procedure. Sorry for the inconvenience. https://status.keygen.sh/incident/492144 Wed, 08 Jan 2025 07:17:18 -0000 https://status.keygen.sh/incident/492144#9fa094d8bd96f92f15c1960ad042d127e0b88d859f425c45e8b9625d6d655219 We're seeing a volume increase this year across multiple large accounts, and we need to upgrade our database infrastructure to support this increase moving forward. We will be taking an outage tonight to facilitate the upgrade and changeover. We will attempt to minimize downtime, but expect at least 5 minutes of downtime during the changeover. Request timing may increase or become sporadic throughout the day as we prepare the database for the upgrade procedure. Sorry for the inconvenience. https://status.keygen.sh/incident/492144 Wed, 08 Jan 2025 05:41:36 -0000 https://status.keygen.sh/incident/492144#9fa094d8bd96f92f15c1960ad042d127e0b88d859f425c45e8b9625d6d655219 We're seeing a volume increase this year across multiple large accounts, and we need to upgrade our database infrastructure to support this increase moving forward. We will be taking an outage tonight to facilitate the upgrade and changeover. We will attempt to minimize downtime, but expect at least 5 minutes of downtime during the changeover. Request timing may increase or become sporadic throughout the day as we prepare the database for the upgrade procedure. Sorry for the inconvenience. https://status.keygen.sh/incident/492144 Wed, 08 Jan 2025 05:41:36 -0000 https://status.keygen.sh/incident/492144#9fa094d8bd96f92f15c1960ad042d127e0b88d859f425c45e8b9625d6d655219 We're seeing a volume increase this year across multiple large accounts, and we need to upgrade our database infrastructure to support this increase moving forward. We will be taking an outage tonight to facilitate the upgrade and changeover. We will attempt to minimize downtime, but expect at least 5 minutes of downtime during the changeover. Request timing may increase or become sporadic throughout the day as we prepare the database for the upgrade procedure. Sorry for the inconvenience. https://status.keygen.sh/incident/492144 Wed, 08 Jan 2025 05:41:36 -0000 https://status.keygen.sh/incident/492144#9fa094d8bd96f92f15c1960ad042d127e0b88d859f425c45e8b9625d6d655219 We're seeing a volume increase this year across multiple large accounts, and we need to upgrade our database infrastructure to support this increase moving forward. We will be taking an outage tonight to facilitate the upgrade and changeover. We will attempt to minimize downtime, but expect at least 5 minutes of downtime during the changeover. Request timing may increase or become sporadic throughout the day as we prepare the database for the upgrade procedure. Sorry for the inconvenience. https://status.keygen.sh/incident/492144 Wed, 08 Jan 2025 04:16:10 -0000 https://status.keygen.sh/incident/492144#9fa094d8bd96f92f15c1960ad042d127e0b88d859f425c45e8b9625d6d655219 We're seeing a volume increase this year across multiple large accounts, and we need to upgrade our database infrastructure to support this increase moving forward. We will be taking an outage tonight to facilitate the upgrade and changeover. We will attempt to minimize downtime, but expect at least 5 minutes of downtime during the changeover. Request timing may increase or become sporadic throughout the day as we prepare the database for the upgrade procedure. Sorry for the inconvenience. https://status.keygen.sh/incident/492144 Wed, 08 Jan 2025 04:16:10 -0000 https://status.keygen.sh/incident/492144#9fa094d8bd96f92f15c1960ad042d127e0b88d859f425c45e8b9625d6d655219 We're seeing a volume increase this year across multiple large accounts, and we need to upgrade our database infrastructure to support this increase moving forward. We will be taking an outage tonight to facilitate the upgrade and changeover. We will attempt to minimize downtime, but expect at least 5 minutes of downtime during the changeover. Request timing may increase or become sporadic throughout the day as we prepare the database for the upgrade procedure. Sorry for the inconvenience. https://status.keygen.sh/incident/492144 Wed, 08 Jan 2025 04:16:10 -0000 https://status.keygen.sh/incident/492144#9fa094d8bd96f92f15c1960ad042d127e0b88d859f425c45e8b9625d6d655219 We're seeing a volume increase this year across multiple large accounts, and we need to upgrade our database infrastructure to support this increase moving forward. We will be taking an outage tonight to facilitate the upgrade and changeover. We will attempt to minimize downtime, but expect at least 5 minutes of downtime during the changeover. Request timing may increase or become sporadic throughout the day as we prepare the database for the upgrade procedure. Sorry for the inconvenience. https://status.keygen.sh/incident/492144 Wed, 08 Jan 2025 02:30:00 -0000 https://status.keygen.sh/incident/492144#9fa094d8bd96f92f15c1960ad042d127e0b88d859f425c45e8b9625d6d655219 We're seeing a volume increase this year across multiple large accounts, and we need to upgrade our database infrastructure to support this increase moving forward. We will be taking an outage tonight to facilitate the upgrade and changeover. We will attempt to minimize downtime, but expect at least 5 minutes of downtime during the changeover. Request timing may increase or become sporadic throughout the day as we prepare the database for the upgrade procedure. Sorry for the inconvenience. https://status.keygen.sh/incident/492144 Wed, 08 Jan 2025 02:30:00 -0000 https://status.keygen.sh/incident/492144#9fa094d8bd96f92f15c1960ad042d127e0b88d859f425c45e8b9625d6d655219 We're seeing a volume increase this year across multiple large accounts, and we need to upgrade our database infrastructure to support this increase moving forward. We will be taking an outage tonight to facilitate the upgrade and changeover. We will attempt to minimize downtime, but expect at least 5 minutes of downtime during the changeover. Request timing may increase or become sporadic throughout the day as we prepare the database for the upgrade procedure. Sorry for the inconvenience. https://status.keygen.sh/incident/492144 Wed, 08 Jan 2025 02:30:00 -0000 https://status.keygen.sh/incident/492144#9fa094d8bd96f92f15c1960ad042d127e0b88d859f425c45e8b9625d6d655219 We're seeing a volume increase this year across multiple large accounts, and we need to upgrade our database infrastructure to support this increase moving forward. We will be taking an outage tonight to facilitate the upgrade and changeover. We will attempt to minimize downtime, but expect at least 5 minutes of downtime during the changeover. Request timing may increase or become sporadic throughout the day as we prepare the database for the upgrade procedure. Sorry for the inconvenience. https://status.keygen.sh/incident/445793 Sat, 26 Oct 2024 07:30:50 +0000 https://status.keygen.sh/incident/445793#b7df8e006a4458e3b48e7a55ec7f73001e421bf0f201eed074175fe80a6fb16b Maintenance completed https://status.keygen.sh/incident/445793 Sat, 26 Oct 2024 07:30:50 +0000 https://status.keygen.sh/incident/445793#b7df8e006a4458e3b48e7a55ec7f73001e421bf0f201eed074175fe80a6fb16b Maintenance completed https://status.keygen.sh/incident/445793 Sat, 26 Oct 2024 07:30:50 +0000 https://status.keygen.sh/incident/445793#b7df8e006a4458e3b48e7a55ec7f73001e421bf0f201eed074175fe80a6fb16b Maintenance completed https://status.keygen.sh/incident/445793 Sat, 26 Oct 2024 07:00:00 -0000 https://status.keygen.sh/incident/445793#c710d2ffe273dae623fd2348ec513aa95c92bcfd5df9dd7980cb997fd8f5acc6 We will be performing scheduled maintenance on our Postgres datastores on 2024-10-26 at 07:00:00 UTC. During the changeover there may be up to 5 minutes of downtime. https://status.keygen.sh/incident/445793 Sat, 26 Oct 2024 07:00:00 -0000 https://status.keygen.sh/incident/445793#c710d2ffe273dae623fd2348ec513aa95c92bcfd5df9dd7980cb997fd8f5acc6 We will be performing scheduled maintenance on our Postgres datastores on 2024-10-26 at 07:00:00 UTC. During the changeover there may be up to 5 minutes of downtime. https://status.keygen.sh/incident/445793 Sat, 26 Oct 2024 07:00:00 -0000 https://status.keygen.sh/incident/445793#c710d2ffe273dae623fd2348ec513aa95c92bcfd5df9dd7980cb997fd8f5acc6 We will be performing scheduled maintenance on our Postgres datastores on 2024-10-26 at 07:00:00 UTC. During the changeover there may be up to 5 minutes of downtime. https://status.keygen.sh/incident/440345 Thu, 10 Oct 2024 02:30:00 +0000 https://status.keygen.sh/incident/440345#5c1aa88ea8026ef660a897f088feeccb9e7ba1180f2bf7512edc41a1eb9d17e4 Maintenance completed https://status.keygen.sh/incident/440345 Thu, 10 Oct 2024 02:30:00 +0000 https://status.keygen.sh/incident/440345#5c1aa88ea8026ef660a897f088feeccb9e7ba1180f2bf7512edc41a1eb9d17e4 Maintenance completed https://status.keygen.sh/incident/440345 Thu, 10 Oct 2024 02:30:00 +0000 https://status.keygen.sh/incident/440345#5c1aa88ea8026ef660a897f088feeccb9e7ba1180f2bf7512edc41a1eb9d17e4 Maintenance completed https://status.keygen.sh/incident/440345 Thu, 10 Oct 2024 02:00:00 -0000 https://status.keygen.sh/incident/440345#e82dd5e082bb79a5d0325751a4e03097467b8cad0ca06c8efe0442f9b401d2d7 We will be performing a scheduled upgrade of our Redis datastores on October 10th 2024 at 2am UTC (i.e. the 9th @ 9pm CST). During the upgrade and changeover there may be up to 5 minutes of downtime. https://status.keygen.sh/incident/440345 Thu, 10 Oct 2024 02:00:00 -0000 https://status.keygen.sh/incident/440345#e82dd5e082bb79a5d0325751a4e03097467b8cad0ca06c8efe0442f9b401d2d7 We will be performing a scheduled upgrade of our Redis datastores on October 10th 2024 at 2am UTC (i.e. the 9th @ 9pm CST). During the upgrade and changeover there may be up to 5 minutes of downtime. https://status.keygen.sh/incident/440345 Thu, 10 Oct 2024 02:00:00 -0000 https://status.keygen.sh/incident/440345#e82dd5e082bb79a5d0325751a4e03097467b8cad0ca06c8efe0442f9b401d2d7 We will be performing a scheduled upgrade of our Redis datastores on October 10th 2024 at 2am UTC (i.e. the 9th @ 9pm CST). During the upgrade and changeover there may be up to 5 minutes of downtime. https://status.keygen.sh/incident/422353 Sun, 01 Sep 2024 01:00:00 +0000 https://status.keygen.sh/incident/422353#325f09c48551d7b7efd6fdd5875906b33d90625dcee42613a889251f4adbfee8 Maintenance completed https://status.keygen.sh/incident/422353 Sun, 01 Sep 2024 01:00:00 +0000 https://status.keygen.sh/incident/422353#325f09c48551d7b7efd6fdd5875906b33d90625dcee42613a889251f4adbfee8 Maintenance completed https://status.keygen.sh/incident/422353 Sun, 01 Sep 2024 01:00:00 +0000 https://status.keygen.sh/incident/422353#325f09c48551d7b7efd6fdd5875906b33d90625dcee42613a889251f4adbfee8 Maintenance completed https://status.keygen.sh/incident/422353 Sun, 01 Sep 2024 00:00:00 -0000 https://status.keygen.sh/incident/422353#f3d08247babb2a092a58f0117d36263f88d52803cb1c9459716eb6486bcbef9c We started rotating the TLS certificates for our Redis databases. There may be up to 5 minutes of downtime while the services restart. https://status.keygen.sh/incident/422353 Sun, 01 Sep 2024 00:00:00 -0000 https://status.keygen.sh/incident/422353#f3d08247babb2a092a58f0117d36263f88d52803cb1c9459716eb6486bcbef9c We started rotating the TLS certificates for our Redis databases. There may be up to 5 minutes of downtime while the services restart. https://status.keygen.sh/incident/422353 Sun, 01 Sep 2024 00:00:00 -0000 https://status.keygen.sh/incident/422353#f3d08247babb2a092a58f0117d36263f88d52803cb1c9459716eb6486bcbef9c We started rotating the TLS certificates for our Redis databases. There may be up to 5 minutes of downtime while the services restart. https://status.keygen.sh/incident/407981 Sun, 11 Aug 2024 00:30:00 +0000 https://status.keygen.sh/incident/407981#beb81571e277c18b52ef711fe3b2e4553bc1e9f1bd467712f61d0ee013ebc026 Maintenance completed https://status.keygen.sh/incident/407981 Sun, 11 Aug 2024 00:30:00 +0000 https://status.keygen.sh/incident/407981#beb81571e277c18b52ef711fe3b2e4553bc1e9f1bd467712f61d0ee013ebc026 Maintenance completed https://status.keygen.sh/incident/407981 Sun, 11 Aug 2024 00:30:00 +0000 https://status.keygen.sh/incident/407981#beb81571e277c18b52ef711fe3b2e4553bc1e9f1bd467712f61d0ee013ebc026 Maintenance completed https://status.keygen.sh/incident/407981 Sun, 11 Aug 2024 00:00:00 -0000 https://status.keygen.sh/incident/407981#5c68f06c30e7914aba1bfa58a674970729bd30001fd9f30a309ceebc0d07b479 We will be performing scheduled maintenance on our Redis databases at 2024-08-11 00:00:00 UTC. There may be up to 5 minutes of downtime as we do a changeover, but it should largely go unnoticed. https://status.keygen.sh/incident/407981 Sun, 11 Aug 2024 00:00:00 -0000 https://status.keygen.sh/incident/407981#5c68f06c30e7914aba1bfa58a674970729bd30001fd9f30a309ceebc0d07b479 We will be performing scheduled maintenance on our Redis databases at 2024-08-11 00:00:00 UTC. There may be up to 5 minutes of downtime as we do a changeover, but it should largely go unnoticed. https://status.keygen.sh/incident/407981 Sun, 11 Aug 2024 00:00:00 -0000 https://status.keygen.sh/incident/407981#5c68f06c30e7914aba1bfa58a674970729bd30001fd9f30a309ceebc0d07b479 We will be performing scheduled maintenance on our Redis databases at 2024-08-11 00:00:00 UTC. There may be up to 5 minutes of downtime as we do a changeover, but it should largely go unnoticed. https://status.keygen.sh/incident/407561 Sat, 10 Aug 2024 08:00:00 +0000 https://status.keygen.sh/incident/407561#6a5ec108a304546d15387b5797d74102803eec0c3cb17c90725de2803b1589a7 Maintenance completed https://status.keygen.sh/incident/407561 Sat, 10 Aug 2024 08:00:00 +0000 https://status.keygen.sh/incident/407561#6a5ec108a304546d15387b5797d74102803eec0c3cb17c90725de2803b1589a7 Maintenance completed https://status.keygen.sh/incident/407561 Sat, 10 Aug 2024 08:00:00 +0000 https://status.keygen.sh/incident/407561#6a5ec108a304546d15387b5797d74102803eec0c3cb17c90725de2803b1589a7 Maintenance completed https://status.keygen.sh/incident/407561 Sat, 10 Aug 2024 07:00:00 -0000 https://status.keygen.sh/incident/407561#8c7aec7c5e458b4be98a67ca1da71dd5f283d06354f32d964703ff825c877dd9 We will be performing scheduled maintenance on our Postgres databases at 2024-08-10 07:00:00 UTC. There may be up to 5 minutes of downtime as we do a changeover, but it should largely go unnoticed. https://status.keygen.sh/incident/407561 Sat, 10 Aug 2024 07:00:00 -0000 https://status.keygen.sh/incident/407561#8c7aec7c5e458b4be98a67ca1da71dd5f283d06354f32d964703ff825c877dd9 We will be performing scheduled maintenance on our Postgres databases at 2024-08-10 07:00:00 UTC. There may be up to 5 minutes of downtime as we do a changeover, but it should largely go unnoticed. https://status.keygen.sh/incident/407561 Sat, 10 Aug 2024 07:00:00 -0000 https://status.keygen.sh/incident/407561#8c7aec7c5e458b4be98a67ca1da71dd5f283d06354f32d964703ff825c877dd9 We will be performing scheduled maintenance on our Postgres databases at 2024-08-10 07:00:00 UTC. There may be up to 5 minutes of downtime as we do a changeover, but it should largely go unnoticed. https://status.keygen.sh/ Fri, 12 Jul 2024 15:41:35 +0000 https://status.keygen.sh/#c640128c05b0e512bcd012c2b334ceb551c5ff94a59d7faa89c0f5c3660572fc Webhooks recovered https://status.keygen.sh/ Fri, 12 Jul 2024 15:36:32 +0000 https://status.keygen.sh/#c640128c05b0e512bcd012c2b334ceb551c5ff94a59d7faa89c0f5c3660572fc Webhooks went down https://status.keygen.sh/ Fri, 12 Jul 2024 15:31:37 +0000 https://status.keygen.sh/#eca6ef0c9228e8b4a6950acc233450522cacd2c9e9dc2d50c5dd7bb06d232c7b Webhooks recovered https://status.keygen.sh/ Fri, 12 Jul 2024 15:29:44 +0000 https://status.keygen.sh/#e507c08276fad609d35549270e4107ef2aed92b2114c8ee4c16872d58a04482e Licensing API recovered https://status.keygen.sh/ Fri, 12 Jul 2024 15:28:13 +0000 https://status.keygen.sh/#e507c08276fad609d35549270e4107ef2aed92b2114c8ee4c16872d58a04482e Licensing API went down https://status.keygen.sh/ Fri, 12 Jul 2024 15:26:35 +0000 https://status.keygen.sh/#eca6ef0c9228e8b4a6950acc233450522cacd2c9e9dc2d50c5dd7bb06d232c7b Webhooks went down https://status.keygen.sh/incident/371330 Sun, 26 May 2024 00:30:00 +0000 https://status.keygen.sh/incident/371330#1466bddd4769dc2a666aa0392fe02abe3df247545a71b17bf151cb4285002f6e Maintenance completed https://status.keygen.sh/incident/371330 Sun, 26 May 2024 00:30:00 +0000 https://status.keygen.sh/incident/371330#1466bddd4769dc2a666aa0392fe02abe3df247545a71b17bf151cb4285002f6e Maintenance completed https://status.keygen.sh/incident/371330 Sun, 26 May 2024 00:30:00 +0000 https://status.keygen.sh/incident/371330#1466bddd4769dc2a666aa0392fe02abe3df247545a71b17bf151cb4285002f6e Maintenance completed https://status.keygen.sh/incident/371330 Sun, 26 May 2024 00:00:00 -0000 https://status.keygen.sh/incident/371330#29341780c4414419c2568bac3086ded8b3660d5f5959cb0448821fa96d444835 We will be performing scheduled maintenance on our Redis datastores on 2024-05-26 00:00:00 UTC. During the changeover process, there may be up to 5 minutes of downtime. https://status.keygen.sh/incident/371330 Sun, 26 May 2024 00:00:00 -0000 https://status.keygen.sh/incident/371330#29341780c4414419c2568bac3086ded8b3660d5f5959cb0448821fa96d444835 We will be performing scheduled maintenance on our Redis datastores on 2024-05-26 00:00:00 UTC. During the changeover process, there may be up to 5 minutes of downtime. https://status.keygen.sh/incident/371330 Sun, 26 May 2024 00:00:00 -0000 https://status.keygen.sh/incident/371330#29341780c4414419c2568bac3086ded8b3660d5f5959cb0448821fa96d444835 We will be performing scheduled maintenance on our Redis datastores on 2024-05-26 00:00:00 UTC. During the changeover process, there may be up to 5 minutes of downtime. https://status.keygen.sh/incident/334436 Sat, 09 Mar 2024 07:30:00 +0000 https://status.keygen.sh/incident/334436#bd2b7e21cb2bdc1835ab7de8dcbc4b6d9fbdeca3b1885a107875b793aa8115db Maintenance completed https://status.keygen.sh/incident/334436 Sat, 09 Mar 2024 07:30:00 +0000 https://status.keygen.sh/incident/334436#bd2b7e21cb2bdc1835ab7de8dcbc4b6d9fbdeca3b1885a107875b793aa8115db Maintenance completed https://status.keygen.sh/incident/334436 Sat, 09 Mar 2024 07:30:00 +0000 https://status.keygen.sh/incident/334436#bd2b7e21cb2bdc1835ab7de8dcbc4b6d9fbdeca3b1885a107875b793aa8115db Maintenance completed https://status.keygen.sh/incident/334436 Sat, 09 Mar 2024 07:00:00 -0000 https://status.keygen.sh/incident/334436#54927b5d2ae2801b4b53da5607c14d8214262900b9079256b05bdb3b57044096 We will be performing scheduled maintenance on our Postgres datastores on 2024-03-09 07:00:00 UTC. During the changeover there may be up to 5 minutes of downtime. https://status.keygen.sh/incident/334436 Sat, 09 Mar 2024 07:00:00 -0000 https://status.keygen.sh/incident/334436#54927b5d2ae2801b4b53da5607c14d8214262900b9079256b05bdb3b57044096 We will be performing scheduled maintenance on our Postgres datastores on 2024-03-09 07:00:00 UTC. During the changeover there may be up to 5 minutes of downtime. https://status.keygen.sh/incident/334436 Sat, 09 Mar 2024 07:00:00 -0000 https://status.keygen.sh/incident/334436#54927b5d2ae2801b4b53da5607c14d8214262900b9079256b05bdb3b57044096 We will be performing scheduled maintenance on our Postgres datastores on 2024-03-09 07:00:00 UTC. During the changeover there may be up to 5 minutes of downtime. https://status.keygen.sh/incident/333245 Wed, 28 Feb 2024 16:24:00 -0000 https://status.keygen.sh/incident/333245#8510c05c85d0c08a735cb8f4b71edf6e724ab8f2aa9fb91db0ca66b4485153e1 The deployment has been rolled back and API performance has returned to normal. https://status.keygen.sh/incident/333245 Wed, 28 Feb 2024 16:24:00 -0000 https://status.keygen.sh/incident/333245#8510c05c85d0c08a735cb8f4b71edf6e724ab8f2aa9fb91db0ca66b4485153e1 The deployment has been rolled back and API performance has returned to normal. https://status.keygen.sh/incident/333245 Wed, 28 Feb 2024 16:17:00 -0000 https://status.keygen.sh/incident/333245#7f083516c5ff43e91430abba33b8626077821db38928737732e75413639744f5 We are aware of degraded API performance due to a bad deploy. We are working on rolling back the changes now. We apologize for the inconvenience. https://status.keygen.sh/incident/333245 Wed, 28 Feb 2024 16:17:00 -0000 https://status.keygen.sh/incident/333245#7f083516c5ff43e91430abba33b8626077821db38928737732e75413639744f5 We are aware of degraded API performance due to a bad deploy. We are working on rolling back the changes now. We apologize for the inconvenience. https://status.keygen.sh/incident/322421 Tue, 06 Feb 2024 20:04:00 -0000 https://status.keygen.sh/incident/322421#85b20b34c2eec955264963d89ab22f32a1404a90178d4cdbad07ab378e78f8e8 We were able to locate the source of the problem on our end that was limiting our scaling under certain loads. We've deployed a fix and are now accepting all traffic. If there's a silver lining, it's that we fixed a long standing performance problem that reared itself sporadically, and now we know why. We're going to write up a complete postmortem soon. Thanks so much for your patience, and I sincerely apologize for the problems this outage has caused all your businesses over the last 36 hours. It's been a nightmare, but we came out stronger. https://status.keygen.sh/incident/322421 Tue, 06 Feb 2024 20:04:00 -0000 https://status.keygen.sh/incident/322421#85b20b34c2eec955264963d89ab22f32a1404a90178d4cdbad07ab378e78f8e8 We were able to locate the source of the problem on our end that was limiting our scaling under certain loads. We've deployed a fix and are now accepting all traffic. If there's a silver lining, it's that we fixed a long standing performance problem that reared itself sporadically, and now we know why. We're going to write up a complete postmortem soon. Thanks so much for your patience, and I sincerely apologize for the problems this outage has caused all your businesses over the last 36 hours. It's been a nightmare, but we came out stronger. https://status.keygen.sh/incident/322421 Tue, 06 Feb 2024 20:04:00 -0000 https://status.keygen.sh/incident/322421#85b20b34c2eec955264963d89ab22f32a1404a90178d4cdbad07ab378e78f8e8 We were able to locate the source of the problem on our end that was limiting our scaling under certain loads. We've deployed a fix and are now accepting all traffic. If there's a silver lining, it's that we fixed a long standing performance problem that reared itself sporadically, and now we know why. We're going to write up a complete postmortem soon. Thanks so much for your patience, and I sincerely apologize for the problems this outage has caused all your businesses over the last 36 hours. It's been a nightmare, but we came out stronger. https://status.keygen.sh/incident/322421 Tue, 06 Feb 2024 12:14:00 -0000 https://status.keygen.sh/incident/322421#4454d249bdf949013e486097ac04ac512c38a4d76df5e6bcb14f53807a533e64 Services have stabilized, but we are continuing to load shed the DDoS. We are continuing to monitor the incident and scale up infrastructure to handle the next traffic spike without downtime. https://status.keygen.sh/incident/322421 Tue, 06 Feb 2024 12:14:00 -0000 https://status.keygen.sh/incident/322421#4454d249bdf949013e486097ac04ac512c38a4d76df5e6bcb14f53807a533e64 Services have stabilized, but we are continuing to load shed the DDoS. We are continuing to monitor the incident and scale up infrastructure to handle the next traffic spike without downtime. https://status.keygen.sh/incident/322421 Tue, 06 Feb 2024 12:14:00 -0000 https://status.keygen.sh/incident/322421#4454d249bdf949013e486097ac04ac512c38a4d76df5e6bcb14f53807a533e64 Services have stabilized, but we are continuing to load shed the DDoS. We are continuing to monitor the incident and scale up infrastructure to handle the next traffic spike without downtime. https://status.keygen.sh/incident/322421 Tue, 06 Feb 2024 10:06:00 -0000 https://status.keygen.sh/incident/322421#dce7783abab5cc2b66def11f610ddf6fb6521a09f7c1db9ae76549c971ac0ee9 We're aware of the same issue happening again. We're working on restoring services. https://status.keygen.sh/incident/322421 Tue, 06 Feb 2024 10:06:00 -0000 https://status.keygen.sh/incident/322421#dce7783abab5cc2b66def11f610ddf6fb6521a09f7c1db9ae76549c971ac0ee9 We're aware of the same issue happening again. We're working on restoring services. https://status.keygen.sh/incident/322421 Tue, 06 Feb 2024 10:06:00 -0000 https://status.keygen.sh/incident/322421#dce7783abab5cc2b66def11f610ddf6fb6521a09f7c1db9ae76549c971ac0ee9 We're aware of the same issue happening again. We're working on restoring services. https://status.keygen.sh/incident/322421 Mon, 05 Feb 2024 21:01:00 -0000 https://status.keygen.sh/incident/322421#4b4bc2cfe4b9b8140616e8dc5237a8f9a7b7c9856ad1e7328f83039a6a46013b Today, we experienced an unintentional DDoS. It was not an attack as we originally reported (we were unaware of its nature). It was largely due to an unexpected increase in API volume, which caused a cascading series of events that ultimately caused an outage that lasted over 5 hours. This is unacceptable, and I apologize. The timestamps in this root cause analysis will be in CST. At 12:30am CST, we experienced a large, sudden surge in legitimate traffic. This caused our autoscaler to kick in and scale up infrastructure. But because the source of the traffic surge was legitmate traffic, we were unable to rate limit effectively. By the time our infrastructure was scaled up enough to sustain the traffic, multiple systems had began failing due to a backup in job processessing. To make matters worse, the source of the traffic had an aggressive retry algorithm configured, where instead of backing off exponentially, it sped up, which effectively DDoS'd us with the volume of traffic. Three hours elapsed before we were made aware of the incident. At 4am CST, we were able to page an engineer to investigate. This was a terrible response time, and we'll cover what went wrong here at the end. But at 4am, we were aware of and unraveling the incident, and formulating a response. We noticed some outlier IPs i.r.t. volume, and we immediately began blacklisting the IP addresses that were sending higher-than-normal API volume. But that wasn't enough. And there were too many IPs. The volume was distributed, and as far as we could tell, it was legitimate traffic. This made things complicated. To prevent further strain on our systems, we enabled Cloudflare's "Under Attack" mode, which effectively blocked the majority of API requests (since an API client cannot solve a Cloudflare "challenge response"). We wished there was another way, but Cloudflare's WAF is not very friendly to API-only products. We ultimately did this so that our systems could recover, and to give us time to determine next steps. Our Redis instances were at capacity due to the number of jobs to process, and this "pause" gave our systems time to recover. Around 5am, once our systems were sufficiently recovered, we disabled Cloudflare's "Under Attack" mode. But the DDoS was still ongoing, and we determined the source of the traffic was originating from a single account. We began sending challenge responses to requests matching that account, effectively blocking traffic until we could deal with it in a more controlled fashion. After getting in contact with the account owner, we determined that the traffic was indeed legitimate. We began letting in a percentage of the traffic until their applications were able to stabilize and cease retries (which was effectively the source of the DDoS). We then began receiving reports that customers were experiencing intermittent signature verification failures. After investigation, this was due to Cloudflare modifying the Date header we send, which is a part of a response's cryptographic signature. It was intermittent because it was dependent upon the current time. For example, let's assume a request has a timestamp of: 2024-02-04T23:01:10.993239 This results in us sending the following Date header: Date: Sun, 04 Feb 2024 23:01:10 GMT Unfortunately, the Date header is rewritten by Cloudflare to the following: Date: Sun, 04 Feb 2024 23:01:11 GMT Note the difference in the second portion. By the time Cloudflare sent the response, the Date had changed, because we were at milisecond .993239. This change in Date is what was causing the signature verification failures, since the Date header is used in the response signature to prevent timing attacks. At 5am, service was restored for all accounts, and API volume was back to a normal level. At this point, we disabled Cloudflare's CDN to resolve the signature verification issue. Our request logs and event logs were delayed for another hour before normalizing. It's unacceptable that it took over 3 hours to acknowledge the incident, and ultimately 5 hours of downtime, and I apologize for that. I was the on-call engineer, and I failed that duty. A few things went wrong: 1. Our pager service was not configured to send a phone call due to an oversight in the on-call configuration. 2. The on-call engineer's phone was placed on DnD mode (do-not-disturb) at 12am CST. 3. The pager service's text notifications were not whitelisted from DnD mode. 4. The email alerts were not seen due to DnD mode being enabled (I was asleep). 5. There were no escalation policies for paging more aggresively. I sincerely apologize for this incident. I feel terrible. I feel defeated. I haven't slept. It's been a day of putting out fires in the aftermath, as I'm sure it has been for you. A lot of things went wrong, and I'm still trying to piece things together. But the above is what I do know now. I'm looking at ways to prevent this from happening again. From scaling up infrastructure, to increasing datastore capacity, to finding a new WAF provider that is compatible with our API. I'm in talks with multiple providers. I understand licensing is a vital part of a software business, and we regret this happened today. I'm sorry. https://status.keygen.sh/incident/322421 Mon, 05 Feb 2024 21:01:00 -0000 https://status.keygen.sh/incident/322421#4b4bc2cfe4b9b8140616e8dc5237a8f9a7b7c9856ad1e7328f83039a6a46013b Today, we experienced an unintentional DDoS. It was not an attack as we originally reported (we were unaware of its nature). It was largely due to an unexpected increase in API volume, which caused a cascading series of events that ultimately caused an outage that lasted over 5 hours. This is unacceptable, and I apologize. The timestamps in this root cause analysis will be in CST. At 12:30am CST, we experienced a large, sudden surge in legitimate traffic. This caused our autoscaler to kick in and scale up infrastructure. But because the source of the traffic surge was legitmate traffic, we were unable to rate limit effectively. By the time our infrastructure was scaled up enough to sustain the traffic, multiple systems had began failing due to a backup in job processessing. To make matters worse, the source of the traffic had an aggressive retry algorithm configured, where instead of backing off exponentially, it sped up, which effectively DDoS'd us with the volume of traffic. Three hours elapsed before we were made aware of the incident. At 4am CST, we were able to page an engineer to investigate. This was a terrible response time, and we'll cover what went wrong here at the end. But at 4am, we were aware of and unraveling the incident, and formulating a response. We noticed some outlier IPs i.r.t. volume, and we immediately began blacklisting the IP addresses that were sending higher-than-normal API volume. But that wasn't enough. And there were too many IPs. The volume was distributed, and as far as we could tell, it was legitimate traffic. This made things complicated. To prevent further strain on our systems, we enabled Cloudflare's "Under Attack" mode, which effectively blocked the majority of API requests (since an API client cannot solve a Cloudflare "challenge response"). We wished there was another way, but Cloudflare's WAF is not very friendly to API-only products. We ultimately did this so that our systems could recover, and to give us time to determine next steps. Our Redis instances were at capacity due to the number of jobs to process, and this "pause" gave our systems time to recover. Around 5am, once our systems were sufficiently recovered, we disabled Cloudflare's "Under Attack" mode. But the DDoS was still ongoing, and we determined the source of the traffic was originating from a single account. We began sending challenge responses to requests matching that account, effectively blocking traffic until we could deal with it in a more controlled fashion. After getting in contact with the account owner, we determined that the traffic was indeed legitimate. We began letting in a percentage of the traffic until their applications were able to stabilize and cease retries (which was effectively the source of the DDoS). We then began receiving reports that customers were experiencing intermittent signature verification failures. After investigation, this was due to Cloudflare modifying the Date header we send, which is a part of a response's cryptographic signature. It was intermittent because it was dependent upon the current time. For example, let's assume a request has a timestamp of: 2024-02-04T23:01:10.993239 This results in us sending the following Date header: Date: Sun, 04 Feb 2024 23:01:10 GMT Unfortunately, the Date header is rewritten by Cloudflare to the following: Date: Sun, 04 Feb 2024 23:01:11 GMT Note the difference in the second portion. By the time Cloudflare sent the response, the Date had changed, because we were at milisecond .993239. This change in Date is what was causing the signature verification failures, since the Date header is used in the response signature to prevent timing attacks. At 5am, service was restored for all accounts, and API volume was back to a normal level. At this point, we disabled Cloudflare's CDN to resolve the signature verification issue. Our request logs and event logs were delayed for another hour before normalizing. It's unacceptable that it took over 3 hours to acknowledge the incident, and ultimately 5 hours of downtime, and I apologize for that. I was the on-call engineer, and I failed that duty. A few things went wrong: 1. Our pager service was not configured to send a phone call due to an oversight in the on-call configuration. 2. The on-call engineer's phone was placed on DnD mode (do-not-disturb) at 12am CST. 3. The pager service's text notifications were not whitelisted from DnD mode. 4. The email alerts were not seen due to DnD mode being enabled (I was asleep). 5. There were no escalation policies for paging more aggresively. I sincerely apologize for this incident. I feel terrible. I feel defeated. I haven't slept. It's been a day of putting out fires in the aftermath, as I'm sure it has been for you. A lot of things went wrong, and I'm still trying to piece things together. But the above is what I do know now. I'm looking at ways to prevent this from happening again. From scaling up infrastructure, to increasing datastore capacity, to finding a new WAF provider that is compatible with our API. I'm in talks with multiple providers. I understand licensing is a vital part of a software business, and we regret this happened today. I'm sorry. https://status.keygen.sh/incident/322421 Mon, 05 Feb 2024 21:01:00 -0000 https://status.keygen.sh/incident/322421#4b4bc2cfe4b9b8140616e8dc5237a8f9a7b7c9856ad1e7328f83039a6a46013b Today, we experienced an unintentional DDoS. It was not an attack as we originally reported (we were unaware of its nature). It was largely due to an unexpected increase in API volume, which caused a cascading series of events that ultimately caused an outage that lasted over 5 hours. This is unacceptable, and I apologize. The timestamps in this root cause analysis will be in CST. At 12:30am CST, we experienced a large, sudden surge in legitimate traffic. This caused our autoscaler to kick in and scale up infrastructure. But because the source of the traffic surge was legitmate traffic, we were unable to rate limit effectively. By the time our infrastructure was scaled up enough to sustain the traffic, multiple systems had began failing due to a backup in job processessing. To make matters worse, the source of the traffic had an aggressive retry algorithm configured, where instead of backing off exponentially, it sped up, which effectively DDoS'd us with the volume of traffic. Three hours elapsed before we were made aware of the incident. At 4am CST, we were able to page an engineer to investigate. This was a terrible response time, and we'll cover what went wrong here at the end. But at 4am, we were aware of and unraveling the incident, and formulating a response. We noticed some outlier IPs i.r.t. volume, and we immediately began blacklisting the IP addresses that were sending higher-than-normal API volume. But that wasn't enough. And there were too many IPs. The volume was distributed, and as far as we could tell, it was legitimate traffic. This made things complicated. To prevent further strain on our systems, we enabled Cloudflare's "Under Attack" mode, which effectively blocked the majority of API requests (since an API client cannot solve a Cloudflare "challenge response"). We wished there was another way, but Cloudflare's WAF is not very friendly to API-only products. We ultimately did this so that our systems could recover, and to give us time to determine next steps. Our Redis instances were at capacity due to the number of jobs to process, and this "pause" gave our systems time to recover. Around 5am, once our systems were sufficiently recovered, we disabled Cloudflare's "Under Attack" mode. But the DDoS was still ongoing, and we determined the source of the traffic was originating from a single account. We began sending challenge responses to requests matching that account, effectively blocking traffic until we could deal with it in a more controlled fashion. After getting in contact with the account owner, we determined that the traffic was indeed legitimate. We began letting in a percentage of the traffic until their applications were able to stabilize and cease retries (which was effectively the source of the DDoS). We then began receiving reports that customers were experiencing intermittent signature verification failures. After investigation, this was due to Cloudflare modifying the Date header we send, which is a part of a response's cryptographic signature. It was intermittent because it was dependent upon the current time. For example, let's assume a request has a timestamp of: 2024-02-04T23:01:10.993239 This results in us sending the following Date header: Date: Sun, 04 Feb 2024 23:01:10 GMT Unfortunately, the Date header is rewritten by Cloudflare to the following: Date: Sun, 04 Feb 2024 23:01:11 GMT Note the difference in the second portion. By the time Cloudflare sent the response, the Date had changed, because we were at milisecond .993239. This change in Date is what was causing the signature verification failures, since the Date header is used in the response signature to prevent timing attacks. At 5am, service was restored for all accounts, and API volume was back to a normal level. At this point, we disabled Cloudflare's CDN to resolve the signature verification issue. Our request logs and event logs were delayed for another hour before normalizing. It's unacceptable that it took over 3 hours to acknowledge the incident, and ultimately 5 hours of downtime, and I apologize for that. I was the on-call engineer, and I failed that duty. A few things went wrong: 1. Our pager service was not configured to send a phone call due to an oversight in the on-call configuration. 2. The on-call engineer's phone was placed on DnD mode (do-not-disturb) at 12am CST. 3. The pager service's text notifications were not whitelisted from DnD mode. 4. The email alerts were not seen due to DnD mode being enabled (I was asleep). 5. There were no escalation policies for paging more aggresively. I sincerely apologize for this incident. I feel terrible. I feel defeated. I haven't slept. It's been a day of putting out fires in the aftermath, as I'm sure it has been for you. A lot of things went wrong, and I'm still trying to piece things together. But the above is what I do know now. I'm looking at ways to prevent this from happening again. From scaling up infrastructure, to increasing datastore capacity, to finding a new WAF provider that is compatible with our API. I'm in talks with multiple providers. I understand licensing is a vital part of a software business, and we regret this happened today. I'm sorry. https://status.keygen.sh/incident/322421 Mon, 05 Feb 2024 15:44:00 -0000 https://status.keygen.sh/incident/322421#2dabb311a4f57ebdafe71c409226dbf19d1fc36b918c2443b7a6929ebd4f125b API availability has returned to normal, and response signatures should now be working as expected. The bulk of the traffic has been mitigated, but we are continuing to monitor the situation as overall request volume is still elevated. If another surge occurs, we will update this report. We have a lot of data in our backlog, so request/event logs and webhooks may be delayed. https://status.keygen.sh/incident/322421 Mon, 05 Feb 2024 15:44:00 -0000 https://status.keygen.sh/incident/322421#2dabb311a4f57ebdafe71c409226dbf19d1fc36b918c2443b7a6929ebd4f125b API availability has returned to normal, and response signatures should now be working as expected. The bulk of the traffic has been mitigated, but we are continuing to monitor the situation as overall request volume is still elevated. If another surge occurs, we will update this report. We have a lot of data in our backlog, so request/event logs and webhooks may be delayed. https://status.keygen.sh/incident/322421 Mon, 05 Feb 2024 15:44:00 -0000 https://status.keygen.sh/incident/322421#2dabb311a4f57ebdafe71c409226dbf19d1fc36b918c2443b7a6929ebd4f125b API availability has returned to normal, and response signatures should now be working as expected. The bulk of the traffic has been mitigated, but we are continuing to monitor the situation as overall request volume is still elevated. If another surge occurs, we will update this report. We have a lot of data in our backlog, so request/event logs and webhooks may be delayed. https://status.keygen.sh/incident/322421 Mon, 05 Feb 2024 13:50:00 -0000 https://status.keygen.sh/incident/322421#f7b460fff0426b49caa32e9abe986b65a3239d40beed91d00881cf4a74746848 We've employed Cloudflare to mitigate the issue, but this has had an effect on signature verifications. Due to the way Cloudflare operates, the Date header Cloudflare overwrites may not match the Date header Keygen sends, resulting in signature verification errors. If possible, use the Keygen-Date header for signature verification purposes, instead of Date, to mitigate this issue until we're able to resolve this issue with Cloudflare. https://status.keygen.sh/incident/322421 Mon, 05 Feb 2024 13:50:00 -0000 https://status.keygen.sh/incident/322421#f7b460fff0426b49caa32e9abe986b65a3239d40beed91d00881cf4a74746848 We've employed Cloudflare to mitigate the issue, but this has had an effect on signature verifications. Due to the way Cloudflare operates, the Date header Cloudflare overwrites may not match the Date header Keygen sends, resulting in signature verification errors. If possible, use the Keygen-Date header for signature verification purposes, instead of Date, to mitigate this issue until we're able to resolve this issue with Cloudflare. https://status.keygen.sh/incident/322421 Mon, 05 Feb 2024 13:50:00 -0000 https://status.keygen.sh/incident/322421#f7b460fff0426b49caa32e9abe986b65a3239d40beed91d00881cf4a74746848 We've employed Cloudflare to mitigate the issue, but this has had an effect on signature verifications. Due to the way Cloudflare operates, the Date header Cloudflare overwrites may not match the Date header Keygen sends, resulting in signature verification errors. If possible, use the Keygen-Date header for signature verification purposes, instead of Date, to mitigate this issue until we're able to resolve this issue with Cloudflare. https://status.keygen.sh/incident/322421 Mon, 05 Feb 2024 11:27:00 -0000 https://status.keygen.sh/incident/322421#2e2f9dcbb677d189d76a2cb71e9204cdd2a060b9b4d8fb6af4975c5ee7b1c366 We are still under an active DDoS attack, but we have put in place mitigations. Most API traffic has returned to normal, but we are still working with our infrastructure providers on a full resolution. Some accounts may still be inaccessible due to the nature of the attack. Thank you for your patience. https://status.keygen.sh/incident/322421 Mon, 05 Feb 2024 11:27:00 -0000 https://status.keygen.sh/incident/322421#2e2f9dcbb677d189d76a2cb71e9204cdd2a060b9b4d8fb6af4975c5ee7b1c366 We are still under an active DDoS attack, but we have put in place mitigations. Most API traffic has returned to normal, but we are still working with our infrastructure providers on a full resolution. Some accounts may still be inaccessible due to the nature of the attack. Thank you for your patience. https://status.keygen.sh/incident/322421 Mon, 05 Feb 2024 11:27:00 -0000 https://status.keygen.sh/incident/322421#2e2f9dcbb677d189d76a2cb71e9204cdd2a060b9b4d8fb6af4975c5ee7b1c366 We are still under an active DDoS attack, but we have put in place mitigations. Most API traffic has returned to normal, but we are still working with our infrastructure providers on a full resolution. Some accounts may still be inaccessible due to the nature of the attack. Thank you for your patience. https://status.keygen.sh/incident/322421 Mon, 05 Feb 2024 10:12:00 -0000 https://status.keygen.sh/incident/322421#f4ff61ad26915ed94cad879fc17f1223231cc932604ad37323b397b9604aaf9a As of 23:48:02, we have been experiencing a major DDoS attack. We're working with our infrastructure provider on a resolution. Thank you for your patience. https://status.keygen.sh/incident/322421 Mon, 05 Feb 2024 10:12:00 -0000 https://status.keygen.sh/incident/322421#f4ff61ad26915ed94cad879fc17f1223231cc932604ad37323b397b9604aaf9a As of 23:48:02, we have been experiencing a major DDoS attack. We're working with our infrastructure provider on a resolution. Thank you for your patience. https://status.keygen.sh/incident/322421 Mon, 05 Feb 2024 10:12:00 -0000 https://status.keygen.sh/incident/322421#f4ff61ad26915ed94cad879fc17f1223231cc932604ad37323b397b9604aaf9a As of 23:48:02, we have been experiencing a major DDoS attack. We're working with our infrastructure provider on a resolution. Thank you for your patience. https://status.keygen.sh/incident/317300 Thu, 25 Jan 2024 02:39:00 -0000 https://status.keygen.sh/incident/317300#62ed0b907000dd6761504e7d3aa4952c0c75766cbb3956659f62d082f34a355c Everything looks to be back to normal. Please reach out to support@keygen.sh if you encounter any issues, or if you have further questions about what occurred. https://status.keygen.sh/incident/317300 Thu, 25 Jan 2024 02:39:00 -0000 https://status.keygen.sh/incident/317300#62ed0b907000dd6761504e7d3aa4952c0c75766cbb3956659f62d082f34a355c Everything looks to be back to normal. Please reach out to support@keygen.sh if you encounter any issues, or if you have further questions about what occurred. https://status.keygen.sh/incident/317300 Thu, 25 Jan 2024 02:39:00 -0000 https://status.keygen.sh/incident/317300#62ed0b907000dd6761504e7d3aa4952c0c75766cbb3956659f62d082f34a355c Everything looks to be back to normal. Please reach out to support@keygen.sh if you encounter any issues, or if you have further questions about what occurred. https://status.keygen.sh/incident/317300 Wed, 24 Jan 2024 22:37:00 -0000 https://status.keygen.sh/incident/317300#ed269937fcfc2beed6a66e30cba357faa341735f6bb24e9ca7ab25b47ba5b5db Today, we attempted to deploy an upgrade to our application that effected encrypted columns in our databases. Unfortunately, during the rollout, we discovered that lookups on certain encrypted attributes, such as license key, began to fail. We quickly found the cause of the problem, rolled back, and eventually deployed a fix. During this time, you may have received 401 or 404 responses indicating that a license key did not exist, when it actually did. We are continuing to monitor the situation and will be keeping this status update open for the next couple hours. No data was lost during this incident; only lookups were effected. https://status.keygen.sh/incident/317300 Wed, 24 Jan 2024 22:37:00 -0000 https://status.keygen.sh/incident/317300#ed269937fcfc2beed6a66e30cba357faa341735f6bb24e9ca7ab25b47ba5b5db Today, we attempted to deploy an upgrade to our application that effected encrypted columns in our databases. Unfortunately, during the rollout, we discovered that lookups on certain encrypted attributes, such as license key, began to fail. We quickly found the cause of the problem, rolled back, and eventually deployed a fix. During this time, you may have received 401 or 404 responses indicating that a license key did not exist, when it actually did. We are continuing to monitor the situation and will be keeping this status update open for the next couple hours. No data was lost during this incident; only lookups were effected. https://status.keygen.sh/incident/317300 Wed, 24 Jan 2024 22:37:00 -0000 https://status.keygen.sh/incident/317300#ed269937fcfc2beed6a66e30cba357faa341735f6bb24e9ca7ab25b47ba5b5db Today, we attempted to deploy an upgrade to our application that effected encrypted columns in our databases. Unfortunately, during the rollout, we discovered that lookups on certain encrypted attributes, such as license key, began to fail. We quickly found the cause of the problem, rolled back, and eventually deployed a fix. During this time, you may have received 401 or 404 responses indicating that a license key did not exist, when it actually did. We are continuing to monitor the situation and will be keeping this status update open for the next couple hours. No data was lost during this incident; only lookups were effected. https://status.keygen.sh/incident/301275 Tue, 19 Dec 2023 03:30:00 +0000 https://status.keygen.sh/incident/301275#763664b1c202199b3c6b755a94e558a91c53c6f14c8b9c77928c658dc0578147 Maintenance completed https://status.keygen.sh/incident/301275 Tue, 19 Dec 2023 03:30:00 +0000 https://status.keygen.sh/incident/301275#763664b1c202199b3c6b755a94e558a91c53c6f14c8b9c77928c658dc0578147 Maintenance completed https://status.keygen.sh/incident/301275 Tue, 19 Dec 2023 03:30:00 +0000 https://status.keygen.sh/incident/301275#763664b1c202199b3c6b755a94e558a91c53c6f14c8b9c77928c658dc0578147 Maintenance completed https://status.keygen.sh/incident/301275 Tue, 19 Dec 2023 03:00:00 -0000 https://status.keygen.sh/incident/301275#b4b14c2e971fd20186d08e115d212cbc022550cc9cb6e3273c4b69737c3dc582 We will be performing a scheduled upgrade of our Postgres datastores on December 19th 2023 at 3am UTC (i.e. the 18th @ 9pm CST). During the upgrade and changeover there may be up to 30 minutes of downtime. https://status.keygen.sh/incident/301275 Tue, 19 Dec 2023 03:00:00 -0000 https://status.keygen.sh/incident/301275#b4b14c2e971fd20186d08e115d212cbc022550cc9cb6e3273c4b69737c3dc582 We will be performing a scheduled upgrade of our Postgres datastores on December 19th 2023 at 3am UTC (i.e. the 18th @ 9pm CST). During the upgrade and changeover there may be up to 30 minutes of downtime. https://status.keygen.sh/incident/301275 Tue, 19 Dec 2023 03:00:00 -0000 https://status.keygen.sh/incident/301275#b4b14c2e971fd20186d08e115d212cbc022550cc9cb6e3273c4b69737c3dc582 We will be performing a scheduled upgrade of our Postgres datastores on December 19th 2023 at 3am UTC (i.e. the 18th @ 9pm CST). During the upgrade and changeover there may be up to 30 minutes of downtime. https://status.keygen.sh/incident/302160 Mon, 18 Dec 2023 17:00:00 -0000 https://status.keygen.sh/incident/302160#da1f28934fe22c81df8ff3dacefdad60c3714bbfc624a6787fd7af18788fe93b Today at 7am CST, we experienced a partial outage due to elevated API volume. We scaled up our infrastructure to accommodate the unexpected volume, but due to other factors, we continued to see elevated error rates. This was due to our event backlog filling up faster than we could process (due to multiple factors including webhook delivery failures), eventually leading to an out-of-memory failure on our Redis datastores, resulting in further failed requests. We resolved the webhook-related failures, which relieved pressure on our Redis datastores, and scaled up our worker formation to process the backlog as quick a possible. Full service was restored at 8am CST, but the event logs were still delayed. As of 10am CST, the backlog has been cleared and service is operating normally. We apologize for the inconvenience. As part of our database upgrade later tonight, we will be better prepared for unexpected spikes in volume like this moving forward. https://status.keygen.sh/incident/302160 Mon, 18 Dec 2023 17:00:00 -0000 https://status.keygen.sh/incident/302160#da1f28934fe22c81df8ff3dacefdad60c3714bbfc624a6787fd7af18788fe93b Today at 7am CST, we experienced a partial outage due to elevated API volume. We scaled up our infrastructure to accommodate the unexpected volume, but due to other factors, we continued to see elevated error rates. This was due to our event backlog filling up faster than we could process (due to multiple factors including webhook delivery failures), eventually leading to an out-of-memory failure on our Redis datastores, resulting in further failed requests. We resolved the webhook-related failures, which relieved pressure on our Redis datastores, and scaled up our worker formation to process the backlog as quick a possible. Full service was restored at 8am CST, but the event logs were still delayed. As of 10am CST, the backlog has been cleared and service is operating normally. We apologize for the inconvenience. As part of our database upgrade later tonight, we will be better prepared for unexpected spikes in volume like this moving forward. https://status.keygen.sh/incident/302160 Mon, 18 Dec 2023 17:00:00 -0000 https://status.keygen.sh/incident/302160#da1f28934fe22c81df8ff3dacefdad60c3714bbfc624a6787fd7af18788fe93b Today at 7am CST, we experienced a partial outage due to elevated API volume. We scaled up our infrastructure to accommodate the unexpected volume, but due to other factors, we continued to see elevated error rates. This was due to our event backlog filling up faster than we could process (due to multiple factors including webhook delivery failures), eventually leading to an out-of-memory failure on our Redis datastores, resulting in further failed requests. We resolved the webhook-related failures, which relieved pressure on our Redis datastores, and scaled up our worker formation to process the backlog as quick a possible. Full service was restored at 8am CST, but the event logs were still delayed. As of 10am CST, the backlog has been cleared and service is operating normally. We apologize for the inconvenience. As part of our database upgrade later tonight, we will be better prepared for unexpected spikes in volume like this moving forward. https://status.keygen.sh/incident/282013 Sat, 11 Nov 2023 07:30:00 +0000 https://status.keygen.sh/incident/282013#f347f01189f0b52322e0e6743dabf87c30f0369f70c54e2a9367beb9b3235b1c Maintenance completed https://status.keygen.sh/incident/282013 Sat, 11 Nov 2023 07:30:00 +0000 https://status.keygen.sh/incident/282013#f347f01189f0b52322e0e6743dabf87c30f0369f70c54e2a9367beb9b3235b1c Maintenance completed https://status.keygen.sh/incident/282013 Sat, 11 Nov 2023 07:30:00 +0000 https://status.keygen.sh/incident/282013#f347f01189f0b52322e0e6743dabf87c30f0369f70c54e2a9367beb9b3235b1c Maintenance completed https://status.keygen.sh/incident/282013 Sat, 11 Nov 2023 07:00:00 -0000 https://status.keygen.sh/incident/282013#4c19ec2ea18174eb6acf9c2c73a693f6c7c2585fe6dae307c88f928301963b4c We will be performing scheduled maintenance on our Postgres datastores on November 11th 2023 at 7am UTC. During the changeover there may be up to 5 minutes of downtime. https://status.keygen.sh/incident/282013 Sat, 11 Nov 2023 07:00:00 -0000 https://status.keygen.sh/incident/282013#4c19ec2ea18174eb6acf9c2c73a693f6c7c2585fe6dae307c88f928301963b4c We will be performing scheduled maintenance on our Postgres datastores on November 11th 2023 at 7am UTC. During the changeover there may be up to 5 minutes of downtime. https://status.keygen.sh/incident/282013 Sat, 11 Nov 2023 07:00:00 -0000 https://status.keygen.sh/incident/282013#4c19ec2ea18174eb6acf9c2c73a693f6c7c2585fe6dae307c88f928301963b4c We will be performing scheduled maintenance on our Postgres datastores on November 11th 2023 at 7am UTC. During the changeover there may be up to 5 minutes of downtime. https://status.keygen.sh/incident/248726 Sat, 26 Aug 2023 07:30:00 +0000 https://status.keygen.sh/incident/248726#25cd8944d94d6dbfd09d2a13f0097af89a8ec2f2bb48808ff13faeddb66995c3 Maintenance completed https://status.keygen.sh/incident/248726 Sat, 26 Aug 2023 07:30:00 +0000 https://status.keygen.sh/incident/248726#25cd8944d94d6dbfd09d2a13f0097af89a8ec2f2bb48808ff13faeddb66995c3 Maintenance completed https://status.keygen.sh/incident/248726 Sat, 26 Aug 2023 07:30:00 +0000 https://status.keygen.sh/incident/248726#25cd8944d94d6dbfd09d2a13f0097af89a8ec2f2bb48808ff13faeddb66995c3 Maintenance completed https://status.keygen.sh/incident/248726 Sat, 26 Aug 2023 07:00:00 -0000 https://status.keygen.sh/incident/248726#b654c4389cf262486356814edefe1968e04b3bde5565f96cad2f2069a686c4ca We will be performing scheduled maintenance on our Postgres datastores on August 26th 2023 at 7am UTC. During the changeover there may be up to 5 minutes of downtime, but it should largely go unnoticed. https://status.keygen.sh/incident/248726 Sat, 26 Aug 2023 07:00:00 -0000 https://status.keygen.sh/incident/248726#b654c4389cf262486356814edefe1968e04b3bde5565f96cad2f2069a686c4ca We will be performing scheduled maintenance on our Postgres datastores on August 26th 2023 at 7am UTC. During the changeover there may be up to 5 minutes of downtime, but it should largely go unnoticed. https://status.keygen.sh/incident/248726 Sat, 26 Aug 2023 07:00:00 -0000 https://status.keygen.sh/incident/248726#b654c4389cf262486356814edefe1968e04b3bde5565f96cad2f2069a686c4ca We will be performing scheduled maintenance on our Postgres datastores on August 26th 2023 at 7am UTC. During the changeover there may be up to 5 minutes of downtime, but it should largely go unnoticed. https://status.keygen.sh/incident/239802 Sun, 06 Aug 2023 00:30:00 +0000 https://status.keygen.sh/incident/239802#5541812653f9ee21d12338b6e4847e900f159d1c15b5c3f312a02a6055d8c0bd Maintenance completed https://status.keygen.sh/incident/239802 Sun, 06 Aug 2023 00:30:00 +0000 https://status.keygen.sh/incident/239802#5541812653f9ee21d12338b6e4847e900f159d1c15b5c3f312a02a6055d8c0bd Maintenance completed https://status.keygen.sh/incident/239802 Sun, 06 Aug 2023 00:30:00 +0000 https://status.keygen.sh/incident/239802#5541812653f9ee21d12338b6e4847e900f159d1c15b5c3f312a02a6055d8c0bd Maintenance completed https://status.keygen.sh/incident/239802 Sun, 06 Aug 2023 00:00:00 -0000 https://status.keygen.sh/incident/239802#c85025364b7e0ba8c39d76c12fa8fe012026b6fa0e81ca41926e1e145d25ec09 We will be upgrading our Redis datastores on August 6th 2023 at 12am UTC. During the changeover process, there may be up to 5 minutes of downtime. https://status.keygen.sh/incident/239802 Sun, 06 Aug 2023 00:00:00 -0000 https://status.keygen.sh/incident/239802#c85025364b7e0ba8c39d76c12fa8fe012026b6fa0e81ca41926e1e145d25ec09 We will be upgrading our Redis datastores on August 6th 2023 at 12am UTC. During the changeover process, there may be up to 5 minutes of downtime. https://status.keygen.sh/incident/239802 Sun, 06 Aug 2023 00:00:00 -0000 https://status.keygen.sh/incident/239802#c85025364b7e0ba8c39d76c12fa8fe012026b6fa0e81ca41926e1e145d25ec09 We will be upgrading our Redis datastores on August 6th 2023 at 12am UTC. During the changeover process, there may be up to 5 minutes of downtime. https://status.keygen.sh/incident/216375 Sat, 10 Jun 2023 00:30:00 +0000 https://status.keygen.sh/incident/216375#c4128989c5b883cab3f88f97ea67d4499da28c399eeb12d411529734b8fce7a3 Maintenance completed https://status.keygen.sh/incident/216375 Sat, 10 Jun 2023 00:30:00 +0000 https://status.keygen.sh/incident/216375#c4128989c5b883cab3f88f97ea67d4499da28c399eeb12d411529734b8fce7a3 Maintenance completed https://status.keygen.sh/incident/216375 Sat, 10 Jun 2023 00:30:00 +0000 https://status.keygen.sh/incident/216375#c4128989c5b883cab3f88f97ea67d4499da28c399eeb12d411529734b8fce7a3 Maintenance completed https://status.keygen.sh/incident/216375 Sat, 10 Jun 2023 00:00:00 -0000 https://status.keygen.sh/incident/216375#c0dad15fb663fd3518fd18a90c0a74cfb5f86d487da77c6a978346f4fda9c951 We will be performing scheduled maintenance on our Postgres datastores on June 10th 2023 at 12am UTC. During the changeover there may be up to 5 minutes of downtime, but it should largely go unnoticed. https://status.keygen.sh/incident/216375 Sat, 10 Jun 2023 00:00:00 -0000 https://status.keygen.sh/incident/216375#c0dad15fb663fd3518fd18a90c0a74cfb5f86d487da77c6a978346f4fda9c951 We will be performing scheduled maintenance on our Postgres datastores on June 10th 2023 at 12am UTC. During the changeover there may be up to 5 minutes of downtime, but it should largely go unnoticed. https://status.keygen.sh/incident/216375 Sat, 10 Jun 2023 00:00:00 -0000 https://status.keygen.sh/incident/216375#c0dad15fb663fd3518fd18a90c0a74cfb5f86d487da77c6a978346f4fda9c951 We will be performing scheduled maintenance on our Postgres datastores on June 10th 2023 at 12am UTC. During the changeover there may be up to 5 minutes of downtime, but it should largely go unnoticed. https://status.keygen.sh/incident/193603 Mon, 10 Apr 2023 04:30:00 +0000 https://status.keygen.sh/incident/193603#5de00ba239ed9318d0eed0ef80602e228f043da8bfe14e2539a9cb3116909dd2 Maintenance completed https://status.keygen.sh/incident/193603 Mon, 10 Apr 2023 04:30:00 +0000 https://status.keygen.sh/incident/193603#5de00ba239ed9318d0eed0ef80602e228f043da8bfe14e2539a9cb3116909dd2 Maintenance completed https://status.keygen.sh/incident/193603 Mon, 10 Apr 2023 04:30:00 +0000 https://status.keygen.sh/incident/193603#5de00ba239ed9318d0eed0ef80602e228f043da8bfe14e2539a9cb3116909dd2 Maintenance completed https://status.keygen.sh/incident/193603 Mon, 10 Apr 2023 04:00:00 -0000 https://status.keygen.sh/incident/193603#81a62c028c499c0013d5fe4a4d5fd0ac0c69bca20ea4f00ffba5fa0ff38744cb We will be upgrading our Redis datastores on Tuesday, April 10th 2023 at 4am UTC (10pm CST, 8pm PST). During the changeover process, there may be up to 5 minutes of downtime. https://status.keygen.sh/incident/193603 Mon, 10 Apr 2023 04:00:00 -0000 https://status.keygen.sh/incident/193603#81a62c028c499c0013d5fe4a4d5fd0ac0c69bca20ea4f00ffba5fa0ff38744cb We will be upgrading our Redis datastores on Tuesday, April 10th 2023 at 4am UTC (10pm CST, 8pm PST). During the changeover process, there may be up to 5 minutes of downtime. https://status.keygen.sh/incident/193603 Mon, 10 Apr 2023 04:00:00 -0000 https://status.keygen.sh/incident/193603#81a62c028c499c0013d5fe4a4d5fd0ac0c69bca20ea4f00ffba5fa0ff38744cb We will be upgrading our Redis datastores on Tuesday, April 10th 2023 at 4am UTC (10pm CST, 8pm PST). During the changeover process, there may be up to 5 minutes of downtime. https://status.keygen.sh/incident/188017 Sat, 25 Mar 2023 07:30:00 +0000 https://status.keygen.sh/incident/188017#c88b05c201364321b8df28da2b6a1783e6894d0301eedbacdaa7fc0b67a0cd6a Maintenance completed https://status.keygen.sh/incident/188017 Sat, 25 Mar 2023 07:30:00 +0000 https://status.keygen.sh/incident/188017#c88b05c201364321b8df28da2b6a1783e6894d0301eedbacdaa7fc0b67a0cd6a Maintenance completed https://status.keygen.sh/incident/188017 Sat, 25 Mar 2023 07:30:00 +0000 https://status.keygen.sh/incident/188017#c88b05c201364321b8df28da2b6a1783e6894d0301eedbacdaa7fc0b67a0cd6a Maintenance completed https://status.keygen.sh/incident/188017 Sat, 25 Mar 2023 07:00:00 -0000 https://status.keygen.sh/incident/188017#27db759f11d87c0b3535bf672086c8da026e03cec27ef30601d530cac22e2166 Routine database maintenance. The API may be unavailable for up to 5 minutes while we do a changeover. Time is UTC. https://status.keygen.sh/incident/188017 Sat, 25 Mar 2023 07:00:00 -0000 https://status.keygen.sh/incident/188017#27db759f11d87c0b3535bf672086c8da026e03cec27ef30601d530cac22e2166 Routine database maintenance. The API may be unavailable for up to 5 minutes while we do a changeover. Time is UTC. https://status.keygen.sh/incident/188017 Sat, 25 Mar 2023 07:00:00 -0000 https://status.keygen.sh/incident/188017#27db759f11d87c0b3535bf672086c8da026e03cec27ef30601d530cac22e2166 Routine database maintenance. The API may be unavailable for up to 5 minutes while we do a changeover. Time is UTC. https://status.keygen.sh/incident/150963 Sun, 15 Jan 2023 00:30:00 +0000 https://status.keygen.sh/incident/150963#5d549d9392de1e54ce9533dd0d8f73f80965c0af2fad1df21071922bd1129e3a Maintenance completed https://status.keygen.sh/incident/150963 Sun, 15 Jan 2023 00:30:00 +0000 https://status.keygen.sh/incident/150963#5d549d9392de1e54ce9533dd0d8f73f80965c0af2fad1df21071922bd1129e3a Maintenance completed https://status.keygen.sh/incident/150963 Sun, 15 Jan 2023 00:30:00 +0000 https://status.keygen.sh/incident/150963#5d549d9392de1e54ce9533dd0d8f73f80965c0af2fad1df21071922bd1129e3a Maintenance completed https://status.keygen.sh/incident/150963 Sun, 15 Jan 2023 00:00:00 -0000 https://status.keygen.sh/incident/150963#2d57454c24cf0664300bcc5f9682ed11b746386d8a532f70f56a1a4cbde7035b Routine database maintenance. The API may be unavailable for up to 5 minutes as we do a changeover. Scheduled time is 2023-01-15 00:00:00 UTC. https://status.keygen.sh/incident/150963 Sun, 15 Jan 2023 00:00:00 -0000 https://status.keygen.sh/incident/150963#2d57454c24cf0664300bcc5f9682ed11b746386d8a532f70f56a1a4cbde7035b Routine database maintenance. The API may be unavailable for up to 5 minutes as we do a changeover. Scheduled time is 2023-01-15 00:00:00 UTC. https://status.keygen.sh/incident/150963 Sun, 15 Jan 2023 00:00:00 -0000 https://status.keygen.sh/incident/150963#2d57454c24cf0664300bcc5f9682ed11b746386d8a532f70f56a1a4cbde7035b Routine database maintenance. The API may be unavailable for up to 5 minutes as we do a changeover. Scheduled time is 2023-01-15 00:00:00 UTC. https://status.keygen.sh/incident/129277 Sun, 30 Oct 2022 00:30:00 +0000 https://status.keygen.sh/incident/129277#5ca4fd88a3a4739bee5cdaaad8bb5302325f38739e748a6ebb5c7f670936251d Maintenance completed https://status.keygen.sh/incident/129277 Sun, 30 Oct 2022 00:30:00 +0000 https://status.keygen.sh/incident/129277#5ca4fd88a3a4739bee5cdaaad8bb5302325f38739e748a6ebb5c7f670936251d Maintenance completed https://status.keygen.sh/incident/129277 Sun, 30 Oct 2022 00:30:00 +0000 https://status.keygen.sh/incident/129277#5ca4fd88a3a4739bee5cdaaad8bb5302325f38739e748a6ebb5c7f670936251d Maintenance completed https://status.keygen.sh/incident/129277 Sun, 30 Oct 2022 00:00:00 -0000 https://status.keygen.sh/incident/129277#597f7f2179bc217c677da3394b24cdde05b80489fb155740771c5edda82d30ae We'll be performing routine database maintenance on our Redis cluster. There may be a small amount of downtime while we do a change over, but this maintenance should largely go unnoticed. Scheduled time is 2022-10-30 00:00:00 UTC. https://status.keygen.sh/incident/129277 Sun, 30 Oct 2022 00:00:00 -0000 https://status.keygen.sh/incident/129277#597f7f2179bc217c677da3394b24cdde05b80489fb155740771c5edda82d30ae We'll be performing routine database maintenance on our Redis cluster. There may be a small amount of downtime while we do a change over, but this maintenance should largely go unnoticed. Scheduled time is 2022-10-30 00:00:00 UTC. https://status.keygen.sh/incident/129277 Sun, 30 Oct 2022 00:00:00 -0000 https://status.keygen.sh/incident/129277#597f7f2179bc217c677da3394b24cdde05b80489fb155740771c5edda82d30ae We'll be performing routine database maintenance on our Redis cluster. There may be a small amount of downtime while we do a change over, but this maintenance should largely go unnoticed. Scheduled time is 2022-10-30 00:00:00 UTC. https://status.keygen.sh/incident/118976 Sat, 24 Sep 2022 08:00:00 +0000 https://status.keygen.sh/incident/118976#41cf7f6238bde4c4af34e22fec1c8ca76b8f1fd303a1c0643a191199ea35f0e0 Maintenance completed https://status.keygen.sh/incident/118976 Sat, 24 Sep 2022 08:00:00 +0000 https://status.keygen.sh/incident/118976#41cf7f6238bde4c4af34e22fec1c8ca76b8f1fd303a1c0643a191199ea35f0e0 Maintenance completed https://status.keygen.sh/incident/118976 Sat, 24 Sep 2022 08:00:00 +0000 https://status.keygen.sh/incident/118976#41cf7f6238bde4c4af34e22fec1c8ca76b8f1fd303a1c0643a191199ea35f0e0 Maintenance completed https://status.keygen.sh/incident/118976 Sat, 24 Sep 2022 07:00:00 -0000 https://status.keygen.sh/incident/118976#997948d6ff539f420ab88e97930e6cec3bc6ac631ddf4f7827ea6b2f0f8cf075 We will be performing maintenance on our Postgres cluster. No downtime is expected, but some requests may be delayed for a moment while we switch over to our HA standbys. Scheduled time is 2022-09-24 07:00:00 UTC. https://status.keygen.sh/incident/118976 Sat, 24 Sep 2022 07:00:00 -0000 https://status.keygen.sh/incident/118976#997948d6ff539f420ab88e97930e6cec3bc6ac631ddf4f7827ea6b2f0f8cf075 We will be performing maintenance on our Postgres cluster. No downtime is expected, but some requests may be delayed for a moment while we switch over to our HA standbys. Scheduled time is 2022-09-24 07:00:00 UTC. https://status.keygen.sh/incident/118976 Sat, 24 Sep 2022 07:00:00 -0000 https://status.keygen.sh/incident/118976#997948d6ff539f420ab88e97930e6cec3bc6ac631ddf4f7827ea6b2f0f8cf075 We will be performing maintenance on our Postgres cluster. No downtime is expected, but some requests may be delayed for a moment while we switch over to our HA standbys. Scheduled time is 2022-09-24 07:00:00 UTC. https://status.keygen.sh/incident/112994 Fri, 26 Aug 2022 15:27:00 -0000 https://status.keygen.sh/incident/112994#984244549636057fda7a36404c6ba69e0bc40722c85ace37372e93734402997b Looks to have been short lived. We’ll continue to monitor the situation, but it looks to be resolved. https://status.keygen.sh/incident/112994 Fri, 26 Aug 2022 15:27:00 -0000 https://status.keygen.sh/incident/112994#984244549636057fda7a36404c6ba69e0bc40722c85ace37372e93734402997b Looks to have been short lived. We’ll continue to monitor the situation, but it looks to be resolved. https://status.keygen.sh/incident/112994 Fri, 26 Aug 2022 15:27:00 -0000 https://status.keygen.sh/incident/112994#984244549636057fda7a36404c6ba69e0bc40722c85ace37372e93734402997b Looks to have been short lived. We’ll continue to monitor the situation, but it looks to be resolved. https://status.keygen.sh/incident/112994 Fri, 26 Aug 2022 15:23:00 -0000 https://status.keygen.sh/incident/112994#9681d08b50fe4e4fb5816978a9f82ccee36c5dcae66bdc70d8c58c18a7d5e226 We are experiencing DNS issues again due to an upstream provider. We are working to resolve the issue. https://status.keygen.sh/incident/112994 Fri, 26 Aug 2022 15:23:00 -0000 https://status.keygen.sh/incident/112994#9681d08b50fe4e4fb5816978a9f82ccee36c5dcae66bdc70d8c58c18a7d5e226 We are experiencing DNS issues again due to an upstream provider. We are working to resolve the issue. https://status.keygen.sh/incident/112994 Fri, 26 Aug 2022 15:23:00 -0000 https://status.keygen.sh/incident/112994#9681d08b50fe4e4fb5816978a9f82ccee36c5dcae66bdc70d8c58c18a7d5e226 We are experiencing DNS issues again due to an upstream provider. We are working to resolve the issue. https://status.keygen.sh/incident/112108 Wed, 24 Aug 2022 01:00:00 -0000 https://status.keygen.sh/incident/112108#0987fe937501f1644b6f5d662a416d0df400c3d7df077319669a45d2bc18aa0b This is now resolved. https://status.keygen.sh/incident/112108 Wed, 24 Aug 2022 01:00:00 -0000 https://status.keygen.sh/incident/112108#0987fe937501f1644b6f5d662a416d0df400c3d7df077319669a45d2bc18aa0b This is now resolved. https://status.keygen.sh/incident/112108 Wed, 24 Aug 2022 01:00:00 -0000 https://status.keygen.sh/incident/112108#0987fe937501f1644b6f5d662a416d0df400c3d7df077319669a45d2bc18aa0b This is now resolved. https://status.keygen.sh/incident/112108 Tue, 23 Aug 2022 23:28:00 -0000 https://status.keygen.sh/incident/112108#fb7329dfde0a5d73e8fba52bf089f89dcc79d33f0a52432556688d1080b6658a We are investigating issues with an upstream DNS provider. More information is available here: https://status.heroku.com/ We are working with Heroku to resolve the issue. https://status.keygen.sh/incident/112108 Tue, 23 Aug 2022 23:28:00 -0000 https://status.keygen.sh/incident/112108#fb7329dfde0a5d73e8fba52bf089f89dcc79d33f0a52432556688d1080b6658a We are investigating issues with an upstream DNS provider. More information is available here: https://status.heroku.com/ We are working with Heroku to resolve the issue. https://status.keygen.sh/incident/112108 Tue, 23 Aug 2022 23:28:00 -0000 https://status.keygen.sh/incident/112108#fb7329dfde0a5d73e8fba52bf089f89dcc79d33f0a52432556688d1080b6658a We are investigating issues with an upstream DNS provider. More information is available here: https://status.heroku.com/ We are working with Heroku to resolve the issue. https://status.keygen.sh/incident/98608 Wed, 06 Jul 2022 02:30:00 +0000 https://status.keygen.sh/incident/98608#a3f4b580b73ed2a016b73901d3e14fa681cbd5c963276d0cadee7b108957e0da Maintenance completed https://status.keygen.sh/incident/98608 Wed, 06 Jul 2022 02:30:00 +0000 https://status.keygen.sh/incident/98608#a3f4b580b73ed2a016b73901d3e14fa681cbd5c963276d0cadee7b108957e0da Maintenance completed https://status.keygen.sh/incident/98608 Wed, 06 Jul 2022 02:30:00 +0000 https://status.keygen.sh/incident/98608#a3f4b580b73ed2a016b73901d3e14fa681cbd5c963276d0cadee7b108957e0da Maintenance completed https://status.keygen.sh/incident/98608 Wed, 06 Jul 2022 02:00:00 -0000 https://status.keygen.sh/incident/98608#97dee54f9a7c817a6d89393a8aa5dee4e7239705fd332070f8d4a948fd933730 We will be performing scheduled software upgrades on our production databases. Potential downtime is up to 30 minutes but expected to be much shorter. Scheduled time is 02:00:00 UTC timezone. https://status.keygen.sh/incident/98608 Wed, 06 Jul 2022 02:00:00 -0000 https://status.keygen.sh/incident/98608#97dee54f9a7c817a6d89393a8aa5dee4e7239705fd332070f8d4a948fd933730 We will be performing scheduled software upgrades on our production databases. Potential downtime is up to 30 minutes but expected to be much shorter. Scheduled time is 02:00:00 UTC timezone. https://status.keygen.sh/incident/98608 Wed, 06 Jul 2022 02:00:00 -0000 https://status.keygen.sh/incident/98608#97dee54f9a7c817a6d89393a8aa5dee4e7239705fd332070f8d4a948fd933730 We will be performing scheduled software upgrades on our production databases. Potential downtime is up to 30 minutes but expected to be much shorter. Scheduled time is 02:00:00 UTC timezone. https://status.keygen.sh/incident/93468 Thu, 09 Jun 2022 00:00:00 +0000 https://status.keygen.sh/incident/93468#173a08d88610ded36ca16602691832560bc8d32096ba0f8ed06bd484f64f3402 Maintenance completed https://status.keygen.sh/incident/93468 Wed, 08 Jun 2022 15:30:00 -0000 https://status.keygen.sh/incident/93468#22f9f939efb71972c0a5e3817920696a92193905fe7310e8a4161e1d97820295 We're currently performing maintenance on request logs and other types of metric logs. Dashboard metrics and logs may be intermittently unavailable until complete. We apologize for any inconvenience. https://status.keygen.sh/incident/82730 Sat, 23 Apr 2022 07:30:00 +0000 https://status.keygen.sh/incident/82730#46451a337330b32406ced8974355eafc9700e4a4b5938fb927e7cd6a5afa8749 Maintenance completed https://status.keygen.sh/incident/82730 Sat, 23 Apr 2022 07:30:00 +0000 https://status.keygen.sh/incident/82730#46451a337330b32406ced8974355eafc9700e4a4b5938fb927e7cd6a5afa8749 Maintenance completed https://status.keygen.sh/incident/82730 Sat, 23 Apr 2022 07:30:00 +0000 https://status.keygen.sh/incident/82730#46451a337330b32406ced8974355eafc9700e4a4b5938fb927e7cd6a5afa8749 Maintenance completed https://status.keygen.sh/incident/82730 Sat, 23 Apr 2022 07:00:00 -0000 https://status.keygen.sh/incident/82730#e840f02bc343c2f11434e01f52c3c54b775ba8f10faa232a88d9268cfe9ee4a4 Scheduled maintenance window for our databases. The service may be unavailable for a couple minutes as we do a changeover. https://status.keygen.sh/incident/82730 Sat, 23 Apr 2022 07:00:00 -0000 https://status.keygen.sh/incident/82730#e840f02bc343c2f11434e01f52c3c54b775ba8f10faa232a88d9268cfe9ee4a4 Scheduled maintenance window for our databases. The service may be unavailable for a couple minutes as we do a changeover. https://status.keygen.sh/incident/82730 Sat, 23 Apr 2022 07:00:00 -0000 https://status.keygen.sh/incident/82730#e840f02bc343c2f11434e01f52c3c54b775ba8f10faa232a88d9268cfe9ee4a4 Scheduled maintenance window for our databases. The service may be unavailable for a couple minutes as we do a changeover. https://status.keygen.sh/incident/79461 Sun, 03 Apr 2022 00:15:00 +0000 https://status.keygen.sh/incident/79461#73002f3af35e11e45b18d87c034d4f195ff4a7e792d8c2c70d7ee34eb55d53f0 Maintenance completed https://status.keygen.sh/incident/79461 Sun, 03 Apr 2022 00:15:00 +0000 https://status.keygen.sh/incident/79461#73002f3af35e11e45b18d87c034d4f195ff4a7e792d8c2c70d7ee34eb55d53f0 Maintenance completed https://status.keygen.sh/incident/79461 Sun, 03 Apr 2022 00:15:00 +0000 https://status.keygen.sh/incident/79461#73002f3af35e11e45b18d87c034d4f195ff4a7e792d8c2c70d7ee34eb55d53f0 Maintenance completed https://status.keygen.sh/incident/79461 Sun, 03 Apr 2022 00:00:00 -0000 https://status.keygen.sh/incident/79461#d1b1b0eee25d1f4901be4735daad9e30c4754a822dac5c25be91d84c2eba8ae9 We'll be performing maintenance on our Redis datastores. There may be a few minutes of downtime while we do a changeover. https://status.keygen.sh/incident/79461 Sun, 03 Apr 2022 00:00:00 -0000 https://status.keygen.sh/incident/79461#d1b1b0eee25d1f4901be4735daad9e30c4754a822dac5c25be91d84c2eba8ae9 We'll be performing maintenance on our Redis datastores. There may be a few minutes of downtime while we do a changeover. https://status.keygen.sh/incident/79461 Sun, 03 Apr 2022 00:00:00 -0000 https://status.keygen.sh/incident/79461#d1b1b0eee25d1f4901be4735daad9e30c4754a822dac5c25be91d84c2eba8ae9 We'll be performing maintenance on our Redis datastores. There may be a few minutes of downtime while we do a changeover. https://status.keygen.sh/incident/72656 Thu, 24 Feb 2022 17:35:00 -0000 https://status.keygen.sh/incident/72656#9141f4431593c30bf83d9ce834d8526b72c909be76f1e6ca5bfb691ea9bdfbbb Webhooks have fully recovered. All systems are back to normal. https://status.keygen.sh/incident/72656 Thu, 24 Feb 2022 17:35:00 -0000 https://status.keygen.sh/incident/72656#9141f4431593c30bf83d9ce834d8526b72c909be76f1e6ca5bfb691ea9bdfbbb Webhooks have fully recovered. All systems are back to normal. https://status.keygen.sh/incident/72656 Thu, 24 Feb 2022 17:35:00 -0000 https://status.keygen.sh/incident/72656#9141f4431593c30bf83d9ce834d8526b72c909be76f1e6ca5bfb691ea9bdfbbb Webhooks have fully recovered. All systems are back to normal. https://status.keygen.sh/incident/72656 Thu, 24 Feb 2022 17:18:00 -0000 https://status.keygen.sh/incident/72656#b5240b7a7d0cbb22769bd6eb1fba2e818055e74603c51f97d7facac20ace7b2b All services have recovered. It may take awhile to work through our webhook backlog. We will continue to monitor the situation. https://status.keygen.sh/incident/72656 Thu, 24 Feb 2022 17:18:00 -0000 https://status.keygen.sh/incident/72656#b5240b7a7d0cbb22769bd6eb1fba2e818055e74603c51f97d7facac20ace7b2b All services have recovered. It may take awhile to work through our webhook backlog. We will continue to monitor the situation. https://status.keygen.sh/incident/72656 Thu, 24 Feb 2022 17:18:00 -0000 https://status.keygen.sh/incident/72656#b5240b7a7d0cbb22769bd6eb1fba2e818055e74603c51f97d7facac20ace7b2b All services have recovered. It may take awhile to work through our webhook backlog. We will continue to monitor the situation. https://status.keygen.sh/incident/72656 Thu, 24 Feb 2022 17:16:00 -0000 https://status.keygen.sh/incident/72656#8b2f1a3203ff75d556036bb7e1ddcc179e3f0aabe256c461ef169d3c5e012d8a Platform functionality is starting to recover. We will continue to monitor as things recover further. Thank you again for your patience. https://status.keygen.sh/incident/72656 Thu, 24 Feb 2022 17:16:00 -0000 https://status.keygen.sh/incident/72656#8b2f1a3203ff75d556036bb7e1ddcc179e3f0aabe256c461ef169d3c5e012d8a Platform functionality is starting to recover. We will continue to monitor as things recover further. Thank you again for your patience. https://status.keygen.sh/incident/72656 Thu, 24 Feb 2022 17:16:00 -0000 https://status.keygen.sh/incident/72656#8b2f1a3203ff75d556036bb7e1ddcc179e3f0aabe256c461ef169d3c5e012d8a Platform functionality is starting to recover. We will continue to monitor as things recover further. Thank you again for your patience. https://status.keygen.sh/incident/72656 Thu, 24 Feb 2022 17:10:00 -0000 https://status.keygen.sh/incident/72656#a8897dacc16714ae81ec7d0a5028ab7a68694964b45e07b0db8d0dcf0c275b69 We are still waiting on Heroku to restore platform functionality. We have heard reports of some applications recovering, but Keygen is largely still down. Since we have been down for over 30 minutes, we are initiating disaster recovery procedures. We sincerely apologize for the trouble. https://status.keygen.sh/incident/72656 Thu, 24 Feb 2022 17:10:00 -0000 https://status.keygen.sh/incident/72656#a8897dacc16714ae81ec7d0a5028ab7a68694964b45e07b0db8d0dcf0c275b69 We are still waiting on Heroku to restore platform functionality. We have heard reports of some applications recovering, but Keygen is largely still down. Since we have been down for over 30 minutes, we are initiating disaster recovery procedures. We sincerely apologize for the trouble. https://status.keygen.sh/incident/72656 Thu, 24 Feb 2022 17:10:00 -0000 https://status.keygen.sh/incident/72656#a8897dacc16714ae81ec7d0a5028ab7a68694964b45e07b0db8d0dcf0c275b69 We are still waiting on Heroku to restore platform functionality. We have heard reports of some applications recovering, but Keygen is largely still down. Since we have been down for over 30 minutes, we are initiating disaster recovery procedures. We sincerely apologize for the trouble. https://status.keygen.sh/incident/72656 Thu, 24 Feb 2022 16:36:00 -0000 https://status.keygen.sh/incident/72656#f0bca151923350d96f50d6dc9fc6b5fd9970e037eb56c14e983fb548716aacf1 We are currently experiencing an outage with an upstream infrastructure provider, Heroku. We are attempting to resolve the issue as quickly as possible. We will post updates here when we have them. Heroku's status is available here: https://status.heroku.com. Comments: https://news.ycombinator.com/item?id=30456688. https://status.keygen.sh/incident/72656 Thu, 24 Feb 2022 16:36:00 -0000 https://status.keygen.sh/incident/72656#f0bca151923350d96f50d6dc9fc6b5fd9970e037eb56c14e983fb548716aacf1 We are currently experiencing an outage with an upstream infrastructure provider, Heroku. We are attempting to resolve the issue as quickly as possible. We will post updates here when we have them. Heroku's status is available here: https://status.heroku.com. Comments: https://news.ycombinator.com/item?id=30456688. https://status.keygen.sh/incident/72656 Thu, 24 Feb 2022 16:36:00 -0000 https://status.keygen.sh/incident/72656#f0bca151923350d96f50d6dc9fc6b5fd9970e037eb56c14e983fb548716aacf1 We are currently experiencing an outage with an upstream infrastructure provider, Heroku. We are attempting to resolve the issue as quickly as possible. We will post updates here when we have them. Heroku's status is available here: https://status.heroku.com. Comments: https://news.ycombinator.com/item?id=30456688. https://status.keygen.sh/incident/69133 Sat, 12 Feb 2022 11:00:00 +0000 https://status.keygen.sh/incident/69133#9206955150d9995957cbb677e799596c3b3a125a7d507ab09ca791338000e159 Maintenance completed https://status.keygen.sh/incident/69133 Sat, 12 Feb 2022 11:00:00 +0000 https://status.keygen.sh/incident/69133#9206955150d9995957cbb677e799596c3b3a125a7d507ab09ca791338000e159 Maintenance completed https://status.keygen.sh/incident/69133 Sat, 12 Feb 2022 11:00:00 +0000 https://status.keygen.sh/incident/69133#9206955150d9995957cbb677e799596c3b3a125a7d507ab09ca791338000e159 Maintenance completed https://status.keygen.sh/incident/69133 Sat, 12 Feb 2022 07:00:00 -0000 https://status.keygen.sh/incident/69133#1d01bc40dda232aaacbf5bb5b448ee966e6834b443ec29e638b0b29da220a04d Scheduled maintenance window for our databases. The service may be unavailable for a couple minutes as we do a changeover. https://status.keygen.sh/incident/69133 Sat, 12 Feb 2022 07:00:00 -0000 https://status.keygen.sh/incident/69133#1d01bc40dda232aaacbf5bb5b448ee966e6834b443ec29e638b0b29da220a04d Scheduled maintenance window for our databases. The service may be unavailable for a couple minutes as we do a changeover. https://status.keygen.sh/incident/69133 Sat, 12 Feb 2022 07:00:00 -0000 https://status.keygen.sh/incident/69133#1d01bc40dda232aaacbf5bb5b448ee966e6834b443ec29e638b0b29da220a04d Scheduled maintenance window for our databases. The service may be unavailable for a couple minutes as we do a changeover. https://status.keygen.sh/incident/68781 Mon, 31 Jan 2022 19:00:00 +0000 https://status.keygen.sh/incident/68781#d76663c1a7205a2cc77e3d4da9fff2f12b0b992cf37b0085e8bbc5488bf0edc1 Maintenance completed https://status.keygen.sh/incident/68781 Mon, 31 Jan 2022 18:00:00 -0000 https://status.keygen.sh/incident/68781#ed390fc70f5fcdf55d8d4de3760301c9c6b85bf674eb59116cfc8dcf18b198e1 This migration should largely go unnoticed. But for clients running an outdated version of OpenSSL, clients may begin receiving CERT_HAS_EXPIRED errors. More info here: https://keygen.sh/docs/api/connectivity/ https://status.keygen.sh/incident/67359 Sun, 23 Jan 2022 00:15:00 +0000 https://status.keygen.sh/incident/67359#43d50acf3b42a1a4135177c35e990546507c18cbffa5d555b4651d8b1464c0b5 Maintenance completed https://status.keygen.sh/incident/67359 Sun, 23 Jan 2022 00:15:00 +0000 https://status.keygen.sh/incident/67359#43d50acf3b42a1a4135177c35e990546507c18cbffa5d555b4651d8b1464c0b5 Maintenance completed https://status.keygen.sh/incident/67359 Sun, 23 Jan 2022 00:15:00 +0000 https://status.keygen.sh/incident/67359#43d50acf3b42a1a4135177c35e990546507c18cbffa5d555b4651d8b1464c0b5 Maintenance completed https://status.keygen.sh/incident/67359 Sun, 23 Jan 2022 00:00:00 -0000 https://status.keygen.sh/incident/67359#bbe26b16d0f794bddaa381637ad388dc62c91aff2191023e1118462634e39d3f Scheduled maintenance window for our cache datastores. The service may be unavailable for a couple minutes as we do a changeover. https://status.keygen.sh/incident/67359 Sun, 23 Jan 2022 00:00:00 -0000 https://status.keygen.sh/incident/67359#bbe26b16d0f794bddaa381637ad388dc62c91aff2191023e1118462634e39d3f Scheduled maintenance window for our cache datastores. The service may be unavailable for a couple minutes as we do a changeover. https://status.keygen.sh/incident/67359 Sun, 23 Jan 2022 00:00:00 -0000 https://status.keygen.sh/incident/67359#bbe26b16d0f794bddaa381637ad388dc62c91aff2191023e1118462634e39d3f Scheduled maintenance window for our cache datastores. The service may be unavailable for a couple minutes as we do a changeover. https://status.keygen.sh/incident/58967 Sun, 21 Nov 2021 04:10:00 +0000 https://status.keygen.sh/incident/58967#91bdf689a570b6b56f3abb495791cd3e8384f238aecf25e27e08cb23566ca3fe Maintenance completed https://status.keygen.sh/incident/58967 Sun, 21 Nov 2021 04:10:00 +0000 https://status.keygen.sh/incident/58967#91bdf689a570b6b56f3abb495791cd3e8384f238aecf25e27e08cb23566ca3fe Maintenance completed https://status.keygen.sh/incident/58967 Sun, 21 Nov 2021 04:10:00 +0000 https://status.keygen.sh/incident/58967#91bdf689a570b6b56f3abb495791cd3e8384f238aecf25e27e08cb23566ca3fe Maintenance completed https://status.keygen.sh/incident/58967 Sun, 21 Nov 2021 03:20:00 -0000 https://status.keygen.sh/incident/58967#8bbd511fe1acc5ecfcca2b47dba253b1efa19ddc70005f95e02a490be3bdc7b9 We will be performing scheduled software upgrades to our production databases. Expected downtime is up to 30 minutes. Scheduled time is 03:30:00 UTC timezone. https://status.keygen.sh/incident/58967 Sun, 21 Nov 2021 03:20:00 -0000 https://status.keygen.sh/incident/58967#8bbd511fe1acc5ecfcca2b47dba253b1efa19ddc70005f95e02a490be3bdc7b9 We will be performing scheduled software upgrades to our production databases. Expected downtime is up to 30 minutes. Scheduled time is 03:30:00 UTC timezone. https://status.keygen.sh/incident/58967 Sun, 21 Nov 2021 03:20:00 -0000 https://status.keygen.sh/incident/58967#8bbd511fe1acc5ecfcca2b47dba253b1efa19ddc70005f95e02a490be3bdc7b9 We will be performing scheduled software upgrades to our production databases. Expected downtime is up to 30 minutes. Scheduled time is 03:30:00 UTC timezone. https://status.keygen.sh/incident/55704 Sun, 03 Oct 2021 00:15:00 +0000 https://status.keygen.sh/incident/55704#53903d3c113f95832ae67ddc7dc29c4ef1961be28c12840f3fca9ebdd3f88061 Maintenance completed https://status.keygen.sh/incident/55704 Sun, 03 Oct 2021 00:15:00 +0000 https://status.keygen.sh/incident/55704#53903d3c113f95832ae67ddc7dc29c4ef1961be28c12840f3fca9ebdd3f88061 Maintenance completed https://status.keygen.sh/incident/55704 Sun, 03 Oct 2021 00:00:00 -0000 https://status.keygen.sh/incident/55704#dab9a5eb25869156b821b73823c9ba8a653586d4d1598e6a1b845b1f4e78ef74 Routine database maintenance. The API may be unavailable for up to 5 minutes. https://status.keygen.sh/incident/55704 Sun, 03 Oct 2021 00:00:00 -0000 https://status.keygen.sh/incident/55704#dab9a5eb25869156b821b73823c9ba8a653586d4d1598e6a1b845b1f4e78ef74 Routine database maintenance. The API may be unavailable for up to 5 minutes. https://status.keygen.sh/incident/55557 Fri, 01 Oct 2021 04:27:27 -0000 https://status.keygen.sh/incident/55557#286c97e73d241af68617fe5b39679b01ff28e426148d05f39088294ee5d7ddb1 Due to the widespread issues caused by Lets Encrypt and lack of support for outdated OpenSSL clients, we have moved to a Comodo TLS certificate. Moving forward, this should resolve all TLS connection issues caused by the expired Lets Encrypt root certificate, DST Root CA X3. We unfortunately experienced around 10–15 minutes of downtime during the migration to the new CA due to a misconfiguration on our end, but everything should be green now. Please reach out if you continue to experience any issues. https://status.keygen.sh/incident/55497 Thu, 30 Sep 2021 15:26:41 -0000 https://status.keygen.sh/incident/55497#c9638f6f221a6be6fe46745bfe9edb4436784c08647da00cf65a8951b4fc69e4 We are seeing reports of some customers receiving an expired TLS cerificate error when connecting to our APIs. Lets Encrypt's older root cert, DST Root CA X3, expired today, and we are working with customers to resolve this issue for affected devices. To resolve, Let's Encrypt's new root cert, ISRG Root X1, needs to be trusted. Unfortunately, this is currently out of our control and older clients must be updated to trust the new CA Please upgrade your ca-certificates. For example — on Unix, you will need to upgrade the ca-certificates package; on macOS, you may need to reboot, reinstall OpenSSL >= 1.1 and then upgrade curl to the latest version via Homebrew. Additional reading and steps to resolve the issues on devices that do not have an up-to-date certificate chain: https://letsencrypt.org/docs/dst-root-ca-x3-expiration-september-2021/ Other discussions and links that may help: - https://twitter.com/Scott_Helme/status/1443293844292919304 - https://community.letsencrypt.org/t/help-thread-for-dst-root-ca-x3-expiration-september-2021/149190/424 - https://github.com/mperham/sidekiq/issues/5008#issuecomment-931543300 - https://www.kobelnet.ch/2021/09/30/old-lets-encrypt-root-certificate-expiration-workaround - https://twitter.com/search?q=letsencrypt&src=typed_query&f=live We have marked this incident as resolved because it is a client issue, not a server issue. Though, we will continue to monitor if there's anything we can do on our end to lessen its impact. Please reach out if you need assistance upgrading.