Incidents reported on status page for Xchart https://status.xchart.com/ https://d1lppblt9t2x15.cloudfront.net/logos/bad06b7be02ee4cb1eb5a5cdfa422b7e.png https://status.xchart.com/ en https://status.xchart.com/incident/608510 Tue, 24 Jun 2025 21:44:47 -0000 https://status.xchart.com/incident/608510#96e11495da2c61f042f01cd7ced36fb59c416590e9a735354e153f51293e8362 Xchart.com Public Site and Case Manager recovered. https://status.xchart.com/incident/608510 Tue, 24 Jun 2025 21:44:47 -0000 https://status.xchart.com/incident/608510#96e11495da2c61f042f01cd7ced36fb59c416590e9a735354e153f51293e8362 Xchart.com Public Site and Case Manager recovered. https://status.xchart.com/incident/608510 Tue, 24 Jun 2025 21:39:36 -0000 https://status.xchart.com/incident/608510#d5d38b47bce82bca19bc2db5de58949bb72e15e9e7252f7dfb8b25cca914b9e6 Xchart.com Public Site and Case Manager went down. https://status.xchart.com/incident/608510 Tue, 24 Jun 2025 21:39:36 -0000 https://status.xchart.com/incident/608510#d5d38b47bce82bca19bc2db5de58949bb72e15e9e7252f7dfb8b25cca914b9e6 Xchart.com Public Site and Case Manager went down. https://status.xchart.com/incident/602108 Thu, 12 Jun 2025 21:14:00 -0000 https://status.xchart.com/incident/602108#2881ffc0bfeb1708d78d4f8442f5fc78488ac87a1a7810aa79f1811ab885e08b Today, a major **global outage** affected Google Cloud — the infrastructure provider that powers Xchart and countless other internet services. During this time, many of our users were unable to access the platform or experienced degraded performance. We first detected the issue around **11:08 AM PST** and updated [our status page](https://status.xchart.com) at **11:16 AM**. We started seeing some signs of recovery around **12:19 PM**, but it was not until around **1:43 PM**, access to Xchart was fully restored for most customers. You can find Google’s official incident summary [here](https://status.cloud.google.com/incidents/ow5i3PPK96RduMcb1SsW), and a broader look at the impact across the internet in [this article from CNBC](https://www.cnbc.com/2025/06/12/google-cloud-and-other-internet-services-are-reporting-outages.html). --- ## A Broader Reflection Xchart depends on Google Cloud to deliver the speed, reliability, and scale our customers expect — and most days, that infrastructure is a huge strength. We simply couldn’t build and support this platform the way we do without it. But centralization has its tradeoffs. When a major provider like Google Cloud experiences an issue, the ripple effects are felt far and wide. Today was a clear reminder of that. --- ## Thank You We just want to say thank you for your patience. So many of you were incredibly understanding, kind, and patient as we worked through this with no real ability to “fix” it ourselves. It means a lot. We’re grateful for your trust, and we’ll keep doing everything we can to make Xchart the most reliable anesthesia charting system we know how to build. Stay paperless, **Henrik Joreteg** CEO, Xchart, Inc. https://status.xchart.com/incident/602108 Thu, 12 Jun 2025 21:14:00 -0000 https://status.xchart.com/incident/602108#2881ffc0bfeb1708d78d4f8442f5fc78488ac87a1a7810aa79f1811ab885e08b Today, a major **global outage** affected Google Cloud — the infrastructure provider that powers Xchart and countless other internet services. During this time, many of our users were unable to access the platform or experienced degraded performance. We first detected the issue around **11:08 AM PST** and updated [our status page](https://status.xchart.com) at **11:16 AM**. We started seeing some signs of recovery around **12:19 PM**, but it was not until around **1:43 PM**, access to Xchart was fully restored for most customers. You can find Google’s official incident summary [here](https://status.cloud.google.com/incidents/ow5i3PPK96RduMcb1SsW), and a broader look at the impact across the internet in [this article from CNBC](https://www.cnbc.com/2025/06/12/google-cloud-and-other-internet-services-are-reporting-outages.html). --- ## A Broader Reflection Xchart depends on Google Cloud to deliver the speed, reliability, and scale our customers expect — and most days, that infrastructure is a huge strength. We simply couldn’t build and support this platform the way we do without it. But centralization has its tradeoffs. When a major provider like Google Cloud experiences an issue, the ripple effects are felt far and wide. Today was a clear reminder of that. --- ## Thank You We just want to say thank you for your patience. So many of you were incredibly understanding, kind, and patient as we worked through this with no real ability to “fix” it ourselves. It means a lot. We’re grateful for your trust, and we’ll keep doing everything we can to make Xchart the most reliable anesthesia charting system we know how to build. Stay paperless, **Henrik Joreteg** CEO, Xchart, Inc. https://status.xchart.com/incident/602108 Thu, 12 Jun 2025 21:14:00 -0000 https://status.xchart.com/incident/602108#2881ffc0bfeb1708d78d4f8442f5fc78488ac87a1a7810aa79f1811ab885e08b Today, a major **global outage** affected Google Cloud — the infrastructure provider that powers Xchart and countless other internet services. During this time, many of our users were unable to access the platform or experienced degraded performance. We first detected the issue around **11:08 AM PST** and updated [our status page](https://status.xchart.com) at **11:16 AM**. We started seeing some signs of recovery around **12:19 PM**, but it was not until around **1:43 PM**, access to Xchart was fully restored for most customers. You can find Google’s official incident summary [here](https://status.cloud.google.com/incidents/ow5i3PPK96RduMcb1SsW), and a broader look at the impact across the internet in [this article from CNBC](https://www.cnbc.com/2025/06/12/google-cloud-and-other-internet-services-are-reporting-outages.html). --- ## A Broader Reflection Xchart depends on Google Cloud to deliver the speed, reliability, and scale our customers expect — and most days, that infrastructure is a huge strength. We simply couldn’t build and support this platform the way we do without it. But centralization has its tradeoffs. When a major provider like Google Cloud experiences an issue, the ripple effects are felt far and wide. Today was a clear reminder of that. --- ## Thank You We just want to say thank you for your patience. So many of you were incredibly understanding, kind, and patient as we worked through this with no real ability to “fix” it ourselves. It means a lot. We’re grateful for your trust, and we’ll keep doing everything we can to make Xchart the most reliable anesthesia charting system we know how to build. Stay paperless, **Henrik Joreteg** CEO, Xchart, Inc. https://status.xchart.com/incident/602108 Thu, 12 Jun 2025 21:14:00 -0000 https://status.xchart.com/incident/602108#2881ffc0bfeb1708d78d4f8442f5fc78488ac87a1a7810aa79f1811ab885e08b Today, a major **global outage** affected Google Cloud — the infrastructure provider that powers Xchart and countless other internet services. During this time, many of our users were unable to access the platform or experienced degraded performance. We first detected the issue around **11:08 AM PST** and updated [our status page](https://status.xchart.com) at **11:16 AM**. We started seeing some signs of recovery around **12:19 PM**, but it was not until around **1:43 PM**, access to Xchart was fully restored for most customers. You can find Google’s official incident summary [here](https://status.cloud.google.com/incidents/ow5i3PPK96RduMcb1SsW), and a broader look at the impact across the internet in [this article from CNBC](https://www.cnbc.com/2025/06/12/google-cloud-and-other-internet-services-are-reporting-outages.html). --- ## A Broader Reflection Xchart depends on Google Cloud to deliver the speed, reliability, and scale our customers expect — and most days, that infrastructure is a huge strength. We simply couldn’t build and support this platform the way we do without it. But centralization has its tradeoffs. When a major provider like Google Cloud experiences an issue, the ripple effects are felt far and wide. Today was a clear reminder of that. --- ## Thank You We just want to say thank you for your patience. So many of you were incredibly understanding, kind, and patient as we worked through this with no real ability to “fix” it ourselves. It means a lot. We’re grateful for your trust, and we’ll keep doing everything we can to make Xchart the most reliable anesthesia charting system we know how to build. Stay paperless, **Henrik Joreteg** CEO, Xchart, Inc. https://status.xchart.com/incident/602108 Thu, 12 Jun 2025 21:14:00 -0000 https://status.xchart.com/incident/602108#2881ffc0bfeb1708d78d4f8442f5fc78488ac87a1a7810aa79f1811ab885e08b Today, a major **global outage** affected Google Cloud — the infrastructure provider that powers Xchart and countless other internet services. During this time, many of our users were unable to access the platform or experienced degraded performance. We first detected the issue around **11:08 AM PST** and updated [our status page](https://status.xchart.com) at **11:16 AM**. We started seeing some signs of recovery around **12:19 PM**, but it was not until around **1:43 PM**, access to Xchart was fully restored for most customers. You can find Google’s official incident summary [here](https://status.cloud.google.com/incidents/ow5i3PPK96RduMcb1SsW), and a broader look at the impact across the internet in [this article from CNBC](https://www.cnbc.com/2025/06/12/google-cloud-and-other-internet-services-are-reporting-outages.html). --- ## A Broader Reflection Xchart depends on Google Cloud to deliver the speed, reliability, and scale our customers expect — and most days, that infrastructure is a huge strength. We simply couldn’t build and support this platform the way we do without it. But centralization has its tradeoffs. When a major provider like Google Cloud experiences an issue, the ripple effects are felt far and wide. Today was a clear reminder of that. --- ## Thank You We just want to say thank you for your patience. So many of you were incredibly understanding, kind, and patient as we worked through this with no real ability to “fix” it ourselves. It means a lot. We’re grateful for your trust, and we’ll keep doing everything we can to make Xchart the most reliable anesthesia charting system we know how to build. Stay paperless, **Henrik Joreteg** CEO, Xchart, Inc. https://status.xchart.com/incident/602093 Thu, 12 Jun 2025 20:54:00 -0000 https://status.xchart.com/incident/602093#3d6b181a7d024f3b4d0a809a1279564ea3210acc2b47236ed3165d796b764a64 We're seeing elevated error rates but people are able to log in, etc. We'll be sending out a notification email describing what happened. Global Google Cloud outage. https://status.xchart.com/incident/602093 Thu, 12 Jun 2025 20:54:00 -0000 https://status.xchart.com/incident/602093#3d6b181a7d024f3b4d0a809a1279564ea3210acc2b47236ed3165d796b764a64 We're seeing elevated error rates but people are able to log in, etc. We'll be sending out a notification email describing what happened. Global Google Cloud outage. https://status.xchart.com/incident/602093 Thu, 12 Jun 2025 20:54:00 -0000 https://status.xchart.com/incident/602093#3d6b181a7d024f3b4d0a809a1279564ea3210acc2b47236ed3165d796b764a64 We're seeing elevated error rates but people are able to log in, etc. We'll be sending out a notification email describing what happened. Global Google Cloud outage. https://status.xchart.com/incident/602093 Thu, 12 Jun 2025 20:54:00 -0000 https://status.xchart.com/incident/602093#3d6b181a7d024f3b4d0a809a1279564ea3210acc2b47236ed3165d796b764a64 We're seeing elevated error rates but people are able to log in, etc. We'll be sending out a notification email describing what happened. Global Google Cloud outage. https://status.xchart.com/incident/602093 Thu, 12 Jun 2025 20:54:00 -0000 https://status.xchart.com/incident/602093#3d6b181a7d024f3b4d0a809a1279564ea3210acc2b47236ed3165d796b764a64 We're seeing elevated error rates but people are able to log in, etc. We'll be sending out a notification email describing what happened. Global Google Cloud outage. https://status.xchart.com/incident/602093 Thu, 12 Jun 2025 20:54:00 -0000 https://status.xchart.com/incident/602093#3d6b181a7d024f3b4d0a809a1279564ea3210acc2b47236ed3165d796b764a64 We're seeing elevated error rates but people are able to log in, etc. We'll be sending out a notification email describing what happened. Global Google Cloud outage. https://status.xchart.com/incident/602093 Thu, 12 Jun 2025 20:54:00 -0000 https://status.xchart.com/incident/602093#3d6b181a7d024f3b4d0a809a1279564ea3210acc2b47236ed3165d796b764a64 We're seeing elevated error rates but people are able to log in, etc. We'll be sending out a notification email describing what happened. Global Google Cloud outage. https://status.xchart.com/incident/601777 Thu, 12 Jun 2025 18:16:00 -0000 https://status.xchart.com/incident/601777#d681158f4b025fd64b26c594f591ae98d2134fe2115d41d0b82e97a880a328b7 Affecting lots of sites not just Xchart. https://status.xchart.com/incident/601777 Thu, 12 Jun 2025 18:16:00 -0000 https://status.xchart.com/incident/601777#d681158f4b025fd64b26c594f591ae98d2134fe2115d41d0b82e97a880a328b7 Affecting lots of sites not just Xchart. https://status.xchart.com/incident/601777 Thu, 12 Jun 2025 18:16:00 -0000 https://status.xchart.com/incident/601777#d681158f4b025fd64b26c594f591ae98d2134fe2115d41d0b82e97a880a328b7 Affecting lots of sites not just Xchart. https://status.xchart.com/incident/601777 Thu, 12 Jun 2025 18:16:00 -0000 https://status.xchart.com/incident/601777#d681158f4b025fd64b26c594f591ae98d2134fe2115d41d0b82e97a880a328b7 Affecting lots of sites not just Xchart. https://status.xchart.com/incident/601777 Thu, 12 Jun 2025 18:16:00 -0000 https://status.xchart.com/incident/601777#d681158f4b025fd64b26c594f591ae98d2134fe2115d41d0b82e97a880a328b7 Affecting lots of sites not just Xchart. https://status.xchart.com/incident/601647 Thu, 12 Jun 2025 17:57:00 -0000 https://status.xchart.com/incident/601647#f06a74648cbd17d3a60136bbbf8320b5d6e9bcc7c31e605727b9c4f4dcf30488 Xchart Internal API went down. https://status.xchart.com/incident/600262 Tue, 10 Jun 2025 17:30:31 -0000 https://status.xchart.com/incident/600262#6b08970db4f1bc892605f54ec1c15a749ba69b1e9d469ed976de4c22c4b21c5d Xchart Internal API recovered. https://status.xchart.com/incident/600262 Tue, 10 Jun 2025 17:25:47 -0000 https://status.xchart.com/incident/600262#8d8580ddf253baec3229ea741261333a9432522e0588a6eac432c4f8345a7409 Xchart Internal API went down. https://status.xchart.com/incident/558079 Wed, 07 May 2025 12:14:12 -0000 https://status.xchart.com/incident/558079#2e07486affea46edda4608f82652451439a424348554a42c9edfdfbd2c1f19fd Xchart Enterprise Dashboard recovered. https://status.xchart.com/incident/558079 Wed, 07 May 2025 12:10:51 -0000 https://status.xchart.com/incident/558079#c351facbf3196b98c2b68a4f1836add8deb1ad59d1fc1c37b8fd560b3b78e9d1 Xchart Enterprise Dashboard went down. https://status.xchart.com/incident/552947 Mon, 28 Apr 2025 21:48:55 -0000 https://status.xchart.com/incident/552947#e791d8fb48aa6e5e8e77a10bfc8d5b2c9e7db3b8a17a6ed7010a878fdd7eff91 Xchart Internal API recovered. https://status.xchart.com/incident/552947 Mon, 28 Apr 2025 21:45:36 -0000 https://status.xchart.com/incident/552947#dddbe36cf11ff3fc0477a126aa79e1a621a8ddfb19e871156515112e5f9e458d Xchart Internal API went down. https://status.xchart.com/incident/552947 Mon, 28 Apr 2025 18:40:53 -0000 https://status.xchart.com/incident/552947#320d5769aa153aa80f5f3bfaf1faa66ad972a9c438d2007192ac5c933423f6b7 Xchart Internal API recovered. https://status.xchart.com/incident/552947 Mon, 28 Apr 2025 18:36:13 -0000 https://status.xchart.com/incident/552947#e8ee581356d51515766eb35bcce3428ea4647cc834525da9e0f69f38a13e6d95 Xchart Internal API went down. https://status.xchart.com/incident/552947 Mon, 28 Apr 2025 08:15:42 -0000 https://status.xchart.com/incident/552947#ea0848605cfb290a893c457f115828be2af2af1a5cd3e5270e2f19701b828fa4 Xchart Legal Site recovered. https://status.xchart.com/incident/552947 Mon, 28 Apr 2025 08:12:24 -0000 https://status.xchart.com/incident/552947#03ef4a62b77603cac74d9277a3dda79b8c6e711252df6cd987bd0f92a62ce844 Xchart Legal Site went down. https://status.xchart.com/incident/552947 Mon, 28 Apr 2025 08:11:28 -0000 https://status.xchart.com/incident/552947#a9f7104e1089c93a8b4c1c8a48dd1e6c901d1b72b18bd3e9afa6533cbfcc322a Xchart Legal Site recovered. https://status.xchart.com/incident/552947 Mon, 28 Apr 2025 08:06:22 -0000 https://status.xchart.com/incident/552947#bdd32738d77ee71f7da2cba78f26a2c5029992df697acc4c5fc05332ba4ca87c Xchart Legal Site went down. https://status.xchart.com/incident/546636 Thu, 17 Apr 2025 05:09:27 -0000 https://status.xchart.com/incident/546636#041e3f10451c691cd9a5dcfcd49e23725ff219856ac0b5d63ab2eee4132dd5d2 Xchart Internal API recovered. https://status.xchart.com/incident/546636 Thu, 17 Apr 2025 05:06:14 -0000 https://status.xchart.com/incident/546636#47d3ddb000b99d568394136c70ed5e350db30467f35910ebb9d610343485f473 Xchart Internal API went down. https://status.xchart.com/incident/546636 Thu, 17 Apr 2025 03:49:31 -0000 https://status.xchart.com/incident/546636#3df93b75fabf0661eb75d66c43996d8b4f666c81ae12ed3c91b8f7bc8c8e4c6f Xchart Internal API recovered. https://status.xchart.com/incident/546636 Thu, 17 Apr 2025 03:45:39 -0000 https://status.xchart.com/incident/546636#6c961c0defa075dc586ccdf7289ee0506b193a524426cc73f8dfae0bda40552d Xchart Internal API went down. https://status.xchart.com/incident/542910 Thu, 10 Apr 2025 04:18:48 -0000 https://status.xchart.com/incident/542910#a986ebfdb0af389a547dabdc0f97a79b1092e746c35126e408df67fd04185040 Xchart Internal API recovered. https://status.xchart.com/incident/542910 Thu, 10 Apr 2025 04:15:13 -0000 https://status.xchart.com/incident/542910#15a5bcb49ad3254c29501d3809986734270bebe2d7304042c4d0433cd96172e8 Xchart Internal API went down. https://status.xchart.com/incident/451770 Mon, 28 Oct 2024 08:41:24 -0000 https://status.xchart.com/incident/451770#a0049d98a570f1407bb6d175ee1f855a12a841e5184f3c58a15cb4a0ae867b2f Case Editor recovered. https://status.xchart.com/incident/451770 Mon, 28 Oct 2024 08:35:46 -0000 https://status.xchart.com/incident/451770#7aa0cbffe72477ff091c4bc111736ceed278224b6cd7d8cd481a240a6be4513e Case Editor went down. https://status.xchart.com/incident/432354 Fri, 20 Sep 2024 17:40:39 -0000 https://status.xchart.com/incident/432354#6728318b90836a62a8999f5dd3cfa00ab3b2bf0590059b107aadfe5e7a48732c Xchart Internal API recovered. https://status.xchart.com/incident/432354 Fri, 20 Sep 2024 17:37:03 -0000 https://status.xchart.com/incident/432354#8a2273fc681b193c88ef4db5954f1cc3f6fd0575d4bb9884bce615f2a1a6c082 Xchart Internal API went down. https://status.xchart.com/incident/432354 Fri, 20 Sep 2024 17:32:33 -0000 https://status.xchart.com/incident/432354#6fe8d34ca3bc0fc5f7d5e416a191c9213e158372ed133aefd85850099760f776 Xchart Internal API recovered. https://status.xchart.com/incident/432354 Fri, 20 Sep 2024 17:29:08 -0000 https://status.xchart.com/incident/432354#98e58af57da99dfe966147a8a5a09775aef4c6b93c101defb5738a062422f976 Xchart Internal API went down. https://status.xchart.com/incident/432354 Fri, 20 Sep 2024 17:00:39 -0000 https://status.xchart.com/incident/432354#79425345c47aeb3e6ce180465813144a8ec0226d90dd6e129b6f7732b77b5880 Xchart Internal API recovered. https://status.xchart.com/incident/432354 Fri, 20 Sep 2024 16:45:02 -0000 https://status.xchart.com/incident/432354#76148e7df12456190f38d44d6bc400d53fff1818fc5cbcf6b420ceae37be711c Xchart Internal API went down. https://status.xchart.com/incident/432345 Fri, 20 Sep 2024 16:15:00 -0000 https://status.xchart.com/incident/432345#e879ac8565a305941784039b415ecdeefc595f79f74179eac5247a1ce050e539 We're having an issue related to google cloud. We believe this issue is now resolved. https://status.xchart.com/incident/432345 Fri, 20 Sep 2024 16:15:00 -0000 https://status.xchart.com/incident/432345#e879ac8565a305941784039b415ecdeefc595f79f74179eac5247a1ce050e539 We're having an issue related to google cloud. We believe this issue is now resolved. https://status.xchart.com/incident/432345 Fri, 20 Sep 2024 16:15:00 -0000 https://status.xchart.com/incident/432345#e879ac8565a305941784039b415ecdeefc595f79f74179eac5247a1ce050e539 We're having an issue related to google cloud. We believe this issue is now resolved. https://status.xchart.com/incident/432330 Fri, 20 Sep 2024 15:39:00 -0000 https://status.xchart.com/incident/432330#f660b38078f0cdc769ba3b1ce3d016a7192688d0e1678f6b11aa1bca7df8b16e Xchart Internal API went down. https://status.xchart.com/incident/423756 Tue, 03 Sep 2024 22:46:51 -0000 https://status.xchart.com/incident/423756#79cc17ffaa6bd1c3a3f175be5f27841fe0b2d0c166082ad3baf768258a477a68 Xchart Internal API recovered. https://status.xchart.com/incident/423756 Tue, 03 Sep 2024 22:39:23 -0000 https://status.xchart.com/incident/423756#d1b861484c7c891e9c083d4008606ba77e29e38d13314b54fdf6001299867c13 Xchart Internal API went down. https://status.xchart.com/incident/421514 Fri, 30 Aug 2024 02:05:49 -0000 https://status.xchart.com/incident/421514#e512971fe7aa4b20d666865c6fdf0f6b1ee93e191311641ffb02c3532dadba72 Xchart Internal API recovered. https://status.xchart.com/incident/421514 Fri, 30 Aug 2024 02:02:38 -0000 https://status.xchart.com/incident/421514#2108d558b360cb06d78c259971c090c41cf0924f4f0b2d36acd6f0827e0e7e89 Xchart Internal API went down. https://status.xchart.com/incident/421514 Fri, 30 Aug 2024 01:06:48 -0000 https://status.xchart.com/incident/421514#8a328b3e294854b5d4a961dfdd079a935dfaebe7150a2483174fd11583c0b081 Xchart Internal API recovered. https://status.xchart.com/incident/421514 Fri, 30 Aug 2024 01:03:34 -0000 https://status.xchart.com/incident/421514#bb9ae7bfabd4e48a41079c1dc735ca464d3612b80f704718014980e2e6aa0712 Xchart Internal API went down. https://status.xchart.com/incident/416819 Wed, 21 Aug 2024 03:02:23 -0000 https://status.xchart.com/incident/416819#4c71271ba8e75061433258e510a6ee3d1ae3258b53bf2c5e10fa4b17069022d6 Xchart Internal API recovered. https://status.xchart.com/incident/416819 Wed, 21 Aug 2024 02:59:09 -0000 https://status.xchart.com/incident/416819#e6e82b56d82546f4209697dbf4c3a0aa41567c5c8e09223207a7f9cfa948891c Xchart Internal API went down. https://status.xchart.com/incident/411474 Fri, 09 Aug 2024 17:12:15 -0000 https://status.xchart.com/incident/411474#bea116ade1d184d27241b25c3329b0100e37c296cdf96c16509b94134a76c216 Case Manager and Xchart Legal Site recovered. https://status.xchart.com/incident/411474 Fri, 09 Aug 2024 17:12:15 -0000 https://status.xchart.com/incident/411474#bea116ade1d184d27241b25c3329b0100e37c296cdf96c16509b94134a76c216 Case Manager and Xchart Legal Site recovered. https://status.xchart.com/incident/411474 Fri, 09 Aug 2024 17:09:22 -0000 https://status.xchart.com/incident/411474#94ed1889c919e86f08a4faf9e6977baf67be6a00a45e98383926e87fb9d8cdbd Xchart.com Public Site, Case Editor, and Xchart Patient Forms recovered. https://status.xchart.com/incident/411474 Fri, 09 Aug 2024 17:09:22 -0000 https://status.xchart.com/incident/411474#94ed1889c919e86f08a4faf9e6977baf67be6a00a45e98383926e87fb9d8cdbd Xchart.com Public Site, Case Editor, and Xchart Patient Forms recovered. https://status.xchart.com/incident/411474 Fri, 09 Aug 2024 17:09:22 -0000 https://status.xchart.com/incident/411474#94ed1889c919e86f08a4faf9e6977baf67be6a00a45e98383926e87fb9d8cdbd Xchart.com Public Site, Case Editor, and Xchart Patient Forms recovered. https://status.xchart.com/incident/411474 Fri, 09 Aug 2024 17:03:31 -0000 https://status.xchart.com/incident/411474#1ae999b2afea8a1a5e122d5a5def0c314df96167746edc3d54b8a06c9a16a82a Case Editor and Case Manager went down. https://status.xchart.com/incident/411474 Fri, 09 Aug 2024 17:03:31 -0000 https://status.xchart.com/incident/411474#1ae999b2afea8a1a5e122d5a5def0c314df96167746edc3d54b8a06c9a16a82a Case Editor and Case Manager went down. https://status.xchart.com/incident/411474 Fri, 09 Aug 2024 17:02:41 -0000 https://status.xchart.com/incident/411474#769a8508cefb675d791d8b2202ff2fbdb9a6b7bbf8967a38bbdf17486c5d591d Xchart.com Public Site, Xchart Patient Forms, and Xchart Legal Site went down. https://status.xchart.com/incident/411474 Fri, 09 Aug 2024 17:02:41 -0000 https://status.xchart.com/incident/411474#769a8508cefb675d791d8b2202ff2fbdb9a6b7bbf8967a38bbdf17486c5d591d Xchart.com Public Site, Xchart Patient Forms, and Xchart Legal Site went down. https://status.xchart.com/incident/411474 Fri, 09 Aug 2024 17:02:41 -0000 https://status.xchart.com/incident/411474#769a8508cefb675d791d8b2202ff2fbdb9a6b7bbf8967a38bbdf17486c5d591d Xchart.com Public Site, Xchart Patient Forms, and Xchart Legal Site went down. https://status.xchart.com/incident/372323 Tue, 21 May 2024 01:15:42 -0000 https://status.xchart.com/incident/372323#5fe273ed461c636af149e5b6cede4c03bc0066262233c833d9fe219bd94bdba7 Xchart Internal API recovered. https://status.xchart.com/incident/372323 Tue, 21 May 2024 01:12:18 -0000 https://status.xchart.com/incident/372323#3a8e3dce8f37b864aa610b78918d686f1f82787b1fcc40bfd11c0049e488e10a Xchart Internal API went down. https://status.xchart.com/incident/372106 Mon, 20 May 2024 14:34:16 -0000 https://status.xchart.com/incident/372106#1e17527d3ea8ab38d18c500891eed61e8ae2ade51c55a53b1ca01b6fd2ca465a Case Editor recovered. https://status.xchart.com/incident/372106 Mon, 20 May 2024 14:25:17 -0000 https://status.xchart.com/incident/372106#9601fb35c94d963bb28353bdbf1e5d7e0b0a25bb73e468667b1b88d459dd47c8 Case Editor went down. https://status.xchart.com/incident/355005 Sat, 13 Apr 2024 19:06:29 -0000 https://status.xchart.com/incident/355005#5e72e7239e0c7ee0e7af6a2d8d4558e9760ec3bcd2a998cf375100758f152d1e Xchart Internal API recovered. https://status.xchart.com/incident/355005 Sat, 13 Apr 2024 19:03:12 -0000 https://status.xchart.com/incident/355005#5bbb076182f886cc5624e895faabbcfe02584c77caf761fcbfecb2444915e812 Xchart Internal API went down. https://status.xchart.com/incident/332614 Tue, 27 Feb 2024 11:22:59 -0000 https://status.xchart.com/incident/332614#c31820880ea8a63dbdd5f81ee8e1653c132a92065b200e0897b6afe53db9e88d Xchart Internal API recovered. https://status.xchart.com/incident/332614 Tue, 27 Feb 2024 11:19:41 -0000 https://status.xchart.com/incident/332614#e81941b978ab498dd44d650e55592a1bde0d7a8771f4aaecf3ef4eb06ca5a235 Xchart Internal API went down. https://status.xchart.com/incident/328596 Sun, 18 Feb 2024 16:15:08 -0000 https://status.xchart.com/incident/328596#8353a8a429b37b76af9fb392f3f2eea0255034de3a7138c0c8e64c603f24c366 Xchart Internal API recovered. https://status.xchart.com/incident/328596 Sun, 18 Feb 2024 16:09:57 -0000 https://status.xchart.com/incident/328596#34f0988f7a9848f95f0b633ac3f28f8e3150c3d7fb31cc5f8f1299aa46f0fe80 Xchart Internal API went down. https://status.xchart.com/incident/315449 Sat, 20 Jan 2024 14:04:24 -0000 https://status.xchart.com/incident/315449#bf52efbcf814e91ec0922fec947f9a73b96f331841c8c7a85e90eea0e572fbcf Xchart Internal API recovered. https://status.xchart.com/incident/315449 Sat, 20 Jan 2024 14:01:07 -0000 https://status.xchart.com/incident/315449#bef82a0effe7736010f6c6a1007f9ecbd3c1258c7b8a7f98539600f0dacdf9db Xchart Internal API went down. https://status.xchart.com/incident/292429 Sun, 26 Nov 2023 17:04:48 -0000 https://status.xchart.com/incident/292429#c58ab836b786e7c36197166a45369fa2e8721e0595c2288e8ffbb6111d39535c Xchart Internal API recovered. https://status.xchart.com/incident/292429 Sun, 26 Nov 2023 16:56:29 -0000 https://status.xchart.com/incident/292429#60cbc836c2f9ac0500dfbf889c1318342c484f1eaf77dde1cf25e07f2eb5f3f1 Xchart Internal API went down. https://status.xchart.com/incident/276384 Mon, 23 Oct 2023 07:30:29 -0000 https://status.xchart.com/incident/276384#b2f8cea630a29eae3d49875435ab5d0bdd8798c7691474e106efe572d725863f Xchart Internal API recovered. https://status.xchart.com/incident/276384 Mon, 23 Oct 2023 07:09:53 -0000 https://status.xchart.com/incident/276384#afc365c9fd8856b11c25e68849ff899be485e764688f5a405dc68f94916c82bc Xchart Internal API went down. https://status.xchart.com/incident/275701 Sat, 21 Oct 2023 07:20:38 -0000 https://status.xchart.com/incident/275701#78c0d8dec776ee50234571f949f259569cf64c29fa2aa29c5bf5b6f0a3264a99 Xchart Internal API recovered. https://status.xchart.com/incident/275701 Sat, 21 Oct 2023 07:09:55 -0000 https://status.xchart.com/incident/275701#2ee7eda5449d1842a1cb47d03bffa4381fa2607e1e80bb51962993afc0ee7455 Xchart Internal API went down. https://status.xchart.com/incident/267829 Tue, 03 Oct 2023 20:44:00 -0000 https://status.xchart.com/incident/267829#d121ef88189e83caf099f33c7f063813c5ca6dbd8487682070f9f8208e9fcef9 Hello everyone. Unfortunately we had a period of about 4 minutes today during which our main database was unreachable. It occured between 12:19-12:23 PST. Even though the database was unreachable for only a short time, the impact was significant, especially if you happened to be in the middle of a case. ## What was the impact? If you were _not_ signed in during the incident, there is no known impact to your account or your data. If you _were_ signed in, you would have likely found yourself suddenly redirected to the sign in page and during that period, you would not have been able to immediately sign back in again. Additionally, even though the database was only "off" for a short period, it does take these systems a bit of time to spin up again. So it could have been a bit more than 4 minutes during which you would not have been able to log in. This is not the type of experience we want our customers to have with our product. We're sorry for the disruption. ## How did it happen? Even though each organization's clinical database is split into their own data stores, there is a shared database for handling user accounts, etc. That shared database is what became unreachable. - We were shutting down an old, unused cloud server. - We didn't know that Google Cloud used to link certain types of servers to certain database services, and we didn't know that our setup fit that criteria. - When we disabled the old/unused server, unbeknownst to us, Google Cloud also automatically disabled the entire database. ## What did we do to fix it? As soon as we realized something was wrong after making the change, we immediately reenabled the service that we just shut down. This restored functionality. The problem was, that anyone who had been signed in, would have been automatically signed out the next time your session was renewed (which is every 3 minutes). ## Why did we want to shut it down? And why do it during the day while there are surgeries going on? We shut it down because it was a housekeeping item raised during our recent security audit. We shut it down, even in the middle of the work day like this, because it was unused and we were not aware there was this hidden link. ## What could we have done to prevent this? The fact that disabling this old service would have any impact on our database was hard to predict. But that doesn't change the fact that it is our responsibility to keep things running as best as we can. We will take steps to carefully unlink these systems before we attempt to remove the old service again, and when we do, it will be during off-hours. I take full responsibility for the incident. My personal apologies to those who were affected. If you were negatively affected or you'd like to discuss this further you can email me directly: henrik@xchart.com and I'll do my best to help however I can. Henrik Joreteg CEO Xchart, Inc. https://status.xchart.com/incident/267829 Tue, 03 Oct 2023 20:44:00 -0000 https://status.xchart.com/incident/267829#d121ef88189e83caf099f33c7f063813c5ca6dbd8487682070f9f8208e9fcef9 Hello everyone. Unfortunately we had a period of about 4 minutes today during which our main database was unreachable. It occured between 12:19-12:23 PST. Even though the database was unreachable for only a short time, the impact was significant, especially if you happened to be in the middle of a case. ## What was the impact? If you were _not_ signed in during the incident, there is no known impact to your account or your data. If you _were_ signed in, you would have likely found yourself suddenly redirected to the sign in page and during that period, you would not have been able to immediately sign back in again. Additionally, even though the database was only "off" for a short period, it does take these systems a bit of time to spin up again. So it could have been a bit more than 4 minutes during which you would not have been able to log in. This is not the type of experience we want our customers to have with our product. We're sorry for the disruption. ## How did it happen? Even though each organization's clinical database is split into their own data stores, there is a shared database for handling user accounts, etc. That shared database is what became unreachable. - We were shutting down an old, unused cloud server. - We didn't know that Google Cloud used to link certain types of servers to certain database services, and we didn't know that our setup fit that criteria. - When we disabled the old/unused server, unbeknownst to us, Google Cloud also automatically disabled the entire database. ## What did we do to fix it? As soon as we realized something was wrong after making the change, we immediately reenabled the service that we just shut down. This restored functionality. The problem was, that anyone who had been signed in, would have been automatically signed out the next time your session was renewed (which is every 3 minutes). ## Why did we want to shut it down? And why do it during the day while there are surgeries going on? We shut it down because it was a housekeeping item raised during our recent security audit. We shut it down, even in the middle of the work day like this, because it was unused and we were not aware there was this hidden link. ## What could we have done to prevent this? The fact that disabling this old service would have any impact on our database was hard to predict. But that doesn't change the fact that it is our responsibility to keep things running as best as we can. We will take steps to carefully unlink these systems before we attempt to remove the old service again, and when we do, it will be during off-hours. I take full responsibility for the incident. My personal apologies to those who were affected. If you were negatively affected or you'd like to discuss this further you can email me directly: henrik@xchart.com and I'll do my best to help however I can. Henrik Joreteg CEO Xchart, Inc. https://status.xchart.com/incident/267829 Tue, 03 Oct 2023 20:44:00 -0000 https://status.xchart.com/incident/267829#d121ef88189e83caf099f33c7f063813c5ca6dbd8487682070f9f8208e9fcef9 Hello everyone. Unfortunately we had a period of about 4 minutes today during which our main database was unreachable. It occured between 12:19-12:23 PST. Even though the database was unreachable for only a short time, the impact was significant, especially if you happened to be in the middle of a case. ## What was the impact? If you were _not_ signed in during the incident, there is no known impact to your account or your data. If you _were_ signed in, you would have likely found yourself suddenly redirected to the sign in page and during that period, you would not have been able to immediately sign back in again. Additionally, even though the database was only "off" for a short period, it does take these systems a bit of time to spin up again. So it could have been a bit more than 4 minutes during which you would not have been able to log in. This is not the type of experience we want our customers to have with our product. We're sorry for the disruption. ## How did it happen? Even though each organization's clinical database is split into their own data stores, there is a shared database for handling user accounts, etc. That shared database is what became unreachable. - We were shutting down an old, unused cloud server. - We didn't know that Google Cloud used to link certain types of servers to certain database services, and we didn't know that our setup fit that criteria. - When we disabled the old/unused server, unbeknownst to us, Google Cloud also automatically disabled the entire database. ## What did we do to fix it? As soon as we realized something was wrong after making the change, we immediately reenabled the service that we just shut down. This restored functionality. The problem was, that anyone who had been signed in, would have been automatically signed out the next time your session was renewed (which is every 3 minutes). ## Why did we want to shut it down? And why do it during the day while there are surgeries going on? We shut it down because it was a housekeeping item raised during our recent security audit. We shut it down, even in the middle of the work day like this, because it was unused and we were not aware there was this hidden link. ## What could we have done to prevent this? The fact that disabling this old service would have any impact on our database was hard to predict. But that doesn't change the fact that it is our responsibility to keep things running as best as we can. We will take steps to carefully unlink these systems before we attempt to remove the old service again, and when we do, it will be during off-hours. I take full responsibility for the incident. My personal apologies to those who were affected. If you were negatively affected or you'd like to discuss this further you can email me directly: henrik@xchart.com and I'll do my best to help however I can. Henrik Joreteg CEO Xchart, Inc. https://status.xchart.com/incident/244670 Fri, 11 Aug 2023 21:15:00 -0000 https://status.xchart.com/incident/244670#851160936febbe23d94e645d5b6d3e93597755118da7d786c6fcfbdbfc50a30e Hey everyone, as some of you noticed, Xchart experienced some downtime today. You can see it as a red blip here on [our official status page](https://status.xchart.com/). Access to our API was unavailable to some of our customers for up to an hour. It took a total of 1 hour and 7 minutes to fully resolve according to our status monitoring. Today's outage is the single longest outage we've experienced to date. For this, we are truly sorry. The last time a core part of the service was unavailable in the last 90 days was a 5 minute stretch where servers in Asia, Europe, and Australia were unable to reach our API. I wanted to take the time to write up an explanation of what happened. Because frankly, it's not what we want our customers to experience. ## What was the problem? **It was caused by a missing DNS record.** The DNS system is how a browser looks up a domain name like: xchart.com and determines what server to ask for the data from. So the domain for our internal data api, wasn't resolving because there wasn't a record there to be found. If you were in the affected group, your computer wasn't getting anything back at all from our data API because it couldn't reach it, at all. Our servers were running just fine, but the request from your computer just wouldn't even have been sent to our servers. ## Why was the DNS record missing? This is the part that gets frustrating to us. We made, what should have been an inconsequential change. We were trying to edit the DNS record for our internal API to have a shorter cache time. This was a small pre-requisite step in preparation for the actual change which was planned to happen over the weekend. This seemed like a small enough change that we felt comfortable doing it during normal business hours. Plus, it's something you want to do before the actual change so that old caches with the old time have a chance to clear out. The web interface we're using to make these changes doesn't let us directly change a record in place. Instead, you have to create a new one, and delete the old one. We didn't want to risk there ever not being one there, so we created the new duplicate one first, then subsequently deleted the old one. We made the change, everything seemed to work fine, no big deal, no problem, it seemed to have worked exactly as expected. But then our monitoring tools started sending alerts that the system was unreachable. We immediately knew something bad had happened because this should've been a totally harmless change and according to the screens we were looking at a record had remained in place the entire time. We tried undoing the change and replacing the old record and we tried with various tools to see if the record was properly in place, but several of them kept saying it was still not there. At this point, we desperately tried to contact support at this vendor. Thankfully they got back to us reasonable fast with this simple response: > "Hi, This should now be resolved. Please re-try." We pointed out the issue still was intermittently happening and the explanation was: > "It should eventually stabilise. Since the change has been made from our end just now, it could take some time." and when we asked what the cause was, we got this: > "The DNS record was saved in our database, but not on our DNS provider. I added it there." It turns out, that the web interface we were using to make this change was lying to us. It was showing that the change had been successful when in reality it wasn't. **Wow.** Once it was actually in place, we spent the next half hour just trying to confirm from various sources that things were in fact working, and notifying the customers who had reached out to us, to "please try it again." ## What we learned, and what we're going change **What worked well:** - Having a status page that auto-updates based on its current tests against our production systems. - Our monitoring alerted us of the issue long before any customer did. **What failed:** - The DNS service provider we use showed us incorrect information. The record was missing, but it said it wass in place. - Despite the seemingly innocuous nature of the change, we likely should not have been makin DNS changes during normal operating hours. **What we'll do different:** - DNS changes, no matter how innocuous should be reserved for weekends or at least off hours. - We will switch to a differen DNS service provider. Most likely we'll move this to Google Cloud as well. That's where the majority of our infrastructure is running already and we have been extremely impressed with their overall reliability with their other services we use. ## A Personal Apology As CEO the buck stops with me. I'm sorry for what happened today. If you're going to blame anyone, blame me. We're grateful for your trust and continued business. We will continue to strive to build the best possible anesthesia charting system we know how. Stay Paperless, Henrik Joreteg, CEO Xchart, Inc. https://status.xchart.com/incident/244670 Fri, 11 Aug 2023 19:00:18 -0000 https://status.xchart.com/incident/244670#aca30e137ea1c2eb5e013657a42f1755e136088e3d936ed044b0bb9de4aafdc9 Xchart Internal API recovered. https://status.xchart.com/incident/244670 Fri, 11 Aug 2023 17:50:17 -0000 https://status.xchart.com/incident/244670#68399fb678254f86fe87e39a4c2f16e315c64cab5ff03d9c6753090fec703f7e Xchart Internal API went down. https://status.xchart.com/incident/215470 Sat, 03 Jun 2023 06:21:32 -0000 https://status.xchart.com/incident/215470#069b7f4142affcaf830a3ac3371f3d924db85881c578fedfd87c8105b7743cf5 Xchart Internal API recovered. https://status.xchart.com/incident/215470 Sat, 03 Jun 2023 06:17:51 -0000 https://status.xchart.com/incident/215470#18f00eddf2e6e819c65269d46112c8fadfda9a785b8a799569e88155bddcf741 Xchart Internal API went down. https://status.xchart.com/incident/203958 Fri, 05 May 2023 18:39:31 -0000 https://status.xchart.com/incident/203958#3192b71f775a1a3cbbddedc3a27b54d09ac9abd5808cc99894747b2db92fab6a Xchart Internal API recovered. https://status.xchart.com/incident/203958 Fri, 05 May 2023 18:35:16 -0000 https://status.xchart.com/incident/203958#8fb4eba7706165a7e64c05afc9d1cf6561394d609b230ad05c1448cb637e7a93 Xchart Internal API went down.