All systems are operational

Past Incidents

Monday 12th March 2018

No incidents reported

Sunday 11th March 2018

(Sunday 11th March 2018 23:36:49 UTC)
API Post-mortem: High latency

Over the past few days, we have been experiencing several short periods of high latency during excessive bursts of traffic. We would like to apologize for any negative effect this has had on the immediate delivery of notifications to your users.

After attempting several remediation strategies, we have been able to find a solution that appears to be working quite well, by utilizing a Redis caching mechanism to drastically reduce load on the database cluster. We will be closely monitoring latency over the next few days to confirm that the issue is indeed resolved.

Saturday 10th March 2018

(Saturday 10th March 2018 10:56:33 UTC)
Notification Service Notification Service health has stabilized

The Notification Service has returned to normal operational capacity.

(Saturday 10th March 2018 10:53:21 UTC)
API API health has stabilized

The API has returned to normal operational capacity.

(Saturday 10th March 2018 10:51:47 UTC)
Website Website health has stabilized

The Website has returned to normal operational capacity.

(Saturday 10th March 2018 10:36:20 UTC)
Notification Service Notification Service high latency

We're investigating an issue that is causing high latency in the Notification Service.

(Saturday 10th March 2018 10:22:39 UTC)
Website Website high latency

We're investigating an issue that is causing high latency in the Website.

(Saturday 10th March 2018 10:01:37 UTC)
API API high latency

We're investigating an issue that is causing high latency in the API.

(Saturday 10th March 2018 09:35:38 UTC)
Notification Service Notification Service health has stabilized

The Notification Service has returned to normal operational capacity.

(Saturday 10th March 2018 09:27:36 UTC)
API API health has stabilized

The API has returned to normal operational capacity.

(Saturday 10th March 2018 09:27:02 UTC)
Website Website health has stabilized

The Website has returned to normal operational capacity.

(Saturday 10th March 2018 09:10:25 UTC)
Notification Service Notification Service high latency

We're investigating an issue that is causing high latency in the Notification Service.

(Saturday 10th March 2018 08:56:48 UTC)
Website Website high latency

We're investigating an issue that is causing high latency in the Website.

(Saturday 10th March 2018 08:50:27 UTC)
API API high latency

We're investigating an issue that is causing high latency in the API.

(Saturday 10th March 2018 07:03:11 UTC)
API API health has stabilized

The API has returned to normal operational capacity.

(Saturday 10th March 2018 06:46:22 UTC)
Website Website health has stabilized

The Website has returned to normal operational capacity.

(Saturday 10th March 2018 06:20:11 UTC)
Website Website high latency

We're investigating an issue that is causing high latency in the Website.

(Saturday 10th March 2018 06:14:14 UTC)
API API high latency

We're investigating an issue that is causing high latency in the API.

(Saturday 10th March 2018 05:10:44 UTC)
API API health has stabilized

The API has returned to normal operational capacity.

(Saturday 10th March 2018 05:08:01 UTC)
Website Website health has stabilized

The Website has returned to normal operational capacity.

(Saturday 10th March 2018 05:05:13 UTC)
Notification Service Notification Service health has stabilized

The Notification Service has returned to normal operational capacity.

(Saturday 10th March 2018 04:35:55 UTC)
Notification Service Notification Service high latency

We're investigating an issue that is causing high latency in the Notification Service.

(Saturday 10th March 2018 04:33:09 UTC)
Website Website high latency

We're investigating an issue that is causing high latency in the Website.

(Saturday 10th March 2018 04:27:15 UTC)
API API high latency

We're investigating an issue that is causing high latency in the API.

Friday 9th March 2018

(Friday 9th March 2018 15:44:00 UTC)
API High latency root cause under investigation

Our service experienced high latency for short periods of up to 20 minutes in the past 24 hours. Investigation is still underway and we are attempting various solutions to prevent high latency from reoccurring. No API requests were dropped during these periods of high latency.

(Friday 9th March 2018 15:30:57 UTC)
API API health has stabilized

The API has returned to normal operational capacity.

(Friday 9th March 2018 15:29:15 UTC)
Website Website health has stabilized

The Website has returned to normal operational capacity.

(Friday 9th March 2018 15:10:19 UTC)
Notification Service Notification Service health has stabilized

The Notification Service has returned to normal operational capacity.

(Friday 9th March 2018 14:48:11 UTC)
Notification Service Notification Service high latency

We're investigating an issue that is causing high latency in the Notification Service.

(Friday 9th March 2018 14:39:10 UTC)
Website Website high latency

We're investigating an issue that is causing high latency in the Website.

(Friday 9th March 2018 14:36:19 UTC)
API API high latency

We're investigating an issue that is causing high latency in the API.

(Friday 9th March 2018 04:37:31 UTC)
API API health has stabilized

The API has returned to normal operational capacity.

(Friday 9th March 2018 04:30:03 UTC)
Notification Service Notification Service health has stabilized

The Notification Service has returned to normal operational capacity.

(Friday 9th March 2018 02:48:08 UTC)
Notification Service Notification Service high latency

We're investigating an issue that is causing high latency in the Notification Service.

(Friday 9th March 2018 02:47:25 UTC)
API API high latency

We're investigating an issue that is causing high latency in the API.

Thursday 8th March 2018

(Thursday 8th March 2018 18:15:30 UTC)
Website Website health has stabilized

The Website has returned to normal operational capacity.

(Thursday 8th March 2018 18:14:15 UTC)
API API health has stabilized

The API has returned to normal operational capacity.

(Thursday 8th March 2018 17:55:23 UTC)
Website Website high latency

We're investigating an issue that is causing high latency in the Website.

(Thursday 8th March 2018 17:27:22 UTC)
API API high latency

We're investigating an issue that is causing high latency in the API.