All systems are operational

Past Incidents

Saturday 9th June 2018

No incidents reported

Friday 8th June 2018

(Friday 8th June 2018 05:34:45 UTC)
API Post-mortem: High API latency

High latency has affected the API for seven short bursts of 2 - 3 minutes each from 2:00AM UTC to 4:00AM UTC. No requests were dropped during this time. We have managed to stabilize the service. Further incidents will be posted if there are any future developments. We apologize for any inconvenience caused.

(Friday 8th June 2018 05:06:39 UTC)
API API health has stabilized

The API has returned to normal operational capacity.

(Friday 8th June 2018 03:04:15 UTC)
API API high latency

We're investigating an issue that is causing high latency in the API.

Thursday 7th June 2018

No incidents reported

Wednesday 6th June 2018

(Wednesday 6th June 2018 21:37:50 UTC)
Notification Service Update: Notification service outage

The MQTT notification service (Android) has experienced downtime in the past hours due to a large-scale Denial of Service attack on our service by an unknown party. We have been unsuccessful in blocking the attack. Instead, we have greatly over-provisioned compute resources to support the immense number of connections introduced by the attack.

Service has currently been restored. At this time we are continuing to monitor the attack as it is still ongoing. We are also investigating ways to detect and block Denial of Service attacks automatically to prevent similar incidents in the future.

Further incidents will be posted if there are any future developments. We truly apologize for any inconvenience caused by this Denial of Service attack.

(Wednesday 6th June 2018 21:30:17 UTC)
Notification Service Notification Service health check passed

The HTTP endpoint is now operating as expected.

(Wednesday 6th June 2018 14:37:50 UTC)
Notification Service Notification Service health check failed

We're investigating an issue that is causing the HTTP health check to fail for the Notification Service.

(Wednesday 6th June 2018 00:28:00 UTC)
Notification Service Post-mortem: Notification service partial outage

Notification service partial outage has affected the service for a period of less than 30 minutes due to unexpected load on the cluster. We have managed to stabilize the service. Further incidents will be posted if there are any future developments. We apologize for any inconvenience caused.

Tuesday 5th June 2018

(Tuesday 5th June 2018 16:36:09 UTC)
Notification Service Notification Service health check passed

The HTTP endpoint is now operating as expected.

(Tuesday 5th June 2018 16:15:46 UTC)
Notification Service Notification Service health check failed

We're investigating an issue that is causing the HTTP health check to fail for the Notification Service.