APIPost-mortem: API & Notification service high latency & downtime (April 11, 2020)
First off, we'd like to sincerely apologize for any impact this incident has had on your app's notification experience. Our core mission at Pushy has always been providing a fast, reliable, always-on notification service you can rely on 100% of the time. We have unfortunately failed to meet that expectation today.
The Pushy Pro API & notification service have experienced high latency and partial downtime over the past 8 hours due to a large-scale, sophisticated Denial of Service attack on our API. After further investigation and multiple attempts, we have been successful in blocking the attack and ensuring it will be blocked in the future to prevent any similar incidents.
The incident is now resolved and service is restored. Once again, we truly apologize for any inconvenience and negative impact caused by this incident.
We thank you for choosing Pushy. Please contact us if you have any questions or concerns.
(Saturday 11th April 2020 21:06:13 UTC)
APIAPI health has stabilized
The API has returned to normal operational capacity.
(Saturday 11th April 2020 19:16:54 UTC)
APIAPI: Incident reclassified as DDoS attack - fix in progress
We've identified a DDoS attack on the API which is causing this incident. We're working on restoring availability ASAP and blocking this attack vector. Thank you for your patience.
(Saturday 11th April 2020 18:37:42 UTC)
Notification ServiceNotification Service high latency
We're investigating an issue that is causing high latency in the Notification Service.
(Saturday 11th April 2020 18:20:06 UTC)
APIAPI: Issue identified & fix in progress
We've identified the issue causing the incident and are working on a fix ASAP. Thank you for your patience.
(Saturday 11th April 2020 14:07:04 UTC)
APIAPI high latency
We're investigating an issue that is causing high latency in the API.