The Pushy database cluster is still recovering after an AWS EC2 / EBS volume outage in the 'us-east-1b' region, where the primary Pushy database is hosted. For a period of 4-5 hours, our primary database was completely unreachable due to an I/O cutoff from its EBS volumes, and so we had failed back to a secondary database quickly, within 10 minutes of the incident manifesting.
5 hours later, when AWS has restored connectivity to EBS volumes within the us-east-1b region, the primary database has gone out of sync, and is currently undergoing a full resync. This is causing slightly degraded performance due to increased resource contention to serve both API requests and to resync the primary database server.
The Pushy API is currently operational, although there are temporary periods of degraded performance while latency spikes. The resync is expected to complete within the next 2 hours, after which API latency should return to expected levels of ~15ms average.
We deeply apologize for the impact and inconvenience this has had on your reliable notification delivery experience. Thanks for your patience and understanding as we work to continue resolving this issue ASAP. We will post an update when it is fully resolved.
