Maintenance Window Continuation
Incident Report for Papertrail
Postmortem

At around 3:30 a.m. PDT on Wednesday, September 5, our hosting provider performed scheduled maintenance to machines hosting a number of Papertrail’s virtual server instances. These security updates included a configuration change that consequently impaired performance on those instances and affected the backup instances we had provisioned to cover the maintenance window.

As a result, for a period of 48 hours, Papertrail users experienced delays in log processing and alerts.

Over the past month, we’ve taken significant steps to increase the durability and performance of our platform as we continue to build the next generation of Papertrail. This level of service interruption is not the bar we aim to set; therefore, we are in the process of reducing our dependence on a single hosting provider.

Posted Sep 12, 2018 - 17:46 PDT

Resolved
This incident has been resolved.
Posted Sep 06, 2018 - 20:44 PDT
Monitoring
We are almost done burning the backlog, ETA 1 hour.
Posted Sep 06, 2018 - 19:38 PDT
Update
There is no ETA at the moment. Log viewer may be 2-3 hours behind.
Posted Sep 06, 2018 - 13:08 PDT
Update
Disk issues resolved. Moving ETA to 4:00 PM UTC.
Posted Sep 06, 2018 - 07:56 PDT
Update
We are dealing with disk space issues, moving ETA to 2:30 PM UTC
Posted Sep 06, 2018 - 05:48 PDT
Update
ETA to resolution is 1:30 PM UTC.
Posted Sep 06, 2018 - 04:01 PDT
Update
We're 76% done with the backlog, revised ETA is 3 hours from now.
Posted Sep 06, 2018 - 01:42 PDT
Investigating
We are currently investigating this issue.
Posted Sep 05, 2018 - 23:57 PDT
This incident affected: Log Search and Search Alerts.