Maintenance Window Continuation
Incident Report for Papertrail
Postmortem

At around 3:30 a.m. PDT on Wednesday, September 5, our hosting provider performed scheduled maintenance to machines hosting a number of Papertrail’s virtual server instances. These security updates included a configuration change that consequently impaired performance on those instances and affected the backup instances we had provisioned to cover the maintenance window.

As a result, for a period of 48 hours, Papertrail users experienced delays in log processing and alerts.

Over the past month, we’ve taken significant steps to increase the durability and performance of our platform as we continue to build the next generation of Papertrail. This level of service interruption is not the bar we aim to set; therefore, we are in the process of reducing our dependence on a single hosting provider.

Posted 3 months ago. Sep 12, 2018 - 17:46 PDT

Resolved
This incident has been resolved.
Posted 3 months ago. Sep 06, 2018 - 20:44 PDT
Monitoring
We are almost done burning the backlog, ETA 1 hour.
Posted 3 months ago. Sep 06, 2018 - 19:38 PDT
Update
There is no ETA at the moment. Log viewer may be 2-3 hours behind.
Posted 3 months ago. Sep 06, 2018 - 13:08 PDT
Update
Disk issues resolved. Moving ETA to 4:00 PM UTC.
Posted 3 months ago. Sep 06, 2018 - 07:56 PDT
Update
We are dealing with disk space issues, moving ETA to 2:30 PM UTC
Posted 3 months ago. Sep 06, 2018 - 05:48 PDT
Update
ETA to resolution is 1:30 PM UTC.
Posted 3 months ago. Sep 06, 2018 - 04:01 PDT
Update
We're 76% done with the backlog, revised ETA is 3 hours from now.
Posted 3 months ago. Sep 06, 2018 - 01:42 PDT
Investigating
We are currently investigating this issue.
Posted 3 months ago. Sep 05, 2018 - 23:57 PDT
This incident affected: Log Search and Search Alerts.