Papertrail Status

Sep 18, 2018

No incidents reported today.

Sep 17, 2018

No incidents reported.

Sep 16, 2018

No incidents reported.

Sep 15, 2018

No incidents reported.

Sep 14, 2018

No incidents reported.

Sep 13, 2018

No incidents reported.

Sep 12, 2018

No incidents reported.

Sep 11, 2018

No incidents reported.

Sep 10, 2018

No incidents reported.

Sep 9, 2018

No incidents reported.

Sep 8, 2018

No incidents reported.

Sep 7, 2018

Small Delay in Log Viewing

Resolved - Everything is green across the board. Thanks for coming along for the ride.

While we don't expect this to be true, shoot an email to support@papertrailapp.com if anything looks out of place.
Sep 7, 13:40 PDT

Monitoring - All the queues are clear and we're 100% live. We'll keep an eye on things to make sure they stay that way. Stay tuned for the conclusion to this epic saga.
Sep 7, 13:27 PDT

Update - There was a slight speed bump in that some folks ended up with larger delays but the news in the end is good. Some will still have a delay of up to 10 minutes while others will have no delay at all. Stay tuned as we're expecting everyone to burn down to zero soon.
Sep 7, 12:00 PDT

Identified - We've discovered an issue regarding logs taking longer than usual to be made available within the Viewer. We're aware of what's causing the delay (an effect of the microcode patches from our recent maintenance window) and working on bolstering the processes to eliminate the delay altogether. Currently, the delay can be between 2 and 5 minutes depending on what slot your account falls into within the pipeline.
Sep 7, 10:58 PDT

Sep 6, 2018

Maintenance Window Continuation

Postmortem - Read details
Sep 12, 17:46 PDT

Resolved - This incident has been resolved.
Sep 6, 20:44 PDT

Monitoring - We are almost done burning the backlog, ETA 1 hour.
Sep 6, 19:38 PDT

Update - There is no ETA at the moment. Log viewer may be 2-3 hours behind.
Sep 6, 13:08 PDT

Update - Disk issues resolved. Moving ETA to 4:00 PM UTC.
Sep 6, 07:56 PDT

Update - We are dealing with disk space issues, moving ETA to 2:30 PM UTC
Sep 6, 05:48 PDT

Update - ETA to resolution is 1:30 PM UTC.
Sep 6, 04:01 PDT

Update - We're 76% done with the backlog, revised ETA is 3 hours from now.
Sep 6, 01:42 PDT

Investigating - We are currently investigating this issue.
Sep 5, 23:57 PDT

Sep 5, 2018

Hosting Provider Server Restarts

Completed - The scheduled maintenance has been completed.
Sep 5, 23:45 PDT

Update - We are about 70% done with the backlog burndown. ETA to resolution is 3 hours. Will continue providing updates as they come.
Sep 5, 23:24 PDT

Update - Backlog continues to burn down. Will update with more information and ETA to full resolution in 90 minutes.
Sep 5, 21:51 PDT

Update - We've reached the upper bounds of the additional capacity we can add. We're about 60% of the way.
Sep 5, 21:17 PDT

Update - We are provisioning extra capacity on top of the extra capacity to see if we can clear the backlog quicker.
Sep 5, 18:22 PDT

Update - 52% of the backlog is cleared.
Sep 5, 16:56 PDT

Update - Around 47% of the backlog has been cleared.
Sep 5, 15:44 PDT

Update - Datastore backlog is 30% processed, we continue to process the backlog as fast as possible.
Sep 5, 13:10 PDT

Update - Datastore backlog is 23% processed, we continue to process the backlog as fast as possible.
Sep 5, 12:24 PDT

Update - Filtering is now fully operational. We continue to work on the datastore inserter backlog.
Sep 5, 10:38 PDT

Update - 82% of the filtering layer backlog has been processed. Roughly 5% of the log search data backlog has processed.
Sep 5, 09:30 PDT

Update - We are extending the maintenance window as we continue to investigate. Backlog is being processed at the moment.
Sep 5, 07:26 PDT

Update - Catching up with the backlog is taking longer than expected - we are investigating. There will be a delay in log viewing and search alerts.
Sep 5, 05:53 PDT

Update - Capacity is coming back online, we're focusing on burning down any queue lag.
Sep 5, 05:13 PDT

Update - Our provider has taken down more than expected capacity. There is a slight delay in log viewing. We are trying to bring this under control now.
Sep 5, 04:32 PDT

In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Sep 5, 03:45 PDT

Scheduled - Our hosting provider will be restarting a significant number of servers during this time window.
Log collection and the website are not impacted. Log Search and Search Alerts performance might be degraded during this time. We are working with our hosting provider to minimize the impact to our service.
Aug 31, 08:18 PDT

Sep 4, 2018

No incidents reported.