Delays in log viewer
Incident Report for Papertrail
Resolved
This incident has been resolved.
Posted 23 days ago. Nov 20, 2018 - 22:26 PST
Update
We are all caught up! Alerts should be arriving on time. Delayed search alerts will be sent when they get processed.
Posted 23 days ago. Nov 20, 2018 - 22:25 PST
Update
We are 95% complete. ETA 40 min
Posted 23 days ago. Nov 20, 2018 - 21:48 PST
Update
About 85% done with the backlog processing.
Posted 24 days ago. Nov 20, 2018 - 20:54 PST
Update
Doesn't seem we're going to hit this estimate. ETA for realtime log search is 75min.
Posted 24 days ago. Nov 20, 2018 - 20:30 PST
Update
We are about 40% done, so it's looking like we might be able to process this within the next 90 minutes.
Posted 24 days ago. Nov 20, 2018 - 19:15 PST
Update
Rough ETA is 2 hours from now. Will refine as more data comes through.
Posted 24 days ago. Nov 20, 2018 - 18:17 PST
Update
We've now deployed significant extra capacity to burn down the backlog as quickly as possible. ETA coming soon.
Posted 24 days ago. Nov 20, 2018 - 18:09 PST
Update
We've just finished processing our filtering backlog. We're now continuing to push the data to be searchable. Posting an ETA for that within 30 minutes as we deploy extra capacity and measure the ingestion rates.
Posted 24 days ago. Nov 20, 2018 - 17:47 PST
Update
At the current rate, it looks like the filtering backlog will be done in an hour. We'll publish a revised ETA as soon as we finish that backlog.
Posted 24 days ago. Nov 20, 2018 - 16:45 PST
Update
We are continuing to work on a fix for this issue.
Posted 24 days ago. Nov 20, 2018 - 16:44 PST
Update
We continue to make good progress on the filter backlog. ETA is 3 hours, but we'll have a clearer picture as soon as we can clear out the filter backlog.
Posted 24 days ago. Nov 20, 2018 - 15:50 PST
Update
We've found a way to process part of the backlog quicker. Our unofficial ETA is 4 hours to resolution for log search. More info in an hour.
Posted 24 days ago. Nov 20, 2018 - 14:46 PST
Update
The backlog is continuing to shrink. We don't have an ETA for 'live' yet, but the delays are decreasing steadily and we should have a clearer picture, soon.
Posted 24 days ago. Nov 20, 2018 - 14:03 PST
Update
The backlog is draining, though slower than we originally anticipated. We've tasked as many systems to processing as we can and are forging ahead.
Posted 24 days ago. Nov 20, 2018 - 12:42 PST
Update
Provisioning Papertrail as a Heroku add-on is functioning, again. Continuing to consume the backlog of logs received.
Posted 24 days ago. Nov 20, 2018 - 10:22 PST
Update
We're continuing to drain the queues. In the interim, we're still seeing provisioning failures for Papertrail as a Heroku add-on and are investigating.
Posted 24 days ago. Nov 20, 2018 - 10:11 PST
Identified
Engineering has identified the cause of the blockage. The pipeline is burning through the backlog.
Posted 24 days ago. Nov 20, 2018 - 09:50 PST
Update
We are experiencing contention issues in our filtering layer. We're investigating.
Posted 24 days ago. Nov 20, 2018 - 06:31 PST
Update
We are continuing to investigate this issue.
Posted 24 days ago. Nov 20, 2018 - 06:24 PST
Investigating
We are currently investigating this issue.
Posted 24 days ago. Nov 20, 2018 - 06:03 PST
This incident affected: Log Search and Search Alerts.