Syslog client reconnects
Incident Report for Papertrail
Resolved
This incident has been resolved.
Posted 4 months ago. Aug 30, 2018 - 09:34 PDT
Update
All delayed alerts have been sent. We'll continue to monitor the system's performance before declaring it resolved.
Posted 4 months ago. Aug 30, 2018 - 09:24 PDT
Update
Log search is caught up. We continue to process alerts.
Posted 4 months ago. Aug 30, 2018 - 09:12 PDT
Update
Delays are now affecting around 5% of customers, up to 20 minutes. Delayed alerts are still being sent.
Posted 4 months ago. Aug 30, 2018 - 09:08 PDT
Update
We continue to process the backlog. Around 30% of customers are still delayed by up to 90 minutes.
Posted 4 months ago. Aug 30, 2018 - 07:26 PDT
Update
We are trying to accelerate backlog completion traffic ramps up. Affected hosts keep exhibiting great performance, despite the increased load.
Posted 4 months ago. Aug 30, 2018 - 06:05 PDT
Update
We are continuing to monitor for any further issues.
Posted 4 months ago. Aug 30, 2018 - 03:41 PDT
Monitoring
Connections are back to normal and we are burning down the backlog. The fix seems to brought a pronounced benefit to our database.
Posted 4 months ago. Aug 30, 2018 - 03:40 PDT
Identified
We have identified a potential fix for the database problems.
Posted 4 months ago. Aug 30, 2018 - 03:04 PDT
Update
Inbound connections have stabilized. We are waiting for our provider to give us updates. We are burning down the backlog of messages we've received.
Posted 4 months ago. Aug 30, 2018 - 02:15 PDT
Update
We have engaged our networking provider to assist us in troubleshooting.
Posted 4 months ago. Aug 30, 2018 - 00:54 PDT
Update
Connections are starting to normalize. We continue investigating and dealing with the backlog. Expect new syslog endpoints creation to be delayed by an unknown time.
Posted 4 months ago. Aug 30, 2018 - 00:51 PDT
Update
We are still investigating. We are seeing more successful connections coming in.
Posted 4 months ago. Aug 29, 2018 - 23:55 PDT
Update
Log clients are continuing to have connection issues
Posted 4 months ago. Aug 29, 2018 - 21:17 PDT
Investigating
We are currently investigating this issue.
Posted 4 months ago. Aug 29, 2018 - 20:57 PDT
This incident affected: Log Collection and Log Search.