Get email notifications whenever Papertrail creates, updates or resolves an incident.
Get text message notifications whenever Papertrail creates or resolves an incident.
Get webhook notifications whenever Papertrail creates an incident, updates an incident, resolves an incident or changes a component status.
Visit our support site.
All Systems Operational
Website
Operational
Log Collection
Operational
Log Search
Operational
Search Alerts
Operational
Operational
Degraded Performance
Partial Outage
Major Outage
Maintenance
Past Incidents
Feb 18, 2019
No incidents reported today.
Feb 17, 2019
No incidents reported.
Feb 16, 2019
Resolved -
Search Alerts have been returned to normal behavior.
Feb 16, 02:21 PST
Feb 16, 02:21 PST
Monitoring -
Starting at 03:00 UTC on Feb 16th Search Alerts were delayed and we are continuing to work through the backlog.
Nearly all of the search alerts that would normally run between 03:00 UTC and 5:45 UTC were not run. A small number of search alerts that would normally run during that time searched a span of days rather than minutes or hours and may have resulted in alerts that were false positives.
The backlog of search alerts dating back to 5:45 UTC is being processed now.
Feb 16, 00:12 PST
Nearly all of the search alerts that would normally run between 03:00 UTC and 5:45 UTC were not run. A small number of search alerts that would normally run during that time searched a span of days rather than minutes or hours and may have resulted in alerts that were false positives.
The backlog of search alerts dating back to 5:45 UTC is being processed now.
Feb 16, 00:12 PST
Identified -
We've identified the problem and we're working to get Search Alerts back to realtime.
Feb 15, 23:14 PST
Feb 15, 23:14 PST
Update -
We continue to investigate.
Feb 15, 21:50 PST
Feb 15, 21:50 PST
Update -
We are continuing to investigate and search alerts remain delayed.
Feb 15, 20:36 PST
Feb 15, 20:36 PST
Investigating -
Search alerts are delayed.
Feb 15, 19:21 PST
Feb 15, 19:21 PST
Feb 14, 2019
Resolved -
Additional resources were added to the logs and logs6 endpoints and they have been stable for a week. We will continue to monitor, but do not expect any further customer impact from slow TLS handshakes.
Feb 14, 14:14 PST
Feb 14, 14:14 PST
Monitoring -
The logs and logs6 endpoints have been experiencing increased load recently, leading to slower TLS handshakes during peak periods.
UDP senders and plaintext TCP senders are not affected. Some TLS senders may see increased queueing and delays during peak times, depending on how often they reconnect to Papertrail. Some senders could even drop messages, in the absence of client-side queueing.
We're adding capacity to handle the increased load, but it may be a few days before that is online. In the meantime, anyone affected by the slower TLS handshakes is advised to create a new destination, which will be allocated on a different cluster.
Jan 28, 15:57 PST
UDP senders and plaintext TCP senders are not affected. Some TLS senders may see increased queueing and delays during peak times, depending on how often they reconnect to Papertrail. Some senders could even drop messages, in the absence of client-side queueing.
We're adding capacity to handle the increased load, but it may be a few days before that is online. In the meantime, anyone affected by the slower TLS handshakes is advised to create a new destination, which will be allocated on a different cluster.
Jan 28, 15:57 PST
Feb 13, 2019
No incidents reported.
Feb 12, 2019
Resolved -
From 19:17 UTC to 20:05 UTC search alerts were delayed.
Feb 12, 15:33 PST
Feb 12, 15:33 PST
Monitoring -
Search alerts have returned to normal.
Feb 12, 13:06 PST
Feb 12, 13:06 PST
Investigating -
Search alerts have been delayed since 19:17 UTC.
Feb 12, 12:03 PST
Feb 12, 12:03 PST
Feb 11, 2019
No incidents reported.
Feb 10, 2019
Resolved -
Accrued usage (in bytes) and "last event seen" counters are working as of 07:25 UTC on 11 Feb 2019. Bytes received after that time and sources seen after that time will be tracked as they were before this incident.
Feb 10, 23:37 PST
Feb 10, 23:37 PST
Identified -
We've identified the blockage and have returned logs to real-time status. Some 24-hour search alerts may not reflect accurate message counts.
Accrued usage (in bytes) and "last event seen" counters may not reflect accurate information for some customers. Engineering is actively working on restoring this functionality. This does not impact log processing/ingestion, search, or alerts.
Feb 10, 22:07 PST
Accrued usage (in bytes) and "last event seen" counters may not reflect accurate information for some customers. Engineering is actively working on restoring this functionality. This does not impact log processing/ingestion, search, or alerts.
Feb 10, 22:07 PST
Update -
Engineering is still investigating. We've determined the scope to be contained to larger-volume customers. The effect is a cease of new logs in the Events viewer as well as Search Alerts not triggering. Log collection is NOT impacted so when the delay catches up, logs received during this period will become available.
Feb 10, 20:57 PST
Feb 10, 20:57 PST
Update -
A subset of customers will see delay in events viewer. We are still investigating the cause of the log processing delay.
Feb 10, 19:48 PST
Feb 10, 19:48 PST
Update -
We are still investigating the cause of the log processing delay and trying to process the backlog.
Feb 10, 19:01 PST
Feb 10, 19:01 PST
Investigating -
We are currently investigating log processing delay.
Feb 10, 17:38 PST
Feb 10, 17:38 PST
Feb 9, 2019
No incidents reported.
Feb 8, 2019
No incidents reported.
Feb 7, 2019
No incidents reported.
Feb 6, 2019
No incidents reported.
Feb 5, 2019
No incidents reported.
Feb 4, 2019
No incidents reported.