Get email notifications whenever Papertrail creates, updates or resolves an incident.
Get webhook notifications whenever Papertrail creates an incident, updates an incident, resolves an incident or changes a component status.
Visit our support site.
All Systems Operational
Website
Operational
Log Collection
Operational
Log Search
Operational
Search Alerts
Operational
Operational
Degraded Performance
Partial Outage
Major Outage
Maintenance
Past Incidents
Nov 16, 2018
No incidents reported today.
Nov 15, 2018
No incidents reported.
Nov 14, 2018
Resolved -
All archives have been backfilled and current archives will continue to sync successfully.
Papertrail Support (support@papertrailapp.com) will be happy to dive into any outstanding archiving issues from here on out.
Nov 14, 20:54 PST
Papertrail Support (support@papertrailapp.com) will be happy to dive into any outstanding archiving issues from here on out.
Nov 14, 20:54 PST
Monitoring -
We've fixed the issue regarding archive pushes not authenticating even though the test was successful, and back-fill is progressing successfully across the board. New archives will also push successfully as they are generated.
We'll monitor this for a bit and if nothing appears out of the ordinary, we'll mark this as resolved.
Nov 14, 15:18 PST
We'll monitor this for a bit and if nothing appears out of the ordinary, we'll mark this as resolved.
Nov 14, 15:18 PST
Investigating -
Most customers should be seeing archives update in their respective buckets. Some are still having authentication issues during the back-fill process, even though the bucket association has been rebuilt. We've engaged AWS and will update when we know more.
Nov 14, 12:23 PST
Nov 14, 12:23 PST
Update -
Back-filling has begun for hours starting at 00:00 to 13:00 UTC. This should encompass all potential missing archives.
Nov 14, 09:24 PST
Nov 14, 09:24 PST
Monitoring -
Engineering determined the root cause to be an issue with stale region data that was more strictly enforced after the migration to our new archiving pipeline. This caused perfectly good credentials to return as invalid, the S3 integration to be disabled, and the email to go out erroneously. Engineering has cleared the stale region data and released all the failure locks for buckets attached to Papertrail accounts.
If you haven't explicitly "re-enabled" (via Settings > Archives), there should be no need at this point. If archives still are not working, credentials should be double-checked. Papertrail Support will be more than happy to debug if that's the case.
Engineering is also beginning work on back-filling archives that were missed during this period.
Nov 14, 08:48 PST
If you haven't explicitly "re-enabled" (via Settings > Archives), there should be no need at this point. If archives still are not working, credentials should be double-checked. Papertrail Support will be more than happy to debug if that's the case.
Engineering is also beginning work on back-filling archives that were missed during this period.
Nov 14, 08:48 PST
Identified -
We've determined the issue stemmed from Papertrail recording the bucket as being a part of a different region than in which it resided.
If you received an email, click "Re-enable" within Settings > Archives and pushing to S3 will start functioning again. We're working on silently handling this, as well, for those who were affected.
Nov 14, 08:24 PST
If you received an email, click "Re-enable" within Settings > Archives and pushing to S3 will start functioning again. We're working on silently handling this, as well, for those who were affected.
Nov 14, 08:24 PST
Investigating -
Engineering is investigating an issue where some customers received reports their S3 bucket credentials were incorrect and pushing archives to said bucket was disabled.
All core Papertrail functionality is intact and Papertrail-hosted archives are available.
Nov 14, 08:19 PST
All core Papertrail functionality is intact and Papertrail-hosted archives are available.
Nov 14, 08:19 PST
Nov 13, 2018
Completed -
The scheduled maintenance has been completed.
Nov 13, 20:30 PST
Nov 13, 20:30 PST
Update -
Archive generation using the legacy pipeline has been paused. Generation using the new pipeline kicks off at around 03:00 UTC where it will begin with the 00:00-00:59 UTC time window. Once we're confident in the process, maintenance will conclude.
Nov 13, 16:29 PST
Nov 13, 16:29 PST
In progress -
Scheduled maintenance is currently in progress. We will provide updates as necessary.
Nov 13, 16:00 PST
Nov 13, 16:00 PST
Scheduled -
Our Engineering team will be deploying potentially breaking changes to Papertrail's archiving pipeline. Depending on the current configuration of an account's archive intervals (daily or hourly), this might bring breaking changes:
- For those who receive HOURLY archives: expect no changes. The Papertrail UI and API for archives will remain the same and function as they did previously.
- For those who receive DAILY archives: these accounts will be converted to hourly. Please update any API implementations to account for this.
To confirm to which interval an account is currently set, log in to Papertrail and head to Settings => Archives.
After the migration is complete, 100% of accounts will have hourly archives. Please keep in mind that Papertrail Support will no longer be able to override the interval. The standard archive interval across all accounts under all circumstances (including users of Papertrail as a Heroku add-on) will be hourly.
If you have any questions or concerns or would like assistance in translating your API calls to conform to the hourly interval, please email us: support@papertrailapp.com.
Nov 7, 10:46 PST
- For those who receive HOURLY archives: expect no changes. The Papertrail UI and API for archives will remain the same and function as they did previously.
- For those who receive DAILY archives: these accounts will be converted to hourly. Please update any API implementations to account for this.
To confirm to which interval an account is currently set, log in to Papertrail and head to Settings => Archives.
After the migration is complete, 100% of accounts will have hourly archives. Please keep in mind that Papertrail Support will no longer be able to override the interval. The standard archive interval across all accounts under all circumstances (including users of Papertrail as a Heroku add-on) will be hourly.
If you have any questions or concerns or would like assistance in translating your API calls to conform to the hourly interval, please email us: support@papertrailapp.com.
Nov 7, 10:46 PST
Nov 12, 2018
No incidents reported.
Nov 11, 2018
No incidents reported.
Nov 10, 2018
No incidents reported.
Nov 9, 2018
No incidents reported.
Nov 8, 2018
No incidents reported.
Nov 7, 2018
No incidents reported.
Nov 6, 2018
No incidents reported.
Nov 5, 2018
Resolved -
All customers are caught up and users should see live logs.
Nov 5, 12:53 PST
Nov 5, 12:53 PST
Identified -
Engineering is tracking an issue affecting the real-time status of log processing for some customers. Those who are affected are currently seeing delays of as low as 1 minute and as high as 30 minutes when searching for recent logs or tailing.
There is no impact on logs being collected. There is potential impact to search alert performance, though the impact is greatly dependent on the frequency of matched events and the alert's window.
Nov 5, 11:01 PST
There is no impact on logs being collected. There is potential impact to search alert performance, though the impact is greatly dependent on the frequency of matched events and the alert's window.
Nov 5, 11:01 PST
Nov 4, 2018
No incidents reported.
Nov 3, 2018
No incidents reported.
Nov 2, 2018
No incidents reported.