Service Under Maintenance
Website   Under Maintenance
Log Collection   Operational
Log Search   Operational
Search Alerts   Operational
Operational
Degraded Performance
Partial Outage
Major Outage
Maintenance
Scheduled Maintenance
Update - We will be undergoing scheduled maintenance during this time.
Nov 16, 16:52 PST
Scheduled - On Monday at around 15:00 UTC, Papertrail, along with the rest of the SolarWinds DevOps product line, will undergo migration to a new, unified authentication system. Once the migration is complete, all logins will be routed through https://my.solarwinds.cloud. Read more about why, what this new experience will look like, and what to expect on a day-to-day basis here: https://blog.papertrailapp.com/common-login-screen/

There are four crucial pieces of information to keep in mind:

1. Logged-in users will be logged out. This will likely manifest as redirection to the login page or the Events viewer stopping live tail or search functions.
2. Customers with outstanding new-user invitations will see them automatically expire. New invitations will need to be sent.
3. Papertrail as a service–including Log collection, filtering, archiving, and alerting–will remain entirely functional.
4. Heroku users are exempt from this conversion. Access to Papertrail via the Heroku Dashboard (http://dashboard.heroku.com) will remain the same.

We're expecting this process to only take a few minutes but are allotting extra time to allow the dust to fully settle. Papertrail Support (support@papertrailapp.com) is standing by to assist during and after the migration.
Nov 16, 14:31 PST
Past Incidents
Nov 17, 2018

No incidents reported today.

Nov 16, 2018

No incidents reported.

Nov 15, 2018

No incidents reported.

Nov 14, 2018
Resolved - All archives have been backfilled and current archives will continue to sync successfully.

Papertrail Support (support@papertrailapp.com) will be happy to dive into any outstanding archiving issues from here on out.
Nov 14, 20:54 PST
Monitoring - We've fixed the issue regarding archive pushes not authenticating even though the test was successful, and back-fill is progressing successfully across the board. New archives will also push successfully as they are generated.

We'll monitor this for a bit and if nothing appears out of the ordinary, we'll mark this as resolved.
Nov 14, 15:18 PST
Investigating - Most customers should be seeing archives update in their respective buckets. Some are still having authentication issues during the back-fill process, even though the bucket association has been rebuilt. We've engaged AWS and will update when we know more.
Nov 14, 12:23 PST
Update - Back-filling has begun for hours starting at 00:00 to 13:00 UTC. This should encompass all potential missing archives.
Nov 14, 09:24 PST
Monitoring - Engineering determined the root cause to be an issue with stale region data that was more strictly enforced after the migration to our new archiving pipeline. This caused perfectly good credentials to return as invalid, the S3 integration to be disabled, and the email to go out erroneously. Engineering has cleared the stale region data and released all the failure locks for buckets attached to Papertrail accounts.

If you haven't explicitly "re-enabled" (via Settings > Archives), there should be no need at this point. If archives still are not working, credentials should be double-checked. Papertrail Support will be more than happy to debug if that's the case.

Engineering is also beginning work on back-filling archives that were missed during this period.
Nov 14, 08:48 PST
Identified - We've determined the issue stemmed from Papertrail recording the bucket as being a part of a different region than in which it resided.

If you received an email, click "Re-enable" within Settings > Archives and pushing to S3 will start functioning again. We're working on silently handling this, as well, for those who were affected.
Nov 14, 08:24 PST
Investigating - Engineering is investigating an issue where some customers received reports their S3 bucket credentials were incorrect and pushing archives to said bucket was disabled.

All core Papertrail functionality is intact and Papertrail-hosted archives are available.
Nov 14, 08:19 PST
Nov 13, 2018
Completed - The scheduled maintenance has been completed.
Nov 13, 20:30 PST
Update - Archive generation using the legacy pipeline has been paused. Generation using the new pipeline kicks off at around 03:00 UTC where it will begin with the 00:00-00:59 UTC time window. Once we're confident in the process, maintenance will conclude.
Nov 13, 16:29 PST
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
Nov 13, 16:00 PST
Scheduled - Our Engineering team will be deploying potentially breaking changes to Papertrail's archiving pipeline. Depending on the current configuration of an account's archive intervals (daily or hourly), this might bring breaking changes:

- For those who receive HOURLY archives: expect no changes. The Papertrail UI and API for archives will remain the same and function as they did previously.

- For those who receive DAILY archives: these accounts will be converted to hourly. Please update any API implementations to account for this.

To confirm to which interval an account is currently set, log in to Papertrail and head to Settings => Archives.

After the migration is complete, 100% of accounts will have hourly archives. Please keep in mind that Papertrail Support will no longer be able to override the interval. The standard archive interval across all accounts under all circumstances (including users of Papertrail as a Heroku add-on) will be hourly.

If you have any questions or concerns or would like assistance in translating your API calls to conform to the hourly interval, please email us: support@papertrailapp.com.
Nov 7, 10:46 PST
Nov 12, 2018

No incidents reported.

Nov 11, 2018

No incidents reported.

Nov 10, 2018

No incidents reported.

Nov 9, 2018

No incidents reported.

Nov 8, 2018

No incidents reported.

Nov 7, 2018

No incidents reported.

Nov 6, 2018

No incidents reported.

Nov 5, 2018
Resolved - All customers are caught up and users should see live logs.
Nov 5, 12:53 PST
Identified - Engineering is tracking an issue affecting the real-time status of log processing for some customers. Those who are affected are currently seeing delays of as low as 1 minute and as high as 30 minutes when searching for recent logs or tailing.

There is no impact on logs being collected. There is potential impact to search alert performance, though the impact is greatly dependent on the frequency of matched events and the alert's window.
Nov 5, 11:01 PST
Nov 4, 2018

No incidents reported.

Nov 3, 2018

No incidents reported.