All archives have been backfilled and current archives will continue to sync successfully.
Papertrail Support (support@papertrailapp.com) will be happy to dive into any outstanding archiving issues from here on out.
Posted Nov 14, 2018 - 20:54 PST
Monitoring
We've fixed the issue regarding archive pushes not authenticating even though the test was successful, and back-fill is progressing successfully across the board. New archives will also push successfully as they are generated.
We'll monitor this for a bit and if nothing appears out of the ordinary, we'll mark this as resolved.
Posted Nov 14, 2018 - 15:18 PST
Investigating
Most customers should be seeing archives update in their respective buckets. Some are still having authentication issues during the back-fill process, even though the bucket association has been rebuilt. We've engaged AWS and will update when we know more.
Posted Nov 14, 2018 - 12:23 PST
Update
Back-filling has begun for hours starting at 00:00 to 13:00 UTC. This should encompass all potential missing archives.
Posted Nov 14, 2018 - 09:24 PST
Monitoring
Engineering determined the root cause to be an issue with stale region data that was more strictly enforced after the migration to our new archiving pipeline. This caused perfectly good credentials to return as invalid, the S3 integration to be disabled, and the email to go out erroneously. Engineering has cleared the stale region data and released all the failure locks for buckets attached to Papertrail accounts.
If you haven't explicitly "re-enabled" (via Settings > Archives), there should be no need at this point. If archives still are not working, credentials should be double-checked. Papertrail Support will be more than happy to debug if that's the case.
Engineering is also beginning work on back-filling archives that were missed during this period.
Posted Nov 14, 2018 - 08:48 PST
Identified
We've determined the issue stemmed from Papertrail recording the bucket as being a part of a different region than in which it resided.
If you received an email, click "Re-enable" within Settings > Archives and pushing to S3 will start functioning again. We're working on silently handling this, as well, for those who were affected.
Posted Nov 14, 2018 - 08:24 PST
Investigating
Engineering is investigating an issue where some customers received reports their S3 bucket credentials were incorrect and pushing archives to said bucket was disabled.
All core Papertrail functionality is intact and Papertrail-hosted archives are available.