We've fixed the issue regarding archive pushes not authenticating even though the test was successful, and back-fill is progressing successfully across the board. New archives will also push successfully as they are generated.
We'll monitor this for a bit and if nothing appears out of the ordinary, we'll mark this as resolved.
Posted 5 months ago. Nov 14, 2018 - 15:18 PST
Most customers should be seeing archives update in their respective buckets. Some are still having authentication issues during the back-fill process, even though the bucket association has been rebuilt. We've engaged AWS and will update when we know more.
Posted 5 months ago. Nov 14, 2018 - 12:23 PST
Back-filling has begun for hours starting at 00:00 to 13:00 UTC. This should encompass all potential missing archives.
Posted 5 months ago. Nov 14, 2018 - 09:24 PST
Engineering determined the root cause to be an issue with stale region data that was more strictly enforced after the migration to our new archiving pipeline. This caused perfectly good credentials to return as invalid, the S3 integration to be disabled, and the email to go out erroneously. Engineering has cleared the stale region data and released all the failure locks for buckets attached to Papertrail accounts.
If you haven't explicitly "re-enabled" (via Settings > Archives), there should be no need at this point. If archives still are not working, credentials should be double-checked. Papertrail Support will be more than happy to debug if that's the case.
Engineering is also beginning work on back-filling archives that were missed during this period.
Posted 5 months ago. Nov 14, 2018 - 08:48 PST
We've determined the issue stemmed from Papertrail recording the bucket as being a part of a different region than in which it resided.
If you received an email, click "Re-enable" within Settings > Archives and pushing to S3 will start functioning again. We're working on silently handling this, as well, for those who were affected.
Posted 5 months ago. Nov 14, 2018 - 08:24 PST
Engineering is investigating an issue where some customers received reports their S3 bucket credentials were incorrect and pushing archives to said bucket was disabled.
All core Papertrail functionality is intact and Papertrail-hosted archives are available.