Alert notification issues

Incident Report for Papertrail

Resolved

This incident has been resolved.

Posted Oct 05, 2020 - 12:01 PDT

Monitoring

Slack appears to be recovering. As of 1732 UTC, we have re-enabled Slack Alerts.

All alerts have been processing without issues.

Posted Oct 05, 2020 - 10:51 PDT

Update

Slack Alerts are still not being delivered. We are continuing to monitor the status of their APIs.

All other alerts are being delivered without issues.

Posted Oct 05, 2020 - 10:20 PDT

Update

Slack Alerts are still not being delivered. We are continuing to monitor the status of their APIs.

All other alerts are being delivered without issues.

Posted Oct 05, 2020 - 09:45 PDT

Identified

Between 1400 UTC and 1554 UTC we experienced intermittent issues delivering search alerts and not all search alerts may have been delivered.

Since 1554 UTC we have placed mitigations to stop delivering Slack alerts while the Slack API is experiencing issues. Other Search Alerts are being delivered as expected.

Posted Oct 05, 2020 - 09:03 PDT

Investigating

We are currently investigating this issue.

Posted Oct 05, 2020 - 08:36 PDT

This incident affected: Search Alerts.