tag:www.papertrailstatus.com,2005:/historyPapertrail Status - Incident History2024-03-18T19:30:33-07:00Papertrailtag:www.papertrailstatus.com,2005:Incident/200635722024-02-23T23:46:30-08:002024-02-23T23:46:30-08:00Search<p><small>Feb <var data-var='date'>23</var>, <var data-var='time'>23:46</var> PST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Feb <var data-var='date'>23</var>, <var data-var='time'>22:42</var> PST</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p>tag:www.papertrailstatus.com,2005:Incident/199846622024-02-14T08:20:38-08:002024-02-14T08:20:38-08:00papertrailapp.com is unavailable<p><small>Feb <var data-var='date'>14</var>, <var data-var='time'>08:20</var> PST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Feb <var data-var='date'>14</var>, <var data-var='time'>08:07</var> PST</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Feb <var data-var='date'>14</var>, <var data-var='time'>07:45</var> PST</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p>tag:www.papertrailstatus.com,2005:Incident/193293992023-12-05T04:01:26-08:002023-12-05T04:01:51-08:00Search Alerts Delayed<p><small>Dec <var data-var='date'> 5</var>, <var data-var='time'>04:01</var> PST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Dec <var data-var='date'> 5</var>, <var data-var='time'>03:42</var> PST</small><br><strong>Update</strong> - The backlog has been processed. Search Alerts were enabled at 11:38 UTC. Some search alerts may fire initially due to the search window extending back to the start of this incident rather than the normal window (1 minute, 10 minutes, 1hr, etc.) but they will return to normal operation after that initial search with the larger window.</p><p><small>Dec <var data-var='date'> 5</var>, <var data-var='time'>02:50</var> PST</small><br><strong>Update</strong> - We continue to process the backlog and once that is complete we will enable Search Alerts.</p><p><small>Dec <var data-var='date'> 5</var>, <var data-var='time'>01:38</var> PST</small><br><strong>Monitoring</strong> - The logs visible in the viewer are up to date. More of the backlog needs to be processed before search alerts can be turned back on.</p><p><small>Dec <var data-var='date'> 5</var>, <var data-var='time'>00:23</var> PST</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p>tag:www.papertrailstatus.com,2005:Incident/191549562023-11-16T20:00:00-08:002023-11-17T13:21:56-08:00Bad network interface on logs7.papertrailapp.com (169.46.82.187)<p><small>Nov <var data-var='date'>16</var>, <var data-var='time'>20:00</var> PST</small><br><strong>Resolved</strong> - Between 04:05 and 09:40 UTC on November 17, 2023, the IP address 169.46.82.187, associated with one of the backend servers for logs7.papertrailapp.com, encountered a hardware network interface issue. <br /><br />The other 3 IP address that serves logs7.papertrailapp.com remained operational.<br /><br />Customer impact for users that happen to use the 169.46.82.187 IP address for logs7.papertrailapp.com:<br /><br />1. Port-Based Log Destinations and UDP users<br />Logs would have been dropped.<br /><br />2. TCP users<br />Logs would have experienced a 6-hour delay, then sent all at once around 09:35-40 UTC. This delay will lead to incorrect timestamps in the logs, reflecting the time of arrival rather than the time of generation.<br /><br />Importantly, there is no impact on Heroku customers and those utilizing logs7.papertrailapp.com as an HTTP log destination.</p>tag:www.papertrailstatus.com,2005:Incident/184454822023-09-09T00:00:00-07:002023-09-09T02:06:10-07:00Partial delayed log processing from 06:50 UTC to 07:12 UTC<p><small>Sep <var data-var='date'> 9</var>, <var data-var='time'>00:00</var> PDT</small><br><strong>Resolved</strong> - Starting at 06:50 UTC some logs were delayed. Log processing caught back up by 07:12 UTC.</p>tag:www.papertrailstatus.com,2005:Incident/181796782023-08-17T19:26:07-07:002023-08-17T19:26:07-07:00Search Alerts are Delayed and Increased Errors<p><small>Aug <var data-var='date'>17</var>, <var data-var='time'>19:26</var> PDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Aug <var data-var='date'>17</var>, <var data-var='time'>18:52</var> PDT</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Aug <var data-var='date'>17</var>, <var data-var='time'>18:22</var> PDT</small><br><strong>Update</strong> - We are continuing to investigate this issue.</p><p><small>Aug <var data-var='date'>17</var>, <var data-var='time'>18:16</var> PDT</small><br><strong>Investigating</strong> - Starting at 12:30 UTC search alerts are delayed.</p>tag:www.papertrailstatus.com,2005:Incident/178731392023-07-16T19:09:23-07:002023-07-16T19:09:23-07:00Search Alerts and Log Search delayed<p><small>Jul <var data-var='date'>16</var>, <var data-var='time'>19:09</var> PDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jul <var data-var='date'>16</var>, <var data-var='time'>18:20</var> PDT</small><br><strong>Monitoring</strong> - Starting at 12:16 UTC Search Alerts and Log Search are delayed.</p>tag:www.papertrailstatus.com,2005:Incident/178530192023-07-14T04:31:29-07:002023-07-14T04:31:29-07:00Updates to or new port based Log Destinations are delayed<p><small>Jul <var data-var='date'>14</var>, <var data-var='time'>04:31</var> PDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jul <var data-var='date'>14</var>, <var data-var='time'>03:36</var> PDT</small><br><strong>Identified</strong> - The issue has been identified and a fix is being implemented.</p>tag:www.papertrailstatus.com,2005:Incident/177952392023-07-07T18:31:14-07:002023-07-07T18:31:14-07:00Delayed log search and search alerts from 23:20 UTC on Friday to 01:20 UTC on Saturday<p><small>Jul <var data-var='date'> 7</var>, <var data-var='time'>18:31</var> PDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jul <var data-var='date'> 7</var>, <var data-var='time'>17:42</var> PDT</small><br><strong>Monitoring</strong> - Starting at 23:20 UTC on Friday July 7th log search and search alerts are delayed for some customers.</p>tag:www.papertrailstatus.com,2005:Incident/177406202023-07-01T15:00:00-07:002023-07-01T16:11:28-07:00Log Search and Search Alerts delayed from 22:04 UTC to 22:29 UTC<p><small>Jul <var data-var='date'> 1</var>, <var data-var='time'>15:00</var> PDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p>tag:www.papertrailstatus.com,2005:Incident/176679772023-06-28T19:01:13-07:002023-06-28T19:01:13-07:00Scheduled Maintenance<p><small>Jun <var data-var='date'>28</var>, <var data-var='time'>19:01</var> PDT</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Jun <var data-var='date'>28</var>, <var data-var='time'>18:00</var> PDT</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Jun <var data-var='date'>23</var>, <var data-var='time'>14:37</var> PDT</small><br><strong>Scheduled</strong> - On Wednesday 28th between 18:00 PDT and 19:00 PDT we will be performing maintenance. During this timeframe, logins and invitations may intermittently fail. If this happens a retry should resolve the issue.</p>tag:www.papertrailstatus.com,2005:Incident/168102592023-04-10T07:56:16-07:002023-04-10T07:56:16-07:00Issue with sending mail for invites and password reset<p><small>Apr <var data-var='date'>10</var>, <var data-var='time'>07:56</var> PDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Apr <var data-var='date'>10</var>, <var data-var='time'>07:54</var> PDT</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p>tag:www.papertrailstatus.com,2005:Incident/167934252023-04-08T12:30:00-07:002023-04-08T13:36:30-07:00Log collection to part of one endpoint interrupted<p><small>Apr <var data-var='date'> 8</var>, <var data-var='time'>12:30</var> PDT</small><br><strong>Resolved</strong> - Logs sent to logs7.papertrailapp.com may have been delayed if they were sent between 19:37 UTC and 20:00 UTC.</p>tag:www.papertrailstatus.com,2005:Incident/162177032023-02-21T01:22:10-08:002023-02-21T10:06:28-08:00Delayed log search and search alerts<p><small>Feb <var data-var='date'>21</var>, <var data-var='time'>01:22</var> PST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Feb <var data-var='date'>21</var>, <var data-var='time'>00:26</var> PST</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p>tag:www.papertrailstatus.com,2005:Incident/160766552023-02-09T09:24:24-08:002023-02-09T09:24:24-08:00Search Alerts Issues<p><small>Feb <var data-var='date'> 9</var>, <var data-var='time'>09:24</var> PST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Feb <var data-var='date'> 9</var>, <var data-var='time'>07:38</var> PST</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Feb <var data-var='date'> 9</var>, <var data-var='time'>06:37</var> PST</small><br><strong>Identified</strong> - The issue has been identified and a fix has been implemented.</p><p><small>Feb <var data-var='date'> 9</var>, <var data-var='time'>05:57</var> PST</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p>tag:www.papertrailstatus.com,2005:Incident/159352232023-02-08T10:39:24-08:002023-02-08T10:39:24-08:00Planned maintenance<p><small>Feb <var data-var='date'> 8</var>, <var data-var='time'>10:39</var> PST</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Feb <var data-var='date'> 8</var>, <var data-var='time'>07:00</var> PST</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Jan <var data-var='date'>25</var>, <var data-var='time'>11:48</var> PST</small><br><strong>Scheduled</strong> - Log processing (Log Search and Search Alerts) will be delayed while planned maintenance is performed. The website may also be unavailable intermittently.</p>tag:www.papertrailstatus.com,2005:Incident/160275992023-02-04T14:16:08-08:002023-02-04T14:16:08-08:00Email Delivery Issue<p><small>Feb <var data-var='date'> 4</var>, <var data-var='time'>14:16</var> PST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Feb <var data-var='date'> 4</var>, <var data-var='time'>13:57</var> PST</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Feb <var data-var='date'> 4</var>, <var data-var='time'>12:24</var> PST</small><br><strong>Identified</strong> - We have identified an issue with our email provider. <br />All email communications (account signup, alerts with mail notification) are currently affected.</p><p><small>Feb <var data-var='date'> 4</var>, <var data-var='time'>11:07</var> PST</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p>tag:www.papertrailstatus.com,2005:Incident/159551832023-01-27T14:17:46-08:002023-01-27T14:17:46-08:00Search Alerts are Delayed<p><small>Jan <var data-var='date'>27</var>, <var data-var='time'>14:17</var> PST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jan <var data-var='date'>27</var>, <var data-var='time'>13:41</var> PST</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Jan <var data-var='date'>27</var>, <var data-var='time'>12:51</var> PST</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p>tag:www.papertrailstatus.com,2005:Incident/159352162023-01-26T13:19:35-08:002023-01-26T13:19:35-08:00Planned maintenance<p><small>Jan <var data-var='date'>26</var>, <var data-var='time'>13:19</var> PST</small><br><strong>Completed</strong> - The scheduled maintenance has been completed.</p><p><small>Jan <var data-var='date'>26</var>, <var data-var='time'>12:01</var> PST</small><br><strong>In progress</strong> - Scheduled maintenance is currently in progress. We will provide updates as necessary.</p><p><small>Jan <var data-var='date'>25</var>, <var data-var='time'>11:46</var> PST</small><br><strong>Scheduled</strong> - The website and log search may be unavailable intermittently while planned maintenance is performed.</p>tag:www.papertrailstatus.com,2005:Incident/157838992023-01-07T21:03:23-08:002023-01-13T12:33:38-08:00Archived logs delayed from 2023-01-06 at 9:05 AM to 2023-01-07 at 3:10 AM<p><small>Jan <var data-var='date'> 7</var>, <var data-var='time'>21:03</var> PST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jan <var data-var='date'> 6</var>, <var data-var='time'>11:30</var> PST</small><br><strong>Update</strong> - We are continuing to monitor as we process the backlog of logs sent to be archived.</p><p><small>Jan <var data-var='date'> 6</var>, <var data-var='time'>01:54</var> PST</small><br><strong>Monitoring</strong> - Log archives are delayed.</p>tag:www.papertrailstatus.com,2005:Incident/133273772022-11-17T21:56:08-08:002022-11-17T21:56:08-08:00Search Alerts Delayed<p><small>Nov <var data-var='date'>17</var>, <var data-var='time'>21:56</var> PST</small><br><strong>Resolved</strong> - From 04:50 to 05:50 some search alerts were delayed.</p><p><small>Nov <var data-var='date'>17</var>, <var data-var='time'>21:48</var> PST</small><br><strong>Monitoring</strong> - Since 04:50 UTC Search Alerts have been delayed.</p>tag:www.papertrailstatus.com,2005:Incident/114287792022-10-04T13:30:30-07:002022-10-04T13:30:30-07:00Delayed log search and search alerts<p><small>Oct <var data-var='date'> 4</var>, <var data-var='time'>13:30</var> PDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Oct <var data-var='date'> 4</var>, <var data-var='time'>13:09</var> PDT</small><br><strong>Monitoring</strong> - Logs are delayed by 6 minutes or less. We are processing the backlog.</p>tag:www.papertrailstatus.com,2005:Incident/112869852022-09-27T08:07:01-07:002022-09-27T08:07:01-07:00Accidental status set on Statuspage for "Search Alerts"<p><small>Sep <var data-var='date'>27</var>, <var data-var='time'>08:07</var> PDT</small><br><strong>Resolved</strong> - An accidental status for "Search Alerts" was at 8AM UTC and was resolved 7 hours later.</p>tag:www.papertrailstatus.com,2005:Incident/111839492022-09-19T14:17:35-07:002022-09-19T14:17:35-07:00Live Tail, Log Search and Search Alerts are delayed.<p><small>Sep <var data-var='date'>19</var>, <var data-var='time'>14:17</var> PDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Sep <var data-var='date'>19</var>, <var data-var='time'>14:13</var> PDT</small><br><strong>Monitoring</strong> - The backlog has been processed.</p><p><small>Sep <var data-var='date'>19</var>, <var data-var='time'>13:53</var> PDT</small><br><strong>Investigating</strong> - Log processing is delayed. The Live Tail, Log Search and Search Alerts features will have delays.</p>tag:www.papertrailstatus.com,2005:Incident/111493022022-09-16T03:24:34-07:002022-09-16T03:24:34-07:00Transient failures when sending emails since 04:30 UTC 16Sep2022<p><small>Sep <var data-var='date'>16</var>, <var data-var='time'>03:24</var> PDT</small><br><strong>Resolved</strong> - As of 09:00 UTC delivery of notifications from search alerts returned to normal.</p><p><small>Sep <var data-var='date'>16</var>, <var data-var='time'>00:00</var> PDT</small><br><strong>Identified</strong> - Not all notifications sent by search alerts are being delivered. All notifications types are impacted, not just email.</p><p><small>Sep <var data-var='date'>15</var>, <var data-var='time'>23:30</var> PDT</small><br><strong>Monitoring</strong> - Starting roughly 2 hours ago not all emails sent by search alerts are being delivered. Our email provider has not yet provided an ETA to restore normal service. https://status.mailgun.com/incidents/m85fxdtc7tvs</p>