tag:www.papertrailstatus.com,2005:/historyPapertrail Status - Incident History2025-07-12T21:20:53-07:00Papertrailtag:www.papertrailstatus.com,2005:Incident/257517862025-07-09T16:54:06-07:002025-07-09T16:54:06-07:00Some logs delivered from the Heroku AddOn may not arrive.<p><small>Jul <var data-var='date'> 9</var>, <var data-var='time'>16:54</var> PDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jul <var data-var='date'> 9</var>, <var data-var='time'>16:31</var> PDT</small><br><strong>Identified</strong> - The issue has been identified and a fix is being implemented.</p>tag:www.papertrailstatus.com,2005:Incident/257163422025-07-07T03:44:32-07:002025-07-07T03:44:32-07:00Delayed log ingestion and alerts<p><small>Jul <var data-var='date'> 7</var>, <var data-var='time'>03:44</var> PDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jul <var data-var='date'> 7</var>, <var data-var='time'>01:36</var> PDT</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p>tag:www.papertrailstatus.com,2005:Incident/254234472025-06-10T18:42:29-07:002025-06-10T18:42:29-07:00Logs from some of the applications on Heroku platform are not received in Papertrail<p><small>Jun <var data-var='date'>10</var>, <var data-var='time'>18:42</var> PDT</small><br><strong>Resolved</strong> - Log delivery from Heroku to Papertrail has returned to normal. If it appears that Heroku isn't sending logs to your Papertrail account please follow the instructions at https://status.heroku.com/incidents/2822 and/or contact Heroku support.</p><p><small>Jun <var data-var='date'>10</var>, <var data-var='time'>11:55</var> PDT</small><br><strong>Update</strong> - The Heroku incident is not yet resolved but log delivery from Heroku to Papertrail is nearly back to normal.</p><p><small>Jun <var data-var='date'>10</var>, <var data-var='time'>01:28</var> PDT</small><br><strong>Monitoring</strong> - We are observing an issue with log delivery from Heroku to Papertrail due to an ongoing Heroku incident. This may affect a subset of customers using the Heroku plug-in to send logs to Papertrail. More details on the Heroku incident are available at https://status.heroku.com/incidents/2822.</p>tag:www.papertrailstatus.com,2005:Incident/248954002025-05-06T03:26:56-07:002025-05-06T03:26:56-07:00Alert Delivery<p><small>May <var data-var='date'> 6</var>, <var data-var='time'>03:26</var> PDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>May <var data-var='date'> 6</var>, <var data-var='time'>03:26</var> PDT</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>May <var data-var='date'> 6</var>, <var data-var='time'>03:09</var> PDT</small><br><strong>Identified</strong> - The issue has been identified and a fix is being implemented.</p><p><small>May <var data-var='date'> 6</var>, <var data-var='time'>02:20</var> PDT</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p>tag:www.papertrailstatus.com,2005:Incident/243847262025-03-18T16:04:57-07:002025-03-18T16:04:57-07:00Delays in log ingestion<p><small>Mar <var data-var='date'>18</var>, <var data-var='time'>16:04</var> PDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Mar <var data-var='date'>18</var>, <var data-var='time'>15:50</var> PDT</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Mar <var data-var='date'>18</var>, <var data-var='time'>15:12</var> PDT</small><br><strong>Identified</strong> - The issue has been identified and a fix is being implemented.</p><p><small>Mar <var data-var='date'>18</var>, <var data-var='time'>13:41</var> PDT</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p>tag:www.papertrailstatus.com,2005:Incident/241757542025-02-27T07:26:43-08:002025-02-27T07:26:43-08:00Delay in log ingestion<p><small>Feb <var data-var='date'>27</var>, <var data-var='time'>07:26</var> PST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Feb <var data-var='date'>27</var>, <var data-var='time'>07:26</var> PST</small><br><strong>Update</strong> - We are continuing to monitor for any further issues.</p><p><small>Feb <var data-var='date'>27</var>, <var data-var='time'>07:17</var> PST</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Feb <var data-var='date'>27</var>, <var data-var='time'>07:17</var> PST</small><br><strong>Identified</strong> - The issue has been identified and a fix is being implemented.</p><p><small>Feb <var data-var='date'>27</var>, <var data-var='time'>06:45</var> PST</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p>tag:www.papertrailstatus.com,2005:Incident/236875692025-01-24T13:30:00-08:002025-01-24T15:33:59-08:00Some Search Alerts searched partial data.<p><small>Jan <var data-var='date'>24</var>, <var data-var='time'>13:30</var> PST</small><br><strong>Resolved</strong> - From 21:49 to 22:43 on 2025-01-24 UTC some Search Alerts searched partial data</p>tag:www.papertrailstatus.com,2005:Incident/230743372024-12-04T19:04:19-08:002024-12-04T19:04:19-08:00Delay in log ingestion<p><small>Dec <var data-var='date'> 4</var>, <var data-var='time'>19:04</var> PST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Dec <var data-var='date'> 4</var>, <var data-var='time'>18:49</var> PST</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Dec <var data-var='date'> 4</var>, <var data-var='time'>18:05</var> PST</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p>tag:www.papertrailstatus.com,2005:Incident/228091482024-11-08T18:09:41-08:002024-11-08T18:09:41-08:00Delayed log ingestion<p><small>Nov <var data-var='date'> 8</var>, <var data-var='time'>18:09</var> PST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Nov <var data-var='date'> 8</var>, <var data-var='time'>17:58</var> PST</small><br><strong>Identified</strong> - The issue has been identified and a fix is being implemented.</p><p><small>Nov <var data-var='date'> 8</var>, <var data-var='time'>17:42</var> PST</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p>tag:www.papertrailstatus.com,2005:Incident/226194142024-10-21T07:44:24-07:002024-10-21T07:44:24-07:00papertrailapp.com is unavailable<p><small>Oct <var data-var='date'>21</var>, <var data-var='time'>07:44</var> PDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Oct <var data-var='date'>21</var>, <var data-var='time'>07:33</var> PDT</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Oct <var data-var='date'>21</var>, <var data-var='time'>07:30</var> PDT</small><br><strong>Identified</strong> - The issue has been identified and a fix is being implemented.</p><p><small>Oct <var data-var='date'>21</var>, <var data-var='time'>06:46</var> PDT</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p>tag:www.papertrailstatus.com,2005:Incident/225687922024-10-16T14:02:58-07:002024-10-16T14:02:58-07:00Log ingestion delayed<p><small>Oct <var data-var='date'>16</var>, <var data-var='time'>14:02</var> PDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Oct <var data-var='date'>16</var>, <var data-var='time'>13:49</var> PDT</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Oct <var data-var='date'>16</var>, <var data-var='time'>13:17</var> PDT</small><br><strong>Identified</strong> - The issue has been identified and a fix is being implemented.</p><p><small>Oct <var data-var='date'>16</var>, <var data-var='time'>12:37</var> PDT</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p>tag:www.papertrailstatus.com,2005:Incident/218291982024-08-20T16:17:02-07:002024-08-20T16:17:02-07:00Connections to web site delayed or timing out<p><small>Aug <var data-var='date'>20</var>, <var data-var='time'>16:17</var> PDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Aug <var data-var='date'>20</var>, <var data-var='time'>15:55</var> PDT</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Aug <var data-var='date'>20</var>, <var data-var='time'>15:49</var> PDT</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p>tag:www.papertrailstatus.com,2005:Incident/217649222024-08-16T07:54:48-07:002024-08-16T07:54:48-07:00Logs sent to logs.papertrailapp.com are delayed<p><small>Aug <var data-var='date'>16</var>, <var data-var='time'>07:54</var> PDT</small><br><strong>Resolved</strong> - From 14:24 UTC to 14:45 UTC new connections to logs.papertrailapp.com were delayed.</p><p><small>Aug <var data-var='date'>16</var>, <var data-var='time'>07:42</var> PDT</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p>tag:www.papertrailstatus.com,2005:Incident/214207702024-07-15T07:00:00-07:002024-07-15T09:36:54-07:00Some log destinations on logs7.papertrailapp.com unavailable<p><small>Jul <var data-var='date'>15</var>, <var data-var='time'>07:00</var> PDT</small><br><strong>Resolved</strong> - On 2025-07-15 from 14:10 to 14:22 UTC some log destinations on logs7.papertrailapp.com were unavailable.</p>tag:www.papertrailstatus.com,2005:Incident/203077612024-03-20T00:31:20-07:002024-03-20T00:31:20-07:00Log Collection is delayed<p><small>Mar <var data-var='date'>20</var>, <var data-var='time'>00:31</var> PDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Mar <var data-var='date'>20</var>, <var data-var='time'>00:05</var> PDT</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p>tag:www.papertrailstatus.com,2005:Incident/200635722024-02-23T23:46:30-08:002024-02-23T23:46:30-08:00Search<p><small>Feb <var data-var='date'>23</var>, <var data-var='time'>23:46</var> PST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Feb <var data-var='date'>23</var>, <var data-var='time'>22:42</var> PST</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p>tag:www.papertrailstatus.com,2005:Incident/199846622024-02-14T08:20:38-08:002024-02-14T08:20:38-08:00papertrailapp.com is unavailable<p><small>Feb <var data-var='date'>14</var>, <var data-var='time'>08:20</var> PST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Feb <var data-var='date'>14</var>, <var data-var='time'>08:07</var> PST</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Feb <var data-var='date'>14</var>, <var data-var='time'>07:45</var> PST</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p>tag:www.papertrailstatus.com,2005:Incident/193293992023-12-05T04:01:26-08:002023-12-05T04:01:51-08:00Search Alerts Delayed<p><small>Dec <var data-var='date'> 5</var>, <var data-var='time'>04:01</var> PST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Dec <var data-var='date'> 5</var>, <var data-var='time'>03:42</var> PST</small><br><strong>Update</strong> - The backlog has been processed. Search Alerts were enabled at 11:38 UTC. Some search alerts may fire initially due to the search window extending back to the start of this incident rather than the normal window (1 minute, 10 minutes, 1hr, etc.) but they will return to normal operation after that initial search with the larger window.</p><p><small>Dec <var data-var='date'> 5</var>, <var data-var='time'>02:50</var> PST</small><br><strong>Update</strong> - We continue to process the backlog and once that is complete we will enable Search Alerts.</p><p><small>Dec <var data-var='date'> 5</var>, <var data-var='time'>01:38</var> PST</small><br><strong>Monitoring</strong> - The logs visible in the viewer are up to date. More of the backlog needs to be processed before search alerts can be turned back on.</p><p><small>Dec <var data-var='date'> 5</var>, <var data-var='time'>00:23</var> PST</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p>tag:www.papertrailstatus.com,2005:Incident/191549562023-11-16T20:00:00-08:002023-11-17T13:21:56-08:00Bad network interface on logs7.papertrailapp.com (169.46.82.187)<p><small>Nov <var data-var='date'>16</var>, <var data-var='time'>20:00</var> PST</small><br><strong>Resolved</strong> - Between 04:05 and 09:40 UTC on November 17, 2023, the IP address 169.46.82.187, associated with one of the backend servers for logs7.papertrailapp.com, encountered a hardware network interface issue. <br /><br />The other 3 IP address that serves logs7.papertrailapp.com remained operational.<br /><br />Customer impact for users that happen to use the 169.46.82.187 IP address for logs7.papertrailapp.com:<br /><br />1. Port-Based Log Destinations and UDP users<br />Logs would have been dropped.<br /><br />2. TCP users<br />Logs would have experienced a 6-hour delay, then sent all at once around 09:35-40 UTC. This delay will lead to incorrect timestamps in the logs, reflecting the time of arrival rather than the time of generation.<br /><br />Importantly, there is no impact on Heroku customers and those utilizing logs7.papertrailapp.com as an HTTP log destination.</p>tag:www.papertrailstatus.com,2005:Incident/184454822023-09-09T00:00:00-07:002023-09-09T02:06:10-07:00Partial delayed log processing from 06:50 UTC to 07:12 UTC<p><small>Sep <var data-var='date'> 9</var>, <var data-var='time'>00:00</var> PDT</small><br><strong>Resolved</strong> - Starting at 06:50 UTC some logs were delayed. Log processing caught back up by 07:12 UTC.</p>tag:www.papertrailstatus.com,2005:Incident/181796782023-08-17T19:26:07-07:002023-08-17T19:26:07-07:00Search Alerts are Delayed and Increased Errors<p><small>Aug <var data-var='date'>17</var>, <var data-var='time'>19:26</var> PDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Aug <var data-var='date'>17</var>, <var data-var='time'>18:52</var> PDT</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Aug <var data-var='date'>17</var>, <var data-var='time'>18:22</var> PDT</small><br><strong>Update</strong> - We are continuing to investigate this issue.</p><p><small>Aug <var data-var='date'>17</var>, <var data-var='time'>18:16</var> PDT</small><br><strong>Investigating</strong> - Starting at 12:30 UTC search alerts are delayed.</p>tag:www.papertrailstatus.com,2005:Incident/178731392023-07-16T19:09:23-07:002023-07-16T19:09:23-07:00Search Alerts and Log Search delayed<p><small>Jul <var data-var='date'>16</var>, <var data-var='time'>19:09</var> PDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jul <var data-var='date'>16</var>, <var data-var='time'>18:20</var> PDT</small><br><strong>Monitoring</strong> - Starting at 12:16 UTC Search Alerts and Log Search are delayed.</p>tag:www.papertrailstatus.com,2005:Incident/178530192023-07-14T04:31:29-07:002023-07-14T04:31:29-07:00Updates to or new port based Log Destinations are delayed<p><small>Jul <var data-var='date'>14</var>, <var data-var='time'>04:31</var> PDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jul <var data-var='date'>14</var>, <var data-var='time'>03:36</var> PDT</small><br><strong>Identified</strong> - The issue has been identified and a fix is being implemented.</p>tag:www.papertrailstatus.com,2005:Incident/177952392023-07-07T18:31:14-07:002023-07-07T18:31:14-07:00Delayed log search and search alerts from 23:20 UTC on Friday to 01:20 UTC on Saturday<p><small>Jul <var data-var='date'> 7</var>, <var data-var='time'>18:31</var> PDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jul <var data-var='date'> 7</var>, <var data-var='time'>17:42</var> PDT</small><br><strong>Monitoring</strong> - Starting at 23:20 UTC on Friday July 7th log search and search alerts are delayed for some customers.</p>tag:www.papertrailstatus.com,2005:Incident/177406202023-07-01T15:00:00-07:002023-07-01T16:11:28-07:00Log Search and Search Alerts delayed from 22:04 UTC to 22:29 UTC<p><small>Jul <var data-var='date'> 1</var>, <var data-var='time'>15:00</var> PDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p>