tag:status.calltrackingmetrics.com,2005:/historyCallTrackingMetrics Status - Incident History2024-03-28T07:00:30-04:00CallTrackingMetricstag:status.calltrackingmetrics.com,2005:Incident/202698582024-03-15T11:40:36-04:002024-03-15T11:40:36-04:00Elevated 500's with Reports<p><small>Mar <var data-var='date'>15</var>, <var data-var='time'>11:40</var> EDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Mar <var data-var='date'>15</var>, <var data-var='time'>11:16</var> EDT</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Mar <var data-var='date'>15</var>, <var data-var='time'>11:13</var> EDT</small><br><strong>Identified</strong> - The issue has been identified and a fix is being implemented.</p><p><small>Mar <var data-var='date'>15</var>, <var data-var='time'>11:10</var> EDT</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p>tag:status.calltrackingmetrics.com,2005:Incident/201794862024-03-07T09:24:03-05:002024-03-07T09:24:03-05:00Investigating Potential Issue with Delayed Transcriptions<p><small>Mar <var data-var='date'> 7</var>, <var data-var='time'>09:24</var> EST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Mar <var data-var='date'> 6</var>, <var data-var='time'>16:13</var> EST</small><br><strong>Investigating</strong> - We are aware of intermittent errors occurring with transcriptions. Your patience is appreciated while we look into the cause. We do not anticipate any loss of data; however the completion of the transcriptions might be delayed.</p>tag:status.calltrackingmetrics.com,2005:Incident/199162272024-02-05T18:50:45-05:002024-02-05T18:50:45-05:00An issue with some CTM tracking numbers exists.<p><small>Feb <var data-var='date'> 5</var>, <var data-var='time'>18:50</var> EST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Feb <var data-var='date'> 5</var>, <var data-var='time'>18:36</var> EST</small><br><strong>Monitoring</strong> - Our technical partner has identified the cause and deployed a repair. Normal operation of voice calls is being reported. We will continue to monitor our systems.</p><p><small>Feb <var data-var='date'> 5</var>, <var data-var='time'>16:25</var> EST</small><br><strong>Investigating</strong> - We are aware of issues with a small subset of CTM tracking numbers in which a caller receives a busy signal and cannot connect. This cause of this issue is with systems that neither CTM nor our Direct Technical Partners control. We ask for your patience while this issue on the broader telecom network is being addressed. We will update via this channel as we receive new information.</p>tag:status.calltrackingmetrics.com,2005:Incident/197953392024-01-22T10:54:59-05:002024-01-22T10:54:59-05:00Investigating Potential Issue affecting Logins and Access to Features<p><small>Jan <var data-var='date'>22</var>, <var data-var='time'>10:54</var> EST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jan <var data-var='date'>22</var>, <var data-var='time'>10:42</var> EST</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Jan <var data-var='date'>22</var>, <var data-var='time'>10:39</var> EST</small><br><strong>Identified</strong> - The issue has been identified and a fix is being implemented.</p><p><small>Jan <var data-var='date'>22</var>, <var data-var='time'>10:27</var> EST</small><br><strong>Investigating</strong> - We are aware of intermittent errors occurring that affect login and access to features. Our incident response team is investigating.</p>tag:status.calltrackingmetrics.com,2005:Incident/194027462023-12-11T14:22:06-05:002023-12-11T14:22:06-05:00Issue with purchasing numbers<p><small>Dec <var data-var='date'>11</var>, <var data-var='time'>14:22</var> EST</small><br><strong>Resolved</strong> - Tracking numbers can be purchased using the inventory look up once again.</p><p><small>Dec <var data-var='date'>11</var>, <var data-var='time'>11:41</var> EST</small><br><strong>Identified</strong> - We are aware of issues affecting the ability to purchase numbers within our platform. We are working on a resolution that will return the ability to purchase numbers to you. In the interim, you may request a number by visiting https://app.calltrackingmetrics.com/tracking_numbers/buy_new.</p>tag:status.calltrackingmetrics.com,2005:Incident/192607762023-12-01T11:43:43-05:002023-12-01T11:43:43-05:00Investigating Issue with Transcriptions<p><small>Dec <var data-var='date'> 1</var>, <var data-var='time'>11:43</var> EST</small><br><strong>Resolved</strong> - The issues with transcriptions have been resolved. Please report any issues to our Technical Support team at https://www.calltrackingmetrics.com/support.</p><p><small>Dec <var data-var='date'> 1</var>, <var data-var='time'>09:32</var> EST</small><br><strong>Monitoring</strong> - A fix has been put in place and we are monitoring performance. Transcriptions that did not happen are being retried and should be populated over the course of today.</p><p><small>Dec <var data-var='date'> 1</var>, <var data-var='time'>08:51</var> EST</small><br><strong>Investigating</strong> - We are aware of intermittent errors occurring on the service. Our incident response team is working with our transcription provider to solve the issue as quickly as possible. Additional updates will be posted as information is received.</p>tag:status.calltrackingmetrics.com,2005:Incident/190580932023-11-08T10:49:42-05:002023-11-08T10:49:42-05:00Issue affecting AskAI<p><small>Nov <var data-var='date'> 8</var>, <var data-var='time'>10:49</var> EST</small><br><strong>Resolved</strong> - Services have been restored and operations are back to normal. Thank you for your patience.</p><p><small>Nov <var data-var='date'> 8</var>, <var data-var='time'>10:43</var> EST</small><br><strong>Monitoring</strong> - Our technology partner has identified the issue affecting AskAI and has implemented a fix. Normal performance should be recovering if it has not already done so. Please visit https://status.openai.com/ for progress.</p><p><small>Nov <var data-var='date'> 8</var>, <var data-var='time'>09:39</var> EST</small><br><strong>Investigating</strong> - We are aware of an issue that may affect AskAI within CTM. Our technology partner is experiencing an outage and is investigating the cause. Please visit https://status.openai.com/ for progress.</p>tag:status.calltrackingmetrics.com,2005:Incident/189821172023-10-31T18:15:22-04:002023-10-31T18:15:33-04:00Investigating Issue with our Transcription Service<p><small>Oct <var data-var='date'>31</var>, <var data-var='time'>18:15</var> EDT</small><br><strong>Resolved</strong> - We observed elevated errors for approximately 1 hour . A process is running to transcribe all impacted recordings and workflows</p><p><small>Oct <var data-var='date'>31</var>, <var data-var='time'>15:10</var> EDT</small><br><strong>Investigating</strong> - We are aware of problems with our transcription service. Our incident response team is investigating.</p>tag:status.calltrackingmetrics.com,2005:Incident/188155492023-10-16T13:07:30-04:002023-10-16T13:07:30-04:00Intermittent issues marking Smart Dialer calls as completed<p><small>Oct <var data-var='date'>16</var>, <var data-var='time'>13:07</var> EDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Oct <var data-var='date'>16</var>, <var data-var='time'>11:49</var> EDT</small><br><strong>Monitoring</strong> - Between 11:34 and 11:42 ET we experienced intermittent failures when marking Smart Dialer calls as completed. This was due to a networking issue with a caching server. The issue is resolved and we are monitoring.</p><p><small>Oct <var data-var='date'>16</var>, <var data-var='time'>11:47</var> EDT</small><br><strong>Identified</strong> - The issue has been identified and a fix has been implemented.</p>tag:status.calltrackingmetrics.com,2005:Incident/187003152023-10-11T14:57:58-04:002023-10-11T14:57:58-04:00Delays with recording availability<p><small>Oct <var data-var='date'>11</var>, <var data-var='time'>14:57</var> EDT</small><br><strong>Resolved</strong> - All Intelligence and Transcriptions that were recoverable have been recovered. Programmable Voice Recording and Intelligence services are operating normally at this time.</p><p><small>Oct <var data-var='date'> 5</var>, <var data-var='time'>15:56</var> EDT</small><br><strong>Monitoring</strong> - New Voice recordings are now operating normally; our upstream carrier is still investigating impact to recordings & transcriptions during the impact window.</p><p><small>Oct <var data-var='date'> 5</var>, <var data-var='time'>12:16</var> EDT</small><br><strong>Investigating</strong> - Our upstream provider has identified an issue causing some delays in recording availability. They are working to identify the cause.</p>tag:status.calltrackingmetrics.com,2005:Incident/185029702023-09-14T14:34:37-04:002023-09-14T14:34:37-04:00CallTrackingMetrics Marketing Site Outage<p><small>Sep <var data-var='date'>14</var>, <var data-var='time'>14:34</var> EDT</small><br><strong>Resolved</strong> - Our vendor has identified and resolved the network issue affecting our marketing site. Our CallTrackingMetrics platform was never affected.</p><p><small>Sep <var data-var='date'>14</var>, <var data-var='time'>13:39</var> EDT</small><br><strong>Identified</strong> - Our vendor is working on a networking issue. Our CallTrackingMetrics platform remains unaffected.<br /><br />To login you can go directly to https://app.calltrackingmetrics.com/</p><p><small>Sep <var data-var='date'>14</var>, <var data-var='time'>13:23</var> EDT</small><br><strong>Investigating</strong> - Our marketing site is experiencing an outage. Our CallTrackingMetrics platform is unaffected.<br /><br />To login you can go directly to https://app.calltrackingmetrics.com/</p>tag:status.calltrackingmetrics.com,2005:Incident/177543862023-07-03T14:12:14-04:002023-07-03T14:12:14-04:00Investigating Potential Issue with Elevated 502 errors.<p><small>Jul <var data-var='date'> 3</var>, <var data-var='time'>14:12</var> EDT</small><br><strong>Resolved</strong> - Services have been restored and operations are back to normal. Thank you for your patience.</p><p><small>Jul <var data-var='date'> 3</var>, <var data-var='time'>13:50</var> EDT</small><br><strong>Monitoring</strong> - Root cause has been identified and mitigation steps have been implemented. We are still working on a full resolution and teams are monitoring error rates as they continue to improve.</p><p><small>Jul <var data-var='date'> 3</var>, <var data-var='time'>13:33</var> EDT</small><br><strong>Investigating</strong> - We are aware of intermittent errors occurring on the service. Our incident response team is investigating.</p>tag:status.calltrackingmetrics.com,2005:Incident/177536332023-07-03T12:05:53-04:002023-07-03T12:05:53-04:00Investigating Potential Issue - Elevated 500 errors<p><small>Jul <var data-var='date'> 3</var>, <var data-var='time'>12:05</var> EDT</small><br><strong>Resolved</strong> - Services have been restored and operations are back to normal. Thank you for your patience.</p><p><small>Jul <var data-var='date'> 3</var>, <var data-var='time'>12:01</var> EDT</small><br><strong>Monitoring</strong> - Root cause has been identified and mitigation steps have been implemented. We are still working on a full resolution and teams are monitoring error rates as they continue to improve.</p><p><small>Jul <var data-var='date'> 3</var>, <var data-var='time'>11:40</var> EDT</small><br><strong>Investigating</strong> - We are aware of intermittent errors occurring on the service. Our incident response team is investigating.</p>tag:status.calltrackingmetrics.com,2005:Incident/176321932023-06-20T12:10:16-04:002023-06-20T12:10:16-04:00Issue with queued calls<p><small>Jun <var data-var='date'>20</var>, <var data-var='time'>12:10</var> EDT</small><br><strong>Resolved</strong> - Our technical partner has resolved the issue with the queueing of calls.</p><p><small>Jun <var data-var='date'>20</var>, <var data-var='time'>11:56</var> EDT</small><br><strong>Investigating</strong> - Our Technical Partner has informed us of an issue which is causing calls to terminate abruptly within 10 seconds when routed through a queue. "Our monitoring systems have detected a potential issue with dequeue calls. Our engineering team has been alerted and is actively investigating. We will update as soon as we have more information."</p>tag:status.calltrackingmetrics.com,2005:Incident/174229602023-05-30T13:35:41-04:002023-05-30T13:35:41-04:00Investigating Potential Issue<p><small>May <var data-var='date'>30</var>, <var data-var='time'>13:35</var> EDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>May <var data-var='date'>30</var>, <var data-var='time'>13:20</var> EDT</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring progress. We're currently processing jobs for reporting and events.</p><p><small>May <var data-var='date'>30</var>, <var data-var='time'>12:57</var> EDT</small><br><strong>Identified</strong> - The issue has been identified and a fix is being implemented.</p><p><small>May <var data-var='date'>30</var>, <var data-var='time'>12:44</var> EDT</small><br><strong>Investigating</strong> - We are aware of intermittent errors occurring on the service. Our incident response team is investigating.</p>tag:status.calltrackingmetrics.com,2005:Incident/172964062023-05-18T21:49:33-04:002023-05-18T21:49:33-04:00Browser Softphone<p><small>May <var data-var='date'>18</var>, <var data-var='time'>21:49</var> EDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>May <var data-var='date'>18</var>, <var data-var='time'>20:21</var> EDT</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>May <var data-var='date'>18</var>, <var data-var='time'>19:59</var> EDT</small><br><strong>Update</strong> - We are seeing partial recovery</p><p><small>May <var data-var='date'>18</var>, <var data-var='time'>19:46</var> EDT</small><br><strong>Identified</strong> - The issue has been identified and a fix is being implemented.</p><p><small>May <var data-var='date'>18</var>, <var data-var='time'>19:32</var> EDT</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p>tag:status.calltrackingmetrics.com,2005:Incident/169332012023-04-21T16:10:35-04:002023-04-21T16:10:35-04:00Investigating Potential Issue<p><small>Apr <var data-var='date'>21</var>, <var data-var='time'>16:10</var> EDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Apr <var data-var='date'>21</var>, <var data-var='time'>15:59</var> EDT</small><br><strong>Monitoring</strong> - We have identified and taken corrective action. We are monitoring performance.</p><p><small>Apr <var data-var='date'>21</var>, <var data-var='time'>15:55</var> EDT</small><br><strong>Investigating</strong> - We are aware of intermittent errors occurring on the service. Our incident response team is investigating.</p>tag:status.calltrackingmetrics.com,2005:Incident/167245932023-04-02T06:56:54-04:002023-04-02T06:56:54-04:00Elevated 502 Errors<p><small>Apr <var data-var='date'> 2</var>, <var data-var='time'>06:56</var> EDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Apr <var data-var='date'> 1</var>, <var data-var='time'>23:56</var> EDT</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Apr <var data-var='date'> 1</var>, <var data-var='time'>23:53</var> EDT</small><br><strong>Identified</strong> - The issue has been identified and a fix is being implemented.</p><p><small>Apr <var data-var='date'> 1</var>, <var data-var='time'>23:52</var> EDT</small><br><strong>Update</strong> - We are continuing to investigate this issue.</p><p><small>Apr <var data-var='date'> 1</var>, <var data-var='time'>23:30</var> EDT</small><br><strong>Update</strong> - We are continuing to investigate this issue.</p><p><small>Apr <var data-var='date'> 1</var>, <var data-var='time'>23:17</var> EDT</small><br><strong>Investigating</strong> - We are currently investigating this issue.</p>tag:status.calltrackingmetrics.com,2005:Incident/163247452023-03-02T12:00:39-05:002023-03-02T12:00:39-05:00Intermittent 502 errors occurring for white-labeled hosts<p><small>Mar <var data-var='date'> 2</var>, <var data-var='time'>12:00</var> EST</small><br><strong>Resolved</strong> - Services have been restored and operations are back to normal. Thank you for your patience.</p><p><small>Mar <var data-var='date'> 2</var>, <var data-var='time'>11:03</var> EST</small><br><strong>Update</strong> - This is affecting a small subset of users. We are still working on the issue and you may connect to https://app.calltrackngmetrics.com.</p><p><small>Mar <var data-var='date'> 2</var>, <var data-var='time'>10:41</var> EST</small><br><strong>Update</strong> - You may connect to our main site https://app.calltrackingmetrics.com without errors.</p><p><small>Mar <var data-var='date'> 2</var>, <var data-var='time'>10:21</var> EST</small><br><strong>Update</strong> - We are continuing to investigate this issue.</p><p><small>Mar <var data-var='date'> 2</var>, <var data-var='time'>10:19</var> EST</small><br><strong>Investigating</strong> - We are aware of intermittent 502 errors occurring for white-labeled accounts. Our incident response team is investigating.</p>tag:status.calltrackingmetrics.com,2005:Incident/163243632023-03-02T10:17:25-05:002023-03-02T10:17:25-05:00Intermittent issues searching for and purchasing phone numbers<p><small>Mar <var data-var='date'> 2</var>, <var data-var='time'>10:17</var> EST</small><br><strong>Resolved</strong> - Services have been restored and operations are back to normal. Thank you for your patience.</p><p><small>Mar <var data-var='date'> 2</var>, <var data-var='time'>10:04</var> EST</small><br><strong>Monitoring</strong> - Root cause has been identified and mitigation steps have been implemented. We are still working on a full resolution and teams are monitoring error rates as they continue to improve.</p><p><small>Mar <var data-var='date'> 2</var>, <var data-var='time'>09:42</var> EST</small><br><strong>Identified</strong> - The issue has been identified and a fix is being implemented.</p><p><small>Mar <var data-var='date'> 2</var>, <var data-var='time'>09:42</var> EST</small><br><strong>Update</strong> - We are continuing to investigate this issue.</p><p><small>Mar <var data-var='date'> 2</var>, <var data-var='time'>09:40</var> EST</small><br><strong>Investigating</strong> - We are aware of intermittent issues searching for and purchasing phone numbers. Our incident response team is investigating.</p>tag:status.calltrackingmetrics.com,2005:Incident/161017672023-02-11T10:15:14-05:002023-02-11T10:28:59-05:00Access to the CTM Service<p><small>Feb <var data-var='date'>11</var>, <var data-var='time'>10:15</var> EST</small><br><strong>Resolved</strong> - We experienced a brief period of elevated 500 errors between 10:04 and 10:12</p>tag:status.calltrackingmetrics.com,2005:Incident/160336772023-02-05T12:10:22-05:002023-02-05T12:12:33-05:00Elevated 500/502 errors<p><small>Feb <var data-var='date'> 5</var>, <var data-var='time'>12:10</var> EST</small><br><strong>Resolved</strong> - Starting at 11:36 and ending at 12:08 we experienced elevated 502 errors. This was due to a recovery process that was putting too much load on our databases related to the incident on Thursday. Our team is working on expanding our footprint to support increased load spikes like this now. The issue is resolved and we will not repeat this.</p>tag:status.calltrackingmetrics.com,2005:Incident/160093762023-02-02T19:41:54-05:002023-02-02T19:41:54-05:00Issues with connecting to CTM<p><small>Feb <var data-var='date'> 2</var>, <var data-var='time'>19:41</var> EST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Feb <var data-var='date'> 2</var>, <var data-var='time'>17:24</var> EST</small><br><strong>Update</strong> - Our system is currently stable. Activities that were received during the incident will eventually appear in the log. New activities will appear in real time. We will continue to monitor performance.</p><p><small>Feb <var data-var='date'> 2</var>, <var data-var='time'>16:36</var> EST</small><br><strong>Monitoring</strong> - We are seeing recovery and are monitoring performance. We will update again at 5:00 pm EST.</p><p><small>Feb <var data-var='date'> 2</var>, <var data-var='time'>16:36</var> EST</small><br><strong>Update</strong> - We are continuing to work on a fix for this issue.</p><p><small>Feb <var data-var='date'> 2</var>, <var data-var='time'>16:03</var> EST</small><br><strong>Identified</strong> - At the moment, our issues have returned. Our team is working on the resolution. The next scheduled update is at 4:30 pm EST.</p><p><small>Feb <var data-var='date'> 2</var>, <var data-var='time'>15:34</var> EST</small><br><strong>Update</strong> - Site performance has improved, we continue to monitor. Our next update will be at 4:30 pm EST.</p><p><small>Feb <var data-var='date'> 2</var>, <var data-var='time'>15:04</var> EST</small><br><strong>Update</strong> - We continue to monitor site performance. Thank you for your patience. Our next update will be at 3:30 pm EST.</p><p><small>Feb <var data-var='date'> 2</var>, <var data-var='time'>14:29</var> EST</small><br><strong>Update</strong> - We are seeing recovery and continue to monitor performance. We will update again at 3:00 pm EST.</p><p><small>Feb <var data-var='date'> 2</var>, <var data-var='time'>14:25</var> EST</small><br><strong>Monitoring</strong> - Site availability has been restored. Our team continues to monitor as a regional outage in AWS infrastructure caused a database failure.</p><p><small>Feb <var data-var='date'> 2</var>, <var data-var='time'>14:08</var> EST</small><br><strong>Update</strong> - We appreciate your patience as we continue to investigate the cause of the issue. We will update again at 2:30 pm EST.</p><p><small>Feb <var data-var='date'> 2</var>, <var data-var='time'>13:36</var> EST</small><br><strong>Update</strong> - We are continuing to investigate the issue.</p><p><small>Feb <var data-var='date'> 2</var>, <var data-var='time'>13:27</var> EST</small><br><strong>Investigating</strong> - We are currently investigating the issue.</p>tag:status.calltrackingmetrics.com,2005:Incident/159799482023-01-30T05:30:00-05:002023-01-30T20:19:04-05:00Google Call Extensions/Call Only Ads<p><small>Jan <var data-var='date'>30</var>, <var data-var='time'>05:30</var> EST</small><br><strong>Resolved</strong> - From 5:27pm EST to 8:04pm EST retrieval of Google Click data was limited. Our teams were alerted to the issue and worked on a solution. During this time call delivery was not impacted but retrieval of the additional Click data was not available. You may notice phone calls during this period missing attribution data.</p>tag:status.calltrackingmetrics.com,2005:Incident/158207382023-01-11T12:38:56-05:002023-01-11T12:42:29-05:00Users experienced 500 errors when using CallTrackingMetrics<p><small>Jan <var data-var='date'>11</var>, <var data-var='time'>12:38</var> EST</small><br><strong>Resolved</strong> - We received reports of 500 errors when trying to perform various functions within CTM. The incident began at 12:11 p.m and was resolved at 12:39 pm. All times Eastern.</p>