tag:status.txt2give.co,2005:/historyTxt2Give Status - Incident History2024-03-28T06:00:04-05:00Txt2Givetag:status.txt2give.co,2005:Incident/49757052020-08-30T11:56:14-05:002020-08-30T11:56:14-05:00Text Message Delivery Issues<p><small>Aug <var data-var='date'>30</var>, <var data-var='time'>11:56</var> CDT</small><br><strong>Resolved</strong> - Our messaging provider is seeing traffic restored and messages processing as expected. They are considering this incident resolved.</p><p><small>Aug <var data-var='date'>30</var>, <var data-var='time'>09:39</var> CDT</small><br><strong>Identified</strong> - As of Aug 30, 07:33 CDT, our messaging provider started investigating a text message delivery incident that was affecting the entire industry. They continue working closely with industry partners and vendors to bring about a resolution. All updates to this issue will be reported here as we get them.</p>tag:status.txt2give.co,2005:Incident/44028052020-06-19T16:27:16-05:002020-06-19T16:27:20-05:00Canadian Toll Free Text Message Outage<p><small>Jun <var data-var='date'>19</var>, <var data-var='time'>16:27</var> CDT</small><br><strong>Resolved</strong> - As of 12:20 PM Central Standard Time, this has been resolved.</p><p><small>Jun <var data-var='date'>18</var>, <var data-var='time'>16:49</var> CDT</small><br><strong>Investigating</strong> - Text messages seem to still not be routing to Canadian users from our Toll Free numbers. We are working with our provider to figure out the issue, and will keep you updated here.</p>tag:status.txt2give.co,2005:Incident/44014302020-06-18T15:34:20-05:002020-06-18T15:34:20-05:00Canadian Text Message Routing<p><small>Jun <var data-var='date'>18</var>, <var data-var='time'>15:34</var> CDT</small><br><strong>Resolved</strong> - We have resolved this issue. If you see any other issues please contact support.</p><p><small>Jun <var data-var='date'>18</var>, <var data-var='time'>14:41</var> CDT</small><br><strong>Update</strong> - This is currently only affecting accounts that have both a local and toll-free number. We've identified the issue and are working to resolve it.</p><p><small>Jun <var data-var='date'>18</var>, <var data-var='time'>13:24</var> CDT</small><br><strong>Identified</strong> - We are experiencing an outage for all of our Canadian phone numbers. We are working with our provider to resolve the issue. We will continue giving updates here as they become available.</p>tag:status.txt2give.co,2005:Incident/37107532020-03-22T11:59:42-05:002020-03-22T11:59:42-05:00Partial Data Loss<p><small>Mar <var data-var='date'>22</var>, <var data-var='time'>11:59</var> CDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Mar <var data-var='date'>22</var>, <var data-var='time'>11:59</var> CDT</small><br><strong>Update</strong> - We haven't seen or heard of any issues related to the data loss, so we are marking this issue as resolved.</p><p><small>Mar <var data-var='date'>19</var>, <var data-var='time'>15:39</var> CDT</small><br><strong>Monitoring</strong> - The data restore has completed and everything looks good. We will continue monitoring things for a few more hours before marking this incident resolved.<br /><br />Thank you for your patience while we restored the data and please let us know if you see any anomalies in your data.</p><p><small>Mar <var data-var='date'>19</var>, <var data-var='time'>14:00</var> CDT</small><br><strong>Update</strong> - We've tested the restore process and things look good. We will be taking the system down for approximately 5-10 minutes to run the restore process on the production server.<br /><br />Once it's back up we will look everything over to ensure things are as expected, and give another update then.</p><p><small>Mar <var data-var='date'>19</var>, <var data-var='time'>11:55</var> CDT</small><br><strong>Identified</strong> - This incident will be where you can stay up-to-date on the data recovery task at hand that we notified you about yesterday via email.<br /><br />We have retrieved a backup from our provider, and are working to backfill the data that was lost. We estimate this will take several more hours.<br /><br />In the meantime, all donation services continue to operate normally.</p>tag:status.txt2give.co,2005:Incident/36944742020-03-16T13:35:30-05:002020-03-16T13:35:30-05:00Server Infrastructure Issue<p><small>Mar <var data-var='date'>16</var>, <var data-var='time'>13:35</var> CDT</small><br><strong>Resolved</strong> - We have word from our provider that all issues have been resolved.</p><p><small>Mar <var data-var='date'>16</var>, <var data-var='time'>12:01</var> CDT</small><br><strong>Monitoring</strong> - Our infrastructure provider had an outage starting around 11:14 AM today. They quickly found and resolved the issue. They are monitoring the resolution to make sure there are no other remaining issues. We will give the all-clear once they do.</p>tag:status.txt2give.co,2005:Incident/32021622019-11-10T22:49:37-06:002019-11-10T22:49:37-06:00Database issue<p><small>Nov <var data-var='date'>10</var>, <var data-var='time'>22:49</var> CST</small><br><strong>Resolved</strong> - Around 6 PM central time, one of our databases that holds login session data reached its storage limit which resulted in 500 server errors.<br /><br />Normally we receive alerts for this kind of thing, but the alerts weren't being received properly by the on-call staff.<br /><br />The alerts were finally received around 10:30 PM central time, and the database issue was resolved.<br /><br />We will be looking into why the database filled up so quickly, which is abnormal, and why the alerts were not being received properly.<br /><br />We are very sorry for this issue and will do our best to put measures in place so it won't happen again.</p>tag:status.txt2give.co,2005:Incident/31654282019-10-31T11:10:54-05:002019-10-31T11:10:54-05:00Application Service Provider Down<p><small>Oct <var data-var='date'>31</var>, <var data-var='time'>11:10</var> CDT</small><br><strong>Resolved</strong> - Our provider has yet to update us on the status, but all indicators on our end are looking good, so we are calling this issue resolved. If we have any other issues from our provider we will create a new incident here.</p><p><small>Oct <var data-var='date'>31</var>, <var data-var='time'>10:32</var> CDT</small><br><strong>Monitoring</strong> - Our provider has not given the all-clear yet, but from our end things seem to be working again. We will give another status update once they have officially given the all-clear.</p><p><small>Oct <var data-var='date'>31</var>, <var data-var='time'>10:24</var> CDT</small><br><strong>Identified</strong> - The issue has been identified and a fix is being implemented.</p><p><small>Oct <var data-var='date'>31</var>, <var data-var='time'>10:23</var> CDT</small><br><strong>Investigating</strong> - Our application service provider is currently experiencing issues. This is affecting all of our web services, including our donation engine and dashboard.<br /><br />We will update here once the issue has been resolved on their end.</p>tag:status.txt2give.co,2005:Incident/30695082019-10-04T20:58:16-05:002019-10-04T20:58:16-05:00Some giving numbers are unresponsive<p><small>Oct <var data-var='date'> 4</var>, <var data-var='time'>20:58</var> CDT</small><br><strong>Resolved</strong> - The carrier found the issue and has fixed the issue. This incident has been resolved.</p><p><small>Oct <var data-var='date'> 4</var>, <var data-var='time'>16:35</var> CDT</small><br><strong>Identified</strong> - A few of our customers have reported that their giving numbers are unresponsive. We've identified that the problem lies with our carrier, and are working with them to resolve this issue.</p>tag:status.txt2give.co,2005:Incident/30682342019-10-04T11:05:31-05:002019-10-04T11:05:31-05:00Connection timeouts<p><small>Oct <var data-var='date'> 4</var>, <var data-var='time'>11:05</var> CDT</small><br><strong>Resolved</strong> - We've figured out the cause for the timeouts and have fixed it. We'll be keeping an eye on this to make sure we don't see any more issues.</p><p><small>Oct <var data-var='date'> 4</var>, <var data-var='time'>09:22</var> CDT</small><br><strong>Investigating</strong> - We're currently seeing connection timeouts across our platform. We're currently investigating this and will update you shortly.</p>tag:status.txt2give.co,2005:Incident/30581922019-10-01T17:16:09-05:002019-10-01T17:16:09-05:00No reply from some inbound text messages<p><small>Oct <var data-var='date'> 1</var>, <var data-var='time'>17:16</var> CDT</small><br><strong>Resolved</strong> - This incident has been resolved and all giving numbers are working as expected.</p><p><small>Oct <var data-var='date'> 1</var>, <var data-var='time'>16:38</var> CDT</small><br><strong>Update</strong> - We've identified the issue and are working to get this resolved.</p><p><small>Oct <var data-var='date'> 1</var>, <var data-var='time'>16:38</var> CDT</small><br><strong>Identified</strong> - Some of our giving numbers aren't responding to inbound text messages. We are looking into the issue with our provider and will provide an update as soon as we can.</p>tag:status.txt2give.co,2005:Incident/29808392019-09-10T09:21:45-05:002019-09-10T09:21:45-05:00PayPal Payments<p><small>Sep <var data-var='date'>10</var>, <var data-var='time'>09:21</var> CDT</small><br><strong>Resolved</strong> - The PayPal errors have seemed to subsided, so we're marking this incident resolved. If you're using PayPal as your payment gateway and still see issues, please contact support.</p><p><small>Sep <var data-var='date'> 9</var>, <var data-var='time'>15:03</var> CDT</small><br><strong>Investigating</strong> - We are seeing some PayPal payment issues on our platform that seem to be originating from PayPal's side. We are investigating and will keep you updated here.</p>tag:status.txt2give.co,2005:Incident/24254042019-05-20T15:58:59-05:002019-05-20T15:58:59-05:00Online Donations Issue<p><small>May <var data-var='date'>20</var>, <var data-var='time'>15:58</var> CDT</small><br><strong>Resolved</strong> - From 10:46 PM CST on 5/19 to 2:24 PM CST on 5/20, there was a bug preventing online donations from going through. The issue has been resolved and all online donations should now have no issues.</p>tag:status.txt2give.co,2005:Incident/20950622019-05-20T15:56:45-05:002019-05-20T15:56:45-05:00TouchPoint Sync Issues<p><small>May <var data-var='date'>20</var>, <var data-var='time'>15:56</var> CDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Apr <var data-var='date'> 9</var>, <var data-var='time'>08:00</var> CDT</small><br><strong>Update</strong> - We are continuing to monitor for any further issues.</p><p><small>Dec <var data-var='date'>13</var>, <var data-var='time'>12:13</var> CST</small><br><strong>Monitoring</strong> - TouchPoint has informed us that as of 9:46 AM Central Time, the issue has been resolved. We will continue monitoring donations to make sure they are pushed to TouchPoint successfully.</p><p><small>Dec <var data-var='date'>12</var>, <var data-var='time'>10:35</var> CST</small><br><strong>Identified</strong> - TouchPoint syncing is currently not working. We've contacted TouchPoint and there is a problem with their API. Their team is working on resolving the issue. We'll post updates as we get them from TouchPoint.</p>tag:status.txt2give.co,2005:Incident/19870962018-10-22T21:08:54-05:002018-10-22T21:08:54-05:00Text Message Provider Delays<p><small>Oct <var data-var='date'>22</var>, <var data-var='time'>21:08</var> CDT</small><br><strong>Resolved</strong> - Our provider has indicated that their service is no longer experiencing delays and things are back to normal.</p><p><small>Oct <var data-var='date'>22</var>, <var data-var='time'>17:38</var> CDT</small><br><strong>Identified</strong> - Our text messaging provider has been experiencing text message delays today, averaging about 5 minutes. We will monitor their status and post updates here.</p>tag:status.txt2give.co,2005:Incident/19278912018-09-24T12:41:40-05:002018-09-24T12:41:40-05:00Incoming messages for new donors<p><small>Sep <var data-var='date'>24</var>, <var data-var='time'>12:41</var> CDT</small><br><strong>Resolved</strong> - This issue has been resolved.</p><p><small>Sep <var data-var='date'>24</var>, <var data-var='time'>10:31</var> CDT</small><br><strong>Monitoring</strong> - A fix has been implemented and everything looks good. We will continue monitoring for a bit before giving the all-clear.</p><p><small>Sep <var data-var='date'>24</var>, <var data-var='time'>10:11</var> CDT</small><br><strong>Identified</strong> - We've found an issue with text messages being processed for new donors. We've identified the issue and are working on a solution now.</p>tag:status.txt2give.co,2005:Incident/18690862018-08-30T14:20:51-05:002018-08-30T14:20:51-05:00Timeouts to payments provider<p><small>Aug <var data-var='date'>30</var>, <var data-var='time'>14:20</var> CDT</small><br><strong>Resolved</strong> - We haven't seen any other problems with timeouts in the past several days now, so we are resolving this issue.</p><p><small>Aug <var data-var='date'>23</var>, <var data-var='time'>13:40</var> CDT</small><br><strong>Monitoring</strong> - An alternate server cluster has been deployed by our payments provider that seems to not have the same networking issues as the standard servers. We've switched our code to point to this new cluster until our hosting provider can get the issue resolved. <br /><br />Our payments provider has told us that they aren't seeing any timeouts from this new cluster, and since we've pointed to the new cluster we haven't seen any either.<br /><br />Once the issue has been resolved with the standard servers and has proven to not have any timeout issues, we will switch back.</p><p><small>Aug <var data-var='date'>20</var>, <var data-var='time'>11:03</var> CDT</small><br><strong>Identified</strong> - A small percentage of transactions to our payments provider are timing out. We're working with them to resolve this issue. The problem seems to be a networking issue between our hosting provider and payments provider, with whom we are also working.<br /><br />We'll update this as we make progress.</p>tag:status.txt2give.co,2005:Incident/17276792018-05-13T13:08:37-05:002018-05-13T13:08:37-05:00Short Code provider issues<p><small>May <var data-var='date'>13</var>, <var data-var='time'>13:08</var> CDT</small><br><strong>Resolved</strong> - We've seen no more issues since our vendor's fix earlier, so we are marking this resolved.</p><p><small>May <var data-var='date'>13</var>, <var data-var='time'>10:28</var> CDT</small><br><strong>Monitoring</strong> - Our provider said the issue should be resolved, and we will be monitoring things on our end as well.</p><p><small>May <var data-var='date'>13</var>, <var data-var='time'>09:55</var> CDT</small><br><strong>Identified</strong> - Our provider has identified the issue and is working on a solution.</p><p><small>May <var data-var='date'>13</var>, <var data-var='time'>09:24</var> CDT</small><br><strong>Investigating</strong> - Our short code vendor is currently experiencing issues with their system. We are in contact with them and will provide updates as we get them.</p>tag:status.txt2give.co,2005:Incident/17136262018-05-03T09:09:16-05:002018-05-03T09:09:16-05:00Step-up authentication using SMS<p><small>May <var data-var='date'> 3</var>, <var data-var='time'>09:09</var> CDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>May <var data-var='date'> 2</var>, <var data-var='time'>17:59</var> CDT</small><br><strong>Update</strong> - This issue has been resolved.</p><p><small>May <var data-var='date'> 2</var>, <var data-var='time'>17:59</var> CDT</small><br><strong>Update</strong> - We are continuing to investigate this issue.</p><p><small>May <var data-var='date'> 2</var>, <var data-var='time'>11:24</var> CDT</small><br><strong>Investigating</strong> - Our authentication provider is having issues with sending SMS messages for our step-up authentication. They are investigating. In the meantime, if you'd like to change your step-up authentication method to Google Authenticator, please contact support and they will help.</p>tag:status.txt2give.co,2005:Incident/16650352018-04-03T11:48:01-05:002018-04-03T11:48:01-05:00Short code issues<p><small>Apr <var data-var='date'> 3</var>, <var data-var='time'>11:48</var> CDT</small><br><strong>Resolved</strong> - We've reverted our temporary fix and short codes are still routing correctly. This issue is resolved.</p><p><small>Mar <var data-var='date'>29</var>, <var data-var='time'>13:51</var> CDT</small><br><strong>Update</strong> - We've gotten word from our provider that the fix has been made on their end, and the DNS changes are propagating now. We'll hold off on removing our temporary fix until we're sure their change has propagated. We'll update here once our temporary fix has been removed and we're all clear.</p><p><small>Mar <var data-var='date'>29</var>, <var data-var='time'>11:16</var> CDT</small><br><strong>Monitoring</strong> - We've put a temporary fix in place while our provider sorts things out. All short code text messages should be processing correctly now. Once our provider has fixed the issue on their end we will remove the temporary fix and resolve this issue.</p><p><small>Mar <var data-var='date'>29</var>, <var data-var='time'>11:08</var> CDT</small><br><strong>Investigating</strong> - We're seeing an issue with our short code provider which is causing short code messages to not be processed. We're working with our provider now to resolve the issue.</p>tag:status.txt2give.co,2005:Incident/16500552018-03-21T02:38:52-05:002018-03-21T02:38:52-05:00Login screen is down<p><small>Mar <var data-var='date'>21</var>, <var data-var='time'>02:38</var> CDT</small><br><strong>Resolved</strong> - Dashboard logins are now working again. You'll be receiving and email with details of what happened, as well as instructions on how to get logged in going forward.</p><p><small>Mar <var data-var='date'>20</var>, <var data-var='time'>11:35</var> CDT</small><br><strong>Update</strong> - Email logins have been restored. We're still working on SMS logins.</p><p><small>Mar <var data-var='date'>20</var>, <var data-var='time'>10:22</var> CDT</small><br><strong>Investigating</strong> - We're looking into a problem with the login screen for the dashboard. This does not affect the ability for donors to make donations.</p>tag:status.txt2give.co,2005:Incident/16485842018-03-19T15:11:54-05:002018-03-19T15:11:54-05:00Server Issues<p><small>Mar <var data-var='date'>19</var>, <var data-var='time'>15:11</var> CDT</small><br><strong>Resolved</strong> - We received notification that the new server is online and everything should be back to normal. Downtime was approximately 9 minutes.</p><p><small>Mar <var data-var='date'>19</var>, <var data-var='time'>15:09</var> CDT</small><br><strong>Identified</strong> - Our hosting provider is having hardware issues with one of our servers. They are attempting to move our application to a different server. We expect this to be a quick process, and will give updates as we get them here.</p>tag:status.txt2give.co,2005:Incident/15588842018-01-10T11:33:16-06:002018-01-10T11:33:16-06:00Text message delays to Verizon customers<p><small>Jan <var data-var='date'>10</var>, <var data-var='time'>11:33</var> CST</small><br><strong>Resolved</strong> - Our provider has given the all clear.</p><p><small>Jan <var data-var='date'> 8</var>, <var data-var='time'>15:29</var> CST</small><br><strong>Investigating</strong> - Our upstream SMS provider has identified delays sending messages to Verizon customers. The link to their status on this issue can be found here:
<br />
<br />https://status.twilio.com/incidents/33vds1q3cf45</p>tag:status.txt2give.co,2005:Incident/15419922018-01-08T15:27:00-06:002018-01-08T15:27:00-06:00PayPal issues with small percentage of donors<p><small>Jan <var data-var='date'> 8</var>, <var data-var='time'>15:27</var> CST</small><br><strong>Resolved</strong> - We believe the fix we implemented has resolved this particular incident.</p><p><small>Dec <var data-var='date'>23</var>, <var data-var='time'>00:41</var> CST</small><br><strong>Monitoring</strong> - We've deployed a fix that should take care of the issues that were affecting some users donating to organizations with the PayPal gateway. We will continue to monitor this fix to ensure it is 100% effective.</p><p><small>Dec <var data-var='date'>21</var>, <var data-var='time'>16:28</var> CST</small><br><strong>Update</strong> - We're continuing to work on a solution for this small subset of users. More info will be posted as we make more progress.</p><p><small>Dec <var data-var='date'>19</var>, <var data-var='time'>14:50</var> CST</small><br><strong>Identified</strong> - We've found an issue that is affecting a small percentage of donors trying to donate to organizations using the PayPal Express Checkout Gateway. The group of affected donors seems to be related to email addresses that are considered undeliverable by our system.
<br />
<br />We're working on a fix and will update here as we make progress.</p>tag:status.txt2give.co,2005:Incident/14460492017-10-26T10:49:36-05:002017-10-26T10:49:37-05:00Canadian transactions failing<p><small>Oct <var data-var='date'>26</var>, <var data-var='time'>10:49</var> CDT</small><br><strong>Resolved</strong> - There was a currency conversion related issue that affected all Canadian transactions starting early this morning around 2:30 AM Central Time, causing the transactions to fail. We identified and fixed the issue at 10:30 AM Central Time.</p>tag:status.txt2give.co,2005:Incident/13912042017-10-01T12:05:12-05:002017-10-01T12:05:12-05:00Intermittent errors<p><small>Oct <var data-var='date'> 1</var>, <var data-var='time'>12:05</var> CDT</small><br><strong>Resolved</strong> - Everything is looking back to normal, so we are marking this issue resolved. The root cause is yet to be determined, but from what we can tell it appears one server dyno started misbehaving. We will be digging into this to try and figure out what caused that.</p><p><small>Oct <var data-var='date'> 1</var>, <var data-var='time'>11:48</var> CDT</small><br><strong>Monitoring</strong> - We believe the issue has been resolved. We're monitoring closely.</p><p><small>Oct <var data-var='date'> 1</var>, <var data-var='time'>11:44</var> CDT</small><br><strong>Identified</strong> - We're seeing some intermittent errors when trying to give by text. We've identified the issue are working to resolve it.</p>