Infrastructure Updates, August 2023
Incident Report for GlobalTill
Resolved
This incident has been resolved.
Posted Aug 21, 2023 - 10:02 PDT
Monitoring
GlobalTill Update: Email Delivery Improvement

We're pleased to bring you a positive update regarding our recent email delivery challenges.

Since our transition to an alternative email provider, we've seen substantial improvements in our email deliverability. As of our latest metrics, almost 98% of outgoing mail is successfully showing as delivered (see https://safe.dsh.re/download/80ade639-646f-4f8d-9ce6-f691b217b940).

While we are encouraged by these results, our team continues to diligently monitor the situation to ensure a full and consistent recovery.
Posted Aug 13, 2023 - 16:48 PDT
Identified
Email Issues

Problem: Some of our users have faced challenges with the delivery of emails from the GlobalTill platform. Our investigation revealed that a significant portion of AWS IP addresses, from which our platform sends emails, were blacklisted by Barracuda Networks and potentially others. This resulted in low email deliverability.

Status Update: On August 7th-8th, we successfully completed a failover process to an alternative email provider. As of August 9th, we've seen improved deliverability rates. We're actively monitoring to guarantee complete recovery. Please be advised that email statistics in the GT backend (under Settings > Advanced > Email Log), including metrics on opens, deliveries, and bounces, may not reflect accurate data until subsequent updates are finalized. We will post another update on this when completed.

Reporting

Problem: Our recent updates caused hiccups in GlobalTill's legacy reporting system. While no changes were made directly by our team, we discovered that some dependencies had undocumented changes. This caused an inability for reporting parameters to transition from the web interface to our background job system.

Status Update: We have addressed most of these challenges and are closely monitoring our logs to ensure the stability and accuracy of the reporting system moving forward.

What to do if you're still having problems

If you are still facing any challenges with reporting or email delivery, we're here to help. Please get in touch through the regular support channels or complete this form.

Again, we sincerely apologize for any disruption.

The GlobalTill Support Team
Posted Aug 11, 2023 - 21:41 PDT
This incident affected: Core (Admin / Back-end, Background Processing).