Tonight at 10pm we will be upgrading the operating system on the Sonic customer FTP server ftp.sonic.net. We expect the service downtime to be brief and no longer than 1 hour in total.

Update: This maintenance has been completed successfully.

Update (12:55AM): This maintenance has been completed.

Tonight, starting at 11:59 PM, we will be conducting maintenance on core networking equipment in the LA area. While we anticipate no downtime for end users, there might be occasional routing instability as traffic is rerouted from the affected equipment. The maintenance window for this operation is estimated to be 4 hours.

UPDATE: Maintenance complete.

Tonight, beginning at 10PM PST, we will be applying updates to multiple systems.  Any downtime to affected services should be brief, and we expect to complete the maintenance within 2 hours.

The systems include, among others:

• VPN servers
• Various public facing applications

At 11am our back-end IMAP/POP3 cluster entered a critical state which lead to an interruption in those services, as well as other services that rely on our mail infrastructure. The initial cause of the failure was a routine maintenance procedure that involved dropping traffic to a portion of the cluster. While the remaining cluster should have been able to run temporarily with a smaller group, that quickly turned out to not be the case. The remaining servers began to fail intermittently as they tried to shift traffic to account for the sudden increase in load. This would have caused noticeable mail client issues, and it also led to service availability interruptions on both our webmail and our voicemail platforms.

As soon as the problem was detected, we acted by aborting the maintenance. This was followed by additional resources being added to the cluster to prevent further disruptions. As of 11:34am service was fully restored. We do not expect loss of email to have resulted from this. As always we will look into improving our metrics and analytics to improve our response time.

Yesterday afternoon at approximately 4:45PM we completed the turn up and migration of services to a new set of geographically redundant proxy servers used by our internal mail infrastructure to communicate between the various clusters that support email services. Unfortunately one of the new proxies was left out of an ACL which prevented that proxy from being able to connect to the mail forwarding cluster.  This resulted in about 50% of forwarded mail (typically configured in the “Forwarding” section of Sonic Member Tools) being rejected.  These rejected messages would have resulted in non-deliverable messages being returned to the original sender instead of being delivered as intended.

We apologize for any issues this may have caused and will review our systems for any opportunities to improve monitoring to ensure that if something like this happens again it is immediately detected and resolved.

Tonight, beginning at 10PM PST, we will be applying updates to multiple systems, including our customer shell server and VPN servers. Any downtime to affected services should be very brief, and we expect to complete the maintenance within 2 hours.

Update: maintenance is complete.

Tonight, beginning at 10PM PST, we will be applying updates to multiple systems.  Any downtime to affected services should be brief, and we expect to complete the maintenance within 2 hours.

Update: Work is complete.

Tonight (3/14/24) beginning at 11pm we will be conducting maintenance on various Sonic systems. Downtime on individual systems will be limited to a few minutes each, and the maintenance work overall is scheduled to last for approximately one hour.

Update: 12:10 AM Maintenance is complete.

March 6th, at 12:01 AM we will be performing maintenance on our phone switch. No voice service interruptions are expected as a result of this maintenance. The window for this maintenance is 3 hours.
~Network Engineering