Sonic.net Status

Trouble with the Sonic.net Website.

Fri Oct 10 10:05:08 PDT 2008 — Trouble with the Sonic.net Website. We are currently experiencing trouble with our corp.sonic.net Web Server that serves part of the Sonic.net Home Page and all of our Blogs. We are working on the problem and will have it back soon. -Augie, William, and Kelsey.

Update: We have restored services and determined the problem being a hardware failure. We apologize for the inconvenience this may have caused. -Don and William

Posted in Maintenance, Network | 2 Comments »

Mail Storage Maintenance.

Tonight at Midnight we will be performing maintenance on the NetApp Filers that store Customer E-Mail; there is no expected downtime during this period and will take less than an hour to be completed.

During this period we will be adding additional capacity to these Filers; this capacity will allow for future growth of Customer E-Mail storage and improved performance for all of Sonic.net’s E-Mail Customers.

–Augie, Don, William, Sal, and Kelsey.

Update: maintenance has been completed; no problems were encountered (other than starting later than scheduled); capacity was nearly doubled on our E-Mail Storage System.  –Augie

Posted in Maintenance | No Comments »

Tagged with: e-mail , email , filer , netapp , storage .

Webmail IMAP performance problems solved.

Separate from our earlier post about slow imap.sonic.net performance (http://corp.sonic.net/status/2008/09/26/imap-performance-problems-solved/) - we have also received reports of slow Webmail IMAP performance and timeouts when Customers were using the Webmail clients on http://webmail.sonic.net.

We believe we have isolated the problem, which was a bug in our IMAP Proxy software, and have not received any reports of new problems since the beginning of the week when we implemented a fix for the problem software.

If you see timeouts when using http://webmail.sonic.net, please contact Technical Support (support@sonic.net or 1.707.547.3400) immediately, and provide the error message you receive and the time at which the problem occurred.

Posted in Uncategorized | 2 Comments »

Webmail Web Site Time-Warp.

A misguided attempt to update some software on our Webmail Cluster inadvertently took the software, associated web pages, and server configuration back to January of this year.

As a result Customers would have seen inconsistent or broken behavior while trying to access the website from around 2:30am to 8:00am, at which point the data was restored from backups.

We applogize for any inconvenience this caused to our Customers; we will be reviewing our documented procedures so that this type of mistake does not occur in the future.

–Augie

Posted in Uncategorized | No Comments »

Tagged with: atmail , nutsmail , squirrelmail , webmail .

Emergency Router Maintenance.

At 3:40PM this afternoon we will be performing an emergency router reload on one of our ATM customer aggregation routers. All connected Business-T and FRATM customers will experience approximately 5 minutes of downtime during the reload. -Tim, Nathan, Matt and Jared

Posted in Uncategorized | No Comments »

San Jose POP Retrofit

This Thursday, Oct 2 at 12:01 AM we will begin work retrofitting the backbone of our San Jose POP. This retrofit will enhance the redundancy and capacity of this POP to accomodate our growth and new CLEC-based products. Currently this POP does not directly serve any customers, so there should be no customer impact. We will route customer traffic away from the POP, so you may see longer routes to Internet hosts than normal during this time. We expect the operation to take 60-90 minutes.

-Jared and Nathan

Posted in Network | No Comments »

DSL Subscriber Rebalancing

On Tuesday September 30 at 12:01 AM we will be moving several hundred DSL subscribers from one of our DSL aggregation routers to another, to balance network load. This move should take less than 5 minutes and will only cause a brief service interruption.

-Jared

Update: Rebalancing completed. Downtime for affected customers was under 5 minutes.

Posted in Network | No Comments »

IMAP performance problems solved.

We believe we have fixed the recent IMAP performance problems; let us know if you have any trouble with imap.sonic.net still.

For the past few days we have received a number of reports from customers concerned about periodic slow IMAP performance to imap.sonic.net; the problem would manifest as a mail client behaving very slowly when reading e-mail or timing out all together.

The periodic nature of the problem made it very difficult to troubleshoot as by the time a customer would call to alert us to the problem it would then mysteriously go away.

We believe we have solved the problem and have since implemented much more detailed monitoring of the imap.sonic.net IMAP cluster.

Note: we have seen a handful of reports of slow Webmail performance; those servers are completely separate from the imap.sonic.net cluster, so if you are seeing Webmail performance problems, please note the time it occurred, the error you saw if any and send an e-mail to support@sonic.net .

Posted in Uncategorized | 5 Comments »

Updates to password changing tool

In the interest of customer security we have updated our password changing tool to email main account holders when either the main account, or any mailbox under that account has the password changed.  The email also will go to any “invoice to email” addresses you have on the account.  We hope that this is useful in notifying customers immediately if their account or mailboxes have been compromised.

Posted in Member Tools, New Features | 3 Comments »

System Authentication Failure

This afternoon, around 4:30PM we experienced a problem in our main authentication system. This problem effected a handful of user accounts and associated mailboxes. Account and mailbox owners would have seen authentication failures and possible timeouts when attempting to connect to services such as shell, pop/imap, and webmail services. The underlying problem has been fixed.  Customers may have seen approximately 10 minutes of authentication failure. If you are experiencing problems with authentication, please call Sonic.net Support at (707) 522-1000. We appologize for the inconvenience this has caused. –Don and Jared

Posted in Network | No Comments »