Update(12:15AM): This maintenance is now complete.
Beginning tonight at midnight I will be performing maintenance on equipment that serves a small portion of legacy DSL customers in northern California. Expected downtime for this operation is around 15 minutes.
– Robbie
Sonic will be closed Monday, 7/27/2015 from 4pm to 10pm to celebrate Sonic’s 21st birthday! We will return Tuesday to our normal working hours, and we will call back anyone who left us a message during the closure. Our 24×7 network operations team will still be on call for our Enterprise, dedicated circuit and colocation customers.
Tonight, July 23 starting at 11:59pm, we will performing maintenance on equipment serving a small subset of legacy DSL customers in Northern California. Expected downtime is less than 20 minutes.
Update(12:38am): Maintenance complete, downtime for affected customers was less than 5 minutes.
-Tomoc
Tonight beginning at 11:59PM PDT we will be performing a software upgrade of various components of our voice network in Santa Rosa. No service impact is expected from this, but users may experience sign-outs/session time-outs on voice portals during the maintenance. Estimated time is around ~1 hour.
-Brandon
Update: Maintenance is now complete
-Brandon
This morning a bug was encountered during last nights maintenance that was causing Fusion Voice to not work. Â We have made the appropriate changes to fix the issues with certain customers and all voice should be corrected as of 10:45am. Â Sorry for the inconvenience.
-Brandon
This morning starting at 10am we began noticing a performance impacting issue on our IMAP cluster. We are currently diagnosing the problem and hope to have the system back up and running as soon as possible. This will cause issues for mail retrieval on local clients as well as webmail.
-SOC
Update: We’ve traced the problem back to high CPU utilization on one of the heads units in one of the two NFS filer clusters used to store mail spools. At this time, it isn’t entirely clear what is causing the problem and we’re working to trace it further to the source and determine what options we to resolve the situation. In the meantime, POP, IMAP and Webmail services may have intermittent availability problems. No stored mail, or inbound mail, is being deferred or lost. -Kelsey, William and the rest of the SOC.
Update: All services have been back to normal since 11:45AM. The problem filer is still showing elevated usage but is no longer running one of the CPUs at 100% – we still do not know the root cause but suspect it may have been one or more mail server processes stuck in a byzantine failure. We’re continuing to investigate and monitor the situation and have contacted our vendor for assistance. -Kelsey and William
Update(3:04AM): This maintenance is now complete.
Beginning tonight at midnight I will be performing intrusive maintenance on equipment serving Fusion and FlexLink customers in the Sebastopol area. Downtown for this operation can be up to an hour.
– Robbie
We are currently experiencing an issue with a piece of equipment that serves T1 customers out of Sebastopol. Â We are working on the issue as fast as possible to get everything back up. Â We will update as things progress
-Brandon
Update: Â Service has been restored
-Brandon
Beginning tonight at 12am I will be conducting a code upgrade on a piece of gear in our phone system. Â There should be no downtime, at most in-progress calls may experience a drop. Â Maintenance should last around ~1hr
-Brandon
Maintenance is now complete
-Brandon
Today, July 16, at 10:38am, fiber facilities serving a subset of Fusion/FlexLink customers in the greater Los Angeles area went down. We are working to restore service as quickly as possible.
Update(11:35am): We have routed traffic around the affected circuits and were able to restore traffic to almost all subscribers. We are working to restore facilities to the final affected office as quickly as possible.
Update(10:37am): Outages yesterday were caused by multiple fiber cable cuts in Southern California. Most facilities have been replaced or repaired, but repairs on one remaining cable is still underway. We continue to work to restore service to the last affected subscribers.
Update(10:53am): ETR for final repairs is 2 hours.
-Tomoc, NOC