Sonic.net Status

Pogowave finishes network address translation migration

Today is the day that the wireless department will turn off network address translation on our last Pogowave station; The Golden Apple Ranch on Fitzpatrick Lane.

This is our largest and most diverse station, featuring three low band access points.

We think that this can be performed in less than 30 minutes of down time.  Thank you for your patience during this transition that will allow for improved customer support for all Sonic.net Pogowave customers.

Posted in Maintenance, Network | No Comments »

DSL DHCP Server Issues

Early this morning four of our DHCP servers started having issues responding quickly to DCHP requests.  These simultaneous failures overwhelmed our ability to migrate load to our hot-standby servers but we were able to put several work-arounds in place to mitigate the issues.  These failures were initially believed to be consistent with disk pre-failure scenarios where a single disk’s performance is impacted and the RAID system has yet to fail the disk.  However, upon further investigation it was revealed that these failures were triggered by scheduled SMART tests.  Ironically, the SMART tests were recently enabled to help us detect and replace failing disks before their failure triggered a service impacting event.

At this time, all DHCP services have been returned to normal.

-Nathan, Don, and Kelsey

Posted in Uncategorized | 2 Comments »

ATM Customer Aggregation Router Reload

This upcoming Thursday, Oct 1 at 12:01 AM, we will be performing a maintenance reload on our ATM customer aggregation routers. This will result in 5-10 minutes of downtime for Business-T and FRATM customers.

-Jared

Update: The router reloads have been completed without incident. All Business-T and FRATM customers should be back up at this time. Total downtime was less than 10 minutes.

Posted in Uncategorized | No Comments »

DSL Outage, Santa Rosa

Starting at approximately 9:50 this evening, some areas of Santa Rosa began experiencing intermittent DSL connectivity.   We are working with ATT to resolve the issue.    As a result of this outage, the telephone wait for Technical Support is unusually high.  We expect resolution shortly, but do not have an ETR .

-Kim,  Support

Update: AT&T has diagnosed the problem as a failing card at an ATM switch in Northern California.  This outage persists. We currently have no estimated time of repair. AT&T has reset the card which seems to have temporarily restored service to some customers while they work toward a permanent solution.

Update:  Service should now be fully restored to all customers affected by this outage.  If you are still having connectivity issues please contact Sonic.net technical support.

Posted in Uncategorized | 3 Comments »

ADSL2+ Linecard Maintenance

Tonight, this Tuesday, Sep 15 at 12:01 AM, we will be performing a software upgrade of the linecards that provide ADSL2+ service out of our Downtown Santa Rosa CO. This will cause a brief disruption of service to ADSL2+ customers in that area. Affected customers will experience approximately 5 minutes of downtime as the cards reboot onto the new software.

-Jared

Update: The ADSL2+ cards have all been upgraded to the new software and are up and functioning normally.

Posted in Uncategorized | No Comments »

DSL Customer Migration

This evening, Saturday September 12, at 12:01 AM, we will be migrating some of our dynamic IP DSL subscribers from a heavily loaded DSL aggregation router to a newer, less loaded one. Affected customers may see an interruption in their Internet connectivity for 5-10 minutes. We apologize for the short lead time on this announcement, but we feel it is better to address the load issue before the weekend. Thank you for your understanding.

-Jared
Update: The migration has been completed without incident and all affected customers appear to be up and running normally at this time.

Posted in Uncategorized | 1 Comment »

E-mail infrastructure maintenance parts three and four.

Update (01:10) : Mail migration done; should be faster accessing e-mail for those of you that were migrated. -Augie

This Saturday (12, September) at 12:01 AM we will be performing the last service impacting e-mail migrations from our existing e-mail storage infrastructure to our new (and greatly improved) storage system.

When it is completed this migration will mark a substantial improvement in our e-mail systems performance and availability for all of our customers.

During the maintenance window (12:01am – 1:00am) all customer e-mail systems will be unavailable; this includes POP, IMAP, and Webmail.

No e-mail will be lost during this time. -Augie

Posted in Maintenance | No Comments »

Large DoS Attack

This afternoon at 12:15 PM a huge DoS attack was aimed at a customer on our Sonic Telecom CLEC network. The DoS attack was large enough to destabilize routing in our CLEC network, so customers may have seen 10-15 minutes of spotty connectivity as we tracked down and blocked the DoS. At this time, the attack has been blocked and all services are functioning normally.

-Jared and Nathan

Posted in Uncategorized | 2 Comments »

E-mail infrastructure maintenance part two.

Update (0100) : E-mail maintenance done; total down time was around 28 minutes; no e-mail was lost; let us know if you see any problems. – Augie

This Saturday (05, September) at 12:01 AM we will be performing the second of four service impacting e-mail migrations from our existing e-mail storage infrastructure to our new (and greatly improved) storage system.

When it is completed this migration will mark a substantial improvement in our e-mail systems performance and availability for all of our customers.

During the maintenance window (12:01am – 1:00am) all customer e-mail systems will be unavailable; this includes POP, IMAP, and Webmail.

No e-mail will be lost during this time. -Augie

Posted in Maintenance | 2 Comments »

DHCP Server Failure

This morning at approximately 8 AM our Support staff became aware that one of our DHCP servers was not serving DHCP leases properly, in such a way that our monitoring systems were not able to detect the problem. Our Support team escalated the problem to the Operations staff and we were able to remedy the problem by rebooting the partially-locked server. Service was restored by approximately 8:28 AM. We apologize for this outage and will be looking into ways of extending our monitoring so this unusual type of failure will not be missed again.

-Jared and the Support team

Update: The DHCP server has begun to fail again, so we have removed it from service. DHCP should be functioning normally again. If you are having problems obtaining a DHCP lease, please try rebooting your modem and any computers or routers that are having problems.

Posted in Network | 4 Comments »