Month: December 2009

ATM Customer Aggregation Router Reload

This evening, Tuesday, Dec 29 at 12:01 AM, we will be performing a maintenance reload on our ATM customer aggregation routers. This will result in 5-10 minutes of downtime for Business-T and FRATM customers.

-Jared

Update: The scheduled router reloads have been completed and all affected customers should be back online. Downtime for affected customers was < 10 minutes.

Sonic.net Closing at 6pm

Sonic.net is closing today at 6:00PM for the holidays. Technical Support will be reopening Saturday morning at 8:00AM.  Happy holidays from all of us at Sonic.net!

Call Center Problem

Today at approximately 1:30 pm phone lines in to the Sonic.net office are reporting all circuits busy. We have reported this outage to our vendor who has not yet diagnosed the trouble.

This outage affects inbound and outbound calls through our office phone system. No other services are affected by this outage.

As always, we are are available via email to support@sonic.net

Update:  it appears the root cause here was a failed ATM card at AT&T.  As of 5:37pm we believe our phone system is fully operational again. Our vendors are continuing to monitor the situation closely. — John F and Jasper

ATM Switch Reboot

This morning at approximately 5:01 AM, one of our ATM switches suffered an interface card lockup. We are about to initiate an emergency reboot of the ATM switch chassis to clear the problem. The ATM switch reboot will disrupt DSL, Business-T, and FRATM connectivity to some customers in the Bay Area, as well as the Stockton, Chico, and San Diego areas.

-Jared

Update: The ATM switch reboot was completed at 5:33 AM. All service through that ATM switch is now restored.

DDoS Attack

Starting at about 2:24 PM today, a distributed denial of service attack was launched at one of our DSL subscribers. This DDoS was large enough to disrupt service to the Redback SMS DSL aggregation router that served this customer and several hundred others. We have blocked the DDoS attack at our edge, thus restoring service to the affected customers.

-Jared

DDoS Attack

Starting at about 1:33 PM today, a distributed denial of service attack was launched at one of our DSL subscribers. This DDoS was large enough to disrupt service to the Redback SMS DSL aggregation router that served this customer and several hundred others. We blocked the DDoS attack at our edge within 5 minutes, thus restoring service to the affected customers.

-Jared

Offices Closed for our Holiday Party

Our offices will be closed from 12pm to 5pm today for our 15th Annual Holiday Party.  Should you need assistance, please leave a voice mail or email support@sonic.net.  As always, Network and Systems Operations are available on their respective hot lines.  -Sonic.net

The website is down

We apologize for the delay in posting this notification, but our systems operations staff are very busy trying to fix the problem. Thank you for your patience, web services should be restored soon!

Update:  Starting at 7:30AM this morning a DDOS directed at one of our hosted websites started to destabilize the cluster causing intermittent failures.  Our initial response was misdirected as we thought it to be a known failure scenario with our hardware load balancers.  We rebooted the load balancers and all services came back up and we believed that the issue had been resolved.  It wasn’t until several subsequent failures occurred that the root cause of the problem was correctly identified and fixed.  Services were fully restored shortly after 9:00AM.  -Kelsey, William, Joey and Nathan

Unexpected T1 Aggregation Router Reload

Today at 9:58AM one of the routers that serves T1 customers in the Sonoma County area unexpectedly reloaded. Service has been restored at this time and we are investigating the cause of the reload. Approximate downtime was 5 minutes.

-Tim and Nathan