Month: July 2010

Closed this Saturday, 11am-6pm

Our team will be taking time off to celebrate a great year and our company 16th birthday, so technical support will be closed from 11am to 6pm this Saturday the 31st.  Support will be open from 6pm to 10pm and Sunday as usual. Sorry for any inconvenience.

Our normal 24x7x365 schedule is still applicable for our on-call network operations team for dedicated business and colocation customers.

Non-intrusive Power Maintenance

This evening, beginning at 10PM, we will be performing non-intrusive power maintenance on equipment at 200 Paul Avenue in San Francisco. We will be moving power sources between redundant loads and do not expect the work to have any impact on customer traffic.

-Tim, Nathan, Jared, and Matt

Core switch maintenance

This evening, Tuesday July 20th at 11PM we will be replacing the failed blade in our core switch. There should be no effect aside from a brief hit in accessibility to news.sonic.net. Any impact should be less than 10 minutes.

-Jared

Update: The card replacement was completed shortly after 11PM, and the core switch was restored to full service at approximately 11:15PM and is functioning normally at this time.

Sonic.net News cluster problem.

10:28 news.sonic.net is back on-line — we apologize for the inconvenience.

The Sonic.net News cluster (news.sonic.net) suffered a hardware problem and is currently being repaired; we will update you as changes occur.

Python upgraded on web cluster

Python has been upgraded to version 2.6 on our webcluster and shell server.
Access the new version by calling python2.6 instead of python.
MySQLdb has also been installed along with this upgrade.

-William

Network Instability

At 12:55pm today it was brought to our attention customers on our network were experiencing intermittent reachability to parts of the internet.  We determined the source of this to be a routing issue with one of our upstream providers. At approximately 1:08 we routed traffic away from the affected upstream, thus mitigating the issue.

The upstream provider has since fixed their routing issue, but we are performing further tests on the link before returning network traffic back to it.

-Matt and Jared

Edit: Fixed incorrectly stated times in the above message. -Matt

Web Cluster Issues

Update: Our web cluster was the target of a large DDoS; the targeted host has been isolated and the cluster has been stabilized.

We are currently experiencing issues with our web server cluster. We are investigating the problem and will provide an update shortly.

-Tim, Nathan, and William

DSL Subscriber Migration

This evening, Thursday July 15, at 12:01AM, we will be migrating all DSL subscribers in the Sacramento area to a new OC-12 circuit. Affected customers may experience up to 10 minutes of downtime while the migration is performed.

-Tim, Jared, and Nathan

Update: This migration has been completed and all affected customers should be back online at this time.