Archive for November, 2011

Customer SQL Server Outage

Wednesday, November 30th, 2011

This morning, at 8:32 AM, one of our customer MySQL servers crashed due to an apparent bug in MySQL.  Additionally, after the crash it was failing to restart.  The trouble was isolated to a specific user's database, which when manually removed, allowed the MySQL server to start normally.  The server ...

San Jose DSL outage

Tuesday, November 29th, 2011

We have confirmed with AT&T that there is currently a DSL outage in the San Jose area. They are working to resolve the issue but we do not currently have an ETR. Symptoms are that customers have sync but are unable to get on line.   Update: The outage should be clear. ...

DSL Aggregation Router Reload

Monday, November 28th, 2011

Tonight, Tuesday November 29 at 12:01 AM, we will be performing a maintenance reload on several legacy DSL aggregation routers that serve some of our customers in the Bay Area and Los Angeles areas. Affected customers will experience 5-10 minutes of downtime as the routers reboot. -Jared Update: The maintenance reloads have ...

Legacy DSL Service Degradation

Saturday, November 12th, 2011

This evening, our support team began tracking an issue causing poor DSL performance for legacy DSL subscribers served by a certain SMS. Upon further investigation, we found a port on our ATM switch to be failing, and causing loss for traffic going across it. We moved the SMS to a ...

VPN Concentrator Authentication Difficulties

Friday, November 11th, 2011

Earlier this afternoon, we made some changes to the authentication service used by our VPN Concentrator, which resulted in an inability to establish new connections to the VPN Concentrator. We tracked down and fixed the configuration problem that caused this issue, and all VPN sessions should be able to connect ...

Brief LATA 3 DSL Outage

Tuesday, November 8th, 2011

At approximately 3:09PM, while performing routine cable maintenance, a telco technician accidentally pulled an incorrect fiber jumper causing a brief outage. This would have affected legacy DSL service for customers in the Sacramento area. The mistake was immediately corrected and service was restored less than a minute later. We apologize ...

Seismic Retrofit at our Santa Rosa Datacenter

Tuesday, November 8th, 2011

The raised floor bracing and cabinet mounting systems at our Santa Rosa datacenter are currently undergoing a seismic retrofit.  The work is being handled by a contractor that specializes in raised floor environments and will not cause an interruption in any of our services.   Colo customers should note that ...

Fusion Voice 3-1-1 Service

Tuesday, November 8th, 2011

All Fusion Voice subscribers are able to access 3-1-1 services where available.  Please note that  3-1-1 services are available only in a limited set of cities.  For more information about 3-1-1, and the list of cities where 3-1-1 services are available, please see our wiki.

Flexlink Ethernet Customer Reconfiguration

Thursday, November 3rd, 2011

Tonight, Friday November 4 at 12:01 AM, we will be reconfiguring approximately 50 Flexlink Ethernet customers to smooth the way for future network upgrades. This work will disrupt connectivity to those customers for less than 5 minutes apiece. No work or changes will be necessary for these customers to restore ...

Northern California Calling Issues

Tuesday, November 1st, 2011

We were notified this morning of issues dialing numbers in Northern California on our Fusion Voice service. The issue was traced to one of our upstream phone providers in this area. Calls are now routed out redundant capacity and the issue is resolved. There was no impact to 911 service. -Nathan