Archive for the ‘Network’ Category

San Jose POP Retrofit

Tuesday, September 30th, 2008

This Thursday, Oct 2 at 12:01 AM we will begin work retrofitting the backbone of our San Jose POP. This retrofit will enhance the redundancy and capacity of this POP to accomodate our growth and new CLEC-based products. Currently this POP does not directly serve any customers, so there should ...

DSL Subscriber Rebalancing

Monday, September 29th, 2008

On Tuesday September 30 at 12:01 AM we will be moving several hundred DSL subscribers from one of our DSL aggregation routers to another, to balance network load. This move should take less than 5 minutes and will only cause a brief service interruption. -Jared Update: Rebalancing completed. Downtime for affected customers ...

System Authentication Failure

Monday, September 22nd, 2008

This afternoon, around 4:30PM we experienced a problem in our main authentication system. This problem effected a handful of user accounts and associated mailboxes. Account and mailbox owners would have seen authentication failures and possible timeouts when attempting to connect to services such as shell, pop/imap, and webmail services. The ...

VPN Concentrator Certificate Expiry

Monday, September 15th, 2008

Today the identity certificate for our VPN concentrator expired. This would have prevented new VPN sessions from establishing. We have renewed the certificate, and the VPN concentrator is accepting connections normally now. -Jared and Nathan

LATA1 AT&T ATM Outage

Thursday, September 4th, 2008

At 5:03 PM today, all of our DSL and Business-T subscribers in LATA1 went offline. The problem appeared to be internal to AT&T's ATM network. As of 5:12 PM, the problem in AT&T's network appears to have been resolved, and DSL and Business-T customers are back online. We are continuing ...

ATM Switch Maintenance

Wednesday, September 3rd, 2008

Tonight at midnight we will be performing potentially invasive maintenance on one of our ATM switches. We will be reloading the primary route processor in the ATM switch and it should automatically fail over to the backup. Impact should be only a few seconds as the failover occurs, though it ...

BroadLink wireless outage

Wednesday, August 27th, 2008

BroadLink is a wireless carrier partner for a group of customers in Santa Rosa and some surrounding areas.  BroadLink has had a failure of a critical component in the core of their network, and they are currently attempting repair. We have scheduled a meeting for an hour and a half from ...

Massive DDoS Attack

Tuesday, August 12th, 2008

Approx 30 mins ago a Distributed Denial of Service attack was aimed at one of our colo customers. The NOC quickly identified and blocked the DDoS, but the attack ramped up to over 700 Mbps for several minutes. This traffic congested one of our transit links for 5-10 minutes, possibly ...

DSL Subscriber Rebalancing

Monday, August 11th, 2008

On Wednesday August 13 at 12:01 AM we will be moving several hundred DSL subscribers from one of our DSL aggregation routers to another, to balance network load. This move should take less than 5 minutes and will only cause a brief service interruption. -Jared Update: The migration has been completed. Downtime ...

Inbound DDoS

Friday, August 1st, 2008

We recently experienced an inbound DDoS attack that was directed at one of our customers. The attack was promptly blocked at our edge and any connectivity issues related to it should be resolved. Customers may have experienced slightly degraded performance during this time. -Dusty, Nathan, and Jared