Month: June 2010

CLEC Intrusive Maintenance

This evening, beginning at 12:01AM, we will be performing maintenance on equipment serving a subset of Fusion and FlexLink customers. Affected customers may experience up to 30 minutes of downtime while the work is performed.

-Tim, Nathan, and Jacob

ATM Switch Maintenance

Tonight, Wednesday June 30 between 1 AM and 2 AM, we will be performing a maintenance reload of one of our ATM switches to resolve a diagnostic bug we have found. Affected customers will experience about 5 minutes of downtime as the switch reloads onto new code.

-Jared

Update: The ATM switch reload has been completed without incident. Total downtime for affected customers was less than 10 minutes.

CLEC Intrusive Maintenance

This evening, beginning at 12:01AM, we will be performing maintenance on equipment serving a subset of Fusion and FlexLink customers. Affected customers may experience up to 30 minutes of downtime while the work is performed.

-Tim, Nathan, and Jacob

DSL Subscriber Migration

This evening, Tuesday June 29, at 12:01 AM, we will be migrating some of our dynamic IP DSL subscribers from one of our ATM OC-12 circuits to a less loaded one. Affected customers may see an interruption in their Internet connectivity for 5-10 minutes.

-Jared and Nathan

Update: This migration has been postponed to Wednesday June 30, at 12:01 AM.

Update: This migration has been completed successfully. All affected customers should be back online at this time.

CLEC Intrusive Maintenance

This evening, beginning at 12:01AM, we will be performing maintenance on equipment serving a subset of Fusion and FlexLink customers. Affected customers may experience up to 30 minutes of downtime while the work is performed.

-Tim, Nathan, and Jacob

DSL Subscriber Migration

This evening, Friday June 25, at 12:01 AM, we will be migrating some of our dynamic IP DSL subscribers from one of our ATM OC-12 circuits to less loaded one. Affected customers may see an interruption in their Internet connectivity for 5-10 minutes.

-Jared and Nathan

Update: The subscriber migration has been completed without any undue impact. All affected customers should be back online at this time.

CLEC Intrusive Maintenance

This evening, beginning at 12:01AM, we will be performing maintenance on equipment serving Fusion and FlexLink customers. Affected customers may experience up to 30 minutes of downtime while the work is performed.

-Tim, Nathan, and Jacob

Scheduled server and power maintenance.

Tomorrow evening (24, June) starting at 10pm Pacific Time we will be changing the power configuration on the NetApp storage filers that store customer e-mail; these filers are redundantly powered and we expect no down time for them during the maintenance window. During this time we will also perform server maintenance that may cause brief service interruption to our customer facing Member Tools.

–Augie

Recursive DNS Clusters Redeployed

Since the cluster failure last month we had been running our rercursive name servers in a sub optimal (but still redundant) configuration.  Last Friday, we completed the re-deployment and once again have a multiple layers of redundancy with both our primary and secondary recursive name servers announced into our network in Santa Rosa, San Francisco and Los Angeles.  The old Alteon hardware load balancers that used to handle 208.201.224.11 in Santa Rosa and San Francisco have been replaced with Linux LVS/Quagga BGP servers that do not suffer from the same limitations that caused the original cluster failure.  While each of the real servers could handle many times our total DNS load on its own, neither of the Alteons that were responsible for directing traffic for 208.201.224.11 could handle it after some routine network configuration shifted traffic inside of our network.  Apart from a small hiccup due to a BGP misconfiguration Friday morning, the new clusters have performed perfectly. –Kelsey, Nathan and Jared

CLEC Intrusive Maintenance

This Tuesday, June 22, beginning at 12:01AM, we will be performing maintenance on equipment serving Fusion and FlexLink customers in Sausalito, San Mateo, San Jose, and Millbrae. Affected customers should expect less than 30 minutes of downtime.

-Tim, Nathan, and Jacob