System Maintenance

June 25, 2015 – 2:48 pm by Grant Keller

UPDATE: Work complete.


Tonight starting at 11:59pm System Operations will be running updates that will effect the following services:

  • Webmail
  • Membertools
  • SMTP
  • FTP
  • IPv6 tunnels
  • customer websites

All of the interuptions should be momentary.

Grant and SOC

Fusion Intrusive Maintenance – San Francisco

June 25, 2015 – 2:13 pm by Brandon Gile

Beginning tonight at 12pm I will be conducting intrusive maintenance for a very small portion of fusion customers in San Francisco.  Estimated downtime is ~20 mins.



This is now complete


Emergency Router Maintenance – ASE/Airport Business Park Fiber

June 25, 2015 – 1:45 pm by rabrown

Update (12:40AM): This maintenance is now complete.

Beginning tonight at midnight I will be doing maintenance on a router that serves ASE and business park fiber customers. Expected downtime is less than 1 hour.

– Robbie

Fiber Outage – ASE/Airpot Fiber

June 25, 2015 – 7:52 am by Brandon Gile

Service has been restored



We have an outage affecting customers being served off of ASE and the Airport Business park.  We are currently looking into the issue and will update soon.




Fusion/FlexLink Outage – Danville

June 23, 2015 – 9:38 pm by tomoc

This evening, June 23 starting at 9:23pm, circuits serving a subset of Danville Fusion/FlexLink subscribers went down. We are working to restore service as quickly as possible.

Update(2:10am): One of the affected circuits has been repaired. Service has been restored for all affected users.


Non-Intrusive Maintenance – Flexlink/Fusion Customers

June 23, 2015 – 3:39 pm by Tim Jackson

This maintenance is now complete.

Beginning at 11:59PM PDT on Thursday 6/25/15 we will be performing some policy updates on routers serving customers in the Bay Area and Los Angeles. These changes are not expected to be traffic impacting.

-Tim J.

Fiber DIA Customers – Intrusive Network Maintenance – 6/25/15

June 23, 2015 – 3:37 pm by Tim Jackson

This maintenance is now complete.

* Clarification: This maintenance will begin at 11:59PM PDT on 6/25/15.

On Thursday 6/25/15 we will be performing a software upgrade of a router serving Fiber DIA customers. This maintenance is expected to last 15-20 minutes.

-Tim J.

IMAP/POP3 outage

June 22, 2015 – 4:46 pm by Grant Keller

Due to the ongonig logging issues we have seen today, there was a brief outage of our IMAP/POP3 services. As of right now both are up and running. Total down time was 25 minutes.


Internal Log Server Issues

June 22, 2015 – 9:52 am by Kelsey

One of our central syslog servers had a series of raid failures that lead to filesystem corruption late last night.  Normally, this would only be an issue for our backend systems but at approximately 8:30AM this morning local logging on our MX servers, which forward logs to that  syslog server, blocked.  This caused inbound mail handling to stop along with other services including the redis sentinel cluster that is used to store configuration information.  That, in turn, caused the membertools to have issues since they depend upon some of the configuration stored in redis.  The MX servers were all back up and running by 9:15AM.  We’re continuing to work on restoring the failed syslog server but there is no customer impact at this time.  – Sonic Ops

Fusion/FlexLink Outage

June 16, 2015 – 4:20 pm by tomoc

Update (4:52PM): All affected customers should be back online at this point.

Today, June 16, starting at 3:35pm, configuration changes on equipment serving Fusion/FlexLink subscribers in northern California caused connectivity issues for a subset of subscribers.

We are working to restore service to all affected subscribers as soon as possible and expect all connectivity to be restored within 15 minutes.

-Tomoc and NOC