Non-intrusive Voice Maintenance

April 22, 2014 – 8:25 am by Tim Jackson

Tonight (4/22/2014) beginning at 11:59PM PDT we will begin performing a software upgrade on portions of our voice network in Santa Rosa. No customer service impact is expected from this.

-Tim J.

Emergency Fusion/FlexLink Maintenance

April 12, 2014 – 10:56 am by tomoc

Tonight, April 12, starting at 11:59PM, we will be performing intrusive emergency maintenance on equipment serving Fusion and FlexLink customers in the Cotati area. We estimate this maintenance will be completed within two hours.

-Tomoc and Clay

Heartbleed update

April 11, 2014 – 5:25 pm by Grant Keller

Hello,

All vulnerable keys have now been replaced. We still recommend changing your account password to mitigate the damage if this exploit was attacked on our system. The systems included in this last round of updates where:

  • legacy-webmail.sonic.net
  • forums.sonic.net
  • wiki.sonic.net

- Grant and SOC

Non-intrusive Voice Maintenance

April 10, 2014 – 12:54 pm by Tim Jackson

Tonight (4/10/2014) beginning at 11:59PDT we will be performing non-intrusive maintenance on our SBC in Santa Rosa. No customer impact is expected from this.

-Tim J

The Heartbleed Bug and You: Change Your Passwords!

April 9, 2014 – 5:33 pm by Kelsey

We always keep your privacy and security in mind.

By now you’d be hard pressed to have missed coverage of The Heartbleed bug in OpenSSL.  At this point, Sonic.net is joining many other providers and recommending that you change your passwords for your online services.  This is important for high value accounts like banking and finance or other accounts that protect your personal information and data.  Do not forget to change your ISP and email account passwords!  These are especially important since access to your email account can be used to gain access into most of your online services.

We do not have any reason to believe that we, or any of our users, were targeted.  However, this attack was undetectable and the cautious response is to assume that sensitive information has been leaked.  In the interest of full disclosure we are providing a complete list of affected services and systems.  It should be noted that all of the vulnerable services support PFS wherever possible and should our private keys have been leaked, they cannot be used to decrypt any past traffic in most cases.

Customers may change their passwords in the membertools using the password tool.

If you have any questions, please post them in our forums.

As of 21:45 on April 7th,  all vulnerable systems had received an update to fix this bug.

The following sites and services were vulnerable:

  • imap.sonic.net (login credentials were not vulnerable, only keys)
  • pop.sonic.net (login credentials were not vulnerable, only keys)
  • mail.sonic.net (login credentials and mailflow in/out)
  • legacy-webmail.sonic.net
  • webmail.sonic.net
  • forums.sonic.net
  • wiki.sonic.net
  • corp.sonic.net
  • newsignup.sonic.net
  • public-api.sonic.net (used by mobile apps)
  • fusionbroadband.com (used by our wholesale partners and customers)
  • srapi.sonic.net (used by our wholesale partners)

All of these systems have had their certificates replaced except for the following which are still pending reissue by our CAs:

UPDATE: All systems have had their keys replaced.

  • legacy-webmail.sonic.net
  • wiki.sonic.net
  • forums.sonic.net

The following sites and services were not vulnerable due to running an early version of OpenSSL:

  • mx.sonic.net (inbound mail)
  • members.sonic.net
  • signup.sonic.net
  • listman.sonic.net

Pop/Imap servers key update.

April 9, 2014 – 5:25 pm by Grant Keller

In a few moments we will be updating the Pop and Imap servers’ ssl keys due to the Heartbleed bug. Fortunately the way our servers are configured they where not vulnerable  to exposing authentication information due to this bug, but there is a chance that the keys where compromised. Expect a brief interruption while the keys are installed, after which incoming mail service will be restored.

 

- Grant and SOC

Webmail.sonic.net Certificate update

April 9, 2014 – 2:01 pm by Grant Keller

UPDATE: The new certificate has been installed.

In the next few minutes we will be pushing a new SSL certificate to our webmail servers. There will be a brief service interruption while this is done. This is one of the last few steps required in cleaning up after the Heartbleed openssl vulnerability.

- Grant and SOC

Server Updates

April 8, 2014 – 5:13 pm by joemuller

Update:

All work is complete as of 1:00am on Wednesday, April 9th.

Tonight, April 8th, at 11:59pm, System Operations will be updating and rebooting several systems, including load balancers. This may cause a brief interruption in any web-based services and mail.

– Joe and the SOC

OpenSSL Heartbleed Bug

April 7, 2014 – 10:20 pm by Kelsey

A serious bug in OpenSSL was announced this afternoon known as the Heartbleed Bug.  An attacker, armed with the ability to exploit this bug is able to remotely read the contents of the memory of a vulnerably server.  This exposes the potential for an attacker to acquire the private key used to both encrypt the traffic and identify the server allowing them to eavesdrop on traffic as well as impersonate the server.  For a more in depth explanation of the bug and its affects see heartbleed.com  We have updated our servers with a local version of OpenSSL that disables Hearbeats to prevent an exploit pending new packages released by our OS upstream which fully resolves the issue.   -Kelsey

Update: April 8th, 17:35.  All affected public web and application servers received the fix from our OS upstream shortly after the original MOTD was posted yesterday.  Today, we’ve worked on wrapping up the upgrades on less critical systems and have reissued certificates for the bulk of the systems which had potentially exposed private keys. Ironically, we’re still waiting for all of our EV certs to be reissued.  The severity of this exploit can’t be underestimated as even earlier today Yahoo’s servers we’re still vulnerable exposing user names and passwords for the taking with little effort.  All users who run secure services should ensure that their systems are properly patched and consider having their certificates reissued by their CA.  -Kelsey and Grant

ATM Customer Aggregation Router Reload

April 7, 2014 – 10:58 am by tomoc

Tonight, April 7, starting at 11:59PM, we will be performing a maintenance reload on our customer facing ATM aggregation routers. This will result in 5-10 minutes of downtime for Business-T and FRATM customers.

-Tomoc