Sonic.net CEO Blog

Sebastopol construction

The first thing that comes to mind when I try to describe the Sebastopol central office is “cute”. It reminds me of Sonic.net’s old datacenter; crowded, small, but with purpose. It’s a plucky little CO.

Yesterday we began the installation process in Sebastopol for our next generation products: FlexLink for medium sized businesses and enterprise and Fusion for small business and home. The first cabinet went in (low ceiling!), and the drilling has begun. Two holes down this afternoon, two to go.

We should be bolted down tomorrow, then equipment rack-in, power turn-up, and copper termination. The last part will probably take the longest, as we are pre-building a lot of copper capacity, and it is all done by hand.

Posted in Access | 3 Comments »

One nice new feature

Click for the comic intro to the Chrome browser

For a number of years, my desktop PC has spent most of it’s time running just three basic programs. A web browser, an email client and a ssh client (unix remote terminal).

So does it matter what the OS is? Not really, as long as it’s reliable and secure. What matters is those three programs.

I’ve been using Firefox and Thunderbird as my primary applications for some time, and I have a love/hate relationship with each. They’ve both got shortcomings and real issues, but so far, they’re each the best option.

With the browser being one of these important applications, I am always interested when something new comes along to shake up the status quo. Without new entrants, there is little innovation, so Google’s Chrome browser is an interesting new entry. Alongside the IE/Firefox duopoly, it’s not the only other browser, there are Safari and Opera - that’s about all that comes to mind.

But, Chrome has Google behind it - and they have been thinking of the web itself as applications for a long time. This concept, plus a tidy interface and some nice features look like a win so far.

My favorite feature so far is ability to grab a tab and tear it off into a new window. In IE and Firefox, you’re stuck in the application, viewing one tab at a time. When you are watching something (a video, for example) and want to view it while doing something else, you’ve got to open a whole new window, move that URL in, and start the video over.

Chrome lets you grab any tab and drag it out, creating a new window, without interruption of the video or application. Perfect! I’ve got my windows concept back and can juggle “applications” (web pages) however I like on my desktop!

Chrome is beta, but you can use it alongside your current browser without any problems. Play with both, compare, and use what you like best. Then, as new releases of Firefox, Safari, etc come out, try them too - hopefully we will continue to see innovation driven by competition.

Related articles elsewhere:

• 7 Really Awesome Things About Google Chrome
• Images: How Google’s Chrome browser shines
• Google Chrome Features That Are Missing in your Favorite Web Browser

Posted in Software | 4 Comments »

Close shave

Negative Exposure Assessment Drilling Protocol Training

I wonder what the people at the grocery store thought when staff member Matt Merner purchased more than twenty cans of shaving cream. I suspect they guessed he was planning some sort of late night pranks.

That would be an incorrect assumption. Matt was actually preparing for our upcoming telco central office installations.

To prepare for the deployment of our equipment in eighteen additional new telco central offices, a group of our staff was trained in negative exposure assessment procedures. This is a mandatory process for our handling of floor tiles which contain or which are presumed to contain asbestos.

Many telco central offices were built before the hazards of asbestos were well known. As a result, the buildings contain insulation, fireproofing, pipe coverings, conduits and other products which may contain asbestos. The item we deal with is the floor tiles, which we drill through in order to mount our equipment cabinets on the concrete floor.

The solution: Better buy Barbasol!

Staff member Clay Carley is our resident expert, an officially certified “competent person”. Asbestos handling is obviously heavily regulated (learn more here), and Clay’s been through the required training for this task. Clay trains the staff members who will be working with or near the process. (This includes curious CEOs.)

The basic process involves lots of wet paper towels, shaving cream and a few Ziplock baggies. The shaving cream forms a barrier, capturing asbestos fibers which may be released by the drilling process. Really high tech.

Construction in Sebastopol is scheduled to begin next week. To keep the floor smooth and soft, we’ve opted for Barbasol with Aloe.

Related

• Article: What Are The Risks Of Asbestos Exposure?
• Video: How Barbasol makes shaving cream.

Posted in Facilities | 1 Comment »

Slaughtering the hogs

How much Internet is too much? Apparently it’s 250 gigabytes, enough Internet content to fill up a $55 hard disk drive.

Comcast made news today by announcing a usage cap for Internet users. You can read more about it at PC Magazine. See also the DSLReports coverage.

The reason for the cap isn’t economical, it’s technical. In a shared physical topology, there must be management of usage in order to prevent performance problems due to congestion. Cable networks today are Hybrid Fiber Coax (HFC) networks, where the video and data is carried on fiber to distribution nodes which serve 500 to 2000 homes. All of these homes are on a common coaxial cable network, and share the capacity of the network. An image is worth a thousand words, so please view a simple HFC network diagram now.

This is a bit before my time, but a cable network is like a telephone party line. Common until around the 1940s, shared party line telephone service was how most homes received telephone service. It was cost effective because they didn’t have to run wire from every home back to the central office. Instead, it ran from house to house, so the circuit was shared. Telephone companies abandoned party line configurations over fifty years ago, and this has given them/us a big edge over cable.

For the last few years, Comcast has managed heavy usage by warning customers who used “too much”, without defining what too much was. This practice has been called an “invisible cap”. My guess is that this invisible cap is actually more effective than one which is well defined and documented. If you know how much you’re allowed to use, it’s possible to use bandwidth monitoring software to run up to but not over the limit. When the limit was unknown, users simply lived in fear and would presumably curtail their usage. Notably, users in locations where Comcast was the only broadband option would really be motivated to avoid getting the boot, as they’d have nowhere else to go for broadband Internet.

Of course, telco’s response was to point this weakness out in the ads many of us fondly remember. PacBell was criticized by cable companies who claimed they really didn’t have a problem. In fact they do, and the issue then and now is the same, and caps are the only real solution. (see that diagram again, and think “shared network”. There’s only so much to go around between the 500-2000 homes.)

The other solution they tried was filtering peer to peer traffic. They got in trouble with the FCC for this, and the new openly documented and disclosed cap is the result. Also notable, the illegal filtering of P2P traffic by Comcast is what really kicked the net neutrality cause into high gear. The folks at Save the Internet are continuing to fight for uniform and unfettered Internet access.

So, sit back and relax, watch the old PacBell ads and enjoy the trip down memory lane. Comcast has finally stuck the pigs, and they sure are squealing!

Related articles

• It’s official: Comcast starts 250GB bandwidth caps October 1
• Comcast to dam and damn the Internet with usage caps
• FCC to Comcast: Throttling Broadband is Unlawful
• Groups cheer reports of FCC acting against Comcast
• Comcast: The More You Surf, The Slower Your Speed

Posted in Access | 8 Comments »

Punt!

Sonic.net and BroadLink staff re-engineer BL network

As mentioned in the status blog (formerly system MOTD), BroadLink Communications had a system failure in a critical component of their network today. A hard drive in their core router failed, and while they do have spare parts, they were unable to rebuild it.

BroadLink’s wireless towers in Santa Rosa continue to serve a small number of customers, many of whom still cannot obtain DSL broadband service due to their locations. While BroadLink is a business in decline, it does provide a very valuable service to some under served locations.

Designed over ten years ago, the network is based upon 802.11 wireless access points which serve intelligent Linux CPE at customer premises. The network’s core converts bridged Ethernet to ATM PVCs, one for each customer. (A PVC is a permanent virtual circuit - think of it as a unique customer’s connection inside a larger pipe.) The ATM is fed via a T3 to the Sonic.net network, where we provide IP routing to the Internet. Provisioning and automated management of the per customer configurations is all via the ATM PVCs. This system was designed prior to the viability of more modern solutions like dot1q vlan tagging, PPPoE or MAC RADIUS auth, which could provide similar functionality. The ATM PVC configuration was a very innovative and tidy solution, and if anything, it was before it’s time.

So, in summary we’ve got Ethernet on the WAN (the wireless), which feeds into a magic one of a kind box (The “Red-C”) that converts it into ATM and PVCs. It’s handed off to Sonic.net, where we terminate the ATM into a Redback SMS, basically in the same way that a DSL customer is provisioned. In this way, the hundreds of wireless customers are managed in our systems in the same way that nearly 50,000 DSL loops are managed. Out the other side: Ethernet. Hmm. Basically exactly what went in, but with per-customer provisioning, locking, diagnostics and management in the middle.

The Red-C died - hard drive failure. So, we pulled it and the Redback out, and are simply routing BroadLink’s entire bridged network to the Internet. See the pic for the combined Sonic.net and BroadLink response team working on these changes to the network this afternoon.

This change is transparent to the end-user, but not really sustainable for the long term due to the inability to provision, diagnose, lock, disconnect and manage individual end-users. But, everyone is online, and we will address the bigger picture another day.

The photo includes, top to bottom, Jason Kane, Sonic.net wireless product manager, Tim McAllister, BroadLink board member, Nathan Patrick, Sonic.net network architect, Scott Woods, former BroadLink engineer (Thanks Scott!) and Josh, BroadLink’s technician. Unfortunately, CEO Warren Linney is at Burning Man and is unreachable. Warren, I hope you remembered your googles, water, sunscreen, and a spare cloned backup hard disk drive!

Related articles by Zemanta

• Hold Onto that 56k Modem, Trust Me

Posted in Access | 1 Comment »

Critical systems: Power backup

Image via Wikipedia

Obviously, an ISP doesn’t function without electricity, so we’ve got big investments in redundant power here.

A datacenter power system consists of multiple inputs which are arbitrated by a transfer switch, and multiple loads such as UPS systems and air conditioners (CRACs).

The primary input is PG&E, and the transfer switch monitors the quality of this input. If the utility power goes offline or fades, the transfer switch sends a signal to the starter on the generator, which powers up automatically. Once the generator power output is online and stable, which typically takes twenty to thirty seconds, the transfer switch physically swings a huge set of contacts over to the new input, transferring the load.

The UPS systems and their batteries carry the datacenter computing load during this startup and transfer, while CRAC loads are dropped during the transition. A datacenter can’t function for long without cooling, so the entire generator and transfer switch system must function as designed in order to stay online.

The generator itself is the really cool bit of this whole setup. For those who are into engines, it’s a 24 liter V-12 Detroit Diesel, with twin turbochargers. That’s a full two liters of per cylinder - imagine a piston and cylinder the size of a 2 liter soda bottle. Now, gang up twelve of them. It’s a huge engine. At full throttle it generates over one thousand horsepower, and three quarters of a megawatt of power.

In our five years at our Apollo Way location, the generator has only been called on to respond to a power outage twice. PG&E has done a great job for us, delivering quite reliable power. But, we still must test fire the generator once every week, top up it’s fuel every few months, and trade out old fuel for new periodically. It’s full generating capacity is totally load tested every few years by hooking it up to a massive resistor/heater bank. The maintenance and load testing is critical to assure that the power will be there when we do need it.

Posted in Facilities | 1 Comment »

Next generation product pricing

Fusion Bundle Logo Concept

As discussed previously, we have been working for some time toward the launch of new next generation products. As we get nearer deployment, some of the details are firming up.

First, bundling. This is a hot topic - some customers really like bundles, and some really do not. We believe in providing as many options as we can, so our next generation products will be available both with and without other services bundled. Of note, you do NOT need to have a voice telephone service for these products, and in fact at this point our initial offering does not include voice. The voice offering is likely to arrive sometime late this year.

Second, a name. Our current tentative name for the family of products is “Fusion”. Maybe that’s “Sonic.net Fusion Broadband Internet”, or “Fusion: Next Generation Products”, etc - it’s a working concept at this point. The Fusion concept encompasses all of the products that will be available. If we stick with this name, the product is likely to have an atom logo, where each electron in orbit represents an additional bundled product. Opt for broadband only, or add in voice, TV, or mobile. Each adds a ring to the orbit.

Finally and most important, pricing. Here are the initial launch products and prices. Note that these are standalone, delivered on a dedicated copper pair, so unlike today’s DSL, you don’t need to have a voice line and associated costs.

Residential locations, dynamic IP:

• 6 Mbps/1 Mbps $45/mo
• 10 Mbps/1 Mbps $65/mo
• 18 Mbps/1 Mbps $80/mo

Residential locations, static IP:

• 6 Mbps/1 Mbps $70/mo
• 10 Mbps/1 Mbps $90/mo
• 18 Mbps/1 Mbps $105/mo

Business locations, dynamic IP:

• 6 Mbps/1 Mbps $70/mo
• 10 Mbps/1 Mbps $90/mo
• 18 Mbps/1 Mbps $105/mo

Business locations, static IP:

• 6 Mbps/1 Mbps $80/mo
• 10 Mbps/1 Mbps $100/mo
• 18 Mbps/1 Mbps $115/mo

Bundling offers the opportunity to drive costs downward - for example, adding voice service (when available) reduces the monthly cost of both products by a combined total of $20/mo. Adding television saves another $10/mo. At this time, bundle savings for adding mobile have not been set.

Product speeds are tiered based upon the capabilities of the loop itself. So for example, the max downstream speed of the 6/1Mbps product is between 4-6Mbps, the 10/1 between 7-10Mbps, 18/1, 11-18Mbps. Maximum speed is based upon the line’s electrical capability to carry ADSL2+ data. This rate of speed will be faster than legacy ADSL1 would be for the same CO based loop.

For customers near downtown Santa Rosa, these products will be available in just a couple weeks. About ten additional cities plus expanded Santa Rosa coverage will arrive in the coming months.

Oh, and yes, the free clip art atom that I’ve used here has one too many electrons in orbit. The max would be four. Broadband, voice, TV and mobile.

Posted in Access, Mobile, TV, Voice | 38 Comments »

New UPS on site

If you think of a UPS (uninterruptible power supply) as just a small box that sits under your desk, you have got to see the monsters we use in the datacenter.

Sonic.net took delivery today of the first two boxes that make up the UPS portion of our new 500KVA Mitsubishi monster. Total delivered weight, 7200 pounds - nearly four tons. The heavier of the two containers by itself was over 4000 pounds. Here’s a pic of the staff and driver getting ready to roll it out on our dock. The batteries and battery cabinets, which are very heavy (the batteries are mostly lead) will be here another day.

The Mitsubishi offers 450KVA of capacity at it’s rate power factor of 90%. This gives us lots of room to grow. Today our Leibert provides 104KVA, and our PowerWare 128KVA. Together these two are just about 1/2 the size of the Mitsubishi. Like I said, lots of room to grow.

Congrats on this next stage of the project to Russ Irving and Kelsey Cummings. It’s been a long term project, and it’s great to see hardware on site.

Next up, waiting on the city permit, structural engineering, rigging and finally turn-up.

Posted in Facilities | 1 Comment »

Opinions on personalized email domains?

I’m interested in feedback on a concept.

We are researching the idea of offering personalized surname based email and web addresses. This would mean that a customer could select an email address like firstname@lastname.com, and an optional web address of firstname.lastname.com for example.

Each would forward to a customer’s usual mailbox or website, so there would be be no change to existing addresses or settings - just a new personalized address in addition.

Over 35,000 surnames are available, matching over 60% of individuals.

What do you think? Is a surname based email address and web address worth a few bucks a month? Comment here with your opinion please!

Posted in Uncategorized | 12 Comments »

More bees!

As I’ve written previously, I keep bees. Yesterday I got a call from Oak Grove Elementary School in Graton about a swarm of bees in one of their trees. The nice folks at Beekind referred them because Sonic.net has a bucket truck which I can borrow, so we can reach swarms which have settled high in a tree.

I’ve captured two swarms already this season, and have three hives running now, and that’s about as many as I want to manage. So, I called my friends John and Chris Mason at Emtu Wines. They lost both of their hives last winter, presumably to colony collapse.

I sent John up in the bucket this time, while Chris and I observed from below. The swarm wasn’t nearly as large as the one from Occidental, perhaps three pounds of bees, about 10,000. Boxed up and away they went, into one of the empty Emtu hives. Hopefully the bees will be successful there, and none of the elementary school kids will risk anaphylaxis.

Posted in Uncategorized | No Comments »