The system design and the work today was supervised by Russ Irving, our staff power system expert. The work was accomplished without any interruption in service to our datacenter. During the transition, we had a second standby generator and transfer switch wired in to the power and cooling systems via a carefully orchestrated process.

This equipment uses outside air to cool and compress freon, which is then piped inside to large air handlers in the data center. See our cooling system video for an overview.

Each of the cooling towers has 200 tons of cooling capacity, for a total of 400 RT (refrigeration tons) in the system. Either of the two towers can accommodate our current data center cooling requirement, so we have redundant capacity, allowing for failure or maintenance.

What’s 200 tons of cooling capacity mean? From Wikipedia:

The unit ton is used in refrigeration and air conditioning to measure heat absorption. Prior to the introduction of mechanical refrigeration, cooling was accomplished by delivering ice. Installing one ton of refrigeration replaced the daily delivery of one ton of ice.

In North America, a standard ton of refrigeration is 12,000 BTU/h = 200 BTU/min ≈ 3,517 W. This is approximately the power required to melt one short ton (2,000 lb) of ice at 0 °C in 24 hours, thus representing the delivery of 1 ton of ice per day.

So, if I’m doing my math right, with both towers at full capacity we could chill the equivalent of about one point four million watts. That would be an awful lot of trucks full of ice!

Related articles

• Cooling is key

Here are a couple examples of some simple cable lacing used to manage a set of large cables. Yup, I tied these myself. The third image is a cable end itself, with an AMP Champ tool and the connector just after termination.

Check out Wikipedia’s cable lacing article out for more infomation. For additional info, tools and supplies (you can do this at home!), see Tecra Tools.

Lifting cabinets into SNFCCA06 CO

This was a tricky CO that was held up because there is no freight elevator, and the staircase is just too tight to get the cabinet up. These seismic rated cabinets weight in at around 300 pounds with the welded frame only, so they are a bit of work to install. Once each cabinet is bolted down, the solid sides and locking doors are installed.

Today, the build-out of Fusion and FlexLink in San Francisco is over half complete. If you would like service in San Francisco for delivery within the next month, you can order Fusion (residential or small biz ADSL2+) or order FlexLink (business ADSL2+, T1 and Ethernet).

The new custom engineered breaker panel board arrived this week, and we now have most of the components on site. Construction has begun on the physical mounting of the equipment in our power room. We are excited about the new power delivery capacity that this project will provide, allowing for over double our current power load.

If you’re interested in seeing the images in the gallery below, you can click for a medium sized version, then click on the medium one for full size.

Efficient datacenter cooling results in reduced costs

Our green datacenter cooling system has established a great track record since deployment. Our 2008 total utility costs are projected to come in very near 2006 levels, despite huge growth of equipment in the datacenter.

So, more servers in 2008, but far less power used to cool them. That is green and cost effective! Our growth in power consumption is ongoing, but the trend line has taken a nice step downward due to the investment in efficiency.

For more info on the innovative Bell Products Core4 system at Sonic.net, see this article.

For those interested in the technical reason for the failure, during the multiple power transitions from utility to generator and back, the variable frequency drives (VFDs) on the four redundant air handlers sensed an over-voltage condition and shut down to protect themselves. To address this, they have now been re-configured; if they have a failure now, they will wait eighty seconds for power to stabilize and re-start automatically.

The interesting thing though was that this presented an opportunity to see what really happens in a large datacenter without AC for a brief period of time. Total cooling downtime was 15 30 minutes, and during that time, the temperature rose 15 degrees. The room is typically kept at 69 degrees fahrenheit, so this pushed the ambient room temperature to about 85.

Meanwhile, in-cabinet temperatures for cabinets with a lot of equipment in them nearly touched 100 degrees F. That’s just ten to twenty degrees prior to when we expect equipment to begin failing, so this was a close call for us.

Datacenters are challenging environments to design. You need fully physically redundant Internet connections, plus fire suppression, physical and electronic security, power backup and redundant cooling. We’re very pleased with the efficiency of our new AC system and it’s VFDs, and it’s clear how critical it is from this incident.

Negative Exposure Assessment Drilling Protocol Training

I wonder what the people at the grocery store thought when staff member Matt Merner purchased more than twenty cans of shaving cream. I suspect they guessed he was planning some sort of late night pranks.

That would be an incorrect assumption. Matt was actually preparing for our upcoming telco central office installations.

To prepare for the deployment of our equipment in eighteen additional new telco central offices, a group of our staff was trained in negative exposure assessment procedures. This is a mandatory process for our handling of floor tiles which contain or which are presumed to contain asbestos.

Many telco central offices were built before the hazards of asbestos were well known. As a result, the buildings contain insulation, fireproofing, pipe coverings, conduits and other products which may contain asbestos. The item we deal with is the floor tiles, which we drill through in order to mount our equipment cabinets on the concrete floor.

The solution: Better buy Barbasol!

Staff member Clay Carley is our resident expert, an officially certified “competent person”. Asbestos handling is obviously heavily regulated (learn more here), and Clay’s been through the required training for this task. Clay trains the staff members who will be working with or near the process. (This includes curious CEOs.)

The basic process involves lots of wet paper towels, shaving cream and a few Ziplock baggies. The shaving cream forms a barrier, capturing asbestos fibers which may be released by the drilling process. Really high tech.

Construction in Sebastopol is scheduled to begin next week. To keep the floor smooth and soft, we’ve opted for Barbasol with Aloe.

Related

• Article: What Are The Risks Of Asbestos Exposure?
• Video: How Barbasol makes shaving cream.

Image via Wikipedia

Obviously, an ISP doesn’t function without electricity, so we’ve got big investments in redundant power here.

A datacenter power system consists of multiple inputs which are arbitrated by a transfer switch, and multiple loads such as UPS systems and air conditioners (CRACs).

The primary input is PG&E, and the transfer switch monitors the quality of this input. If the utility power goes offline or fades, the transfer switch sends a signal to the starter on the generator, which powers up automatically. Once the generator power output is online and stable, which typically takes twenty to thirty seconds, the transfer switch physically swings a huge set of contacts over to the new input, transferring the load.

The UPS systems and their batteries carry the datacenter computing load during this startup and transfer, while CRAC loads are dropped during the transition. A datacenter can’t function for long without cooling, so the entire generator and transfer switch system must function as designed in order to stay online.

The generator itself is the really cool bit of this whole setup. For those who are into engines, it’s a 24 liter V-12 Detroit Diesel, with twin turbochargers. That’s a full two liters of per cylinder – imagine a piston and cylinder the size of a 2 liter soda bottle. Now, gang up twelve of them. It’s a huge engine. At full throttle it generates over one thousand horsepower, and three quarters of a megawatt of power.

In our five years at our Apollo Way location, the generator has only been called on to respond to a power outage twice. PG&E has done a great job for us, delivering quite reliable power. But, we still must test fire the generator once every week, top up it’s fuel every few months, and trade out old fuel for new periodically. It’s full generating capacity is totally load tested every few years by hooking it up to a massive resistor/heater bank. The maintenance and load testing is critical to assure that the power will be there when we do need it.