In disappointment, there is opportunity. . .

I was personally greatly disappointed with the news coming out of last week that the Uptime Institute had branded Microsoft and Google as the enemy to traditional data center operators.  To be truthful, I did not give the reports much credit especially given our long and successful relationship with that organization.  However, when our representatives to the event returned and corroborated the story, I have to admit that I felt more than  a bit let down.

As reported elsewhere, there are some discrepancies in how our mission was portrayed versus the reality of our position.   One of the primary messages of our cloud initiatives is that there is a certain amount of work/information that you will want to be accessed via the cloud, and there is some work/information that you want to keep privately.  Its why we call it SOFTWARE + SERVICES.  There’s quite a few things people just would not feel comfortable running in the cloud.   We are doing this (data center construction and operation)  because the market, competitive forces, and our own research is driving us there.   I did want to address some of the misconceptions coming out of that meeting however:

On PUE, Measurement, and our threat to the IT industry

The comments that Microsoft and Google are the biggest threat to the IT industry and that Microsoft is “making the industry look bad by putting our facilities in areas that would bring the PUE numbers down” are very interesting.  First as mentioned before, please revisit our Software + Services strategy, its kind of hard to be a threat if we are openly acknowledging the need for corporate data centers in our expressed strategy.   I can assure you that we have no intention of making anyone look “bad”, nor do we in any way market our PUE values.  We are not a data center real estate firm and we do not lease out our space where this might even remotely be a factor. 

While Microsoft believes in Economization (both water and air-side), not all of our facilities employ this technology.  In fact, if a criticism does exist its that we believe that its imperative to widen your environmental envelopes as open as you can.  Simply stated – run your facilities hotter!

The fact of the matter is that Microsoft has invested in both technology and software to allow us to run our environments more aggressively than a traditional data center environment.   We understand that certain industries have very specific requirements around the operation of storage of information which drive and dictate certain physical reliability and redundancy needs.   I have been very vocal around getting the Best PUE for your facility.  Our targets are definitely unrealistic for the industry at large but the goal of driving the most efficiency you can out of your facilities is something everyone should be focused on.

It was also mentioned that we do not measure our facilities over time which is patently untrue.   We have years and years worth of measured information for our facilities with multiple measurements per day.  We have been fairly public about this and have produced specifics on numbers (including the Uptime Symposium last year) which makes this somewhat perplexing. 

On Bullying the Industry

If the big cloud players are trying to bully the industry with  money and resources, I guess I have to ask – To what end?  Does this focus on energy efficiency equate to something bad?  Aside from the obvious corporate responsibility of using resources wisely and lowering operating costs, the visibility we are bringing to this space is not inherently bad.  Given the energy constraints we are seeing across the planet, a focus on energy efficiency is a good thing. 

Lets not Overreact, There is yet hope

While many people (external and internal) approached me about pulling out of the Uptime organization entirely or even suggesting that we create a true non-for-profit end user forum, motivated by technology and operations issues alone, I think its more important to stay the course.   As an industry we have so much yet to accomplish.  We are at the beginning of some pretty radical changes in both technology, operations, and software that will define our industry in the coming decades.   Now is not the time to splinter but instead redouble our efforts to work together in the best interests of all involved.

Instead of picking apart the work done by the Green Grid and attacking the PUE metric by and large, I would love to see Uptime and Green Grid working together to give some real guidance.  Instead of calling out that PUE’s of 1.2 are unrealistic for traditional data center operators, would it not be more useful for Uptime and Green Grid to produce PUE targets and ranges associated with each Uptime Tier?   In my mind that would go along way to drive the standardization of reporting and reduce ridiculous marketing claims of PUE.

This industry is blessed with two organizations full of smart people attacking the same problem set.  We will continue our efforts through the Microsoft Data Center Experience (MDX) events, conferences, and white-papers to share what we are doing in the most transparent way possible.


Out of the Box Paradox – Manifested (aka Chicago Area Data Center begins its journey)

Comment on Microsoft’s official launch of the Chicago facility and the announcement of another Microsoft Data Center Experience Conference in the Chicago facility.


With modern conventional thinking and untold management consultants coaching people to think outside the box, I find it humorous that we have actually physically manifested an “Out of the Box Paradox” in Chicago.  

What is an Out of the Box Paradox you ask?  Well I will refer to Wikipedia on this one for a great example:

“The encouragement of thinking outside the box, however, has possibly become so popular that thinking inside the box is starting to become more unconventional.  This kind of “going against the grain means going with the grain” mentality causes a paradox in that there may be no such thing as conventionality when unconventionality becomes convention.”

The funny part here is that we are actually doing this with….you guessed it…..boxes. Today we finished the first phase of construction and we are rolling into the testing of container-based deployments.  Our facility in Chicago is our first purpose-built data center to accommodate containers on a large scale.  It has been an incredibly interesting journey.  The challenges of solving things that have never been done before are many.  We even had to create our own container specification, one specifically with the end-user in mind to ensure we maximized the cost and efficiency gains possible, not to mention standard blocking and tackling issues like standardizing power, water, network and other interfaces.  All sorts of interesting things have been discovered, corrected, and perfected.  From electrical harmonics issues to streamlining materials movement, to whole new operational procedures.

Chicago Container Spaces with load banks

The facility is already simply amazing and it’s a wonder to behold. Construction kicked off only one year ago and when completed it will have the capacity to scale to hundreds of thousands of servers which can be deployed (and de-commissioned as needed) very quickly.  The joke we use internally is that this is not your mother’s data center.  You get that impression from the first moment you step into the “hangar bay” on the first floor. The “hangar’s” first floor will house the container deployments and I can assure you it is like no data center you have ever seen.  It’s one more step to the industrialization of the IT world, or at least the cloud-scale operations space.  To be fair, and it’s important to note, only one half of the total facility is ready at this point, but even half of this facility is significant in terms of total capacity.

That “Industrialization of IT” is one of the core tenets of my mission at Microsoft. Throwing smart bodies at dumb problems is not really smart at all. The real quest is how to drive innovation and automation into everything that you do to reduce the amount of work that needs to be performed by humans.  Dedicate your smart people for solving hard problems.  It’s more than a mission, it’s a philosophy deeply rooted in our organization.  Besides, industry numbers tell us that humans are the leading cause of outages in data center facilities. 🙂 Our Chicago facility is a huge step forward to driving that industrialization increasingly forward.  It truly represents an evolution and demonstrates what could happen when you blend the power of software and breakthrough innovative design and engineering. Even for buildings!

 Chicago Container Spines being constructed

I have watched with much interest the back and forth on containers in the media, in the industry, and the interesting uses being proposed by the industry. The fact of the matter is that Containers are a great “Out of the Box Paradox” that really should not be terribly shocking to the industry at large. 

The idea of “containment” is almost as old as mechanical engineering and thermodynamics itself. Containment gives you the ability to manage the heat or lack thereof more effectively in individual ecosystems. Forward looking designers have been doing “containment” for a long time. So going back to the paradox that “out of the box, is in the box thinking” shift, the concept is not terribly new.  It’s the application at our scale and specifically to the data center world which is most interesting.  

It allows us to get out of the traditional decision points common to the data center industry in that certain infrastructure decisions actually reside in the container itself, which allows for a much quicker refresh cycle of key components and the ability to swap out for the next greatest technology rapidly.  Therefore, by default it allows us to deploy our capital infrastructure costs much more closely aligned with actual need versus the large step functions one normally sees in data center construction (build a large expensive facility, and fill it up over time versus build capacity out as you need it).   This allows you to better manage costs, better manage your business, and give you the best possible ramp for technology refresh.  You don’t particularly care if its AC or DC, if it’s water cooled or air cooled.  Our metrics are simple – Give us the best performing, most efficient, lowest TCO technology to meet our needs. If today that’s AC, great.  Tomorrow DC?  Fantastic.  Do I want to be able to do a bake-off between the two?  Sure. I don’t have to reinvest huge funds in my facilities to make those changes. 

For those of you with real lives and have not been following the whole container debates here is a quick recap –

  1. Microsoft is using standard 40 foot shipping containers for the deployment of servers in support of the software + services strategy and in support of our cloud services infrastructure initiatives.
  2. The containers can house as many as 2500 servers achieving a density of 10 times the amount of compute in the equivalent space in a traditional data center.
  3. We believe containers offer huge advantages at scale in terms of both initial capital and ongoing operating costs.
  4. This idea has met some resistance in the industry. As highlighted by my interesting back and forth with Eric Lai from Computerworld magazine. Original article can be found here, with my “Anthills” response found here.
  5. Chicago represents one of the first purpose-built container-built facilities ever.

To be clear, as I have said in the past, containers are not for everyone, but they are great for us.

The other thing which is important is the energy efficiency of the containers. Now I want to be careful here as the reporting of efficiency numbers can be a dangerous exercise in the blogo-sphere. But our testing shows that our containers in Chicago can deliver an average PUE of 1.22 with an AVERAGE ANNUAL PEAK PUE of 1.36. I break these two numbers out separately because there is still some debate (at least in the circles I travel in) on which of these metrics is more meaningful.  Regardless of your position on which is more meaningful, you have to admit those numbers are pretty darn compelling. 


For the purists and math-heads out there, Microsoft includes house lighting and office loads in our PUE calculation. They are required to run the facility so we count them as overhead.

On the “Sustainability” side of containers it’s also interesting to note that shipping 2500 servers in one big container has a positive reduction on the CO2 related to transportation, let alone the amount of packaging material eliminated.

So in my mind, containers are driving huge cost and efficiency (read also as cost benefits in addition to “green” benefits) gains for the business.  This is an extremely important point, as Microsoft expands its data center infrastructure, it is supremely important that we follow an established smart growth methodology for our facilities that is designed to prevent overbuilding—and thus avoid associated costs to the environment and to our shareholders.  We are a business after all.  We must do all of this while also meeting the rapidly growing demand for Microsoft’s Online and Live services.

Containers, and this new approach is definitely a change in how facilities have traditionally been developed, and as a result many people in our industry are intimidated by it.  But they shouldn’t be. Data center’s have not changed in fundamental design for decades.  Sometimes change is good. The exposure to any new idea is always met with resistance, but with a little education things change over time.

In that vein we are looking at holding our second Microsoft Data Center Experience (MDX) event in Chicago in the Spring/Summer 2009.  Our first event held in San Antonio, was basically an opportunity for a couple hundred Microsoft enterprise customers to tour our facilities, ask all the questions they wanted, interact with our Data Center experts (mechanical, electrical, operations, facilities management, etc.), and generally get a feel to our approach. It’s not that ours is the right way, or the wrong way…..just our way.  Think of an Operations event for Operations people, by Operations people. 

It’s not glamorous, there are no product pitches, no slick brochures, no hardware hunks or booth babes, but hopefully it’s interesting.  That first event was hugely successful with incredible feedback from our customers. As a result, we decided to do the same thing in Chicago with the very first container data center.  Which of course makes things a bit tricky.  While the facility will be going through a vigorous testing phase from effectively now moving forward, we thought it better to ensure that any and all construction activity be formally complete before we go moving large groups of people through our facility to ensure safety.  Plus, I don’t think I have enough hard hats and safety gear for you all.  

So if you attended MDX-San Antonio and really want to drill deeper in on Containers, in a facility custom built for them, or would like to attend just to ask questions, look for details on it from your Microsoft account management team or your local Microsoft sales office for details next Spring. (Although it’s not a sales event, you are more likely to reach someone there faster than calling into Global Foundation Services directly, after all we have a global infrastructure to run.)