Through an idea and force of will, he created an industry…

This week the Data Center Industry got the terrible news it knew might be coming for some time.   That Ken Brill, founder of the Uptime Institute had passed away.  Many of us knew that Ken had been ill for some time and although it may sound silly, were hoping he could somehow pull through it.   Even as ill as he was, Ken was still sending and receiving emails and staying in touch with this industry that quite frankly he helped give birth to.  

I was recently asked about Ken and his legacy for a Computerworld article and it really caused me to stop and re-think his overall legacy and gift to the rest of us in the industry.  Ken Brill was a pioneering, courageous, tenacious, visionary who through his own force of will saw the inefficiencies in a nascent industry and helped craft it into what it is today.

Throughout his early career experience Ken was able to see the absolute silo’ing of information, best practices, and approaches that different enterprises were developing around managing their mission critical IT spaces.    While certainly not alone in the effort, he became the strongest voice and champion to break down those walls, help others through the process and build a network of people who would share these ideas amongst each other.  Before long an industry was born.   Sewn together through his sometimes delicate, sometimes not so delicate cajoling and through it all his absolute passion for the Data Center industry at large.

One of the last times Ken and I got to speak in person.In that effort he also created and permeated the language that the industry uses as commonplace.   Seeing a huge gap in terms of how people communicated and compared mission critical capabilities he became the klaxon of the Tiering system which essentially normalized the those conversations across the Data Center Industry.   While some (including myself) have come to think it’s a time to re-define how we classify our mission critical spaces, we all have to pay homage to the fact that Ken’s insistence and drive for the Tiering system created a place and a platform to even have such conversations.  

One of Ken’s greatest strengths was his adaptability.   For example, Ken and I did not always agree.   I remember an Uptime Fellows meeting back in 2005 or 2006 or so in Arizona.  In this meeting I started talking about the benefits of modularization and reduced infrastructure requirements augmented by better software.   Ken was incredulous and we had significant conversations around the feasibility of such an approach.   At another meeting we discussed the relative importance or non-importance of a new organization called ‘The Green Grid’ (Smile)and if Uptime should closely align itself with those efforts.   Through it all Ken was ultimately adaptable. Whether it was giving those ideas light for conversation amongst the rest of the Uptime community via audio blogs, or other means, Ken was there to have a conversation.

In an industry where complacency has become commonplace, where people rarely question established norms, it was always comforting to know that Ken was there acting the firebrand, causing the conversation to happen.   This week we lost one of the ‘Great Ones’ and I for one will truly miss him.  To his family my deepest sympathies, to our industry I ask, “Who will take his place?”

 

\Mm

In disappointment, there is opportunity. . .

I was personally greatly disappointed with the news coming out of last week that the Uptime Institute had branded Microsoft and Google as the enemy to traditional data center operators.  To be truthful, I did not give the reports much credit especially given our long and successful relationship with that organization.  However, when our representatives to the event returned and corroborated the story, I have to admit that I felt more than  a bit let down.

As reported elsewhere, there are some discrepancies in how our mission was portrayed versus the reality of our position.   One of the primary messages of our cloud initiatives is that there is a certain amount of work/information that you will want to be accessed via the cloud, and there is some work/information that you want to keep privately.  Its why we call it SOFTWARE + SERVICES.  There’s quite a few things people just would not feel comfortable running in the cloud.   We are doing this (data center construction and operation)  because the market, competitive forces, and our own research is driving us there.   I did want to address some of the misconceptions coming out of that meeting however:

On PUE, Measurement, and our threat to the IT industry

The comments that Microsoft and Google are the biggest threat to the IT industry and that Microsoft is “making the industry look bad by putting our facilities in areas that would bring the PUE numbers down” are very interesting.  First as mentioned before, please revisit our Software + Services strategy, its kind of hard to be a threat if we are openly acknowledging the need for corporate data centers in our expressed strategy.   I can assure you that we have no intention of making anyone look “bad”, nor do we in any way market our PUE values.  We are not a data center real estate firm and we do not lease out our space where this might even remotely be a factor. 

While Microsoft believes in Economization (both water and air-side), not all of our facilities employ this technology.  In fact, if a criticism does exist its that we believe that its imperative to widen your environmental envelopes as open as you can.  Simply stated – run your facilities hotter!

The fact of the matter is that Microsoft has invested in both technology and software to allow us to run our environments more aggressively than a traditional data center environment.   We understand that certain industries have very specific requirements around the operation of storage of information which drive and dictate certain physical reliability and redundancy needs.   I have been very vocal around getting the Best PUE for your facility.  Our targets are definitely unrealistic for the industry at large but the goal of driving the most efficiency you can out of your facilities is something everyone should be focused on.

It was also mentioned that we do not measure our facilities over time which is patently untrue.   We have years and years worth of measured information for our facilities with multiple measurements per day.  We have been fairly public about this and have produced specifics on numbers (including the Uptime Symposium last year) which makes this somewhat perplexing. 

On Bullying the Industry

If the big cloud players are trying to bully the industry with  money and resources, I guess I have to ask – To what end?  Does this focus on energy efficiency equate to something bad?  Aside from the obvious corporate responsibility of using resources wisely and lowering operating costs, the visibility we are bringing to this space is not inherently bad.  Given the energy constraints we are seeing across the planet, a focus on energy efficiency is a good thing. 

Lets not Overreact, There is yet hope

While many people (external and internal) approached me about pulling out of the Uptime organization entirely or even suggesting that we create a true non-for-profit end user forum, motivated by technology and operations issues alone, I think its more important to stay the course.   As an industry we have so much yet to accomplish.  We are at the beginning of some pretty radical changes in both technology, operations, and software that will define our industry in the coming decades.   Now is not the time to splinter but instead redouble our efforts to work together in the best interests of all involved.

Instead of picking apart the work done by the Green Grid and attacking the PUE metric by and large, I would love to see Uptime and Green Grid working together to give some real guidance.  Instead of calling out that PUE’s of 1.2 are unrealistic for traditional data center operators, would it not be more useful for Uptime and Green Grid to produce PUE targets and ranges associated with each Uptime Tier?   In my mind that would go along way to drive the standardization of reporting and reduce ridiculous marketing claims of PUE.

This industry is blessed with two organizations full of smart people attacking the same problem set.  We will continue our efforts through the Microsoft Data Center Experience (MDX) events, conferences, and white-papers to share what we are doing in the most transparent way possible.

/Mm

DataCenter Think Tanks Sheepish on Weightloss

LoseWeight.jpg

Matt Stansbury over at Data Center Facilities Pro posted an interesting post regarding a panel containing Uptime’s Ken Brill.  The note warns folks on the use of PUE as a benchmarking standard between data centers.

I can’t say I really disagree with what he says. In my mind, self measurement is always an intensely personal thing.  To me, PUE is a great self-measurement tool to drive towards power efficiency in your data center.  Do you include lighting?  Do you include your mechanical systems?  To me those questions are not all that dissimilar to the statement, “I am losing weight”.  Are you weighing yourself nude? in the your underwear?  with your shoes on? 

I do think the overall PUE metric could go a little farther to fully define what *MUST* be in the calculation, especially if you are going to use it comparatively.  But those who want to use this metric in some kind of competitive game are completely missing the point.   This is ultimately about using the power resources you have to its highest possible efficiency.    As I have stated over and over, and as recently as the recent Data Center Dynamics conference in Seattle.  Every Data Center is different.  If I tried to compare the efficiency of one of our latest generation facilities in San Antonio or Dublin to a facility built 10 years ago, assuming we made sure that we were comparing apples to apples with like systems included, of course the latest generation facilities would be better off.   A loss of 5 pounds on an Olympic runner with 4% body fat compared to a loss of 5 pounds on professional sumo wrestler have dramatically different effects (or non effects). 

Everyone knows I am a proponent of PUE/DCiE.  So when you read this understand where my baggage is carried.   To me the use of either, or, or both of these is a matter of audience.   Engineers love efficiency.  Business Managers understand overhead.   Regardless the measurement is consistent and more importantly the measurement is happening with some regularity. This is more important than anything.

If we are going to attempt to use PUE for full scale facility comparison a couple of things have to happen.   At Microsoft we measure PUE aggressively and often.  This speaks to the time element that Ken mentions in his talk in the post.   It would be great for the Green Grid or Uptime or anyone to produce the “Imperial Standard”.  One could even think that these groups could earn some extra revenue by certifying facilities to the “Imperial PUE standard”.  This would include minimum measurement cylces (once a day, twice a day, average for a year, peak for a year, etc).  Heavens knows it would be a far more useful metric for measuring data centers than the current LEEDS certifications. But thats another post for another time.  Seriously, the time element is hugely important.  Measuring your data center once at midnight in January while experiencing the coldest winter on record might make great marketing, but it doesnt mean much.

As an industry we have to face the fact that there are morons amongst us.  This, of course  is made worse if people are trying to advertise PUE as a competitive advantage due mostly to the fact that this means that they have engaged marketing people to “enhance” the message.    Ken’s mention of someone announcing that their PUE of .8 should instantly flag that person as an idiot and you should hand them an Engvallian sign.    But even barring these easy to identify examples we must remember that any measurement can be gamed.  In fact, I would go so far as to say that gaming measurements is the national pastime of all businesses. 

Ultimately I just chalk this up to another element of “Green-washing” that our industry is floating in.

eue

Ken also talks about the use of the word “Power” being incorrect and that because it is a point in time measurement versus an over time measurement and that we should be focused on “Energy”. According to Ken this could ultimately doom the measurement on the whole.  I think this is missing the point entirely on two fronts.   First whether you call it power or energy, the naming semantics dont really matter.  They matter to english professors and people writing white papers, but it terms of actually doing something, it has no effect.  The simple act of measuring is the most critical concept here.   Measure something, get better.   Whether you like PUE, DCIE or whether you want to adopt “Energy” and call it EUE and embrace a picture of a sheep with power monitoring apparatus attached to its back, the name doesnt really matter. )Though I must admit, a snappy mascot might actually drive more people to measure.  Just do something!

My informal polling at speaking engagements continues on the state of the industry and I am sad to say, the amount of people actively measuring power consumption remains less than 10% (let alone measuring for efficiency!), and if anything the number seems to be declining.  

In my mind, as an end-user, the thrash that we see coming from the standards bodies and think tank organizations like Uptime, Green Grid, and others should really stop bickering over whose method of calculation is better or has the best name.  We have enough challenge getting the industry to adopt ANY KIND of measurement.  To confuse matters more and argue the finer points of absurdity is only going to further magnify this thrash and ensure we continue to confuse most data center operators into more non-action .  As an industry we are heading down a path with our gun squarely aimed at out foot.   If we are not careful, the resultant wound is going to end up in amputation.

– MM

Struggling with CADE, McKinsey / Uptime Metric (RE-POST)

percentage.jpg

This is a Re-post of my original blog post on May 5th regarding tortured thoughts around the CADE Data Center Metric put forward by McKinsey. This has relevance to my next post and I am placing it here for your convenience.

 

I guess I should start out this post with the pre-emptive statement that as a key performance indicator I support the use of CADE or metrics that tie both facilities and IT into a single metric.  In fact we have used a similar metric internally at Microsoft.  But the fact is at the end of the day I believe that any such metrics must be useful and actionable.  Maybe its because I have to worry about Operations as well.  Maybe its because I don’t think you roll the total complexity of running a facility with one metric.  In short, I don’t think dictating yet another metric, especially one that doesn’t lend itself to action, is helpful.

As some of you know I recently gave keynote speeches at both DataCenter World and the 2008 Uptime Symposium.  Part of those speeches included a simple query of the audience of how many people are measuring energy efficiency in their facilities.  Now please keep in mind that the combined audience of both engagements numbered between 2000-2400 datacenter professionals.  Arguably these are the 2400 that really view data centers as a serious business within their organizations.  These are folks whose full time jobs are running and supporting data center environments for some of the most important companies around the world.   At each conference less than 10% of them raised their hands.   The fact that many in the industry including Ken Brill at the Uptime Institute, Green Grid, and others have been preaching about measurement for at least the last three years and less than 10% of the industry has accepted this best practice is troublesome.  

Whether you believe in measuring PUE or DCIE, you need to be measuring *something* in order to even get one variable of the CADE metric.  Given this lack of instrumentation and\or process within those firms most motivated to do so speaks in large part of the lack of success this metric is going to have over time.  It therefore follows, if they are not measuring efficiency, they likely don’t understand their total facility utilization (electrically speaking).  The IT side may have an easier way of getting variables for system utilization, but how many firms have host level performance agents in place? 

I want to point out that I am speaking to the industry in general.  Companies like ours who are investing hundreds of millions of dollars get the challenges and requirements in this space.  Its not a nice to have, its a requirement.  But when you extend this to the rest of the industry, there is a massive gap in this space.

Here are some interesting scenarios that when extended to the industry may break or complicate the CADE metric:

  • As you cull out dead servers in your environment, your utilization will drop accordingly and as a result the metric will remain unchanged.  The components of CADE are not independent. Dead servers are removed so that Average server utilization goes up then Data Center Utilization goes down showing proportionally so there is no change and if anything PUE goes up which means the metric may actually go up. Keep in mind that all results are good when kept in context of one another.
  • Hosting Providers like Savvis, Equinix, Dupont Fabros, Digital Realty Trust, and the army of others will be exempt from participating.  They will need to report back of house numbers to the their  customers (effectively PUE).    They do not have access to their customers server information It seems to me that CADE reporting in hosted environments will be difficult if not impossible.  As the design of their facilities will need to play a large part of the calculation this makes effective tracking difficult.  Additionally, overall utilization will be measured at what level?
  • If hosters exempted, then it gives CADE a very limited application or shelf-life.  You have to own the whole problem for it to be effective.  
  • As I mentioned, I think CADE has strong possibilities for those firms who own their entire stack.   But most of the data centers in the world would probably not fall into “all-in” scenario bucket.

I cant help but think we are putting the cart before the horse in this industry.  CADE may be a great way to characterize data center utilization but its completely useless if the industry isnt even measuring the basics.  I have come to the realization that this industry does a wonderful job in telling its members WHAT to do, but lacks to follow-up with the HOW.  CADE is meant for higher level consumption.  Specifically those execs who lack the technical skill-sets to make heads or tails of efficiencies and how they relate to overall operations.   For them, this metric is perfect. But we have a considerable way to go before the industry at large gets there.

Regardless, I strongly suggest each and everyone adopt the take away at Symposium….Measure, measure, measure.