Reflections on my visit to the US Army War College

dl-ar-317.jpg

This week I had the honor and privilege of being a guest at the US Army War College as part of their 2008 Strategy Implementation Seminar.  The program focuses on strategy and leadership to further prepare the graduates for their future roles as leaders of the United States Military.  The program culminates in a special seminar that brings in outside leaders, academics, and essentially a cross section of the American public and exposes the graduating members of the course to the rich diversity of thought found in our society. 

Conversely it is an incredible opportunity to expose those same industry leaders and cross section of the American public to the leadership talent and depth of the military.  While I am certain that the graduates take away interesting nuggets of perspective, information, thoughts, approaches, and methodologies from the guests, I would have to say the education the guests get in return is equally if not more valuable.

This event had a very profound affect on me as a leader, as an individual, and as a United States citizen. 

The dedication, talent, intelligence and sheer mental fortitude present in these students was astounding.  Their grasp of issues as far ranging as economics, politics (both domestic and foreign), law, and of course the military were simply complete. Perhaps more importantly their understanding of the interplay of these topics and the resultant gaps would put most of the people I deal with to shame as ignorant simpletons.

The Seminar brings in a host of authors, strategists, industry and military leaders to a single location for an intense and well regimented program for thought and discourse on a wide variety of topics.  There is a very strong non-attribution component to the conversations which allows the participants, speakers, panel members, and guests to be open and forthright in the exchange of ideas.  It is truly powerful stuff.  The conversations were never timid, always insightful, and you felt that all points of view got an equal share of the spotlight. Something many people might not expect from such an event.

As a leader, the lessons learned from this event will stay with me forever.  While there are always nuggets of leadership theory one can glean from any such effort (and I certainly picked up a few here), the event exposed me to different types of interactions, and greatly increased my ability to manage my ‘situational awareness’. 

I had a great many conversations with students and guests alike.  The interactions with the other guests provided a wonderful opportunity to network with people across many walks of life.  An experience most of us never truly get.  To interact with professors, think tank analysts, politicians, industry leaders, news reporters, all engaged in active debate was personally very gratifying for me. 

More impacting was the interaction with the students. The students were not all Army.  There was representation from the Marines, Navy, Air Force, State Department, and others.   Most of the students I talked to had been to Iraq or Afghanistan.   In many cases multiple times.   These folks had stories that would make you laugh, make you cry, make your heart sing for joy, and take you to some dark places, and above all allow you, to for a brief crystalline moment, see into and through the eyes of those who defend this country.   To gauge the quality of the men and women putting it on the line.  For me this was a powerful personal experience.

If you are ever so fortunate as to be invited to this event, Go! I can guarantee that It will be one of the richest and most rewarding experiences of your life. Kudos to the Department of Defense and US Army War College for building such a strong and robust program.

And finally, a great big “Hoo-Ah!” to my all compatriots in Seminar 6. I enjoyed your hospitality and willingness to bring me into your fold.  You have taught me a truly important lesson, “Always remember to Start slow, then taper off.”   :)

-Mike Manos

Data Center Leadership video posted at TechEd

clip_image001

I recently did a panel at TechEd in Orlando.  While there Lewis Curtis caught me for a moment to shoot some questions at me around Data Center leadership.  It may or may not be interesting to you (My money is on the latter :) ).  But if you are interested the link can be found below:

On Datacenter Leadership

 

-Mm

Green Grid Data Center Indicator + CADE = Something useful!

There are times when two concepts merge and the result makes something better than the whole.  It is not unlike the old television commercial where two people collide into each other.  One eating a chocolate bar, the other a vat of peanut butter.   The resulting lines are television gold:

“Your chocolate is in my peanut butter!”

“Your peanut butter is on my chocolate!”

“HEY!” (in unison with smiles)

I have been anxiously awaiting for the Green Grid to publish their work on the Data Center Indicator tool.  My good friend Christian Belady and the incredible folks in the Technical workgroups came up with something that made me smile and gave CADE a way to be a viable metric. 

The Data Center Indicator Tool gives you a visual representation across all factors important to operating and measuring a data center.  Its no secret that  I get quite passionate about the need to measure your data center.   The lack of strict and rigorous uniform measurement across the data center industry is one of the biggest tragedies we are waiting to inflict upon ourselves. 

This tool is not necessarily for beginners as it assumes you have a good set of data and active measurement already in place.   However, in terms of quickly identifying trends and understanding your environment, I find it quite unique and interesting.   In fact, many of the same factors represented are rolled up into the CADE metric.  

image

The white paper which has been published on the Green Grid Site (White Paper #15)  Is a great way to have a holistic view at your environment over time and is even suitable for executives not familiar with the intricacies of Data Center or Mission Critical environment facilities.    If one takes the rolled up percentage in CADE and combines it with this type of Graph you have a great KPI, and a mechanism which makes the information actionable.  That dear readers is what any facilities manager can use.

-Mm

Struggling with CADE, McKinsey / Uptime Metric (RE-POST)

percentage.jpg

This is a Re-post of my original blog post on May 5th regarding tortured thoughts around the CADE Data Center Metric put forward by McKinsey. This has relevance to my next post and I am placing it here for your convenience.

 

I guess I should start out this post with the pre-emptive statement that as a key performance indicator I support the use of CADE or metrics that tie both facilities and IT into a single metric.  In fact we have used a similar metric internally at Microsoft.  But the fact is at the end of the day I believe that any such metrics must be useful and actionable.  Maybe its because I have to worry about Operations as well.  Maybe its because I don’t think you roll the total complexity of running a facility with one metric.  In short, I don’t think dictating yet another metric, especially one that doesn’t lend itself to action, is helpful.

As some of you know I recently gave keynote speeches at both DataCenter World and the 2008 Uptime Symposium.  Part of those speeches included a simple query of the audience of how many people are measuring energy efficiency in their facilities.  Now please keep in mind that the combined audience of both engagements numbered between 2000-2400 datacenter professionals.  Arguably these are the 2400 that really view data centers as a serious business within their organizations.  These are folks whose full time jobs are running and supporting data center environments for some of the most important companies around the world.   At each conference less than 10% of them raised their hands.   The fact that many in the industry including Ken Brill at the Uptime Institute, Green Grid, and others have been preaching about measurement for at least the last three years and less than 10% of the industry has accepted this best practice is troublesome.  

Whether you believe in measuring PUE or DCIE, you need to be measuring *something* in order to even get one variable of the CADE metric.  Given this lack of instrumentation and\or process within those firms most motivated to do so speaks in large part of the lack of success this metric is going to have over time.  It therefore follows, if they are not measuring efficiency, they likely don’t understand their total facility utilization (electrically speaking).  The IT side may have an easier way of getting variables for system utilization, but how many firms have host level performance agents in place? 

I want to point out that I am speaking to the industry in general.  Companies like ours who are investing hundreds of millions of dollars get the challenges and requirements in this space.  Its not a nice to have, its a requirement.  But when you extend this to the rest of the industry, there is a massive gap in this space.

Here are some interesting scenarios that when extended to the industry may break or complicate the CADE metric:

  • As you cull out dead servers in your environment, your utilization will drop accordingly and as a result the metric will remain unchanged.  The components of CADE are not independent. Dead servers are removed so that Average server utilization goes up then Data Center Utilization goes down showing proportionally so there is no change and if anything PUE goes up which means the metric may actually go up. Keep in mind that all results are good when kept in context of one another.
  • Hosting Providers like Savvis, Equinix, Dupont Fabros, Digital Realty Trust, and the army of others will be exempt from participating.  They will need to report back of house numbers to the their  customers (effectively PUE).    They do not have access to their customers server information It seems to me that CADE reporting in hosted environments will be difficult if not impossible.  As the design of their facilities will need to play a large part of the calculation this makes effective tracking difficult.  Additionally, overall utilization will be measured at what level?
  • If hosters exempted, then it gives CADE a very limited application or shelf-life.  You have to own the whole problem for it to be effective.  
  • As I mentioned, I think CADE has strong possibilities for those firms who own their entire stack.   But most of the data centers in the world would probably not fall into “all-in” scenario bucket.

I cant help but think we are putting the cart before the horse in this industry.  CADE may be a great way to characterize data center utilization but its completely useless if the industry isnt even measuring the basics.  I have come to the realization that this industry does a wonderful job in telling its members WHAT to do, but lacks to follow-up with the HOW.  CADE is meant for higher level consumption.  Specifically those execs who lack the technical skill-sets to make heads or tails of efficiencies and how they relate to overall operations.   For them, this metric is perfect. But we have a considerable way to go before the industry at large gets there.

Regardless, I strongly suggest each and everyone adopt the take away at Symposium….Measure, measure, measure.

Upcoming Speaking Events

speech.gif

For those of you interested, I thought I would outline some of my upcoming speaking events:

August 7th, 2008 – Data Center Dynamics Keynote – Seattle

Presentation: “The Need for Data Center Glasnost”

September 16th, 2008 – Data Center Dynamics Keynote – Chicago

Presentation: “Containers, Fact, Fiction and Fantasy”

October 2, 2008 – Power and Cooling ’08 – London

Presentation: “Think Green, Think Different, Think DATACENTRES”

November 17th, 2008 – 7×24 Exchange Keynote with C. Belady – Palm Springs

Presentation: “Kicking Anthills: The Challenge Landscape of Mission Critical Environments

-Mm

Do Not Tighten Bolts

nuts-and-bolts.jpgI have a pretty incredible job.   My current role has me leading Microsoft’s efforts at designing, constructing, and operating its world-wide Data Center infrastructure in support of our cloud services initiatives, or more correctly “Software + Services”.  

There aren’t many people around tasked with this kind of challenge, in fact the number of companies attempting this challenge can be counted on one or two hands.   One routine question I get asked is ‘what methodology or approach I use to deliver against this challenge?’.  The question of course assumes there is an answer.  As if some there is some book one can run out and purchase to figure it out.   No such book exists. 

The real answer involves the hard work and dedication of an incredibly talented team focused on a single mission.   That is exactly what we have on the Data Center challenge at Microsoft.  I have the world’s most talented team and I am not in the least shy about my confidence in them.   However, even with the raw materials produced by this incredible team,  the technology breakthroughs in Data Center design, the incredible process and automation improvements, the focused reduction of energy consumption and drive for greater energy-use efficiency is not 100% of the formula.

There is one missing element.  Clues to this one missing element can be found in products for sale at your local Target or Walmart store.   Many people may be surprised by this, but I suspect most wont. 

I recently spent a good part of a weekend putting together deck furniture for my home.   It was good quality stuff, it had the required parts and hardware and not unlike other do-it-yourself furniture it had directions that left a lot to be desired. In many ways its like IT Infrastructure or running any IT shop.   You have all the tools, you have all the raw components, but how you put it all together is where the real magic happens, and the directions are usually just as vague on how to do it.

One of the common themes across all steps of the deck furniture pieces was a common refrain, ‘Do Not Tighten Bolts”.   The purpose was to get all of the components together, even if a bit loose, to ensure you had the right shape, all components were in the right place, and then and only then do you tighten the bolts.

If you really want to know the secret to putting together solutions at scale, remember the “Do Not Tighten Bolts” methodology.   Assemble to raw components, ensure you have the right shape and that all components are in the right place, and then “Tighten it down.”   This can be and is an iterative process.   Keep working to get that right shape.  Keep working to find the right component configuration.  Tighten bolts.    As I built my first deck chair, there was significant amounts of trial and error.  The second deck chair however was seamless, even with the same cruddy directions.   Once you learn to ‘Not Tighten’ technique the assembly process is quick and provides you with great learnings.   

Some may feel this approach too simplistic, or lacks the refinement of a polished methodology.   The fact of the matter is that Cloud Services infrastructure takes lots of hard work, great natural resources, and above all flexibility without adherence to dogmatic approach.

That is why I have named this blog “Loose Bolts”.  I will be moving my formal blog activity to this forum and hopefully post interesting topics from time to time.

Thanks so much for reading,

Mike Manos