Cloud Détente – The Cloud Cat and Mouse Papers

Over the last decade or so I have been lucky enough to be placed into a fairly unique position to work internationally deploying global infrastructure for cloud environments.  This work has spanned across some very large companies with a very dedicated focus on building out global infrastructure and managing through those unique challenges.   Strategies may have varied but the challenges faced by them all had some very common themes.   One of the more complex interactions when going through this process is what I call the rolling Cat and Mouse interactions between governments at all levels and these global companies.  

Having been a primary player in these negotiations and the development of measures and counter measures as a result of these interactions, I have come to believe there are some interesting potential outcomes that cloud adopters should think about and understand.   The coming struggle and complexity for managing regulating and policing multi-national infrastructure will not solely impact the large global players, but in a very real way begin to shape how their users will need to think through these socio-political  and geo-political realities. The potential impacts on their business, their adoption of cloud technologies, their resulting responsibilities and measure just how aggressively they look to the cloud for the growth of their businesses.

These observations and predictions are based upon my personal experiences.  So for whatever its worth (good or bad)  this is not the perspective of an academic writing from some ivory tower, rather they are  the observations of someone who has been there and done it.  I probably have enough material to write an entire book on my personal experiences and observations, but I have committed myself to writing a series of articles highlighting what I consider the big things that are being missed in the modern conversation of cloud adoption.  

The articles will highlight (with some personal experiences mixed in) the ongoing battle between Technocrats versus Bureaucrats.  I will try to cover a different angle on many of the big topics out there today such as :

• Big Data versus Big Government
• Rise of Nationalism as a factor in Technology and infrastructure distribution
• The long struggle ahead for managing, regulating, and policing clouds
• The Business, end-users, regulation and the cloud
• Where does the data live? How long does it live? Why Does it Matter?
• Logic versus Reality – The real difference between Governments and Technology companies.
• The Responsibilities of data ownership
• … regarding taxation exposure
• … regarding PII impacts
• … Safe Harbor

My hope is that this series and the topics I raise, while maybe a bit raw and direct, will cause you to think a bit more about the coming impacts on Technology industry at large, the potential coming impacts to small and medium size businesses looking to adopt these technologies, and the developing friction and complexity at the intersection of technology and government.

\Mm

Read Full Post »

Attacking the Cruft

Today the Uptime Institute announced that AOL won the Server Roundup Award.  The achievement has gotten some press already (At Computerworld, PCWorld, and related sites) and I cannot begin to tell you how proud I am of my teams.   One of the more personal transitions and journeys I have made since my experience scaling the Microsoft environments from tens of thousands of servers to hundreds of thousands of servers has been truly understanding the complexity facing a problem most larger established IT departments have been dealing with for years.  In some respects, scaling infrastructure, while incredibly challenging and hard, is in large part a uni-directional problem space.   You are faced with growth and more growth followed by even more growth.  All sorts of interesting things break when you get to big scale. Processes, methodologies, technologies, all quickly fall to the wayside as you climb ever up the ladder of scale.

At AOL I faced a multi-directional problem space in that, as a company and as a technology platform we were still growing.  Added to that there was 27 years of what I call “Cruft”.   I define “Cruft” as years of build-up of technology, processes, politics, fiscal orphaning and poor operational hygiene.  This cruft can act as a huge boat anchor and barrier to an organization to drive agility in its online and IT operations.  On top of this Cruft a layer of what can best be described as lethargy or perhaps apathy can sometimes develop and add even more difficulty to the problem space.

One of the first things I encountered at AOL was the cruft.  In any organization, everyone always wants to work on the new, cool, interesting things. Mainly because they are new and interesting..out of the norm.  Essentially the fun stuff!  But the ability for the organization to really drive the adoption of new technologies and methods was always slowed, gated or in some cases altogether prevented by years interconnected systems, lost owners, servers of unknown purpose lost in the distant historical memory and the like.   This I found in healthy populations at AOL. 

We initially set about building a plan to attack this cruft.   To earnestly remove as much of the cruft  as possible and drive the organization towards agility.  Initially we called this list of properties, servers, equipment and the like the Operations $/-\!+ list. As this name was not very user-friendly it migrated into a series of initiatives grouped the name of Ops-Surdities.   These programs attacked different types of cruft and were at a high level grouped into three main categories:

The Absurdity List – A list of projects/properties/applications that had a questionable value, lack of owner, lack of direction, or the like but was still drawing load and resources from our data centers.   The plan here was to develop action plans for each of the items that appeared on this list.

Power Hog – An effort to audit our data center facilities, equipment, and the like looking for inefficient servers, installations, and /or technology and migrating them to new more efficient platforms or our AOL Cloud infrastructure.  You knew you were in trouble when you had a trophy of a bronze pig appear on your desk or office and that you were marked. 

Ops Hygiene – The sometimes tedious task of tracking down older machines and systems that may have been decomissioned in the past, marked for removal, or were fully depreciated and were never truly removed.  Pure Vampiric load.  You may or may not be surprised how much of this exists in modern data centers.  It’s a common issue I have had with most data center management professionals in the industry.

So here we are, in a timeline measured in under a year, and being told all along the way by“crufty old-timers” that we would never make any progress, my teams have de-comissioned almost 10,000 servers from our environments. (Actually this number is greater now, but the submission deadline for the award was earlier in the year).  What an amazing accomplishment.  What an amazing team!

So how did we do it?

As we will be presenting this in a lot more detail at the Uptime Symposium, I am not going to give away all of our secrets in a blog post and give you a good reason to head to the Uptime event and listen to and ask the primary leaders of this effort how they did it in person.  It may be a good use of that Travel budget your company has been sitting on this year.

What I will share is some guidelines on approach and some things to be wary of if you are facing similar challenges in your organization.

FOCUS AND ATTENTION

I cannot tell you how many I have spoken with that have tried to go after ‘cruft’ like this time and time again and failed.   One of the key drivers for success in my mind is ensuring that there is focus and attention on this kind of project at all levels, across all organizations, and most importantly from the TOP.   To often executives give out blind directives with little to no follow through and assume this kind of thing gets done.   They are generally unaware of the natural resistance to this kind of work there is in most IT organizations.    Having a motivated, engaged, and focused leadership on these types of efforts goes and extraordinarily long way to making headway here.  

BEWARE of ORGANIZATIONAL APATHY

The human factors that stack up against a project like this are impressive.  While they may not be openly in revolt over such projects there is a natural resistance to getting things done.  This work is not sexy.  This work is hard.  This work is tedious.  This likely means going back and touching equipment and kit that has not been messed with for a long time.   You may have competing organizational priorities which place this kind of work at the bottom of the workload priority list.   In addition to having Executive buy in and focus, make sure you have some really driven people running these programs.  You are looking for CAN DO people, not MAKE DO people.

TECHNOLOGY CAN HELP, BUT ITS NOT YOUR HEAVY LIFTER

Probably a bit strange for a technology blog to say, but its true.  We have an incredible CMDB and Asset System at AOL.  This was hugely helpful to the effort in really getting to the bottom of the list.   However no amount of Technology in place will be able to perform the myriad of tasks required to actually make material movement on this kind of work.   Some of it requires negotiation, some of it requires strength of will, some of it takes pure persistence in running these issues down…working with the people.  Understanding what is still required, what can be moved.  This requires people.   We had great technologies in place from the perspective of knowing where are stuff was, what it did, and what it was connected to.  We had great technologies like our Cloud to move some of these platforms to ultimately.    However, you need to make sure you don’t go to far down the people trap.  I have a saying in my organization – There is a perfect number of project managers and security people in any organization.  Where the work output and value delivered is highest.   What is that number?  Depends – but you definitely know when you have one too many of each.

MAKE IT FUN IF YOU CAN

From the brass pigs, to minor celebrations each month as we worked through the process we ensured that the attention given the effort was not negative. Sure it can be tough work, but you are at the end of the day substantially investing in the overall agility of your organization.  Its something to be celebrated.    In fact at the completion of our aggressive goals the primary project leads involved did a great video (which you can see here) to highlight and celebrate the win.   Everyone had a great laugh and a ton of fun doing what was ultimately a tough grind of work.  If you are headed to Symposium I strongly encourage you to reach out to my incredible project leads.  You will be able to recognize them from the video….without the mustaches of course!

\Mm

Read Full Post »

ATC Ribbon Cutting

In my previous post I had mentioned how extremely proud I was of the Technology teams here at AOL in delivering a truly state of the art Data Center facility with some incredible ground breaking technology.  As I mentioned the facility was actually in production use faster than we could get the ribbon cutting ceremony scheduled.  I thought I would share a small slice of the pictures of the internal Ribbon Cutting Event.

___

Alex Gounares, former fellow Microsoft alum and AOL CTO and I presided over the celebration.   In this photo, Alex and I talk over some of the technologies used in our cloud with one our cloud engineers.  As the facility is based upon pre-racked technologies and modular facility and network build components it allows for significant cost and capital optimization. this allows us to build only when demand and growth dictates the need. All machines in the background are live and have been live for a few weeks.

___

After receiving two very large scissors which were remarkably sharp and precise for their size we were ready to go.   A few short words about the phenomenal job our teams performed and it was time for some ribbon to kiss raised floor.

 

 

___

At the end of the day the real reason why this project was such a success really breaks down to the team responsible for this incredible win.   An effort like this took incredibly smart people from different organizations working together to make this a reality.    The achievement is even more impressive in my mind when you think about the fact that in many cases our 90 day to live timeframe included design and execution on the go!   My guess is our next one may be significantly faster without all that design time. The true heroes of ATC are below!

 

\Mm

(Special thanks goes out to Krysta Scharlach for the permission and use of her pictures in this post)

Read Full Post »

Chaos Monkeys, Donkeys and the Innovation of Action

Last week I once again had the pleasure of speaking at the Uptime Institute’s Symposium.  As one of the premiere events in the Data Center industry it is definitely one of those conferences that is a must attend to get a view into what’s new, what’s changing, and where we are going as an industry.  Having attended the event numerous times in the past, this year I set out on my adventure with a slightly different agenda.

Oh sure I would definitely attend the various sessions on technology, process, and approach.  But this time I was also going with the intent to listen equally to the presenters as well as the scuttlebutt, side conversations, and hushed whispers of the attendees.   Think of it as a cultural experiment in being a professional busy body.  As I wove my way around from session to session I was growing increasingly anxious that while the topics were of great quality, and discussed much needed areas of improvement in our technology sector – most of them were issues we have covered, talked about and have been dealing with as an industry for many years.   In fact I was hard pressed to find anything of real significance in the new category.   These thoughts were mirrored in those side conversations and hushed whispers I heard around the various rooms as well.

One of the new features of Symposium is that the 451 Group has opted to expand the scope of the event to be more far reaching covering all aspects of the issues facing our industry.   It has brought in speakers from Tier 1 Research and other groups that have added an incredible depth to the conference.    With that depth came some really good data.   In many respects the data reflected (in my interpretation) that while technology and processes are improving in small pockets, our industry ranges from stagnant to largely slow to act.  Despite mountains of data showing energy efficiency benefits, resulting cost benefits, and the like we just are not moving the proverbial ball down the field.

In a purely unscientific poll I was astounded to find out that some of the most popular sessions were directly related to those folks who have actually done something.  Those that took the new technologies (or old technologies) and put them into practice were roundly more interesting than more generic technology conversations.   Giving very specific attention to detail on the how they accomplished the tasks at hand, what they learned, what they would do differently.   Most of these “favorites” were not necessarily in those topics of “bleeding edge” thought leadership but specifically the implementation of technologies and approaches we have talked about the event for many years.   If I am honest, one of those sessions that surprised me the most was our own.   AOL had the honor of winning an IT Innovation Award from Uptime and as a result the teams responsible for driving our cloud and virtualization platforms were allowed to give a talk about what we did, what the impact was and how it all worked out.   I was surprised because I was not sure how many people would come to this side session and listen to presentation or find the presentation relevant.  Of course I thought it was relevant (We were after all going to get a nifty plaque for the achievement) but to my surprise the room was packed full, ran out of chairs, and had numerous people standing for the presentation.   During the talk we had a good interaction of questions from the audience and after the talk we were inundated with people coming up to specifically dig into more details.  We had many comments around the usefulness of the talk because we were giving real life experiences in making the kinds of changes that we as an industry have been talking about for years.  Our talk and adaption of technology even got a little conversation in some of the Industry press such as Data Center Dynamics.

Another session that got incredible reviews was the presentation by Andrew Stokes of Deutsche Bank who guided the audience through their adoption of 100% free air cooled data center in the middle of New York City.  Again, the technology here was not new (I had built large scale facilities using this in 2007) – but it was the fact that Andrew and the folks at Deutsche Bank actually went out and did something.   Not someone from those building large-scale cloud facilities, not some new experimental type of server infrastructure.  Someone who used this technology servicing IT equipment that everyone uses, in a fairly standard facility who actually went ahead and did something Innovative.  They put into practice something that others have not. Backed back facts, and data, and real life experiences the presentation went off incredibly and was roundly applauded by those I spoke with as one of the most eye-opening presentations of the event.

By listening the audiences, the hallway conversations, and the multitude of networking opportunities throughout the event a pattern started to emerge,  a pattern that reinforced the belief that I was already coming to in my mind.   Despite a myriad of talk on very cool technology, application, and evolving thought leadership innovations – the most popular and most impactful sessions seemed to center on those folks who actually did something, not with the new bleeding edge technologies, but utilizing those recurring themes that have carried from Symposium to Symposium over the years.   Air Side economization?  Not new.   Someone (outside Google, Microsoft, Yahoo, etc) doing it?  Very New-Very Exciting.  It was what I am calling the Innovation of ACTION.  Actually doing those things we have talked about for so long.

While this Innovation of Action had really gotten many people buzzing at the conference there was still a healthy population of people who were downplaying those technologies.  Downplaying their own ability to do those things.    Re-stating the perennial dogmatic chant that these types of things (essentially any new ideas post 2001 in my mind) would never work for their companies.

This got me thinking (and a little upset) about our industry.  If you listen to those general complaints, and combine it with the data that we have been mostly stagnant in adopting these new technologies – we really only have ourselves to blame.   There is a pervasive defeatist attitude amongst a large population of our industry who view anything new with suspicion, or surround it with the fear that it will ultimately take their jobs away.  Even when the technologies or “new things” aren’t even very new any more.  This phenomenon is clearly visible in any conversation around ‘The Cloud’ and its impact on our industry.    The data center professional should be front and center on any conversation on this topic but more often than not self-selects out of the conversation because they view it more as an application thing, or more IT than data center thing.   Which is of course complete bunk.   Listening to those in attendance complain that the ‘Cloud’ is going to take their jobs away, or that only big companies like Google , Amazon, Rackspace, or  Microsoft would ever need them in the future were driving me mad.   As my keynote at Uptime was to be centered around a Cloud survival guide – I had to change my presentation to account for what I was hearing at the conference.

In my talk I tried to focus on what I felt to be emerging camps at the conference.    To the first, I placed a slide prominently featuring Eeyore (from Winnie the Pooh fame) and captured many of the quotes I had heard at the conference referring to how the Cloud, and new technologies were something to be mistrusted rather than an opportunity to help drive the conversation.     I then stated that we as an industry were an industry of donkeys.  That fact seems to be backed up by data.   I have to admit, I was a bit nervous calling a room full of perhaps the most dedicated professionals in our industry a bunch of donkeys – but I always call it like I see it.

I contrasted this with those willing to evolve their thought forward, embrace that Innovation of Action by highlighting the Cloud example of Netflix.   When Netflix moved heavily into the cloud they clearly wanted to evolve past the normal IT environment and build real resiliency into their product.   They did so by creating a rogue process (on purpose) call the Chaos Monkey which randomly shut down processes and wreaked havoc in their environment.   At first the Chaos Monkey was painful, but as they architected around those impacts their environments got stronger.   This was no ordinary IT environment.  This was something similar, but new.  The Chaos Monkey creates Action, results in Action and on the whole moves the ball forward.

Interestingly after my talk I literally have dozens of people come up and admit they had been donkeys and offered to reconnect next year to demonstrate what they had done to evolve their operations.

My challenge to the audience at Uptime, and ultimately my challenge to you the industry is to stop being donkeys.   Lets embrace the Innovation of Action and evolve into our own versions of Chaos Monkeys.    Lets do more to put the technologies and approaches we have talked about for so long into action.    Next Year at Uptime (and across a host of other conferences) lets highlight those things that we are doing.  Lets put our Chaos Monkeys on display.

As you contemplate your own job – whether IT or Data Center professional….Are you a Donkey or Chaos Monkey?

\Mm

Read Full Post »

Now Hiring -Apply Within

Joining a company during a big transition is a pretty exciting thing.   The amount of chaos and work to get done is immense especially when you are trying to fundamentally re-shape something that exists to best in class and ultimately industry leading again.  While I can’t tell you specifically what I am thinking yet, we have already begun making at some of the necessary changes to deliver on a big mission.   The overall content strategy that AOL is marching down is going to drive some very interesting infrastructure and operations related challenges (You may have seen the recent purchase of Huffington Post and some others) with even greater challenges ahead.  From time to time I am going to post some positions that I have open in my organization in case any of my readers may be interested or know someone that is interested.  I know some of you read this for more of my industry related thoughts – but you might be interested some of the things we are doing here as well.  If not, you can skip to the next post. 

So without further ado…

NOC Engineers

I am dramatically expanding the size of our Network Operations Center and capabilities – Do you know someone smart who is interested in breaking into one of the top ten largest Internet companies (as measured by infrastructure) and wants to get a good feel for managing Internet scale operations, multiple data center locations, and tens of thousands of machines?   Are you a returning veteran looking for your next career move?   Are you a college student wondering how to break into this space at scale? This is an area that is going to evolve greatly in the coming weeks and months within the organization and its a chance to get some great experience  and be at the ground floor of a company significantly driving automation and the next big thing.   This would include significant career paths into multiple technology directions.  These positions are based in Dulles, VA – so we would be looking at the Virginia, DC, and Maryland area for candidates.  I am extremely motivated to fill these roles quickly.

CDN/EDGE/Applications Engineering Manager

We are looking for strong candidates to manage a team that will be an integral part of our overall future delivery strategy specifically targeted at distributed CDN and Edge computing capabilities and help drive various application engineering efforts within the organization.   This position reports up through our Network Engineering organization but will be deeply embedded into other engineering teams, product teams, and architecture efforts.

DNS Engineer

We are looking for DNS engineers to join the engineering teams and help drive the future of our name space management and technologies along with some operational related responsibilities managing our global DNS infrastructure.  As we are significantly adding to the business from the content aggregation side of the house the DNS function will be an integral growth area for the business.

If you or someone you might know is interested in these postions please reach out to me via comment on this blog post (I’ll keep it private) so I can tell you where to send your resumes to.   I am not interested in sourcing these positions from outside agencies at this time.

\Mm

Tags: Hiring, DNS Engineers, NOC Engineers, CDN Engineers, Edge Engineers, Application Engineering, AOL

 

Read Full Post »

I’ve Got Mail….A new Aol.

You may have seen the announcement today about my recent decision and move to join the new leadership team at Aol.  To some of my friends in the Technorati, and most specifically the Valley, this move probably seems very contrarian.  Having built some of the largest cloud infrastructure’s in the world, re-aligning operational processes at massive scale, Aol at first stroke may seem an odd choice.  I have worked in some of the largest multi-national companies in the world, I have successfully (and unsuccessfully) launched start-ups, have been a cost center and carried a P&L.  I think I have a pretty good understanding of the range and complexity of challenges (especially from a technology perspective)  from small business to large.   Across the spectrum of these types and sized companies you get a different feel.   Different cultures.  Different attitudes.    Different Vibes.

Aol is aggressively moving to redefine itself in the industry, to significantly transform and morph itself into a world that Aol itself helped create and define over 25 years ago.   There is no arguing that the first true scale challenges in dealing with the Internet at large were experienced by those first AOL’ers as they had to deal with numbers of users never before seen in our industry.  They pushed the boundaries of technology, they pushed the boundary of operations, they created whole new paradigms.  To reinvent itself in a market with such competition, such diversity is a huge challenge.

One of the most surprising things to me is that Vibe-thing I talked about a few moements ago. When walking around the company you cannot help but notice that definitely has more of a technology start-up feel to it.   Its palpable.  One of the folks I ran into called it a start-around.   A combination of a Start-up and a Turn-around.  Perhaps thats the best description I have to describe that vibe.   Sure things have been tough, sure there is alot of legacy to work through, but the level of commitment to those folks that are here is incredible.  Moreso than that.  Its a culture of beleivers.  Its all the self-sacrafice and personal investment you find in a startup, but with a team of seasoned veterans.  Its quite unique in my experience.

As I mentioned, Aol has long held a place of respect in terms of Operational best practices at scale, and a culture that recognized the importance of technology in the delivery of its mission.  Tim Armstrong, the CEO and Google veteran, has built an incredible team of passionate technology veterans from places like Google, Microsoft, and others.  The mission is focused.  The mission is deliberate.  The mission is clear.   The mission is hard.   Its a huge challenge.   Its the kind of challenge I love.   If you think its impossible you are only encouraging my energy more.  I could have taken a safe bet.  But where is the excitement?  Where is the challenge?  As the saying goes, “A ship is safe in the harbour, but thats not what ships are for!”   This ship is setting sail and my commitment is that not only will we find a new world, we will define it!

In the coming days/weeks/months, I hope to share many of the exciting things we will be endeavoring to accomplish and give you a real taste of some of the big changes I will be attempting.   As always, technology and operational processes will be key to the success of the mission the company is on and I have some very definite ideas on how we can leap frog current thinking in this space and ensure that our technology and operational approach is no only a strategic value to the business, but also industry leading in execution.

\Mm

Tags: Michael Manos, AOL, Data Centers, Operations, start-around

Read Full Post »

Site Selection,Data Center Clustering and their Interaction

I have written many times on the importance of the site selection for data centers and its growing importance when one considers the regulatory and legislative efforts underway globally.   Those who make their living in this space know that this is going to have a significant impact on the future landscape of these electronic bit factories.   The on-going long term operational costs over the life of the facility,  their use of natural resources (such as power) and what they house and protect (PII data or Personally Identifiable Information) are even now significantly impacting this process for many large global firms, and is making its way into the real estate community.  This is requiring a series of crash courses in information security, power regulation and rate structures, and other complex issues for many in the Real Estate community. 

In speaking to a bunch of friends in the Real Estate side of the business, I thought it might be interesting to take a few of these standard criteria head on in an open discussion.   For this post I think I will take on two of the elemental ones in data center site selection. We will look at one major item, and one item that is currently considered a minor factor that is quickly on the rise in terms of its overall importance.  Namely Power and Water, respectively.

Watts the Big Deal?

Many think that power cost alone is the primary driver for data centers, and while it is always a factor there are many other facets that come into play underneath that broader category of Power.   While Site Selection Factors are always considered highly confidential I thought it might highlight some of the wider arcs in this category.

One such category getting quite a bit of attention is Power  Generation Mix.   The Generation mix is important because it is essentially the energy sources responsible how that area or region gets its energy.  Despite what politicians would lead you to believe, once an electron is on the grid it is impossible to tell from which source it came.   So ‘Green Energy’ and its multitude of definitions is primarily determined by the mix of energy sources for a given region.   A windmill for example does not generate an electron with a tiny label saying that is sourced from ‘green’ or ‘renewable’ sources.   Understanding the generation mix of your power will allow you to forecast and predict potential Carbon output as a result of Data Center Carbon production.   The Environmental Protection Agency in the US, produces a metric called the Carbon Emission Factor which can be applied to your consumption to assist you in calculating your carbon output and is based upon the generation mix of the areas you are looking to site select in.   Whether you are leasing or building your own facility you will likely find yourself falling into a mandatory compliance in terms of reporting for this kind of thing.

So you might be thinking, ‘Great, I just need to find the areas that have cheap power and a good Carbon Emission Factor right?’  The answer is no.  Many Site Selection processes that I see emerging in the generic space start and stop right at this line.   I would however advocate that one takes the next logical step which is to look at the relationship of these factors together and over a long period of time.

Generation Mix has long been considered to be a ‘Forever’ kind of thing.  The generation sources within a region, rarely changed, or have rarely changed over time.   But that is of course changing significantly in the new era that we live in.

Lets take the interplay (both historical and moving forward) of the Power Cost and its relationship with the Generation Mix.  As humans we like to think in simplistic terms.  Power costs for a specific region are ‘so many cents per kilowatt hour’ this changes based upon whether you are measured at a residential, commercial, or industrial rate schedule.   The rate schedule is a function of how much power you ultimately consume or promise to consume to the local utility.   The reality of course is much more complicated than that.   Power rates fluctuate constantly based upon the overall mix.   Natural disasters, regulation, etc. can have a significant impact on power cost over time.   Therefore its generally wise to look at the Generation Mix Price Volatility through the longer term lens of history and see how a region’s power costs oscillate between these types of events.     However you decide to capture or benchmark this it is a factor that should be considered. 

This is especially true when you take this Volatility factor and apply it the changing requirements of Carbon Reporting and impacts.  While the United States is unlikely to have a law similar to the CRC in the UK (Carbon Reduction Commitment), it will see legislation and regulation impacting the energy producers.  

You might be asking yourself, ‘Who cares if they go after those big bad energy companies and force them to put more ‘green power in their mixes’.  Well lets think about the consequences of these actions to you the user, and why its important to your site selection activity.

As the energy producers are regulated to create a more ‘green’ mix into their systems, two things will happen.  The first of course is that rates will rise.  The energy producers will need to sink large amounts of capital to invest into these technologies, plants, research and development, etc to come to alignment with the legal requirements they are being regulated to.   This effect will be uneven as many areas around the globe have quite a disparate mix of energy from region to region.   This will also mean that ‘greener’ power will likely result in ‘more expensive power’.   Assessing an area for the potential impacts to these kinds of changes is definitely important in a data center build scenario as you likely have a desire to ensure that your facility has the longest possible life which could span a couple of decades.  The second thing which may be a bit harder to guess at, is ‘which technology’ is a given region or area likely to pick and its resulting carbon output impact.   While I have a definite approach to thinking through such things, this is essentially the beginning of the true secret sauce to site selection expertise and the help you may require if you don’t have an internal group to go through this kind of data and modeling.  This is going to have an interesting impact on the ‘clustering’ effect that happens in our industry at large.

We have seen many examples like Quincy, Washington and San Antonio, Texas where the site selection process has led to many Data Center providers locating in the same area to benefit from this type of analysis (even if not directly exposed to the criteria).  There is a story (that I don’t know if its true or not) that in the early days when a new burger chain was looking to expand where it would place its restaurants, it used the footprint of its main competitor as its guide. The thinking was that they probably had a very scientific method for that selection and they would receive that same ancillary benefit without the cost and effort.   Again, not sure if that is true or not, but its definitely something likely to happen in our industry. 

In many markets these types of selections are in high demand.   Ascent Corporation out of St. Louis is in the process of building a modern facility just down the street from the Microsoft Mega-Facility near Chicago.   While Ascent was a part of the original Microsoft effort to build at that location, there has been an uptick in interest for being close to that facility for the same reasons as I have outlined here.  The result is their CH2 facility is literally a stones throw from the Microsoft Behemoth.  The reasons? Proximity to power, fiber, and improved water infrastructure are already there in abundance.  The facility even boasts container capabilities just like its neighbor.   The Elmhurst Electrical Substation sits directly across the highway from the facility with the first set of transmission poles within easy striking distance.  

The Generation mix of that area has a large nuclear component which has little to no carbon impact, and generates long term stability in terms of power cost fluctuations.   According to Phil Horstmann, President of Ascent, their is tremendous interest in the site and one of the key draws is the proximity of its nearby neighbor.  In the words of one potential tenant ‘Its like the decision to go to IBM in the 80s.  Its hard to argue against a location where Microsoft or Google has placed one of its facilities.’

This essentially dictates that there will be increasing demand on areas where this analysis is done or has been perceived to be done.   This is especially true where co-location and hosting providers can align their interests with those commercial locations where there is market demand.  While those that follow first movers will definitely benefit from these decisions (especially those without dedicated facility requirements), first movers continue to have significant advantage if they can get this process correct.

Tying into the power conversation is that of water.  With the significant drive for economization (whether water based or air-based)  water continues to be a factor.  What many people don’t understand is that in many markets the discharge water is clean to dump into the sewage system and to ‘dirty’ to discharge to retention ponds.  This causes all kinds of potential issues and understanding the underlying water landscape is important.   The size of the metropolitan sewage environments, ability to dig your own well efforts, the local water table and aquifer issues, your intended load and resulting water requirements, how the local county, muncipality, or region views discharge in general and which chemicals and in what quantities is important to think about today.  However, as the use of water increases in terms of its potential environmental scrutiny – water is quickly rising on the site selection radar of many operators and those with long term holds.

I hope this brief talk was helpful.  I hope to post a few other key factors and a general discussion in the near future.  

\Mm

Read Full Post »

Rolling Clouds – My Move into the Mobile Cloud

As many of you saw in my last note, I have officially left Digital Realty Trust to address some personal things.   While I get those things in order I am not sitting idling by.   I am extremely happy to announce that I have taken a role at Nokia as their VP of Service Operations.  In this role I will have global responsibility for the strategy, operation and run of infrastructure aspects for Nokia’s new cloud and mobile services platforms.

Its an incredibly exciting role especially when you think of the fact that the number of mobile hand-held’s around the world are increasingly becoming the interface by which people are consuming information.  Whether that be Navigation-based applications or other content related platforms your phone is becoming your gateway to the world. 

I am also very excited by the fact that there are some fierce competitors in this space as well.  Once again I will be donning my armor and doing battle with my friends at Google.   Their Droid platform is definitely interesting and it will be interesting to see how that develops.  I have a great amount of respect for Urs Hoelze and their cloud platform is something I am fairly familiar with .  I will also be doing battle with the folks from Apple (and interestingly my good friend Olivier Sanche).  Apple definitely has the high end hand-held market here in the US, but its experience in Cloud platforms and operations is not very sophisticated just yet.  On some levels I guess I am even competing against the infrastructure and facilities I built out at Microsoft at least as it relates to the mobile world.  Those are some meaty competitors and as you have seen before, I love a good fight.

In my opinion, Nokia has some very interesting characteristics that position it extremely well if not atop the fray in this space.   First there is no arguing about Nokia penetration of hand-held devices across the world.  Especially in markets like India, China, South America, and other emerging Internet-using populations.    Additionally these emerging economies are skipping past ground-based wired technologies to wireless connectivity.   As a result of that, Nokia has an incredible presence already in those markets.   Their OVI platform today already has a significant population of users (measured at least in the 10s of millions) and so scale at the outset is definitely there.    When I think about the challenge that Google has in getting device penetration out there, or Apples high-end (and mostly US) only approach you can see the opportunity.    I am extremely excited to get going.

Hope you will join me for an incredible ride!

\Mm

Read Full Post »

A Digital Adieu

Those who follow the news from Digital Realty Trust closely may have recently read that I have decided to leave the company to focus a bit more on some personal work/life balance issues.  With this move comes a new role that I will talk more of in the coming days and weeks.

I would like to take a moment and talk about my time and experience at Digital and what I believe to be some industry ground breaking work that is being done there.   The first thing that strikes me about the company is the quality and dedication of the people.  The staff within the organization are incredibly committed to both providing the best product  (in terms of engineering and construction) along with an obsessive regimen around Operations.  In my role running all aspects of design, construction, and operations, this passion showed through every single day.    It was a delight to work with such motivated people. 

From the outside it might be difficult to gauge just how significant this operation truly is.   As many of you know I have run some large programs before, but they all pale in comparison to the size, scope, and complexity of the work happening at DLR.  Its one thing to be building a couple of very large facilities and quite another to be building out tens upon tens of data center construction initiatives across the world.   There simply is no organization in the world that has to construct, manage and operate more data centers, period.   In addition to these “block and tackling” items there is also a healthy focus on modularization and evolving data center design and prototyping.   This focus is not just about driving additional efficiencies in power and cooling, but also in cost, and time to deploy.  A true intersection of business requirements.  On top of all this you add the Pod Architecture Services program and Build to Suit program which additionally extend Digitals capabilities to those looking to build “Do it Yourself” (DIY) Data Centers.   In short, it was a ton of fun with incredible opportunities for growth.

In my time at the company I have focused on driving additional streamlining efforts and operational rigor across the board and have helped set the engineering direction of the company.   This work has already begun to pay some significant dividends and I am sure will likely continue well into the future.   But let me be clear – The success of these initiatives will be delivered by a top rate team with few peers in the industry.  

In short, Digital was a great experience and I feel blessed in having made some life-long friends there as well.   So as I start a new chapter in my life, a bid fond adieu to a Data Center Juggernaut and look boldly forward to what is to come, for me and for Digital.

\Mm

Read Full Post »

Private Clouds – Not just a Cost and Technology issue, Its all about trust, the family jewels, corporate value, and identity

I recently read a post by my good friend James Hamilton at Amazon regarding Private Clouds.   James and I worked closely together at Microsoft and he was always a good source for out of the box thinking and challenging the status quo.    While James post found here, speaks to the Private Cloud initiative being what amounts to be an evolutionary dead end, I would have to respectfully disagree.

James’ post starts out by correctly pointing out that at scale the large cloud players have the resources and incentive to achieve some pretty incredible cost savings.  From an infrastructure perspective he is dead on.  But I don’t necessarily agree that this innovation will never reach the little guy.  In my role at Digital Realty Trust I think I might have a pretty unique perspective on the infrastructure developments both at the “big” guys along with what most corporate enterprises have available to them from a leasing or commercial perspective.  

Companies like Digital Realty Trust,  Equinix, Terramark, Dupont Fabros, and a host of others in the commercial data center space are making huge advancements in this space as well.   The free market economy has now placed an importance on low PUE highly efficient buildings.   You are starting to see these firms commission buildings with Commission PUEs Sub 1.4.   Compared to most existing data center facilities this is a huge improvement.  Likewise these firms are incented to hire mechanical and electrical experts.  This means that this same expertise is available to the enterprise through leasing arrangements.  Where James is potentially correct is at that next layer of IT specific equipment.

This is an area where there is an amazing amount of innovation happening by Amazon, Google, and Microsoft.   But even here in this space there are firms stepping up to provide solutions to bring extensive virtualization and cloud-like capabilities to bear.    Companies like Hexagrid have software solutions offerings that are being marketed to typical co-location and hosting firms to do the same thing.  Hexagrid and others are focusing on the software and hardware combinations to deliver full service solutions for those companies in this space.    In fact (as some comments on James’ blog mention) there is a lack of standards and a fear of vendor lock-in by choosing one of the big firms.  Its an interesting thought to ponder if a software+hardware solution offered to the hundreds of co-location players and hosting firms might be more of a universal solution without fear of lockdown.  Time will tell.

But this brings up one of the key criticisms that this is not just about cost and technology.   I believe what is really at stake here is much more than that.   James makes great points on greater resource utilization of the big cloud players and how much more efficient they are at utilizing their infrastructure.   To which i will snarkly (and somewhat tongue-in-cheek) say to that, “SO WHAT!”     Do enterprises really care about this?  Do they really optimize for this?  I mean if you pull back that fine veneer of politically correct answers  and “green-suitable” responses is that what their behavior in REAL LIFE is indicative of?    NO.

This was a huge revelation for me when I moved into my role at Digital.  When I was at Microsoft, I optimized for all of the things that James mentions because it made sense to do when you owned the whole pie.   In my role at Digital I have visibility into tens of data centers, across hundreds of customers that span just about every industry.  There is not, nor has there been a massive move (or any move for that matter) to become more efficient in the utilization of their resources.   We have had years of people bantering about how wonderful, cool, and how revolutionary a lot of this stuff is, but world wide Data center utilization levels have remained abysmally low.   Some providers bank on this.  Over subscription of their facilities is part of their business plan.  They know companies will lease and take down what they think they need, and never take it down in REALITY.   

So if this technology issue is not a motivating factor what is?  Well cost is always part of the equation.   The big cloud providers will definitely deliver cost savings, but private clouds could also deliver cost savings as well.   More importantly however, Private clouds will allow companies to retain their identity and uniqueness, and keep what makes them competitively them –Them.

I don’t so much see it as a Private cloud or Public cloud kind of thing but more of a Private Cloud AND Public cloud kind of thing.   To me it looks more an exercise of data abstraction.   The Public offerings will clearly offer infrastructure benefits in terms of cost, but will undoubtedly lock a company into that single solution.  The IT world has been bit before by putting all their eggs in a single basket and the need for flexibility will remain more key.    Therefore you might begin to see Systems Integrators, Co-location and hosting firms, and others build their own platforms, or much more likely, build platforms that umbrella over the big cloud players to give enterprises the best of both worlds. 

Additionally we must keep in mind that  the biggest resistance to the adoption of the cloud is not technology or cost but RISK and TRUST.  Do you, Mr. CIO, trust Google to run all of your infrastructure? your applications?  Do you Mrs. CIO, Trust Microsoft or Amazon to do the same for you?    The answer is not a blind yes or no.   Its a complicated set of minor yes responses and no responses.   They might feel comfortable outsourcing mail operations, but not the data warehouse manifesting decades of customer information.     The Private cloud approach will allow you to spread your risk.   It will allow you to maintain those aspects of the business that are core to the company. 

The cloud is an interesting place, today.  It is dominated by technologists.  Extremely smart engineering people who like to optimize and solve for technological challenges.  The actual business adoption of this technology set has yet to be fully explored.   Just wait until the “Business” side of the companies get their hooks into this technology set and start placing other artificial constraints, or optimizations around other factors.  There are thousands of different motivators out in the world.  Once they starts to happen earnest.  I think what you will find is a solution that looks more like a hybrid solution than the pure plays we dream about today.

Even if you think my ideas and thoughts on this topic is complete BS, I would remind you of something that I have told my teams for a very long time.  “There is no such thing as a temporary data center.”  This same mantra will hold true for the cloud.  If you believe that the Private Cloud will be a passing and temporary thing, just keep in mind that there will be systems and solutions build to this technology approach thus imbuing it with a very very long life.  

\Mm

Read Full Post »