The Weight of Technical Liberty…Cutting the Cruft

Over the next few months, it’s my sincere desire to share with you some of the amazing technology accomplishments currently underway at First Data and how we are attempting to change the industry.  In any conversation about the future, you must begin by framing the past.   As you may or may not know First Data is a company that was founded in 1971.   It is a company hallmarked in its early years by significant technology innovation with a number of ‘firsts’ in the enablement of credit card processing across the globe. 

Throughout the years the company grew both organically as well as through large numbers of mergers and acquisitions on a global scale which ultimately enabled it to become the international leader it is today.  I will spare folks a deeper commercial of the company only to state that today it has more scale and technology reach than any other company like it in the #Fintech space. 

I share this information because it’s that unencumbered growth over decades of acquisition, an evolving and changing regulatory and compliance field of requirements, and a historically growing list of platforms and services that ultimately led to the largest trove of ‘Cruft’ I have ever been challenged with in my personal career. It’s a challenge 45 years in the making. 

As you may recall I first defined ‘Cruft’ while engaged at the Turn-Around at AOL:

Cruft is defined as years of build-up of technology, processes, politics, fiscal orphaning, and poor operational hygiene that ultimately impede technical agility and operation.  Additionally, Cruft can create an acidic cloud of lethargy or apathy in the workforce that ultimately sucks the energy out of innovation from within.

When I originally defined the term I was referring to the work we accomplished attacking the Cruft in a different organization which ultimately led to the company winning the Uptime Institute’s “Server Round-Up” Award. That award was created to promote full IT and Facilities integration and improve overall energy efficiency.  While recognized for the energy efficiency improvement, it was really a by-product of other technological and organizational wins for the company.

Our work on ‘attacking the Cruft’ at First Data has resulted in similar, in fact, greater energy cost savings, but more importantly it has reduced and continues to reduce the operational complexity of our environments.  Attacking the Cruft problem along the technology, process, and hygiene axes have resulted in some very powerful and significant results.  While we are far from completing the task, the last twenty-four (24) months have yielded some mind-numbing progress.

Is this really my metric? So Not Technical…

The first challenge I had was trying to find a way to truly quantify the reductions in a metric everyone could understand.   Simply counting servers was not enough, it could not account for other devices like storage equipment, network equipment, and other kit that does not easily fold into that definition.   Measuring power usage decreases, while absolutely telling the effort from a purely technical perspective, obfuscated the tremendous amount of work and passion the teams poured into modernizing our plant.  Many of the consumers of the information of our modernization efforts are not technology or energy wonks.   We had to come up with a metric that was universal.  That everyone, even non-technical people could understand and visualize.   In the end, we settled on the ‘ton’. 

I know what you are thinking…the ton?  As in… like..weight?

Yes. 

It’s not as cool as measuring in megawatts, or measured computational capacity, or MIPS, or IOPs, or whatever metric is fashionable these days, but it is universal.  Additionally, the scale of the work output would just get lost.   So what did we achieve over the last 24 months?

  1. We removed 220+ tons of IT Equipment from our global data centers.
  2. We consolidated and shutdown 5 data centers across the world; and have an aggressive plan to continue to consolidate more.
  3. We employed large-scale internal virtualization technology, open source cloud technologies, and are building a hybridized cloud controller that has resulted in moving nearly 75% of our physical distributed server environments to a virtualized footprint. (I will share more on that in a different post).

There were significant other achievements as well which we can discuss at a later date.  But as I said, we had to set the framework of what the starting position was.   We still have a mountain of work in this space to do but the momentum has started and passions have been ignited.  Those passions are blowing away that “acidic cloud” that results from Cruft.  The results speak for themselves.  That is an incredible amount of work to achieve in just 24 months.  It’s not just about establishing a set of technical goals for an organization to achieve.  As a leader it’s about ensuring that you have created the fertile soil for those changes to take place and have empowered your people to make decisions along that alignment. 

Of course, none of this could have been achieved if the firm from the top down was dedicated to driving this kind of significant change.   First Data is truly blessed with a board and leadership team who not only understand technology, they have lived it, they have managed it, they have won with it.   It’s a very unique set of variables that have been toggled.

While tonnage may be an easier metric for non-techies to understand how much equipment was removed,  it is hard to grasp just how much 220 tons actually represents.  As these efforts over the last two years have created more operational simplicity giving us the freedom and liberty to expand and explore new technology approaches it is only fitting to associate it with the Statue of Liberty.  Which by coincidence also weighs 220 tons.  Visualize that.

\Mm