Spooky Action Issues Ahead for the Just-out-of-Reachables

I had an interesting conversation this week with a good friend of mine who works at one of the big Internet companies. We hadn’t talked in a good while so it turned out to be a wide ranging conversation that meandered back and forth from our mutual experiences building globally scaled technology platforms, past companies, current companies, to the possibility of getting together soon for beers when we happened to be in the same part of the world at the same time.  It ultimately landed on the topic of EMV and some of the unique challenges in the Payments space.  While the questions were initially focused on consumer experiences, it quickly devolved (as they always do with my friends) into a technology and business conversation. 

One of my most substantial learnings moving from ‘Big Internet’ to the Payments and Fintech space is that the Internet guys do not have a lock on scale or its challenges.  I used to live in that world of hubris where all the tough and hard challenges were exclusively found, created, and solved within a hundred miles of the Pacific Ocean shoreline.  That is not to say that they do not solve those things, its just not as exclusive as they believe. Reality has a way of kicking you in the pants.  I know it did mine.

Whether it be geo-diversity in infrastructure or software platforms the need for global reach and footprint balanced against localization is not only important in an ever-shrinking world, its mandatory.  This is especially true in Fintech.   One does not get to focus singly on Business to Business Scenarios or Business to Consumer scenarios separately, you have to focus on both.  Throw in country by country regulatory requirements around privacy, commerce, data access, taxation, industry and customer oversight and governance and you end up with a giant three dimensional jigsaw puzzle of international complexity.    All of that is before you get to really hard stuff… interfacing with the customers and merchants. Now our jigsaw puzzle has reached the 4th dimension and I am pretty sure quantum mechanics and spooky action at a distance come into play in there somewhere.

‘Hey Mike, Why isn’t everyone moved over to the Chip and Pin (or EMV) in the United States yet?’.  The inference was that somehow the Banks, companies like mine, and others were not doing enough to protect the consumer.   To really answer his question I had to first talk about the complexity of the Point of Sale and merchant eco-system that exists between his purchase of a slice of pizza and its ultimate funds settlement to his bank.   It may seem straightforward to folks who do not understand all of the pieces, but it can actually be pretty complex.  

The ends are pretty simple –on one side you have the consumer interfacing with the merchant.  On the other is the bank or credit card provider where the funds are ultimately provided.   Most people conceptually understand those two parts.  However the parts in the middle get a little tangled up.

The merchant may get its payments servicing from any one of a different number of types of businesses.  The first way would be through a direct payment processor like FirstData, a second could be through an independent sales organization (or ISO) who can have relationships with a single or multiple payment processors, a third way could be through merchant’s banking partner directly who in turn has relationships with a payment processor, another way would be through an Integrated Software Vendor (or ISV) who adds additional software capabilities before it hits a payment processor.  In these examples the ISO, ISV, and Banking partner still interface at some point with a payment processor before connecting to the Credit Card Associations (or schemes ) like American Express, Discover, Mastercard, or Visa, and then ultimately the bank.  The path any particular transaction could take could cross many different providers.  Depending on the route and merchant up, there could be some significant distance between the consumer end and the settlement end on the bank.  Additionally the question of who owns the full transaction path gets fuzzy as all of these players intersect with each other in the transaction flow.

In trying to answer the question about EMV adoption the solution could be complex.  In many cases the ISO and the ISV may have additional software in the mix that is not EMV compliant and the merchant must wait for their service to be compliant before they can adopt. In many cases the point of sale device they use may not be EMV compliant either and they are waiting on their business partner to provide one that is.  Then of course there is normal human apathy or a lack of desire to spend the money to upgrade on the merchants behalf.  All of these things play a part. 

While many focus on the added security for the consumer with the EMV cards, the impact on the merchant is slightly different.  It reality represents a shift in liability away from the banks and credit card companies to the merchants themselves.  If a merchant does not adopt the technology, any fraud related activity from their store will be their responsibility and will need to take the loss rather than the bank.  Some merchants feel like they are willing to take that risk given their customer and product mix.  The last category are merchants, who like most consumers, don’t understand the complexity nor do they want to and generally ignore it.  

While great strides have been made to drive EMV adoption, we are sitting woefully short of the adoption to those folks listed in the categories above.   I call these folks the ‘Just-Out-Of-Reachables’ and they have been a hard nut to crack for the industry at large.  It has tried for multiple years to prepare, communicate and execute for this. To be fair these interactions have not stopped commerce, and have only put the individual merchants at risk for fraud for the most part.

There is however a potentially more fatal issue on the horizon that will stop transactions and could directly impact the merchants and consumers alike.   While somewhat technical in nature it has to do with a move from SHA1 to SHA2 hash certificates.  Many solutions in the marketplace were initially written with the SHA1 security specifications and dependencies.  Over time that certificate has proven to be less secure and the industry at large has been told to migrate to SHA2 certificates.  While many of the big players have made the jump from a technology perspective to the SHA2 standard, the issue significantly overlaps and shares the same challenges with reaching out and solving for the ‘Just-Out-Of-Reachables’.  Only in this case, once the SHA1 certificates expire, their point of sale systems will stop working altogether.  Working with the browser forums has resulted in some firms getting small extensions to SHA1 certificate expirations, but it has not been uniformly consistent with some firms getting differing lengths of extensions, and some none at all.   Its a real issue for the industry at large and we should start to see these impacts over the next few months. 

It will be interesting to see how this plays out especially given the potential public exposure of it. 


The Weight of Technical Liberty…Cutting the Cruft

Over the next few months, it’s my sincere desire to share with you some of the amazing technology accomplishments currently underway at First Data and how we are attempting to change the industry.  In any conversation about the future, you must begin by framing the past.   As you may or may not know First Data is a company that was founded in 1971.   It is a company hallmarked in its early years by significant technology innovation with a number of ‘firsts’ in the enablement of credit card processing across the globe. 

Throughout the years the company grew both organically as well as through large numbers of mergers and acquisitions on a global scale which ultimately enabled it to become the international leader it is today.  I will spare folks a deeper commercial of the company only to state that today it has more scale and technology reach than any other company like it in the #Fintech space. 

I share this information because it’s that unencumbered growth over decades of acquisition, an evolving and changing regulatory and compliance field of requirements, and a historically growing list of platforms and services that ultimately led to the largest trove of ‘Cruft’ I have ever been challenged with in my personal career. It’s a challenge 45 years in the making. 

As you may recall I first defined ‘Cruft’ while engaged at the Turn-Around at AOL:

Cruft is defined as years of build-up of technology, processes, politics, fiscal orphaning, and poor operational hygiene that ultimately impede technical agility and operation.  Additionally, Cruft can create an acidic cloud of lethargy or apathy in the workforce that ultimately sucks the energy out of innovation from within.

When I originally defined the term I was referring to the work we accomplished attacking the Cruft in a different organization which ultimately led to the company winning the Uptime Institute’s “Server Round-Up” Award. That award was created to promote full IT and Facilities integration and improve overall energy efficiency.  While recognized for the energy efficiency improvement, it was really a by-product of other technological and organizational wins for the company.

Our work on ‘attacking the Cruft’ at First Data has resulted in similar, in fact, greater energy cost savings, but more importantly it has reduced and continues to reduce the operational complexity of our environments.  Attacking the Cruft problem along the technology, process, and hygiene axes have resulted in some very powerful and significant results.  While we are far from completing the task, the last twenty-four (24) months have yielded some mind-numbing progress.

Is this really my metric? So Not Technical…

The first challenge I had was trying to find a way to truly quantify the reductions in a metric everyone could understand.   Simply counting servers was not enough, it could not account for other devices like storage equipment, network equipment, and other kit that does not easily fold into that definition.   Measuring power usage decreases, while absolutely telling the effort from a purely technical perspective, obfuscated the tremendous amount of work and passion the teams poured into modernizing our plant.  Many of the consumers of the information of our modernization efforts are not technology or energy wonks.   We had to come up with a metric that was universal.  That everyone, even non-technical people could understand and visualize.   In the end, we settled on the ‘ton’. 

I know what you are thinking…the ton?  As in… like..weight?


It’s not as cool as measuring in megawatts, or measured computational capacity, or MIPS, or IOPs, or whatever metric is fashionable these days, but it is universal.  Additionally, the scale of the work output would just get lost.   So what did we achieve over the last 24 months?

  1. We removed 220+ tons of IT Equipment from our global data centers.
  2. We consolidated and shutdown 5 data centers across the world; and have an aggressive plan to continue to consolidate more.
  3. We employed large-scale internal virtualization technology, open source cloud technologies, and are building a hybridized cloud controller that has resulted in moving nearly 75% of our physical distributed server environments to a virtualized footprint. (I will share more on that in a different post).

There were significant other achievements as well which we can discuss at a later date.  But as I said, we had to set the framework of what the starting position was.   We still have a mountain of work in this space to do but the momentum has started and passions have been ignited.  Those passions are blowing away that “acidic cloud” that results from Cruft.  The results speak for themselves.  That is an incredible amount of work to achieve in just 24 months.  It’s not just about establishing a set of technical goals for an organization to achieve.  As a leader it’s about ensuring that you have created the fertile soil for those changes to take place and have empowered your people to make decisions along that alignment. 

Of course, none of this could have been achieved if the firm from the top down was dedicated to driving this kind of significant change.   First Data is truly blessed with a board and leadership team who not only understand technology, they have lived it, they have managed it, they have won with it.   It’s a very unique set of variables that have been toggled.

While tonnage may be an easier metric for non-techies to understand how much equipment was removed,  it is hard to grasp just how much 220 tons actually represents.  As these efforts over the last two years have created more operational simplicity giving us the freedom and liberty to expand and explore new technology approaches it is only fitting to associate it with the Statue of Liberty.  Which by coincidence also weighs 220 tons.  Visualize that.


Dinner with @FirstData CTO at GATech Career Fair on 9/13


We will have representatives from my First Data technology teams at the Georgia Tech 2016 Career Fair on September 13th, 2016.   If you are a current Georgia Tech student or alum looking for some incredible opportunities its definitely worth stopping by to see what we are all about!  We have a bunch of great positions open in the Atlanta, Denver, New York and a host of other markets.

In addition we will be choosing 10 resumes at random for a private dinner with me, First Data’s Chief Technology Officer.  Think of it as an opportunity to grow your network, ask career related questions, or an opportunity to just get a free bite to eat.   Just you, me, food, and drink.   All you have to do is ensure that you drop your resume off at the First Data booth to enter.

You can tweet or follow #GATechWork4FD for more information as well.