Spooky Action Issues Ahead for the Just-out-of-Reachables

I had an interesting conversation this week with a good friend of mine who works at one of the big Internet companies. We hadn’t talked in a good while so it turned out to be a wide ranging conversation that meandered back and forth from our mutual experiences building globally scaled technology platforms, past companies, current companies, to the possibility of getting together soon for beers when we happened to be in the same part of the world at the same time.  It ultimately landed on the topic of EMV and some of the unique challenges in the Payments space.  While the questions were initially focused on consumer experiences, it quickly devolved (as they always do with my friends) into a technology and business conversation. 

One of my most substantial learnings moving from ‘Big Internet’ to the Payments and Fintech space is that the Internet guys do not have a lock on scale or its challenges.  I used to live in that world of hubris where all the tough and hard challenges were exclusively found, created, and solved within a hundred miles of the Pacific Ocean shoreline.  That is not to say that they do not solve those things, its just not as exclusive as they believe. Reality has a way of kicking you in the pants.  I know it did mine.

Whether it be geo-diversity in infrastructure or software platforms the need for global reach and footprint balanced against localization is not only important in an ever-shrinking world, its mandatory.  This is especially true in Fintech.   One does not get to focus singly on Business to Business Scenarios or Business to Consumer scenarios separately, you have to focus on both.  Throw in country by country regulatory requirements around privacy, commerce, data access, taxation, industry and customer oversight and governance and you end up with a giant three dimensional jigsaw puzzle of international complexity.    All of that is before you get to really hard stuff… interfacing with the customers and merchants. Now our jigsaw puzzle has reached the 4th dimension and I am pretty sure quantum mechanics and spooky action at a distance come into play in there somewhere.

‘Hey Mike, Why isn’t everyone moved over to the Chip and Pin (or EMV) in the United States yet?’.  The inference was that somehow the Banks, companies like mine, and others were not doing enough to protect the consumer.   To really answer his question I had to first talk about the complexity of the Point of Sale and merchant eco-system that exists between his purchase of a slice of pizza and its ultimate funds settlement to his bank.   It may seem straightforward to folks who do not understand all of the pieces, but it can actually be pretty complex.  

The ends are pretty simple –on one side you have the consumer interfacing with the merchant.  On the other is the bank or credit card provider where the funds are ultimately provided.   Most people conceptually understand those two parts.  However the parts in the middle get a little tangled up.

The merchant may get its payments servicing from any one of a different number of types of businesses.  The first way would be through a direct payment processor like FirstData, a second could be through an independent sales organization (or ISO) who can have relationships with a single or multiple payment processors, a third way could be through merchant’s banking partner directly who in turn has relationships with a payment processor, another way would be through an Integrated Software Vendor (or ISV) who adds additional software capabilities before it hits a payment processor.  In these examples the ISO, ISV, and Banking partner still interface at some point with a payment processor before connecting to the Credit Card Associations (or schemes ) like American Express, Discover, Mastercard, or Visa, and then ultimately the bank.  The path any particular transaction could take could cross many different providers.  Depending on the route and merchant up, there could be some significant distance between the consumer end and the settlement end on the bank.  Additionally the question of who owns the full transaction path gets fuzzy as all of these players intersect with each other in the transaction flow.

In trying to answer the question about EMV adoption the solution could be complex.  In many cases the ISO and the ISV may have additional software in the mix that is not EMV compliant and the merchant must wait for their service to be compliant before they can adopt. In many cases the point of sale device they use may not be EMV compliant either and they are waiting on their business partner to provide one that is.  Then of course there is normal human apathy or a lack of desire to spend the money to upgrade on the merchants behalf.  All of these things play a part. 

While many focus on the added security for the consumer with the EMV cards, the impact on the merchant is slightly different.  It reality represents a shift in liability away from the banks and credit card companies to the merchants themselves.  If a merchant does not adopt the technology, any fraud related activity from their store will be their responsibility and will need to take the loss rather than the bank.  Some merchants feel like they are willing to take that risk given their customer and product mix.  The last category are merchants, who like most consumers, don’t understand the complexity nor do they want to and generally ignore it.  

While great strides have been made to drive EMV adoption, we are sitting woefully short of the adoption to those folks listed in the categories above.   I call these folks the ‘Just-Out-Of-Reachables’ and they have been a hard nut to crack for the industry at large.  It has tried for multiple years to prepare, communicate and execute for this. To be fair these interactions have not stopped commerce, and have only put the individual merchants at risk for fraud for the most part.

There is however a potentially more fatal issue on the horizon that will stop transactions and could directly impact the merchants and consumers alike.   While somewhat technical in nature it has to do with a move from SHA1 to SHA2 hash certificates.  Many solutions in the marketplace were initially written with the SHA1 security specifications and dependencies.  Over time that certificate has proven to be less secure and the industry at large has been told to migrate to SHA2 certificates.  While many of the big players have made the jump from a technology perspective to the SHA2 standard, the issue significantly overlaps and shares the same challenges with reaching out and solving for the ‘Just-Out-Of-Reachables’.  Only in this case, once the SHA1 certificates expire, their point of sale systems will stop working altogether.  Working with the browser forums has resulted in some firms getting small extensions to SHA1 certificate expirations, but it has not been uniformly consistent with some firms getting differing lengths of extensions, and some none at all.   Its a real issue for the industry at large and we should start to see these impacts over the next few months. 

It will be interesting to see how this plays out especially given the potential public exposure of it. 

\Mm

The Weight of Technical Liberty…Cutting the Cruft

Over the next few months, it’s my sincere desire to share with you some of the amazing technology accomplishments currently underway at First Data and how we are attempting to change the industry.  In any conversation about the future, you must begin by framing the past.   As you may or may not know First Data is a company that was founded in 1971.   It is a company hallmarked in its early years by significant technology innovation with a number of ‘firsts’ in the enablement of credit card processing across the globe. 

Throughout the years the company grew both organically as well as through large numbers of mergers and acquisitions on a global scale which ultimately enabled it to become the international leader it is today.  I will spare folks a deeper commercial of the company only to state that today it has more scale and technology reach than any other company like it in the #Fintech space. 

I share this information because it’s that unencumbered growth over decades of acquisition, an evolving and changing regulatory and compliance field of requirements, and a historically growing list of platforms and services that ultimately led to the largest trove of ‘Cruft’ I have ever been challenged with in my personal career. It’s a challenge 45 years in the making. 

As you may recall I first defined ‘Cruft’ while engaged at the Turn-Around at AOL:

Cruft is defined as years of build-up of technology, processes, politics, fiscal orphaning, and poor operational hygiene that ultimately impede technical agility and operation.  Additionally, Cruft can create an acidic cloud of lethargy or apathy in the workforce that ultimately sucks the energy out of innovation from within.

When I originally defined the term I was referring to the work we accomplished attacking the Cruft in a different organization which ultimately led to the company winning the Uptime Institute’s “Server Round-Up” Award. That award was created to promote full IT and Facilities integration and improve overall energy efficiency.  While recognized for the energy efficiency improvement, it was really a by-product of other technological and organizational wins for the company.

Our work on ‘attacking the Cruft’ at First Data has resulted in similar, in fact, greater energy cost savings, but more importantly it has reduced and continues to reduce the operational complexity of our environments.  Attacking the Cruft problem along the technology, process, and hygiene axes have resulted in some very powerful and significant results.  While we are far from completing the task, the last twenty-four (24) months have yielded some mind-numbing progress.

Is this really my metric? So Not Technical…

The first challenge I had was trying to find a way to truly quantify the reductions in a metric everyone could understand.   Simply counting servers was not enough, it could not account for other devices like storage equipment, network equipment, and other kit that does not easily fold into that definition.   Measuring power usage decreases, while absolutely telling the effort from a purely technical perspective, obfuscated the tremendous amount of work and passion the teams poured into modernizing our plant.  Many of the consumers of the information of our modernization efforts are not technology or energy wonks.   We had to come up with a metric that was universal.  That everyone, even non-technical people could understand and visualize.   In the end, we settled on the ‘ton’. 

I know what you are thinking…the ton?  As in… like..weight?

Yes. 

It’s not as cool as measuring in megawatts, or measured computational capacity, or MIPS, or IOPs, or whatever metric is fashionable these days, but it is universal.  Additionally, the scale of the work output would just get lost.   So what did we achieve over the last 24 months?

1. We removed 220+ tons of IT Equipment from our global data centers.
2. We consolidated and shutdown 5 data centers across the world; and have an aggressive plan to continue to consolidate more.
3. We employed large-scale internal virtualization technology, open source cloud technologies, and are building a hybridized cloud controller that has resulted in moving nearly 75% of our physical distributed server environments to a virtualized footprint. (I will share more on that in a different post).

There were significant other achievements as well which we can discuss at a later date.  But as I said, we had to set the framework of what the starting position was.   We still have a mountain of work in this space to do but the momentum has started and passions have been ignited.  Those passions are blowing away that “acidic cloud” that results from Cruft.  The results speak for themselves.  That is an incredible amount of work to achieve in just 24 months.  It’s not just about establishing a set of technical goals for an organization to achieve.  As a leader it’s about ensuring that you have created the fertile soil for those changes to take place and have empowered your people to make decisions along that alignment. 

Of course, none of this could have been achieved if the firm from the top down was dedicated to driving this kind of significant change.   First Data is truly blessed with a board and leadership team who not only understand technology, they have lived it, they have managed it, they have won with it.   It’s a very unique set of variables that have been toggled.

While tonnage may be an easier metric for non-techies to understand how much equipment was removed,  it is hard to grasp just how much 220 tons actually represents.  As these efforts over the last two years have created more operational simplicity giving us the freedom and liberty to expand and explore new technology approaches it is only fitting to associate it with the Statue of Liberty.  Which by coincidence also weighs 220 tons.  Visualize that.

\Mm

Just like the NASA Stereo-B Probe…

On August 23rd 2016, NASA announced that its DSN (Deep Space Network) had re-established connection with the Stereo-B probe whose joint mission (with its counterpart probe Stereo-A) was to orbit the sun.  The purpose of the twin probes managed by the Johns Hopkins University Applied Physics Laboratory in Maryland was to use the two probes to see the entire star. Not just the side facing our planet, but the other side as well.  The folks lost communication with the probe, but never gave up.  Two years later (just a few days ago in fact) they were able to re-connect to the silent space adventurer and the team is hard at work trying to get things back into working order.

How does that have anything to do with the LooseBolts blog you ask?  There are some interesting corollaries to our prodigal space mission and my ramblings here on Loosebolts.   The obvious one of course is that its been about two years since my last post.   I didn’t time it that way but it is kind of spooky.   Similarly just as Stereo-B had its counterpart in the Stereo-A mission, its not like I have been really silent as I have been maintaining a blog internally to my work at First Data.  That internal blog has been blasting out all kinds of interesting, pithy, technical, optimistic, critical, business-relevant, and some non-business-relevant posts throughout the entirety of the time.

My internal blog is called Random Thoughts.  The dual blog system (one focused internally and the other focused externally) is a mechanism I have used for a very long time throughout my career.  Like our twin solar satellites, Random Thoughts has been buzzing away on the far side of the sun accomplishing its half of the mission.   Loosebolts however, had to take a backseat until I could get some of my house in order. 

To be honest my ‘Random Thoughts’ are not really all that random and are almost exclusively focused primarily on the work at hand inside the firm.  In my opinion Communication is the number one factor whether you will have an effective organization.  This importance is magnified 1000 times if you are actively engaged in the turn-around of the business.    Communication defines expectations. Communication defines concerns.  Communication defines Priority and Importance.  Communication defines direction.  Most importantly – Communications defines culture. 

This last bit is incredibly important when engaging in re-inventing a company which essentially defined the first and second generations of an industry. A company that has been in operation almost 40 years and whose next evolution maintains as a basic precept that technology will be a defining factor in its paradigm shift. The target of so many disrupters trying to out disrupt itself and everyone else at the same time.  A company that has scale beyond any other in the #Fintech space in terms of reach but one that also has to contend with significant legacy technology decisions. 

Communications are in full effect. In addition to the internal blog, I also send out a personal note to the entire technology organization summarizing the work accomplished and challenges discovered each week every Friday.  We have even deployed our own internal Global Technology TV Station focused around our execution.

On the Technology side, we have been hard at work doing some incredibly cool things. Things you cannot do without scale to start.   Problems, opportunities, and a vantage point really unimaginable for most.  Of course it will all be about execution. The execution (both looking ahead and looking behind) is well underway. 

While I have personally participated in some very large turn-around’s and technology strategy shifts before – the learnings and challenges here have truly changed my perspective across a number of areas.  The strange result of dropping an Internet minded technologist in the middle of Financial Services with the executive support required to make the change.  I hope to share many of these with you here again on Loosebolts.  We are getting ready for some interesting transitions and revelations. 

Communication initiated to Loosebolts blog….

Connection Established.

Sending queued messages…. 

\Mm

On the Move: First Data

Well after a bit of time in stealth I am finally able to announce that I have taken the position of Chief Technology Officer at First Data. 

After being asked to join the Turn-Around team at AOL and driving some amazing results over the past four years, it was time for a change.  I absolutely loved my time there and the people were amazing.  Success has a quality all on its own and it was an incredibly personally rewarding experience for me to be a part of something that unique.

The move to First Data is an incredibly exciting move for me for many different reasons but one of the key drivers for me is that I feel that this industry is ripe for change.  It’s a move for me from running and building large scale Internet products and infrastructure to the Financial Services Industry.  

For those of you who may be be unaware of who First Data is, or what they do, its probably easiest to think of it this way – one out of every two credit card or debit card transactions across the world touches our infrastructure at some point in the transaction process. From a transactional scale perspective its very similar to what I have been used to companies likes AOL, Microsoft, and Disney.  The difference being of course that these transactions are a little more important than checking your favorite sports scores, or getting your e-mail. 

My challenge of course is to drive automation. To build a platform that makes infrastructure a decisive and differentiating platform to launch products and services.  To create a learning infrastructure and software eco-system that gets smarter over time.   In large part how do you blend the agility of the Internet with the maturity and complexity of the Financial Services Industry.   Sure it’s a complex technical problem space, but it has some very interesting business and regulatory challenges as well.   In many respects dealing with Safe Harbor, Regulatory and tax has been part of my job for many years.  The challenge now is to take that automation to the next level.  

To that end I have to say that First Data is assembling an amazingly formidable team to drive this change.  I will be reporting to the President of First Data, Guy Chiarello.  Guy is a universally respected Technology leader in the Financial Services industry with top posts at Morgan Stanley and JP Morgan Chase.  Technology will be key to the success of the company and the leadership team is a unique blend of technology savvy leaders from across the world. 

The new adventure begins!

You can follow the link to the official press announcement.

Along with the initial pickup from the Wall Street Journal.

\Mm