[Warning: This post includes a lot of information about modern computer systems. I’ve written it for the average person, but you may not be interested in it at all. It’s important to understand why the US government should not have a useless Terrorist Screening Database (TSDB) four years after the project officially started in 2003, six years after the 9/11 attacks. If you find your eyes glazing over, just skip ahead to the next post in this series. I find this topic interesting, but you may not.]
The obstacles to building the terrorist watch list are the same facing private corporations during mergers and acquisitions. When company A acquires company B, two organizations must now merge incompatible storehouses of information. If company A uses a particular piece of software to manage customer information, and company B uses a different application, the two sets of information won’t match.
Different definitions of "important"
Nowhere is this pain felt more keenly than the customer database—or, more accurately, databases. Without a standard way to describe customers, the two companies’ customer relations management (CRM) systems will have different “data models.” There will be overlap in obvious places: for example, everyone needs the name of a customer. However, the two systems will have many key differences: one CRM application might record detailed information about marketing campaigns; the other CRM application might provide a far sketchier picture of marketing effectiveness.
To complicate matters, no company ever leaves critical applications (CRM, accounting, HR, etc.) untouched. Because these systems manage important information, they need to include the information that’s important to that company. No application ever perfectly describes that information for every possible organization, so some “customization” is necessary to add or modify information. Companies often spend millions of dollars to perform these customizations, so you can see how important they are.
Therefore, mergers and acquisitions immediately create an information management crisis. Day-to-day business operations depend on information that is now stored in incompatible systems. The business and legal costs of leaving the information in this Balkanized form are too high, but the scope of the problem—potentially, millions of records in incompatible databases—is daunting.
Hired guns to the rescue
At this point, corporations look to a combination of software and specialists for help. Many software companies exist purely to provide tools for merging different databases. Each of these software vendors usually has its own specialization, such as reconciling product catalogues, or migrating data from one popular accounting system to another.
However, these software tools are never going to provide a “pushbutton” solution. Company A won’t be able to install the software, click a button, and Voila! the problem disappears.
One obstacle is customization: it’s extremely difficult to preserve the customizations as you move information from system X to system Y, or from both systems into some separate, combined data repository. The software might be able to follow your instructions for handling the added information. However, unless you are dangerously overconfident about the software’s reliability (or, for that matter, your own instructions), you’ll have to double-check the results.
Therefore, another major obstacle to reconciling databases is “quality assurance.” All too frequently, migrated information is garbled, duplicated, or missing. It’s the way of the electronic world, so you’ll need to roll up your sleeves and start digging through a staggering amount of information about sales opportunities, customers, financial transactions, and everything else in these different databases.
To increase the chances of success, while ensuring that the project doesn’t drag on forever, companies hire “systems integrators” (SIs). Most corporations don’t have these kinds of experts permanently on staff, waiting to be called. Better to hire people who handle these projects every day, have a deep understanding of the systems in question, and know the do’s and don’ts that can be learned only through hard experience.
Not all problems are technical
Now that you have the software and specialists in place, these projects may still fail, not for any technical reason. The corporation itself may sabotage the project. Without clear project goals—what data must be preserved, what might be discarded to simplify matters, how quickly does the project need to be done, etc.—expensive systems integration projects might grind on for months or years without completion.
The people within the corporation who really understand the data models might be reluctant to add to the other work they need to perform their participation in the messy, difficult, and lengthy quality assurance process. For example, only the sales managers really know what some of the information about business opportunities really means, so the project can’t succeed without their help. Unfortunately, the sales managers are usually backing towards the door, unwilling to get so tied up in this project that they can’t do their day jobs, making money for the company.
When these problems arise, you’ll inevitably hear a favorite corporate buzzword, “ownership.” Who owns this project? Who owns the data model? Who owns the reports that need to be generated, once the information is finally available?
“Ownership” has two facets, assigning and assuming these responsibilities. A high-level decision-maker has to appoint someone to be responsible. If that person’s duties and authority aren’t clear, there’s no real “ownership” possible, and the project will fail. If you pick the wrong person to assume responsibility, the project will fail. If the importance and urgency of the project isn’t clear to everyone, and not phrased in ways that will motivate them, the project will fail.
When companies hire specialists to handle these projects, they’re often looking for people who can advise them how to handle these all-too-human failings. However, even the best advice will go unheeded if there isn’t a person with real clout—usually someone high in the organization chart—keeping a careful eye on the project.
“Hard” does not mean “impossible”
These projects might sound Herculean, but they’re hardly impossible. The work may take months or even years, but it does get done in hundreds of corporations every year. Feeling the competition snapping at their heels, companies find a way to make these projects successful.
Therefore, you might expect the US government, facing the terrorist threat, would also find a way to make the Terrorist Screening Database (TSDB) and its by-product, the terrorist watch list, successful. A TSDB with 755,000 entries is, by definition, a failure, since its purpose is to identify and track the members of small but dangerous terrorist cells. So what happened?
Next: Some possible reasons for the TSDB’s failure.
Another excellent post, KD.
Different definitions of "important"
It's amazing how relative "easy" is in platform migration. Companies merging the same, customized CRM systems spend mere hundreds of thousands or millions of dollars, instead of tens of millions to merge completely different platforms or implement them from scratch.
Hired guns to the rescue
Part of the problem is the supposed ease-of-use of the various CRM systems themselves. SAP & PeopleSoft, in particular, were designed so that non-programmers can "customize" them. Thus in the mid-to-late-'90s, many people without MIS or similar degrees/experience followed the siren-song into the field. As a result the technical abilities of some of them is often stunningly low. I know CRM developers who can't import a comma-delimited ACSII file. Although there are certainly exceptions, this, sadly, seems to be the rule.
All of the CRM systems that I've seen have user-interface issues, which are usually very difficult to improve. Of course, user-interface issues are the some of the most important considerations in designing any system, which can plague even good programmers, but having non-technical developers working on a system seems to aggravate the problem.
Not all problems are technical
I would add turf-protection to another typical, potential pitfall. Someone (DBAs & programmers, their managers & directors) always loses, but still has the ability to sabotage the process.
Posted by: Mike | 10/29/2007 at 09:58
"The work may take months or even years, but it does get done in hundreds of corporations every year."
Some still fail. A company I worked for abandoned a CRM implementation after 2 years & $25M.
Of course, success in this particular project isn't, and shouldn't be just measured in the usual, on-time, and on-budget way. Most companies don't have the resources that our government should be able to apply, or expect the apparent forgiveness that the American citizen has that the shareholder does not. Failure is less excusable.
Posted by: Mike | 10/29/2007 at 10:24