EA872 Week2: Why we need Foundation: A case of "Data Integration Gone Wrong"

This week I'm going to take a small break from the material and present a real case where not having a good foundation results in complex efforts to integrate data at the wrong place at the wrong time.

A company has multiple transaction systems tracking down contracts, campaign, and contacts for running down "ads" at different locations. Typically, this would be a straight case of integrating the data, finding the lowest grain, and rolling/aggregating as needed across the different hierarchies needed. The problem, in this case, is that each of the contracts, campaign, contacts, and locations appear to be at different grain levels with no clear joins or direct way to aggregate data meaningfully. After much exploration and tracking down how data relates from these different objects, it turns out that it cannot be joined in a conventional way because there are not enough integration points. Here is an example: one can track down the count of locations an ad was played, but the connection to contacts is only based on a pool of locations associated with a campaign assigned to multiple contacts. So you have the same count of locations to all contacts which does not make sense!

Simply put, the problem here is that as the business grew, there has been a disconnect of what modifications need to be done to integrate the different data silos coming from different business units so an effort to revising the foundation and consolidating these silos was not undertaken by the responsible teams. I can only deduce that there is no strong EA team or the EA architecture foundation has not been in place as the business grow and expanded. Hence, the typical scenario where one ends up with data silos that need to be integrated at the end of the data processing stages (user end) meaning the need of additional workarounds and quite often hybrid additional tools that need to be purchased and multiple implementation efforts (inefficient resource allocation). This is way EA is always important and needs to be the driving force behind coupling business goals and objectives to IT efforts and implementations especially in business integration scenarios (whether from acquisitions or from growing business units and functionalities).

One interesting approach I read about this week is in chapter 1 of Enterprise Integration Patterns:
http://www.enterpriseintegrationpatterns.com/patterns/messaging/Chapter1.html
Under Loose Coupling, the chapter discusses loose-coupling and tight-coupling with advantages and disadvantage of each approach. This is something the can be considered at an EA planning level or part of EA architecture foundation.