I think this most often happens when a corporation or agency creates a new sub-corporation or sub-agency that requires much of the same data as the parent, but slightly different in format or content. Instead of sharing the established data as an enterprise asset, the 'new' group replicates the established data, makes changes / enhancements then struggles to stay in sync with the parent schema. Tah-Da - information silo created!

Thank you for your well written comments. To tell you the truth, 7 years ago, I would have said something similar.

So your point is that there is nothing that can be done to resolve data isolation, we should just live with it. Perhaps that was fine under the prior art paradigm. What I am saying is that under this prior art paradigm, we isolated all this data that never should have been isolated.

Organizations are collectively paying billions of dollars annually attempting to live with that problem.

Under the prior art, we didn’t know how to design and enforce referential integrity between databases. We didn’t know that our universe of data had a finite boundary composed of very specific master data. We didn’t understand that entire databases could be transformed with the addition of four boundary data entities. We didn’t know that there was a permanent solution for data integration that once a database was transformed that it would become a part of a network of integrated databases. We just could not conceive of integrating a database in a week or less. We just took the wrong path!

But, the natural state of data is to be integrated. We unintentionally added isolation into our data architectures by ignoring even the most fundamental interactions between our data models. You will be amazed at the difference of working with truly integrated data.
The future is very bright for those that think outside the box!

Thank you again for your articulate comments.

A large US based computer game concern has acquired companies in Europe and Asia. These companies all have their own independently developed databases with data models and data in a number of languages including English, French, German, Mandarin and Cantonese. As many of the non-native English speaking companies cater for the English speaking market they typically have a combination of languages in the one data base. Many of the companies sell on-line and whilst many customers have bought from more than one of those companies, each company has identified them independently and much of the data is out of date as the customer data often only gets updated as the customer buys another game. Some of the companies make no attempt to match a customer from one purchase to the next and simply keep a new copy of customer data for each purchase. Some of the companies list game titles in one table with a many to many relationship to the gaming platform. Others consider the game title to be the concatenation of the name and platform so there is no hard link between "GameX Xbox" and "GameX Playstation". Add to this product variants such as "GameX Deluxe Edition" and some games may even be completely renamed for different platforms. (Ever used one of those movie services where they recommend to you the DVD version even though you have just watched the Bluray version?)

The English only speaking US execs wish to get regular updates on game sales over all companies by platform by demographic in order to make timely decisions on pricing, promotion, etc and when to cut products. They are also interested in trends on platform usage. For example, if many customers have more than 1 major platform then it could be more profitable to only release games for one platform as the customer can use it either way.

If someone has an integration solution to above problem that only takes days, I would certainly like to hear about it.

Ian: Your hypothetical reminders me of some of my past projects!

Data Integration by design is an entirely different paradigm for data integration. As such it is difficult to respond in a few short paragraphs, but I’ll give it a try.

The data architecture for this new paradigm was designed from the top down. The data universe, that is, the totality of metadata and of data was our model. It was discovered that integrated data was the natural state of data and that it was our data modeling methods that imparted islands of disparate data into our data architectures. Perhaps more important was the discovery that the data universe was finite and therefore has an actual boundary of metadata and of master data. All data models should include these boundary data entities to anchor the data model relative to all other so designed data models. A new form of entity relationship (commonality relationship) links these boundary data entities between data models. The result is the first ever integrated network of data models. Upon instantiation, the result is an integrated network of databases as long as these boundary database tables are properly populated. The population of master data is very important to this paradigm.

Your “hypothetical” is a great example of what can be done with this new paradigm. The data integration by design paradigm focuses upon enhancing the native master data of each database. Each database to be integrated represents a consistent and cohesive data set with the exception that the master data is somewhat deficient. Essentially, with the data integration by design paradigm, each database, in its entirety, is recast by a new standardized set of master data while retaining its native set of master data as well. This recast enhances the master data of the existing database and provides multiple access paths deep into the database.

It is important to understand that the governance of boundary master data needs to be provided from outside of the individual organizations in order to transcend the organizational boundaries. The boundary master data providers have much of the information that the native master data sets do not and as such will resolve some of your master data issues.

The data integration by design paradigm leaves the existing operational database intact. Because of this data integration in place, most of the issues you enumerate are of no importance to me from a data integration point of view. The boundary data entities are applied to the data models. The data access paths are formed between databases.

The network of integrated databases may be considered as a distributed ODS. Now, within each database, may be formed one or more fact tables to support reporting. The fact tables are also integrated across databases as well as with the operational transactions of the parent database. The fact tables from multiple databases may be consolidated into a single database if desired.

Since there are no transformations anywhere within the data architecture, all the data sets are integrated at a high-level. More data integration will follow for each database where the data governance is within the scope of the organization.

This is my attempt at a quick explanation of a paradigm that is extremely foreign to most data architects and data modelers. This paradigm is based upon numerous new discoveries and certainly a complete understanding on your part is not possible here.

May I suggest that you visit my web site and review the information there as well: www.strins.com.

Thanks

Can't agree more, Doug.
What causes data isolation? bad planning, lack of basic data principles, disiplines, procedures, standardards, oversight/governance.

Hi Rob

Islands of Disparate Data - occurs not just because of poor DB Design. I've seen it occur quite a bit when modelers are unable to articulate the business language via the data model. Also, it is not uncommon for modelers to not even make attempts to a) Avoid Semantic Mismatches (e.g., let's just drop items into a structure, regardless of its business meaning or intent....what's the harm?) or b) Not pay attention to terms that are more Application Oriented, than on the same level of grain of the overall Business Model, etc. All of these things contribute to a poor Business/Logical Model, which ultimately leads to a poor DB design.

Also, sad to say, there is the element of politics. It is not uncommon for modelers to instantiate terms that are not consistent with the Business Language first, then attempt to force-fit the data requirements into those terms, regardless of whether those terms are truly relevant and representative of the data.

Last, but not least, Corporate America does not always have the ability to recognize that IT teams are not truly collaborating and building synergy across teams and within groups; or embracing a 'better solution', when the decision-makers decide to use one that is just 'good enough'.....so where's the incentive?

In my experience lack of data ownership is the lead cause. In some cases abandonment of data ownership. Service Orientated Architecture can solve some of this issues if managed from a data level. This can only work if the quality of the data is part of the data owner’s job description. Data ownership is not a sideline job.
On table level you can combine data in single tables but columns cannot easily be shared between owners.
I witnessed a group of persons responsible for managing client limits leaving the bank to be replaced by new staff that simply started changing limits without being trained on the system. The chaos that followed was then blamed on the system.

In larger corporations, I think that the teams building solutions simply don't know that data is already governed by the enterprise. Without a comprehensive metadata strategy, how would they know what data standards exist or where identical data is stored?

There are a few problems. But the biggest problem is the business itself and the architecture that it wants to enforce. If a business would have general architecture principles (as Togaf proposes) one could imagine that there would be a principle like "Redundancy of data must be avoided".

A principle like this could cause serious extra work in a project, so it depends again on the business how strict it would be enforced.

But there's something else that needs to be taken into consideration: applications come and go, but databases grow like a tree. So, it would be a good idea to design your data in a more abstract level. And with each design step it should be taken into consideration how easy it is to make future adjustments.

What I am trying to say is that most companies get the IT system they wanted in the past. How sad.

What causes islands of disparate data? Lack of Data Stewardship.

Data management policies focus on warehousing, modeling, governance, reporting, etc. How many organizations focus on data stewardship? There are generally no attempts to gather and hold knowledge on the how-what-where-who of data in an organization. It would be nice to have a data steward who can guide the answers to the key questions:
What data is needed?
Where is the data?
Who owns it?
How do we get it?

Are there any organizations that have created a group of data stewards whose primary role is to "know" where all the pools of data are and how to harness them?

I have visited your blog, and the solutions- they are all good- natural state of data is to be integrated- proper use of data models- making them universal is the answer..