The Need for Common Metadata
Having painted the intelligent business picture, that so many companies seem to want, there is one foundation stone that must be laid first if this vision is to ever become a reality. That foundation stone has to be common business metadata. By that I mean common data names, common data definitions, common data integrity rules such that all data items in dimensions and metrics used in a data warehousing system are absolutely consistent. If there is one pre-requisite to integrating BI into operational business processes it has to be that the BI is trustworthy and has common understanding. Trusted metrics, trusted meaning, trusted unambiguous data names and an audit trail of how data got to be in the data warehouse and/or the data marts that store it. Without this, integrating BI into operations could cause untold damage to business operations especially if automated analysis and actions cause business changes to be made.
Over the next several articles I will discuss in detail the components of intelligent business. However before I do that, let’s look at this requirement of common metadata in more detail.
What is common metadata? This is the very basis of data sharing. It is common definitions of data items, common data names, common integrity rules, common reference data (e.g. code set values…..), common mappings, and common transformations for all subjects (person, product, location….), subject attributes, transactional data, dimensional data and metrics associated with data that are considered ‘official’ or ‘standard’ within an organisation. These ‘official’ and common data items (sometimes called data assets) can be used not only in business intelligence systems, but also in operational systems. The point here is that for BI to be at the centre of enterprise operations, with automated alerts, recommendations and action messages being fired off on detection of exceptions, then that BI had better be correct and understood. So the pre-requisite to intelligent business from a BI perspective is to resolve all inconsistencies in data naming, data definitions and data integrity rules across all BI systems by making these systems adhere to common data definitions.
People have to resolve these discrepancies in logical data models, physical data models, BI tool business views (e.g. Business Objects universes) etc. if there is ever to be success here. To quote Michael Brackett “building a common data architecture may seem expensive but not building one is even more expensive”.[2]
So a common business model is needed. The benefits of a common business model include:
- A better understanding of data
- Increases awareness and reduces uncertainty about data
- Formally named, comprehensively defined, properly structured and high quality data
- Deliver incremental short term benefits while moving towards a long term goal
- Reduces volume of data through removal of redundant data
- Reduces need for transformations if data is common
- Improves productivity as people spend less time struggling to understand data and more time using it to do their jobs
Therefore the very first step involved in intelligent business is to put in place a common business model and resolve BI inconsistencies by repairing existing BI systems to adhere to that common business model. In that sense, all BI data stores and BI tools need to:
- Use common naming, definitions and structuring of common data across all BI systems
Use common data integrity rules on common data items across all BI systems
We also need to:
- Identify (discover) the definitions of disparate data. This means finding the metadata in these source systems and integrating the metadata. Given the disparate data in source systems and indeed the discrepancies across existing BI systems, it would seem obvious to that data integration software (ETL software) could be used to integrate the disparate metadata describing disparate source data on these source systems and bring it into a repository for cross reference to a common business model. This is exactly what Informatica have done for example with their latest SuperGlue metadata integration product (See Figure 2)
Figure 2 Informatica SuperGlue
The above screen shot shows how a metadata product can collect metadata and integrate it such that you have an “inventory” of data items using the Informatica SuperGlue product. Also database vendors like IBM in DB2 (the Masala project) are looking at this issue as are other data integration vendors such as MetaMatrix and BEA with Liquid Data.
- Map the definitions of disparate data (cross referencing data) in source system models to the definitions data items defined in the common business model such that all data is correctly sourced and that we have full metadata lineage between source data and common data used in BI systems. Note that this is metadata to metadata mapping.
Figure 3 shows how metadata is mapped from source metamodel to the common metamodel in BEA Liquid Data.
Figure 3 BEA Liquid Data
It would also seem that doing this in an industry standard way would be desireable. To that end the Object Management Group Model Drivel Architecture (MDA) – reference - would seem to be the most appropriate standard for this task. Metamatrix are one of the few vendors that have grasped this already and support it in their product.
- Assessing data quality in disparate data
- Generate/ define cleanup and translation rules for disparate data to get it into a common state
- Integrate disparate data to create a common trusted set of business intelligence
While all these steps seem somewhat formal they nevertheless make it possible to resolve inconsistencies in BI systems and resolve inconsistencies between source operational systems and BI systems so that we reach a point where BI is trusted and clearly understood. At this point it becomes “safe” to then go to the next step which is to look at integrating BI into business operations.
To conclude, a common business model is the foundation stone to consistency and integrated BI. A model driven data integration strategy is the way to achieve consistent common metadata. This is done by discovering data definitions in source systems and using data integration software to integrate the metadata and create a data asset registry that contains an inventory of disparate data items as well as to create (reverse engineer) source models. From here we map source system metadata (definitions describing source system data) from source system models to the common business model (metadata describing BI system data) so that we know how the lineage works. Having done this, we can then define transforms to get the data from an unofficial state into a high quality commonly understood state.
Once data is consistently defined in BI systems we are ready to integrate BI into the enterprise to empower operational business processes with just in time BI. This is the next step in the roadmap to intelligent business and I will explore how this can be done next month on www.businessintelligence.com.
[1] Reference TDWI Report Series Building the Real-Time Enterprise by Colin White and Wayne Eckerson
[2] Source: “Data Sharing – Using A Common Data Architecture” (ISBN 0-471-30993-1)
About the AuthorMike Ferguson is Managing Director of Intelligent Business Strategies Limited,
a leading information technology analyst and consulting company.
As an analyst and consultant he specializes in enterprise
business intelligence, enterprise business integration, and
enterprise portals. Mike can be contacted at (44) (0) 1625
520700 or e-mail at .
|
