Treat Different Data Types Separately

To build a reliable CDI solution, it is imperative to treat different types of data separately, such as master reference data, relationship data or transaction data. Master reference data is the foundational entity data (such as name and address) that is critical for uniquely identifying a customer across multiple systems and channels. Without a persistent and trustworthy hub of customer reference or profile data that serves as the “system of record”, other types of data can not be aggregated reliably. Ideally, such a master store should create and maintain the best-of-breed record for each customer culled from all relevant internal and external data sources –at the cell or attribute level – along with the associated cross-reference keys. This store then becomes the best source of truth for customer profile information for all downstream operational and analytical applications.

The next type of data is relationship or hierarchy data. This type of data defines the relationships among various entities (such as individual to organization, organization to organization, or individuals within households). Relationship data can be managed reliably across different sources only after the underlying conflicts of master (entity) data have been resolved. Most of the custom solutions deployed have fixed relationships among entities embedded in the system’s data model, which makes it hard for IT to manage changes in customer relationships and affiliations.

The third type of data is transaction or activity data (such as amount withdrawn from an account). Although there are significant challenges in managing large volumes of transaction data, there is usually little conflict in reconciling such data since there is an unambiguous system of record for each type of transaction. The key issue lies in attaching these transactions correctly to the same customer across multiple CRM touch-points and then aggregating them accurately for other applications to consume (such as the average account balance). Note that transactions can be aggregated for the right customer or household only after the ambiguities of the associated master and relationship data have been removed.

Essentially, without treating different data types separately and establishing a reliable foundation of master data at the start, a trustworthy CDI platform can not be built. Yet, none of the data tools maintain separation of data types. For instance, ETL tools neither recognize nor treat master data apart from other types of data. EII tools assume that all federated data results are clean and unambiguous; in fact, they rely on an external source to provide correct cross-reference keys and global IDs to accurately join the results of a federated query. DQ tools provide ad-hoc cleansing of a source but do not recognize data types nor offer on-going management of data changes.