What is data harmonisation?
Having access to clean, high-quality data allows you to analyse sales, marketing efforts, and other factors that contribute to your company’s success. The data harmonisation process:
- Delivers data in the way that you analyse it internally (internal language), as well as the way outside vendors and partners need it (external language)
- Creates hierarchies that allow for big-picture views, and ensures that hierarchies are consistent across data sources
- Provides enough granularity to make decisions, but not so much detail that it’s difficult to sort through the data
Step 1: Aligning data with master data management
Data often comes from multiple sources, each with its own nomenclature and structure. The first step in harmonisation is to standardise the data and create hierarchies using a master list, which is a single source of truth about the data.
Aligning the data creates standardised language and hierarchies for:
- Product names: “Product X” vs. “Prod. X”
- Product categories, brands, and sub-brands (typically based on the client’s list of SKUs)
- Time periods: for example, measuring quarters in months vs. weeks
- Geographies: cities, regions, states, countries, etc.
- Currencies
- Channels: for example, classifying a store as pharmacy vs. retail
- Customer 360 data: internal data about customers including transaction, satisfaction, and other data sets
- Macroeconomic data: standardising all macroeconomic information
- Metrics and KPIs: harmonising across business units
- Advertising data: for example, campaigns for corporate vs. individual brands
Data harmonisation utilises master data to align data within sources (e.g. standardising product names within a sales database) as well as across sources (e.g. reconciling social media data that may report weekly data as Sunday-Saturday, vs. retail channel data that may report weekly sales as Monday-Sunday).
Step 2: Build an Aggregate Information Model
After the data is harmonised, the vendor creates an Aggregate Information Model that provides hierarchies for each type of data. These hierarchies allow a big-picture view (e.g. product categories) as well as details (e.g. specific products within those category), all in a consistent format. This model also lets you see correlations among data, such as how brand awareness from survey data correlates to sales.
Metrics can be aggregated for different attributes, depending on what factors are most critical for you. You can also derive metrics for future analysis, and pre-calculate metrics to be used earlier in the process, allowing you to quickly detect anomalies and trends before the data even gets to the analysts.
The risks of not harmonising the data
Without harmonisation, you won’t have an accurate picture of sales, trends, and other metrics. It will be difficult to roll up the data for a macro view, or break it down to get detailed insights. Perhaps most importantly, you may miss opportunities (and threats) because your data is spread out, in disparate formats, and unorganised by hierarchies. As a result, you may make incorrect (and costly) decisions, report inaccurate quarterly results, lose market share, and even put people’s jobs at risk.
A one-time investment
Once master data is created, it can be leveraged multiple times across numerous departments to harmonise each department’s data in a continuous manner. With incremental updates of data, the master data also improves. Having a single source of truth means individual units don’t have to develop their own systems, which are often expensive, error-prone, and inconsistent from one unit to the next. Instead, teams from marketing, customer service, commercialisation and other units can profit and capitalise on one harmonised data set.
Best practices in harmonisation
Harmonisation is typically a mix of automated steps (often using artificial intelligence) and manual efforts, with leading vendors automating 60 percent or more of the process. The goal is to use artificial intelligence as much as possible in order to reduce errors and shorten the time to insight.
In addition, good partners will:
- Create data models that meet future plans as well as immediate needs
- Offer deep industry and category expertise, which saves you time
- Provide a no-code environment that lets data analysts harmonise the data directly
Kantar: The Marketing Data Experts
At Kantar, we help leading organisations make smarter, faster business decisions. We have significant industry and category expertise in retail, CPG and many other areas, which gives us a distinct advantage in generating and harmonising marketing data (including our data as well as data from others). Kantar’s specialists have a thorough understanding of the complexity and nuances of marketing data sets, and routinely handle every step of the data insights process, including data engineering with Olympus, our platform with built-in artificial intelligence and machine learning.