Are you driving your organisation with dirty data?

Ross McIntosh, Director, International Sales at VisionWare discusses why organisations need to be using clean data if they want to keep the CRM engine running smoothly

When you buy a new car, you are looking for optimal performance. But what if your car dealership or mechanic suggested that you take the oil from your old car and put it in the new one before you hit the road? Clearly, you wouldn’t think this is such a good idea—old oil will just serve to impact the performance and reliability of your new pride and joy and diminish the experience and bonding with your new car.

So why would it be any different in the case of your investments in a new CRM system or your new data analytics platform?

Taking your old, ‘dirty data’ that has been accumulating over the years in your systems – with all its gaps and duplicate information – and dropping it into a new CRM system or data platform will certainly lead to sub-par performance and negatively impact your ability to achieve the outcomes expected.

Fortunately, you can get a filter for your old data, thereby getting value without loss of performance—the key is making the right decisions in advance of the new investment. There are steps you can take to clean up your data to make it “like new” and not be disappointed with the results and poor adoption.


Data Quality

It’s crucial to gain insight into the quality—and limitations—of the data that is available to you. Start by looking at the processes that are generating the data in the first place. It’s likely that you’re gathering data from a range of different systems and departments within the organisation on the front end, and those systems will be capturing data that is fit for that purpose, but likely have certain limitations, properties, and processes that will have an impact on the completeness or consistency of the customer data itself.

Assessing this quality in each of your separate systems will inform what kinds of data cleansing will be needed downstream, while also helping you identify any shortcomings—let’s call them opportunities for optimisation—in your existing data management processes.

Four main factors contribute to data quality:

  • Completeness of data: Often, different departments within your organisation will have fragmented siloes of information. Health services, the justice department, and the housing department, for example, will capture discrete data points about your citizens, and may leave other information out of their records. Only by matching and merging the data from these disparate sources can you achieve completeness of data. Another important consideration in your data being fit for purpose is coverage—how much of the population does your dataset actually cover?
  • Consistency of data: Where completeness of data means having all the pieces of information on a given citizen within each department, consistency of data means having the same information, in the same format, using the same codes, across all departments. This means that if one department has a citizen’s address and contact preference information, that data should be shared across the entire organisation, enabling you to fulfil your customers’ expectations of ‘telling you once.’ Similarly, for consent management and meeting General Data Protection Regulation (GDPR) obligations, which will take effect May 25 of this year, you need to provide a consistent way to interact with the citizen—one that is based on their consent preferences.
  • Accuracy of data: Accuracy, of course, means that the data in your CRM system or data platform should be correct. This means spelling Kelley vs. Kelly correctly, and knowing whether a citizen lives in apartment 2 or apartment B. One note of caution: It’s easy to get caught up in trying to have 100 percent accurate data, but some information will need updating frequently to remain accurate, and some won’t. We’re striving for fit for purpose, rather than perfect, here.
  • Currency of data: Experience tells us that data tends to degrade over time—in other words, data quality will typically decline the longer it is left un-managed. In some cases, such as with social care or health information, the data for each citizen needs to be updated in or close to real-time in order to best serve the citizen. With other data, such as address information, less frequent updating may be acceptable.

Together, these elements result in how much you can trust of your data, which is key to its value and the insight created that is the basis for your decisions.



Part of what contributes to dirty data is the duplicate information that accumulates over time. Within each department in your organisation, it’s expected that some duplication will occur. This issue is compounded when you look at data across your organisation where a record can span across multiple departments.

To clean up duplicate information, your organisation must explore how the data is going to interact between systems; how it will blend to create the single, “golden record” for each citizen that will generate a more complete and accurate picture. This blending process also gives your organisation the opportunity to discover relationships hidden inside your existing data sources. Can you bring together all the disparate information you have on a citizen called Kelley Smith and match it to create the golden record? Can you differentiate the Kelley Smith that lives on Main Street and the one that lives on Park Avenue?

The right analytics platform can help improve the customer experience, offering a deep understanding of how the golden record needs to be structured to maximise value.


The 360-Degree View

There are some credit-check companies that offer services to “clean up” your data. For a fee, organisations can send them large files of data, which the credit-check companies will compare and verify against their own data, telling you whether your information is correct.

The problem with offerings like these is that they are one-off engagements, and, while they can help improve data quality at a specific point in time, the continued use of these services is both expensive and inefficient to replicate across all of the sources of data. Additionally, what is the impact if the data is updated the day after their verification? Or if the data that you currently hold is more current than their data? With this approach, you would be immediately out of date—which is just what you were trying to avoid!

The key is that they don’t offer all the broader capabilities of managing your own data: de-duplication, matching, continuous real-time updates, or a way to push the completed, “golden view” record back out in a controlled manner to the agreed systems in your enterprise. Using a platform to manage your data, as opposed to simply addressing the symptoms of data quality, provides a more efficient and sustainable solution for leveraging the data assets throughout your organisation.

A truly robust data management platform will address all the facets discussed here, cleaning up the dirty data before it’s put into a new CRM, to ensure optimal performance. The result is a 360-degree, “golden view” of each citizen’s information in the system, aggregated from your own internal data, which can be easily and continuously monitored and updated over the years.

Related reading