Enquire Now
DQ Blog CRM Data Cleansing What is Data Deduplication?

What is Data Deduplication?

Martin Doyle July 27th, 2011 CRM Data Cleansing, Data Quality

What is Data Deduplication and what are the benefits?

Data Deduplication is the removal of unwanted records from a file or database; cross matching is the process of comparing one database against another to identify relationships which lead to accurate data integration/migration and construction of a single customer view.

How does it work?

To identify similar records, data is reduced to a common form (standardised), then specialist phonetic (Fonetix™) algorithms are applied, which link matching record groups together and identify their % similarity index.

Once the relationships between potentially matching records have been identified and ratified, the process of improving information quality begins:

  • Duplicate record identification and removal
  • Orphaned data recovery
  • Perfect record creation
  • Assigning group numbers
  • Cross linking
  • Flagging

What are the benefits?

The benefits are clear – avoidance of mailing waste, accurate analysis, better decision-making and improved brand image. Creating duplicate free data provides you with a Single Customer View, allows you to automate repetitive data management tasks, deliver Master Data Management projects with ease, allowing for better informed decisions and ultimately business data you can trust.

The downside is that data quality is tricky to measure, monitor and correct. Whilst prevention is better than cure, businesses simply don’t take the action required to attack the root causes or consider the downstream impacts of poor data quality.

Our Data Deduplication Software is easy to use and fast. Developed over the past 20 years and used by many companies globally. For a Free Trial of our Match Deduplication Software or our DedupleExpress software and see for yourself.


Written by Martin Doyle

Martin is CEO and founder of DQ Global, a Data Quality Software company based in the UK. With an engineering background, Martin previously ran a CRM Software business. He has gained a wealth of knowledge and experience over the years and has established himself as a Data Quality Improvement Evangelist and an industry expert.