A report on a recent business survey found that, on average, out of every £6 spent out of their budget, departments wasted £1 because of poor data quality. Business people are aware that having clean data that gives accurate information helps them swim away from their competitors, but many still struggle to keep their heads above water rather than invest in data quality solutions.

Would you put toxic fuel in your car and still expect it to take you wherever you want to go? It won't happen, will it? Toxic data is the same; a business running on toxic data simply can't perform.

So why keep on putting toxic data into the analysis engines and expect world beating results? It's not difficult to clean data before you use it. All you need are the right tools – a one off investment that could make all the difference.

Big data is here, and it's getting bigger by the second. It is bringing significant challenges as well as opportunities. The horizons of master data management are changing at record speed.

How loyal are your customers? Do your senior managers know? Can you measure this? If your data gives you a single customer view you can see at a glance how much repeat business you are getting, and what is attracting the most loyal customers.

All businesses need to be able to adapt to market demand. Many astute business leaders seem to have an instinct for what is going to work well and what direction to take next. That's the way it appears to outsiders, but the truth is that their gut feel has most likely evolved from the accurate information from clean data available for them to study.

Got a direct mail campaign coming up? The marketers have prepared an excellent mailing with an eye-catching design, a tempting offer, a powerful call to action and prominent contact details. Everything is lined up to handle the responses and the business is holding its breath for an onslaught of interest.

But even with all this in hand, bad list preparation can virtually kill the campaign, waste the marketing budget and cause lasting damage to the reputation of the business. A bad mailing list that hasn't been checked by data cleansing software can even take you outside the law and result in costly fines.

The Hidden Dangers of Defective Data

The surface dangers of bad data are obvious: it renders mailings a waste of time and money, can cause reputation damage and seriously interfere with customer relationships.

But there are other dangers lurking beneath the surface that could bite harshly into your bottom line and deliver false information that lures you into making business decisions that are financially damaging.

Firstly, I promise you won’t need to be a scientist or engineer to understand this. And yes, it is relevant to how data decays from order (high quality) to chaos (low quality).

There is a ubiquitous phenomenon we all instinctively accept that data has an unerring ability to go from high quality to low quality.

This phenomenon - with energy - is defined by the 2^nd Law of thermodynamics; which loosely states that energy has an absolute and unfailing tendency to go from "more concentrated" to "less concentrated". It kind of "spreads out" and gets "diluted". Some examples are:

• Energy flows from a higher temperature to a lower temperature (heat exchange)
• Energy flows from a higher pressure to a lower pressure (expansion).
• Energy flows from a higher voltage potential to a lower voltage potential (electric current).
• Energy flows from a higher gravitational potential to a lower gravitational potential (falling objects).
• Water flows and falls from higher elevation to a lower elevation (downhill).

Basically, energy always goes from high concentrations to low concentrations and when the transfer stops there is a state of equilibrium, when it is said to be at its maximum entropy.

In science, "Entropy" is defined as a measure of unusable energy. As usable energy decreases and unusable energy increases, "entropy" increases. So, as usable energy is irretrievably lost, disorganization, randomness and chaos increase.

In the context of this article, it sort of validates why our, once orderly databases - if left to their own devices - rapidly decay into a disorderly, untrusted, fragmented and duplicated mess.

Entropy may therefore be thought of as a measure of the usefulness of data or information. Eventually all of the data in our organizations just gets less useful; until finally, it becomes mostly useless. It has reached a point of equilibrium, or its maximum entropy, where it has no further potential to be actively used, for say marketing, or, for informed decision making.

Sounds like what happens to any database when neglected and left to decay naturally to me?

Unlike energy though, unfortunately, as yet, we cannot scientifically measure the degree of data entropy as I don't believe there are any universally accepted units of data chaos or disorder. It does sound like a good legal term though for disciplinary action… "You are guilty of generating 3.5 units of disorder in my CRM and 4.2 units in my ERP system, you are sentenced to x years of data entry”.

So what can we learn from this?

Well if we borrow from science and again stretch the energy metaphors to apply to data and information, it seems pretty obvious that if we wish to reverse data chaos and overcome data decay, we need to apply some effort and actually do some work!

In science, work is defined as (force x distance moved) e.g. the work or effort required to lift a weight, compress a gas, pump water uphill etc., or, in the case of data, we might consider it the work or effort required to change its state from “A RIGHT STATE”, to “THE RIGHT STATE”.

Basically, if we are to change the state of data within business applications into a state which is fit for use, there is hard work to be done! There can be no more excuses or corporate slacking; because, when it comes to: refreshing, standardizing, formatting, validating, suppressing, deduping and enhancing your data. All of which are incidentally verbs, action is the key.

Data does not clean itself

Unless you take action, things simply stop happening, or don't start, when there is equilibrium or maximum entropy. Putting data back into a fit for use state requires work, hard work.

It will be worth the input of physical and emotional energy though as businesses will be rewarded with high value data yielding high value returns. Basically, things happen when high energy high value data is allowed to move from high potential to low potential through its use.

Action is always the key.

In the case of corporate data, it requires effort from everyone:

• Business Leaders need to lead a culture of Corporate Data Responsibility (CDR), where trusted data is the norm and accurate information a corporate imperative.
• Management to implement CDR through a data governance culture where data is skilfully curated to deliver business information and organisational insight.
• I.T. to ensure CDR where any data migrations, data integrations and data processing take place to guarantee they are co-ordinated, repeatable and correct all of the time.
• Data workers to ensure CDR through data which are captured correctly, first time and every time so it is fit for use by all upstream consumers in the data and information demand chain.

All of this combined effort means better business; reduced entropic waste, reduced operational friction, reduced data scrap and re-work. It leads to: actionable information, which in turn drives better decisions, which creates, greater shareholder value, greater sustainability, happier employees and much, much higher profits!

Martin Doyle is a Data Quality Improvement Evangelist and the CEO of DQ Global, which is a UK-based data quality software and services vendor providing data cleansing, international address and email verification, data deduplication, and data matching solutions for Customer Relationship Management, Single Customer View, and Master Data Management. DQ Global has worked with over 500 businesses worldwide on a variety of projects, providing their clients with improved data quality, making their data fit for business use, and enabling them to trust their data and make decisions based on a foundation of fact.

Listen to the full interview here:

http://www.ocdqblog.com/home/the-johari-window-of-data-quality.html

When it comes to data quality improvement, I believe you must take the approach a doctor might take, in that you must diagnose before you prescribe.

Good content is data which is fit for purpose.  It should be:

A recent report from Enterprise Data Management Council from "What's In and What's Out” in data quality.  Take a look at our DQ360 product which will help you with your data quality issues.

As we move into 2012, businesses that capitalise on their data, create single customers views and master their enterprise data will have a distinct advantage over their competition. They will survive and thrive, whilst those who don’t will be at a severe disadvantage and could become extinct.

In an ideal world we would have a universal product naming convention.  In practice this rarely happens and it's possible for different users in the same organisation to give different names to the same product.

The most difficult type of data to control at point of entry is free form text. We recently saw 254,577 variations in a free form entry field for an international address which contravened published internal data entry guidelines.

We recently stumbled across this unintended data quality error as a result of what should have been a simple data processing task.