The Hidden Dangers of Defective Data

The surface dangers of bad data are obvious: it renders mailings a waste of time and money, can cause reputation damage and seriously interfere with customer relationships.

But there are other dangers lurking beneath the surface that could bite harshly into your bottom line and deliver false information that lures you into making business decisions that are financially damaging.

Firstly, I promise you won’t need to be a scientist or engineer to understand this. And yes, it is relevant to how data decays from order (high quality) to chaos (low quality).

There is a ubiquitous phenomenon we all instinctively accept that data has an unerring ability to go from high quality to low quality.

This phenomenon - with energy - is defined by the 2^nd Law of thermodynamics; which loosely states that energy has an absolute and unfailing tendency to go from "more concentrated" to "less concentrated". It kind of "spreads out" and gets "diluted". Some examples are:

• Energy flows from a higher temperature to a lower temperature (heat exchange)
• Energy flows from a higher pressure to a lower pressure (expansion).
• Energy flows from a higher voltage potential to a lower voltage potential (electric current).
• Energy flows from a higher gravitational potential to a lower gravitational potential (falling objects).
• Water flows and falls from higher elevation to a lower elevation (downhill).

Basically, energy always goes from high concentrations to low concentrations and when the transfer stops there is a state of equilibrium, when it is said to be at its maximum entropy.

In science, "Entropy" is defined as a measure of unusable energy. As usable energy decreases and unusable energy increases, "entropy" increases. So, as usable energy is irretrievably lost, disorganization, randomness and chaos increase.

In the context of this article, it sort of validates why our, once orderly databases - if left to their own devices - rapidly decay into a disorderly, untrusted, fragmented and duplicated mess.

Entropy may therefore be thought of as a measure of the usefulness of data or information. Eventually all of the data in our organizations just gets less useful; until finally, it becomes mostly useless. It has reached a point of equilibrium, or its maximum entropy, where it has no further potential to be actively used, for say marketing, or, for informed decision making.

Sounds like what happens to any database when neglected and left to decay naturally to me?

Unlike energy though, unfortunately, as yet, we cannot scientifically measure the degree of data entropy as I don't believe there are any universally accepted units of data chaos or disorder. It does sound like a good legal term though for disciplinary action… "You are guilty of generating 3.5 units of disorder in my CRM and 4.2 units in my ERP system, you are sentenced to x years of data entry”.

So what can we learn from this?

Well if we borrow from science and again stretch the energy metaphors to apply to data and information, it seems pretty obvious that if we wish to reverse data chaos and overcome data decay, we need to apply some effort and actually do some work!

In science, work is defined as (force x distance moved) e.g. the work or effort required to lift a weight, compress a gas, pump water uphill etc., or, in the case of data, we might consider it the work or effort required to change its state from “A RIGHT STATE”, to “THE RIGHT STATE”.

Basically, if we are to change the state of data within business applications into a state which is fit for use, there is hard work to be done! There can be no more excuses or corporate slacking; because, when it comes to: refreshing, standardizing, formatting, validating, suppressing, deduping and enhancing your data. All of which are incidentally verbs, action is the key.

Data does not clean itself

Unless you take action, things simply stop happening, or don't start, when there is equilibrium or maximum entropy. Putting data back into a fit for use state requires work, hard work.

It will be worth the input of physical and emotional energy though as businesses will be rewarded with high value data yielding high value returns. Basically, things happen when high energy high value data is allowed to move from high potential to low potential through its use.

Action is always the key.

In the case of corporate data, it requires effort from everyone:

• Business Leaders need to lead a culture of Corporate Data Responsibility (CDR), where trusted data is the norm and accurate information a corporate imperative.
• Management to implement CDR through a data governance culture where data is skilfully curated to deliver business information and organisational insight.
• I.T. to ensure CDR where any data migrations, data integrations and data processing take place to guarantee they are co-ordinated, repeatable and correct all of the time.
• Data workers to ensure CDR through data which are captured correctly, first time and every time so it is fit for use by all upstream consumers in the data and information demand chain.

All of this combined effort means better business; reduced entropic waste, reduced operational friction, reduced data scrap and re-work. It leads to: actionable information, which in turn drives better decisions, which creates, greater shareholder value, greater sustainability, happier employees and much, much higher profits!

Martin Doyle is a Data Quality Improvement Evangelist and the CEO of DQ Global, which is a UK-based data quality software and services vendor providing data cleansing, international address and email verification, data deduplication, and data matching solutions for Customer Relationship Management, Single Customer View, and Master Data Management. DQ Global has worked with over 500 businesses worldwide on a variety of projects, providing their clients with improved data quality, making their data fit for business use, and enabling them to trust their data and make decisions based on a foundation of fact.

Listen to the full interview here:

http://www.ocdqblog.com/home/the-johari-window-of-data-quality.html

When it comes to data quality improvement, I believe you must take the approach a doctor might take, in that you must diagnose before you prescribe.

Good content is data which is fit for purpose.  It should be:

A recent report from Enterprise Data Management Council from "What's In and What's Out” in data quality.  Take a look at our DQ360 product which will help you with your data quality issues.

As we move into 2012, businesses that capitalise on their data, create single customers views and master their enterprise data will have a distinct advantage over their competition. They will survive and thrive, whilst those who don’t will be at a severe disadvantage and could become extinct.

In an ideal world we would have a universal product naming convention.  In practice this rarely happens and it's possible for different users in the same organisation to give different names to the same product.

The most difficult type of data to control at point of entry is free form text. We recently saw 254,577 variations in a free form entry field for an international address which contravened published internal data entry guidelines.

Primary Data Quality Issues

We think you probably already know the answers; however, here are some of the questions you might want to ask yourself.

• Who is responsible for overall Data Quality in your business?
• Do you have a set of data quality guidelines (a DQ-Plan) for your business?
• Does everyone who touches your data know about and adhere to the Data Quality Guidelines?
• Has everyone been trained in the importance of good data management principles and understand the consequences of dirty data?

As an alternative to positive data management predictions for 2011 a prominent thought leader Jill Dyché has put out her Anti-Predictions for 2011.

Whilst pessimistic, there is a definite ring of truth and I'm confident a number of you reading this will agree in whole or part. We'll be intersted in your thoughts.

The data quality manager has to have the most thankless task in the organisation, but the main problem seems to be that there is confusion about (a) the difference between data and information and (b) ownership differences between the data content (things stored in the electronic filing cabinet) and the data container (the electronic filing cabinet).

The need for accurate business intelligence is becoming increasingly critical as decision-making becomes more complex daily and customers demand ever higher levels of service.

The function of the data warehouse is to consolidate and reconcile information from disparate business units and systems to present a consistent picture of the business at a single point in time.

Reliance on uncertain or obsolete names and addresses damages your credibility and reduces customer loyalty. UK and international address correction guarantees your communications reach their intended targets; and you avoid wasting resources on unwanted or undeliverable mail.