In today’s world, data is generated and gathered from a growing number of streams at a pace we have never seen before. With so much data now available, how do we ensure it is used to generate insights, and not insults?
Fit for use data
The way we use data and the insights it provides has changed dramatically over the years. Throughout the COVID-19 pandemic, the government has continuously told us that every decision it makes is driven by “the data”, such as infection rates, hospital admissions and deaths. These insights have been used to make critical decisions, such as when to enter and leave lockdown, which have impacted people’s lives, jobs, and education.
Car manufacturers are using live diagnostic data to predict faults before they happen, gift ideas are being suggested to consumers based on their online habits and news stories are being collated personally for us based on our interests.
However, in order to ensure that these insights are correct, they need to be informed by “fit for use data.”
Why do we want insights?
Businesses are being increasingly led by insights for three main reasons;
- They help drive revenue,
- They promote less waste,
- They mitigate risk.
This is all true, providing the data being used is reliable. While they may go hand in hand, insights and data are two different things. Data provides us with pieces of information without any context, it is only through the holistic analysis of that data that we are then provided with the insight, which can then be used to inform decisions. Take the infamous “beer and nappies” story. A convenience store spotted there was an odd correlation between nappies being bought alongside beer. Upon analysis, it was found that fathers were stopping by on their way home from work to purchase some nappies and also bought beer. The store then displayed beer and nappies next to each other by the checkouts and saw the sales increase exponentially.
The problem with insults…
The last thing any business will want to do is insult their customers or clients. The impact of doing so is obvious, it can cause a breach of trust, damage relationships with clients or damage a businesses’ brand or reputation in the eyes of the customer. As the amount of data being collected increases, so does the risk of insulting someone and impacting your businesses’ reputation.
How do we avoid insults?
The best way to ensure the quality of your data, and avoid insults, is to check it at every stage of the journey.
- Data capture – At this stage, prevention is better than cure. With low code and no code collection prevalent and the growth of citizen developers, businesses must ensure that they verify the quality of their data before it enters the Dataverse. Verifying and fixing it at the point of capture means it will be correct at the very first step. This requires things like PCF Controls, reference data such as address confirmation, and process control. Doing this at the beginning of the data journey will also minimise cost as attempting to rectify any issues later down the line can be expensive.
- Data in flight – This is when data is streamed or migrated into another system in structured or unstructured forms. This part of the journey requires business rules, validation, transformation, and formatting in order to ensure that when the data is uploaded into the other system, it’s fit for use.
- Data at rest – This is when the data reaches its destination and is now being stored, whether that’s in a data lake, Cosmos DB, or blob storage. This is the “cure” stage of the journey, which could be more expensive than the “prevention” stage. This stage requires processes like deduplication, verification, suppression, enhancing and more.
- Analysis – The final stage is the analysis of the data. This is where the insights are derived and cognitive services, data bricks, and stream analytics are formed. This requires data that is fit for use. Without the correct data, this is where wrong decisions are made with a high degree of certainty, and you can end up insulting your customers, rather than providing insight.
At DQ Global™, we have been ensuring that customer data is fit for purpose for 25 years. We have a range of industry leading products that help businesses improve revenue, reduce costs, and mitigate risk. You can find out more about our data quality solutions here.
You can also watch our CEO Martin Doyle’s full talk “Customer Insights or Insults” talk at this year’s Scottish Summit here.
Check out our free “DQ Guide to Data Quality in 2021” here, which looks at data quality as a business problem and the steps you can take to fix it.
