I’ve been working with data for over 20 years now and the way we are using data became even more and more intense. We evolved from standard spreadsheet data, to IOT, digital body language and even bio metrical data. The amounts of data we generate on a daily basis is incredible, giving us even more data to work with. We rely more and more on data algorithms. Data mining and data modelling became the norm, influencing our daily lives and jobs. There is just one problem ... are we using the right data or even the data right?
In the perfect storm of storage, processing power and data science, AI became the new holy grail and companies worldwide are starting to implement it in their operations. That’s where reality kicks in. AI is not fixing your data issues; it is exposing them. AI feeds on data to produce new insights and when the ingested data is wrong, biased, or even non-existing even AI will not solve anything. Bringing us back to the essence of every data related exercise “shit in, shit out”. How do we fix this? Plain and simple. Data hygiene and transparency.
Value doesn’t come out of data. The way we interpret it does. Therefore, always start with the question “What problem are we trying to solve?”. Don’t look at big data, look for smart data. Ones you have identified the possible data you are looking for start to map where to find it. This is the part where you will need to break company data silos, where you will need to start normalizing data and you will realize that you missed so many opportunities in the past because you were not able to look behind your own desk or department.
You will have already done a tremendous job by aligning data sources coming out of different systems creating a new master data file. Now comes the time to analyse this pool and learn from it. This is the part where you will realize that bad data originated from bad habits and low hygiene. You will learn that this incredible CRM system is not reflecting field reality for 2 reasons.
It was built by technicians and you forgot to first look at the real field challenges and by outcome you are missing valuable data on the one hand and have too many useless data on the other. Second obvious insight; the system only works if you enter data in it and maintain it over time. Human interaction is the biggest problem in data mining. Changing the way we maintain data hygiene is crucial for the success of any data related exercise. If you can not motivate your staff to log the info in the systems, you will never be able to trust the data coming out of it!
Ok, you did it. You have your master file, you have trained your people to maintain clean data influx, you even identified new missing data objects and are working on it. Great! Now start implementing a POC (Proof Of Execution). This will help you to learn, fine-tune and build confidence that this data project will start delivering benefits to ALL. Don’t rush, it will take time, especially with AI the algorithms need to learn over time. Better data generates better insights, new insights will generate new data, .... just make sure that data hygiene is maintained and improve the model while testing it.
Finally, never forget data is there to help. Not to replace a human mind or emotion! After all, never forget even the strongest insights come from what we have been feeding it and at the end of the game some human common sense or even ethical decisions will be needed.
