Why we use Evidence Based, not Data Driven

Rod Jacka - Wednesday, July, 05, 2017

The adjective ‘data driven’ has become very popular in marketing circles in past years. A quick search in Wikipedia returns its definition as:

The adjective data-driven means that progress in an activity is compelled by data, rather than by intuition or personal experience. It is often labeled as business jargon for what scientists call evidence-based decision making.

Whilst this statement takes a less than flattering poke at those who use the phrase data driven, in my opinion the differences run far deeper.

At Panalysis we prefer to use the phrase “evidence based” for a number of reasons. Here are a few of them.

The Data Made Me Do It

Data driven implies that the data is in control.

Whilst surprising insights and efficiencies can and do emerge when data analysis techniques are used, ultimately humans need to make decisions based on the analysis of the data. Data is a means to an end and not the end itself.

Data driven also implies that we inhabit a deterministic universe. Laplace’s Demon has many well known arguments against a deterministic universe. The world and how human beings interact with it is simply way too complicated and far too erratic for predictions to be made purely on observed past data.

All Models are Wrong but Some are Useful

To use data you must create models and all models have some bias. George E. P. Box was a great statistician to whom the quote “All models are wrong but some are useful” is attributed. Whilst there is argument within the statistics community as to the level of how much this statement applies as a general approach, it accurately summarises the key issue.

If the model is useful it will help to increase the accuracy of prediction and reduce the error and waste from poorly targeted campaigns. Models are models and nothing more. As much as the world of machine learning is taking great leaps forward and the idea of artificial intelligence is making a huge resurgence in marketing right now, they are still just models. Regardless of the complexity of the model, inputs predict outcomes. Yes they do offer high levels of efficiency compared to alternatives, but we need to weigh the benefits and the costs against the outcomes.

A strong analogy can be seen in the concept that the map is not the territory Whilst the map is potentially a good representation of the territory and can be used to predict outcomes it will contain errors that reduce its overall accuracy. For the map to be completely accurate it would need to be at a 1:1 scale which Lewis Caroll humorously portrayed in Sylvie and Bruno Concluded, where the map was at a scale of a mile to a mile and due to how impractical the map was at this scale as it stopped the sunlight, the characters realised that the country itself made for the best map.

You Need the Data!

Unless you have the data you can’t include it in your model. It is quite simply not possible to gather all of the data that you will need to create the perfect model. Even if you could get all of the data as mentioned above, we don’t live in a deterministic universe. A quick look at Chaos theory will give you a small glimpse into the key issues; one small error in the data collected or the missing data makes a huge difference in the outcome.

The Past Doesn’t Predict the Future

Regardless of how much and how well you have collected data it is about the past. You simply can’t collect data in the future. We can certainly use past data to understand what has worked and what hasn’t but the world is too complex for this to be a reliable indicator of the future.

Human Know How and Experience

No amount of data could have predicted the iPod. Whilst the cult of Steve Jobs runs deep, the creation of this device led to a number of innovations such as the iPhone and other smart phones. If we are to take a data driven approach to this we would have needed to make a prediction that the combination of small hard drives, the availability of miniaturisation of other electronic components and the increasing popularity of the digitisation of music would lead to to the iPod. The creation of the iPod took creativity, careful assessment of the potential for commercial opportunity and a leap of faith to make this happen.

The Case for the Evidence Based Approach

Ultimately the future can’t be known in advance. We can make predictions about what will happen and some of these will hold to be true but the vast majority will not be. For an excellent summary of this topic I strongly recommend Philip Tetlock’s book Superforecasting.

We use data to guide (rather than drive) our decisions and we incorporate the opinions of the customers we work with into our practice.

An evidence based approach:

  • Gathers data (in its broadest sense) and weighs this according to its credibility
  • Analyses and interprets this data
  • Creates hypotheses that can be tested
  • Designs experiments that are used to test these hypotheses
  • Accounts for our many cognitive biases
  • Runs these experiments and assesses the results
  • Documents what is learnt and then plans the next steps.

Our emphasis is on creating testable hypotheses from which our clients can learn what does and doesn’t work for their businesses.

Human Factors

For people to make decisions they need to comprehend the factors that will influence that decision. Complex models are less likely to be trusted and acted upon than more easily understood models. In our experience, using an iterative evidence based approach leads to greater long term success.

In closing, data can be considered evidence in decision making but it isn’t the only thing that should be considered.

1 thought on “Why we use Evidence Based, not Data Driven”


Your email address will not be published. Required fields are marked *