Why being statistically correct never hurt anyone…
Businesses are now, thanks to the ease of and access to web analytics tools, using data insights in more and more of their decision making. We’re evaluating traffic sources to understand where our customers are coming from, we’re viewing automated funnels to identify improvement areas in our online shopping processes and we’re measuring and attributing marketing spend to identify the best media options. Some are also running smart hypothesis tests to validate new ideas.
The question is however, are the reports we are using and making decisions from statistically correct? The reality is that in some cases they’re not – and here’s why.
Statistics involves complex problem solving … for a reason.
Statistics for dummies was written for those who want to simplify the complex in order to understand it. It’s a great starter for those who want to understand the basics of statistics, so I highly recommend it (and others like it). The challenge with data interpretation is similar – we try to simplify it in order to make decisions, but this can introduce problems if not done properly. We need to accept that sometimes things are just complex and they require a considered approach to getting the right answer. Learning and using statistical methods to analyse data is necessary for anyone interpreting data and presenting or using results for decision making.
The ‘averageness’ of the average.
A large number of the metrics that are used in web analytics are simple averages – i.e. average time on site, average pages viewed, average sales. Using an average makes the assumption that the distribution of data resembles a “normal distribution” (remember the bell curve). If your data doesn’t match this then the average can be very misleading. We see this day in and day out. The data in web analytics tools like Google Analytics is often taken from distributions that better resemble exponential, gamma or others where averages make little sense.
Take for example the following scenario:
- 90% of an eCommerce site’s daily sales were be between $119 and $5,271
- The most common daily sale ranged between $450-$550 per day.
- However when averaged, daily sales were reported as $1,608 per day
The result: When comparing your daily sales to the yearly average you are going to be disappointed on far more days than you will be happy.
A typical distribution of sales for an e-commerce website.
Data recorded is not necessarily a good measure for analysis.
There are a few good examples of what we mean by this. What we’re seeing in user behaviour today is that we’re multi-device-talented’’- we’re using different devices and browsers simultaneously, but this behaviour is being tracked separately.
Adding to this, not all data that is being recorded into Google Analytics, and other tools is human in origin. Robots and other automated applications can skew the data significantly.
Don’t even get me started on time series analysis. However, this is another important factor as often the data that is analysed in web analytics changes over time – seasonality and other external factors can have a significant influence on short and long term results.
These examples demonstrate one main point: some people using today’s user-friendly analytics tools may not be analysing your company data using statistically accurate methods and great opportunities may be being missed.
Accurate reporting is critical in helping businesses make better decisions on where to place spend and where to focus efforts. By integrating statistics-led-thinking into your reporting, you’ll get a more disciplined approach to analysis, better formed hypothesis testing, and reduced errors arising from data analysis. The good news is this can be done quickly and relatively easily.
If you want help to ensure you’re getting your reporting right, connect with us. The odds are you’ll be better off.