What is statistically significant data?
One of the very first things I learned while working at Social Fulcrum in South Boston, MA was the importance of statistically significant data in marketing and advertising. I have always understood how to calculate conversion rates and find the better performer but I was unaware of this “stat-sig” concept. Here is how Investopedia defines something as significantly significant:
Statistically significant is the likelihood that a relationship between two or more variables is caused by something other than random chance. Statistical hypothesis testing is used to determine whether the result of a data set is statistically significant. This test provides a p-value, representing the probability that random chance could explain the result; in general, a p-value of 5% or lower is considered to be statistically significant.
Wait, what? Ok, here is it again in normal-person English:
Specifically, a set of data becomes statistically significant when the set is large enough to accurately represent the phenomenon or population sample being studied. A data set is deemed to be statistically significant if the probability of the phenomenon being random is less than one out of every 20, which is why the p-value is set at 5%.
Uhh, ok. Let’s be honest, that can still be a bit confusing. Here is my version of how it was explained to me:
Say you were doing an experiment to see if quarters, when flipped in the air, are more likely to land on heads than tails. After flipping the coin ten times you get the following results:
Heads: 6 (60% conversion rate)
Tails: 4 (40% conversion rate)
Heads converted better than tails. Eureka! Heads is the winner, right? Would you be confident enough to bet $1,000 on the next ten flips that heads would be the winner again? Probably not. There is just not enough data to say that we would expect a similar outcome if we ran the test again.
Let’s say you have a lot of time on your hands and you flipped the coin one million times and the results were the following:
Heads: 550,000 (55%)
Tails: 450,000 (45%)
Heads converted better than tails again but this time there is enough data to confidently say that heads is the statistically significant winner and will continue to convert better than tails.
Quick side note: the coin-flipping example is just a simple example of how statistically significant data works; it is not meant for applications for randomization and games of chance. We’ll get into a real-world scenario in just two seconds.
How is statistically significant data used for marketing?
Statistically significant data is critical in marketing because it gives us 100% certainty that the information being used for optimizations and data-driven decisions will improve conversion rates moving forward. Statistically significant data has a multitude of applications in marketing and one of its most common uses is for A/B Testing. For those of you who are unaware of A/B Testing, it is when you test two similar components (e.g. audiences, landing pages, etc.) and change one thing to see if one variant will outperform the other.
For lead generation campaigns (Facebook Ads), I am constantly running side-experiments to ensure that we are targeting the right audience to acquire the cheapest and highest qualified leads. My most recent test is to learn if a lookalike audience made up of Lead pixel events will convert better than a lookalike audience made up of Purchase pixel events. My hypothesis is that the Purchase audience will perform better because these users have shown interest in the product. Here is the latest data:
After the first two days of advertising, the Lead lookalike outperformed the Purchase lookalike by 71%; however, the Kissmetrics AB Significance Test gave it a 86% certainty score meaning there was not enough data to call the Lead lookalike audience the statistically significant winner. The trend quickly shifted on the third day when the Purchase lookalike audience converted 188% better than the Lead lookalike audience and the Purchase lookalike audience is now converting 43% better in the fourth day of testing (all data). Just like on the second day, the certainty score is at 87% right now, which means there is a 13% chance that the trend can turn again.
The lesson of this scenario is that you need to wait until there is enough data to make statistically significant decisions regardless of how obvious the trend might seem. But what happens when you are testing a variety of variants and you cannot get a significantly significant winner? Here are a few scenarios where it is ok to end an experiment early:
- Getting costly – As shown previously, it is best to get a significant amount of data before you can crown a winner but sometimes two variants will perform the same. Instead of spending time and effort trying to find a stat-sig winner move onto your next test and throw both variants into the mix.
- Winners are crushing it – It is always smart to compare your results to benchmark metrics for your industry; I have been sourcing Wordstream’s benchmark data for the last year and I highly recommend it. You can use this data to determine if your sales funnel is over-performing your industry average. If your variants are outperforming your industry benchmarks than leverage them all – you do not need to only pick one.
- All bad performers – Of course, if your variants are significantly underperforming your industry benchmarks, you might want to end things early so you can rethink your marketing strategy. Sometimes it is not the variants you are testing that could be causing bad results such as the platform you are using, the audience you are targeting, or the sales funnel you have established.
There are going to be some scenarios where there is not enough data to have a statistically significant winner. One of the best examples is a startup running A/B tests for their newsletter that only has 100 subscribers. If you get stuck in a scenario like that, I recommend that you expand your test across several campaigns with a specific theme and compare the overall data.
In summary, statistically significant data is one of the best ways to ensure that the changes you are making to your marketing strategy and sales funnel will put you in the best possible chance for success.
Do you need help figuring out if your data is statistically significantly? Here at Perfect Pixel Marketing, we use proven marketing tactics and data-driven strategies to drive the best possible ROI for our clients. Send us a message and we will help you get on the right path to success.
