Is Your Split Test Data Lying To You?
While I do have a respect for “the numbers” and a preference for decisions to be driven by data there is somtimes an inherent distrust of what the numbers say.
Sometimes an A/B split test will have a winner that seems legitimate with a 95% confidence level in only 2 or 3 days. The test has most likely gotten enough conversions for the traffic that it received to determine a mathematic winner.
Conversion Optimization Red Flags
Every test you do needs to be validated and there are a few red flags that lead to shady data, mistrust and bad decisions made from bad data or trusting marketers.
The first red flag to look for in this situation is the traffic.
Traffic has a lot of variables that your testing tool can not see.
For example, the traffic on Saturday is different than the traffic on Tuesday. Middle of the week traffic compared to weekend traffic is vastly different. Even time of day can have a difference in traffic. There are a ton of fluctuations in traffic. To eliminate those variations in day and time, never end a test after a few days. Always allow for a full two weeks of traffic at minimum to smooth out any inconsistencies in traffic across day of the week.
Don’t Mix The Streams
Another factor to be mindful of is your traffic source. Combining different sources of traffic like PPC traffic, direct traffic, house traffic from an email send each have a different a intent and level of relationship with you and your brand, which will lead to vastly different conversion rates. When all of these different traffic sources are mixed together into a single test you get a mix of different visitor motivation levels which results in a blended conversion rate.
Beware of Sample Size
The issue I’ve seen with many do it for you split testing tools, yes even the bigger ones you have to pay for, is that none of them factor in sample size. So may times a test can reach a 95% confidence level and the total amount of visitors that ran through all variations is too small to confidently say it’s a valid test. Even if the testing tool says there is a winner.
The tragedy is that many marketers made decision thinking the data they have is clean and they never validate the numbers against these and other validity threats to testing.
Be diligent and mindful of these potential data dangers and you’ll have more predictable results going forward. We’ll explore more threats to your data in future posts.
