Regardless if you’re new to split testing or if you’ve been testing for a while a question that keeps popping up in the back of your head is, how long should you let a test run?
If you run a test for too short a time frame you risk not knowing if it will hold up over time.
If you let a test run for too long, you’re wasting time and traffic that you could be sending to the better performing page.
There are a ton of online calculators out there that can estimate how long your test needs to run, Visual Website Optimizer has a simple tool for calculating how long a test should run.
Simply enter:
- The current conversion rate of your control page
- Your expected conversion improvement
- The total number of variations in the test including your control
- The average number of visitors per day for all variations
- The percentage of visitors included in the test
And you’ll get an estimate of how many days your test needs to run to reach a confidence level of 95%
The problem with this is it’s almost impossible to know a few key pieces that will drastically effect the length of your test.
Let’s say you had an existing conversion rate of 2% and were testing the control and one other variation and sending traffic evenly to both treatments with 200 visitors a day. If your expected conversion increase as 20% you’d have to run your test for 196 days to reach a confidence level that would have an impact on your business.
Now lets say you added a variation so you had the control vs. Version B & Version C. Three variations total, that will cost you an additional 98 days for a total of 294 days to run your test.
Realistically no business has the luxury of letting a test run this long, not to mention the fluctuations of traffic over such a long period of time that would add validity threats to your data causing you to question the final outcome after devoting so long to testing.
There are three things you can do to shorten the length of time a test needs to run.
1. Increase the amount of traffic to the test, which is not always possible and may increase your costs for that traffic.
2. Decrease the number of variations you test (limiting to just two). You saw the additional 98 days needed by just adding one more treatment to the test.
3. Increase the projected improvement. This is a tricky one. So in the same example lets say your test actually got a 30% lift from the control as opposed to the estimated 20% you expected. This would be significant for the overall time needed to run this test. With all else being equal, 2 treatments and 200 average visitors per day the time needed to run this test drops way down to just 31 days as opposed to the original 196 day estimate.
The key to a bigger improvement increase is to test bigger changes. So many people only test small changes and once they see an improvement, even a small one, they accept that and make changes based on it, expecting the improvement to hold over time, when it usually does not.
