SEO Split Testing and Statistical Significance
We decided not to calculate statistical significance for split tests, because the split test graph/results are based on averages of the test group clicks, and the control group clicks, with a daily difference then calculated. That daily difference is then averaged out again pre and post change periods (blue and black line). With all this calculation we concluded the statistical significance calculation (based on the daily difference data points) would not be meaningful or true.
So, upon creating the split tests, we added the option to create separate time-based group tests for both the control and test group. This then allows you to do three things:
Use the SEO split test to see how the test group performs against the control group over the same time period.
See how much the test group itself has improved, by checking the test group test.
Check the statistical significance of the test and control group separately, by looking at the associated group tests to make sure each group hits statistical significance.
This is also summarised in a video : https://www.loom.com/share/3ba3caafbb28420fbeb93fa995c14221
So, upon creating the split tests, we added the option to create separate time-based group tests for both the control and test group. This then allows you to do three things:
Use the SEO split test to see how the test group performs against the control group over the same time period.
See how much the test group itself has improved, by checking the test group test.
Check the statistical significance of the test and control group separately, by looking at the associated group tests to make sure each group hits statistical significance.
This is also summarised in a video : https://www.loom.com/share/3ba3caafbb28420fbeb93fa995c14221
Updated on: 20/05/2025
Thank you!