For example, If you are redesigning the site architecture of your product page templates, you should consider making it noticeably different from both a visual and back-end (code structure) perspective. While user research or on-page A/B testing may have led to the new architecture or design, it’s still unclear whether the proposed changes will impact rankings.
This should be the most common reason that you run an SEO split test. Given all of the subjectivity of the pre-test and post-test analysis, you want to make sure your variation yields a different enough result to be confident that the variation did in fact have a significant impact. Of course, with bigger changes, comes bigger risks.
While larger sites have the luxury of testing smaller things, they are still at the mercy of their own guesswork. For less robust sites, if you are going to run an SEO split test on a template, it needs to be different enough not only for users to behave differently but for Google to evaluate and rank your page differently as well.
Communicating experimentation for SEO split-tests
Regardless of your SEO expertise, communicating with stakeholders about experimentation requires a skill set of its own.
The expectations with testing are highly volatile. Some people expect every test to be a winner. Some expect you to give them definitive answers on what will work better. Unfortunately, these are false expectations. To avoid them, you need to establish realistic expectations early on for your manager, client, or whoever you are running a split test for.
Expectation 1: Most of your tests will fail
This understanding is a pillar of all successful experimentation programs. For people not close to the subject, it’s also the hardest pill to swallow. You have to get them to accept the fact that the time and effort that goes into the first iteration of a test will most likely lead to an inconclusive or losing test.
The most valuable aspect of experimentation and split-testing is the iterative process each test undergoes. The true outcome of successful experimentation, regardless if it’s SEO split-testing or other types, is the culmination of multiple tests that lead to gradual increases in major KPIs.
Expectation 2: You are working with probabilities, not sure things
This expectation applies especially to SEO split-testing, as you are utilizing a variety of metrics as indirect signals of success. This helps people understand that, even if you reach 99% significance, there are no guarantees of the results once the winning variation is implemented.
This principle also gives you wiggle-room for pre-test and post-test analysis. That doesn’t mean you can manipulate the data in your favor, but does mean you don’t need to spend hours and hours coming up with an empirically data-driven projection. It also allows you to utilize your subjective expert opinion based on all the metrics you are analyzing to determine success.
Expectation 3: You need a large enough sample size
Without a large enough sample size, you shouldn’t even entertain the idea of running an SEO split test unless your stakeholders are patient enough to wait several months for results.
Sam Nenzer, a consultant for SearchPilot and Distilled, explains how to know if you have enough traffic for testing:
“Over the course of our experience with SEO split testing, we’ve generated a rule of thumb: if a site section of similar pages doesn’t receive at least 1,000 organic sessions per day in total, it’s going to be very hard to measure any uplift from your split test.”
Therefore, if your site doesn’t have the right traffic, you may want to default to low-risk implementations or competitive research to validate your ideas.
Expectation 4: The goal of experimentation is to mitigate risk with the potential of performance improvement
The key term here is “potential” performance improvement. If your test yields a winning variation, and you implement it across your site, don’t expect the same results to happen as you saw during the test. The true goal for all testing is to introduce new ideas to your site with very low risk and potential for improved metrics.
For example, if you are updating the architecture or code of a PDP template to accommodate a Google algorithm change, the goal isn’t necessarily to increase organic traffic. The goal is to reduce the negative impact you may see from the algorithm change.
Let your stakeholders know that you can also utilize split-testing to improve business value or internal efficiencies. This includes things like releasing code updates that users never see, or a URL/CMS update for groups of pages or several microsites at a time.
While it’s tempting to run an SEO split test, it’s vital that you understand the inherent risks of it to ensure that you’re getting the true value you need out of it. This will help inform you on when the scenario calls for a split test or an alternative approach. You also need to be communicating experimentation with realistic expectations from the get-go.
There are major inherent risks of engaging with SEO split-testing that you don’t see with on-page tests that CRO usually runs, including wasted resources and non scalable results.
Some of the scenarios where you should feel confident in engaging with an SEO split test include where you’re uncertain of keyword and query performance, proof-of-concept and risk mitigation for larger-scale websites, justification for ideas that require robust resources, and when you’re considering making big changes to your templates.
And remember, one of the biggest challenges of experimentation is properly communicating it to others. Everyone has different expectations for testing, so you need to get ahead of it and address those expectations right away.
If there are other scenarios for or risks associated with SEO split-testing that you’ve seen in your own work, please share in the comments below.
Sign up for The Moz Top 10, a semimonthly mailer updating you on the top ten hottest pieces of SEO news, tips, and rad links uncovered by the Moz team. Think of it as your exclusive digest of stuff you don’t have time to hunt down but want to read!