This document discusses Branko Rihtman and his background in SEO, science, and warzone experience. It then goes on to explain the scientific method and how to properly conduct SEO experiments and analyze data according to that method. Key steps discussed include defining a question, gathering resources, forming a hypothesis, performing experiments, analyzing and interpreting data, publishing results, and getting other perspectives on findings. Problems with SEO experimentation like lack of control and limited signals are addressed.

1. Branko Rihtman, June 2010

2. @neyne
About me – Branko Rihtman
 SEO
 Scientist
 Warzone Experience
 Becks
http://www.seo-scientist.com http://www.whiteweb.com

3. @neyne
The Scientific Method
1.Define the question
2.Gather information and resources (observe)
3.Form hypothesis
4.Perform experiment and collect data
5.Analyze and interpret data
6.Publish results
7.Retest (frequently done by other scientists)

4. @neyne
Define the question
Usually done after observation of a phenomena
• Sudden change in locations (over-optimization penalty, brand algo, etc.)
• Anomalies in SERPs
• Someone’s blog post (footer links, second link doesn’t count, etc.)
• Social media
• Things to remember:
• You don’t have time to test everything
• You don’t have the ability to test everything
• What is the ROI of your test ?
• Every change you do on your website is
a potential experiment!
"The most exciting phrase to hear in science -the one that
heralds new discoveries- is not "Eureka!" but "That's funny...".
(I. Asimov)

5. @neyne
Gather information and resources
• Related blog posts
• Problem: non-standardized terminology
• -950 Penalty
• Over-optimization penalty
• OOP
• Unbalanced link equation
• Footer link abuse
• Sidebar link abuse
• Boilerplate linking
• Forums
• Problem: misinterpretation
• Social Media
•

6. @neyne
Perform experiment and collect data
Things to remember:
• Monitor your testing grounds
• Testing keyword choice:
• [sdkfsdjwekrwle] – 0 results
• [payday loans] – 5.5M results
• [translational remedy] – 340K results
• Document everything!
Problems:
 Lack of control over the system
 Limited output signal (ranking, title,
snippet)
 Non-conformity of results
 No reliable controls

7. @neyne
Solution
Multi-directional experiments
change
- Reaction X State A State B Reaction X
- change

8. @neyne
Interpret data and draw
conclusions
 Does the conclusion agree with expectations ?
 Does the conclusion have an alternative
explanation ?
 Does the conclusion agree with other existing
data ? (if not, what was different ?)
 Bounce the findings off someone
Important: No definite conclusions!!! Only an increasing level of certainty that your
conclusion supports (or defies) the hypothesis.

9. @neyne
Data Analysis
 Statistical analysis
 Sample size
 Significance (one and two-tailed tests)
 Standard Deviation
 Correlation
Average is not an end-all measure
Get help! Talk to a professional!
People will call you out on crap! (especially if it is self-serving)
.

10. @neyne
Avoiding the personal bias
Jean Louis Théodore Géricault
The Epsom Derby
“We observe what we expect to observe,
until shown otherwise”
Ludwick Fleck

11. @neyne
Go Social!
Define the question Gather information and
resources
Forum
Twitter Social Media
Publish results Interpret data and draw
conclusions
Find an independent blogger you
can trust with data Bounce the findings off someone
Market your findings who will not steal them
Market your findings

12. @neyne
The SEO Testing Secret Ingredient
Ethanol

13. @neyne
Thank You!