This document provides an overview of using data analysis techniques like Causal Impact and Prophet to analyze SEO performance data. Causal Impact can be used to measure the impact of a single change on a metric like traffic or conversions by comparing actual data to a predicted counterfactual. Prophet allows removing noise from overall growth trends by incorporating additional factors like search volume or economic indicators. The document outlines examples, limitations, and resources to get started with these techniques.

1. Cutting through the
noise in SEO with data.

2. 10 million organic visits
£8 billion every month
13+ million customers
6+ million indexed URLs
@reinahanada @Wise

3. Data is difficult.

4. ‘I’m so grateful that this
hell will be over
tomorrow. I can
proudly say, I do not
want to do this ever
again.’
- Reina, 2015

6. Data show us
what human eyes
struggle to see 👀

7. My goal:
By the end of this,
you’ll be able to:
Causal Impact + Tensorflow Probability
E.g. measuring the effect of Algo update
Before & after test 🧪
Prophet with regressor
E.g. removing noise from growth factors
Remove noise from data 📊

8. “It’s statistically
significant.”
🧪

9. Statistical significant =
very unlikely to have occurred
simply by chance
You can feel confident that it’s real, not
that you just got lucky or unlucky

10. Before & After test with
Causal Impact.

11. Causal impact = Reality vs ‘what if’?

12. Suitable for:
Any change in
a single day
Did Google Algo update impact
our traffic or position?
Algorithm update
Did updating the CTA improve our
conversion?
Changing CTA / design
Did optimisation help improve our
position?
On-page optimisation

13. January 10, 2022 - Migration to new landing page

14. First visits and new users are dropping 🧪
MIGRATION

15. Is this our fault? Do we revert back?

16. Conversions
down?
Traffic down? CVR down?
Rankings drop?
Search volume
drop?
Design
change
Tracking issue
(secret option)
Seasonality
External factors
Algo update
SERP change
On page
change/
error
Competitor
activity
Do nothing
What could have
gone wrong?

17. Data +
exact date
Causal
impact
(python
package)
Graph +
report

18. Data +
exact date
Causal
impact
(python
package)
Graph +
report

19. Ingredients = data + the exact date 🍳

20. Ranking?
Search volume?
Conversion rate?

21. Data +
exact date
Causal
impact
(python
package)
Graph +
report

23. Ranking?
Search volume?
Conversion rate?

25. Data +
exact date
Causal
impact
(python
package)
Graph +
report

26. First visits = YEP, we are losing visits 😭
…and it’s proven to be statistically significant

27. Real data vs predicted data
Orange shade = 95% probability

28. Difference between real vs predicted

29. Accumulated differences
Overall impact = negative

30. Report: “it’s statistically significant”
p-value = 0.01 (only 1% chance that this is accidental)

31. Always check: p-value
p-value = the probability of getting this result by
chance. Closer this gets to 0%, more likely that it
is not by chance
p-value < 0.05 = it’s statistically significant
p-value > 0.05 = it’s NOT (= just accident?)

32. New users = Drop is not statistically significant 😇

33. 2 weeks lag - change the date

35. 2 weeks lag = Drop is statistically significant 😭

36. 4 weeks lag = Drop is statistically significant 💩

37. Maybe drop in search trend? 🧪
Google Trend = ‘transfert argent’ (remittance)

39. New users (4 weeks lag) with search trend
Drop is not statistically significant 😮💨
More
than 5%

40. We are losing new users,
but it might be due to drop in
search demand 🧪

41. Next step =
Keep the new design,
but optimise

42. Judgement =
Use common sense +
a pinch of pessimism

43. Python package: tfcausalimpact
Language Python
Author WillianFuks (CausalImpact by Google +
Tensorflow by Google)
Ingredients Data + exact date of the change
Regressor Optional (as many as you want)
Used for Impact of the change on a single day

44. Troubleshooting 🧪🔥
Google spreadsheet
● Dates should be in ascending order, in YYYY-MM-DD
● Numbers should not have dots (1,000 → 1000)
● Make sure there are no empty cells
● Useful formula for Google Trends: =if(len(B2),B2,C1)
Python
● When it fails - start from importing data again
● Check how data looks with data.head() or print(data)
● Check all data types are float or integers by data.dtypes

45. Causal impact =
measuring the impact of
1 change on 1 date

46. Removing external noise
with Prophet.

47. Prophet = forecasting based on trend 📈
FORECAST
(We won’t use)
TREND

48. Suitable for:
Overall trend
Is the traffic growing? Is it due to
search demand?
Growth (+ relationship
with price changes, etc)
Is there steady growth as we publish
more blog articles?
Growth in blogs

49. Our US blog is growing 🚀
Are we actually growing???

50. Or is it because… USD is at all time high? 🧪

51. Data +
regressors
Prophet
(python
package)
Graphs

52. Data +
regressors
Prophet
(python
package)
Graphs

53. Ranking?
Search volume?
Conversion rate?

54. Data +
regressors
Prophet
(python
package)
Graphs

58. Data +
regressors
Prophet
(python
package)
Graphs

59. Growth trend without the impact of USD FX rate
We are growing! 😍
TREND

60. Trend with the impact of USD FX rate
…We are stagnating! 💩
TREND

61. We are growing,
but growth might be coming
from exchange rate 😅

62. We are growing,
but we should be
growing more 🧪

63. REMINDER:
Use common sense +
a pinch of pessimism

64. Next step = focus on growth

65. Python package: Prophet
Language Python
Author Facebook
Ingredients Data (12+ months)
Regressor Optional (as many as you want)
Used for Forecast / time series trend

66. Troubleshooting 🧪🔥
Google spreadsheet
● You can have missing data - but preferably not
● Do not name a column named ‘trend’ - it’s reserved
● (Everything as I’ve said in Causal Impact)
Python
● (Everything as I’ve said in Causal Impact)

67. Prophet =
measuring trend over time

68. Important concept:
Multicollinearity.

69. Always check this before
running Causal Impact /
Prophet! 🚨

70. Multicollinearity = two regressors are correlated
Adds noise + confuses the model 😵💫

71. How to spot multicollinearity 🔎
Common sense
✅ No coding needed
❌ Inaccurate
VIF
(Variance inflation factors)
✅ Accurate
❌ Coding needed
VIF colab notebook + testing method in the slide

72. VIF - Remove any variables with VIF > 1.5
(1.5 is VERY conservative; can be 2-5)

73. Python package: VIF from Statsmodels
Language Python
Author Open source
Data Data

74. How to run VIF in 3 easy steps 💥
1. Google sheet with all the data (variables)
2. Open Colab Notebook and change here
3. Click buttons

75. Keep on running until you get all < 1.5

76. My goal:
And now, you should
be able to:
Causal Impact + Tensorflow Probability
E.g. measuring the effect of Algo update
Before & after test 🧪
Prophet with regressor
E.g. removing noise from growth factors
Remove noise in data 📊
✅
✅

77. Cheatsheet
Causal
impact
● Changed template
● Changed CTA
● Google Algorithm update
● Fee change of product
● Russian war on Ukraine
Prophet ● Traffic & Google trend relationship
● Overall trend with seasonality
● Effect of inflation

78. Limitations + cautions
Causal
impact
● Highly dependent on the data points
(provide as much data; consider time lag)
● Not good for multiple changes happening at
once / changes happening over time
● Very slow to run
● Small numbers (e.g. CvR) - hard to detect
Prophet ● Sensitive to seasonality - should provide
multiple seasons
● Automatically tries to ‘fit the model’ - so we
cannot specify to ‘prioritise’ one regressor
over another
● Weak on outlier / large impact events

79. Colab notebook links
Causal
impact
https://colab.research.google.com/drive/1MdYf-
78Lt1NicCPQ3ax2Orxfgzy_nsUn#scrollTo=ZzAIj
GO0OHjd
Prophet https://colab.research.google.com/drive/1RdDN
D2I81KeFhIXejCg0h4V3h3zo0t0m#scrollTo=3ky
jYeMK2DzK
VIF https://colab.research.google.com/drive/1b18CT
9bEcGqaqPibb_b8BSIPPa7lHZpo#scrollTo=7JU
8VAPOj6Z2

80. Isn’t this too easy to be true?
🧪

81. ‘Sometimes the simplest tools are pure and
effective. You only use a complex technique if
there is no simpler way.
It is the principles of analysis (the logic, the
conclusions) that are the most powerful.’

82. Data analysis is easy,
coming up with variables
is difficult.

83. Data analysis is easy,
coming up with a story
is difficult.

84. Data = creativity 🎨

85. SEOs = creative 😎

86. SEO + data = 🔥

88. wi.se/seo