Advanced Experimentation Workshop presented by James Spittal, CEO, Web Marketing ROI & Holly Joshi, Head of Customer Success APAC, Optimizely. Watch the recorded talk here https://www.youtube.com/watch?v=_Fv15lQOGyE or https://optimizely.wistia.com/medias/3hnx60ue8u Hands on workshop covering the most common and challenging use cases in advanced experimentation today. Inclusive of data challenges, integrations, multi-page, customised and URL testing. ABOUT MODERNIZE 2018: Be a part of the future. Shape the customer experience with Accenture, Forrester, Qantas, REA, Intrepid and more -- and learn how leading organisations are harnessing their people, processes, and technologies to exceed customer’s expectations. Visit: https://www.optimizely.com/anz/modernize2018

1. experience
Advanced Experimentation Workshop
James Spittal, CEO, Web Marketing ROI
Holly Joshi, Sr. Customer Success Manager, Optimizely
Mansoor Omar, Technical Support Engineer, Optimizely

2. 1. Tackling data problems
• Misconceptions/theories
• Data between GA and Optimizely
• How to ensure what you’re seeing is accurate
2. Overcoming common challenges faced operationally
• Writing regular expressions
• Cross-domain Optimizely
3. Making your campaign development faster and more efficient
• Optimizely Extensions
• Reducing flickering
Agenda
experience

3. CRO/MarTech Leadership = GSD
experience

4. experience

6. experience
All About The Data
What Optimizely measures and integrating with your data systems

7. 1. Resolving the misconceptions
2. Implementation is the solution
3. Get the most common/most critical
right
Optimizely Data
experience

8. 73%
… the percentage of Google Analytics implementations that are wrong (leading to
incorrect traffic numbers, incorrect revenue, etc).
Source: https://econsultancy.com/blog/10925-80-of-online-retailers-are-using-google-
analytics-incorrectly

10. Apples & Oranges
experience

11. If you’re not comparing the same thing –
you’re not going to see the same numbers.
Apples & Oranges
experience

12. Google Analytics Session Time-Outs (and
Optimizely End User Id)
By default, a Google Analytics session
expires after 30 minutes of user inactivity.
This is entirely different to how Optimizely
tracks a “unique visitor” (tied to Optimizely
End User Id cookie, 10 year expiry).
experience

13. Impressions (new Optimizely measurement
model) – How it actually works
Optimizely logs an event whenever an experiment activates (this can be seen as a ‘decision’
under the hood as an XMLHTTPRequest)
The decision carries a few pieces of information, including:
Visitor ID
The ID of the experiment / campaign which is activating
The ID of the variation the visitor is allocated to
UUID: A unique decision ID (used for deduplication)
Again, decision events are generated whenever an experiment activates, even if the visitor
has been targeted by the same experiment previously. So in a non-technical sense, you can
imagine that a decision is generated each time an experience is delivered to a visitor.

14. Impressions (new Optimizely measurement
model)
select
count(distinct(UUID))
from
decisions;
This deduplication on UUID is important because Optimizely clients sometimes send multiple
decisions for the same experiment activation (this is due to the batching behavior of the web
snippet).
Typically, impressions are allocated on an annual basis. So we just count up the number of
unique decision events, subtract them from the annual quota, and voila, you've got your
"impressions remaining" count for the year.
experience

15. Optimizely Results Screen Example

16. Pages misfiring/misactivating
experience

17. Audience Conditions – Custom JavaScript
experience

18. QA everything (DIY)
1. Make a spreadsheet of all of your
events, pages and audiences
2. Create a QA campaign for each, to verify
it is triggering when intended and not
triggering when it shouldn’t.
experience

19. Optimizely Accelerate
experience

20. Optimizely Accelerate
Total cost around $15k AUD, includes full events/pages/URLs/audiences QA and campaign build,
delivered by approved Optimizely Solutions Partners.
experience

21. experience
URLs
Custom, Cross-Domain, Multi-Page

22. 1. Regular expressions (versus Simple
Match, Exact Match and Substring Match)
2. Multi-page tests
3. Making Optimizely work cross-domain
(and/or cross-project)
Pages & URLs
experience

23. 1. Matching too freely
2. ‘example.com’ Matches (Substring) for:
yourdomain.com.au/example.com
Common ‘Pages’ mistakes
experience

24. Regular Expressions Cheat Sheet
experience

25. Regular Expressions Practice

26. 1. This makes Optimizely cross-domain
work
2. In short - Injects an iFrame that allows
local storage/cookie to traverse
3. Put waitForOriginSync into your Project
JS
WaitForOriginSync
experience

27. WaitForOriginSync
experience

28. experience
Campaign Development
Extensions, Efficiencies and Performance

29. 1. Extensions = more campaigns, faster
• Extensions 101
• Why extensions can be tricky
• Extension of extensions
2. Experiment flickering minimisation
Campaign Development
experience

30. Extensions

31. Extensions are extendable!
experience

32. 1. Templates, modularity, performance,
strategy, operational savings…
2. Object- orientated
101
experience

33. 1. How many input fields should we have?
2. Why is my extension not showing up/not
accessible?
3. Where did my extension go?
4. Data drops out of an extension
Challenges
experience

34. 1. Translation
[:en]
Hello World
[:zh]
(Traditional Chinese)
2. Markdown
3. Extending extensions
Cool Use Cases
experience

35. Page Flash/Flickering
https://media.giphy.com/media/wdh1SvEn0E06I/giphy.g
if

37. 1. WaitForElement & ES6
2. WaitUntil
3. See the Optimzely Developer Docs
4. Async vs. Sync
Document object model
challenges
experience

38. jQuery – what you need to know about
Optimizely X
Var $ = window.optimizely.get(‘jQuery’) [Optimizely jQuery] when
instrumenting jQuery in your Optimizely campaigns goes wrong versus
var $ = window.jQuery;
var $ = window.jQuery || window.optimizely.get(‘jQuery’);
experience

39. 1. Use ‘utils’ (WaitForElement, etc)
2. Use ES6/Native JavaScript
3. Asynchronous versus synchronous
Most important things to know
experience

40. JavaScript Hell

41. Asynchronous versus synchronous

42. Don’t always believe what Google
PageSpeed Insights tells you 
experience

43. Document Object Model
experience

44. experience
Takeaways
Recap & Resources

45. 1. If you need help with data analysis
problems or auditing your GA data, speak
to an Optimizely Solutions Partner or
GACP
2. Manny and Holly are available for on-
demand Optimizely support for Optimizely
customers
Next steps
experience

46. experience
Questions?
james@webmarketingroi.com.au
James Spittal
THANK YOU