The document discusses scaling experimentation from a few to over a hundred experiments in a software environment, detailing setups involving SDKs and analytics. It highlights challenges faced at different scales of experimentation, emphasizing the need for centralized metrics tracking. It also outlines examples and considerations for managing multiple experiments effectively.

2. 2
Scaling Fullstack
Going from 1 to 100+ experiments
Asa Schachar
Engineering Manager
Ali Rizvi
Staff Software Engineer
Thilo Richter
Solutions Architect Manager

3. 1 to 10 Experiments

5. Simple Experimentation Setup
Application
SDK
Datafile
A
B
Remote
Configuration
Analytics

6. Application Backend
app.optimizely.com
Analytics
Backend
Frontend Application
app.optimizely.com
Your App
SDK
Datafile
Optimizely User
Your Users
Events
A
B

7. Application Backend
app.optimizely.com
Analytics
Backend
Frontend Application
app.optimizely.com

8. Application Backend
app.optimizely.com
Analytics
Backend
Frontend Application
app.optimizely.com
SDKSDK
SDK

9. Installing SDKs
Application Backend
app.optimizely.com
Analytics
Backend
Frontend Application
app.optimizely.com
SDKSDK
SDK

10. Datafile Polling
Application
SDK
Datafile
A
B
5 min
CDN

11. Datafile Polling
Application
SDK
Datafile
A
B
Remote
Configuration
Analytics
CDN
5 min

12. 10 to 100
Experiments

14. Example #1
SDK Stateless
Bucketing Logic
app thread #1
Analytics Backend
Request #1SDK
SDK app thread #2 Request #2
SDK app thread #3 Request #3

15. Example #1
Analytics Backend
app thread #1 Request #1
app thread #2 Request #2
Wrappe
r
Stateful, Thread-safe,
Singleton Wrapper
Wrapper
SDK
SDK Stateless Bucketing
Logic

16. Example #2

17. Example #2

18. Example #2

19. Example #2

20. 100+
Experiments

23. 100%
100%
Local
(local computer)
Staging
(staging website)
Production
(production website)
Rollback
23
0% 0%
Environments

25. What are your challenges?

26. Challenges of 1-10
Experiments
1.
2.

27. Challenges of 10-100
Experiments
1. Centralized Metrics Tracking
2.

28. Challenges of 100+
Experiments
1.
2.

29. Q&A