1. Scaling Applications and
Mission Critical Events

2. TO FAIL WELL, FAIL FIRST
BEFORE IT MATTERS, & FAIL
AGAIN BEYOND EXPECTATION.

3. WHO ARE WE
Michael Meyers
Managing Director
Jeremy Andrews
Founder & CEO
➔ CTO 1st venture backed
Drupal startup
➔ CTO 1st top 100 Drupal site,
#1 contributor to D7
➔ VP @ Acquia - Dev
Relations, Marketing,
Products
➔ Advisor to Drupal Assoc.
➔ 15 Years with Drupal
➔ Performance, scalability &
application security expert
➔ Linux firewalls and
KernelTrap.org
➔ Orig. author Drupal installer,
Throttle, pager logic, etc
➔ Maintain Memcache module
➔ Active Drupal contributor
for 18 years

4. #2 All-time Drupal Contributor
12% of Current
Top 50 Contributors
Architects of Fortune 500 sites
Leaders
from Drupal 4 to Drupal 8
Leaders in DevOps and
Automated Testing
Consulted for every major
Drupal platform provider

5. Tag1 wrote the book on
performance and scalability
Leaders
from Drupal 4 to Drupal 8
Leaders in DevOps and
Automated Testing
Consulted for every major
Drupal platform provider

6. IF TALK TIME ALLOWS
Mention some other load testing
clients w/ stats to establish &
demonstrate world class
expertise…
➔ ACLU 85x
➔ PAC12
➔ WhiteHouse.gov
➔ Fortune Top 50
➔ Performance book

7. AGENDA
What you get in 30 minutes.
➔ Strategy, tools & technologies
for load testing
➔ Best practices: an iterative
approach to testing & tuning
➔ Load test and performance
tuning tips and tricks
➔ How to plan for success &
mitigate risk
➔ Q&A

9. Record Holiday Sales & Revenue in the US

10. 光棍节 Scaling Foreo for Global B2

11. Speed Matters
It’s not just about downtime
➔ Amazon
◆ A page load slow down of just 1s
could cost $1.6 billion in sales
each year
➔ Google
◆ A 400ms slowdown results in 440
million abandoned sessions per
month - lots of lost ad revenue
➔ Walmart
◆ For every 100 ms of improvement,
incremental revenue grew 1%
Source:
https://medium.com/@vikigreen/impact-of-slow-page-load-time-on-website-performance-40d5c9ce568a

12. Testing
Life Cycle
Iterate until risk is mitigated
Implement
Solutions
Goals &
Objectives
Test Strategy
Test DesignResults Analysis
Test Execution

13. Goals &
Objectives
Test Strategy
Test DesignResults Analysis
Test Execution
Goals
& Objectives
Business outcomes,
performance targets
Build a model for
business $uccess
Implement
Solutions

14. FOREO: 2019 Holiday Shopping Season Goals
➔ Business Goals
◆ Greatly exceed prior year’s traffic and sales metrics for maximum YoY growth
◆ Test and tune new Drupal 8 site
➔ Margin
◆ Support at least a 50% increase over projections
➔ Performance Objectives
◆ Maintain <2s page load times under peak loads

15. Goals &
Objectives
Test DesignResults Analysis
Test Execution
Test Strategy
Test Strategy
Scope Limitations
Tooling
Software & Env. Set Up
Implement
Solutions

16. II: Test StrategyStrategy: Scope Limitations
➔ Risk mitigation - you can’t do everything
➔ Identify and focus in on “critical paths”
➔ What aren’t you going to test?
➔ How are you going to test 3rd parties, APIs, etc.
➔ Destructive tests that create content & content removal

17. II: Test Strategy
● Risk Mitigation Planning: Balancing time, money, costs, etc.
○ How are you going to test APIs, 3rd party systems, etc?
● Picking the right tools
● Set up your Load Testing Environment
● Set up your Load Test Target
● Proactive, not Reactive: make testing a part of your dev process.
Strategy: Picking the right tools - Locust
➔ Write tests & load plan in Python
➔ Leverage any Python library for integrations and testing
➔ Better than JMeter and other tools for creating plans and patterns
➔ Extensible flask-based UI
➔ Easy to distribute load generation

18. II: Test Strategy
● Risk Mitigation Planning: Balancing time, money, costs, etc.
○ How are you going to test APIs, 3rd party systems, etc?
● Picking the right tools
● Set up your Load Testing Environment
● Set up your Load Test Target
● Proactive, not Reactive: make testing a part of your dev process.
Strategy: Set up Load Testing Environment
➔ Setup Locust
◆ How-to: https://tag1consulting.com/blog/distributed-load-testing-locust

19. II: Test Strategy
● Risk Mitigation Planning: Balancing time, money, costs, etc.
○ How are you going to test APIs, 3rd party systems, etc?
● Picking the right tools
● Set up your Load Testing Environment
● Set up your Load Test Target
● Proactive, not Reactive: make testing a part of your dev process.
Strategy: Set up Load Test Target(s)
➔ Replica of Production is ideal, but not always an option
➔ Full stack & processes: CDN, reverse proxies, cron
➔ 3rd party APIs and systems
➔ Plan ahead, notify everyone (hosting platform, 3rd parties/partners, etc.)
➔ Monitor everything

20. FOREO: DevOps & Performance
Testing
➔ Be proactive: really high traffic & really high stakes!
➔ Performance testing baked into to dev processes
➔ Integrated into CI/CD pipeline
➔ Auto accept/reject PRs based on performance

21. Test Design
Goals &
Objectives
Test Strategy
Test DesignResults Analysis
Test Execution
Test Design
Load models
&
Create Tests
Implement
Solutions

22. II: Test Strategy
● Risk Mitigation Planning: Balancing time, money, costs, etc.
○ How are you going to test APIs, 3rd party systems, etc?
● Picking the right tools
● Set up your Load Testing Environment
● Set up your Load Test Target
● Proactive, not Reactive: make testing a part of your dev process.
Test Design: Load Models
➔ Running tests independently
➔ Ratios for running tests concurrently
➔ Incremental ramp up of load

23. II: Test Strategy
● Risk Mitigation Planning: Balancing time, money, costs, etc.
○ How are you going to test APIs, 3rd party systems, etc?
● Picking the right tools
● Set up your Load Testing Environment
● Set up your Load Test Target
● Proactive, not Reactive: make testing a part of your dev process.
Test Design: Creating Tests
➔ Check http headers, and page content, etc.
➔ Test static content as a % of total, roughly 10% after first pass
➔ Data parameterization and mapping
➔ Test your tests!
➔ Consistency is key - don’t make major changes when iterating

24. II: Test Strategy
● Risk Mitigation Planning: Balancing time, money, costs, etc.
○ How are you going to test APIs, 3rd party systems, etc?
● Picking the right tools
● Set up your Load Testing Environment
● Set up your Load Test Target
● Proactive, not Reactive: make testing a part of your dev process.
Test Design: Creating Tests in Locust
➔ How-to: https://tag1consulting.com/blog/drupal-loadtest-locust
➔ Code your load model as a Python script
➔ Take advantage of Python libraries

25. FOREO: Creating Tests
➔ Evaluate functional paths & analytics determine critical paths
◆ View category & product pages, add to cart, complete purchase
◆ Browse (non product) pages e.g., About, FAQ, etc.
◆ Auth: Support, Register products
➔ Handle localization for translated content
◆ Accommodate variances (e.g., submit button changes)

26. Test Execution
Goals &
Objectives
Test Strategy
Test DesignResults Analysis
Test Execution
Test
Execution
Establish Baselines
Scale incrementally
Monitor
Implement
Solutions

27. II: Test Strategy
● Risk Mitigation Planning: Balancing time, money, costs, etc.
○ How are you going to test APIs, 3rd party systems, etc?
● Picking the right tools
● Set up your Load Testing Environment
● Set up your Load Test Target
● Proactive, not Reactive: make testing a part of your dev process.
Test Execution
➔ Scale up load incrementally - don’t go too fast, too soon
➔ Work through your load models
➔ Consider the length & timing of your test runs

28. II: Test Strategy
● Risk Mitigation Planning: Balancing time, money, costs, etc.
○ How are you going to test APIs, 3rd party systems, etc?
● Picking the right tools
● Set up your Load Testing Environment
● Set up your Load Test Target
● Proactive, not Reactive: make testing a part of your dev process.
Test Execution: Monitoring in real-time
➔ Check everything from your infra to the edge
➔ Logging and monitoring
➔ Check test output: Locust logs, http headers and page content
➔ Confirm tests are hitting all aspects of your application

29. Results
Analysis
& Reporting
Goals &
Objectives
Test Strategy
Test DesignResults Analysis
Test Execution
Result
Analysis
Identify
improvements
Triage solutions
Reporting
Implement
Solutions

30. II: Test Strategy
● Risk Mitigation Planning: Balancing time, money, costs, etc.
○ How are you going to test APIs, 3rd party systems, etc?
● Picking the right tools
● Set up your Load Testing Environment
● Set up your Load Test Target
● Proactive, not Reactive: make testing a part of your dev process.
Results Analysis & Triage
➔ Focus on the major bottleneck - don’t over optimize
➔ Low hanging fruit

31. Reporting
Leaders
from Drupal 4 to Drupal 8
Leaders in DevOps and
Automated Testing
Consulted for every major
Drupal platform provider

32. Prioritizing
Solutions
Leaders
from Drupal 4 to Drupal 8
Leaders in DevOps and
Automated Testing
Consulted for every major
Drupal platform provider
Phase

33. Recommendations
&
Code Review Checklist
Leaders
from Drupal 4 to Drupal 8
Leaders in DevOps and
Automated Testing
Consulted for every major
Drupal platform provider
Phase

34. Implement
Solutions
Triage & Patch
Goals &
Objectives
Test Strategy
Test DesignResults Analysis
Test Execution
Implement
Solutions
Eliminate Bottlenecks
Low hanging fruit
Incremental updates

35. II: Test Strategy
● Risk Mitigation Planning: Balancing time, money, costs, etc.
○ How are you going to test APIs, 3rd party systems, etc?
● Picking the right tools
● Set up your Load Testing Environment
● Set up your Load Test Target
● Proactive, not Reactive: make testing a part of your dev process.
Implement Solutions
➔ Fix a problem, validate the solution.
➔ Do not make a lot of changes!

36. Examples Test Strategy
Test DesignResults Analysis
Test Execution
Implement
Solutions
Goals &
Objectives
Implement
Solutions
Results Analysis
Test Execution

37. EXAMPLE Foreo - Test Execution & Monitoring

38. EXAMPLE Foreo - Results Analysis: Opcode Cache

39. EXAMPLE Foreo - Implement Solution & Test

40. EXAMPLE Foreo - Results Analysis

41. EXAMPLE Foreo - Implement and Test

42. EXAMPLE Foreo - Analyze, Implement, Test...

43. PRO TIPS
✅ Be Proactive, not Reactive
✅ Design & execute different load models
✅ Don’t make assumption - test entire stack
✅ Don’t replicate traffic 1:1 - focus on critical paths
✅ Be methodical
✅ Don’t scale load tests too fast, too soon
✅ Don’t make lots of changes. Make a change, test.
✅ Triage - focus on the biggest bang for your buck
✅ Iterate - bottlenecks block other bottlenecks.

44. QUESTIONS?

45. Subtitle
Add speaker name here
Title slide
Join us for
contribution opportunities
Friday, April 12, 2019
9:00-18:00
Room: 602
Mentored
Contribution
First Time
Contributor Workshop
General
Contribution
#DrupalContributions
9:00-12:00
Room: 606
9:00-18:00
Room: 6A

47. CONTACT US AT ANY TIME
Michael Meyers
m@tag1consulting.com
+1-917-402-8263 (toll free)