The document discusses shifting performance testing left in the development process. It argues that with increased software complexity, testing needs to start earlier to avoid delays. Single user performance testing can be run by developers as part of their normal testing to gain immediate feedback. This involves measuring responsiveness, network traffic, and device vitals under different conditions. While load testing still has value, splitting it up and combining it with functional and responsiveness testing allows more testing to be done earlier in development.

2. Shifting left performance testing
Stop discovering delays on launch day
Amir Rozenberg
Director of Product at Perfecto

3. • Massive changes in SW Dev & Performance
Engineering
• What is appropriate to shift left?
• Demo

4. Massivechanges to SW Dev & PerformanceEngineering
• Digital transformation driving agility
• Scope/TTM/quality Challenges
• Technology complexity balancing to client-side

5. Massivechanges to SW Dev & PerformanceEngineering
• Digital transformation driving agility
• Scope/TTM/quality Challenges
• Technology complexity balancing to client-side
• Scalable engineering squads must adopt frequent testing to avoid "Big
Bang"
• Release delay or go with risk?
• Developers are increasingly participating in testing

6. OK,so what about PerformanceEngineering& Testing
• Developers are increasingly inherently performance-budget aware
• Does Performance Testing = Load Testing ?
• Can dev environments sustain load?
• Can you run load testing per commit? Smoke test? Regression test?
• Does it make sense to run multi-user testing (if you can’t run load)?
• What is the reality?

7. OK,so what about PerformanceEngineering& Testing
Or, can you split it up?
APM/
Production
Monitoring
Functional testing: regression,
smoke?, commit?
Proactive multiplatform
monitoring
Single pane of App
health
Load TestingResponsiveness testing
Logs & triage data

8. OK,so what about PerformanceEngineering& Testing
BTW, this is a pretty generic problem
APM/
Production
Monitoring
Functional testing: regression,
smoke?, commit?
Proactive multiplatform
monitoring
Single pane of App
health
Accessibility Testing
Security Testing
…

9. "Single User PerformanceTesting"
What
• Responsiveness measurement
• OCR- automated eyes on the screen
• Web page timers (W3C): page and object level
• (# of objects, sizes, timing, # of 3rd party service APIs, HTTP responses)..
• Consider for script assertion policy
• Benchmark & trend
• Filmstrip
• Logs
• Network traffic (HAR, PCAP)
• Device vitals (CPU, network, memory..)
• Crash log
• Conditions
• Mobile networks (2G, 3G, changes)
• Resource-intensive apps in background (ex.: Pandora ;))

10. "Single User PerformanceTesting"
What
• Responsiveness measurement
• OCR- automated eyes on the screen
• Web page timers (W3C): page and object level
• (# of objects, sizes, timing, # of 3rd party service APIs, HTTP responses)..
• Consider for script assertion policy
• Benchmark & trend
• Filmstrip
• Logs
• Network traffic (HAR, PCAP)
• Device vitals (CPU, network, memory..)
• Crash log
• Conditions
• Mobile networks (2G, 3G, changes)
• Resource-intensive apps in background (ex.: Pandora ;))
Why
• Developers get instant feedback
• Performance knowledge, analysis in
real time
• No end-of-sprint surprises
• Performance engineers join squads
• Performance appear in sprint
requirements

11. Let’sdivein: Visualrenderingcompletion
Network WiFi 4G 3G
App launch 1.8 2.0 2.6
Login page 1.5 2.8 3.4
Login 4.5 7.2 8.8
8.026E+10
8.026E+10
8.026E+10
8.026E+10
8.026E+10
8.027E+10
8.027E+10
8.027E+10
8.027E+10
8.027E+10
0
50
100
150
200
250
1 3 5 7 9 1113151719212325272931333537394143454749515355
Device Network, CPU
CPU Total (%)#Device Network In - Total (Bytes)#Device

12. Let’sdivein
HAR file
• Objects, sizes, timing (redownload, compression,..)
• Servers and HTTP responses/redirects
• Parallelization
• Etc.
PCAP file
• Packet-level analysis
• Retransmissions
• ..

13. Let’sdivein
Google Lighthouse
• Responsive analysis
• Performance
• Film Strip
• Opportunities
• ..

14. Let’sdivein
Functional + responsiveness testing
 Use cases
 Platforms
X Real world conditions
What is the app responsiveness when
• You’re on 2G?
• Pandora is playing in the background?
• Chrome is checking Celtics-Cavaliers score?
• Battery at 8%?
• .. Multi-screen

16. Is load testingdead to me?
Combine
• Back-end load
• Service API measurement
• Front-end measurement
Backend(s) Content Delivery
Networks
Load
Measurement

17. Let’ssummarize
Complexity,
compliance
Time, $, Resources
Split: What can you
run earlier?
Does it offer insight & value?
Single User Performance
Testing
Embed into the agile
process
Add responsiveness to your load
tests
Track & Benchmark

18. @Amir_Rozenberg