This document provides tips for managing application performance, including: - Choosing the right metrics like time to interact and percentiles to understand performance. - Reasoning about telemetry data by examining time series, histograms, and cumulative distribution functions to identify issues. - Prioritizing the development of performance tools like profilers and predictive models. It also stresses the importance of building alignment through a common performance language, demonstrating impact, and cultivating a performance culture. Finally, it discusses defining a clear performance architecture strategy and team mission to guide work and set a north star.

1. Cheat Codes for Managing
Application Performance
Kwame Thomison
Magnetic Inc.

2. About Me
Meebo: Web based
instant messaging
Facebook: Social
network
Asana: Collaboration
software

3. What makes
managing app
performance so
challenging?

4. What makes managing app
performance so challenging?
It’s not most engineers’ first choice
No true north star for the team
Requires your very best engineers
Unique skillset
Perf work isn’t prioritized early enough

5. Risks of failing to address
those challenges
Business metrics suffer
Burnout
Wasted time
Endless cycle of improvement and regression

6. Agenda
Telemetry and Tooling
Building Alignment
North Star

7. Telemetry and Tooling
Metrics
Reasoning About Data
Building the Right Tools

8. Your architecture

9. What your API expert sees

10. What your product expert
sees

11. What your product expert
sees

12. Your architecture

13. Telemetry and
Tooling:
Metrics

14. Choosing the right metrics
Approximate User Value
Use Percentiles
Correlative Metrics

15. Two Popular Perf Metrics
Time to Interact (TTI)
End to End (E2E)

16. Choosing the right metrics
Approximate User Value
Use Percentiles
Correlative Metrics

17. Source: https://www.dynatrace.com/news/blog/why-averages-suck-and-percentiles-are-great/

18. Source: https://www.dynatrace.com/news/blog/why-averages-suck-and-percentiles-are-great/

19. Source: https://www.dynatrace.com/news/blog/why-averages-suck-and-percentiles-are-great/

20. Choosing the right metrics
Approximate User Value
Use Percentiles
Correlative Metrics

21. Telemetry and
Tooling:
Reasoning About
Data

22. A quick refresher on graphs

23. Time Series

24. Histogram

25. Cumulative Distribution Function

26. Time Series: Slow and
Steady Degradation
• Gradual increase over day/
weeks
• No corresponding increase
in system load
• No discernible pattern when
slicing by demographic
• Slow rollout of a new
service or feature

27. Time Series: The “blip”
• Regression that looks like
noise
• w/w shows a periodic
regression
• No correlation with app
usage
• Cold/warm start
mechanic

28. Histogram: Bimodal
distribution
• Histogram reveals
multiple modes
•No pattern when slicing
by demographic
•Indicates a relatively
expensive code path
being executed part of
the time

29. CDF: Extreme outliers
• The CDF levels off
sharply at p99
• Increases even more at
p99.9
• What’s broken?
• Big fish or celebrities

30. Telemetry and
Tooling: Building
the Right Tools

31. Building the Right Tools: Tool
Type
Static Analyzers
Profilers
Predictive Modelers

32. Building the Right Tools:
Develop Workflow
Example
• Dev in sandbox
• Run test suite
• Push to staging
• Push to prod

33. Building the Right Tools:
Render Pipeline
Example
• DB Query
• Server Processing
• Server Response
• Client CPU

34. 3 Tool Types x
4 Steps in Dev Workflow x
4 Steps in Rendering Pipeline x
= up to 48 tools
So what do you build first?

35. Prioritizing Tools
• Sort by time on the
critical path for rendering
pipeline
• Sort by stages of dev
workflow
• Sort by tool type
• Throw away moonshots

36. Building Alignment
A Common Language
Demonstrating Impact
Performance Culture

37. Building Alignment: A
Common Language
• Perf experts speak a
different language
• Others often uses
neologisms
• Loss of trust over time

38. Building Alignment: A
Common Language
What do you call this?
• JS file
• JS bundle
• Static asset

39. Building Alignment: A
Common Language
Primary concerns for perf
• Why has perf regressed?
• What impact does a
given change have?
• How is perf work
prioritized?

40. How do you get people
speaking the same language?

41. Define a perf grammar!

42. Example
{magnitude} {sequence}
{percentile} {stage of work}

43. -100ms p75 critical path client CPU

44. Building Alignment:
Demonstrating Your Impact
• Display measured TTI on
Every Page Load
• Share Experiments From
Other Companies
• AB Test Your
Optimizations
• Conduct Your Own Study

45. Conducting Your Own
Study: Bucketing Users
• Bucket sessions into perf classes (e.g. < 2s, > 2s)
• Examine business metrics for each perf class
• Infer causal relationship between perf and those
business metrics
• You may be misled into inferring a causal
relationship where one doesn’t exist

46. Conducting Your Own Study:
Artificial Perf Manipulation
• Introduce an artificial perf regression/improvement
• Measure differences between control and experiment
groups
• Infer causal relationship between perf and business metrics
• You’re hurting the UX intentionally
• You might infer a linear relationship where one
doesn’t exist

47. Conducting Your Own Study:
A Longterm Holdout Group
• For a portion of your users, don’t ship any optimizations
• Review business metrics after several weeks/months
• Infer causal relationship between perf and business
metrics
• Potentially very complex to implement

48. Building Alignment:
Cultivating a
Performance
Culture

49. What does a healthy culture of
managing app performance look like?

50. People have a shared definition for
and approach to achieving excellence

51. Cultivating a Performance
Culture
• Clear ownership
• Prioritization
• Tooling and
Documentation
• Levers

52. You’re out to build allies!

53. Unclear Priorities or
Lack of Education or
Lack of Tooling
= Adversaries

54. Shared Priorities and
Performance Education and
Data, Tools, and APIs
= Allies

55. The North
Star

56. North Star
• Architecture Strategy
• Team Mission

57. North Star: Architecture
Strategy
• What’s your perf cost per
incremental feature?
• Use that lens to guide
technology decisions
• Give engineers control
over rendering pipeline
• Pay or Parallelize

60. North Star: Architecture
Strategy
• What’s your perf cost per
incremental feature?
• Use that lens to guide
technology decisions
• Give engineers control
over rendering pipeline
• Pay or Parallelize

61. North Star: Team Mission
• Empower other
engineers
• Work on the hardest
problems
• Minimize perf cost per
incremental feature
• Focus on the big picture

62. Summing it all up
• Tips for Telemetry and Tooling
• Aligning the Company
• Defining a North Star

63. Let’s make the entire web faster!

64. Questions?
kwame@magnetic-inc.com