Etsy's performance culture is led by a team of two that focuses on measuring performance data, building tools to analyze logs and servers, training engineers, and motivating teams. They use tools like Logster to aggregate Apache logs into Graphite for generating graphs of metrics like response times. Profiling tools help identify slow database queries, services, and third party calls. The goal is to prove that improved speed leads to more usage and make all engineers consider performance in their work.

1. Web Performance
Culture and Tools
at Etsy
Mike Brittain
@mikebrittain
Director of Eng., Infrastructure
Etsy

2. All of this started in the last year.

4. Make Performance
Matter

5. $0.20 per item
3.5% transaction fee

6. Focus on the marketplace

7. ecommerce

8. ecommerce
ad supported

9. ecommerce
ad supported
subscription

10. ecommerce
ad supported
subscription
“free”

11. $$$

12. “Prove it.”

13. “Our own years of testing
have conclusively shown that
when speed of a feature or
product improves, usage,
quite simply, goes up.”
http://googleblog.blogspot.com/2009/12/this-week-in-search-121809.html

14. 100+ web, api, jobs
40+ databases
Many more doing lots of other stuff...

15. $$$

16. Performance is fun

17. I want you to tell me
how you feel...

18. 95th percentile
milliseconds
Average

19. e
w r!
en ste
h a
95th percentile w f
is go
is it
h e
T d
milliseconds
m a
Average

20. I am better than you.

21. Make Performance
Matter

22. Performance
Culture

23. Our perf team is two people...

24. Measure

25. Measure
Build tools

26. Measure
Build tools
Train & Coordinate

27. Measure
Build tools
Train & Coordinate
Motivate

28. Measure
Build tools
Train & Coordinate
Motivate
Dedicated

29. Our perf team is two people...
But every engineer thinks
about performance

30. Everything is slow.
Please fix.

33. page generation
time

34. 2011 Goal
800 ms on five pages

39. http://www.etsy.com/blog/news/2011/november-2011-site-performance-report/

40. http://www.etsy.com/blog/news/2011/november-2011-site-performance-report/

41. Performance degradation
is a P2 bug

42. WARNING: This part is hard
Performance degradation
is a P2 bug

43. 2012 Goal
800 ms for all pages

44. Engineering
Denormalize data
Optimize queries
Rewrite from scratch
Caching (memcache, app code)

45. Operations
Network switches
Operating systems
Newer hardware
Additional hardware capacity

46. Product
Identify concerns early
Kill lagging features

48. Performance
Culture

49. Tools

50. How do you make
the graphs?

51. Perf-related data belongs in
your server logs

52. apache_note(“custom_field”, $whatever);
LogFormat %l %t "%r" %>s %b "%{Referer}i"
"%{User-Agent}i" %{custom_field}n

53. LogFormat "%{True-Client-IP}i %l %u %t "%r"
%>s %b "%{Referer}i" "%{User-Agent}i"
%{etsy_user_id}n %{php_bytes}n %{php_microsec}n
%D" combined

54. $timer_start = microtime(true);
...
$timer_diff = microtime(true) - $timer_start;
register_shutdown_function()
apache_note('php_microsec', $timer_diff);

55. web0060 66.249.71.110 - - [24/Aug/2011:04:16:52
+0000] "GET /listing/12189259/tropical-etched-
pair-of-lampwork-glass HTTP/1.1" 200 11034 "-"
"Mozilla/5.0 (compatible; Googlebot/2.1;
+http://www.google.com/bot.html)"
- 13399576 505780 554876

56. Using something else?
time, http method, request uri, response code, referer,
user-agent, response time, response memory,
custom segmentation fields...

57. Quick averages
grep "GET /listing/" access.log |
awk '{sum=sum+$(NF-1)} END {print sum/NR}'

58. Quick graphs
grep "GET /listing/" access.log |
perl -pe "s/.*[.*d{4}:(d{2}):(d{2}):d{2}.*]/1:2/" |
awk '{print $1, $(NF-1)}' > /tmp/pagetimes.dat
gives you...

59. Quick graphs
# /tmp/pagetimes.dat
18:37 251.0
18:38 252.1
18:39 253.5
18:40 251.0
18:45 250.0 and then...

60. Quick graphs
# GNUPLOT
set terminal png
set output 'listings.png'
set yrange [0:2000]
set xdata time
set timefmt "%d/%B/%Y:%H:%M:%S"
set format x "%H:%M"
plot '/tmp/pagetimes.dat' using 1:2 with points

61. Quick graphs

62. Logster
Run by cron
Maintains a cursor on log files
Simple parsing & aggregation
Output to Ganglia or Graphite
github.com/etsy

63. Logster & Graphite
Signed-in vs. Signed-out

64. Response times
responses (pages)
by bucket
time (ms)

65. “Look at your Data”
John Rauser, Velocity 2011
http://www.youtube.com/watch?v=coNDCIMH8bk

67. Idea lifted from
http://www.codinghorror.com/blog/2011/06/
performance-is-a-feature.html

68. Don’t forget the API

69. Measure
performance
data in Production

70. Profiling

71. “I/O Profiler”
Lightweight, inline profiling
Start and end times wrapped around service calls
databases, memcache, apc, 3rd party services, etc.
Enable with query string

72. “I/O Profiler”

73. “I/O Profiler”

74. XHProf
github.com/facebook/xhprof
Deployed on pre-prod machine
Direct access to profiling
Enable with query string

76. Profile against
Production

77. Make performance matter

78. Start small

79. Make graphs and reports

80. Create simple tools

81. Celebrate performance wins

82. Thank you
codeascraft.etsy.com
Mike Brittain
mike@etsy.com
@mikebrittain

83. Jobs at Etsy
We are currently hiring for a number of
positions in engineering and operations.
http://www.etsy.com/careers