Web Performance Culture and Tools at Etsy

Web PerformanceCulture and Toolsat Etsy Mike Brittain @mikebrittain Director of Eng., Infrastructure Etsy

All of this started in the last year.

Make PerformanceMatter

$0.20 per item3.5% transaction fee

Focus on the marketplace

ecommerce

ecommercead supported

ecommercead supportedsubscription

ecommercead supportedsubscription“free”

“Prove it.”

“Our own years of testinghave conclusively shown that when speed of a feature or product improves, usage, quite simply, goes up.” http://googleblog.blogspot.com/2009/12/this-week-in-search-121809.html

100+ web, api, jobs40+ databasesMany more doing lots of other stuﬀ...

Performance is fun

I want you to tell me how you feel...

95th percentilemilliseconds Average

e w r! en ste h a 95th percentile w f is go is it h e T dmilliseconds m a Average

I am better than you.

Make PerformanceMatter

PerformanceCulture

Our perf team is two people...

Measure

MeasureBuild tools

MeasureBuild toolsTrain & Coordinate

MeasureBuild toolsTrain & CoordinateMotivate

MeasureBuild toolsTrain & CoordinateMotivateDedicated

Our perf team is two people... But every engineer thinks about performance

Everything is slow.Please ﬁx.

page generationtime

2011 Goal800 ms on ﬁve pages

http://www.etsy.com/blog/news/2011/november-2011-site-performance-report/

Performance degradationis a P2 bug

WARNING: This part is hardPerformance degradationis a P2 bug

2012 Goal800 ms for all pages

EngineeringDenormalize dataOptimize queriesRewrite from scratchCaching (memcache, app code)

OperationsNetwork switchesOperating systemsNewer hardwareAdditional hardware capacity

ProductIdentify concerns earlyKill lagging features

PerformanceCulture

How do you makethe graphs?

Perf-related data belongs in your server logs

apache_note(“custom_ﬁeld”, $whatever);LogFormat %l %t "%r" %>s %b "%{Referer}i""%{User-Agent}i" %{custom_field}n

LogFormat "%{True-Client-IP}i %l %u %t "%r"%>s %b "%{Referer}i" "%{User-Agent}i"%{etsy_user_id}n %{php_bytes}n %{php_microsec}n%D" combined

$timer_start = microtime(true); ...$timer_diff = microtime(true) - $timer_start;register_shutdown_function()apache_note(php_microsec, $timer_diff);

web0060 66.249.71.110 - - [24/Aug/2011:04:16:52+0000] "GET /listing/12189259/tropical-etched-pair-of-lampwork-glass HTTP/1.1" 200 11034 "-""Mozilla/5.0 (compatible; Googlebot/2.1;+http://www.google.com/bot.html)"- 13399576 505780 554876

Using something else?time, http method, request uri, response code, referer,user-agent, response time, response memory,custom segmentation ﬁelds...

Quick averagesgrep "GET /listing/" access.log | awk {sum=sum+$(NF-1)} END {print sum/NR}

Quick graphsgrep "GET /listing/" access.log | perl -pe "s/.*[.*d{4}:(d{2}):(d{2}):d{2}.*]/1:2/" | awk {print $1, $(NF-1)} > /tmp/pagetimes.dat gives you...

Quick graphs# /tmp/pagetimes.dat18:37 251.018:38 252.118:39 253.518:40 251.018:45 250.0 and then...

Quick graphs# GNUPLOTset terminal pngset output listings.pngset yrange [0:2000]set xdata timeset timefmt "%d/%B/%Y:%H:%M:%S"set format x "%H:%M"plot /tmp/pagetimes.dat using 1:2 with points

Quick graphs

LogsterRun by cronMaintains a cursor on log ﬁlesSimple parsing & aggregationOutput to Ganglia or Graphite github.com/etsy

Logster & Graphite Signed-in vs. Signed-out

Response timesresponses (pages) by bucket time (ms)

“Look at your Data”John Rauser, Velocity 2011 http://www.youtube.com/watch?v=coNDCIMH8bk

Idea lifted fromhttp://www.codinghorror.com/blog/2011/06/performance-is-a-feature.html

Don’t forget the API

Measureperformancedata in Production

Proﬁling

“I/O Proﬁler”Lightweight, inline proﬁlingStart and end times wrapped around service calls databases, memcache, apc, 3rd party services, etc.Enable with query string

“I/O Proﬁler”

XHProfgithub.com/facebook/xhprofDeployed on pre-prod machineDirect access to proﬁlingEnable with query string

Proﬁle againstProduction

Make performance matter

Start small

Make graphs and reports

Create simple tools

Celebrate performance wins

Thank you codeascraft.etsy.comMike Brittainmike@etsy.com@mikebrittain

Jobs at Etsy We are currently hiring for a number of positions in engineering and operations. http://www.etsy.com/careers

http://www.mikebrittain.com	2729
http://dow.ngra.de	1857
http://blog.livedoor.jp	422
http://www.hazrac.org	331
http://localhost	61
http://hazrac.squarespace.com	39
http://a0.twimg.com	35
http://confluence.energy.com.au	31
http://10.64.136.239	20
http://www.ckvinothkumar.com	14
http://ngra.de	14
http://tweetedtimes.com	11
http://planet.cs.ut.ee	10
http://ckvinothkumar.com	10
http://www.linkedin.com	7
http://ec2-79-125-4-101.eu-west-1.compute.amazonaws.com	6
http://us-w1.rockmelt.com	5
http://translate.googleusercontent.com	4
http://lanyrd.com	4
http://parwb06lx	4
http://www.twylah.com	4
http://coderwall.com	4
http://nuevospowerpoints.blogspot.com	3
http://dcxwiki.energy.com.au	3
http://www.hanrss.com	3
http://e-learning.spanish-classes.org	3
https://si0.twimg.com	2
http://ats.cs.ut.ee	2
http://deschleib.tumblr.com	2
http://safe.tumblr.com	1
http://mohamed-0826.csez.zohocorpin.com	1
http://www.alrahmatech.com	1
http://webcache.googleusercontent.com	1
http://mikebrittain.com.	1
http://webcache-exp-test.googleusercontent.com	1

Web Performance Culture and Tools at Etsy

by Mike Brittain

Accessibility

Categories

Upload Details

Usage Rights

35 Embeds 5,646

Statistics

Web Performance Culture and Tools at Etsy Presentation Transcript