This document discusses analyzing log data from Etsy to monitor page generation times and trends over time. A graph is shown displaying typical page generation times between 0-600ms with most times falling between 150-450ms. Monitoring this metric helps ensure site performance remains high.

1. Simple Log Analysis and Trending
Mike Brittain
DIRECTOR OF ENGINEERING, ETSY
@mikebrittain
EXAMPLES: bit.ly/velocity-etsy

2. sa=t&source=web&cd=18&ved=0CHkQFjAR&url=http%3A%2F%2Fwww.etsy.com%2Flisting%2F45184569%2Fchanel-vintage-navy-blue-cotton-dress-w&rct=j&q=vintage%20chanel
%20dress&ei=fMLKTZH0MYTL0AGjpsCDCg&usg=AFQjCNHnkswNxqzqi-Rl6nLVIr7dTg78Rg" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; WOW64; Trident/4.0; GTB7.0; SLCC2; .NET CLR 2.0.50727; .
3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; MDDR; .NET4.0C)" - ur9XOgyn58I8opUn7LDhyqPliXap www.etsy.com ab_new_header=1|ab_db_log_long_queries=1|ab_rawsql_close_statement=1|
ab_bank_transfer=0|ab_listing_favorite_nag=none|ab_shop_chat=0|ab_share2=1|ab_header_css=0|ab_inline_listing_hearts=1|ab_search_shops=1|ab_community_nav_popout=1|ab_boomerang=0|
ab_shopstats_gathering=0 te0sXDV0SfwBpU-qL0w0TnAUIw-x 0 - - 11567688 601120 637356
web0037 1.22.333.444 - - [11/May/2011:17:08:53 +0000] "GET /listing/60547985/half-yard-little-girls-in-kimono-print HTTP/1.1" 200 12389 "http://www.etsy.com/shop/beautifulwork?
section_id=5810576&page=2" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; WOW64; Trident/4.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6
- LVTY9r_Z_IEcEbfKWUJoPTAGHhBx www.etsy.com ab_new_header=1|ab_db_log_long_queries=1|ab_rawsql_close_statement=1|ab_bank_transfer=0|ab_listing_favorite_nag=none|ab_shop_chat=0|ab_shar
ab_header_css=0|ab_inline_listing_hearts=1|ab_search_defaults_allitems=1|ab_search_shops=1|ab_community_nav_popout=1|ab_boomerang=0|ab_shopstats_gathering=0 i57VnazHMo2LEjii5IRjaiaWVO
412157 437768
web0039 1.22.333.444 - - [11/May/2011:17:08:53 +0000] "GET /listing/71705428/fearfully-and-wonderfully-made-8x10?ref=sr_list_10&ga_search_query=pink+and
+green&ga_noautofacet=1&ga_search_type=handmade&ga_facet=handmade%2Fart HTTP/1.1" 200 13232 "http://www.etsy.com/search/handmade/art?q=pink+and+green&noautofacet=1" "Mozilla/5.0 (Wind
AppleWebKit/534.24 (KHTML, like Gecko) Chrome/11.0.696.65 Safari/534.24" - aKR4B5QW0KrN-NBZpWvm0AgkwcD3 www.etsy.com ab_new_header=1|ab_db_log_long_queries=1|ab_rawsql_close_statement
ab_bank_transfer=0|ab_listing_favorite_nag=none|ab_shop_chat=0|ab_share2=1|ab_header_css=0|ab_inline_listing_hearts=1|ab_search_shops=1|ab_community_nav_popout=1|ab_shopstats_gatherin
1mFgWOMRaRG5RN1zcfNZRVfdaDz7 7323622 - - 15124584 662683 690754
web0047 1.22.333.444 - - [11/May/2011:17:08:53 +0000] "GET /listing/31224920/flirt-14k-gold-filled-hook-earrings?ref=sr_gallery_34&ga_noautofacet=1&ga_search_type=handmade&ga_facet=ha
%2Fearrings%2Fpost&ga_search_query= HTTP/1.1" 200 12339 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" - Y8PI3hOiKDJWNz7M_-T_t4mq75zM www.etsy.com ab_n
ab_db_log_long_queries=1|ab_rawsql_close_statement=1|ab_bank_transfer=0|ab_listing_favorite_nag=none|ab_shop_chat=0|ab_share2=1|ab_header_css=0|ab_inline_listing_hearts=1|
ab_search_defaults_allitems=0|ab_search_shops=1|ab_community_nav_popout=1|ab_boomerang=0|ab_shopstats_gathering=0 L6iJ-8gaox1rUhd8F8CVbfhLrMnJ 0 - - 10813080 377090 402038
web0020 1.22.333.444 - - [11/May/2011:17:08:54 +0000] "GET /listing/72395693/sterling-silver-ring-with-rainbow HTTP/1.1" 200 11893 "-" "facebookexternalhit/1.1 (+http://www.facebook.c
externalhit_uatext.php)" - phQhH2hNgWf3AfCQH0SpLqoFL6I1 www.etsy.com ab_new_header=1|ab_db_log_long_queries=1|ab_rawsql_close_statement=1|ab_bank_transfer=0|ab_listing_favorite_nag=no
A
ab_shop_chat=0|ab_share2=1|ab_header_css=0|ab_inline_listing_hearts=1|ab_search_defaults_allitems=0|ab_search_shops=1|ab_community_nav_popout=1|ab_boomerang=0|ab_shopstats_gathering=0
Vs1ZXSz3z9fZ 0 - - 10765744 320118 339176
web0031 1.22.333.444 - - [11/May/2011:17:08:53 +0000] "GET /listing/64397692/spades-and-diamonds-ooak-red-on-custom?ref=v1_other_2 HTTP/1.1" 200 13835 "http://www.etsy.com/listing/725
dots-black-on-white-tank-womens-l?ref=v1_other_2" "Mozilla/5.0 (Macintosh; U; PPC Mac OS X 10.4; en-US; rv:1.9.2.17) Gecko/20110420 Firefox/3.6.17" - vu5b_1x-w87dTJ-NEVYyISdHo0Qf www.
ab_new_header=1|ab_db_log_long_queries=1|ab_rawsql_close_statement=1|ab_bank_transfer=0|ab_listing_favorite_nag=none|ab_shop_chat=0|ab_share2=1|ab_header_css=0|ab_inline_listing_heart
ab_search_shops=1|ab_community_nav_popout=1|ab_your_account_nav_popout=1|ab_new_listing_process=0|ab_boomerang=0|ab_shopstats_gathering=0 eNFgDvlRp9qHk8BYw1skM_efLpik 10551287 - - 152
586038
web0041 1.22.333.444 - - [11/May/2011:17:08:53 +0000] "GET /listing/70834087/custom-engraved-vintage-spoon-bracelet?
ref=sr_gallery_16&ga_search_query=wedding&ga_page=20&ga_search_type=handmade&ga_facet=handmade HTTP/1.1" 200 14874 "http://www.etsy.com/search/handmade?q=wedding&page=20" "Mozilla/5.0
Intel Mac OS X 10_6_7; en-us) AppleWebKit/533.20.25 (KHTML, like Gecko) Version/5.0.4 Safari/533.20.27" - s3dAEruW9OGxqsGII6dZFpVMdOo6 www.etsy.com ab_new_header=1|ab_db_log_long_quer
ab_rawsql_close_statement=1|ab_bank_transfer=0|ab_listing_favorite_nag=none|ab_shop_chat=0|ab_share2=1|ab_header_css=0|ab_inline_listing_hearts=1|ab_search_shops=1|ab_community_nav_po
ab_your_account_nav_popout=1|ab_boomerang=0|ab_shopstats_gathering=0 SHyrUYHqVMKD5KER-cJtiSR6mz9i 14458253 - - 12412104 391399 407669
web0057 1.22.333.444 - - [11/May/2011:17:08:53 +0000] "GET /listing/45184569/chanel-vintage-navy-blue-cotton-dress-w HTTP/1.1" 200 12071 "http://www.google.com/url?
sa=t&source=web&cd=18&ved=0CHkQFjAR&url=http%3A%2F%2Fwww.etsy.com%2Flisting%2F45184569%2Fchanel-vintage-navy-blue-cotton-dress-w&rct=j&q=vintage%20chanel
%20dress&ei=fMLKTZH0MYTL0AGjpsCDCg&usg=AFQjCNHnkswNxqzqi-Rl6nLVIr7dTg78Rg" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; WOW64; Trident/4.0; GTB7.0; SLCC2; .NET CLR 2.0.50727; .
3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; MDDR; .NET4.0C)" - ur9XOgyn58I8opUn7LDhyqPliXap www.etsy.com ab_new_header=1|ab_db_log_long_queries=1|ab_rawsql_close_statement=1|
ab_bank_transfer=0|ab_listing_favorite_nag=none|ab_shop_chat=0|ab_share2=1|ab_header_css=0|ab_inline_listing_hearts=1|ab_search_shops=1|ab_community_nav_popout=1|ab_boomerang=0|
ab_shopstats_gathering=0 te0sXDV0SfwBpU-qL0w0TnAUIw-x 0 - - 11567688 601120 637356
web0037 1.22.333.444 - - [11/May/2011:17:08:53 +0000] "GET /listing/60547985/half-yard-little-girls-in-kimono-print HTTP/1.1" 200 12389 "http://www.etsy.com/shop/beautifulwork?
section_id=5810576&page=2" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; WOW64; Trident/4.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6
- LVTY9r_Z_IEcEbfKWUJoPTAGHhBx www.etsy.com ab_new_header=1|ab_db_log_long_queries=1|ab_rawsql_close_statement=1|ab_bank_transfer=0|ab_listing_favorite_nag=none|ab_shop_chat=0|ab_shar
ab_header_css=0|ab_inline_listing_hearts=1|ab_search_defaults_allitems=1|ab_search_shops=1|ab_community_nav_popout=1|ab_boomerang=0|ab_shopstats_gathering=0 i57VnazHMo2LEjii5IRjaiaWVO
412157 437768
web0039 1.22.333.444 - - [11/May/2011:17:08:53 +0000] "GET /listing/71705428/fearfully-and-wonderfully-made-8x10?ref=sr_list_10&ga_search_query=pink+and
+green&ga_noautofacet=1&ga_search_type=handmade&ga_facet=handmade%2Fart HTTP/1.1" 200 13232 "http://www.etsy.com/search/handmade/art?q=pink+and+green&noautofacet=1" "Mozilla/5.0 (Wind
AppleWebKit/534.24 (KHTML, like Gecko) Chrome/11.0.696.65 Safari/534.24" - aKR4B5QW0KrN-NBZpWvm0AgkwcD3 www.etsy.com ab_new_header=1|ab_db_log_long_queries=1|ab_rawsql_close_statement
ab_bank_transfer=0|ab_listing_favorite_nag=none|ab_shop_chat=0|ab_share2=1|ab_header_css=0|ab_inline_listing_hearts=1|ab_search_shops=1|ab_community_nav_popout=1|ab_shopstats_gatherin

3. PAGE GENERATION TIME (MS)
600
450
300
B
150
0

4. sa=t&source=web&cd=18&ved=0CHkQFjAR&url=http%3A%2F%2Fwww.etsy.com%2Flisting%2F45184569%2Fchanel-vintage-navy-blue-cotton-dress-w&rct=j&q=vintage%20chanel
%20dress&ei=fMLKTZH0MYTL0AGjpsCDCg&usg=AFQjCNHnkswNxqzqi-Rl6nLVIr7dTg78Rg" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; WOW64; Trident/4.0; GTB7.0; SLCC2; .NET CLR 2.0.50727; .
3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; MDDR; .NET4.0C)" - ur9XOgyn58I8opUn7LDhyqPliXap www.etsy.com ab_new_header=1|ab_db_log_long_queries=1|ab_rawsql_close_statement=1|
ab_bank_transfer=0|ab_listing_favorite_nag=none|ab_shop_chat=0|ab_share2=1|ab_header_css=0|ab_inline_listing_hearts=1|ab_search_shops=1|ab_community_nav_popout=1|ab_boomerang=0|
ab_shopstats_gathering=0 te0sXDV0SfwBpU-qL0w0TnAUIw-x 0 - - 11567688 601120 637356
web0037 1.22.333.444 - - [11/May/2011:17:08:53 +0000] "GET /listing/60547985/half-yard-little-girls-in-kimono-print HTTP/1.1" 200 12389 "http://www.etsy.com/shop/beautifulwork?
section_id=5810576&page=2" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; WOW64; Trident/4.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6
- LVTY9r_Z_IEcEbfKWUJoPTAGHhBx www.etsy.com ab_new_header=1|ab_db_log_long_queries=1|ab_rawsql_close_statement=1|ab_bank_transfer=0|ab_listing_favorite_nag=none|ab_shop_chat=0|ab_shar
ab_header_css=0|ab_inline_listing_hearts=1|ab_search_defaults_allitems=1|ab_search_shops=1|ab_community_nav_popout=1|ab_boomerang=0|ab_shopstats_gathering=0 i57VnazHMo2LEjii5IRjaiaWVO
412157 437768
web0039 1.22.333.444 - - [11/May/2011:17:08:53 +0000] "GET /listing/71705428/fearfully-and-wonderfully-made-8x10?ref=sr_list_10&ga_search_query=pink+and
+green&ga_noautofacet=1&ga_search_type=handmade&ga_facet=handmade%2Fart HTTP/1.1" 200 13232 "http://www.etsy.com/search/handmade/art?q=pink+and+green&noautofacet=1" "Mozilla/5.0 (Wind
AppleWebKit/534.24 (KHTML, like Gecko) Chrome/11.0.696.65 Safari/534.24" - aKR4B5QW0KrN-NBZpWvm0AgkwcD3 www.etsy.com ab_new_header=1|ab_db_log_long_queries=1|ab_rawsql_close_statement
ab_bank_transfer=0|ab_listing_favorite_nag=none|ab_shop_chat=0|ab_share2=1|ab_header_css=0|ab_inline_listing_hearts=1|ab_search_shops=1|ab_community_nav_popout=1|ab_shopstats_gatherin
1mFgWOMRaRG5RN1zcfNZRVfdaDz7 7323622 - - 15124584 662683 690754
web0047 1.22.333.444 - - [11/May/2011:17:08:53 +0000] "GET /listing/31224920/flirt-14k-gold-filled-hook-earrings?ref=sr_gallery_34&ga_noautofacet=1&ga_search_type=handmade&ga_facet=ha
%2Fearrings%2Fpost&ga_search_query= HTTP/1.1" 200 12339 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" - Y8PI3hOiKDJWNz7M_-T_t4mq75zM www.etsy.com ab_n
ab_db_log_long_queries=1|ab_rawsql_close_statement=1|ab_bank_transfer=0|ab_listing_favorite_nag=none|ab_shop_chat=0|ab_share2=1|ab_header_css=0|ab_inline_listing_hearts=1|
ab_search_defaults_allitems=0|ab_search_shops=1|ab_community_nav_popout=1|ab_boomerang=0|ab_shopstats_gathering=0 L6iJ-8gaox1rUhd8F8CVbfhLrMnJ 0 - - 10813080 377090 402038
web0020 1.22.333.444 - - [11/May/2011:17:08:54 +0000] "GET /listing/72395693/sterling-silver-ring-with-rainbow HTTP/1.1" 200 11893 "-" "facebookexternalhit/1.1 (+http://www.facebook.c
externalhit_uatext.php)" - phQhH2hNgWf3AfCQH0SpLqoFL6I1 www.etsy.com ab_new_header=1|ab_db_log_long_queries=1|ab_rawsql_close_statement=1|ab_bank_transfer=0|ab_listing_favorite_nag=no
A
ab_shop_chat=0|ab_share2=1|ab_header_css=0|ab_inline_listing_hearts=1|ab_search_defaults_allitems=0|ab_search_shops=1|ab_community_nav_popout=1|ab_boomerang=0|ab_shopstats_gathering=0
Vs1ZXSz3z9fZ 0 - - 10765744 320118 339176
web0031 1.22.333.444 - - [11/May/2011:17:08:53 +0000] "GET /listing/64397692/spades-and-diamonds-ooak-red-on-custom?ref=v1_other_2 HTTP/1.1" 200 13835 "http://www.etsy.com/listing/725
dots-black-on-white-tank-womens-l?ref=v1_other_2" "Mozilla/5.0 (Macintosh; U; PPC Mac OS X 10.4; en-US; rv:1.9.2.17) Gecko/20110420 Firefox/3.6.17" - vu5b_1x-w87dTJ-NEVYyISdHo0Qf www.
ab_new_header=1|ab_db_log_long_queries=1|ab_rawsql_close_statement=1|ab_bank_transfer=0|ab_listing_favorite_nag=none|ab_shop_chat=0|ab_share2=1|ab_header_css=0|ab_inline_listing_heart
ab_search_shops=1|ab_community_nav_popout=1|ab_your_account_nav_popout=1|ab_new_listing_process=0|ab_boomerang=0|ab_shopstats_gathering=0 eNFgDvlRp9qHk8BYw1skM_efLpik 10551287 - - 152
586038
web0041 1.22.333.444 - - [11/May/2011:17:08:53 +0000] "GET /listing/70834087/custom-engraved-vintage-spoon-bracelet?
ref=sr_gallery_16&ga_search_query=wedding&ga_page=20&ga_search_type=handmade&ga_facet=handmade HTTP/1.1" 200 14874 "http://www.etsy.com/search/handmade?q=wedding&page=20" "Mozilla/5.0
Intel Mac OS X 10_6_7; en-us) AppleWebKit/533.20.25 (KHTML, like Gecko) Version/5.0.4 Safari/533.20.27" - s3dAEruW9OGxqsGII6dZFpVMdOo6 www.etsy.com ab_new_header=1|ab_db_log_long_quer
ab_rawsql_close_statement=1|ab_bank_transfer=0|ab_listing_favorite_nag=none|ab_shop_chat=0|ab_share2=1|ab_header_css=0|ab_inline_listing_hearts=1|ab_search_shops=1|ab_community_nav_po
ab_your_account_nav_popout=1|ab_boomerang=0|ab_shopstats_gathering=0 SHyrUYHqVMKD5KER-cJtiSR6mz9i 14458253 - - 12412104 391399 407669
web0057 1.22.333.444 - - [11/May/2011:17:08:53 +0000] "GET /listing/45184569/chanel-vintage-navy-blue-cotton-dress-w HTTP/1.1" 200 12071 "http://www.google.com/url?
sa=t&source=web&cd=18&ved=0CHkQFjAR&url=http%3A%2F%2Fwww.etsy.com%2Flisting%2F45184569%2Fchanel-vintage-navy-blue-cotton-dress-w&rct=j&q=vintage%20chanel
%20dress&ei=fMLKTZH0MYTL0AGjpsCDCg&usg=AFQjCNHnkswNxqzqi-Rl6nLVIr7dTg78Rg" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; WOW64; Trident/4.0; GTB7.0; SLCC2; .NET CLR 2.0.50727; .
3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; MDDR; .NET4.0C)" - ur9XOgyn58I8opUn7LDhyqPliXap www.etsy.com ab_new_header=1|ab_db_log_long_queries=1|ab_rawsql_close_statement=1|
ab_bank_transfer=0|ab_listing_favorite_nag=none|ab_shop_chat=0|ab_share2=1|ab_header_css=0|ab_inline_listing_hearts=1|ab_search_shops=1|ab_community_nav_popout=1|ab_boomerang=0|
ab_shopstats_gathering=0 te0sXDV0SfwBpU-qL0w0TnAUIw-x 0 - - 11567688 601120 637356
web0037 1.22.333.444 - - [11/May/2011:17:08:53 +0000] "GET /listing/60547985/half-yard-little-girls-in-kimono-print HTTP/1.1" 200 12389 "http://www.etsy.com/shop/beautifulwork?
section_id=5810576&page=2" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; WOW64; Trident/4.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6
- LVTY9r_Z_IEcEbfKWUJoPTAGHhBx www.etsy.com ab_new_header=1|ab_db_log_long_queries=1|ab_rawsql_close_statement=1|ab_bank_transfer=0|ab_listing_favorite_nag=none|ab_shop_chat=0|ab_shar
ab_header_css=0|ab_inline_listing_hearts=1|ab_search_defaults_allitems=1|ab_search_shops=1|ab_community_nav_popout=1|ab_boomerang=0|ab_shopstats_gathering=0 i57VnazHMo2LEjii5IRjaiaWVO
412157 437768
web0039 1.22.333.444 - - [11/May/2011:17:08:53 +0000] "GET /listing/71705428/fearfully-and-wonderfully-made-8x10?ref=sr_list_10&ga_search_query=pink+and
+green&ga_noautofacet=1&ga_search_type=handmade&ga_facet=handmade%2Fart HTTP/1.1" 200 13232 "http://www.etsy.com/search/handmade/art?q=pink+and+green&noautofacet=1" "Mozilla/5.0 (Wind
AppleWebKit/534.24 (KHTML, like Gecko) Chrome/11.0.696.65 Safari/534.24" - aKR4B5QW0KrN-NBZpWvm0AgkwcD3 www.etsy.com ab_new_header=1|ab_db_log_long_queries=1|ab_rawsql_close_statement
ab_bank_transfer=0|ab_listing_favorite_nag=none|ab_shop_chat=0|ab_share2=1|ab_header_css=0|ab_inline_listing_hearts=1|ab_search_shops=1|ab_community_nav_popout=1|ab_shopstats_gatherin

5. PAGE GENERATION TIME (MS)
600
450
300
B
150
0

6. A B
web0057 1.22.333.444 - - [11/May/2011:17:08:53 +0000] "GET /listing/45184569/chanel-vintage-navy-blue-cotton-dress-w HTTP/1.1" 200 12071 "http://www.google.com/url?
sa=t&source=web&cd=18&ved=0CHkQFjAR&url=http%3A%2F%2Fwww.etsy.com%2Flisting%2F45184569%2Fchanel-vintage-navy-blue-cotton-dress-w&rct=j&q=vintage%20chanel
%20dress&ei=fMLKTZH0MYTL0AGjpsCDCg&usg=AFQjCNHnkswNxqzqi-Rl6nLVIr7dTg78Rg" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; WOW64; Trident/4.0; GTB7.0; SLCC2; .NET CLR 2.0.
3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; MDDR; .NET4.0C)" - ur9XOgyn58I8opUn7LDhyqPliXap www.etsy.com ab_new_header=1|ab_db_log_long_queries=1|ab_rawsql_close_state
ab_bank_transfer=0|ab_listing_favorite_nag=none|ab_shop_chat=0|ab_share2=1|ab_header_css=0|ab_inline_listing_hearts=1|ab_search_shops=1|ab_community_nav_popout=1|ab_boomerang=
ab_shopstats_gathering=0 te0sXDV0SfwBpU-qL0w0TnAUIw-x 0 - - 11567688 601120 637356
web0037 1.22.333.444 - - [11/May/2011:17:08:53 +0000] "GET /listing/60547985/half-yard-little-girls-in-kimono-print HTTP/1.1" 200 12389 "http://www.etsy.com/shop/beautifulwork
section_id=5810576&page=2" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; WOW64; Trident/4.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Cen
- LVTY9r_Z_IEcEbfKWUJoPTAGHhBx www.etsy.com ab_new_header=1|ab_db_log_long_queries=1|ab_rawsql_close_statement=1|ab_bank_transfer=0|ab_listing_favorite_nag=none|ab_shop_chat=0
ab_header_css=0|ab_inline_listing_hearts=1|ab_search_defaults_allitems=1|ab_search_shops=1|ab_community_nav_popout=1|ab_boomerang=0|ab_shopstats_gathering=0 i57VnazHMo2LEjii5I
412157 437768
web0039 1.22.333.444 - - [11/May/2011:17:08:53 +0000] "GET /listing/71705428/fearfully-and-wonderfully-made-8x10?ref=sr_list_10&ga_search_query=pink+and
+green&ga_noautofacet=1&ga_search_type=handmade&ga_facet=handmade%2Fart HTTP/1.1" 200 13232 "http://www.etsy.com/search/handmade/art?q=pink+and+green&noautofacet=1" "Mozilla/5
AppleWebKit/534.24 (KHTML, like Gecko) Chrome/11.0.696.65 Safari/534.24" - aKR4B5QW0KrN-NBZpWvm0AgkwcD3 www.etsy.com ab_new_header=1|ab_db_log_long_queries=1|ab_rawsql_close_s
ab_bank_transfer=0|ab_listing_favorite_nag=none|ab_shop_chat=0|ab_share2=1|ab_header_css=0|ab_inline_listing_hearts=1|ab_search_shops=1|ab_community_nav_popout=1|ab_shopstats_
1mFgWOMRaRG5RN1zcfNZRVfdaDz7 7323622 - - 15124584 662683 690754
web0047 1.22.333.444 - - [11/May/2011:17:08:53 +0000] "GET /listing/31224920/flirt-14k-gold-filled-hook-earrings?ref=sr_gallery_34&ga_noautofacet=1&ga_search_type=handmade&ga_
%2Fearrings%2Fpost&ga_search_query= HTTP/1.1" 200 12339 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" - Y8PI3hOiKDJWNz7M_-T_t4mq75zM www.etsy.
ab_db_log_long_queries=1|ab_rawsql_close_statement=1|ab_bank_transfer=0|ab_listing_favorite_nag=none|ab_shop_chat=0|ab_share2=1|ab_header_css=0|ab_inline_listing_hearts=1|
ab_search_defaults_allitems=0|ab_search_shops=1|ab_community_nav_popout=1|ab_boomerang=0|ab_shopstats_gathering=0 L6iJ-8gaox1rUhd8F8CVbfhLrMnJ 0 - - 10813080 377090 402038
web0020 1.22.333.444 - - [11/May/2011:17:08:54 +0000] "GET /listing/72395693/sterling-silver-ring-with-rainbow HTTP/1.1" 200 11893 "-" "facebookexternalhit/1.1 (+http://www.fa
externalhit_uatext.php)" - phQhH2hNgWf3AfCQH0SpLqoFL6I1 www.etsy.com ab_new_header=1|ab_db_log_long_queries=1|ab_rawsql_close_statement=1|ab_bank_transfer=0|ab_listing_favorit
ab_shop_chat=0|ab_share2=1|ab_header_css=0|ab_inline_listing_hearts=1|ab_search_defaults_allitems=0|ab_search_shops=1|ab_community_nav_popout=1|ab_boomerang=0|ab_shopstats_gat
Vs1ZXSz3z9fZ 0 - - 10765744 320118 339176
web0031 1.22.333.444 - - [11/May/2011:17:08:53 +0000] "GET /listing/64397692/spades-and-diamonds-ooak-red-on-custom?ref=v1_other_2 HTTP/1.1" 200 13835 "http://www.etsy.com/lis
dots-black-on-white-tank-womens-l?ref=v1_other_2" "Mozilla/5.0 (Macintosh; U; PPC Mac OS X 10.4; en-US; rv:1.9.2.17) Gecko/20110420 Firefox/3.6.17" - vu5b_1x-w87dTJ-NEVYyISdHo
ab_new_header=1|ab_db_log_long_queries=1|ab_rawsql_close_statement=1|ab_bank_transfer=0|ab_listing_favorite_nag=none|ab_shop_chat=0|ab_share2=1|ab_header_css=0|ab_inline_listi
ab_search_shops=1|ab_community_nav_popout=1|ab_your_account_nav_popout=1|ab_new_listing_process=0|ab_boomerang=0|ab_shopstats_gathering=0 eNFgDvlRp9qHk8BYw1skM_efLpik 10551287
586038
web0041 1.22.333.444 - - [11/May/2011:17:08:53 +0000] "GET /listing/70834087/custom-engraved-vintage-spoon-bracelet?
ref=sr_gallery_16&ga_search_query=wedding&ga_page=20&ga_search_type=handmade&ga_facet=handmade HTTP/1.1" 200 14874 "http://www.etsy.com/search/handmade?q=wedding&page=20" "Moz
Intel Mac OS X 10_6_7; en-us) AppleWebKit/533.20.25 (KHTML, like Gecko) Version/5.0.4 Safari/533.20.27" - s3dAEruW9OGxqsGII6dZFpVMdOo6 www.etsy.com ab_new_header=1|ab_db_log_l
ab_rawsql_close_statement=1|ab_bank_transfer=0|ab_listing_favorite_nag=none|ab_shop_chat=0|ab_share2=1|ab_header_css=0|ab_inline_listing_hearts=1|ab_search_shops=1|ab_communit
ab_your_account_nav_popout=1|ab_boomerang=0|ab_shopstats_gathering=0 SHyrUYHqVMKD5KER-cJtiSR6mz9i 14458253 - - 12412104 391399 407669
web0057 1.22.333.444 - - [11/May/2011:17:08:53 +0000] "GET /listing/45184569/chanel-vintage-navy-blue-cotton-dress-w HTTP/1.1" 200 12071 "http://www.google.com/url?

7. $ tail -f access.log

8. Plotting log data for ad hoc analysis and
real-time trending
EXAMPLES: bit.ly/velocity-etsy

9. Performance data belongs in your app logs
(e.g. web server access log)

10. Access log format
Time, remote address, http method, request uri, referrer, user-agent,
response size, response code, execution time, memory consumed,
plus custom fields...
• Signed-in/out (user_id vs. “-”)
• display mode (“desktop” vs. “mobile”)
• i10n/i18n (“en-US”)
• A/B experiment buckets

11. apache_note(“custom_field”, $whatever);
LogFormat %l %t "%r" %>s %b "%{Referer}i"
"%{User-Agent}i" %{custom_field}n ...

12. apache_note(“custom_field”, $whatever);
LogFormat %l %t "%r" %>s %b "%{Referer}i"
"%{User-Agent}i" %{custom_field}n ...
LogFormat "%{True-Client-IP}i %l %u %t "%r" %>s %b
"%{Referer}i" "%{User-Agent}i"
%{display_mode}n %{user_id}n %{php_bytes}n %{php_usec}n %D”

13. LogFormat "%{True-Client-IP}i %l %u %t "%r" %>s %b
"%{Referer}i" "%{User-Agent}i"
%{display_mode}n %{user_id}n %{php_bytes}n %{php_usec}n %D”
web0060 66.249.71.110 - - [11/May/2011:17:08:53 +0000] "GET /listing/
12189259/tropical-etched-pair-of-lampwork-glass HTTP/1.1" 200 11034 "-"
"Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/
bot.html)" desktop - 13399576 505780 554876

14. We have logs. Cool.

15. Ad hoc analysis using the newest,
hottest, bleeding-edge plotting tool.

16. gnuplot!

17. 1986

18. # access.log
web0060 66.249.71.110 - - [11/May/2011:17:08:53 +0000] "GET /listing/
12189259/tropical-etched-pair-of-lampwork-glass HTTP/1.1" 200 11034 "-"
"Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/
bot.html)" desktop - 13399576 505780 554876

19. # access.log
web0060 66.249.71.110 - - [11/May/2011:17:08:53 +0000] "GET /listing/
12189259/tropical-etched-pair-of-lampwork-glass HTTP/1.1" 200 11034 "-"
"Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/
bot.html)" desktop - 13399576 505780 554876
$ cat access.log | awk '/GET /listing/ {print $5, $(NF-1)/1000}'

20. # access.log
web0060 66.249.71.110 - - [11/May/2011:17:08:53 +0000] "GET /listing/
12189259/tropical-etched-pair-of-lampwork-glass HTTP/1.1" 200 11034 "-"
"Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/
bot.html)" desktop - 13399576 505780 554876
$ cat access.log | awk '/GET /listing/ {print $5, $(NF-1)/1000}'
| sed "s/[//"

21. # access.log
web0060 66.249.71.110 - - [11/May/2011:17:08:53 +0000] "GET /listing/
12189259/tropical-etched-pair-of-lampwork-glass HTTP/1.1" 200 11034 "-"
"Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/
bot.html)" desktop - 13399576 505780 554876
$ cat access.log | awk '/GET /listing/ {print $5, $(NF-1)/1000}'
| sed "s/[//" > response_times.dat
# response_times.dat
11/May/2011:17:08:53 505.780
11/May/2011:17:08:54 311.455
11/May/2011:17:08:53 376.745
11/May/2011:17:08:54 322.278

22. # response_times.dat
11/May/2011:17:08:53 505.780
11/May/2011:17:08:54 311.455
11/May/2011:17:08:53 376.745
11/May/2011:17:08:54 322.278
# scatter.gp
plot 'response_times.dat' using 1:2 with points lc rgb "#33ccff",
800 t "SLA" lw 2 lc rgb "#f28705"

23. # scatter.gp
plot 'response_times.dat' using 1:2 with points lc rgb "#33ccff",
800 t "SLA" lw 2 lc rgb "#f28705"

24. #!/usr/local/bin/gnuplot
set terminal pngcairo enhanced font "arial,11" size 800,600
set output 'response_times.png'
# Graph formatting
set title "Listings, generation times (ms)"
set ylabel "millisec"
set key off
set yrange [0:2000]
# Setup time formatting
set xdata time
set timefmt "%d/%B/%Y:%H:%M:%S"
set format x "%H:%M"
plot
'response_times.dat' using 1:2 with points lc rgb "#33ccff",
800 t "SLA" lw 2 lc rgb "#f28705"

25. # scatter.gp
plot 'response_times.dat' using 1:2 with points lc rgb "#33ccff",
800 t "SLA" lw 2 lc rgb "#ff0000"
Median? 95th Percentile?

26. # access.log
web0060 66.249.71.110 - - [11/May/2011:17:08:53 +0000] "GET /listing/
12189259/tropical-etched-pair-of-lampwork-glass HTTP/1.1" 200 11034 "-"
"Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/
bot.html)" desktop - 13399576 505780 554876
$ exercise_for_reader.sh
# response_median_perc95.dat
22/May/2012:15:15:00 417 620
22/May/2012:15:16:00 309 973
22/May/2012:15:17:00 340 860
22/May/2012:15:18:00 303 416

27. # response_median_perc95.dat
22/May/2012:15:15:00 417 620
22/May/2012:15:16:00 309 973
22/May/2012:15:17:00 340 860
22/May/2012:15:18:00 303 416
# trends.gp
plot 'response_median_perc95.dat'
using 1:3 with filledcurve x1 linewidth 1 linecolor rgb "#548c6c" title "95th Percentile",
"" using 1:2 with lines linewidth 2 linecolor rgb "#2192bf" title "Median",
800 title "SLA" linewidth 2 linecolor rgb "#f28705"

28. # trends.gp
plot 'response_median_perc95.dat'
using 1:3 with filledcurve x1 linewidth 1 linecolor rgb "#0060ad" title "95th Percentile",
"" using 1:2 with lines linewidth 2 linecolor rgb "#dd181f" title "Median",
800 title "SLA" linewidth 2 linecolor rgb "#f28705"

29. # access.log
web0060 66.249.71.110 - - [11/May/2011:17:08:53 +0000] "GET /listing/
12189259/tropical-etched-pair-of-lampwork-glass HTTP/1.1" 200 11034 "-"
"Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/
bot.html)" desktop - 13399576 505780 554876
$ php data_buckets.php <filename> <bucket-size>

30. # access.log
web0060 66.249.71.110 - - [11/May/2011:17:08:53 +0000] "GET /listing/
12189259/tropical-etched-pair-of-lampwork-glass HTTP/1.1" 200 11034 "-"
"Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/
bot.html)" desktop - 13399576 505780 554876
$ php data_buckets.php access.log 50 > response_histogram.dat
# response_histogram.dat
# bucket (ms), count
0 0
50 0
100 12
150 412
200 1203
250 2312
300 2078

31. #!/usr/local/bin/gnuplot
set terminal pngcairo enhanced font "arial,11" size 800,600
set output 'histogram.png'
# Graph formatting
set title "Listings response times (ms)"
set key off
set yrange [0:*]
# Histogram rendering
set style data histograms
set style histogram clustered gap 1
set style fill solid 1.00 border lt -1
set boxwidth 0.9 absolute
plot 'response_histogram.dat' using 2:xtic(1)

32. # histogram.gp
plot 'response_histogram.dat' using 2:xtic(1)

33. Warning for Mac / homebrew users
The pngcairo terminal does not work with pango 1.30 (latest).
Use 1.28 or 1.29 instead.
set terminal pngcairo enhanced font "arial,11" size 800,600
“Gnuplot pdfcairo problem”
https://github.com/mxcl/homebrew/issues/11901

34. (Near) Real-Time Trending

35. Logster
Run by cron, maintaining a cursor on log files
Simple parsing & aggregation
Output to Ganglia, Graphite, stdout
Combined log, via syslog
1 minute schedule
Output to Graphite github.com/etsy/logster

36. Report script execution time (ms) segmented by signed-in/out visitors?
web0020 0.0.0.0 - - [11/May/2011:17:08:54 +0000] "GET /listing/72395693/sterling-
silver-ring-with-rainbow HTTP/1.1" 200 11893 "-" "facebookexternalhit/1.1 (+http://
www.facebook.com/externalhit_uatext.php)" desktop 1924124 10765744 320118 339176
web0047 0.0.0.0 - - [11/May/2011:17:08:53 +0000] "GET /listing/31224920/flirt-14k-
gold-filled-hook-earrings HTTP/1.1" 200 12339 "-" "Mozilla/5.0 (compatible;
Googlebot/2.1; +http://www.google.com/bot.html)" desktop - 10813080 377090 402038
web0031 0.0.0.0 - - [11/May/2011:17:08:53 +0000] "GET /listing/64397692/spades-and-
diamonds-ooak-red-on-custom HTTP/1.1" 200 13835 "http://www.etsy.com/listing/
72550205/krylon-dots-black-on-white-tank-womens-l" "Mozilla/5.0 (iPhone; U; CPU like
Mac OS X; en) AppleWebKit/420+ (KHTML, like Gecko) Version/3.0 Mobile/1A543a Safari/
419.3" mobile 8513282 15249296 563978 586038

37. web0020 0.0.0.0 - - [11/May/2011:17:08:54 +0000] "GET /listing/72395693/sterling-
silver-ring-with-rainbow HTTP/1.1" 200 11893 "-" "facebookexternalhit/1.1 (+http://
www.facebook.com/externalhit_uatext.php)" desktop 1924124 10765744 320118 339176
# Parse fields from log file
self.reg = re.compile(‘.* (?P<display_mode>w+) (?P<etsy_user_id>[d-]+)
(?P<php_bytes>d+) (?P<php_usec>d+) (?P<apache_usec>d+)$’)

38. web0020 0.0.0.0 - - [11/May/2011:17:08:54 +0000] "GET /listing/72395693/sterling-
silver-ring-with-rainbow HTTP/1.1" 200 11893 "-" "facebookexternalhit/1.1 (+http://
www.facebook.com/externalhit_uatext.php)" desktop 1924124 10765744 320118 339176
# Collect data from each matching line
# signed_out_ms and signed_in_ms are lists.
fields = regMatch.groupdict()
php_ms = int(fields['php_usec']) / 1000
user_id = fields['user_id']
if (user_id == "-"):
self.signed_out_ms.append(php_ms)
else:
self.signed_in_ms.append(php_ms)

39. web0020 0.0.0.0 - - [11/May/2011:17:08:54 +0000] "GET /listing/72395693/sterling-
silver-ring-with-rainbow HTTP/1.1" 200 11893 "-" "facebookexternalhit/1.1 (+http://
www.facebook.com/externalhit_uatext.php)" desktop 1924124 10765744 320118 339176
return [
MetricObject("signed_in_reqs", len(self.signed_in_ms), "Requests"),
MetricObject("signed_in_ms_min", self.signed_in_ms[0], "ms"),
MetricObject("signed_in_ms_median", percentile(self.signed_in_ms, 0.5), "ms"),
MetricObject("signed_in_ms_perc95", percentile(self.signed_in_ms, 0.95), "ms"),
MetricObject("signed_in_ms_max", self.signed_in_ms[-1], "ms"),
]

40. web0020 0.0.0.0 - - [11/May/2011:17:08:54 +0000] "GET /listing/72395693/sterling-
silver-ring-with-rainbow HTTP/1.1" 200 11893 "-" "facebookexternalhit/1.1 (+http://
www.facebook.com/externalhit_uatext.php)" desktop 1924124 10765744 320118 339176
$ logster --output stdout PerfLogster access.log
signed_out_reqs 8
signed_out_ms_min 201
signed_out_ms_median 227.5
signed_out_ms_perc95 236.9
signed_out_ms_max 239
signed_in_reqs 3
signed_in_ms_min 258
signed_in_ms_median 268
signed_in_ms_perc95 268.0
signed_in_ms_max 268

41. web0020 0.0.0.0 - - [11/May/2011:17:08:54 +0000] "GET /listing/72395693/sterling-
silver-ring-with-rainbow HTTP/1.1" 200 11893 "-" "facebookexternalhit/1.1 (+http://
www.facebook.com/externalhit_uatext.php)" desktop 1924124 10765744 320118 339176
$ logster --output stdout PerfLogster access.log
$ logster --output graphite --graphite-host graphite.example.com:2003
PerfLogster access.log

42. web0020 0.0.0.0 - - [11/May/2011:17:08:54 +0000] "GET /listing/72395693/sterling-
silver-ring-with-rainbow HTTP/1.1" 200 11893 "-" "facebookexternalhit/1.1 (+http://
www.facebook.com/externalhit_uatext.php)" desktop 1924124 10765744 320118 339176
$ logster --output stdout PerfLogster access.log
$ logster --output graphite --graphite-host graphite.example.com:2003
PerfLogster access.log
# Put that into a crontab to run every minute, and...

43. 600
450
300
MEDIAN PAGE GENERATION TIMES (MS)
BY SIGNED-IN/SIGNED-OUT
150
0

44. Thanks!
Code samples and slides at bit.ly/velocity-etsy
github.com/etsy/logster
We are hiring... etsy.com/careers
(or just come talk to me afterward)