• Share
  • Email
  • Embed
  • Like
  • Save
  • Private Content
Simple Log Analysis and Trending
 

Simple Log Analysis and Trending

on

  • 11,805 views

 

Statistics

Views

Total Views
11,805
Views on SlideShare
9,673
Embed Views
2,132

Actions

Likes
26
Downloads
234
Comments
0

14 Embeds 2,132

http://www.dbthink.com 1146
http://www.mikebrittain.com 854
http://lanyrd.com 39
https://twitter.com 27
http://xianguo.com 25
http://feeds.feedburner.com 19
http://www.linkedin.com 9
http://abtasty.com 4
https://si0.twimg.com 3
http://coderwall.com 2
http://www.tuicool.com 1
http://131.253.14.250 1
http://translate.googleusercontent.com 1
https://twimg0-a.akamaihd.net 1
More...

Accessibility

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

    Simple Log Analysis and Trending Simple Log Analysis and Trending Presentation Transcript

    • Simple Log Analysis and Trending Mike Brittain DIRECTOR OF ENGINEERING, ETSY @mikebrittain EXAMPLES: bit.ly/velocity-etsy
    • sa=t&source=web&cd=18&ved=0CHkQFjAR&url=http%3A%2F%2Fwww.etsy.com%2Flisting%2F45184569%2Fchanel-vintage-navy-blue-cotton-dress-w&rct=j&q=vintage%20chanel%20dress&ei=fMLKTZH0MYTL0AGjpsCDCg&usg=AFQjCNHnkswNxqzqi-Rl6nLVIr7dTg78Rg" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; WOW64; Trident/4.0; GTB7.0; SLCC2; .NET CLR 2.0.50727; .3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; MDDR; .NET4.0C)" - ur9XOgyn58I8opUn7LDhyqPliXap www.etsy.com ab_new_header=1|ab_db_log_long_queries=1|ab_rawsql_close_statement=1|ab_bank_transfer=0|ab_listing_favorite_nag=none|ab_shop_chat=0|ab_share2=1|ab_header_css=0|ab_inline_listing_hearts=1|ab_search_shops=1|ab_community_nav_popout=1|ab_boomerang=0|ab_shopstats_gathering=0 te0sXDV0SfwBpU-qL0w0TnAUIw-x 0 - - 11567688 601120 637356web0037 1.22.333.444 - - [11/May/2011:17:08:53 +0000] "GET /listing/60547985/half-yard-little-girls-in-kimono-print HTTP/1.1" 200 12389 "http://www.etsy.com/shop/beautifulwork?section_id=5810576&page=2" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; WOW64; Trident/4.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6- LVTY9r_Z_IEcEbfKWUJoPTAGHhBx www.etsy.com ab_new_header=1|ab_db_log_long_queries=1|ab_rawsql_close_statement=1|ab_bank_transfer=0|ab_listing_favorite_nag=none|ab_shop_chat=0|ab_sharab_header_css=0|ab_inline_listing_hearts=1|ab_search_defaults_allitems=1|ab_search_shops=1|ab_community_nav_popout=1|ab_boomerang=0|ab_shopstats_gathering=0 i57VnazHMo2LEjii5IRjaiaWVO412157 437768web0039 1.22.333.444 - - [11/May/2011:17:08:53 +0000] "GET /listing/71705428/fearfully-and-wonderfully-made-8x10?ref=sr_list_10&ga_search_query=pink+and+green&ga_noautofacet=1&ga_search_type=handmade&ga_facet=handmade%2Fart HTTP/1.1" 200 13232 "http://www.etsy.com/search/handmade/art?q=pink+and+green&noautofacet=1" "Mozilla/5.0 (WindAppleWebKit/534.24 (KHTML, like Gecko) Chrome/11.0.696.65 Safari/534.24" - aKR4B5QW0KrN-NBZpWvm0AgkwcD3 www.etsy.com ab_new_header=1|ab_db_log_long_queries=1|ab_rawsql_close_statementab_bank_transfer=0|ab_listing_favorite_nag=none|ab_shop_chat=0|ab_share2=1|ab_header_css=0|ab_inline_listing_hearts=1|ab_search_shops=1|ab_community_nav_popout=1|ab_shopstats_gatherin1mFgWOMRaRG5RN1zcfNZRVfdaDz7 7323622 - - 15124584 662683 690754web0047 1.22.333.444 - - [11/May/2011:17:08:53 +0000] "GET /listing/31224920/flirt-14k-gold-filled-hook-earrings?ref=sr_gallery_34&ga_noautofacet=1&ga_search_type=handmade&ga_facet=ha%2Fearrings%2Fpost&ga_search_query= HTTP/1.1" 200 12339 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" - Y8PI3hOiKDJWNz7M_-T_t4mq75zM www.etsy.com ab_nab_db_log_long_queries=1|ab_rawsql_close_statement=1|ab_bank_transfer=0|ab_listing_favorite_nag=none|ab_shop_chat=0|ab_share2=1|ab_header_css=0|ab_inline_listing_hearts=1|ab_search_defaults_allitems=0|ab_search_shops=1|ab_community_nav_popout=1|ab_boomerang=0|ab_shopstats_gathering=0 L6iJ-8gaox1rUhd8F8CVbfhLrMnJ 0 - - 10813080 377090 402038web0020 1.22.333.444 - - [11/May/2011:17:08:54 +0000] "GET /listing/72395693/sterling-silver-ring-with-rainbow HTTP/1.1" 200 11893 "-" "facebookexternalhit/1.1 (+http://www.facebook.cexternalhit_uatext.php)" - phQhH2hNgWf3AfCQH0SpLqoFL6I1 www.etsy.com ab_new_header=1|ab_db_log_long_queries=1|ab_rawsql_close_statement=1|ab_bank_transfer=0|ab_listing_favorite_nag=no Aab_shop_chat=0|ab_share2=1|ab_header_css=0|ab_inline_listing_hearts=1|ab_search_defaults_allitems=0|ab_search_shops=1|ab_community_nav_popout=1|ab_boomerang=0|ab_shopstats_gathering=0Vs1ZXSz3z9fZ 0 - - 10765744 320118 339176web0031 1.22.333.444 - - [11/May/2011:17:08:53 +0000] "GET /listing/64397692/spades-and-diamonds-ooak-red-on-custom?ref=v1_other_2 HTTP/1.1" 200 13835 "http://www.etsy.com/listing/725dots-black-on-white-tank-womens-l?ref=v1_other_2" "Mozilla/5.0 (Macintosh; U; PPC Mac OS X 10.4; en-US; rv:1.9.2.17) Gecko/20110420 Firefox/3.6.17" - vu5b_1x-w87dTJ-NEVYyISdHo0Qf www.ab_new_header=1|ab_db_log_long_queries=1|ab_rawsql_close_statement=1|ab_bank_transfer=0|ab_listing_favorite_nag=none|ab_shop_chat=0|ab_share2=1|ab_header_css=0|ab_inline_listing_heartab_search_shops=1|ab_community_nav_popout=1|ab_your_account_nav_popout=1|ab_new_listing_process=0|ab_boomerang=0|ab_shopstats_gathering=0 eNFgDvlRp9qHk8BYw1skM_efLpik 10551287 - - 152586038web0041 1.22.333.444 - - [11/May/2011:17:08:53 +0000] "GET /listing/70834087/custom-engraved-vintage-spoon-bracelet?ref=sr_gallery_16&ga_search_query=wedding&ga_page=20&ga_search_type=handmade&ga_facet=handmade HTTP/1.1" 200 14874 "http://www.etsy.com/search/handmade?q=wedding&page=20" "Mozilla/5.0Intel Mac OS X 10_6_7; en-us) AppleWebKit/533.20.25 (KHTML, like Gecko) Version/5.0.4 Safari/533.20.27" - s3dAEruW9OGxqsGII6dZFpVMdOo6 www.etsy.com ab_new_header=1|ab_db_log_long_querab_rawsql_close_statement=1|ab_bank_transfer=0|ab_listing_favorite_nag=none|ab_shop_chat=0|ab_share2=1|ab_header_css=0|ab_inline_listing_hearts=1|ab_search_shops=1|ab_community_nav_poab_your_account_nav_popout=1|ab_boomerang=0|ab_shopstats_gathering=0 SHyrUYHqVMKD5KER-cJtiSR6mz9i 14458253 - - 12412104 391399 407669web0057 1.22.333.444 - - [11/May/2011:17:08:53 +0000] "GET /listing/45184569/chanel-vintage-navy-blue-cotton-dress-w HTTP/1.1" 200 12071 "http://www.google.com/url?sa=t&source=web&cd=18&ved=0CHkQFjAR&url=http%3A%2F%2Fwww.etsy.com%2Flisting%2F45184569%2Fchanel-vintage-navy-blue-cotton-dress-w&rct=j&q=vintage%20chanel%20dress&ei=fMLKTZH0MYTL0AGjpsCDCg&usg=AFQjCNHnkswNxqzqi-Rl6nLVIr7dTg78Rg" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; WOW64; Trident/4.0; GTB7.0; SLCC2; .NET CLR 2.0.50727; .3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; MDDR; .NET4.0C)" - ur9XOgyn58I8opUn7LDhyqPliXap www.etsy.com ab_new_header=1|ab_db_log_long_queries=1|ab_rawsql_close_statement=1|ab_bank_transfer=0|ab_listing_favorite_nag=none|ab_shop_chat=0|ab_share2=1|ab_header_css=0|ab_inline_listing_hearts=1|ab_search_shops=1|ab_community_nav_popout=1|ab_boomerang=0|ab_shopstats_gathering=0 te0sXDV0SfwBpU-qL0w0TnAUIw-x 0 - - 11567688 601120 637356web0037 1.22.333.444 - - [11/May/2011:17:08:53 +0000] "GET /listing/60547985/half-yard-little-girls-in-kimono-print HTTP/1.1" 200 12389 "http://www.etsy.com/shop/beautifulwork?section_id=5810576&page=2" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; WOW64; Trident/4.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6- LVTY9r_Z_IEcEbfKWUJoPTAGHhBx www.etsy.com ab_new_header=1|ab_db_log_long_queries=1|ab_rawsql_close_statement=1|ab_bank_transfer=0|ab_listing_favorite_nag=none|ab_shop_chat=0|ab_sharab_header_css=0|ab_inline_listing_hearts=1|ab_search_defaults_allitems=1|ab_search_shops=1|ab_community_nav_popout=1|ab_boomerang=0|ab_shopstats_gathering=0 i57VnazHMo2LEjii5IRjaiaWVO412157 437768web0039 1.22.333.444 - - [11/May/2011:17:08:53 +0000] "GET /listing/71705428/fearfully-and-wonderfully-made-8x10?ref=sr_list_10&ga_search_query=pink+and+green&ga_noautofacet=1&ga_search_type=handmade&ga_facet=handmade%2Fart HTTP/1.1" 200 13232 "http://www.etsy.com/search/handmade/art?q=pink+and+green&noautofacet=1" "Mozilla/5.0 (WindAppleWebKit/534.24 (KHTML, like Gecko) Chrome/11.0.696.65 Safari/534.24" - aKR4B5QW0KrN-NBZpWvm0AgkwcD3 www.etsy.com ab_new_header=1|ab_db_log_long_queries=1|ab_rawsql_close_statementab_bank_transfer=0|ab_listing_favorite_nag=none|ab_shop_chat=0|ab_share2=1|ab_header_css=0|ab_inline_listing_hearts=1|ab_search_shops=1|ab_community_nav_popout=1|ab_shopstats_gatherin
    • PAGE GENERATION TIME (MS)600450300 B150 0
    • sa=t&source=web&cd=18&ved=0CHkQFjAR&url=http%3A%2F%2Fwww.etsy.com%2Flisting%2F45184569%2Fchanel-vintage-navy-blue-cotton-dress-w&rct=j&q=vintage%20chanel%20dress&ei=fMLKTZH0MYTL0AGjpsCDCg&usg=AFQjCNHnkswNxqzqi-Rl6nLVIr7dTg78Rg" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; WOW64; Trident/4.0; GTB7.0; SLCC2; .NET CLR 2.0.50727; .3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; MDDR; .NET4.0C)" - ur9XOgyn58I8opUn7LDhyqPliXap www.etsy.com ab_new_header=1|ab_db_log_long_queries=1|ab_rawsql_close_statement=1|ab_bank_transfer=0|ab_listing_favorite_nag=none|ab_shop_chat=0|ab_share2=1|ab_header_css=0|ab_inline_listing_hearts=1|ab_search_shops=1|ab_community_nav_popout=1|ab_boomerang=0|ab_shopstats_gathering=0 te0sXDV0SfwBpU-qL0w0TnAUIw-x 0 - - 11567688 601120 637356web0037 1.22.333.444 - - [11/May/2011:17:08:53 +0000] "GET /listing/60547985/half-yard-little-girls-in-kimono-print HTTP/1.1" 200 12389 "http://www.etsy.com/shop/beautifulwork?section_id=5810576&page=2" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; WOW64; Trident/4.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6- LVTY9r_Z_IEcEbfKWUJoPTAGHhBx www.etsy.com ab_new_header=1|ab_db_log_long_queries=1|ab_rawsql_close_statement=1|ab_bank_transfer=0|ab_listing_favorite_nag=none|ab_shop_chat=0|ab_sharab_header_css=0|ab_inline_listing_hearts=1|ab_search_defaults_allitems=1|ab_search_shops=1|ab_community_nav_popout=1|ab_boomerang=0|ab_shopstats_gathering=0 i57VnazHMo2LEjii5IRjaiaWVO412157 437768web0039 1.22.333.444 - - [11/May/2011:17:08:53 +0000] "GET /listing/71705428/fearfully-and-wonderfully-made-8x10?ref=sr_list_10&ga_search_query=pink+and+green&ga_noautofacet=1&ga_search_type=handmade&ga_facet=handmade%2Fart HTTP/1.1" 200 13232 "http://www.etsy.com/search/handmade/art?q=pink+and+green&noautofacet=1" "Mozilla/5.0 (WindAppleWebKit/534.24 (KHTML, like Gecko) Chrome/11.0.696.65 Safari/534.24" - aKR4B5QW0KrN-NBZpWvm0AgkwcD3 www.etsy.com ab_new_header=1|ab_db_log_long_queries=1|ab_rawsql_close_statementab_bank_transfer=0|ab_listing_favorite_nag=none|ab_shop_chat=0|ab_share2=1|ab_header_css=0|ab_inline_listing_hearts=1|ab_search_shops=1|ab_community_nav_popout=1|ab_shopstats_gatherin1mFgWOMRaRG5RN1zcfNZRVfdaDz7 7323622 - - 15124584 662683 690754web0047 1.22.333.444 - - [11/May/2011:17:08:53 +0000] "GET /listing/31224920/flirt-14k-gold-filled-hook-earrings?ref=sr_gallery_34&ga_noautofacet=1&ga_search_type=handmade&ga_facet=ha%2Fearrings%2Fpost&ga_search_query= HTTP/1.1" 200 12339 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" - Y8PI3hOiKDJWNz7M_-T_t4mq75zM www.etsy.com ab_nab_db_log_long_queries=1|ab_rawsql_close_statement=1|ab_bank_transfer=0|ab_listing_favorite_nag=none|ab_shop_chat=0|ab_share2=1|ab_header_css=0|ab_inline_listing_hearts=1|ab_search_defaults_allitems=0|ab_search_shops=1|ab_community_nav_popout=1|ab_boomerang=0|ab_shopstats_gathering=0 L6iJ-8gaox1rUhd8F8CVbfhLrMnJ 0 - - 10813080 377090 402038web0020 1.22.333.444 - - [11/May/2011:17:08:54 +0000] "GET /listing/72395693/sterling-silver-ring-with-rainbow HTTP/1.1" 200 11893 "-" "facebookexternalhit/1.1 (+http://www.facebook.cexternalhit_uatext.php)" - phQhH2hNgWf3AfCQH0SpLqoFL6I1 www.etsy.com ab_new_header=1|ab_db_log_long_queries=1|ab_rawsql_close_statement=1|ab_bank_transfer=0|ab_listing_favorite_nag=no Aab_shop_chat=0|ab_share2=1|ab_header_css=0|ab_inline_listing_hearts=1|ab_search_defaults_allitems=0|ab_search_shops=1|ab_community_nav_popout=1|ab_boomerang=0|ab_shopstats_gathering=0Vs1ZXSz3z9fZ 0 - - 10765744 320118 339176web0031 1.22.333.444 - - [11/May/2011:17:08:53 +0000] "GET /listing/64397692/spades-and-diamonds-ooak-red-on-custom?ref=v1_other_2 HTTP/1.1" 200 13835 "http://www.etsy.com/listing/725dots-black-on-white-tank-womens-l?ref=v1_other_2" "Mozilla/5.0 (Macintosh; U; PPC Mac OS X 10.4; en-US; rv:1.9.2.17) Gecko/20110420 Firefox/3.6.17" - vu5b_1x-w87dTJ-NEVYyISdHo0Qf www.ab_new_header=1|ab_db_log_long_queries=1|ab_rawsql_close_statement=1|ab_bank_transfer=0|ab_listing_favorite_nag=none|ab_shop_chat=0|ab_share2=1|ab_header_css=0|ab_inline_listing_heartab_search_shops=1|ab_community_nav_popout=1|ab_your_account_nav_popout=1|ab_new_listing_process=0|ab_boomerang=0|ab_shopstats_gathering=0 eNFgDvlRp9qHk8BYw1skM_efLpik 10551287 - - 152586038web0041 1.22.333.444 - - [11/May/2011:17:08:53 +0000] "GET /listing/70834087/custom-engraved-vintage-spoon-bracelet?ref=sr_gallery_16&ga_search_query=wedding&ga_page=20&ga_search_type=handmade&ga_facet=handmade HTTP/1.1" 200 14874 "http://www.etsy.com/search/handmade?q=wedding&page=20" "Mozilla/5.0Intel Mac OS X 10_6_7; en-us) AppleWebKit/533.20.25 (KHTML, like Gecko) Version/5.0.4 Safari/533.20.27" - s3dAEruW9OGxqsGII6dZFpVMdOo6 www.etsy.com ab_new_header=1|ab_db_log_long_querab_rawsql_close_statement=1|ab_bank_transfer=0|ab_listing_favorite_nag=none|ab_shop_chat=0|ab_share2=1|ab_header_css=0|ab_inline_listing_hearts=1|ab_search_shops=1|ab_community_nav_poab_your_account_nav_popout=1|ab_boomerang=0|ab_shopstats_gathering=0 SHyrUYHqVMKD5KER-cJtiSR6mz9i 14458253 - - 12412104 391399 407669web0057 1.22.333.444 - - [11/May/2011:17:08:53 +0000] "GET /listing/45184569/chanel-vintage-navy-blue-cotton-dress-w HTTP/1.1" 200 12071 "http://www.google.com/url?sa=t&source=web&cd=18&ved=0CHkQFjAR&url=http%3A%2F%2Fwww.etsy.com%2Flisting%2F45184569%2Fchanel-vintage-navy-blue-cotton-dress-w&rct=j&q=vintage%20chanel%20dress&ei=fMLKTZH0MYTL0AGjpsCDCg&usg=AFQjCNHnkswNxqzqi-Rl6nLVIr7dTg78Rg" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; WOW64; Trident/4.0; GTB7.0; SLCC2; .NET CLR 2.0.50727; .3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; MDDR; .NET4.0C)" - ur9XOgyn58I8opUn7LDhyqPliXap www.etsy.com ab_new_header=1|ab_db_log_long_queries=1|ab_rawsql_close_statement=1|ab_bank_transfer=0|ab_listing_favorite_nag=none|ab_shop_chat=0|ab_share2=1|ab_header_css=0|ab_inline_listing_hearts=1|ab_search_shops=1|ab_community_nav_popout=1|ab_boomerang=0|ab_shopstats_gathering=0 te0sXDV0SfwBpU-qL0w0TnAUIw-x 0 - - 11567688 601120 637356web0037 1.22.333.444 - - [11/May/2011:17:08:53 +0000] "GET /listing/60547985/half-yard-little-girls-in-kimono-print HTTP/1.1" 200 12389 "http://www.etsy.com/shop/beautifulwork?section_id=5810576&page=2" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; WOW64; Trident/4.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6- LVTY9r_Z_IEcEbfKWUJoPTAGHhBx www.etsy.com ab_new_header=1|ab_db_log_long_queries=1|ab_rawsql_close_statement=1|ab_bank_transfer=0|ab_listing_favorite_nag=none|ab_shop_chat=0|ab_sharab_header_css=0|ab_inline_listing_hearts=1|ab_search_defaults_allitems=1|ab_search_shops=1|ab_community_nav_popout=1|ab_boomerang=0|ab_shopstats_gathering=0 i57VnazHMo2LEjii5IRjaiaWVO412157 437768web0039 1.22.333.444 - - [11/May/2011:17:08:53 +0000] "GET /listing/71705428/fearfully-and-wonderfully-made-8x10?ref=sr_list_10&ga_search_query=pink+and+green&ga_noautofacet=1&ga_search_type=handmade&ga_facet=handmade%2Fart HTTP/1.1" 200 13232 "http://www.etsy.com/search/handmade/art?q=pink+and+green&noautofacet=1" "Mozilla/5.0 (WindAppleWebKit/534.24 (KHTML, like Gecko) Chrome/11.0.696.65 Safari/534.24" - aKR4B5QW0KrN-NBZpWvm0AgkwcD3 www.etsy.com ab_new_header=1|ab_db_log_long_queries=1|ab_rawsql_close_statementab_bank_transfer=0|ab_listing_favorite_nag=none|ab_shop_chat=0|ab_share2=1|ab_header_css=0|ab_inline_listing_hearts=1|ab_search_shops=1|ab_community_nav_popout=1|ab_shopstats_gatherin
    • PAGE GENERATION TIME (MS)600450300 B150 0
    • A Bweb0057 1.22.333.444 - - [11/May/2011:17:08:53 +0000] "GET /listing/45184569/chanel-vintage-navy-blue-cotton-dress-w HTTP/1.1" 200 12071 "http://www.google.com/url?sa=t&source=web&cd=18&ved=0CHkQFjAR&url=http%3A%2F%2Fwww.etsy.com%2Flisting%2F45184569%2Fchanel-vintage-navy-blue-cotton-dress-w&rct=j&q=vintage%20chanel%20dress&ei=fMLKTZH0MYTL0AGjpsCDCg&usg=AFQjCNHnkswNxqzqi-Rl6nLVIr7dTg78Rg" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; WOW64; Trident/4.0; GTB7.0; SLCC2; .NET CLR 2.0.3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; MDDR; .NET4.0C)" - ur9XOgyn58I8opUn7LDhyqPliXap www.etsy.com ab_new_header=1|ab_db_log_long_queries=1|ab_rawsql_close_stateab_bank_transfer=0|ab_listing_favorite_nag=none|ab_shop_chat=0|ab_share2=1|ab_header_css=0|ab_inline_listing_hearts=1|ab_search_shops=1|ab_community_nav_popout=1|ab_boomerang=ab_shopstats_gathering=0 te0sXDV0SfwBpU-qL0w0TnAUIw-x 0 - - 11567688 601120 637356web0037 1.22.333.444 - - [11/May/2011:17:08:53 +0000] "GET /listing/60547985/half-yard-little-girls-in-kimono-print HTTP/1.1" 200 12389 "http://www.etsy.com/shop/beautifulworksection_id=5810576&page=2" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; WOW64; Trident/4.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Cen- LVTY9r_Z_IEcEbfKWUJoPTAGHhBx www.etsy.com ab_new_header=1|ab_db_log_long_queries=1|ab_rawsql_close_statement=1|ab_bank_transfer=0|ab_listing_favorite_nag=none|ab_shop_chat=0ab_header_css=0|ab_inline_listing_hearts=1|ab_search_defaults_allitems=1|ab_search_shops=1|ab_community_nav_popout=1|ab_boomerang=0|ab_shopstats_gathering=0 i57VnazHMo2LEjii5I412157 437768web0039 1.22.333.444 - - [11/May/2011:17:08:53 +0000] "GET /listing/71705428/fearfully-and-wonderfully-made-8x10?ref=sr_list_10&ga_search_query=pink+and+green&ga_noautofacet=1&ga_search_type=handmade&ga_facet=handmade%2Fart HTTP/1.1" 200 13232 "http://www.etsy.com/search/handmade/art?q=pink+and+green&noautofacet=1" "Mozilla/5AppleWebKit/534.24 (KHTML, like Gecko) Chrome/11.0.696.65 Safari/534.24" - aKR4B5QW0KrN-NBZpWvm0AgkwcD3 www.etsy.com ab_new_header=1|ab_db_log_long_queries=1|ab_rawsql_close_sab_bank_transfer=0|ab_listing_favorite_nag=none|ab_shop_chat=0|ab_share2=1|ab_header_css=0|ab_inline_listing_hearts=1|ab_search_shops=1|ab_community_nav_popout=1|ab_shopstats_1mFgWOMRaRG5RN1zcfNZRVfdaDz7 7323622 - - 15124584 662683 690754web0047 1.22.333.444 - - [11/May/2011:17:08:53 +0000] "GET /listing/31224920/flirt-14k-gold-filled-hook-earrings?ref=sr_gallery_34&ga_noautofacet=1&ga_search_type=handmade&ga_%2Fearrings%2Fpost&ga_search_query= HTTP/1.1" 200 12339 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" - Y8PI3hOiKDJWNz7M_-T_t4mq75zM www.etsy.ab_db_log_long_queries=1|ab_rawsql_close_statement=1|ab_bank_transfer=0|ab_listing_favorite_nag=none|ab_shop_chat=0|ab_share2=1|ab_header_css=0|ab_inline_listing_hearts=1|ab_search_defaults_allitems=0|ab_search_shops=1|ab_community_nav_popout=1|ab_boomerang=0|ab_shopstats_gathering=0 L6iJ-8gaox1rUhd8F8CVbfhLrMnJ 0 - - 10813080 377090 402038web0020 1.22.333.444 - - [11/May/2011:17:08:54 +0000] "GET /listing/72395693/sterling-silver-ring-with-rainbow HTTP/1.1" 200 11893 "-" "facebookexternalhit/1.1 (+http://www.faexternalhit_uatext.php)" - phQhH2hNgWf3AfCQH0SpLqoFL6I1 www.etsy.com ab_new_header=1|ab_db_log_long_queries=1|ab_rawsql_close_statement=1|ab_bank_transfer=0|ab_listing_favoritab_shop_chat=0|ab_share2=1|ab_header_css=0|ab_inline_listing_hearts=1|ab_search_defaults_allitems=0|ab_search_shops=1|ab_community_nav_popout=1|ab_boomerang=0|ab_shopstats_gatVs1ZXSz3z9fZ 0 - - 10765744 320118 339176web0031 1.22.333.444 - - [11/May/2011:17:08:53 +0000] "GET /listing/64397692/spades-and-diamonds-ooak-red-on-custom?ref=v1_other_2 HTTP/1.1" 200 13835 "http://www.etsy.com/lisdots-black-on-white-tank-womens-l?ref=v1_other_2" "Mozilla/5.0 (Macintosh; U; PPC Mac OS X 10.4; en-US; rv:1.9.2.17) Gecko/20110420 Firefox/3.6.17" - vu5b_1x-w87dTJ-NEVYyISdHoab_new_header=1|ab_db_log_long_queries=1|ab_rawsql_close_statement=1|ab_bank_transfer=0|ab_listing_favorite_nag=none|ab_shop_chat=0|ab_share2=1|ab_header_css=0|ab_inline_listiab_search_shops=1|ab_community_nav_popout=1|ab_your_account_nav_popout=1|ab_new_listing_process=0|ab_boomerang=0|ab_shopstats_gathering=0 eNFgDvlRp9qHk8BYw1skM_efLpik 10551287586038web0041 1.22.333.444 - - [11/May/2011:17:08:53 +0000] "GET /listing/70834087/custom-engraved-vintage-spoon-bracelet?ref=sr_gallery_16&ga_search_query=wedding&ga_page=20&ga_search_type=handmade&ga_facet=handmade HTTP/1.1" 200 14874 "http://www.etsy.com/search/handmade?q=wedding&page=20" "MozIntel Mac OS X 10_6_7; en-us) AppleWebKit/533.20.25 (KHTML, like Gecko) Version/5.0.4 Safari/533.20.27" - s3dAEruW9OGxqsGII6dZFpVMdOo6 www.etsy.com ab_new_header=1|ab_db_log_lab_rawsql_close_statement=1|ab_bank_transfer=0|ab_listing_favorite_nag=none|ab_shop_chat=0|ab_share2=1|ab_header_css=0|ab_inline_listing_hearts=1|ab_search_shops=1|ab_communitab_your_account_nav_popout=1|ab_boomerang=0|ab_shopstats_gathering=0 SHyrUYHqVMKD5KER-cJtiSR6mz9i 14458253 - - 12412104 391399 407669web0057 1.22.333.444 - - [11/May/2011:17:08:53 +0000] "GET /listing/45184569/chanel-vintage-navy-blue-cotton-dress-w HTTP/1.1" 200 12071 "http://www.google.com/url?
    • $ tail -f access.log
    • Plotting log data for ad hoc analysis and real-time trending EXAMPLES: bit.ly/velocity-etsy
    • Performance data belongs in your app logs (e.g. web server access log)
    • Access log formatTime, remote address, http method, request uri, referrer, user-agent,response size, response code, execution time, memory consumed,plus custom fields...• Signed-in/out (user_id vs. “-”)• display mode (“desktop” vs. “mobile”)• i10n/i18n (“en-US”)• A/B experiment buckets
    • apache_note(“custom_field”, $whatever);LogFormat %l %t "%r" %>s %b "%{Referer}i""%{User-Agent}i" %{custom_field}n ...
    • apache_note(“custom_field”, $whatever);LogFormat %l %t "%r" %>s %b "%{Referer}i""%{User-Agent}i" %{custom_field}n ...LogFormat "%{True-Client-IP}i %l %u %t "%r" %>s %b"%{Referer}i" "%{User-Agent}i"%{display_mode}n %{user_id}n %{php_bytes}n %{php_usec}n %D”
    • LogFormat "%{True-Client-IP}i %l %u %t "%r" %>s %b"%{Referer}i" "%{User-Agent}i"%{display_mode}n %{user_id}n %{php_bytes}n %{php_usec}n %D”web0060 66.249.71.110 - - [11/May/2011:17:08:53 +0000] "GET /listing/12189259/tropical-etched-pair-of-lampwork-glass HTTP/1.1" 200 11034 "-""Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" desktop - 13399576 505780 554876
    • We have logs. Cool.
    • Ad hoc analysis using the newest,hottest, bleeding-edge plotting tool.
    • gnuplot!
    • 1986
    • # access.logweb0060 66.249.71.110 - - [11/May/2011:17:08:53 +0000] "GET /listing/12189259/tropical-etched-pair-of-lampwork-glass HTTP/1.1" 200 11034 "-""Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" desktop - 13399576 505780 554876
    • # access.logweb0060 66.249.71.110 - - [11/May/2011:17:08:53 +0000] "GET /listing/12189259/tropical-etched-pair-of-lampwork-glass HTTP/1.1" 200 11034 "-""Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" desktop - 13399576 505780 554876$ cat access.log | awk /GET /listing/ {print $5, $(NF-1)/1000}
    • # access.logweb0060 66.249.71.110 - - [11/May/2011:17:08:53 +0000] "GET /listing/12189259/tropical-etched-pair-of-lampwork-glass HTTP/1.1" 200 11034 "-""Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" desktop - 13399576 505780 554876$ cat access.log | awk /GET /listing/ {print $5, $(NF-1)/1000} | sed "s/[//"
    • # access.logweb0060 66.249.71.110 - - [11/May/2011:17:08:53 +0000] "GET /listing/12189259/tropical-etched-pair-of-lampwork-glass HTTP/1.1" 200 11034 "-""Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" desktop - 13399576 505780 554876$ cat access.log | awk /GET /listing/ {print $5, $(NF-1)/1000} | sed "s/[//" > response_times.dat# response_times.dat11/May/2011:17:08:53 505.78011/May/2011:17:08:54 311.45511/May/2011:17:08:53 376.74511/May/2011:17:08:54 322.278
    • # response_times.dat11/May/2011:17:08:53 505.78011/May/2011:17:08:54 311.45511/May/2011:17:08:53 376.74511/May/2011:17:08:54 322.278# scatter.gpplot response_times.dat using 1:2 with points lc rgb "#33ccff", 800 t "SLA" lw 2 lc rgb "#f28705"
    • # scatter.gpplot response_times.dat using 1:2 with points lc rgb "#33ccff", 800 t "SLA" lw 2 lc rgb "#f28705"
    • #!/usr/local/bin/gnuplotset terminal pngcairo enhanced font "arial,11" size 800,600set output response_times.png# Graph formattingset title "Listings, generation times (ms)"set ylabel "millisec"set key offset yrange [0:2000]# Setup time formattingset xdata timeset timefmt "%d/%B/%Y:%H:%M:%S"set format x "%H:%M"plot response_times.dat using 1:2 with points lc rgb "#33ccff", 800 t "SLA" lw 2 lc rgb "#f28705"
    • # scatter.gpplot response_times.dat using 1:2 with points lc rgb "#33ccff", 800 t "SLA" lw 2 lc rgb "#ff0000" Median? 95th Percentile?
    • # access.logweb0060 66.249.71.110 - - [11/May/2011:17:08:53 +0000] "GET /listing/12189259/tropical-etched-pair-of-lampwork-glass HTTP/1.1" 200 11034 "-""Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" desktop - 13399576 505780 554876$ exercise_for_reader.sh# response_median_perc95.dat22/May/2012:15:15:00 417 62022/May/2012:15:16:00 309 97322/May/2012:15:17:00 340 86022/May/2012:15:18:00 303 416
    • # response_median_perc95.dat22/May/2012:15:15:00 417 62022/May/2012:15:16:00 309 97322/May/2012:15:17:00 340 86022/May/2012:15:18:00 303 416# trends.gpplot response_median_perc95.dat using 1:3 with filledcurve x1 linewidth 1 linecolor rgb "#548c6c" title "95th Percentile", "" using 1:2 with lines linewidth 2 linecolor rgb "#2192bf" title "Median", 800 title "SLA" linewidth 2 linecolor rgb "#f28705"
    • # trends.gpplot response_median_perc95.dat using 1:3 with filledcurve x1 linewidth 1 linecolor rgb "#0060ad" title "95th Percentile", "" using 1:2 with lines linewidth 2 linecolor rgb "#dd181f" title "Median", 800 title "SLA" linewidth 2 linecolor rgb "#f28705"
    • # access.logweb0060 66.249.71.110 - - [11/May/2011:17:08:53 +0000] "GET /listing/12189259/tropical-etched-pair-of-lampwork-glass HTTP/1.1" 200 11034 "-""Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" desktop - 13399576 505780 554876$ php data_buckets.php <filename> <bucket-size>
    • # access.logweb0060 66.249.71.110 - - [11/May/2011:17:08:53 +0000] "GET /listing/12189259/tropical-etched-pair-of-lampwork-glass HTTP/1.1" 200 11034 "-""Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" desktop - 13399576 505780 554876$ php data_buckets.php access.log 50 > response_histogram.dat# response_histogram.dat# bucket (ms), count0 050 0100 12150 412200 1203250 2312300 2078
    • #!/usr/local/bin/gnuplotset terminal pngcairo enhanced font "arial,11" size 800,600set output histogram.png# Graph formattingset title "Listings response times (ms)"set key offset yrange [0:*]# Histogram renderingset style data histogramsset style histogram clustered gap 1set style fill solid 1.00 border lt -1set boxwidth 0.9 absoluteplot response_histogram.dat using 2:xtic(1)
    • # histogram.gpplot response_histogram.dat using 2:xtic(1)
    • Warning for Mac / homebrew usersThe pngcairo terminal does not work with pango 1.30 (latest).Use 1.28 or 1.29 instead.set terminal pngcairo enhanced font "arial,11" size 800,600“Gnuplot pdfcairo problem”https://github.com/mxcl/homebrew/issues/11901
    • (Near) Real-Time Trending
    • LogsterRun by cron, maintaining a cursor on log filesSimple parsing & aggregationOutput to Ganglia, Graphite, stdoutCombined log, via syslog1 minute scheduleOutput to Graphite github.com/etsy/logster
    • Report script execution time (ms) segmented by signed-in/out visitors?web0020 0.0.0.0 - - [11/May/2011:17:08:54 +0000] "GET /listing/72395693/sterling-silver-ring-with-rainbow HTTP/1.1" 200 11893 "-" "facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)" desktop 1924124 10765744 320118 339176web0047 0.0.0.0 - - [11/May/2011:17:08:53 +0000] "GET /listing/31224920/flirt-14k-gold-filled-hook-earrings HTTP/1.1" 200 12339 "-" "Mozilla/5.0 (compatible;Googlebot/2.1; +http://www.google.com/bot.html)" desktop - 10813080 377090 402038web0031 0.0.0.0 - - [11/May/2011:17:08:53 +0000] "GET /listing/64397692/spades-and-diamonds-ooak-red-on-custom HTTP/1.1" 200 13835 "http://www.etsy.com/listing/72550205/krylon-dots-black-on-white-tank-womens-l" "Mozilla/5.0 (iPhone; U; CPU likeMac OS X; en) AppleWebKit/420+ (KHTML, like Gecko) Version/3.0 Mobile/1A543a Safari/419.3" mobile 8513282 15249296 563978 586038
    • web0020 0.0.0.0 - - [11/May/2011:17:08:54 +0000] "GET /listing/72395693/sterling-silver-ring-with-rainbow HTTP/1.1" 200 11893 "-" "facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)" desktop 1924124 10765744 320118 339176# Parse fields from log fileself.reg = re.compile(‘.* (?P<display_mode>w+) (?P<etsy_user_id>[d-]+)(?P<php_bytes>d+) (?P<php_usec>d+) (?P<apache_usec>d+)$’)
    • web0020 0.0.0.0 - - [11/May/2011:17:08:54 +0000] "GET /listing/72395693/sterling-silver-ring-with-rainbow HTTP/1.1" 200 11893 "-" "facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)" desktop 1924124 10765744 320118 339176# Collect data from each matching line# signed_out_ms and signed_in_ms are lists.fields = regMatch.groupdict()php_ms = int(fields[php_usec]) / 1000user_id = fields[user_id]if (user_id == "-"): self.signed_out_ms.append(php_ms)else: self.signed_in_ms.append(php_ms)
    • web0020 0.0.0.0 - - [11/May/2011:17:08:54 +0000] "GET /listing/72395693/sterling-silver-ring-with-rainbow HTTP/1.1" 200 11893 "-" "facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)" desktop 1924124 10765744 320118 339176return [ MetricObject("signed_in_reqs", len(self.signed_in_ms), "Requests"), MetricObject("signed_in_ms_min", self.signed_in_ms[0], "ms"), MetricObject("signed_in_ms_median", percentile(self.signed_in_ms, 0.5), "ms"), MetricObject("signed_in_ms_perc95", percentile(self.signed_in_ms, 0.95), "ms"), MetricObject("signed_in_ms_max", self.signed_in_ms[-1], "ms"),]
    • web0020 0.0.0.0 - - [11/May/2011:17:08:54 +0000] "GET /listing/72395693/sterling-silver-ring-with-rainbow HTTP/1.1" 200 11893 "-" "facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)" desktop 1924124 10765744 320118 339176$ logster --output stdout PerfLogster access.logsigned_out_reqs 8signed_out_ms_min 201signed_out_ms_median 227.5signed_out_ms_perc95 236.9signed_out_ms_max 239signed_in_reqs 3signed_in_ms_min 258signed_in_ms_median 268signed_in_ms_perc95 268.0signed_in_ms_max 268
    • web0020 0.0.0.0 - - [11/May/2011:17:08:54 +0000] "GET /listing/72395693/sterling-silver-ring-with-rainbow HTTP/1.1" 200 11893 "-" "facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)" desktop 1924124 10765744 320118 339176$ logster --output stdout PerfLogster access.log$ logster --output graphite --graphite-host graphite.example.com:2003 PerfLogster access.log
    • web0020 0.0.0.0 - - [11/May/2011:17:08:54 +0000] "GET /listing/72395693/sterling-silver-ring-with-rainbow HTTP/1.1" 200 11893 "-" "facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)" desktop 1924124 10765744 320118 339176$ logster --output stdout PerfLogster access.log$ logster --output graphite --graphite-host graphite.example.com:2003 PerfLogster access.log# Put that into a crontab to run every minute, and...
    • 600450300 MEDIAN PAGE GENERATION TIMES (MS) BY SIGNED-IN/SIGNED-OUT150 0
    • Thanks! Code samples and slides at bit.ly/velocity-etsy github.com/etsy/logster We are hiring... etsy.com/careers (or just come talk to me afterward)