Simple Log Analysis and Trending

Simple Log Analysis and Trending Mike Brittain DIRECTOR OF ENGINEERING, ETSY @mikebrittain EXAMPLES: bit.ly/velocity-etsy

sa=t&source=web&cd=18&ved=0CHkQFjAR&url=http%3A%2F%2Fwww.etsy.com%2Flisting%2F45184569%2Fchanel-vintage-navy-blue-cotton-dress-w&rct=j&q=vintage%20chanel%20dress&ei=fMLKTZH0MYTL0AGjpsCDCg&usg=AFQjCNHnkswNxqzqi-Rl6nLVIr7dTg78Rg" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; WOW64; Trident/4.0; GTB7.0; SLCC2; .NET CLR 2.0.50727; .3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; MDDR; .NET4.0C)" - ur9XOgyn58I8opUn7LDhyqPliXap www.etsy.com ab_new_header=1|ab_db_log_long_queries=1|ab_rawsql_close_statement=1|ab_bank_transfer=0|ab_listing_favorite_nag=none|ab_shop_chat=0|ab_share2=1|ab_header_css=0|ab_inline_listing_hearts=1|ab_search_shops=1|ab_community_nav_popout=1|ab_boomerang=0|ab_shopstats_gathering=0 te0sXDV0SfwBpU-qL0w0TnAUIw-x 0 - - 11567688 601120 637356web0037 1.22.333.444 - - [11/May/2011:17:08:53 +0000] "GET /listing/60547985/half-yard-little-girls-in-kimono-print HTTP/1.1" 200 12389 "http://www.etsy.com/shop/beautifulwork?section_id=5810576&page=2" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; WOW64; Trident/4.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6- LVTY9r_Z_IEcEbfKWUJoPTAGHhBx www.etsy.com ab_new_header=1|ab_db_log_long_queries=1|ab_rawsql_close_statement=1|ab_bank_transfer=0|ab_listing_favorite_nag=none|ab_shop_chat=0|ab_sharab_header_css=0|ab_inline_listing_hearts=1|ab_search_defaults_allitems=1|ab_search_shops=1|ab_community_nav_popout=1|ab_boomerang=0|ab_shopstats_gathering=0 i57VnazHMo2LEjii5IRjaiaWVO412157 437768web0039 1.22.333.444 - - [11/May/2011:17:08:53 +0000] "GET /listing/71705428/fearfully-and-wonderfully-made-8x10?ref=sr_list_10&ga_search_query=pink+and+green&ga_noautofacet=1&ga_search_type=handmade&ga_facet=handmade%2Fart HTTP/1.1" 200 13232 "http://www.etsy.com/search/handmade/art?q=pink+and+green&noautofacet=1" "Mozilla/5.0 (WindAppleWebKit/534.24 (KHTML, like Gecko) Chrome/11.0.696.65 Safari/534.24" - aKR4B5QW0KrN-NBZpWvm0AgkwcD3 www.etsy.com ab_new_header=1|ab_db_log_long_queries=1|ab_rawsql_close_statementab_bank_transfer=0|ab_listing_favorite_nag=none|ab_shop_chat=0|ab_share2=1|ab_header_css=0|ab_inline_listing_hearts=1|ab_search_shops=1|ab_community_nav_popout=1|ab_shopstats_gatherin1mFgWOMRaRG5RN1zcfNZRVfdaDz7 7323622 - - 15124584 662683 690754web0047 1.22.333.444 - - [11/May/2011:17:08:53 +0000] "GET /listing/31224920/flirt-14k-gold-filled-hook-earrings?ref=sr_gallery_34&ga_noautofacet=1&ga_search_type=handmade&ga_facet=ha%2Fearrings%2Fpost&ga_search_query= HTTP/1.1" 200 12339 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" - Y8PI3hOiKDJWNz7M_-T_t4mq75zM www.etsy.com ab_nab_db_log_long_queries=1|ab_rawsql_close_statement=1|ab_bank_transfer=0|ab_listing_favorite_nag=none|ab_shop_chat=0|ab_share2=1|ab_header_css=0|ab_inline_listing_hearts=1|ab_search_defaults_allitems=0|ab_search_shops=1|ab_community_nav_popout=1|ab_boomerang=0|ab_shopstats_gathering=0 L6iJ-8gaox1rUhd8F8CVbfhLrMnJ 0 - - 10813080 377090 402038web0020 1.22.333.444 - - [11/May/2011:17:08:54 +0000] "GET /listing/72395693/sterling-silver-ring-with-rainbow HTTP/1.1" 200 11893 "-" "facebookexternalhit/1.1 (+http://www.facebook.cexternalhit_uatext.php)" - phQhH2hNgWf3AfCQH0SpLqoFL6I1 www.etsy.com ab_new_header=1|ab_db_log_long_queries=1|ab_rawsql_close_statement=1|ab_bank_transfer=0|ab_listing_favorite_nag=no Aab_shop_chat=0|ab_share2=1|ab_header_css=0|ab_inline_listing_hearts=1|ab_search_defaults_allitems=0|ab_search_shops=1|ab_community_nav_popout=1|ab_boomerang=0|ab_shopstats_gathering=0Vs1ZXSz3z9fZ 0 - - 10765744 320118 339176web0031 1.22.333.444 - - [11/May/2011:17:08:53 +0000] "GET /listing/64397692/spades-and-diamonds-ooak-red-on-custom?ref=v1_other_2 HTTP/1.1" 200 13835 "http://www.etsy.com/listing/725dots-black-on-white-tank-womens-l?ref=v1_other_2" "Mozilla/5.0 (Macintosh; U; PPC Mac OS X 10.4; en-US; rv:1.9.2.17) Gecko/20110420 Firefox/3.6.17" - vu5b_1x-w87dTJ-NEVYyISdHo0Qf www.ab_new_header=1|ab_db_log_long_queries=1|ab_rawsql_close_statement=1|ab_bank_transfer=0|ab_listing_favorite_nag=none|ab_shop_chat=0|ab_share2=1|ab_header_css=0|ab_inline_listing_heartab_search_shops=1|ab_community_nav_popout=1|ab_your_account_nav_popout=1|ab_new_listing_process=0|ab_boomerang=0|ab_shopstats_gathering=0 eNFgDvlRp9qHk8BYw1skM_efLpik 10551287 - - 152586038web0041 1.22.333.444 - - [11/May/2011:17:08:53 +0000] "GET /listing/70834087/custom-engraved-vintage-spoon-bracelet?ref=sr_gallery_16&ga_search_query=wedding&ga_page=20&ga_search_type=handmade&ga_facet=handmade HTTP/1.1" 200 14874 "http://www.etsy.com/search/handmade?q=wedding&page=20" "Mozilla/5.0Intel Mac OS X 10_6_7; en-us) AppleWebKit/533.20.25 (KHTML, like Gecko) Version/5.0.4 Safari/533.20.27" - s3dAEruW9OGxqsGII6dZFpVMdOo6 www.etsy.com ab_new_header=1|ab_db_log_long_querab_rawsql_close_statement=1|ab_bank_transfer=0|ab_listing_favorite_nag=none|ab_shop_chat=0|ab_share2=1|ab_header_css=0|ab_inline_listing_hearts=1|ab_search_shops=1|ab_community_nav_poab_your_account_nav_popout=1|ab_boomerang=0|ab_shopstats_gathering=0 SHyrUYHqVMKD5KER-cJtiSR6mz9i 14458253 - - 12412104 391399 407669web0057 1.22.333.444 - - [11/May/2011:17:08:53 +0000] "GET /listing/45184569/chanel-vintage-navy-blue-cotton-dress-w HTTP/1.1" 200 12071 "http://www.google.com/url?sa=t&source=web&cd=18&ved=0CHkQFjAR&url=http%3A%2F%2Fwww.etsy.com%2Flisting%2F45184569%2Fchanel-vintage-navy-blue-cotton-dress-w&rct=j&q=vintage%20chanel%20dress&ei=fMLKTZH0MYTL0AGjpsCDCg&usg=AFQjCNHnkswNxqzqi-Rl6nLVIr7dTg78Rg" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; WOW64; Trident/4.0; GTB7.0; SLCC2; .NET CLR 2.0.50727; .3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; MDDR; .NET4.0C)" - ur9XOgyn58I8opUn7LDhyqPliXap www.etsy.com ab_new_header=1|ab_db_log_long_queries=1|ab_rawsql_close_statement=1|ab_bank_transfer=0|ab_listing_favorite_nag=none|ab_shop_chat=0|ab_share2=1|ab_header_css=0|ab_inline_listing_hearts=1|ab_search_shops=1|ab_community_nav_popout=1|ab_boomerang=0|ab_shopstats_gathering=0 te0sXDV0SfwBpU-qL0w0TnAUIw-x 0 - - 11567688 601120 637356web0037 1.22.333.444 - - [11/May/2011:17:08:53 +0000] "GET /listing/60547985/half-yard-little-girls-in-kimono-print HTTP/1.1" 200 12389 "http://www.etsy.com/shop/beautifulwork?section_id=5810576&page=2" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; WOW64; Trident/4.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6- LVTY9r_Z_IEcEbfKWUJoPTAGHhBx www.etsy.com ab_new_header=1|ab_db_log_long_queries=1|ab_rawsql_close_statement=1|ab_bank_transfer=0|ab_listing_favorite_nag=none|ab_shop_chat=0|ab_sharab_header_css=0|ab_inline_listing_hearts=1|ab_search_defaults_allitems=1|ab_search_shops=1|ab_community_nav_popout=1|ab_boomerang=0|ab_shopstats_gathering=0 i57VnazHMo2LEjii5IRjaiaWVO412157 437768web0039 1.22.333.444 - - [11/May/2011:17:08:53 +0000] "GET /listing/71705428/fearfully-and-wonderfully-made-8x10?ref=sr_list_10&ga_search_query=pink+and+green&ga_noautofacet=1&ga_search_type=handmade&ga_facet=handmade%2Fart HTTP/1.1" 200 13232 "http://www.etsy.com/search/handmade/art?q=pink+and+green&noautofacet=1" "Mozilla/5.0 (WindAppleWebKit/534.24 (KHTML, like Gecko) Chrome/11.0.696.65 Safari/534.24" - aKR4B5QW0KrN-NBZpWvm0AgkwcD3 www.etsy.com ab_new_header=1|ab_db_log_long_queries=1|ab_rawsql_close_statementab_bank_transfer=0|ab_listing_favorite_nag=none|ab_shop_chat=0|ab_share2=1|ab_header_css=0|ab_inline_listing_hearts=1|ab_search_shops=1|ab_community_nav_popout=1|ab_shopstats_gatherin

PAGE GENERATION TIME (MS)600450300 B150 0

sa=t&source=web&cd=18&ved=0CHkQFjAR&url=http%3A%2F%2Fwww.etsy.com%2Flisting%2F45184569%2Fchanel-vintage-navy-blue-cotton-dress-w&rct=j&q=vintage%20chanel%20dress&ei=fMLKTZH0MYTL0AGjpsCDCg&usg=AFQjCNHnkswNxqzqi-Rl6nLVIr7dTg78Rg" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; WOW64; Trident/4.0; GTB7.0; SLCC2; .NET CLR 2.0.50727; .3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; MDDR; .NET4.0C)" - ur9XOgyn58I8opUn7LDhyqPliXap www.etsy.com ab_new_header=1|ab_db_log_long_queries=1|ab_rawsql_close_statement=1|ab_bank_transfer=0|ab_listing_favorite_nag=none|ab_shop_chat=0|ab_share2=1|ab_header_css=0|ab_inline_listing_hearts=1|ab_search_shops=1|ab_community_nav_popout=1|ab_boomerang=0|ab_shopstats_gathering=0 te0sXDV0SfwBpU-qL0w0TnAUIw-x 0 - - 11567688 601120 637356web0037 1.22.333.444 - - [11/May/2011:17:08:53 +0000] "GET /listing/60547985/half-yard-little-girls-in-kimono-print HTTP/1.1" 200 12389 "http://www.etsy.com/shop/beautifulwork?section_id=5810576&page=2" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; WOW64; Trident/4.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6- LVTY9r_Z_IEcEbfKWUJoPTAGHhBx www.etsy.com ab_new_header=1|ab_db_log_long_queries=1|ab_rawsql_close_statement=1|ab_bank_transfer=0|ab_listing_favorite_nag=none|ab_shop_chat=0|ab_sharab_header_css=0|ab_inline_listing_hearts=1|ab_search_defaults_allitems=1|ab_search_shops=1|ab_community_nav_popout=1|ab_boomerang=0|ab_shopstats_gathering=0 i57VnazHMo2LEjii5IRjaiaWVO412157 437768web0039 1.22.333.444 - - [11/May/2011:17:08:53 +0000] "GET /listing/71705428/fearfully-and-wonderfully-made-8x10?ref=sr_list_10&ga_search_query=pink+and+green&ga_noautofacet=1&ga_search_type=handmade&ga_facet=handmade%2Fart HTTP/1.1" 200 13232 "http://www.etsy.com/search/handmade/art?q=pink+and+green&noautofacet=1" "Mozilla/5.0 (WindAppleWebKit/534.24 (KHTML, like Gecko) Chrome/11.0.696.65 Safari/534.24" - aKR4B5QW0KrN-NBZpWvm0AgkwcD3 www.etsy.com ab_new_header=1|ab_db_log_long_queries=1|ab_rawsql_close_statementab_bank_transfer=0|ab_listing_favorite_nag=none|ab_shop_chat=0|ab_share2=1|ab_header_css=0|ab_inline_listing_hearts=1|ab_search_shops=1|ab_community_nav_popout=1|ab_shopstats_gatherin1mFgWOMRaRG5RN1zcfNZRVfdaDz7 7323622 - - 15124584 662683 690754web0047 1.22.333.444 - - [11/May/2011:17:08:53 +0000] "GET /listing/31224920/flirt-14k-gold-filled-hook-earrings?ref=sr_gallery_34&ga_noautofacet=1&ga_search_type=handmade&ga_facet=ha%2Fearrings%2Fpost&ga_search_query= HTTP/1.1" 200 12339 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" - Y8PI3hOiKDJWNz7M_-T_t4mq75zM www.etsy.com ab_nab_db_log_long_queries=1|ab_rawsql_close_statement=1|ab_bank_transfer=0|ab_listing_favorite_nag=none|ab_shop_chat=0|ab_share2=1|ab_header_css=0|ab_inline_listing_hearts=1|ab_search_defaults_allitems=0|ab_search_shops=1|ab_community_nav_popout=1|ab_boomerang=0|ab_shopstats_gathering=0 L6iJ-8gaox1rUhd8F8CVbfhLrMnJ 0 - - 10813080 377090 402038web0020 1.22.333.444 - - [11/May/2011:17:08:54 +0000] "GET /listing/72395693/sterling-silver-ring-with-rainbow HTTP/1.1" 200 11893 "-" "facebookexternalhit/1.1 (+http://www.facebook.cexternalhit_uatext.php)" - phQhH2hNgWf3AfCQH0SpLqoFL6I1 www.etsy.com ab_new_header=1|ab_db_log_long_queries=1|ab_rawsql_close_statement=1|ab_bank_transfer=0|ab_listing_favorite_nag=no Aab_shop_chat=0|ab_share2=1|ab_header_css=0|ab_inline_listing_hearts=1|ab_search_defaults_allitems=0|ab_search_shops=1|ab_community_nav_popout=1|ab_boomerang=0|ab_shopstats_gathering=0Vs1ZXSz3z9fZ 0 - - 10765744 320118 339176web0031 1.22.333.444 - - [11/May/2011:17:08:53 +0000] "GET /listing/64397692/spades-and-diamonds-ooak-red-on-custom?ref=v1_other_2 HTTP/1.1" 200 13835 "http://www.etsy.com/listing/725dots-black-on-white-tank-womens-l?ref=v1_other_2" "Mozilla/5.0 (Macintosh; U; PPC Mac OS X 10.4; en-US; rv:1.9.2.17) Gecko/20110420 Firefox/3.6.17" - vu5b_1x-w87dTJ-NEVYyISdHo0Qf www.ab_new_header=1|ab_db_log_long_queries=1|ab_rawsql_close_statement=1|ab_bank_transfer=0|ab_listing_favorite_nag=none|ab_shop_chat=0|ab_share2=1|ab_header_css=0|ab_inline_listing_heartab_search_shops=1|ab_community_nav_popout=1|ab_your_account_nav_popout=1|ab_new_listing_process=0|ab_boomerang=0|ab_shopstats_gathering=0 eNFgDvlRp9qHk8BYw1skM_efLpik 10551287 - - 152586038web0041 1.22.333.444 - - [11/May/2011:17:08:53 +0000] "GET /listing/70834087/custom-engraved-vintage-spoon-bracelet?ref=sr_gallery_16&ga_search_query=wedding&ga_page=20&ga_search_type=handmade&ga_facet=handmade HTTP/1.1" 200 14874 "http://www.etsy.com/search/handmade?q=wedding&page=20" "Mozilla/5.0Intel Mac OS X 10_6_7; en-us) AppleWebKit/533.20.25 (KHTML, like Gecko) Version/5.0.4 Safari/533.20.27" - s3dAEruW9OGxqsGII6dZFpVMdOo6 www.etsy.com ab_new_header=1|ab_db_log_long_querab_rawsql_close_statement=1|ab_bank_transfer=0|ab_listing_favorite_nag=none|ab_shop_chat=0|ab_share2=1|ab_header_css=0|ab_inline_listing_hearts=1|ab_search_shops=1|ab_community_nav_poab_your_account_nav_popout=1|ab_boomerang=0|ab_shopstats_gathering=0 SHyrUYHqVMKD5KER-cJtiSR6mz9i 14458253 - - 12412104 391399 407669web0057 1.22.333.444 - - [11/May/2011:17:08:53 +0000] "GET /listing/45184569/chanel-vintage-navy-blue-cotton-dress-w HTTP/1.1" 200 12071 "http://www.google.com/url?sa=t&source=web&cd=18&ved=0CHkQFjAR&url=http%3A%2F%2Fwww.etsy.com%2Flisting%2F45184569%2Fchanel-vintage-navy-blue-cotton-dress-w&rct=j&q=vintage%20chanel%20dress&ei=fMLKTZH0MYTL0AGjpsCDCg&usg=AFQjCNHnkswNxqzqi-Rl6nLVIr7dTg78Rg" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; WOW64; Trident/4.0; GTB7.0; SLCC2; .NET CLR 2.0.50727; .3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; MDDR; .NET4.0C)" - ur9XOgyn58I8opUn7LDhyqPliXap www.etsy.com ab_new_header=1|ab_db_log_long_queries=1|ab_rawsql_close_statement=1|ab_bank_transfer=0|ab_listing_favorite_nag=none|ab_shop_chat=0|ab_share2=1|ab_header_css=0|ab_inline_listing_hearts=1|ab_search_shops=1|ab_community_nav_popout=1|ab_boomerang=0|ab_shopstats_gathering=0 te0sXDV0SfwBpU-qL0w0TnAUIw-x 0 - - 11567688 601120 637356web0037 1.22.333.444 - - [11/May/2011:17:08:53 +0000] "GET /listing/60547985/half-yard-little-girls-in-kimono-print HTTP/1.1" 200 12389 "http://www.etsy.com/shop/beautifulwork?section_id=5810576&page=2" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; WOW64; Trident/4.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6- LVTY9r_Z_IEcEbfKWUJoPTAGHhBx www.etsy.com ab_new_header=1|ab_db_log_long_queries=1|ab_rawsql_close_statement=1|ab_bank_transfer=0|ab_listing_favorite_nag=none|ab_shop_chat=0|ab_sharab_header_css=0|ab_inline_listing_hearts=1|ab_search_defaults_allitems=1|ab_search_shops=1|ab_community_nav_popout=1|ab_boomerang=0|ab_shopstats_gathering=0 i57VnazHMo2LEjii5IRjaiaWVO412157 437768web0039 1.22.333.444 - - [11/May/2011:17:08:53 +0000] "GET /listing/71705428/fearfully-and-wonderfully-made-8x10?ref=sr_list_10&ga_search_query=pink+and+green&ga_noautofacet=1&ga_search_type=handmade&ga_facet=handmade%2Fart HTTP/1.1" 200 13232 "http://www.etsy.com/search/handmade/art?q=pink+and+green&noautofacet=1" "Mozilla/5.0 (WindAppleWebKit/534.24 (KHTML, like Gecko) Chrome/11.0.696.65 Safari/534.24" - aKR4B5QW0KrN-NBZpWvm0AgkwcD3 www.etsy.com ab_new_header=1|ab_db_log_long_queries=1|ab_rawsql_close_statementab_bank_transfer=0|ab_listing_favorite_nag=none|ab_shop_chat=0|ab_share2=1|ab_header_css=0|ab_inline_listing_hearts=1|ab_search_shops=1|ab_community_nav_popout=1|ab_shopstats_gatherin

PAGE GENERATION TIME (MS)600450300 B150 0

A Bweb0057 1.22.333.444 - - [11/May/2011:17:08:53 +0000] "GET /listing/45184569/chanel-vintage-navy-blue-cotton-dress-w HTTP/1.1" 200 12071 "http://www.google.com/url?sa=t&source=web&cd=18&ved=0CHkQFjAR&url=http%3A%2F%2Fwww.etsy.com%2Flisting%2F45184569%2Fchanel-vintage-navy-blue-cotton-dress-w&rct=j&q=vintage%20chanel%20dress&ei=fMLKTZH0MYTL0AGjpsCDCg&usg=AFQjCNHnkswNxqzqi-Rl6nLVIr7dTg78Rg" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; WOW64; Trident/4.0; GTB7.0; SLCC2; .NET CLR 2.0.3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0; MDDR; .NET4.0C)" - ur9XOgyn58I8opUn7LDhyqPliXap www.etsy.com ab_new_header=1|ab_db_log_long_queries=1|ab_rawsql_close_stateab_bank_transfer=0|ab_listing_favorite_nag=none|ab_shop_chat=0|ab_share2=1|ab_header_css=0|ab_inline_listing_hearts=1|ab_search_shops=1|ab_community_nav_popout=1|ab_boomerang=ab_shopstats_gathering=0 te0sXDV0SfwBpU-qL0w0TnAUIw-x 0 - - 11567688 601120 637356web0037 1.22.333.444 - - [11/May/2011:17:08:53 +0000] "GET /listing/60547985/half-yard-little-girls-in-kimono-print HTTP/1.1" 200 12389 "http://www.etsy.com/shop/beautifulworksection_id=5810576&page=2" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; WOW64; Trident/4.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Cen- LVTY9r_Z_IEcEbfKWUJoPTAGHhBx www.etsy.com ab_new_header=1|ab_db_log_long_queries=1|ab_rawsql_close_statement=1|ab_bank_transfer=0|ab_listing_favorite_nag=none|ab_shop_chat=0ab_header_css=0|ab_inline_listing_hearts=1|ab_search_defaults_allitems=1|ab_search_shops=1|ab_community_nav_popout=1|ab_boomerang=0|ab_shopstats_gathering=0 i57VnazHMo2LEjii5I412157 437768web0039 1.22.333.444 - - [11/May/2011:17:08:53 +0000] "GET /listing/71705428/fearfully-and-wonderfully-made-8x10?ref=sr_list_10&ga_search_query=pink+and+green&ga_noautofacet=1&ga_search_type=handmade&ga_facet=handmade%2Fart HTTP/1.1" 200 13232 "http://www.etsy.com/search/handmade/art?q=pink+and+green&noautofacet=1" "Mozilla/5AppleWebKit/534.24 (KHTML, like Gecko) Chrome/11.0.696.65 Safari/534.24" - aKR4B5QW0KrN-NBZpWvm0AgkwcD3 www.etsy.com ab_new_header=1|ab_db_log_long_queries=1|ab_rawsql_close_sab_bank_transfer=0|ab_listing_favorite_nag=none|ab_shop_chat=0|ab_share2=1|ab_header_css=0|ab_inline_listing_hearts=1|ab_search_shops=1|ab_community_nav_popout=1|ab_shopstats_1mFgWOMRaRG5RN1zcfNZRVfdaDz7 7323622 - - 15124584 662683 690754web0047 1.22.333.444 - - [11/May/2011:17:08:53 +0000] "GET /listing/31224920/flirt-14k-gold-filled-hook-earrings?ref=sr_gallery_34&ga_noautofacet=1&ga_search_type=handmade&ga_%2Fearrings%2Fpost&ga_search_query= HTTP/1.1" 200 12339 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" - Y8PI3hOiKDJWNz7M_-T_t4mq75zM www.etsy.ab_db_log_long_queries=1|ab_rawsql_close_statement=1|ab_bank_transfer=0|ab_listing_favorite_nag=none|ab_shop_chat=0|ab_share2=1|ab_header_css=0|ab_inline_listing_hearts=1|ab_search_defaults_allitems=0|ab_search_shops=1|ab_community_nav_popout=1|ab_boomerang=0|ab_shopstats_gathering=0 L6iJ-8gaox1rUhd8F8CVbfhLrMnJ 0 - - 10813080 377090 402038web0020 1.22.333.444 - - [11/May/2011:17:08:54 +0000] "GET /listing/72395693/sterling-silver-ring-with-rainbow HTTP/1.1" 200 11893 "-" "facebookexternalhit/1.1 (+http://www.faexternalhit_uatext.php)" - phQhH2hNgWf3AfCQH0SpLqoFL6I1 www.etsy.com ab_new_header=1|ab_db_log_long_queries=1|ab_rawsql_close_statement=1|ab_bank_transfer=0|ab_listing_favoritab_shop_chat=0|ab_share2=1|ab_header_css=0|ab_inline_listing_hearts=1|ab_search_defaults_allitems=0|ab_search_shops=1|ab_community_nav_popout=1|ab_boomerang=0|ab_shopstats_gatVs1ZXSz3z9fZ 0 - - 10765744 320118 339176web0031 1.22.333.444 - - [11/May/2011:17:08:53 +0000] "GET /listing/64397692/spades-and-diamonds-ooak-red-on-custom?ref=v1_other_2 HTTP/1.1" 200 13835 "http://www.etsy.com/lisdots-black-on-white-tank-womens-l?ref=v1_other_2" "Mozilla/5.0 (Macintosh; U; PPC Mac OS X 10.4; en-US; rv:1.9.2.17) Gecko/20110420 Firefox/3.6.17" - vu5b_1x-w87dTJ-NEVYyISdHoab_new_header=1|ab_db_log_long_queries=1|ab_rawsql_close_statement=1|ab_bank_transfer=0|ab_listing_favorite_nag=none|ab_shop_chat=0|ab_share2=1|ab_header_css=0|ab_inline_listiab_search_shops=1|ab_community_nav_popout=1|ab_your_account_nav_popout=1|ab_new_listing_process=0|ab_boomerang=0|ab_shopstats_gathering=0 eNFgDvlRp9qHk8BYw1skM_efLpik 10551287586038web0041 1.22.333.444 - - [11/May/2011:17:08:53 +0000] "GET /listing/70834087/custom-engraved-vintage-spoon-bracelet?ref=sr_gallery_16&ga_search_query=wedding&ga_page=20&ga_search_type=handmade&ga_facet=handmade HTTP/1.1" 200 14874 "http://www.etsy.com/search/handmade?q=wedding&page=20" "MozIntel Mac OS X 10_6_7; en-us) AppleWebKit/533.20.25 (KHTML, like Gecko) Version/5.0.4 Safari/533.20.27" - s3dAEruW9OGxqsGII6dZFpVMdOo6 www.etsy.com ab_new_header=1|ab_db_log_lab_rawsql_close_statement=1|ab_bank_transfer=0|ab_listing_favorite_nag=none|ab_shop_chat=0|ab_share2=1|ab_header_css=0|ab_inline_listing_hearts=1|ab_search_shops=1|ab_communitab_your_account_nav_popout=1|ab_boomerang=0|ab_shopstats_gathering=0 SHyrUYHqVMKD5KER-cJtiSR6mz9i 14458253 - - 12412104 391399 407669web0057 1.22.333.444 - - [11/May/2011:17:08:53 +0000] "GET /listing/45184569/chanel-vintage-navy-blue-cotton-dress-w HTTP/1.1" 200 12071 "http://www.google.com/url?

$ tail -f access.log

Plotting log data for ad hoc analysis and real-time trending EXAMPLES: bit.ly/velocity-etsy

Performance data belongs in your app logs (e.g. web server access log)

Access log formatTime, remote address, http method, request uri, referrer, user-agent,response size, response code, execution time, memory consumed,plus custom ﬁelds...• Signed-in/out (user_id vs. “-”)• display mode (“desktop” vs. “mobile”)• i10n/i18n (“en-US”)• A/B experiment buckets

apache_note(“custom_ﬁeld”, $whatever);LogFormat %l %t "%r" %>s %b "%{Referer}i""%{User-Agent}i" %{custom_field}n ...

apache_note(“custom_ﬁeld”, $whatever);LogFormat %l %t "%r" %>s %b "%{Referer}i""%{User-Agent}i" %{custom_field}n ...LogFormat "%{True-Client-IP}i %l %u %t "%r" %>s %b"%{Referer}i" "%{User-Agent}i"%{display_mode}n %{user_id}n %{php_bytes}n %{php_usec}n %D”

LogFormat "%{True-Client-IP}i %l %u %t "%r" %>s %b"%{Referer}i" "%{User-Agent}i"%{display_mode}n %{user_id}n %{php_bytes}n %{php_usec}n %D”web0060 66.249.71.110 - - [11/May/2011:17:08:53 +0000] "GET /listing/12189259/tropical-etched-pair-of-lampwork-glass HTTP/1.1" 200 11034 "-""Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" desktop - 13399576 505780 554876

We have logs. Cool.

Ad hoc analysis using the newest,hottest, bleeding-edge plotting tool.

gnuplot!

# access.logweb0060 66.249.71.110 - - [11/May/2011:17:08:53 +0000] "GET /listing/12189259/tropical-etched-pair-of-lampwork-glass HTTP/1.1" 200 11034 "-""Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" desktop - 13399576 505780 554876

# access.logweb0060 66.249.71.110 - - [11/May/2011:17:08:53 +0000] "GET /listing/12189259/tropical-etched-pair-of-lampwork-glass HTTP/1.1" 200 11034 "-""Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" desktop - 13399576 505780 554876$ cat access.log | awk /GET /listing/ {print $5, $(NF-1)/1000}

# access.logweb0060 66.249.71.110 - - [11/May/2011:17:08:53 +0000] "GET /listing/12189259/tropical-etched-pair-of-lampwork-glass HTTP/1.1" 200 11034 "-""Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" desktop - 13399576 505780 554876$ cat access.log | awk /GET /listing/ {print $5, $(NF-1)/1000} | sed "s/[//"

# access.logweb0060 66.249.71.110 - - [11/May/2011:17:08:53 +0000] "GET /listing/12189259/tropical-etched-pair-of-lampwork-glass HTTP/1.1" 200 11034 "-""Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" desktop - 13399576 505780 554876$ cat access.log | awk /GET /listing/ {print $5, $(NF-1)/1000} | sed "s/[//" > response_times.dat# response_times.dat11/May/2011:17:08:53 505.78011/May/2011:17:08:54 311.45511/May/2011:17:08:53 376.74511/May/2011:17:08:54 322.278

# response_times.dat11/May/2011:17:08:53 505.78011/May/2011:17:08:54 311.45511/May/2011:17:08:53 376.74511/May/2011:17:08:54 322.278# scatter.gpplot response_times.dat using 1:2 with points lc rgb "#33ccff", 800 t "SLA" lw 2 lc rgb "#f28705"

# scatter.gpplot response_times.dat using 1:2 with points lc rgb "#33ccff", 800 t "SLA" lw 2 lc rgb "#f28705"

#!/usr/local/bin/gnuplotset terminal pngcairo enhanced font "arial,11" size 800,600set output response_times.png# Graph formattingset title "Listings, generation times (ms)"set ylabel "millisec"set key offset yrange [0:2000]# Setup time formattingset xdata timeset timefmt "%d/%B/%Y:%H:%M:%S"set format x "%H:%M"plot response_times.dat using 1:2 with points lc rgb "#33ccff", 800 t "SLA" lw 2 lc rgb "#f28705"

# scatter.gpplot response_times.dat using 1:2 with points lc rgb "#33ccff", 800 t "SLA" lw 2 lc rgb "#ff0000" Median? 95th Percentile?

# access.logweb0060 66.249.71.110 - - [11/May/2011:17:08:53 +0000] "GET /listing/12189259/tropical-etched-pair-of-lampwork-glass HTTP/1.1" 200 11034 "-""Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" desktop - 13399576 505780 554876$ exercise_for_reader.sh# response_median_perc95.dat22/May/2012:15:15:00 417 62022/May/2012:15:16:00 309 97322/May/2012:15:17:00 340 86022/May/2012:15:18:00 303 416

# response_median_perc95.dat22/May/2012:15:15:00 417 62022/May/2012:15:16:00 309 97322/May/2012:15:17:00 340 86022/May/2012:15:18:00 303 416# trends.gpplot response_median_perc95.dat using 1:3 with filledcurve x1 linewidth 1 linecolor rgb "#548c6c" title "95th Percentile", "" using 1:2 with lines linewidth 2 linecolor rgb "#2192bf" title "Median", 800 title "SLA" linewidth 2 linecolor rgb "#f28705"

# trends.gpplot response_median_perc95.dat using 1:3 with filledcurve x1 linewidth 1 linecolor rgb "#0060ad" title "95th Percentile", "" using 1:2 with lines linewidth 2 linecolor rgb "#dd181f" title "Median", 800 title "SLA" linewidth 2 linecolor rgb "#f28705"

# access.logweb0060 66.249.71.110 - - [11/May/2011:17:08:53 +0000] "GET /listing/12189259/tropical-etched-pair-of-lampwork-glass HTTP/1.1" 200 11034 "-""Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" desktop - 13399576 505780 554876$ php data_buckets.php <filename> <bucket-size>

# access.logweb0060 66.249.71.110 - - [11/May/2011:17:08:53 +0000] "GET /listing/12189259/tropical-etched-pair-of-lampwork-glass HTTP/1.1" 200 11034 "-""Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" desktop - 13399576 505780 554876$ php data_buckets.php access.log 50 > response_histogram.dat# response_histogram.dat# bucket (ms), count0 050 0100 12150 412200 1203250 2312300 2078

#!/usr/local/bin/gnuplotset terminal pngcairo enhanced font "arial,11" size 800,600set output histogram.png# Graph formattingset title "Listings response times (ms)"set key offset yrange [0:*]# Histogram renderingset style data histogramsset style histogram clustered gap 1set style fill solid 1.00 border lt -1set boxwidth 0.9 absoluteplot response_histogram.dat using 2:xtic(1)

# histogram.gpplot response_histogram.dat using 2:xtic(1)

Warning for Mac / homebrew usersThe pngcairo terminal does not work with pango 1.30 (latest).Use 1.28 or 1.29 instead.set terminal pngcairo enhanced font "arial,11" size 800,600“Gnuplot pdfcairo problem”https://github.com/mxcl/homebrew/issues/11901

(Near) Real-Time Trending

LogsterRun by cron, maintaining a cursor on log ﬁlesSimple parsing & aggregationOutput to Ganglia, Graphite, stdoutCombined log, via syslog1 minute scheduleOutput to Graphite github.com/etsy/logster

Report script execution time (ms) segmented by signed-in/out visitors?web0020 0.0.0.0 - - [11/May/2011:17:08:54 +0000] "GET /listing/72395693/sterling-silver-ring-with-rainbow HTTP/1.1" 200 11893 "-" "facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)" desktop 1924124 10765744 320118 339176web0047 0.0.0.0 - - [11/May/2011:17:08:53 +0000] "GET /listing/31224920/flirt-14k-gold-filled-hook-earrings HTTP/1.1" 200 12339 "-" "Mozilla/5.0 (compatible;Googlebot/2.1; +http://www.google.com/bot.html)" desktop - 10813080 377090 402038web0031 0.0.0.0 - - [11/May/2011:17:08:53 +0000] "GET /listing/64397692/spades-and-diamonds-ooak-red-on-custom HTTP/1.1" 200 13835 "http://www.etsy.com/listing/72550205/krylon-dots-black-on-white-tank-womens-l" "Mozilla/5.0 (iPhone; U; CPU likeMac OS X; en) AppleWebKit/420+ (KHTML, like Gecko) Version/3.0 Mobile/1A543a Safari/419.3" mobile 8513282 15249296 563978 586038

web0020 0.0.0.0 - - [11/May/2011:17:08:54 +0000] "GET /listing/72395693/sterling-silver-ring-with-rainbow HTTP/1.1" 200 11893 "-" "facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)" desktop 1924124 10765744 320118 339176# Parse fields from log fileself.reg = re.compile(‘.* (?P<display_mode>w+) (?P<etsy_user_id>[d-]+)(?P<php_bytes>d+) (?P<php_usec>d+) (?P<apache_usec>d+)$’)

web0020 0.0.0.0 - - [11/May/2011:17:08:54 +0000] "GET /listing/72395693/sterling-silver-ring-with-rainbow HTTP/1.1" 200 11893 "-" "facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)" desktop 1924124 10765744 320118 339176# Collect data from each matching line# signed_out_ms and signed_in_ms are lists.fields = regMatch.groupdict()php_ms = int(fields[php_usec]) / 1000user_id = fields[user_id]if (user_id == "-"): self.signed_out_ms.append(php_ms)else: self.signed_in_ms.append(php_ms)

web0020 0.0.0.0 - - [11/May/2011:17:08:54 +0000] "GET /listing/72395693/sterling-silver-ring-with-rainbow HTTP/1.1" 200 11893 "-" "facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)" desktop 1924124 10765744 320118 339176return [ MetricObject("signed_in_reqs", len(self.signed_in_ms), "Requests"), MetricObject("signed_in_ms_min", self.signed_in_ms[0], "ms"), MetricObject("signed_in_ms_median", percentile(self.signed_in_ms, 0.5), "ms"), MetricObject("signed_in_ms_perc95", percentile(self.signed_in_ms, 0.95), "ms"), MetricObject("signed_in_ms_max", self.signed_in_ms[-1], "ms"),]

web0020 0.0.0.0 - - [11/May/2011:17:08:54 +0000] "GET /listing/72395693/sterling-silver-ring-with-rainbow HTTP/1.1" 200 11893 "-" "facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)" desktop 1924124 10765744 320118 339176$ logster --output stdout PerfLogster access.logsigned_out_reqs 8signed_out_ms_min 201signed_out_ms_median 227.5signed_out_ms_perc95 236.9signed_out_ms_max 239signed_in_reqs 3signed_in_ms_min 258signed_in_ms_median 268signed_in_ms_perc95 268.0signed_in_ms_max 268

web0020 0.0.0.0 - - [11/May/2011:17:08:54 +0000] "GET /listing/72395693/sterling-silver-ring-with-rainbow HTTP/1.1" 200 11893 "-" "facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)" desktop 1924124 10765744 320118 339176$ logster --output stdout PerfLogster access.log$ logster --output graphite --graphite-host graphite.example.com:2003 PerfLogster access.log

web0020 0.0.0.0 - - [11/May/2011:17:08:54 +0000] "GET /listing/72395693/sterling-silver-ring-with-rainbow HTTP/1.1" 200 11893 "-" "facebookexternalhit/1.1 (+http://www.facebook.com/externalhit_uatext.php)" desktop 1924124 10765744 320118 339176$ logster --output stdout PerfLogster access.log$ logster --output graphite --graphite-host graphite.example.com:2003 PerfLogster access.log# Put that into a crontab to run every minute, and...

600450300 MEDIAN PAGE GENERATION TIMES (MS) BY SIGNED-IN/SIGNED-OUT150 0

Thanks! Code samples and slides at bit.ly/velocity-etsy github.com/etsy/logster We are hiring... etsy.com/careers (or just come talk to me afterward)

http://www.dbthink.com	1119
http://www.mikebrittain.com	854
http://lanyrd.com	37
http://xianguo.com	25
https://twitter.com	23
http://feeds.feedburner.com	19
http://abtasty.com	4
https://si0.twimg.com	3
http://coderwall.com	2
http://www.tuicool.com	1
http://131.253.14.250	1
http://translate.googleusercontent.com	1
https://twimg0-a.akamaihd.net	1

Simple Log Analysis and Trending

by Mike Brittain on Jun 27, 2012

Accessibility

Categories

Upload Details

Usage Rights

13 Embeds 2,090

Statistics

Simple Log Analysis and Trending Presentation Transcript