The document discusses log files and how they can provide insights for search engine optimization. It provides examples of log file entries and explains what each part of the entry means. It also discusses how log files can be used to analyze search engine crawler behavior, popular pages, error codes, and desktop vs mobile traffic. Tips are provided like segmenting log file data and using it to debug analytics.

1. Log Files
The Overlooked Source of SEO Opportunities
Robin Rozhon

2. Why should care about how search
engines crawl my site?
@RobinRozhon

3. @RobinRozhon
Rossignol Soul 7 HD Skis - Men's - MEC
https://www.mec.ca › All › Snowsports › Backcountry skiing › Gear › Backcountry skis
★★★★★ Rating: 4.1 - 3 reviews - $799.00 - In stock
Soul 7 HD Skis: A fat freeride ski that can serve as your entire quiver. Floaty in powder with
crud-busting power, it's lightweight and easy to turn in all conditions. Powder-turn rocker.

4. 55%
@RobinRozhon
increase in the
revenue YOY

5. SEO crawlers don’t reveal the real
search engines behavior.
@RobinRozhon

6. What’s crawl budget?
@RobinRozhon

7. What’s crawl budget?
Crawl rate limit & Crawl demand
@RobinRozhon

8. Optimizing crawl budget for a 100
pages website doesn’t make sense.
@RobinRozhon

10. The hardest thing?
Getting access to log files.
@RobinRozhon

11. @RobinRozhon

12. Breakdown of a log file

13. A log file is a recording of everything
that goes in and out of a server.
@RobinRozhon

14. 181.224.137.56 - - [28/Feb/2018:20:11:07 -0600] "POST /wp-cron.php?doing_wp_cron=1519870267.7953920364379882812500 HTTP/1.0"
200 - "https://rozhon.com/wp-cron.php?doing_wp_cron=1519870267.7953920364379882812500" "WordPress/4.9.4; https://rozhon.com"
2600:3c03::f03c:91ff:fe08:7a61 - - [28/Feb/2018:20:11:07 -0600] "GET /feed/ HTTP/1.0" 200 6212 "-" "Mozilla/5.0 (Macintosh;
Intel Mac OS X 10_8_2; +feeder.co/crawler) AppleWebKit/537.31 (KHTML, like Gecko) Chrome/37.0.2062.120 Safari/537.36"
66.249.65.104 - - [28/Feb/2018:20:11:29 -0600] "GET / HTTP/1.0" 200 49415 "-" "Mozilla/5.0 (compatible; Googlebot/2.1;
+http://www.google.com/bot.html)"
181.224.137.56 - - [28/Feb/2018:20:21:11 -0600] "POST /wp-cron.php?doing_wp_cron=1519870871.4666459560394287109375 HTTP/1.0"
200 - "https://rozhon.com/wp-cron.php?doing_wp_cron=1519870871.4666459560394287109375" "WordPress/4.9.4; https://rozhon.com"
2600:3c03::f03c:91ff:fe08:7a61 - - [28/Feb/2018:20:21:11 -0600] "GET /feed/ HTTP/1.0" 200 6212 "-" "Mozilla/5.0 (Macintosh;
Intel Mac OS X 10_8_2; +feeder.co/crawler) AppleWebKit/537.31 (KHTML, like Gecko) Chrome/37.0.2062.120 Safari/537.36"
148.64.56.79 - - [28/Feb/2018:20:30:18 -0600] "GET /robots.txt HTTP/1.0" 200 67 "-" "Mozilla/5.0 (compatible;
GrapeshotCrawler/2.0; +http://www.grapeshot.co.uk/crawler.php)"
148.64.56.79 - - [28/Feb/2018:20:30:21 -0600] "GET /blog/crawling-indexing-technical-seo-basics-that-drive-revenue/ HTTP/1.0"
200 45892 "-" "Mozilla/5.0 (compatible; GrapeshotCrawler/2.0; +http://www.grapeshot.co.uk/crawler.php)"
181.224.137.56 - - [28/Feb/2018:20:31:14 -0600] "POST /wp-cron.php?doing_wp_cron=1519871474.0248880386352539062500 HTTP/1.0"
200 - "https://rozhon.com/wp-cron.php?doing_wp_cron=1519871474.0248880386352539062500" "WordPress/4.9.4; https://rozhon.com"
2600:3c03::f03c:91ff:fe08:7a61 - - [28/Feb/2018:20:31:13 -0600] "GET /feed/ HTTP/1.0" 200 6212 "-" "Mozilla/5.0 (Macintosh;
Intel Mac OS X 10_8_2; +feeder.co/crawler) AppleWebKit/537.31 (KHTML, like Gecko) Chrome/37.0.2062.120 Safari/537.36"
181.224.137.56 - - [28/Feb/2018:20:32:44 -0600] "POST /wp-cron.php?doing_wp_cron=1519871564.8240199089050292968750 HTTP/1.0"
200 - "https://rozhon.com/wp-cron.php?doing_wp_cron=1519871564.8240199089050292968750" "WordPress/4.9.4; https://rozhon.com"
8.29.198.26 - - [28/Feb/2018:20:32:44 -0600] "GET /feed/ HTTP/1.0" 200 6212 "-" "Feedly/1.0
(+http://www.feedly.com/fetcher.html; like FeedFetcher-Google)"
2601:600:997f:fcfe:1827:adb1:b306:c39a - - [28/Feb/2018:20:35:36 -0600] "GET
/blog/crawling-indexing-technical-seo-basics-that-drive-revenue/?utm_source=moztop10&utm_medium=email&utm_campaign=moztop10&_h
senc=p2ANqtz-94WR530ovn2zygDfHyy9vLK8rL7eVPTdTPdFlXy9C4zZ9tTHGMZeHFzP-vHe-6mUdyxC-z1xYa2kZ1ZdWDkwE6sxt0MA&_hsmi=60846479
HTTP/1.0" 200 46022 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/64.0.3282.167
Safari/537.36"
2601:600:997f:fcfe:1827:adb1:b306:c39a - - [28/Feb/2018:20:35:36 -0600] "GET
/wp-content/plugins/yet-another-related-posts-plugin/style/widget.css?ver=ebb0b10aabc902f625c6c73e10147729 HTTP/1.0" 200 384
"https://rozhon.com/blog/crawling-indexing-technical-seo-basics-that-drive-revenue/?utm_source=moztop10&utm_medium=email&utm_c
ampaign=moztop10&_hsenc=p2ANqtz-94WR530ovn2zygDfHyy9vLK8rL7eVPTdTPdFlXy9C4zZ9tTHGMZeHFzP-vHe-6mUdyxC-z1xYa2kZ1ZdWDkwE6sxt0MA&_
hsmi=60846479" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/64.0.3282.167
Safari/537.36"
2601:600:997f:fcfe:1827:adb1:b306:c39a - - [28/Feb/2018:20:35:36 -0600] "GET
/wp-content/plugins/jquery-smooth-scroll/css/jss-style.css?ver=ebb0b10aabc902f625c6c73e10147729 HTTP/1.0" 200 372
"https://rozhon.com/blog/crawling-indexing-technical-seo-basics-that-drive-revenue/?utm_source=moztop10&utm_medium=email&utm_c
ampaign=moztop10&_hsenc=p2ANqtz-94WR530ovn2zygDfHyy9vLK8rL7eVPTdTPdFlXy9C4zZ9tTHGMZeHFzP-vHe-6mUdyxC-z1xYa2kZ1ZdWDkwE6sxt0MA&_
hsmi=60846479" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/64.0.3282.167
Safari/537.36"
1.
2.
3.
4.
5.
6.
7.
8.
9.
10.
11.
12.
13.
14.

15. 66.249.65.107 - - [08/Dec/2017:04:54:20 -0400] "GET /contact/
HTTP/1.1" 200 11179 "-" "Mozilla/5.0 (compatible; Googlebot/2.1;
+http://www.google.com/bot.html)"
@RobinRozhon

16. 66.249.65.107 - - [08/Dec/2017:04:54:20 -0400] "GET /contact/
HTTP/1.1" 200 11179 "-" "Mozilla/5.0 (compatible; Googlebot/2.1;
+http://www.google.com/bot.html)"
@RobinRozhon
IP Address (WHO)

17. 66.249.65.107 - - [08/Dec/2017:04:54:20 -0400] "GET /contact/
HTTP/1.1" 200 11179 "-" "Mozilla/5.0 (compatible; Googlebot/2.1;
+http://www.google.com/bot.html)"
@RobinRozhon
Timestamp (WHEN)

18. 66.249.65.107 - - [08/Dec/2017:04:54:20 -0400] "GET /contact/
HTTP/1.1" 200 11179 "-" "Mozilla/5.0 (compatible; Googlebot/2.1;
+http://www.google.com/bot.html)"
@RobinRozhon
Access Request (WHAT)

19. 66.249.65.107 - - [08/Dec/2017:04:54:20 -0400] "GET /contact/
HTTP/1.1" 200 11179 "-" "Mozilla/5.0 (compatible; Googlebot/2.1;
+http://www.google.com/bot.html)"
@RobinRozhon
Status Code (RESULT)

20. 66.249.65.107 - - [08/Dec/2017:04:54:20 -0400] "GET /contact/
HTTP/1.1" 200 11179 "-" "Mozilla/5.0 (compatible; Googlebot/2.1;
+http://www.google.com/bot.html)"
@RobinRozhon
Bytes Transferred (SIZE)

21. 66.249.65.107 - - [08/Dec/2017:04:54:20 -0400] "GET /contact/
HTTP/1.1" 200 11179 "-" "Mozilla/5.0 (compatible; Googlebot/2.1;
+http://www.google.com/bot.html)"
@RobinRozhon
Referrer URL (SOURCE)

22. 66.249.65.107 - - [08/Dec/2017:04:54:20 -0400] "GET /contact/
HTTP/1.1" 200 11179 "-" "Mozilla/5.0 (compatible; Googlebot/2.1;
+http://www.google.com/bot.html)"
@RobinRozhon
User Agent (SIGNATURE)

23. @RobinRozhon

24. @RobinRozhon
66.249.65.107 - - [08/Dec/2017:04:54:20 -0400] "GET /contact/ ...

25. @RobinRozhon
66.249.65.107 - - [08/Dec/2017:04:54:20 -0400] "GET /contact/ ...
66.249.65.107 >> crawl-66-249-65-107.googlebot.com
Rev D Lo k

26. @RobinRozhon
66.249.65.107 - - [08/Dec/2017:04:54:20 -0400] "GET /contact/ ...
66.249.65.107 >> crawl-66-249-65-107.googlebot.com
Rev D Lo k
For d D Lo k

27. @RobinRozhon
66.249.65.107 - - [08/Dec/2017:04:54:20 -0400] "GET /contact/ ...
66.249.65.107 >> crawl-66-249-65-107.googlebot.com
Rev D Lo k
For d D Lo k
crawl-66-249-65-107.googlebot.com >> 66.249.65.107

28. @RobinRozhon
66.249.65.107 - - [08/Dec/2017:04:54:20 -0400] "GET /contact/ ...
66.249.65.107 >> crawl-66-249-65-107.googlebot.com
crawl-66-249-65-107.googlebot.com >> 66.249.65.107
Rev D Lo k
For d D Lo k

29. @RobinRozhon
Which search engines crawl my website?
Google
Bing
Yandex
Baidu
Seznam

30. @RobinRozhon
Which URLs are crawled most often?
/
/products/cycling
/products/snowsports
/product/intense-bike
/contact

31. @RobinRozhon
Which content types are crawled most often?
homepage
pdp
plp
content

32. @RobinRozhon
Which status codes are returned?
200
301
404
500
410

33. @RobinRozhon
Which status codes are returned?
200
301
404
500
410

34. @RobinRozhon
Which URLs return 404?
/en/brand/castelli+continental+crank-brothers+fizik+smith+oakley+spy_plus_/helmets/c/1221?sort=newest 188
/en/avalanche-beacons-and-transceivers/c/1112 183
/en/water-and-board-shorts/c/1666 104
/en/brand/castelli+crank-brothers+oakley+spy_plus_/helmets/helmet-covers-and-accessories/c/1223?sort=newest 102

35. desktop crawler
smartphone crawler
@RobinRozhon
80% 20%
Desktop-first

36. desktop crawler
smartphone crawler
@RobinRozhon
80%20%
Mobile-first

37. Tips

38. Segment data
@RobinRozhon
1)

39. Merge log files with other
data sources
@RobinRozhon
2)

40. Use log file to debug
Google Analytics
@RobinRozhon
3)

41. @RobinRozhon
Mountain Equipment Co-op – MEC – Shop climbing, cycling, running ...
https://www.mec.ca/en/
Canada's go-to place for outdoor gear, know-how and inspiration, MEC is a co-op owned
by the people who shop here. A lifetime membership is $5.

42. @RobinRozhon
Mountain Equipment Co-op – MEC – Shop climbing, cycling, running ...
https://www.mec.ca/
Canada's go-to place for outdoor gear, know-how and inspiration, MEC is a co-op owned
by the people who shop here. A lifetime membership is $5.

43. @RobinRozhon

44. @RobinRozhon

45. @RobinRozhon
Request: https://www.mec.ca/en/
Response: 200 (success)
Referrer: https://www.google.ca/

46. @RobinRozhon
Request: https://www.mec.ca/en/
Response: 200 (success)
Referrer: https://www.google.ca/
Organic Traffic

47. @RobinRozhon
Request: https://www.mec.ca/
Response: 302 (redirect)
Referrer: https://www.google.ca/
Request: https://www.mec.ca/en/
Response: 200 (success)
Referrer: https://www.google.ca/
Organic Traffic

48. @RobinRozhon
Request: https://www.mec.ca/
Response: 302 (redirect)
Referrer: https://www.google.ca/
Request: https://www.mec.ca/en/
Response: 200 (success)
Referrer: https://www.mec.ca/
Request: https://www.mec.ca/en/
Response: 200 (success)
Referrer: https://www.google.ca/
Organic Traffic

49. @RobinRozhon
Request: https://www.mec.ca/
Response: 302 (redirect)
Referrer: https://www.google.ca/
Request: https://www.mec.ca/en/
Response: 200 (success)
Referrer: https://www.mec.ca/
Direct Traffic
Request: https://www.mec.ca/en/
Response: 200 (success)
Referrer: https://www.google.ca/
Organic Traffic

50. Web Application Firewall
(WAF) disabled.
@RobinRozhon

51. Request: https://www.mec.ca/
Response: 302 (redirect)
Referrer: https://www.google.ca/
@RobinRozhon
Request: https://www.mec.ca/
Response: 302 (redirect)
Referrer: https://www.google.ca/
Request: https://www.mec.ca/en/
Response: 200 (success)
Referrer: https://www.mec.ca/
Direct Traffic
Request: https://www.mec.ca/en/
Response: 200 (success)
Referrer: https://www.google.ca/
Organic Traffic

52. Request: https://www.mec.ca/
Response: 302 (redirect)
Referrer: https://www.google.ca/
@RobinRozhon
Request: https://www.mec.ca/en/
Response: 200 (success)
Referrer: https://www.google.ca/
Request: https://www.mec.ca/
Response: 302 (redirect)
Referrer: https://www.google.ca/
Request: https://www.mec.ca/en/
Response: 200 (success)
Referrer: https://www.mec.ca/
Direct Traffic
Request: https://www.mec.ca/en/
Response: 200 (success)
Referrer: https://www.google.ca/
Organic Traffic

53. Request: https://www.mec.ca/
Response: 302 (redirect)
Referrer: https://www.google.ca/
@RobinRozhon
Request: https://www.mec.ca/en/
Response: 200 (success)
Referrer: https://www.google.ca/
Organic Traffic
Request: https://www.mec.ca/
Response: 302 (redirect)
Referrer: https://www.google.ca/
Request: https://www.mec.ca/en/
Response: 200 (success)
Referrer: https://www.mec.ca/
Direct Traffic
Request: https://www.mec.ca/en/
Response: 200 (success)
Referrer: https://www.google.ca/
Organic Traffic

54. Thank you!
@RobinRozhon
rozhon.com/statcitycrawl2018