Como a web funciona e como utilizamos desses conhecimentos para escrever web crawlers.

1. How web crawlers work

2. Gabriel Genê
Software Developer @Jusbrasil
/gabrielgene

3. How the internet works
ServersClients
Internet
(The Web)
(TCP/IP Network)

4. HyperText Transfer Protocol (HTTP)
HTTP Clients
(Web Browser)
HTTP
Request Message
HTTP
Response Message
HTTP Servers
(Web Server)

5. Browser
Client (Browser)
GET URL HTTP/1.1
Host : host:port
HTTP/1.1 200 OK
Server (host:port)
(1) User issues URL from a browser
http://host:port/path/file
(2) Browser sends a request message
(5) Browser formats the
response and displays
(4) Server returns a
response message
(3) Server maps the URL to a
file or program under the
document directory.

6. HTTP Request and Response Messages
hhhhhhhhhhhhhhhh
hhhhhhhhhhhhhhhh
hhhhhhhhhhhhhhhh
bbbbbbbbbbbbbbb
bbbbbbbbbbbbbbb
bbbbbbbbbbbbbbb
bbbbbbbbbbbbbbb
bbbbbbbbbbbbbbb
bbbbbbbbbbbbb
Message Header
A blank line separates the header and body
Message Body (optional)

7. HTTP Request Message Example
GET /doc/book.html HTTP/1.1
Host: www.test101.com
Accept: text/html, */*
Accept-Language: en-us
Accept-Encoding: gzip
User-Agent: Mozilla/4.0
Content-Length: 35
bookId=12345&author=Tan+Ah+Teck
Request Line
A blank line separates the header & body
Request Message Body
Request Headers
Request
Message
Header

8. HTTP Response Message Example
HTTP/1.1 200 OK
Date: Sun, 22 Nov xxxx 01:11:12 GMT
Server: Apache/1.3.29 (Win32)
Content-Length: 35
Content-Type: text/html
<h1>Book 12345</h1>
Status Line
A blank line separates the header & body
Response Message Body
Response Headers
Response
Message
Header

9. HTTP Request Methods
● GET: A client can use the GET request to get a web resource from the
server.
● POST: Used to post data up to the web server.
● PATCH: Ask the server to change the data.
● DELETE: Ask the server to delete the data.

10. How web sites render your data

11. ServerClient
Server side rendering

12. ServerClient
Client side rendering

13. What is a crawler?

16. Search Engine Optimization

18. Google
Crawler
Algorithm
Rank
Your site

19. Crawler Types
● Reverse Engineering
● Headless Browser