How to start with scrappers part 2

1. • Captcha for learning and collecting data

2. GOOLE ENERGY REDUCTION

3. GOOLE ENERGY REDUCTION

4. RECAPTCHA

5. RECAPTCHA

6. RECAPTCHA

7. MACHINE LEARNING
web crawlers part 2

8. ISYOUR BROWSER UNIQUE?
• https://amiunique.org
• https://panopticlick.eff.org

9. ROBOTS.TXT
User-agent: *
Disallow: /cgi-bin/
Disallow: /tmp/
Disallow: /~joe/

10. SITEMAPS
<?xml version="1.0" encoding="UTF-8"?>
<urlset xmlns="http://www.sitemaps.org/schemas/sitemap/0.9">
<url>
<loc>http://www.example.com/</loc>
<lastmod>2005-01-01</lastmod>
<changefreq>monthly</changefreq>
<priority>0.8</priority>
</url>
</urlset>

11. CRAWLERTRAPS
• Captcha
• Hidden fields
• Hidden in CSS

12. ONION ;)
import socks
import socket
from urllib.request import urlopen
socks.set_default_proxy(socks.SOCKS5, "localhost", 9150)
socket.socket = socks.socksocket
print(urlopen('http://icanhazip.com').read())

13. URLOPEN
from urllib.request import urlopen
html = urlopen("http://sampleshop.pl/product/happy-ninja-2/")
print(html.read())

14. “BeautifulSoup tries to make sense of the nonsensical;
it helps format and organize the messy web by fixing
bad HTML and presenting us with easily-traversible
Python objects representing XML structures.”

15. BEAUTIFUL SOUP
from urllib.request import urlopen
from bs4 import BeautifulSoup
html = urlopen("http://sampleshop.pl")
bsObj = BeautifulSoup(html.read(), "html.parser")
print(bsObj.h1)

16. REQUESTS
import requests
from bs4 import BeautifulSoup
session = requests.Session()
headers = {
'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_0)
AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.116 Safari/537.36',
'Accept': ‘text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,
*/*;q=0.8'
}
url = "https://www.whatismybrowser.com/developers/
what-http-headers-is-my-browser-sending"
req = session.get(url, headers = headers)
bsObj = BeautifulSoup(req.text, "html.parser")
print(bsObj.find("table", {"class": "table-striped"}).get_text)

17. REQUESTS
import requests
from bs4 import BeautifulSoup
session = requests.Session()
headers = {
'User-Agent': 'Mozilla/5.0 (Macintosh; Intel Mac OS X 10_12_0)
AppleWebKit/537.36 (KHTML, like Gecko) Chrome/53.0.2785.116 Safari/537.36',
'Accept': ‘text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,
*/*;q=0.8'
}
url = "http://sampleshop.pl/shop"
req = session.get(url, headers = headers)
bsObj = BeautifulSoup(req.text, "html.parser")
print(bsObj.find("p", {"class": "site-description"}).get_text())