This document summarizes a presentation given by Alexander Sibiryakov about Frontera, an open source web crawling framework. Frontera allows building large-scale web crawlers that can crawl billions of pages per month in a distributed manner. It provides abstractions for crawling strategies, message buses, and backend storage. The document describes example uses of Frontera including focused crawls, news analysis, and due diligence. It also outlines the software and hardware requirements and discusses future plans for Frontera.

1. Frontera: open source, large scale web
crawling framework
Alexander Sibiryakov, May 20, 2016, PyData Berlin 2016
sibiryakov@scrapinghub.com

2. • Software Engineer @
Scrapinghub
• Born in Yekaterinburg, RU
• 5 years at Yandex, search
quality department: social and
QA search, snippets.
• 2 years at Avast! antivirus,
research team: automatic false
positive solving, large scale
prediction of malicious
download attempts.
About myself
2

3. • Over 2 billion requests per month
(~800/sec.)
• Focused crawls & Broad crawls
We help turn web content
into useful data
3
{
"content": [
{
"title": {
"text": "'Extreme poverty' to fall below
10% of world population for first time",
"href": "http://www.theguardian.com/
society/2015/oct/05/world-bank-extreme-
poverty-to-fall-below-10-of-world-population-
for-first-time"
},
"points": "9 points",
"time_ago": {
"text": "2 hours ago",
"href": "https://news.ycombinator.com/
item?id=10352189"
},
"username": {
"text": "hliyan",
"href": "https://news.ycombinator.com/
user?id=hliyan"
}
},

4. Broad crawl usages
4
Lead generation
(extracting contact
information)
News analysis
Topical crawling
Plagiarism detection
Sentiment analysis
(popularity, likability)
Due diligence (profile/
business data)
Track criminal activity & find lost persons (DARPA)

5. Saatchi Global Gallery Guide
www.globalgalleryguide.com
• Discover 11K online
galleries.
• Extract general
information, art samples,
descriptions.
• NLP-based extraction.
• Find more galleries on the
web.

6. Frontera recipes
• Multiple websites data collection automation
• «Grep» of the internet segment
• Topical crawling
• Extracting data from arbitrary document

7. Multiple websites data collection
automation
• Scrapers from multiple websites.
• Data items collected and updated.
• Frontera can be used to
• crawl in parallel and scale the process,
• schedule revisiting (within fixed time),
• prioritize the URLs during crawling.

8. «Grep» of the internet segment
• alternative to Google,
• collect the zone files from registrars
(.com/.net/.org),
• setup Frontera in distributed mode,
• implement text processing in spider code,
• output items with matched pages.

9. Topical crawling
• document topic classifier & seeds URL list,
• if document is classified as positive crawler ->
extracted links,
• Frontera in distributed mode,
• topic classifier code put in spider.
Extensions: link classifier, follow/final classifiers.

10. Extracting data from arbitrary
document
• Tough problem. Can’t be solved completely.
• Can be seen as a structured prediction problem:
• Conditional Random Fields (CRF) or
• Hidden Markov Models (HMM).
• Tagged sequence of tokens and HTML tags can be
used to predict the data fields boundaries.
• Webstruct and WebAnnotator Firefox extension.

11. Task
• Spanish web: hosts and
their sizes statistics.
• Only .es ccTLD.
• Breadth-first strategy:
• first 1-click environ,
• 2,
• 3,
• …
• Finishing condition: 100 docs
from host max., all hosts
• Low costs.
11

12. Spanish, Russian, German and
world Web in 2012
12
Domains Web servers Hosts DMOZ*
Spanish (.es) 1,5M 280K 4,2M 122K
Russian
(.ru, .рф, .su) 4,8M 2,6M ? 105K
German (.de) 15,0M 3,7M 20,4M 466K
World 233M 62M 890M 3,9M
Sources: OECD Communications Outlook 2013, statdom.ru
* - current period (October 2015)

13. Solution
• Scrapy (based on Twisted) - network operations.
• Apache Kafka - data bus (offsets, partitioning).
• Apache HBase - storage (random access, linear
scanning, scalability).
• Twisted.Internet - library for async primitives for use in
workers.
• Snappy - efficient compression algorithm for IO-
bounded applications.
13

14. Architecture
Kafka topic
SW
Crawling strategy
workers
Storage workers
14
DB

15. 1. Big and small hosts
problem
• Queue is flooded with
URLs from the same
host.
• → underuse of spider
resources.
• additional per-host
(per-IP) queue and
metering algorithm.
• URLs from big hosts
are cached in memory.
15

16. 2. DDoS DNS service Amazon AWS
Breadth-first strategy →
first visiting of unknown
hosts →
generating huge amount
of DNS reqs.
Recursive DNS server
• on every spider node,
• upstream to Verizon &
OpenDNS.
We used dnsmasq.
16

17. 3. Tuning Scrapy thread pool’а
for efficient DNS resolution
• OS DNS resolver,
• blocking calls,
• thread pool to resolve
DNS name to IP.
• numerous errors and
timeouts 🆘
• A patch for thread
pool size and
timeout adjustment.
17

18. 4. Overloaded HBase region
servers during state check
• 10^3 links per doc,
• state check: CRAWLED/NOT
CRAWLED/ERROR,
• HDDs.
• Small volume 🆗
• With ⬆table size, response
times ⬆
• Disk queue ⬆
• Host-local fingerprint
function for keys in HBase.
• Tuning HBase block cache to
fit average host states into
one block.
18
3Tb of metadata.
URLs, timestamps,…
275 b/doc

19. 5. Intensive network traffic
from workers to services
• Throughput
between workers
and Kafka/HBase
~ 1Gbit/s.
• Thrift compact
protocol for HBase
• Message
compression in
Kafka with Snappy
19

20. 6. Further query and traffic
optimizations to HBase
• State check: lots of
reqs and network
• Consistency
• Local state cache
in strategy worker.
• For consistency,
spider log was
partitioned by
host.
20

21. State cache
• All ops are batched:
– If no key in cache→
read HBase
– every ~4K docs →
flush
• Close to 3M (~1Gb)
elms → flush & cleanup
• Least-Recently-Used
(LRU) 👍
21

22. Spider priority queue (slot)
• Cell:
Array of:
- fingerprint,
- Crc32(hostname),
- URL,
- score
• Dequeueing top N.
• Prone to huge hosts
• Scoring model: document
count per host.
22

23. 7. Problem of big and small
hosts (strikes back!)
• Discovered few very
huge hosts (>20M
docs)
• All queue partitions
were flooded with huge
hosts,
• Two MapReduce jobs:
– queue shuffling,
– limit all hosts to
100 docs MAX.
23

24. Spanish (.es) internet crawl results
• fnac.es, rakuten.es, adidas.es,
equiposdefutbol2014.es,
druni.es,
docentesconeducacion.es -
are the biggest websites
• 68.7K domains found (~600K
expected),
• 46.5M crawled pages overall,
• 1.5 months,
• 22 websites with more than
50M pages
24

25. where are the rest of
web servers?!

26. Bow-tie model
A. Broder et al. / Computer Networks 33 (2000) 309-320
26

27. Y. Hirate, S. Kato, and H. Yamana, Web Structure in 2005
27

28. 12 years dynamics
Graph Structure in the Web — Revisited, Meusel, Vigna, WWW 2014
28

29. • Single-thread Scrapy spider gives 1200 pages/min.
from about 100 websites in parallel.
• Spiders to workers ratio is 4:1 (without content)
• 1 Gb of RAM for every SW (state cache, tunable).
• Example:
• 12 spiders ~ 14.4K pages/min.,
• 3 SW and 3 DB workers,
• Total 18 cores.
Hardware requirements
(distributed backend+spiders)
29

30. Software requirements
30
Single process Distributed spiders
Distributed
backend
Python 2.7+, Scrapy 1.0.4+
sqlite or any other
RDBMS
HBase/RDBMS
- ZeroMQ or Kafka
- - DNS Service

31. • Online operation: scheduling of new batch,
updating of DB state.
• Storage abstraction: write your own backend
(sqlalchemy, HBase is included).
• Run modes: single process, distributed spiders,
dist. backend
• Scrapy ecosystem: good documentation, big
community, ease of customization.
Main features
31

32. • Message bus abstraction (ZeroMQ and Kafka are
available out-of-the box).
• Crawling strategy abstraction: crawling goal, url
ordering, scoring model is coded in separate module.
• Polite by design: each website is downloaded by at most
one spider.
• Canonical URLs resolution abstraction: each document
has many URLs, which to use?
• Python: workers, spiders.
Main features
32

33. References
GitHub: https://github.com/scrapinghub/frontera
RTD: http://frontera.readthedocs.org/
Google groups: Frontera (https://goo.gl/ak9546)
33

34. Future plans
• Python 3 support,
• Docker images,
• Web UI,
• Watchdog solution: tracking
website content changes.
• PageRank or HITS strategy.
• Own HTML and URL parsers.
• Integration into Scrapinghub
services.
• Testing on larger volumes.
34

35. Run your business using Frontera
SCALABLE
OPEN
CUSTOMIZABLE
Made in Scrapinghub
(authors of Scrapy)

36. Contribute!
• Web scale crawler,
• Historically first
attempt in Python,
• Truly resource-
intensive task: CPU,
network, disks.
36

37. We’re hiring!
http://scrapinghub.com/jobs/
37

38. 38
Mandatory sales slide
Crawl the web, at scale
• cloud-based platform
• smart proxy rotator Get data, hassle-free
• off-the-shelf datasets
• turn-key web scraping
try.scrapinghub.com/PDB16

39. Questions?
Thank you!
Alexander Sibiryakov,
sibiryakov@scrapinghub.com