Molescrape is a platform for web crawling and scraping with python. It uses scrapy to schedule the spiders and postprocessing scripts to process the data and ingest it into different databases like MongoDB and Elasticsearch.

1. Web Crawling and Scraping
with Python

2. Introduction
contact@molescrape.com
www.molescrape.com

3. Introduction
contact@molescrape.com
www.molescrape.com

4. Introduction
contact@molescrape.com
www.molescrape.com

5. Why having a full platform?
● Handle multiple projects
● Repeated scraping on same website
● Error monitoring
● Postprocessing allows to change the extracted information
later
contact@molescrape.com
www.molescrape.com

6. Technology
contact@molescrape.com
www.molescrape.com
A
B
C
D
Custom
Processing

7. Newspaper Mining
contact@molescrape.com
www.molescrape.com
Party names mentioned in the titles of newspaper
articles before the German national elections in 2017
Full article (in German): goo.gl/EfvwQP

8. Legal Challenges
● discussed under the term screen scraping
● EU Directive 96/6/EC
● sui generis protection of databases (no level of creativity
needed, only financial investments)
● Court decisions in Germany/Europe usually only about
live requests (i.e. not collection of whole database)
– Ofen: Price search engines for flights
– Meta search engine for vehicle prices
contact@molescrape.com
www.molescrape.com

9. Future Ideas
● Postprocessing: Replace Mosquito Custom Code with
Apache Kafka
● LegalityPipeline: only ingest k% of the data into database,
but mark all data as already read
● Beter integrate broad crawl into infrastructure
contact@molescrape.com
www.molescrape.com

10. Dashboard
contact@molescrape.com
www.molescrape.com

11. Qestions
Qestions
either now or contact@molescrape.com
contact@molescrape.com
www.molescrape.com