This document outlines various tools and techniques for web scraping, including both manual and automated methods. It notes that scraping requires creativity, problem solving, and adapting techniques to individual problems. Various scraping methods are presented, such as copy-paste, regular expressions, HTML parsing, and scraping software. It also addresses ethical and legal considerations around scraping and recommends using APIs when possible to avoid brute force attacks. The goal is to explore the "fine art of scraping" rather than provide set solutions.

1. A
Data Scraper’s
Toolbox
Alice Corona
Data Journalist &
Researcher
@alice_corona
alice-corona.nl
data-journalism-tools.silk.co

2. Despite its fame, scraping is
a highly creative process.
It requires…
• Problem solving
• Lateral thinking
• Planning for efficiency
• Prediction and planning for possible outcomes
and uses
• Methodological choices affect the whole outcome
Credits: Toolbox icons created by Ralf Schmitzer from the Noun Project

3. There cannot be a course on web scraping
teaching you ‘solutions’
We can dive into the fine art of scraping by learning tools
and techniques. Then each individual personality can
adapt, mix and mash this knowledge according to each
individual problem
Credits: Toolbox icons created by Ralf Schmitzer from the Noun Project

4. “process of automatically
collecting information
from the Web”
~ Wikipedia on “Web Scraping”
Credits: Toolbox icons created by Ralf Schmitzer from the Noun Project

5. ?
Credits: Toolbox icons created by Ralf Schmitzer from the Noun Project

6. • Manual copy-paste
• Text grepping / RegEx
• HTML parsing (CSS, XPATH…)
• Web Scraping Software
Credits: Toolbox icons created by Ralf Schmitzer from the Noun Project

7. Copy-Paste
Credits: Toolbox icons created by Ralf Schmitzer from the Noun Project

8. Data Import in Google
Sheets
Credits: Toolbox icons created by Ralf Schmitzer from the Noun Project

9. Chrome’s Scrape-
Similar
Credits: Toolbox icons created by Ralf Schmitzer from the Noun Project

10. X-Path Parsing in
Google Sheets
Credits: Toolbox icons created by Ralf Schmitzer from the Noun Project

11. SearchLink
“SearchLink is a System Service for OS X which
handles searching multiple sources and automatically
generating Markdown links for text”
http://brettterpstra.com/projects/searchlink/
Credits: Toolbox icons created by Ralf Schmitzer from the Noun Project

12. APIs
Credits: Toolbox icons created by Ralf Schmitzer from the Noun Project

13. Scripts
Credits: Toolbox icons created by Ralf Schmitzer from the Noun Project

14. Scraping Software
Credits: Toolbox icons created by Ralf Schmitzer from the Noun Project

15. “Hidden Secrets”
Credits: Toolbox icons created by Ralf Schmitzer from the Noun Project

16. is scraping legal?
Credits: Toolbox icons created by Ralf Schmitzer from the Noun Project

17. is scraping ethical?
Credits: Toolbox icons created by Ralf Schmitzer from the Noun Project

18. Consider…
• Brute force attacks and DoS (prefer APIs,
when available)
• Your use of the data (copyright, privacy…)
Credits: Toolbox icons created by Ralf Schmitzer from the Noun Project

19. A
Data Scraper’s
Toolbox
Alice Corona
Data Journalist &
Researcher
@alice_corona
alice-corona.nl
data-journalism-tools.silk.co