Data scraper's toolbox
•
0 likes•217 views
This document outlines various tools and techniques for web scraping, including both manual and automated methods. It notes that scraping requires creativity, problem solving, and adapting techniques to individual problems. Various scraping methods are presented, such as copy-paste, regular expressions, HTML parsing, and scraping software. It also addresses ethical and legal considerations around scraping and recommends using APIs when possible to avoid brute force attacks. The goal is to explore the "fine art of scraping" rather than provide set solutions.
Recommended
More Related Content
Viewers also liked
Viewers also liked (16)
Similar to Data scraper's toolbox
Similar to Data scraper's toolbox (20)
Recently uploaded
Recently uploaded (20)
Data scraper's toolbox
- 1. A Data Scraper’s Toolbox Alice Corona Data Journalist & Researcher @alice_corona alice-corona.nl data-journalism-tools.silk.co
- 2. Despite its fame, scraping is a highly creative process. It requires… • Problem solving • Lateral thinking • Planning for efficiency • Prediction and planning for possible outcomes and uses • Methodological choices affect the whole outcome Credits: Toolbox icons created by Ralf Schmitzer from the Noun Project
- 3. There cannot be a course on web scraping teaching you ‘solutions’ We can dive into the fine art of scraping by learning tools and techniques. Then each individual personality can adapt, mix and mash this knowledge according to each individual problem Credits: Toolbox icons created by Ralf Schmitzer from the Noun Project
- 4. “process of automatically collecting information from the Web” ~ Wikipedia on “Web Scraping” Credits: Toolbox icons created by Ralf Schmitzer from the Noun Project
- 5. ? Credits: Toolbox icons created by Ralf Schmitzer from the Noun Project
- 6. • Manual copy-paste • Text grepping / RegEx • HTML parsing (CSS, XPATH…) • Web Scraping Software Credits: Toolbox icons created by Ralf Schmitzer from the Noun Project
- 7. Copy-Paste Credits: Toolbox icons created by Ralf Schmitzer from the Noun Project
- 8. Data Import in Google Sheets Credits: Toolbox icons created by Ralf Schmitzer from the Noun Project
- 9. Chrome’s Scrape- Similar Credits: Toolbox icons created by Ralf Schmitzer from the Noun Project
- 10. X-Path Parsing in Google Sheets Credits: Toolbox icons created by Ralf Schmitzer from the Noun Project
- 11. SearchLink “SearchLink is a System Service for OS X which handles searching multiple sources and automatically generating Markdown links for text” http://brettterpstra.com/projects/searchlink/ Credits: Toolbox icons created by Ralf Schmitzer from the Noun Project
- 12. APIs Credits: Toolbox icons created by Ralf Schmitzer from the Noun Project
- 13. Scripts Credits: Toolbox icons created by Ralf Schmitzer from the Noun Project
- 14. Scraping Software Credits: Toolbox icons created by Ralf Schmitzer from the Noun Project
- 15. “Hidden Secrets” Credits: Toolbox icons created by Ralf Schmitzer from the Noun Project
- 16. is scraping legal? Credits: Toolbox icons created by Ralf Schmitzer from the Noun Project
- 17. is scraping ethical? Credits: Toolbox icons created by Ralf Schmitzer from the Noun Project
- 18. Consider… • Brute force attacks and DoS (prefer APIs, when available) • Your use of the data (copyright, privacy…) Credits: Toolbox icons created by Ralf Schmitzer from the Noun Project
- 19. A Data Scraper’s Toolbox Alice Corona Data Journalist & Researcher @alice_corona alice-corona.nl data-journalism-tools.silk.co