This document summarizes web scraping and introduces the Scrapy framework. It defines web scraping as extracting information from websites when APIs are not available or data needs periodic extraction. The speaker then discusses experiments with scraping in Python using libraries like BeautifulSoup and lxml. Scrapy is introduced as a fast, high-level scraping framework that allows defining spiders to extract needed data from websites and run scraping jobs. Key benefits of Scrapy like simplicity, speed, extensibility and documentation are highlighted.

1. Web Scraping with Scrapy
Virendra Rajput
Hacker @Markitty

2. Agenda
● What is web scraping and why it's fun
● My experiments with web scraping
● Getting started with Scrapy
● How Scrapy works and a quick Demo
● Why Scrapy
● Questions

3. What is Web Scraping?
● Extracting information from websites
● Problem:
○ Static websites
○ No access to APIs to extract the data you
need
○ Need to extract data periodically
● Manual solution - go to the website and copy
the required data
● Smarter solution: Web Scraping

4. My Experiments with Scraping

5. Web Scraping in Python
● Download webpage with urllib2, requests
● Parse the page with BeautifulSoup/lxml
● Select with XPath or css selectors

6. Scrapy - fast high Level Screen
Scraping and web crawling
Framework
● Pick a website
● Define the data you want to scrape
● Write the spider to extract the data
● Run the spider
● Store the Data

7. Demo

9. Why Scrapy
● Simplicity
● Fast
● Productive/ Extensible
● Portable
● Well docs & Healthy community
● Commercial Support

10. Advanced Features (built in)
● Interactive shell for trying XPaths (useful for
debugging)
● selecting and extracting data from html
sources
● cleaning and sanitizing the scraped data
● generating feed exports (JSON, CSV)
● media pipeline for downloading stuff
● Middlewares for (cookies, HTTP
compression, cache, user-agent spoofing,
etc)

11. questions
?