This document discusses using web scraping to extract structured data from unstructured sources on the internet. It introduces Scrapy, an open source and customizable Python framework for scraping websites. Scrapy processes requests asynchronously, handles errors and delays, and includes tools for parsing HTML and XML. Examples are given of how Scrapy is used for market analysis, academic research, government projects, and personal side projects involving scraping Philippine news, elections data, and transportation websites. Legal considerations around scraping public data are also briefly addressed.

1. Scraping
the web using
scrapy
Jolo Balbin
Mikko Gozalo

2. – Mitchell Kapor,
a variation of former MIT President Jerome Weisner’s quote
“Getting information off
the Internet is like taking
a drink from a fire
hydrant.”

3. Tons of information
on the internet
News / Rappler, ABS-CBN News, GMA News Online
SOCIAL MEDIA / Facebook, Twitter
TRANSPORTATION / MMDA, WAZE, DOTC WEBSITE
WEATHER / Project NOAH, PAGASA
E-COMMERCE / LAZADA, ZALORA, EBAY, OLX
Government DATA / PHILGEPS

4. Tons of information
on the internet
News / What’s trending? What’s HAPPENING?
SOCIAL MEDIA / What are the people’s sentiment on subject x?
TRANSPORTATION / What’s the traffic like later?
WEATHER / What’s the effect of weather on traffic?
E-COMMERCE / Who’s selling the cheapest item x?
Government DATA / Where are our taxes going?

5. The Problem
Not all data are
structured!
HOW DO WE TURN
UNSTRUCTURED DATA
INTO STRUCTURED
ONES?

6. Web scraping
๏ computer software technique of extracting
information from websites.
๏ focuses more on the transformation of unstructured
data on the web, typically in HTML format, into
structured data that can be stored and analyzed in a
central local database or spreadsheet.
https://en.wikipedia.org/wiki/Web_scraping

7. Conventional Way
๏ Fetch the webpage using urllib, httplib or requests.
๏ Use beautifulsoup4, lxml or regular expressions to get
extract information.
๏ Analyze/store the information!

8. Conventional Way

9. Conventional Way
๏ Blocking! We have to wait for each request to finish
before we move on to the next.
๏ If it encounters an error somewhere, we’re doomed.
Everything will just halt!

10. Conventional Way
๏ We can use threading, gevent or other libraries to
make it asynchronous.
๏ We can wrap parts of the code in try-except blocks
to catch possible Exceptions.

11. Here comes…

12. Why Scrapy?
๏ Processes requests and responses asynchronously.
๏ Customizable!You can override almost everything.
๏ Handles cookies, delays, timeouts, etc so you won’t
have to. No need to reinvent the wheel!
๏ Includes Selectors, a parsing library that can parse
HTML and XML using XPATH or CSS; or you can just
use Beautiful Soup!

13. History of Scrapy
๏ An open source framework to scrape websites
๏ Scrapy was started by Pablo Hoffman and Shane
Evans (2007)
๏ Originally a tool used by Shane’s company
๏ They saw the potential, and open sourced it.

14. Getting Started with scrapy
๏ As easy as pip install Scrapy.
๏ Start a project with scrapy startproject
project_name.
๏ Creating your first spider!

15. Short Demo

16. using Scrapy
at Work

17. At Scrapinghub
๏ Company that provides scraping-related services to
clients around the globe.
๏ Distributed team of 105 people around the world.
๏ Active in contributing to open-source!
๏ Project owner of Scrapy!

18. Academe/Research
๏ A U.S. Department of Energy National Laboratory
Operated by a university in California.
๏ Analyzes relation between product price, energy
efficiency and other product features of typical home
appliances.
๏ Partnered with Scrapinghub for academic research!

19. Market Analytics
๏ A UK company that provides price, promotion and online
product positioning analytics.
๏ Help consumers find the best prices!
๏ Help online retailers compare their prices with other
retailers.
๏ Help brands check if retailers are providing accurate
product information.
๏ Partnered with Scrapinghub for their scraping needs!

20. Government Research
๏ Scrapinghub is participating in DARPA’s Memex.
๏ Crawls the deep web.
๏ Aids in systematically tracking down criminal activity.

21. using Scrapy
FOR SIDE PROJECTS

22. MRT Passenger Traffic
๏ Crawls the MRT3 website using Scrapy.
๏ Downloads the CCTV images for each station.
๏ Approximate the relative passenger traffic for the
certain moment using computer vision!

23. MRT Passenger Traffic
*Line status as of July 1 (Wednesday), 6:40pm

24. MRT Passenger Traffic
*CUBAO STATION status as of July 1

25. MRT Passenger Traffic
*AYALA STATION NB status as of July 1

26. MiniBalita.com
๏ A news reader for Philippine news.
๏ Crawls Philippine news websites such as Rappler,ABS-
CBN News, Inquirer, Spin.ph, etc.
๏ Integrated with TextTeaser to produce “mini” balita.

28. 2013 General Elections
๏ Crawled the 2013 General Elections to find trends.
๏ 70 clustered precincts registered 100% turnout, most
of them in ARMM.
๏ One clustered precinct voted for only one senator.
No one voted for anyone else despite the fact that a
voter may choose up to 12 candidates!

29. Is SCRAPING legal?
๏ Legalities about scraping is a gray area.
๏ Scraping public data is somewhat legal.
๏ Illegality may arise from how the data is used.
๏ Some websites explicitly prohibit scraping.
๏ Always obey robots.txt.

30. End. Any Questions?
Jolo Balbin
Twitter: @mojojolo
http://www.summarizerman.com
Mikko Gozalo
Twitter: @mikkogozalo
http://www.mikkogozalo.com