Your SlideShare is downloading. ×
0
Almost Scraping: Web Scraping without Programming
Almost Scraping: Web Scraping without Programming
Almost Scraping: Web Scraping without Programming
Almost Scraping: Web Scraping without Programming
Almost Scraping: Web Scraping without Programming
Almost Scraping: Web Scraping without Programming
Almost Scraping: Web Scraping without Programming
Almost Scraping: Web Scraping without Programming
Almost Scraping: Web Scraping without Programming
Almost Scraping: Web Scraping without Programming
Almost Scraping: Web Scraping without Programming
Almost Scraping: Web Scraping without Programming
Almost Scraping: Web Scraping without Programming
Almost Scraping: Web Scraping without Programming
Almost Scraping: Web Scraping without Programming
Almost Scraping: Web Scraping without Programming
Almost Scraping: Web Scraping without Programming
Almost Scraping: Web Scraping without Programming
Almost Scraping: Web Scraping without Programming
Almost Scraping: Web Scraping without Programming
Almost Scraping: Web Scraping without Programming
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

Almost Scraping: Web Scraping without Programming

25,088

Published on

Web tools you can use to structure "stuff" on the Internet. Presented by Matt Wynn and Michelle Minkoff at NICAR 2011 at Raleigh in Feb.

Web tools you can use to structure "stuff" on the Internet. Presented by Matt Wynn and Michelle Minkoff at NICAR 2011 at Raleigh in Feb.

Published in: Technology
0 Comments
7 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
25,088
On Slideshare
0
From Embeds
0
Number of Embeds
7
Actions
Shares
0
Downloads
103
Comments
0
Likes
7
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. Almost Scraping: Web Scraping for Non-Programmers Michelle Minkoff, PBSNews.org Matt Wynn, Omaha World-Herald
  • 2. What is Web scraping? <ul><li>The *all-knowing* Wikipedia says: </li></ul><ul><li>“ Web scraping (also called Web harvesting or Web data extraction ) is a computer software technique of extracting information from websites. …Web scraping focuses more on the transformation of unstructured Web content, typically in HTML format, into structured data that can be stored and analyzed in a central local database or spreadsheet. Web scraping is also related to Web automation, which simulates human Web browsing using computer software. Uses of Web scraping include online price comparison, weather data monitoring, website change detection, Web research, Web content mashup and Web data integration.” </li></ul>
  • 3. Why do I want to Web scrape? <ul><li>Journalists like to find stories </li></ul><ul><li>Editors like stories that are exclusive </li></ul><ul><li>Downloading a dataset is like going to a press conference, anyone can grab and use it. </li></ul><ul><li>Web scraping is like an enterprise story, less likely to be picked up by all. </li></ul><ul><li>Puts more control back into your hands </li></ul>
  • 4. What kind of data can I get? <ul><li>Laws (Summary of same-sex marriage laws for each state, pdfs) </li></ul><ul><li>Photos (pictures associated with all players on a team you’re highlighting, all mayoral candidates) </li></ul><ul><li>Recipe ingredients (NYT story about peanut butter) </li></ul><ul><li>Health care (see ProPublica’s Dollars for Docs project) </li></ul><ul><li>Links, images, dates, names, categories, tags, anything with some sort of repeatable structure </li></ul>
  • 5. DownThemAll http://www.downthemall.net
  • 6. Yahoo Pipes http://pipes.yahoo.com/pipes
  • 7. Yahoo Pipes <ul><li>Access and manipulate RSS feeds, which are often a flurry of information </li></ul><ul><li>Sort, filter, combine your information </li></ul><ul><li>Format that info to fit your needs (date formatter) </li></ul>
  • 8. Yahoo Pipes <ul><li>Pair with Versionista, which can create an RSS feed of changes to a Web site to keep tabs on what’s changing. This was done to great effect by ProPublica’s team in late 2009, esp. by Scott Klein and then-intern Brian Boyer, now at Chicago Tribune </li></ul>
  • 9. ScraperWiki http://scraperwiki.com
  • 10. Needlebase http://needlebase.com
  • 11. Needlebase <ul><li>For sites that follow a repetitive formula spanning multiple pages, like index pg &amp; detail page, maybe with a search results page in the middle </li></ul><ul><li>Like a good employee, train it once, and then let it churn. </li></ul>
  • 12. Needlebase <ul><li>Query, select and filter your data in the Web app, then export in format of your choice. </li></ul><ul><li>Can check your data and stay up-to-date on your data set </li></ul><ul><li>Will go more in depth on Needle in Sat.’s hands-on lab at 10 a.m. </li></ul>
  • 13. InfoExtractor http://www.infoextractor.org
  • 14. irobotsoft http://irobotsoft.com
  • 15. Imacros https://addons.mozilla.org/en-US/firefox/addon/imacros-for-firefox/
  • 16. Imacros <ul><li>Record repetitive tasks that you do every day, and keep them as a data set </li></ul><ul><li>Think of it like a bookmark, but if you could include logging in, or entering a search term, as part of that bookmark </li></ul><ul><li>Useful for stats you check every day, scores for your local sports team, stocks if you’re a biz reporter, etc. </li></ul><ul><li>More complex function allows you to extract multiple data points on a page, like from an HTML table. </li></ul>
  • 17. OutwitHub http://www.outwit.com/products/hub
  • 18. OutwitHub <ul><li>Versatile Firefox extension </li></ul><ul><li>Can use it for certain defaults (links, images) </li></ul>
  • 19. OutwitHub <ul><li>Dig through the HTML hierarchy tree </li></ul><ul><ul><li>Structural elements (&lt;h3&gt;) </li></ul></ul><ul><ul><li>Stylistic elements (&lt;strong&gt;) </li></ul></ul><ul><li>Download list of attached files or files themselves </li></ul><ul><li>More options if you buy Pro version </li></ul><ul><li>Will discuss in-depth and use in hands-on lab on Saturday at 10 am </li></ul>
  • 20. Python
  • 21. Wrap-Up <ul><li>Non-programming scrapers can’t do everything, but have the power to get you started. Some say “Program or be programmed,” but this is a compromise. </li></ul><ul><li>Legal permissions still apply, so don’t use scraped info you don’t have the right to. </li></ul><ul><li>Something to consider. How does this apply to what you do every day, and how scraping could contribute to your job? </li></ul><ul><ul><li>“ The businesses that win will be those that understand how to build value from data from wherever it comes. Information isn’t power. The right information is.” – media consultant Neil Perkin wrote in Marketing Week </li></ul></ul>

×