Presentation by Daniel Burseth at the MIT Big Data Explorers "Crash Course" on 9/20/2014. "An end-to-end demonstration of generating, cleaning, and visualizing a “Messy” data Set". http://www.mitbigdataexplorers.com/

1. Daniel Burseth
Co-president MIT Big Data Explorers
dburseth@mit.edu
@dmbnyc
Github: dburseth

2.  Acronyms abound
 Tremendous complexity
 Use building blocks not code

3.  This is easy
EPPM of 10 requires 500 professionals

5.  http://www.nytimes.com/2014/08/18/technology/for-big-data-scientists-hurdle-to-insights-is-janitor-work.
html?emc=eta1&_r=0
Data preparation and cleansing:
• Missing
• Duplicative
• Conventions (dates, time,
geographies)
• Spacing
• Can we measure data
cleanliness?
• What’s our Pareto point?

6.  AWS -> EC2
 Launch instance: ami-c6b61fae (US-EAST)
 Instance type m3.medium
 Connect
 You should see some software on the desktop

7.  Scrape all of Craiglist’s Boston apartment listings using WebHarvy
 Examine, clean, and prepare the data set using OpenRefine
 Map our data and apply filters using Tableau
……all without writing a single line of code.

9.  A hyper-intelligent utility to scrape website
data.
 SysNucleus, makers of USBTrace
 Heavy duty alternatives: Scrapy (scrappy.org),
Beautiful Soup

10. HTTP://SHOUTKEY.COM/WIRE
1. Start Config
2. Click on Hungry Mother –
capture text
3. Click on Hungry Mother –
capture URL
4. Click on Kendall
Square/MIT – capture text
5. Click lasts review–
capture text
CLEAR
1. Mine -> Scrape a list of
similar links
2. Click on Hungry Mother

11.  Let’s start collecting
information in the first sub-page.

12.  Edit Clear
 Navigate into a sub-page
 Start Config
 Set as Next Page Link

13.  Scheduler
 Input keywords
 Puase Inject (word of caution: scraping often violates TOS. Potentially not viable
for apps, commercial purposes!)
 TRY VISITING CRAIGSLIST IN AWS BTW!!
 Proxy
 Database export

14.  Download Craigslist Boston from http://shoutkey.com/glorify
 Look at our data: open Boston Dirty.csv (20k rows of mess!)
 Time to CLEAN: Launch GOOGLE-REFINE.EXE
 Within MOZILLA, navigate to http://127.0.0.1:3333/
 Create Project -> This Computer -> Browse
 Parse by tab
 Create Project

15. 1. First, sort your column.
2. Then, invoke "Re-order rows permanently" in the "Sort" dropdown menu that appears on top of
the middle of the data table.
3. Then invoke Edit cells and Blank down on the Title column.
4. Then on that column, invoke menu Facet > Custom facets and Facet by blank.
5. Select true in that facet, and invoke Remove matching rows in the left most "all" dropdown
menu.
6. Remove the facet.

19.  Then run the “To Number” transform again

23.  Increment the radius to 7
and make judgment calls
along the way.
 Change the Distance
Function and do the same
thing

26.  Looks like we have SOME really expensive real
estate. Data errors????

27.  Boston Clean.csv

28.  Load Boston clean.csv
 “Go to Worksheet”

29.  Great “semantic” example. Tableau understands that this text translates to a
lat/long

30.  Look on the map in the lower right corner
 Let’s “Filter Data”

31.  Under “Measures”, drag “Price” onto size in “Marks”
 Change sum(Price) to avg(Price)
 Drag Price, change to max(price) into Filters and select an “At Most”
 Right click on the filter and show “Quick Filter”
 Drag “City” onto “Label”
 Menu Map -> Map Options
 Click on a node for info and drill down potential

33. 1. Explored various webpage structures and scraped them
2. Exported the data to Refine
3. Parsed columns to extract critical price and location information
4. Used clustering algorithms to merge related geographies
5. Applied filters to identify errant prices
6. Exported the data to Tableau
7. Completed a real cursory mapping visualization

34.  Please come talk to me