The document outlines an exploratory data analysis agenda that includes gathering data from public and social media websites, cleaning messy data using OpenRefine, analyzing data using classic Unix tools like sed, awk, and parallel, and calculating statistics.

1. Exploratory Data Analysis
Aditya Laghate
Twitter: @thinrhino
1

2. Who am I?
• A pseudo geek
• Freelance software consultant
• Wildlife photographer
Twitter: @thinrhino
2

3. Agenda
•
•
•
•
Data gathering
Data cleaning
Usage of classic unix tools
Data analysis
Twitter: @thinrhino
3

4. Data Gathering
• Public data websites
o data.gov.in
o databank.worldbank.org
• Social websites
o facebook.com
o twitter.com
• Blogs / websites /etc via scrapping
Twitter: @thinrhino
4

5. Data cleaning
• Eg: openrefine
o OpenRefine (ex-Google Refine) is a powerful tool for working with messy
data, cleaning it, transforming it from one format into another, extending it
with web services, and linking it to databases like Freebase
o openrefine.org
Twitter: @thinrhino
5

6. Classic Unix Tools
• sed /awk
• Shell scripts
• GNU parallel
o Examples:
o cat rands20M.txt | awk '{s+=$1} END {print s}’
o cat rands20M.txt | parallel --pipe awk '{s+=$1}END{print
s}' | awk '{s+=$1} END {print s}’
o wc -l bigfile.txt
o cat bigfile.txt | parallel
{print s}'
Twitter: @thinrhino
--pipe wc -l | awk '{s+=$1} END
6

7. Data Analysis
Twitter: @thinrhino
7

8. Questions
@thinrhino
me@adityalaghate.in
Twitter: @thinrhino
8