The document discusses analyzing and summarizing data from various sources using Ruby tools. It mentions analyzing weblog data, database transactions, and public performance metrics. The speaker discusses cleaning raw data from sources like IMDB, government databases, and The Guardian newspaper. The talk encourages attendees to collect, clean, and analyze their own data as well as public data to learn something new through investigation and visualization of outliers.

1. Data Stories
Jeremy Hinegardner
Scottish Ruby Conference 2011

2. Data Analysis Anyone?

3. Other People’s Data?

4. Your Own Data?

5. Weblog data
Database transactions
postgres performance metrics

6. Public Data?

7. UN Statistics website
data.gov.uk
data.gov
Scottish Home Survey
The Guardian
IMDB

8. All 3?

9. Collective Intellect
private customer data
internal chat, email lists
our internal data
processing metrics, queue sizes, database
queries, performance metrics.
public data
blogs, boards, tweets

10. I have the __DATA__!
What now?

11. Scrub it.
Get out your Cleaning
Supplies... Ruby

12. Majority of Your Time
Will be Spent Cleaning
The Data.

13. Interesting Data
Cleaning Problems?

14. I want to hear about
them.

15. Cleaning IMDB

16. A whole bunch of .gz
files.

17. Each with its own
slightly different format.

18. Extra junk around the
data.

19. ISO-8859 -> UTF8

20. Dates...

21. Black and White
Black and white
Black & White

22. Country Inconsistencies

23. Why are we doing this?

24. To Learn Something
New!

25. Supercrunchers

26. Outliers

27. Freakonomics

28. Superfreakonomics

29. Science of Fear

30. OK Cupid

31. The Guardian

32. Investigation Time.

33. Internet Movie Database
Title Running Times by
Country
Year made
Country of Origin
Actresses/Actors
Language
Release Dates by
Country Production
Companies
Colour
Genre

34. Ruby + SQLite + R + iTerm

35. Thanks!
Jeremy Hinegardner
jeremy@hinegardner.org
@copiousfreetime