This document outlines the process for analyzing open data, including defining a problem, getting the data, cleaning it, exploring it, analyzing it, and communicating results. It provides tips for getting data using R packages and APIs. Methods are described for cleaning data by addressing outliers and missing values. Techniques are suggested for exploring data through graphing, mapping, and network analysis. The importance of communicating results through stories and sharing the process is emphasized. Examples of case studies and open data groups in Ottawa are also listed.

1. ottawa.theodi.org
@ODI_Ottawa
Rob Davidson

2. nodes-map.theodi.org

3. ottawa.theodi.org
● Define problem or question
● Get the data
● Clean the data
● Explore the data
● Analyze the data
● Communicate results
Data Process

4. ottawa.theodi.org
● Define problem or question
● Get the data
● Clean the data
● Explore the data
● Analyze the data
● Communicate results
Data Process

5. ottawa.theodi.org
● Seoul Subway Data
● 2015 Canadian Federal Election
● Weather Impact on Ottawa Cycling
● Ottawa 311 Data
● Ottawa Crime Statistics
Case study code in Github: https://github.com/robscottd/OpenDataInAction
Case Studies

6. ottawa.theodi.org
Get the Data
The “Good” of getting open data:
Centralized government repositories
Multiple standard formats
The “Bad” of getting open data:
Data is too “clean”, value scrubbed out
Rate-limited/complex APIs

7. ottawa.theodi.org
Get the Data Using R
R basics: read.table, read.csv, read.csv2
Packages: readr, rvest, RSelenium, readxl, rjson
Using APIs: twitteR, httr, jsonlite

8. ottawa.theodi.org
Clean the Data
Tidy the data! Follow Hadley Wickham’s method
Address extreme outliers
Explore outliers graphically (Shiny Gadgets example)
Address missing values
Imputation through MICE, missForest, Hmisc
Or remove the incomplete observations

9. ottawa.theodi.org
Explore the Data: But why not just dive in?
No assumptions, “listen” to the data
Understand data properties
Find patterns in the data
Discover analysis strategies
Begin the visual narrative

10. ottawa.theodi.org
Explore the Data
Graph it
Map it (leaflet)
Network graph it (networkD3,igraph)
Heat map it (d3heatmap,heatmaply)

11. ottawa.theodi.org
Communicate Results
Complete the narrative - what is the story?
Design with audience in mind
Share your process
Publish for easy access and feedback
If possible, provide link to data

12. ottawa.theodi.org
Communicate Results
Shiny
Bookdown
Notebooks

13. ottawa.theodi.org
Case Study
Example:
2015 Canadian Federal Election
http://bit.ly/1RJ8nMi

14. ottawa.theodi.org
It just might not tell you what you want to hear
● Survey design is biased
● Sensors are not identically calibrated
● Merged datasets are not temporally aligned
Data Does Not Lie

15. ottawa.theodi.org
● Data for Good Ottawa
● Open Data Ottawa
● Datafest Ottawa
Sample of Ottawa Open Data Groups

16. ottawa.theodi.org
● Federal Government
● Provincial Government
● Municipal Government
Canadian Open Data Consultations

17. Rob Davidson
@ODI_Ottawa
Ottawa.theodi.org
rob.davidson@veracify.com