Presentation on how I built NYC Taxis: A Day in the Life Visualization

1. NYC Taxis: A Day in the Life
A Data Visualization/Civic Hack by Chris Whong
FutureStack
October 8, 2014 @chris_whong

2. A bit about me…
Data Solutions Architect
Socrata, Inc.
Democratizing Public Data
Urban Planner
Coder

3. Urbanist,
Mapmaker,
Data Junkie

4. OPEN CIVIC
My
Favorit
e
Data
GEOSPATIAL

5. It all started with a tweet…

10. “…you will need to send or bring an
external hard drive with a minimum
capacity of 200 GB to the TLC
offices. The address is listed below.
The hard drive must be brand new,
still in the box and unopened. “

12. Data!

13. Fare Data – 12 CSVs (~2GB
each)
• Payment Type
• Fare
• Tax
• Tip (Credit Card Only)
• Tolls
Trip Data – 12 CSVs (~2GB each)
• Medallion
• Driver
• Pickup Time and Location
• Dropoff Time and Location
173 Million Trips in 2013!

16. • Shared via Torrent and Direct Download
• Urban Data Nerds Rejoice!
• Lots of Interesting Projects and Analysis

19. Driver and Vehicle De - Anonymization
hash = md5(medallionNumber)
Medallion: 6B111958A39B24140C973B262EA9FEA5
Hack License: D3B035A03C8A34DA17488129DA581EE7
There are only 19 million possible values!

20. Passenger Identification
“Riding with the Stars” by Anthony Tockar

21. But what will I do with the data?
I love:
• Data Visualization
• Mapping
• Transportation
• Civic Hacking
• Node.js
• D3.js
• Learning how cities work
SpatioTemporal Data is PERFECT for map animations!

22. Burning Questions:
Where do cabs go?
How much $ to they make?
How much time do they spend empty/full?

23. The Vision:
Pick a single Taxi from the data,
follow it for 24 hours on a map.
Repeat.

24. 1) Get Raw Data
(Isolate a collection of full “cab/days”)
Figure it out myself : HARD!
Ask the internets for help: EASY!
+

26. 2) Turn each trip into a line :
Directions API
Start and End Points IN
Directions with encoded
polylines OUT
Raw Data (Trips & Fares) Node.js Script Trips + Polylines

27. 3) Store the Results in SQLite

28. Getting a Day’s Trips into the
Browser
Backend (heroku):
SQLite Database Absurdly simple node server
/trip
1 day’s trips as geoJSON API
• HTML/CSS/JQuery
• D3.js
• Leaflet.js
• Mapbox
• Moment.js
• Bootstrap
Frontend:

29. GeoJSON in Leaflet via D3

30. GOT DATA.
Now how to visualize it?

33. Follow the Bouncing Ball:

34. Recursive SetTimeout( ) Counts Minutes:
Correct time at Start of Each Trip:

35. Empty/Full Chart w/ Cumulative Earnings

36. Todo:
• Allow pause & inspect
• Show driver shifts & totals
• Show miles traveled
• Add a non-cumulative column chart for
fares

40. Thanks for sponsoring!
&

41. A Few More Examples…

42. Tech
Chops
Design
Eye
DataViz
!
Vision &
StoryTelling

43. Thanks!
@chris_whong
https://github.com/chriswhong
http://chriswhong.com
Support Open Data!