2. Whether you're a new startup looking for investment, or a team at a large company who wants the green light for a new product,
nothing convinces like real running code. But how do you solve the chicken-and-egg problem of filling your early prototype with
real data?
Traffic Photo by TheTruthAbout - http://flic.kr/p/59kPoK
Money Photo by borman818 - http://flic.kr/p/61LYTT
3. As experts in processing large datasets and interpreting charts and graphs, we may think of our data in the same way that a
Bloomberg terminal presents financial information. But information visualisation alone does not make a product.
http://www.flickr.com/photos/financemuseum/2200062668/
4. We need to communicate our understanding of the data to the rest of our product team. We need to be their eyes and ears in the
data - translating human questions into code, and query results into human answers.
5. prototypes are
boundary objects
Instead of communicating across disciplines using language from our own specialisms, we show what we mean in real running
code and designs. We prototype as early as possible, so that we can talk in the language of the product.
http://en.wikipedia.org/wiki/Boundary_object - “allow coordination without consensus as they can allow an actor's local
understanding to be reframed in the context of a some wider collective activity”
http://www.flickr.com/photos/orinrobertjohn/159744546/
6. Prototyping has many potential benefits. We use this triangle to think about how to structure our work and make it clear what
insights we are looking for in a particular project.
7. Novelty
Prototyping has many potential benefits. We use this triangle to think about how to structure our work and make it clear what
insights we are looking for in a particular project.
8. Novelty
lity
id e
F
Prototyping has many potential benefits. We use this triangle to think about how to structure our work and make it clear what
insights we are looking for in a particular project.
9. Novelty
ty De
eli si rab
Fid ilit
y
Prototyping has many potential benefits. We use this triangle to think about how to structure our work and make it clear what
insights we are looking for in a particular project.
10. Novelty
ty De
eli si rab
Fid ilit
y
Prototyping has many potential benefits. We use this triangle to think about how to structure our work and make it clear what
insights we are looking for in a particular project.
11. no more
lorem ipsum
By incorporating analysis and data-science into product design during the prototyping phase, we avoid “lorem ipsum”, the fake
text and made-up data that is often used as a placeholder in design sketches. This helps us understand real-world product use
and find problems earlier.
Photo by R.B. - http://flic.kr/p/8APoN4
12. helping designers explore data
Data can be complex. One of the first things we do when working with a new dataset is create internal toys - “data
explorers” - to help us understand it.
13. Philip Kromer, Infochimps
Flip Kromer of Infochimps describes this process as “hitting the data with the Insight Stick.”
As data scientists, one of our common tasks is to take data from almost any source and apply standard structural techniques to
it without worrying too much about the domain of the data.
14. Philip Kromer, Infochimps
Flip Kromer of Infochimps describes this process as “hitting the data with the Insight Stick.”
As data scientists, one of our common tasks is to take data from almost any source and apply standard structural techniques to
it without worrying too much about the domain of the data.
15. ou can discov er patterns
“With e nough data y t you can't
s using simple counting tha
and fact ophisticated
discover in sma ll data using s
ical and ML a pproaches.” ig on Quora
statist –Dmitriy Ryaboy par
aphrasing Peter Norv
http://b.qr.ae/ijdb2G
Philip Kromer, Infochimps
Flip Kromer of Infochimps describes this process as “hitting the data with the Insight Stick.”
As data scientists, one of our common tasks is to take data from almost any source and apply standard structural techniques to
it without worrying too much about the domain of the data.
16. Here’s a small example of exploring a dataset that I did while working in Nokia’s Location & Commerce division.
17. Searches are goal-driven user behaviour - someone typed something into a search box on a phone. But we can even learn from
activity that isn’t so explicit.
When someone views a Nokia Ovi map on the web or phone, the visuals for the map are served up in square “tiles” from our
servers. We can analyse the number of requests made for each tile and take it as a measure of interest or attention in that part of
the world.
18. Searches are goal-driven user behaviour - someone typed something into a search box on a phone. But we can even learn from
activity that isn’t so explicit.
When someone views a Nokia Ovi map on the web or phone, the visuals for the map are served up in square “tiles” from our
servers. We can analyse the number of requests made for each tile and take it as a measure of interest or attention in that part of
the world.
19. Searches are goal-driven user behaviour - someone typed something into a search box on a phone. But we can even learn from
activity that isn’t so explicit.
When someone views a Nokia Ovi map on the web or phone, the visuals for the map are served up in square “tiles” from our
servers. We can analyse the number of requests made for each tile and take it as a measure of interest or attention in that part of
the world.
20. LA attention heatmap
We built a tool that could calculate metrics for every grid-square of the map of the world, and present heatmaps of
that data on a city level. This view shows which map-tiles are viewed most often in LA using Ovi Maps. It’s calculated
from the server logs of our map-tile servers. You could think of it as a map of the attention our users give to each
tile of LA.
21. LA driving heatmap
This is the same area of California, but instead of map-tile attention it shows the relative number of cars on the road that are
using our navigation features. This gives a whole different view on the city. We can see that it highlights major roads, and it’s
much harder to see where the US coastline occurs. By comparing these two heatmaps we start to understand the meaning and
the potential of these two datasets.
22. But of course a heatmap alone isn’t a product. This is one of the visualisation sketches produced by designer Tom
Coates after investigating the data using the heatmap explorer. It’s much closer to something that could go into a
real product.
23. Tools
These are the tools I’ll be using to demo some of my working processes.
24.
25. Apache Pig makes Hadoop much easier to use by creating map-reduce plans from SQL-like scripts.
32. Realistic cities
generating a dataset of people
moving around town
The first dataset we’ll generate is one you could use to test any system or app involving people moving around the
world - whether it’s an ad-targeting system or a social network.
33. You probably know about Stamen’s beautiful work creating new renderings of OpenStreetMap, including this Toner
style.
34. When they were getting ready to launch their newest tiles called Watercolor, they created this rendering of the access
logs from their Toner tileservers. It shows which parts of the map are most viewed by users of Toner-based apps.
35. Working with data and inspiration from Eric Fischer, Nathaniel Kelso of Stamen generated this map to decide how
deep to pre-render each area of the world to get the maximum hit-rate on their servers. Rendering the full map to
the deepest zoom would have taken years on their servers. The data used as a proxy for the attention of users is a
massive capture of geocoded tweets. The more tweets per square mile, the deeper the zoom will be rendered in that
area.
36. We can go further than geocoded tweets and get a realistic set of POIs that people go to, with timestamps. If you
search for 4sq on the Twitter streaming API you get about 25,000 tweets per hour announcing users’ Foursquare
checkins.
39. And if you view source, the data’s all there in JSON format.
40. Demo:
Gathering Foursquare tweets
So I set up a script to skim the tweets, perform the HTTP requests on 4sq.com and capture the tweet+checkin data as
lines of JSON in files in S3.
41. For this demo I wanted to show just people in San Francisco so I looked up a bounding-box for San Francisco.
42. DEFINE json2tsv `json2tsv.rb` SHIP('/home/hadoop/pig/
json2tsv.rb','/home/hadoop/pig/json.tar');
A = LOAD 's3://mattb-4sq';
B = STREAM A THROUGH json2tsv AS (lat:float, lng:float,
venue, nick, created_at, tweet);
SF = FILTER B BY lat > 37.604031 AND lat < 37.832371 AND
lng > -123.013657 AND lng < -122.355301;
PEOPLE = GROUP SF BY nick;
PEOPLE_COUNTED = FOREACH PEOPLE GENERATE
COUNT(SF) AS c, group, SF;
ACTIVE = FILTER PEOPLE_COUNTED BY c >= 5;
RESULT = FOREACH ACTIVE GENERATE
This pig script loads up the JSON and streams it through a ruby script to turn JSON into Tab-Separated data (because
it’s easier to deal with in pig than JSON).
group,FLATTEN(SF);
STORE RESULT INTO 's3://mattb-4sq/active-sf';
43. venue, nick, created_at, tweet);
SF = FILTER B BY lat > 37.604031 AND lat < 37.832371 AND
lng > -123.013657 AND lng < -122.355301;
PEOPLE = GROUP SF BY nick;
PEOPLE_COUNTED = FOREACH PEOPLE GENERATE
COUNT(SF) AS c, group, SF;
ACTIVE = FILTER PEOPLE_COUNTED BY c >= 5;
RESULT = FOREACH ACTIVE GENERATE
group,FLATTEN(SF);
STORE RESULT INTO 's3://mattb-4sq/active-sf';
We filter the data to San Francisco lat-longs, group the data by username and count it. Then we keep only “active”
users - people with more than 5 checkins.
44. Demo:
Visualising checkins with GeoJSON and KML
You can view the path of one individual user as they arrive at SFO and get their rental car at http://maps.google.com/
maps?q=http:%2F%2Fwww.hackdiary.com%2Fmisc%2Fsampledata-
broton.kml&hl=en&ll=37.625585,-122.398124&spn=0.018015,0.040169&sll=37.0625,-95.677068&sspn=36.8631
78,82.265625&t=m&z=15&iwloc=lyrftr:kml:cFxADtCtq9UxFii5poF9Dk7kA_B4QPBI,g475427abe3071143,,
45.
46. Realistic social networks
generating a dataset of social
connections between people
What about the connections between people? What data could we use as a proxy for a large social graph?
47. Wikipedia is full of data about people and the connections between them.
48. The DBpedia project extracts just the metadata from Wikipedia - the types, the links, the geo-coordinates etc.
49. The DBpedia project extracts just the metadata from Wikipedia - the types, the links, the geo-coordinates etc.
50. It’s available as a public dataset that you can attach to an Amazon EC2 instance and look through.
51. There are many kinds of data in separate files (you can also choose your language).
52. We’re going to start with this one. It tells us what “types” each entity is on Wikipedia, parsed out from their the
Infoboxes on their pages.
54. <Autism> <type> <dbpedia.org/ontology/Disease>
<Autism> <type> <www.w3.org/2002/07/owl#Thing>
<Aristotle> <type> <dbpedia.org/ontology/Philosopher>
<Aristotle> <type> <dbpedia.org/ontology/Person>
<Aristotle> <type> <www.w3.org/2002/07/owl#Thing>
<Aristotle> <type> <xmlns.com/foaf/0.1/Person>
<Aristotle> <type> <schema.org/Person>
<Bill_Clinton> <type> <dbpedia.org/ontology/OfficeHolder>
<Bill_Clinton> <type> <dbpedia.org/ontology/Person>
<Bill_Clinton> <type> <www.w3.org/2002/07/owl#Thing>
<Bill_Clinton> <type> <xmlns.com/foaf/0.1/Person>
<Bill_Clinton> <type> <schema.org/Person>
And these are the ones we’re going to need; just the people.
55.
56. Then we’ll take the file that shows which pages link to which other Wikipedia pages.
57. <http://dbpedia.org/resource/Bill_Clinton> -> Woody_Freeman
<http://dbpedia.org/resource/Bill_Clinton> -> Yasser_Arafat
<http://dbpedia.org/resource/Bill_Dodd> -> Bill_Clinton
<http://dbpedia.org/resource/Bill_Frist> -> Bill_Clinton
<http://dbpedia.org/resource/Bob_Dylan> -> Bill_Clinton
<http://dbpedia.org/resource/Bob_Graham> -> Bill_Clinton
<http://dbpedia.org/resource/Bob_Hope> -> Bill_Clinton
And we’ll try to filter it down to just the human relationships.
58. TYPES = LOAD 's3://mattb/instance_types_en.nt.bz2' USING
PigStorage(' ') AS (subj, pred, obj, dot);
PEOPLE_TYPES = FILTER TYPES BY obj == '<http://xmlns.com/
foaf/0.1/Person>';
PEOPLE = FOREACH PEOPLE_TYPES GENERATE subj;
LINKS = LOAD 's3://mattb/page_links_en.nt.bz2' USING
PigStorage(' ') AS (subj, pred, obj, dot);
SUBJ_LINKS_CO = COGROUP PEOPLE BY subj, LINKS BY subj;
SUBJ_LINKS_FILTERED = FILTER SUBJ_LINKS_CO BY NOT
IsEmpty(PEOPLE) AND NOT IsEmpty(LINKS);
SUBJ_LINKS = FOREACH SUBJ_LINKS_FILTERED GENERATE
FLATTEN(LINKS);
OBJ_LINKS_CO = COGROUP PEOPLE BY subj, SUBJ_LINKS BY obj;
Using pig we load up the types file and filter it to just the people (the entities of type Person from the FOAF
ontology).
OBJ_LINKS_FILTERED = FILTER OBJ_LINKS_CO BY NOT
IsEmpty(PEOPLE) AND NOT IsEmpty(SUBJ_LINKS);
OBJ_LINKS = FOREACH OBJ_LINKS_FILTERED GENERATE
59. TYPES = LOAD 's3://mattb/instance_types_en.nt.bz2' USING
PigStorage(' ') AS (subj, pred, obj, dot);
PEOPLE_TYPES = FILTER TYPES BY obj == '<http://xmlns.com/
foaf/0.1/Person>';
PEOPLE = FOREACH PEOPLE_TYPES GENERATE subj;
LINKS = LOAD 's3://mattb/page_links_en.nt.bz2' USING
PigStorage(' ') AS (subj, pred, obj, dot);
SUBJ_LINKS_CO = COGROUP PEOPLE BY subj, LINKS BY subj;
SUBJ_LINKS_FILTERED = FILTER SUBJ_LINKS_CO BY NOT
IsEmpty(PEOPLE) AND NOT IsEmpty(LINKS);
SUBJ_LINKS = FOREACH SUBJ_LINKS_FILTERED GENERATE
FLATTEN(LINKS);
OBJ_LINKS_CO = COGROUP PEOPLE BY subj, SUBJ_LINKS BY obj;
We filter the links to only those whose subject (originating page) is a person.
OBJ_LINKS_FILTERED = FILTER OBJ_LINKS_CO BY NOT
IsEmpty(PEOPLE) AND NOT IsEmpty(SUBJ_LINKS);
OBJ_LINKS = FOREACH OBJ_LINKS_FILTERED GENERATE
60. OBJ_LINKS_CO = COGROUP PEOPLE BY subj, SUBJ_LINKS BY obj;
OBJ_LINKS_FILTERED = FILTER OBJ_LINKS_CO BY NOT
IsEmpty(PEOPLE) AND NOT IsEmpty(SUBJ_LINKS);
OBJ_LINKS = FOREACH OBJ_LINKS_FILTERED GENERATE
FLATTEN(SUBJ_LINKS);
D_LINKS = DISTINCT OBJ_LINKS;
STORE D_LINKS INTO 's3://mattb/people-graph' USING
PigStorage(' ');
And then filter again to only those links that link to a person.
61. OBJ_LINKS_CO = COGROUP PEOPLE BY subj, SUBJ_LINKS BY obj;
OBJ_LINKS_FILTERED = FILTER OBJ_LINKS_CO BY NOT
IsEmpty(PEOPLE) AND NOT IsEmpty(SUBJ_LINKS);
OBJ_LINKS = FOREACH OBJ_LINKS_FILTERED GENERATE
FLATTEN(SUBJ_LINKS);
D_LINKS = DISTINCT OBJ_LINKS;
STORE D_LINKS INTO 's3://mattb/people-graph' USING
PigStorage(' ');
... and store it.
62. <http://dbpedia.org/resource/Bill_Clinton> -> Woody_Freeman
<http://dbpedia.org/resource/Bill_Clinton> -> Yasser_Arafat
<http://dbpedia.org/resource/Bill_Dodd> -> Bill_Clinton
<http://dbpedia.org/resource/Bill_Frist> -> Bill_Clinton
<http://dbpedia.org/resource/Bob_Dylan> -> Bill_Clinton
<http://dbpedia.org/resource/Bob_Graham> -> Bill_Clinton
<http://dbpedia.org/resource/Bob_Hope> -> Bill_Clinton
This is the result in text.
64. Colours show the results of a “Modularity” analysis that finds the clusters of communities within the graph. For
example, the large cyan group containing Barack Obama is all government and royalty.
67. This is a great book that goes into these techniques in depth. However it’s useful for any networked data, not just
social networks. And it’s useful to anyone, not just startups.
68. This is a great book that goes into these techniques in depth. However it’s useful for any networked data, not just
social networks. And it’s useful to anyone, not just startups.
69. This is a great book that goes into these techniques in depth. However it’s useful for any networked data, not just
social networks. And it’s useful to anyone, not just startups.
70. Realistic ranking
generating a dataset of places
ordered by importance
What if we have all this data about people, places or things but we don’t know whether one thing is more important
than another? We can use public data to rank, compare and score.
71. Wikipedia makes hourly summaries of their web traffic available. Each line of each file shows the language and name
of a page on Wikipedia and how many times it was accessed that hour. We can use that attention as a proxy for the
importance of concepts.
76. Van_Ness_Avenue_%28San_Francisco%29
Recreation_Park_%28San_Francisco%29
Broadway_Tunnel_%28San_Francisco%29
Broadway_Street_%28San_Francisco%29
Carville,_San_Francisco
Union_League_Golf_and_Country_Club_of_San_Francisco
Ambassador_Hotel_%28San_Francisco%29
Columbus_Avenue_%28San_Francisco%29
Grand_Hyatt_San_Francisco
Marina_District,_San_Francisco
Pier_70,_San_Francisco
Victoria_Theatre,_San_Francisco
San_Francisco_Glacier
San_Francisco_de_Ravacayco_District
San_Francisco_church
Lafayette_Park,_San_Francisco,_California
Antioch_University_%28San_Francisco%29
San_Francisco_de_Chiu_Chiu
... which looks like this. There are over 400,000 of them.
77. DATA = LOAD 's3://wikipedia-stats/*.gz' USING
PigStorage(' ') AS (lang, name, count:int, other);
ENDATA = FILTER DATA BY lang=='en';
FEATURES = LOAD 's3://wikipedia-stats/features.txt'
USING PigStorage(' ') AS (feature);
FEATURE_CO = COGROUP ENDATA BY name,
FEATURES BY feature;
FEATURE_FILTERED = FILTER FEATURE_CO BY NOT
IsEmpty(FEATURES) AND NOT IsEmpty(ENDATA);
Using pig we filter the page traffic stats to just the English hits.
FEATURE_DATA = FOREACH FEATURE_FILTERED
GENERATE FLATTEN(ENDATA);
78. FEATURES = LOAD 's3://wikipedia-stats/features.txt'
USING PigStorage(' ') AS (feature);
FEATURE_CO = COGROUP ENDATA BY name,
FEATURES BY feature;
FEATURE_FILTERED = FILTER FEATURE_CO BY NOT
IsEmpty(FEATURES) AND NOT IsEmpty(ENDATA);
FEATURE_DATA = FOREACH FEATURE_FILTERED
GENERATE FLATTEN(ENDATA);
NAMES = GROUP FEATURE_DATA BY name;
We filter the entities down to just those that are geo-features.
COUNTS = FOREACH NAMES GENERATE group,
79. GENERATE FLATTEN(ENDATA);
NAMES = GROUP FEATURE_DATA BY name;
COUNTS = FOREACH NAMES GENERATE group,
SUM(FEATURE_DATA.count) as c;
FCOUNT = FILTER COUNTS BY c > 500;
SORTED = ORDER FCOUNT BY c DESC;
STORE SORTED INTO 's3://wikipedia-stats/
features_out.gz' USING PigStorage('t');
We group and sum the statistics by page-name.
80. Successfully read 442775 records from:
"s3://wikipedia-stats/features.txt"
Successfully read 975017055 records from:
"s3://wikipedia-stats/pagecounts-2012012*.gz"
in 4 hours, 19 minutes and 32 seconds
using 4 m1.small instances.
Using a 4-machine Elastic Mapreduce cluster I can process 50Gb of data containing nearly a billion rows in about
four hours.
81. The Castro 2479
Chinatown 2457
Tenderloin 2276
Mission District 1336
Union Square 1283
Nob Hill 952
Bayview-Hunters Point 916
Alamo Square 768
Russian Hill 721
Ocean Beach 661
San Francisco
Pacific Heights 592
Sunset District 573
neighborhoods
0 750 1500 2250
Here are some results. As you’d expect, the neighbourhoods that rank the highest are the most famous ones. Local
residential neighbourhoods come lower down the scale.
82. Hackney 3428
Camden 2498
Tower Hamlets 2378
Newham 1850
Enfield 1830
Croydon 1796
Islington 1624
Southwark 1603
Lambeth 1354
Greenwich 1316
Hammersmith and Fulham 1268
Haringey 1263 London
Harrow 1183 neighbourhoods
Brent 1140
0 1000 2000 3000
Here it is again for London.
83. To demo this ranking in a data toy that anyone can play with, I built an auto-completer using Elasticsearch. I
transformed the pig output into JSON and made an index.
84. Demo:
A weighted autocompleter with Elasticsearch
I exposed this index through a small Ruby webapp written in Sinatra.
85. So we can easily answer questions like “which of the world’s many Chinatown districts are the best-known?”
86. All code for the workshop:
https://github.com/mattb/where2012-workshop
