SlideShare a Scribd company logo

Geo Analytics Tutorial - Where 2.0 2011

•Download as KEY, PDF•
•
Peter Skomoroch
Peter Skomoroch

O'Reilly Where 2.0 2011 As a result of cheap storage and computing power, society is measuring and storing increasing amounts of information. It is now possible to efficiently crunch Petabytes of data with tools like Hadoop. In this O'Reilly Where 2.0 tutorial, Pete Skomoroch, Sr. Data Scientist at LinkedIn, gives an overview of spatial analytics and how you can use tools like Hadoop, Python, and Mechanical Turk to process location data and derive insights about cities and people. Topics: * Data Science & Geo Analytics * Useful Geo tools and Datasets * Hadoop, Pig, and Big Data * Cleaning Location Data with Mechanical Turk * Spatial Tweet Analytics with Hadoop & Python * Using Social Data to Understand Cities

Technology
1 of 89
Geo Analytics Tutorial Pete Skomoroch Sr. Data Scientist - LinkedIn (@peteskomoroch) #geoanalytics ** Hadoop Intro slides from Kevin Weil, Twitter
Topics ‣ Data Science & Geo Analytics ‣ Useful Geo tools and Datasets ‣ Hadoop, Pig, and Big Data ‣ Cleaning Location Data with Mechanical Turk ‣ Spatial Tweet Analytics with Hadoop & Python ‣ Using Social Data to Understand Cities ‣ Q&A
Topics ‣ Data Science & Geo Analytics ‣ Useful Geo tools and Datasets ‣ Hadoop, Pig, and Big Data ‣ Cleaning Location Data with Mechanical Turk ‣ Spatial Tweet Analytics with Hadoop & Python ‣ Using Social Data to Understand Cities ‣ Q&A
Analytics & Data are Hot Topics
Analytics & Data are Hot Topics
Analytics & Data are Hot Topics
Analytics & Data are Hot Topics
Analytics & Data are Hot Topics
Analytics & Data are Hot Topics
Data Exhaust My Delicious Tags
Data Science * http://www.drewconway.com/zia/?p=2378
Data Visualization ‣ http://www.dataspora.com/blog/
Spatial Analysis Map by Dr. John Snow of London, showing clusters of cholera cases in the 1854 Broad Street cholera outbreak. This was one of the first uses of map-based spatial analysis.
Spatial Analysis • Spatial regression - estimate dependencies between variables • Gravity models - estimate the flow of people, material, or information between locations • Spatial interpolation - estimate variables at unobserved locations based on other measured values • Simulation - use models and data to predict spatial phenomena
Life Span & Food by Zip Code * http://zev.lacounty.gov/news/health/death-by-zip-code * http://www.verysmallarray.com/?p=975
Where Americans Are Moving (IRS Data) ‣ (Jon Bruner) http://jebruner.com/2010/06/the-migration-map/
Facebook Connectivity (Pete Warden) * http://petewarden.typepad.com/searchbrowser/2010/02/how-to-split-up-the-us.html
Topics ‣ Data Science & Geo Analytics ‣ Useful Geo tools and Datasets ‣ Hadoop, Pig, and Big Data ‣ Cleaning Location Data with Mechanical Turk ‣ Spatial Tweet Analytics with Hadoop & Python ‣ Using Social Data to Understand Cities ‣ Q&A
Useful Geo Tools •R, Matlab, SciPy, Commercial Geo Software •R Spatial Pkgs http://cran.r-project.org/web/views/Spatial.html •Hadoop, Amazon EC2, Mechanical Turk •Data Science Toolkit: http://www.datasciencetoolkit.org/ •80% of effort is often in cleaning and processing data
DataScienceToolkit.org •Runs on VM or Amazon EC2 •Street Address to Coordinates •Coordinates to Political Areas •Geodict (text extraction) •IP Address to Coordinates •New UK release on Github
Resources for location data • SimpleGeo • Factual • Geonames • Infochimps • Data.gov • DataWrangling.com
Topics ‣ Data Science & Geo Analytics ‣ Useful Geo tools and Datasets ‣ Hadoop, Pig, and Big Data ‣ Cleaning Location Data with Mechanical Turk ‣ Spatial Tweet Analytics with Hadoop & Python ‣ Using Social Data to Understand Cities ‣ Q&A
Hadoop: Motivation •We want to crunch 1TB of Twitter stream data and understand spatial patterns in Tweets •Data collected from the Twitter “Garden Hose” API last Spring
Data is Getting Big ‣ NYSE: 1 TB/day ‣ Facebook: 20+ TB compressed/day ‣ CERN/LHC: 40 TB/day (15 PB/year!) ‣ And growth is accelerating ‣ Need multiple machines, horizontal scalability
Hadoop ‣ Distributed file system (hard to store a PB) ‣ Fault-tolerant, handles replication, node failure, etc ‣ MapReduce-based parallel computation (even harder to process a PB) ‣ Generic key-value based computation interface allows for wide applicability ‣ Open source, top-level Apache project ‣ Scalable: Y! has a 4000-node cluster ‣ Powerful: sorted a TB of random integers in 62 seconds
MapReduce? cat file | grep geo | sort | uniq -c > ‣ Challenge: how many tweets per output county, given tweets table? ‣ Input: key=row, value=tweet info ‣ Map: output key=county, value=1 ‣ Shuffle: sort by county ‣ Reduce: for each county, sum ‣ Output: county, tweet count ‣ With 2x machines, runs close to 2x faster.
MapReduce? cat file | grep geo | sort | uniq -c > ‣ Challenge: how many tweets per output county, given tweets table? ‣ Input: key=row, value=tweet info ‣ Map: output key=county, value=1 ‣ Shuffle: sort by county ‣ Reduce: for each county, sum ‣ Output: county, tweet count ‣ With 2x machines, runs close to 2x faster.
MapReduce? cat file | grep geo | sort | uniq -c > ‣ Challenge: how many tweets per output county, given tweets table? ‣ Input: key=row, value=tweet info ‣ Map: output key=county, value=1 ‣ Shuffle: sort by county ‣ Reduce: for each county, sum ‣ Output: county, tweet count ‣ With 2x machines, runs close to 2x faster.
MapReduce? cat file | grep geo | sort | uniq -c > ‣ Challenge: how many tweets per output county, given tweets table? ‣ Input: key=row, value=tweet info ‣ Map: output key=county, value=1 ‣ Shuffle: sort by county ‣ Reduce: for each county, sum ‣ Output: county, tweet count ‣ With 2x machines, runs close to 2x faster.
MapReduce? cat file | grep geo | sort | uniq -c > ‣ Challenge: how many tweets per output county, given tweets table? ‣ Input: key=row, value=tweet info ‣ Map: output key=county, value=1 ‣ Shuffle: sort by county ‣ Reduce: for each county, sum ‣ Output: county, tweet count ‣ With 2x machines, runs close to 2x faster.
MapReduce? cat file | grep geo | sort | uniq -c > ‣ Challenge: how many tweets per output county, given tweets table? ‣ Input: key=row, value=tweet info ‣ Map: output key=county, value=1 ‣ Shuffle: sort by county ‣ Reduce: for each county, sum ‣ Output: county, tweet count ‣ With 2x machines, runs close to 2x faster.
MapReduce? cat file | grep geo | sort | uniq -c > ‣ Challenge: how many tweets per output county, given tweets table? ‣ Input: key=row, value=tweet info ‣ Map: output key=county, value=1 ‣ Shuffle: sort by county ‣ Reduce: for each county, sum ‣ Output: county, tweet count ‣ With 2x machines, runs close to 2x faster.
But... ‣ Analysis typically done in Java ‣ Single-input, two-stage data flow is rigid ‣ Projections, filters: custom code ‣ Joins: lengthy, error-prone ‣ n-stage jobs: Hard to manage ‣ Prototyping/exploration requires ‣ analytics in Eclipse? compilation ur doin it wrong...
Enter Pig ‣ High level language ‣ Transformations on sets of records ‣ Process data one step at a time ‣ Easier than SQL?
Why Pig? ‣ Because I bet you can read the following script.
A Real Pig Script ‣ Now, just for fun... the same calculation in vanilla Hadoop MapReduce.
No, seriously.
Pig Simplifies Analysis ‣ The Pig version is: ‣ 5% of the code, 5% of the time ‣ Within 50% of the execution time. ‣ Pig Geo: ‣ Programmable: fuzzy matching, custom filtering ‣ Easily link multiple datasets, regardless of size/structure ‣ Iterative, quick
A Real Example ‣ Fire up your Elastic MapReduce Cluster. ‣ ... or follow along at http://bit.ly/whereanalytics ‣ I used Twitter’s streaming API to store some tweets ‣ Simplest thing: group by location and count with Pig ‣ http://bit.ly/where20pig ‣ Here comes some code!
tweets = LOAD 's3://where20demo/sample-tweets' as ( user_screen_name:chararray, tweet_id:chararray, ... user_friends_count:int, user_statuses_count:int, user_location:chararray, user_lang:chararray, user_time_zone:chararray, place_id:chararray, ...);
tweets = LOAD 's3://where20demo/sample-tweets' as ( user_screen_name:chararray, tweet_id:chararray, ... user_friends_count:int, user_statuses_count:int, user_location:chararray, user_lang:chararray, user_time_zone:chararray, place_id:chararray, ...);
tweets_with_location = FILTER tweets BY user_location != 'NULL';
normalized_locations = FOREACH tweets_with_location GENERATE LOWER(user_location) as user_location;
grouped_tweets = GROUP normalized_locations BY user_location PARALLEL 10;
location_counts = FOREACH grouped_tweets GENERATE $0 as location, SIZE($1) as user_count;
sorted_counts = ORDER location_counts BY user_count DESC;
STORE sorted_counts INTO 'global_location_tweets';
hadoop@ip-10-160-113-142:~$ hadoop dfs -cat /global_location_counts/part* | head -30 brasil 37985 indonesia 33777 brazil 22432 london 17294 usa 14564 são paulo 14238 new york 13420 tokyo 10967 singapore 10225 rio de janeiro 10135 los angeles 9934 california 9386 chicago 9155 uk 9095 jakarta 9086 germany 8741 canada 8201 7696 7121 jakarta, indonesia 6480 nyc 6456 new york, ny 6331
Neat, but... ‣ Wow, that data is messy! ‣ brasil, brazil at #1 and #3 ‣ new york, nyc, and new york ny all in the top 30 ‣ Mechanical Turk to the rescue...
Topics ‣ Data Science & Geo Analytics ‣ Useful Geo tools and Datasets ‣ Hadoop, Pig, and Big Data ‣ Cleaning Location Data with Mechanical Turk ‣ Spatial Tweet Analytics with Hadoop & Python ‣ Using Social Data to Understand Cities ‣ Q&A
Code examples we’ll cover are on Github
You can run them on Elastic MapReduce
Cleaning Twitter Profile Location Names Filter Exact Matches Extract Top Tweet Locations Clean with MTurk Aggregate Context with Hadoop
We will map locations to GeoNames IDs
Start with Location Exact Matches
Use Mechanical Turk to improve results
Workers do simple tasks for a few cents
We constructed the following task
Workers used a Geonames search tool
Location search tool code is on Github
Preparing Data to send to MTurk
We use consensus answers from workers
Processing MTurk Output
Topics ‣ Data Science & Geo Analytics ‣ Useful Geo tools and Datasets ‣ Hadoop, Pig, and Big Data ‣ Cleaning Location Data with Mechanical Turk ‣ Spatial Tweet Analytics with Hadoop & Python ‣ Using Social Data to Understand Cities ‣ Q&A
Tokenizing and Cleaning Tweet Text ‣ Extract Tweet topics with Hadoop + Python + NLTK + Wikipedia
Build Phrase Dictionary with Wikipedia
Streaming Tweet Parser (Python + NLTK)
Parse Tweets and Join to Wikipedia (Pig)
Aggregate by US County for Analysis
Clean Data => Thematic US County Map
Twitter users by county in our sample
“Lady Gaga” Tweets
“Tea Party” Tweets
“Dallas” Tweets
“Stephen Colbert” Tweets
Topics ‣ Data Science & Geo Analytics ‣ Useful Geo tools and Datasets ‣ Hadoop, Pig, and Big Data ‣ Cleaning Location Data with Mechanical Turk ‣ Spatial Tweet Analytics with Hadoop & Python ‣ Using Social Data to Understand Cities ‣ Q&A
LinkedIn Skills
Skills in the Design Industry
Exploring the Spatial Distribution of Skills
People with “Ship Building” Skills
What is the Skill profile of a given city?
Expertise correlated with Santa Clara, CA
Expertise correlated with Los Angeles
Expertise correlated with Washington, DC
Yuba City, CA has 21.3% Unemployment 21.3
Ames, Iowa has 4.7% Unemployment 21.3
Topics ‣ Data Science & Geo Analytics ‣ Useful Geo tools and Datasets ‣ Hadoop, Pig, and Big Data ‣ Cleaning Location Data with Mechanical Turk ‣ Spatial Tweet Analytics with Hadoop & Python ‣ Using Social Data to Understand Cities ‣ Q&A
Questions? Follow me at twitter.com/peteskomoroch datawrangling.com

Recommended

The Data Science Process
The Data Science ProcessThe Data Science Process
The Data Science ProcessVishal Patel
 
Big data mining
Big data miningBig data mining
Big data miningThadsanamoorthy Kajavathanan
 
Big data visualization
Big data visualizationBig data visualization
Big data visualizationAnurag Gupta
 
Data preprocessing PPT
Data preprocessing PPTData preprocessing PPT
Data preprocessing PPTANUSUYA T K
 
Missing Data and Causes
Missing Data and CausesMissing Data and Causes
Missing Data and Causesakanni azeez olamide
 
Data science applications and usecases
Data science applications and usecasesData science applications and usecases
Data science applications and usecasesSreenatha Reddy K R
 
Machine Learning and Data Mining
Machine Learning and Data MiningMachine Learning and Data Mining
Machine Learning and Data MiningTilani Gunawardena PhD(UNIBAS), BSc(Pera), FHEA(UK), CEng, MIESL
 
Introduction to Data Analytics
Introduction to Data AnalyticsIntroduction to Data Analytics
Introduction to Data AnalyticsUtkarsh Sharma
 

Recommended

The Data Science Process
The Data Science ProcessThe Data Science Process
The Data Science ProcessVishal Patel
 
Big data mining
Big data miningBig data mining
Big data miningThadsanamoorthy Kajavathanan
 
Big data visualization
Big data visualizationBig data visualization
Big data visualizationAnurag Gupta
 
Data preprocessing PPT
Data preprocessing PPTData preprocessing PPT
Data preprocessing PPTANUSUYA T K
 
Missing Data and Causes
Missing Data and CausesMissing Data and Causes
Missing Data and Causesakanni azeez olamide
 
Data science applications and usecases
Data science applications and usecasesData science applications and usecases
Data science applications and usecasesSreenatha Reddy K R
 
Machine Learning and Data Mining
Machine Learning and Data MiningMachine Learning and Data Mining
Machine Learning and Data MiningTilani Gunawardena PhD(UNIBAS), BSc(Pera), FHEA(UK), CEng, MIESL
 
Introduction to Data Analytics
Introduction to Data AnalyticsIntroduction to Data Analytics
Introduction to Data AnalyticsUtkarsh Sharma
 
Data preprocessing using Machine Learning
Data preprocessing using Machine Learning Data preprocessing using Machine Learning
Data preprocessing using Machine Learning Gopal Sakarkar
 
Privacy, security and ethics in data science
Privacy, security and ethics in data sciencePrivacy, security and ethics in data science
Privacy, security and ethics in data scienceNikolaos Vasiloglou
 
Statistics for data science
Statistics for data science Statistics for data science
Statistics for data science zekeLabs Technologies
 
Data analytics
Data analyticsData analytics
Data analyticsdavidfergarcia
 
Data analytics vs. Data analysis
Data analytics vs. Data analysisData analytics vs. Data analysis
Data analytics vs. Data analysisDr. C.V. Suresh Babu
 
2.4 rule based classification
2.4 rule based classification2.4 rule based classification
2.4 rule based classificationKrish_ver2
 
Machine Learning-Linear regression
Machine Learning-Linear regressionMachine Learning-Linear regression
Machine Learning-Linear regressionkishanthkumaar
 
Regression analysis in excel
Regression analysis in excelRegression analysis in excel
Regression analysis in excelAwais Salman
 
Wine Quality Analysis Using Machine Learning
Wine Quality Analysis Using Machine LearningWine Quality Analysis Using Machine Learning
Wine Quality Analysis Using Machine LearningMahima -
 
Data preprocessing
Data preprocessingData preprocessing
Data preprocessingGajanand Sharma
 
Project on disease prediction
Project on disease predictionProject on disease prediction
Project on disease predictionKOYELMAJUMDAR1
 
Python for Data Science
Python for Data SciencePython for Data Science
Python for Data ScienceHarri Hämäläinen
 
Exploratory Data Analysis
Exploratory Data AnalysisExploratory Data Analysis
Exploratory Data AnalysisUmair Shafique
 
Data analysis with R
Data analysis with RData analysis with R
Data analysis with RShareThis
 
Dimensionality Reduction
Dimensionality ReductionDimensionality Reduction
Dimensionality ReductionKnoldus Inc.
 
data mining
data miningdata mining
data mininguoitc
 
Principal Component Analysis (PCA) and LDA PPT Slides
Principal Component Analysis (PCA) and LDA PPT SlidesPrincipal Component Analysis (PCA) and LDA PPT Slides
Principal Component Analysis (PCA) and LDA PPT SlidesAbhishekKumar4995
 
Presentation on supervised learning
Presentation on supervised learningPresentation on supervised learning
Presentation on supervised learningTonmoy Bhagawati
 
Predictive Analytics - An Overview
Predictive Analytics - An OverviewPredictive Analytics - An Overview
Predictive Analytics - An OverviewMachinePulse
 
Data science presentation
Data science presentationData science presentation
Data science presentationMSDEVMTL
 
Geo data analytics
Geo data analyticsGeo data analytics
Geo data analyticsDaniel Marcous
 
Enrich Gis With Social Media And Open Data
Enrich Gis With Social Media And Open DataEnrich Gis With Social Media And Open Data
Enrich Gis With Social Media And Open DataJan Willem van Eck
 

More Related Content

What's hot

Data preprocessing using Machine Learning
Data preprocessing using Machine Learning Data preprocessing using Machine Learning
Data preprocessing using Machine Learning Gopal Sakarkar
 
Privacy, security and ethics in data science
Privacy, security and ethics in data sciencePrivacy, security and ethics in data science
Privacy, security and ethics in data scienceNikolaos Vasiloglou
 
2.4 rule based classification
2.4 rule based classification2.4 rule based classification
2.4 rule based classificationKrish_ver2
 
Machine Learning-Linear regression
Machine Learning-Linear regressionMachine Learning-Linear regression
Machine Learning-Linear regressionkishanthkumaar
 
Regression analysis in excel
Regression analysis in excelRegression analysis in excel
Regression analysis in excelAwais Salman
 
Wine Quality Analysis Using Machine Learning
Wine Quality Analysis Using Machine LearningWine Quality Analysis Using Machine Learning
Wine Quality Analysis Using Machine LearningMahima -
 
Project on disease prediction
Project on disease predictionProject on disease prediction
Project on disease predictionKOYELMAJUMDAR1
 
Exploratory Data Analysis
Exploratory Data AnalysisExploratory Data Analysis
Exploratory Data AnalysisUmair Shafique
 
Data analysis with R
Data analysis with RData analysis with R
Data analysis with RShareThis
 
Dimensionality Reduction
Dimensionality ReductionDimensionality Reduction
Dimensionality ReductionKnoldus Inc.
 
data mining
data miningdata mining
data mininguoitc
 
Principal Component Analysis (PCA) and LDA PPT Slides
Principal Component Analysis (PCA) and LDA PPT SlidesPrincipal Component Analysis (PCA) and LDA PPT Slides
Principal Component Analysis (PCA) and LDA PPT SlidesAbhishekKumar4995
 
Presentation on supervised learning
Presentation on supervised learningPresentation on supervised learning
Presentation on supervised learningTonmoy Bhagawati
 
Predictive Analytics - An Overview
Predictive Analytics - An OverviewPredictive Analytics - An Overview
Predictive Analytics - An OverviewMachinePulse
 
Data science presentation
Data science presentationData science presentation
Data science presentationMSDEVMTL
 

What's hot (20)

Data preprocessing using Machine Learning
Data preprocessing using Machine Learning Data preprocessing using Machine Learning
Data preprocessing using Machine Learning
 
Privacy, security and ethics in data science
Privacy, security and ethics in data sciencePrivacy, security and ethics in data science
Privacy, security and ethics in data science
 
Statistics for data science
Statistics for data science Statistics for data science
Statistics for data science
 
Data analytics
Data analyticsData analytics
Data analytics
 
Data analytics vs. Data analysis
Data analytics vs. Data analysisData analytics vs. Data analysis
Data analytics vs. Data analysis
 
2.4 rule based classification
2.4 rule based classification2.4 rule based classification
2.4 rule based classification
 
Machine Learning-Linear regression
Machine Learning-Linear regressionMachine Learning-Linear regression
Machine Learning-Linear regression
 
Regression analysis in excel
Regression analysis in excelRegression analysis in excel
Regression analysis in excel
 
Wine Quality Analysis Using Machine Learning
Wine Quality Analysis Using Machine LearningWine Quality Analysis Using Machine Learning
Wine Quality Analysis Using Machine Learning
 
Data preprocessing
Data preprocessingData preprocessing
Data preprocessing
 
Project on disease prediction
Project on disease predictionProject on disease prediction
Project on disease prediction
 
Python for Data Science
Python for Data SciencePython for Data Science
Python for Data Science
 
Exploratory Data Analysis
Exploratory Data AnalysisExploratory Data Analysis
Exploratory Data Analysis
 
Data analysis with R
Data analysis with RData analysis with R
Data analysis with R
 
Dimensionality Reduction
Dimensionality ReductionDimensionality Reduction
Dimensionality Reduction
 
data mining
data miningdata mining
data mining
 
Principal Component Analysis (PCA) and LDA PPT Slides
Principal Component Analysis (PCA) and LDA PPT SlidesPrincipal Component Analysis (PCA) and LDA PPT Slides
Principal Component Analysis (PCA) and LDA PPT Slides
 
Presentation on supervised learning
Presentation on supervised learningPresentation on supervised learning
Presentation on supervised learning
 
Predictive Analytics - An Overview
Predictive Analytics - An OverviewPredictive Analytics - An Overview
Predictive Analytics - An Overview
 
Data science presentation
Data science presentationData science presentation
Data science presentation
 

Viewers also liked

Enrich Gis With Social Media And Open Data
Enrich Gis With Social Media And Open DataEnrich Gis With Social Media And Open Data
Enrich Gis With Social Media And Open DataJan Willem van Eck
 
Data mining in big databases with geo reference and easy web publishing and s...
Data mining in big databases with geo reference and easy web publishing and s...Data mining in big databases with geo reference and easy web publishing and s...
Data mining in big databases with geo reference and easy web publishing and s...MapWindow GIS
 
Atlas Of Cambodia 2007
Atlas Of Cambodia 2007Atlas Of Cambodia 2007
Atlas Of Cambodia 2007Jan-Peter Mund
 
Geo-analytics Architecture - Technologies
Geo-analytics Architecture - TechnologiesGeo-analytics Architecture - Technologies
Geo-analytics Architecture - TechnologiesBlue BRIDGE
 
Big Data: Small Screen Location-Based Services 2.0
Big Data: Small Screen Location-Based Services 2.0 Big Data: Small Screen Location-Based Services 2.0
Big Data: Small Screen Location-Based Services 2.0 Kevin Foreman
 
Citizen science, vgi, geo crowd sourcing, big geo data how they matter to th...
Citizen science, vgi, geo crowd sourcing, big geo data how they matter to th...Citizen science, vgi, geo crowd sourcing, big geo data how they matter to th...
Citizen science, vgi, geo crowd sourcing, big geo data how they matter to th...Maria Antonia Brovelli
 
The Power of Geo Analytics (and maps) to Improve Predictive Analytics in Heal...
The Power of Geo Analytics (and maps) to Improve Predictive Analytics in Heal...The Power of Geo Analytics (and maps) to Improve Predictive Analytics in Heal...
The Power of Geo Analytics (and maps) to Improve Predictive Analytics in Heal...Health Catalyst
 
Leveraging Geo-Spatial (Big) Data for Financial Services Solutions
Leveraging Geo-Spatial (Big) Data for Financial Services SolutionsLeveraging Geo-Spatial (Big) Data for Financial Services Solutions
Leveraging Geo-Spatial (Big) Data for Financial Services SolutionsCapgemini
 
DataMeet 4: Data cleaning & census data
DataMeet 4: Data cleaning & census dataDataMeet 4: Data cleaning & census data
DataMeet 4: Data cleaning & census dataRitvvij Parrikh
 
Charles Birnbaum (Foursquare) «Big data and user generated content LBS»
Charles Birnbaum (Foursquare) «Big data and user generated content LBS»Charles Birnbaum (Foursquare) «Big data and user generated content LBS»
Charles Birnbaum (Foursquare) «Big data and user generated content LBS»e-Legion
 
Lecture8 rocks
Lecture8 rocksLecture8 rocks
Lecture8 rocksairporte
 
Linked (Geo) Data - Adding a Spatial Dimension to the Web of Data
Linked (Geo) Data - Adding a Spatial Dimension to the Web of DataLinked (Geo) Data - Adding a Spatial Dimension to the Web of Data
Linked (Geo) Data - Adding a Spatial Dimension to the Web of DataAndreas Langegger
 
Dw & etl concepts
Dw & etl conceptsDw & etl concepts
Dw & etl conceptsjeshocarme
 
SXSW Keynote - The Game Layer On Top Of The World
SXSW Keynote - The Game Layer On Top Of The WorldSXSW Keynote - The Game Layer On Top Of The World
SXSW Keynote - The Game Layer On Top Of The WorldSeth Priebatsch
 

Viewers also liked (20)

Geo data analytics
Geo data analyticsGeo data analytics
Geo data analytics
 
Enrich Gis With Social Media And Open Data
Enrich Gis With Social Media And Open DataEnrich Gis With Social Media And Open Data
Enrich Gis With Social Media And Open Data
 
Data mining in big databases with geo reference and easy web publishing and s...
Data mining in big databases with geo reference and easy web publishing and s...Data mining in big databases with geo reference and easy web publishing and s...
Data mining in big databases with geo reference and easy web publishing and s...
 
Atlas Of Cambodia 2007
Atlas Of Cambodia 2007Atlas Of Cambodia 2007
Atlas Of Cambodia 2007
 
Get Big Geo Data
Get Big Geo DataGet Big Geo Data
Get Big Geo Data
 
Geo-analytics Architecture - Technologies
Geo-analytics Architecture - TechnologiesGeo-analytics Architecture - Technologies
Geo-analytics Architecture - Technologies
 
Intro to open refine
Intro to open refineIntro to open refine
Intro to open refine
 
Big Data: Small Screen Location-Based Services 2.0
Big Data: Small Screen Location-Based Services 2.0 Big Data: Small Screen Location-Based Services 2.0
Big Data: Small Screen Location-Based Services 2.0
 
Citizen science, vgi, geo crowd sourcing, big geo data how they matter to th...
Citizen science, vgi, geo crowd sourcing, big geo data how they matter to th...Citizen science, vgi, geo crowd sourcing, big geo data how they matter to th...
Citizen science, vgi, geo crowd sourcing, big geo data how they matter to th...
 
The Power of Geo Analytics (and maps) to Improve Predictive Analytics in Heal...
The Power of Geo Analytics (and maps) to Improve Predictive Analytics in Heal...The Power of Geo Analytics (and maps) to Improve Predictive Analytics in Heal...
The Power of Geo Analytics (and maps) to Improve Predictive Analytics in Heal...
 
Leveraging Geo-Spatial (Big) Data for Financial Services Solutions
Leveraging Geo-Spatial (Big) Data for Financial Services SolutionsLeveraging Geo-Spatial (Big) Data for Financial Services Solutions
Leveraging Geo-Spatial (Big) Data for Financial Services Solutions
 
Mobile LBS
Mobile LBSMobile LBS
Mobile LBS
 
NFC, LBS and Smart Screens in Transport
NFC, LBS and Smart Screens in Transport NFC, LBS and Smart Screens in Transport
NFC, LBS and Smart Screens in Transport
 
DataMeet 4: Data cleaning & census data
DataMeet 4: Data cleaning & census dataDataMeet 4: Data cleaning & census data
DataMeet 4: Data cleaning & census data
 
Charles Birnbaum (Foursquare) «Big data and user generated content LBS»
Charles Birnbaum (Foursquare) «Big data and user generated content LBS»Charles Birnbaum (Foursquare) «Big data and user generated content LBS»
Charles Birnbaum (Foursquare) «Big data and user generated content LBS»
 
Lecture8 rocks
Lecture8 rocksLecture8 rocks
Lecture8 rocks
 
Linked (Geo) Data - Adding a Spatial Dimension to the Web of Data
Linked (Geo) Data - Adding a Spatial Dimension to the Web of DataLinked (Geo) Data - Adding a Spatial Dimension to the Web of Data
Linked (Geo) Data - Adding a Spatial Dimension to the Web of Data
 
Big Data in Retail
Big Data in RetailBig Data in Retail
Big Data in Retail
 
Dw & etl concepts
Dw & etl conceptsDw & etl concepts
Dw & etl concepts
 
SXSW Keynote - The Game Layer On Top Of The World
SXSW Keynote - The Game Layer On Top Of The WorldSXSW Keynote - The Game Layer On Top Of The World
SXSW Keynote - The Game Layer On Top Of The World
 

Similar to Geo Analytics Tutorial - Where 2.0 2011

NoSQL at Twitter (NoSQL EU 2010)
NoSQL at Twitter (NoSQL EU 2010)NoSQL at Twitter (NoSQL EU 2010)
NoSQL at Twitter (NoSQL EU 2010)Kevin Weil
 
Analyzing Big Data at Twitter (Web 2.0 Expo NYC Sep 2010)
Analyzing Big Data at Twitter (Web 2.0 Expo NYC Sep 2010)Analyzing Big Data at Twitter (Web 2.0 Expo NYC Sep 2010)
Analyzing Big Data at Twitter (Web 2.0 Expo NYC Sep 2010)Kevin Weil
 
Spatial Analytics, Where 2.0 2010
Spatial Analytics, Where 2.0 2010Spatial Analytics, Where 2.0 2010
Spatial Analytics, Where 2.0 2010Kevin Weil
 
Where20 Spatial Analytics 2010
Where20 Spatial Analytics 2010Where20 Spatial Analytics 2010
Where20 Spatial Analytics 2010seagor
 
Hadoop, Pig, and Twitter (NoSQL East 2009)
Hadoop, Pig, and Twitter (NoSQL East 2009)Hadoop, Pig, and Twitter (NoSQL East 2009)
Hadoop, Pig, and Twitter (NoSQL East 2009)Kevin Weil
 
Hadoop and pig at twitter (oscon 2010)
Hadoop and pig at twitter (oscon 2010)Hadoop and pig at twitter (oscon 2010)
Hadoop and pig at twitter (oscon 2010)Kevin Weil
 
Big Data at Twitter, Chirp 2010
Big Data at Twitter, Chirp 2010Big Data at Twitter, Chirp 2010
Big Data at Twitter, Chirp 2010Kevin Weil
 
Big Data Analytics with Hadoop with @techmilind
Big Data Analytics with Hadoop with @techmilindBig Data Analytics with Hadoop with @techmilind
Big Data Analytics with Hadoop with @techmilindEMC
 
Concurrency in Distributed Systems : Leslie Lamport papers
Concurrency in Distributed Systems : Leslie Lamport papersConcurrency in Distributed Systems : Leslie Lamport papers
Concurrency in Distributed Systems : Leslie Lamport papersSubhajit Sahu
 
Terascale Learning
Terascale LearningTerascale Learning
Terascale Learningpauldix
 
Online learning, Vowpal Wabbit and Hadoop
Online learning, Vowpal Wabbit and HadoopOnline learning, Vowpal Wabbit and Hadoop
Online learning, Vowpal Wabbit and HadoopHéloïse Nonne
 
Hadoop 101 for bioinformaticians
Hadoop 101 for bioinformaticiansHadoop 101 for bioinformaticians
Hadoop 101 for bioinformaticiansattilacsordas
 
COCOA: Communication-Efficient Coordinate Ascent
COCOA: Communication-Efficient Coordinate AscentCOCOA: Communication-Efficient Coordinate Ascent
COCOA: Communication-Efficient Coordinate Ascentjeykottalam
 
Introduction to Big Data
Introduction to Big DataIntroduction to Big Data
Introduction to Big DataAlbert Bifet
 
Processing Big Data in Real-Time - Yanai Franchi, Tikal
Processing Big Data in Real-Time - Yanai Franchi, TikalProcessing Big Data in Real-Time - Yanai Franchi, Tikal
Processing Big Data in Real-Time - Yanai Franchi, TikalCodemotion Tel Aviv
 
Graph processing
Graph processingGraph processing
Graph processingyeahjs
 
Cliff Click Explains GBM at Netflix October 10 2013
Cliff Click Explains GBM at Netflix October 10 2013Cliff Click Explains GBM at Netflix October 10 2013
Cliff Click Explains GBM at Netflix October 10 2013Sri Ambati
 
Hadoop Streaming Tutorial With Python
Hadoop Streaming Tutorial With PythonHadoop Streaming Tutorial With Python
Hadoop Streaming Tutorial With PythonJoe Stein
 

Similar to Geo Analytics Tutorial - Where 2.0 2011 (20)

NoSQL at Twitter (NoSQL EU 2010)
NoSQL at Twitter (NoSQL EU 2010)NoSQL at Twitter (NoSQL EU 2010)
NoSQL at Twitter (NoSQL EU 2010)
 
Analyzing Big Data at Twitter (Web 2.0 Expo NYC Sep 2010)
Analyzing Big Data at Twitter (Web 2.0 Expo NYC Sep 2010)Analyzing Big Data at Twitter (Web 2.0 Expo NYC Sep 2010)
Analyzing Big Data at Twitter (Web 2.0 Expo NYC Sep 2010)
 
Spatial Analytics, Where 2.0 2010
Spatial Analytics, Where 2.0 2010Spatial Analytics, Where 2.0 2010
Spatial Analytics, Where 2.0 2010
 
Where20 Spatial Analytics 2010
Where20 Spatial Analytics 2010Where20 Spatial Analytics 2010
Where20 Spatial Analytics 2010
 
Hadoop, Pig, and Twitter (NoSQL East 2009)
Hadoop, Pig, and Twitter (NoSQL East 2009)Hadoop, Pig, and Twitter (NoSQL East 2009)
Hadoop, Pig, and Twitter (NoSQL East 2009)
 
Hadoop and pig at twitter (oscon 2010)
Hadoop and pig at twitter (oscon 2010)Hadoop and pig at twitter (oscon 2010)
Hadoop and pig at twitter (oscon 2010)
 
Big Data at Twitter, Chirp 2010
Big Data at Twitter, Chirp 2010Big Data at Twitter, Chirp 2010
Big Data at Twitter, Chirp 2010
 
Big Data Analytics with Hadoop with @techmilind
Big Data Analytics with Hadoop with @techmilindBig Data Analytics with Hadoop with @techmilind
Big Data Analytics with Hadoop with @techmilind
 
MapReduce Algorithm Design
MapReduce Algorithm DesignMapReduce Algorithm Design
MapReduce Algorithm Design
 
Concurrency in Distributed Systems : Leslie Lamport papers
Concurrency in Distributed Systems : Leslie Lamport papersConcurrency in Distributed Systems : Leslie Lamport papers
Concurrency in Distributed Systems : Leslie Lamport papers
 
Terascale Learning
Terascale LearningTerascale Learning
Terascale Learning
 
Online learning, Vowpal Wabbit and Hadoop
Online learning, Vowpal Wabbit and HadoopOnline learning, Vowpal Wabbit and Hadoop
Online learning, Vowpal Wabbit and Hadoop
 
Hadoop 101 for bioinformaticians
Hadoop 101 for bioinformaticiansHadoop 101 for bioinformaticians
Hadoop 101 for bioinformaticians
 
COCOA: Communication-Efficient Coordinate Ascent
COCOA: Communication-Efficient Coordinate AscentCOCOA: Communication-Efficient Coordinate Ascent
COCOA: Communication-Efficient Coordinate Ascent
 
Introduction to Big Data
Introduction to Big DataIntroduction to Big Data
Introduction to Big Data
 
Processing Big Data in Real-Time - Yanai Franchi, Tikal
Processing Big Data in Real-Time - Yanai Franchi, TikalProcessing Big Data in Real-Time - Yanai Franchi, Tikal
Processing Big Data in Real-Time - Yanai Franchi, Tikal
 
Graph processing
Graph processingGraph processing
Graph processing
 
Using MapReduce for Large–scale Medical Image Analysis
Using MapReduce for Large–scale Medical Image AnalysisUsing MapReduce for Large–scale Medical Image Analysis
Using MapReduce for Large–scale Medical Image Analysis
 
Cliff Click Explains GBM at Netflix October 10 2013
Cliff Click Explains GBM at Netflix October 10 2013Cliff Click Explains GBM at Netflix October 10 2013
Cliff Click Explains GBM at Netflix October 10 2013
 
Hadoop Streaming Tutorial With Python
Hadoop Streaming Tutorial With PythonHadoop Streaming Tutorial With Python
Hadoop Streaming Tutorial With Python
 

More from Peter Skomoroch

Bridging the AI Gap: Building Stakeholder Support
Bridging the AI Gap: Building Stakeholder SupportBridging the AI Gap: Building Stakeholder Support
Bridging the AI Gap: Building Stakeholder SupportPeter Skomoroch
 
Managing Machines: The New AI Dev Stack
Managing Machines: The New AI Dev StackManaging Machines: The New AI Dev Stack
Managing Machines: The New AI Dev StackPeter Skomoroch
 
Product Management for AI
Product Management for AIProduct Management for AI
Product Management for AIPeter Skomoroch
 
Executive Briefing: Why managing machines is harder than you think
Executive Briefing: Why managing machines is harder than you thinkExecutive Briefing: Why managing machines is harder than you think
Executive Briefing: Why managing machines is harder than you thinkPeter Skomoroch
 
Building Competitive Moats With Data
Building Competitive Moats With DataBuilding Competitive Moats With Data
Building Competitive Moats With DataPeter Skomoroch
 
O'Reilly Strata: Distilling Data Exhaust
O'Reilly Strata: Distilling Data ExhaustO'Reilly Strata: Distilling Data Exhaust
O'Reilly Strata: Distilling Data ExhaustPeter Skomoroch
 
SF Data Science: Developing Data Products
SF Data Science: Developing Data ProductsSF Data Science: Developing Data Products
SF Data Science: Developing Data ProductsPeter Skomoroch
 
Skills, Reputation, and Search
Skills, Reputation, and SearchSkills, Reputation, and Search
Skills, Reputation, and SearchPeter Skomoroch
 
LinkedIn Endorsements: Reputation, Virality, and Social Tagging
LinkedIn Endorsements: Reputation, Virality, and Social TaggingLinkedIn Endorsements: Reputation, Virality, and Social Tagging
LinkedIn Endorsements: Reputation, Virality, and Social TaggingPeter Skomoroch
 
Developing Data Products
Developing Data ProductsDeveloping Data Products
Developing Data ProductsPeter Skomoroch
 
Practical Problem Solving with Data - Onlab Data Conference, Tokyo
Practical Problem Solving with Data - Onlab Data Conference, TokyoPractical Problem Solving with Data - Onlab Data Conference, Tokyo
Practical Problem Solving with Data - Onlab Data Conference, TokyoPeter Skomoroch
 
Street Fighting Data Science
Street Fighting Data ScienceStreet Fighting Data Science
Street Fighting Data SciencePeter Skomoroch
 
Data Mashups -Data Science Summit
Data Mashups -Data Science SummitData Mashups -Data Science Summit
Data Mashups -Data Science SummitPeter Skomoroch
 
Rapid Data Exploration With Hadoop
Rapid Data Exploration With HadoopRapid Data Exploration With Hadoop
Rapid Data Exploration With HadoopPeter Skomoroch
 
Prototyping Data Intensive Apps: TrendingTopics.org
Prototyping Data Intensive Apps: TrendingTopics.orgPrototyping Data Intensive Apps: TrendingTopics.org
Prototyping Data Intensive Apps: TrendingTopics.orgPeter Skomoroch
 

More from Peter Skomoroch (16)

Bridging the AI Gap: Building Stakeholder Support
Bridging the AI Gap: Building Stakeholder SupportBridging the AI Gap: Building Stakeholder Support
Bridging the AI Gap: Building Stakeholder Support
 
Managing Machines: The New AI Dev Stack
Managing Machines: The New AI Dev StackManaging Machines: The New AI Dev Stack
Managing Machines: The New AI Dev Stack
 
Product Management for AI
Product Management for AIProduct Management for AI
Product Management for AI
 
Executive Briefing: Why managing machines is harder than you think
Executive Briefing: Why managing machines is harder than you thinkExecutive Briefing: Why managing machines is harder than you think
Executive Briefing: Why managing machines is harder than you think
 
Building Competitive Moats With Data
Building Competitive Moats With DataBuilding Competitive Moats With Data
Building Competitive Moats With Data
 
O'Reilly Strata: Distilling Data Exhaust
O'Reilly Strata: Distilling Data ExhaustO'Reilly Strata: Distilling Data Exhaust
O'Reilly Strata: Distilling Data Exhaust
 
SF Data Science: Developing Data Products
SF Data Science: Developing Data ProductsSF Data Science: Developing Data Products
SF Data Science: Developing Data Products
 
Skills, Reputation, and Search
Skills, Reputation, and SearchSkills, Reputation, and Search
Skills, Reputation, and Search
 
LinkedIn Endorsements: Reputation, Virality, and Social Tagging
LinkedIn Endorsements: Reputation, Virality, and Social TaggingLinkedIn Endorsements: Reputation, Virality, and Social Tagging
LinkedIn Endorsements: Reputation, Virality, and Social Tagging
 
Developing Data Products
Developing Data ProductsDeveloping Data Products
Developing Data Products
 
Practical Problem Solving with Data - Onlab Data Conference, Tokyo
Practical Problem Solving with Data - Onlab Data Conference, TokyoPractical Problem Solving with Data - Onlab Data Conference, Tokyo
Practical Problem Solving with Data - Onlab Data Conference, Tokyo
 
Street Fighting Data Science
Street Fighting Data ScienceStreet Fighting Data Science
Street Fighting Data Science
 
Data Mashups -Data Science Summit
Data Mashups -Data Science SummitData Mashups -Data Science Summit
Data Mashups -Data Science Summit
 
Rapid Data Exploration With Hadoop
Rapid Data Exploration With HadoopRapid Data Exploration With Hadoop
Rapid Data Exploration With Hadoop
 
Prototyping Data Intensive Apps: TrendingTopics.org
Prototyping Data Intensive Apps: TrendingTopics.orgPrototyping Data Intensive Apps: TrendingTopics.org
Prototyping Data Intensive Apps: TrendingTopics.org
 
Elasticwulf Pycon Talk
Elasticwulf Pycon TalkElasticwulf Pycon Talk
Elasticwulf Pycon Talk
 

Recently uploaded

Unleash Your Potential - Namagunga Girls Coding Club
Unleash Your Potential - Namagunga Girls Coding ClubUnleash Your Potential - Namagunga Girls Coding Club
Unleash Your Potential - Namagunga Girls Coding ClubKalema Edgar
 
Pigging Solutions Piggable Sweeping Elbows
Pigging Solutions Piggable Sweeping ElbowsPigging Solutions Piggable Sweeping Elbows
Pigging Solutions Piggable Sweeping ElbowsPigging Solutions
 
Build your next Gen AI Breakthrough - April 2024
Build your next Gen AI Breakthrough - April 2024Build your next Gen AI Breakthrough - April 2024
Build your next Gen AI Breakthrough - April 2024Neo4j
 
SIEMENS: RAPUNZEL – A Tale About Knowledge Graph
SIEMENS: RAPUNZEL – A Tale About Knowledge GraphSIEMENS: RAPUNZEL – A Tale About Knowledge Graph
SIEMENS: RAPUNZEL – A Tale About Knowledge GraphNeo4j
 
Unlocking the Potential of the Cloud for IBM Power Systems
Unlocking the Potential of the Cloud for IBM Power SystemsUnlocking the Potential of the Cloud for IBM Power Systems
Unlocking the Potential of the Cloud for IBM Power SystemsPrecisely
 
New from BookNet Canada for 2024: BNC BiblioShare - Tech Forum 2024
New from BookNet Canada for 2024: BNC BiblioShare - Tech Forum 2024New from BookNet Canada for 2024: BNC BiblioShare - Tech Forum 2024
New from BookNet Canada for 2024: BNC BiblioShare - Tech Forum 2024BookNet Canada
 
Breaking the Kubernetes Kill Chain: Host Path Mount
Breaking the Kubernetes Kill Chain: Host Path MountBreaking the Kubernetes Kill Chain: Host Path Mount
Breaking the Kubernetes Kill Chain: Host Path MountPuma Security, LLC
 
Making_way_through_DLL_hollowing_inspite_of_CFG_by_Debjeet Banerjee.pptx
Making_way_through_DLL_hollowing_inspite_of_CFG_by_Debjeet Banerjee.pptxMaking_way_through_DLL_hollowing_inspite_of_CFG_by_Debjeet Banerjee.pptx
Making_way_through_DLL_hollowing_inspite_of_CFG_by_Debjeet Banerjee.pptxnull - The Open Security Community
 
My Hashitalk Indonesia April 2024 Presentation
My Hashitalk Indonesia April 2024 PresentationMy Hashitalk Indonesia April 2024 Presentation
My Hashitalk Indonesia April 2024 PresentationRidwan Fadjar
 
Pigging Solutions in Pet Food Manufacturing
Pigging Solutions in Pet Food ManufacturingPigging Solutions in Pet Food Manufacturing
Pigging Solutions in Pet Food ManufacturingPigging Solutions
 
#StandardsGoals for 2024: What’s new for BISAC - Tech Forum 2024
#StandardsGoals for 2024: What’s new for BISAC - Tech Forum 2024#StandardsGoals for 2024: What’s new for BISAC - Tech Forum 2024
#StandardsGoals for 2024: What’s new for BISAC - Tech Forum 2024BookNet Canada
 
08448380779 Call Girls In Friends Colony Women Seeking Men
08448380779 Call Girls In Friends Colony Women Seeking Men08448380779 Call Girls In Friends Colony Women Seeking Men
08448380779 Call Girls In Friends Colony Women Seeking MenDelhi Call girls
 
How to convert PDF to text with Nanonets
How to convert PDF to text with NanonetsHow to convert PDF to text with Nanonets
How to convert PDF to text with Nanonetsnaman860154
 
Understanding the Laravel MVC Architecture
Understanding the Laravel MVC ArchitectureUnderstanding the Laravel MVC Architecture
Understanding the Laravel MVC ArchitecturePixlogix Infotech
 
CloudStudio User manual (basic edition):
CloudStudio User manual (basic edition):CloudStudio User manual (basic edition):
CloudStudio User manual (basic edition):comworks
 
"LLMs for Python Engineers: Advanced Data Analysis and Semantic Kernel",Oleks...
"LLMs for Python Engineers: Advanced Data Analysis and Semantic Kernel",Oleks..."LLMs for Python Engineers: Advanced Data Analysis and Semantic Kernel",Oleks...
"LLMs for Python Engineers: Advanced Data Analysis and Semantic Kernel",Oleks...Fwdays
 
FULL ENJOY 🔝 8264348440 🔝 Call Girls in Diplomatic Enclave | Delhi
FULL ENJOY 🔝 8264348440 🔝 Call Girls in Diplomatic Enclave | DelhiFULL ENJOY 🔝 8264348440 🔝 Call Girls in Diplomatic Enclave | Delhi
FULL ENJOY 🔝 8264348440 🔝 Call Girls in Diplomatic Enclave | Delhisoniya singh
 
Enhancing Worker Digital Experience: A Hands-on Workshop for Partners
Enhancing Worker Digital Experience: A Hands-on Workshop for PartnersEnhancing Worker Digital Experience: A Hands-on Workshop for Partners
Enhancing Worker Digital Experience: A Hands-on Workshop for PartnersThousandEyes
 

Recently uploaded (20)

Unleash Your Potential - Namagunga Girls Coding Club
Unleash Your Potential - Namagunga Girls Coding ClubUnleash Your Potential - Namagunga Girls Coding Club
Unleash Your Potential - Namagunga Girls Coding Club
 
Pigging Solutions Piggable Sweeping Elbows
Pigging Solutions Piggable Sweeping ElbowsPigging Solutions Piggable Sweeping Elbows
Pigging Solutions Piggable Sweeping Elbows
 
Build your next Gen AI Breakthrough - April 2024
Build your next Gen AI Breakthrough - April 2024Build your next Gen AI Breakthrough - April 2024
Build your next Gen AI Breakthrough - April 2024
 
SIEMENS: RAPUNZEL – A Tale About Knowledge Graph
SIEMENS: RAPUNZEL – A Tale About Knowledge GraphSIEMENS: RAPUNZEL – A Tale About Knowledge Graph
SIEMENS: RAPUNZEL – A Tale About Knowledge Graph
 
Unlocking the Potential of the Cloud for IBM Power Systems
Unlocking the Potential of the Cloud for IBM Power SystemsUnlocking the Potential of the Cloud for IBM Power Systems
Unlocking the Potential of the Cloud for IBM Power Systems
 
New from BookNet Canada for 2024: BNC BiblioShare - Tech Forum 2024
New from BookNet Canada for 2024: BNC BiblioShare - Tech Forum 2024New from BookNet Canada for 2024: BNC BiblioShare - Tech Forum 2024
New from BookNet Canada for 2024: BNC BiblioShare - Tech Forum 2024
 
Breaking the Kubernetes Kill Chain: Host Path Mount
Breaking the Kubernetes Kill Chain: Host Path MountBreaking the Kubernetes Kill Chain: Host Path Mount
Breaking the Kubernetes Kill Chain: Host Path Mount
 
Making_way_through_DLL_hollowing_inspite_of_CFG_by_Debjeet Banerjee.pptx
Making_way_through_DLL_hollowing_inspite_of_CFG_by_Debjeet Banerjee.pptxMaking_way_through_DLL_hollowing_inspite_of_CFG_by_Debjeet Banerjee.pptx
Making_way_through_DLL_hollowing_inspite_of_CFG_by_Debjeet Banerjee.pptx
 
E-Vehicle_Hacking_by_Parul Sharma_null_owasp.pptx
E-Vehicle_Hacking_by_Parul Sharma_null_owasp.pptxE-Vehicle_Hacking_by_Parul Sharma_null_owasp.pptx
E-Vehicle_Hacking_by_Parul Sharma_null_owasp.pptx
 
My Hashitalk Indonesia April 2024 Presentation
My Hashitalk Indonesia April 2024 PresentationMy Hashitalk Indonesia April 2024 Presentation
My Hashitalk Indonesia April 2024 Presentation
 
Pigging Solutions in Pet Food Manufacturing
Pigging Solutions in Pet Food ManufacturingPigging Solutions in Pet Food Manufacturing
Pigging Solutions in Pet Food Manufacturing
 
#StandardsGoals for 2024: What’s new for BISAC - Tech Forum 2024
#StandardsGoals for 2024: What’s new for BISAC - Tech Forum 2024#StandardsGoals for 2024: What’s new for BISAC - Tech Forum 2024
#StandardsGoals for 2024: What’s new for BISAC - Tech Forum 2024
 
08448380779 Call Girls In Friends Colony Women Seeking Men
08448380779 Call Girls In Friends Colony Women Seeking Men08448380779 Call Girls In Friends Colony Women Seeking Men
08448380779 Call Girls In Friends Colony Women Seeking Men
 
How to convert PDF to text with Nanonets
How to convert PDF to text with NanonetsHow to convert PDF to text with Nanonets
How to convert PDF to text with Nanonets
 
Understanding the Laravel MVC Architecture
Understanding the Laravel MVC ArchitectureUnderstanding the Laravel MVC Architecture
Understanding the Laravel MVC Architecture
 
The transition to renewables in India.pdf
The transition to renewables in India.pdfThe transition to renewables in India.pdf
The transition to renewables in India.pdf
 
CloudStudio User manual (basic edition):
CloudStudio User manual (basic edition):CloudStudio User manual (basic edition):
CloudStudio User manual (basic edition):
 
"LLMs for Python Engineers: Advanced Data Analysis and Semantic Kernel",Oleks...
"LLMs for Python Engineers: Advanced Data Analysis and Semantic Kernel",Oleks..."LLMs for Python Engineers: Advanced Data Analysis and Semantic Kernel",Oleks...
"LLMs for Python Engineers: Advanced Data Analysis and Semantic Kernel",Oleks...
 
FULL ENJOY 🔝 8264348440 🔝 Call Girls in Diplomatic Enclave | Delhi
FULL ENJOY 🔝 8264348440 🔝 Call Girls in Diplomatic Enclave | DelhiFULL ENJOY 🔝 8264348440 🔝 Call Girls in Diplomatic Enclave | Delhi
FULL ENJOY 🔝 8264348440 🔝 Call Girls in Diplomatic Enclave | Delhi
 
Enhancing Worker Digital Experience: A Hands-on Workshop for Partners
Enhancing Worker Digital Experience: A Hands-on Workshop for PartnersEnhancing Worker Digital Experience: A Hands-on Workshop for Partners
Enhancing Worker Digital Experience: A Hands-on Workshop for Partners
 

Geo Analytics Tutorial - Where 2.0 2011

  • 1. Geo Analytics Tutorial Pete Skomoroch Sr. Data Scientist - LinkedIn (@peteskomoroch) #geoanalytics ** Hadoop Intro slides from Kevin Weil, Twitter
  • 2. Topics ‣ Data Science & Geo Analytics ‣ Useful Geo tools and Datasets ‣ Hadoop, Pig, and Big Data ‣ Cleaning Location Data with Mechanical Turk ‣ Spatial Tweet Analytics with Hadoop & Python ‣ Using Social Data to Understand Cities ‣ Q&A
  • 3. Topics ‣ Data Science & Geo Analytics ‣ Useful Geo tools and Datasets ‣ Hadoop, Pig, and Big Data ‣ Cleaning Location Data with Mechanical Turk ‣ Spatial Tweet Analytics with Hadoop & Python ‣ Using Social Data to Understand Cities ‣ Q&A
  • 4. Analytics & Data are Hot Topics
  • 5. Analytics & Data are Hot Topics
  • 6. Analytics & Data are Hot Topics
  • 7. Analytics & Data are Hot Topics
  • 8. Analytics & Data are Hot Topics
  • 9. Analytics & Data are Hot Topics
  • 10. Data Exhaust My Delicious Tags
  • 11. Data Science * http://www.drewconway.com/zia/?p=2378
  • 12. Data Visualization ‣ http://www.dataspora.com/blog/
  • 13. Spatial Analysis Map by Dr. John Snow of London, showing clusters of cholera cases in the 1854 Broad Street cholera outbreak. This was one of the first uses of map-based spatial analysis.
  • 14. Spatial Analysis • Spatial regression - estimate dependencies between variables • Gravity models - estimate the flow of people, material, or information between locations • Spatial interpolation - estimate variables at unobserved locations based on other measured values • Simulation - use models and data to predict spatial phenomena
  • 15. Life Span & Food by Zip Code * http://zev.lacounty.gov/news/health/death-by-zip-code * http://www.verysmallarray.com/?p=975
  • 16. Where Americans Are Moving (IRS Data) ‣ (Jon Bruner) http://jebruner.com/2010/06/the-migration-map/
  • 17. Facebook Connectivity (Pete Warden) * http://petewarden.typepad.com/searchbrowser/2010/02/how-to-split-up-the-us.html
  • 18. Topics ‣ Data Science & Geo Analytics ‣ Useful Geo tools and Datasets ‣ Hadoop, Pig, and Big Data ‣ Cleaning Location Data with Mechanical Turk ‣ Spatial Tweet Analytics with Hadoop & Python ‣ Using Social Data to Understand Cities ‣ Q&A
  • 19. Useful Geo Tools •R, Matlab, SciPy, Commercial Geo Software •R Spatial Pkgs http://cran.r-project.org/web/views/Spatial.html •Hadoop, Amazon EC2, Mechanical Turk •Data Science Toolkit: http://www.datasciencetoolkit.org/ •80% of effort is often in cleaning and processing data
  • 20. DataScienceToolkit.org •Runs on VM or Amazon EC2 •Street Address to Coordinates •Coordinates to Political Areas •Geodict (text extraction) •IP Address to Coordinates •New UK release on Github
  • 21. Resources for location data • SimpleGeo • Factual • Geonames • Infochimps • Data.gov • DataWrangling.com
  • 22. Topics ‣ Data Science & Geo Analytics ‣ Useful Geo tools and Datasets ‣ Hadoop, Pig, and Big Data ‣ Cleaning Location Data with Mechanical Turk ‣ Spatial Tweet Analytics with Hadoop & Python ‣ Using Social Data to Understand Cities ‣ Q&A
  • 23. Hadoop: Motivation •We want to crunch 1TB of Twitter stream data and understand spatial patterns in Tweets •Data collected from the Twitter “Garden Hose” API last Spring
  • 24. Data is Getting Big ‣ NYSE: 1 TB/day ‣ Facebook: 20+ TB compressed/day ‣ CERN/LHC: 40 TB/day (15 PB/year!) ‣ And growth is accelerating ‣ Need multiple machines, horizontal scalability
  • 25. Hadoop ‣ Distributed file system (hard to store a PB) ‣ Fault-tolerant, handles replication, node failure, etc ‣ MapReduce-based parallel computation (even harder to process a PB) ‣ Generic key-value based computation interface allows for wide applicability ‣ Open source, top-level Apache project ‣ Scalable: Y! has a 4000-node cluster ‣ Powerful: sorted a TB of random integers in 62 seconds
  • 26. MapReduce? cat file | grep geo | sort | uniq -c > ‣ Challenge: how many tweets per output county, given tweets table? ‣ Input: key=row, value=tweet info ‣ Map: output key=county, value=1 ‣ Shuffle: sort by county ‣ Reduce: for each county, sum ‣ Output: county, tweet count ‣ With 2x machines, runs close to 2x faster.
  • 27. MapReduce? cat file | grep geo | sort | uniq -c > ‣ Challenge: how many tweets per output county, given tweets table? ‣ Input: key=row, value=tweet info ‣ Map: output key=county, value=1 ‣ Shuffle: sort by county ‣ Reduce: for each county, sum ‣ Output: county, tweet count ‣ With 2x machines, runs close to 2x faster.
  • 28. MapReduce? cat file | grep geo | sort | uniq -c > ‣ Challenge: how many tweets per output county, given tweets table? ‣ Input: key=row, value=tweet info ‣ Map: output key=county, value=1 ‣ Shuffle: sort by county ‣ Reduce: for each county, sum ‣ Output: county, tweet count ‣ With 2x machines, runs close to 2x faster.
  • 29. MapReduce? cat file | grep geo | sort | uniq -c > ‣ Challenge: how many tweets per output county, given tweets table? ‣ Input: key=row, value=tweet info ‣ Map: output key=county, value=1 ‣ Shuffle: sort by county ‣ Reduce: for each county, sum ‣ Output: county, tweet count ‣ With 2x machines, runs close to 2x faster.
  • 30. MapReduce? cat file | grep geo | sort | uniq -c > ‣ Challenge: how many tweets per output county, given tweets table? ‣ Input: key=row, value=tweet info ‣ Map: output key=county, value=1 ‣ Shuffle: sort by county ‣ Reduce: for each county, sum ‣ Output: county, tweet count ‣ With 2x machines, runs close to 2x faster.
  • 31. MapReduce? cat file | grep geo | sort | uniq -c > ‣ Challenge: how many tweets per output county, given tweets table? ‣ Input: key=row, value=tweet info ‣ Map: output key=county, value=1 ‣ Shuffle: sort by county ‣ Reduce: for each county, sum ‣ Output: county, tweet count ‣ With 2x machines, runs close to 2x faster.
  • 32. MapReduce? cat file | grep geo | sort | uniq -c > ‣ Challenge: how many tweets per output county, given tweets table? ‣ Input: key=row, value=tweet info ‣ Map: output key=county, value=1 ‣ Shuffle: sort by county ‣ Reduce: for each county, sum ‣ Output: county, tweet count ‣ With 2x machines, runs close to 2x faster.
  • 33. But... ‣ Analysis typically done in Java ‣ Single-input, two-stage data flow is rigid ‣ Projections, filters: custom code ‣ Joins: lengthy, error-prone ‣ n-stage jobs: Hard to manage ‣ Prototyping/exploration requires ‣ analytics in Eclipse? compilation ur doin it wrong...
  • 34. Enter Pig ‣ High level language ‣ Transformations on sets of records ‣ Process data one step at a time ‣ Easier than SQL?
  • 35. Why Pig? ‣ Because I bet you can read the following script.
  • 36. A Real Pig Script ‣ Now, just for fun... the same calculation in vanilla Hadoop MapReduce.
  • 38. Pig Simplifies Analysis ‣ The Pig version is: ‣ 5% of the code, 5% of the time ‣ Within 50% of the execution time. ‣ Pig Geo: ‣ Programmable: fuzzy matching, custom filtering ‣ Easily link multiple datasets, regardless of size/structure ‣ Iterative, quick
  • 39. A Real Example ‣ Fire up your Elastic MapReduce Cluster. ‣ ... or follow along at http://bit.ly/whereanalytics ‣ I used Twitter’s streaming API to store some tweets ‣ Simplest thing: group by location and count with Pig ‣ http://bit.ly/where20pig ‣ Here comes some code!
  • 40.
  • 41. tweets = LOAD 's3://where20demo/sample-tweets' as ( user_screen_name:chararray, tweet_id:chararray, ... user_friends_count:int, user_statuses_count:int, user_location:chararray, user_lang:chararray, user_time_zone:chararray, place_id:chararray, ...);
  • 42. tweets = LOAD 's3://where20demo/sample-tweets' as ( user_screen_name:chararray, tweet_id:chararray, ... user_friends_count:int, user_statuses_count:int, user_location:chararray, user_lang:chararray, user_time_zone:chararray, place_id:chararray, ...);
  • 43. tweets_with_location = FILTER tweets BY user_location != 'NULL';
  • 44. normalized_locations = FOREACH tweets_with_location GENERATE LOWER(user_location) as user_location;
  • 45. grouped_tweets = GROUP normalized_locations BY user_location PARALLEL 10;
  • 46. location_counts = FOREACH grouped_tweets GENERATE $0 as location, SIZE($1) as user_count;
  • 47. sorted_counts = ORDER location_counts BY user_count DESC;
  • 48. STORE sorted_counts INTO 'global_location_tweets';
  • 49. hadoop@ip-10-160-113-142:~$ hadoop dfs -cat /global_location_counts/part* | head -30 brasil 37985 indonesia 33777 brazil 22432 london 17294 usa 14564 são paulo 14238 new york 13420 tokyo 10967 singapore 10225 rio de janeiro 10135 los angeles 9934 california 9386 chicago 9155 uk 9095 jakarta 9086 germany 8741 canada 8201 7696 7121 jakarta, indonesia 6480 nyc 6456 new york, ny 6331
  • 50. Neat, but... ‣ Wow, that data is messy! ‣ brasil, brazil at #1 and #3 ‣ new york, nyc, and new york ny all in the top 30 ‣ Mechanical Turk to the rescue...
  • 51. Topics ‣ Data Science & Geo Analytics ‣ Useful Geo tools and Datasets ‣ Hadoop, Pig, and Big Data ‣ Cleaning Location Data with Mechanical Turk ‣ Spatial Tweet Analytics with Hadoop & Python ‣ Using Social Data to Understand Cities ‣ Q&A
  • 52. Code examples we’ll cover are on Github
  • 53. You can run them on Elastic MapReduce
  • 54. Cleaning Twitter Profile Location Names Filter Exact Matches Extract Top Tweet Locations Clean with MTurk Aggregate Context with Hadoop
  • 55. We will map locations to GeoNames IDs
  • 56. Start with Location Exact Matches
  • 57. Use Mechanical Turk to improve results
  • 58. Workers do simple tasks for a few cents
  • 59. We constructed the following task
  • 60. Workers used a Geonames search tool
  • 61. Location search tool code is on Github
  • 62. Preparing Data to send to MTurk
  • 63. We use consensus answers from workers
  • 65. Topics ‣ Data Science & Geo Analytics ‣ Useful Geo tools and Datasets ‣ Hadoop, Pig, and Big Data ‣ Cleaning Location Data with Mechanical Turk ‣ Spatial Tweet Analytics with Hadoop & Python ‣ Using Social Data to Understand Cities ‣ Q&A
  • 66. Tokenizing and Cleaning Tweet Text ‣ Extract Tweet topics with Hadoop + Python + NLTK + Wikipedia
  • 67. Build Phrase Dictionary with Wikipedia
  • 68. Streaming Tweet Parser (Python + NLTK)
  • 69. Parse Tweets and Join to Wikipedia (Pig)
  • 70. Aggregate by US County for Analysis
  • 71. Clean Data => Thematic US County Map
  • 72. Twitter users by county in our sample
  • 77. Topics ‣ Data Science & Geo Analytics ‣ Useful Geo tools and Datasets ‣ Hadoop, Pig, and Big Data ‣ Cleaning Location Data with Mechanical Turk ‣ Spatial Tweet Analytics with Hadoop & Python ‣ Using Social Data to Understand Cities ‣ Q&A
  • 79. Skills in the Design Industry
  • 80. Exploring the Spatial Distribution of Skills
  • 81. People with “Ship Building” Skills
  • 82. What is the Skill profile of a given city?
  • 83. Expertise correlated with Santa Clara, CA
  • 85. Expertise correlated with Washington, DC
  • 86. Yuba City, CA has 21.3% Unemployment 21.3
  • 87. Ames, Iowa has 4.7% Unemployment 21.3
  • 88. Topics ‣ Data Science & Geo Analytics ‣ Useful Geo tools and Datasets ‣ Hadoop, Pig, and Big Data ‣ Cleaning Location Data with Mechanical Turk ‣ Spatial Tweet Analytics with Hadoop & Python ‣ Using Social Data to Understand Cities ‣ Q&A
  • 89. Questions? Follow me at twitter.com/peteskomoroch datawrangling.com

Editor's Notes

  1. \n
  2. \n
  3. \n
  4. \n
  5. \n
  6. \n
  7. \n
  8. \n
  9. \n
  10. \n
  11. \n
  12. \n
  13. \n
  14. \n
  15. \n
  16. \n
  17. \n
  18. \n
  19. \n
  20. \n
  21. \n
  22. \n
  23. \n
  24. \n
  25. \n
  26. \n
  27. \n
  28. \n
  29. \n
  30. \n
  31. \n
  32. \n
  33. \n
  34. \n
  35. \n
  36. \n
  37. \n
  38. \n
  39. \n
  40. \n
  41. \n
  42. \n
  43. \n
  44. \n
  45. \n
  46. \n
  47. \n
  48. \n
  49. \n
  50. \n
  51. \n
  52. \n
  53. \n
  54. \n
  55. \n
  56. \n
  57. \n
  58. \n
  59. \n
  60. \n
  61. \n
  62. \n
  63. \n
  64. \n
  65. \n
  66. \n
  67. \n
  68. \n
  69. \n
  70. \n
  71. \n
  72. \n
  73. \n
  74. \n
  75. \n
  76. \n
  77. \n
  78. \n
  79. \n
  80. \n
  81. \n
  82. \n
  83. \n
  84. \n
  85. \n
  86. \n
  87. \n
  88. \n