Rapid Data Exploration With Hadoop

Rapid Data Exploration With Hadoop Peter Skomoroch Senior Data Scientist @peteskomoroch

Outline • Overview: LinkedIn Biz, Tech, & Analytics • Rapid Data Exploration 101 - Spatial Analytics Pig Code - Trend detection with Pig & Python - R Streaming Example • Deep Dive: Our Data Analysis Approach • Building Data Products • LinkedIn Data Insights

Connect the world’s professionals to make them more productive and successful

Professional Identity

LinkedIn at a glance • Founded in 2003 • #17 site in the US (Alexa) • 60+ million members • First million members = 477 days • Latest million = 9 days • 500K+ company proﬁles • 12+ million small business professionals • In 2009 - 1billion people searches • Average age: 41 • Household income $107,000 • 42% are “decision makers”

How International? • More than 50% international (members in over 200 countries & territories) • 13+ million in Europe • 4+ million in India • 3+ million in UK • #13 site in UK (Alexa)

How do we keep the lights on? • Proﬁtable since 2007 • Valued at over $1B at the last funding round • Subscriptions • Ads • Job Postings • Enterprise Client

Hadoop on LinkedIn 1,400+ members list “Hadoop” on their proﬁle What other skills do they have? •HBase, Lucene, Solr, MapReduce, Nutch... Where are they? Who do they work for? • 36% in Bay Area • 11% Yahoo! • 8% in India • 2% Apache Software Foundation • 6% in NYC • 1% LinkedIn • 4% in Seattle • 1% Google • 4% in Los Angeles • 1% Facebook

Hadoop at LinkedIn

Voldemort Data Storage Compact, compressed, binary data (something like Avro) Type can be any combination of int, double, float, String, Map, List, etc. => Sequence Files Example member definition: { ‘member_id’: ‘int32’, ‘first_name': 'string', ’last_name': ’string’, ‘age’ : ‘int32’ … }

Getting Data In •From Databases (user data, news, jobs etc.) • Need a way to get data reliably periodically • Need tests to verify data • Support for incremental replication • Solution: Transmogrify Driver Program • InputReader: JDBCReader, CSV Reader • Output Writer: JDBCWriter, HDFS writers • From web logs (page views, search, clicks etc) • Weblogs ﬁles are rsynced and loaded up in HDFS • Hadoop jobs for date cleaning and transformation.

Getting Data Out

Giving Back: Open Source http://sna-projects.com/sna/

Analytics Technologies

We Build Things With Data Give smart people great tools, enable them to solve problems

Prototyping Culture

How does Hadoop enable rapid data exploration?

Pig for Spatial Analytics

US County HeatMap

Pig for Trend Detection

Python Streaming Script

Sort Output & Display

R Streaming Also Easy *from http://www.stat.uiowa.edu/~luke/classes/295-hpc/

Let’s Talk Data

Business is recognizing the importance of analytics

What data do we start with?

We can also leverage... • Connection Graph • Company Pages • Recommendations • Talent Match • Address Book Uploads • Web Referrals • Search Logs • 1M+ Twitter Accounts • Proﬁle Views & Activity • Wikipedia Data • Job Postings • Mechanical Turk • LinkedIn Groups • Census, BLS, & Data.gov • LinkedIn Questions • Much more...

How do we think of Analytics? Data Jujitsu

Lots of Medium can be more powerful than Big >

Reconstruct Reality from Data Exhaust

Data Scientist Lessons • Follow the data, avoid assumptions • Sanity check the extremes (0, inﬁnity) • Don’t get mired in rare edge cases • Data Jujitsu: solve easier auxiliary problems • Build smaller consistent samples to test code • Establish a baseline model quickly, iterate often • Use the right tool for the job at hand • Iterate quickly with high level languages

Where did the bankers go?

We’re Hiring! http://sna-projects.com/sna/ pskomoro@linkedin.com @peteskomoroch

Rapid Data Exploration With Hadoop

by Peter Skomoroch on Mar 31, 2010

Accessibility

Categories

Tags

Upload Details

Usage Rights

2 Embeds 16

Statistics

Rapid Data Exploration With Hadoop — Presentation Transcript

http://www.slideshare.net	15
http://www.lmodules.com	1