How Klout is changing the landscape of social media with Hadoop and BI

How Klout is changing thelandscape of social media withHadoop and BIDave MarianiVP Engineering, KloutDenny LeePrincipal Program ManagerMicrosoft

Discover and be recognized for how you influence the world

Klout’s Big Data makes all this possible 15 Social Networks Processed Every Day 120 Terabytes of Data Storage 200,000 Indexed Users Added Every Day 140,000,000 Users Indexed Every Day 1,000,000,000 Social Signals Processed Every Day 30,000,000,000 API Calls Delivered Every Month 54,000,000,000 Rows of Data In Klout Data Warehouse 3

KLOUT DATA ARCHITECTURE THE BEST TOOL FOR THE JOB Registrations DB Klout.com (MySql) (Node.js) Mobile Profile DB (ObjectiveC) Klout API (Scala) (HBase) Signal Collectors Data Partner API(Java/Scala) Enhancement Engine (Mashery) Data Warehouse (PIG/Hive) Search Index (Hive) (Elastic Search) Streams (MongoDB) Monitoring (Nagios) Serving Stores Dashboards (Tableau) Perks Analyics (Scala) Analytics Cubes Event Tracker (SSAS) (Scala)

What is Business Intelligence?• Data Warehousing, OLAP, Dashboards, Reporting• Ability to slice and dice data in an ad-hoc manner• Getting the right data to the right people, at the right time• i.e. Now 5

Why Hadoop + BI? Hadoop BI Requirement & Query Hive Engines Capture & store all data Yes No Support queries against detail data Yes No Support interactive queries & No Yes applications Support BI & visualization tools No Yes 6

An Example: Klout Event Tracker 1 Perform A|B Testing of User Flows 2 Optimize Registration Funnels3 Monitor consumer engagement & retention (DAUs & MAUs)4 Flexibly track and report on user generated events 7

A Flexible, Hierarchical Schema Project: Event: Property Type: Property Value:Collection Captured Attribute Attributeof Events User Action Key ValueHomePage, Source, Google Search Actions, Gender, MaleMobile iOS Location SF +K (Add a topic) event

Event Tracker Architecture event_log tstamp string { project string "project":"plusK", string event session_id bigint "event":"spend", insights3:9003/track/{"project":”plu ks_uid bigint sK","event":”spend”,"session_id":"0", Warehouse ip string "ip":"50.68.47.158", "ks_uid":123456,”type":”add_topic"} json_keys array<string> "kloutId":“123456", json_values “cookie_id":”123456", array<string> "ref":"http://klout.com/", json_text string "type":"add_topic",Tracker API Log Process Cube dt string Klout UI "time":"1338366015" Scala, Flume Analysis Scala, } hr string node.JS Services AJAX UX SELECT { [Measures].[Counter], [Measures].[PreviousPeriodCounter]} ON COLUMNS, will be saved in HDFS at: NON EMPTY CROSSJOIN ( /logs/events_tracking/2012-05-30/0100 exists([Date].[Date].[Date].allmembers, [Date].[Date].&[2012-05-19T00:00:00]:[Date].[Date].&[2012-06- 02T00:00:00]), [Events].[Event].[Event].allmembers ) DIMENSION PROPERTIES MEMBER_CAPTION ON ROWS FROM [ProductInsight] WHERE ({[Projects].[Project].[plusK]})Instrument Collect Persist Query Report 9

Hadoop & BI Together:Query Cube using a Custom App 10

A peek into product insight >A|B test : unsorted vs. Sorted 11

A Peek intoProduct Insights >Projects: MobileiOS 12

Hadoop & BI Together:Query Cube Using Viz App 14

Hadoop & BI Together:Query Hive using CLI 17

HiveQL ExampleSELECT get_json_object(json_text,$.sid) as sid, get_json_object(json_text,$.inc) as inc, get_json_object(json_text,$.status) as status, eventFROM bi.event_logWHERE project=mobile-ios AND dt=20120612 AND get_json_object(json_text,$.v)<>1.5 AND (event = api_error OR event = api_timeout)ORDER BY sid;

Hadoop & BI Together:Query Hive using Excel 20

Why Hadoop + BI? Hadoop BI Requirement & Query Hive Engines Capture & store all data Yes No Support queries against detail data Yes No Support interactive queries & No Yes applications Support BI & visualization tools No Yes 22

Any Questions? 23

http://nosql.mypopescu.com	3513
http://dennyglee.com	1414
http://plurrimi.wordpress.com	696
http://insight.velaia.de	225
http://bigdata.velaia.de	199
http://posidev.com	158
http://feeds.feedburner.com	30
http://www.hanrss.com	10
http://www.linkedin.com	7
http://core-qa.traackr.com	5
http://cafe.naver.com	4
http://xianguo.com	3
http://www.newsblur.com	3
http://tumblr.hootsuite.com	3
http://socialmexican.com	3
http://core.traackr.com	2
http://pinterest.com	2
http://translate.googleusercontent.com	1
https://twitter.com	1
http://webcache.googleusercontent.com	1
http://wpg1proxy1	1
http://dev.newsblur.com	1
http://www.acushare.com	1
https://twimg0-a.akamaihd.net	1
http://127.0.0.1	1
http://174.122.254.42	1

How Klout is changing the landscape of social media with Hadoop and BI

by Denny Lee on Jul 04, 2012

Accessibility

Categories

Upload Details

Usage Rights

26 Embeds 6,286

Statistics

How Klout is changing the landscape of social media with Hadoop and BI Presentation Transcript