Klout changing landscape of social media

How Klout is changing thelandscape of social media withHadoop and BIDave MarianiVP Engineering, KloutDenny LeePrincipal Program ManagerMicrosoft

Klout uses Big Data to unify the social web

Klout’s Big Data makes all this possible 15 Social Networks Processed Every Day 120 Terabytes of Data Storage 200,000 Indexed Users Added Every Day 140,000,000 Users Indexed Every Day 1,000,000,000 Social Signals Processed Every Day 30,000,000,000 API Calls Delivered Every Month 54,000,000,000 Rows of Data In Klout Data Warehouse 3

Scenario and Definitions Project: Event: Category: Property:Collection Captured Attribute Eventof Events User Action Type Attribute +K (Add a topic) event Topic, {Big Data, BI} Gender, {Male} Location {Palo Alto}

Klout Event Tracker 1 Perform A|B Testing of User Flows 2 Optimize User Registration Funnels3 Monitor consumer engagement & retention (DAUs & MAUs)4 Flexibly track and report on user generated events 5

Klout Event Tracker Requirements 3rd Party Hadoop BI Web Requirement & Query Analytics Hive Engines ToolsCapture & store all user and visitor events No Yes NoIntegrate internal Klout Data No Yes NoSupport queries against granular data No Yes NoSupport interactive queries Yes No YesSupport 3rd party BI tools No No Yes“Query-able” by custom apps No No YesTODO: Make this look good and use animation to “blend” the last 2 columns 6

Klout Data Architecture – The Best Tool for the Job Serve Signal Collectors Registrations DB (Java/ (MySql) Data Scala) Enhancement Engine Klout.com (PIG/Hive) Profile DB (Node.js) Klout API (HBase) (Scala) Data Warehouse (Hive) Search Index Mobile (Elastic Search) In (ObjectiveC) Store & Enhance Streams (MongoDB) Monitoring (Nagios) Dashboards (Tableau) Analytics Analyze Cube Perks Analyics (SSAS) (Scala) Event Tracker (Scala)TO DO: Need to animate the red boxes & make this look better +add Instrument, Collect, Persist, Query, Report information 7

TO DO: make this look better +add Instrument, Collect, Persist, Query, Report informationIf possible, merge slides 7 and 8 together 8

A Peek into Product Insights >A|B Test Example for Viral Workflow 9

A Peek into Product Insights >Projects: Mobile iOS 12

Projects > Mobile iOS > Scala/JavaScript API DB.withConnection("cube")(implicit conn => { var sql1 = SQL(""" select [[Date]].[Date]].[Date]].[MEMBER_CAPTION]]] AS date, ... convert(int, [[Measures]].[Counter]]]) AS cnt from openquery(productinsight, ’ SELECT {[Measures].[Counter]} ON COLUMNS, NON EMPTY CROSSJOIN ( exists([Date].[Date].[Date].allmembers, {[Date].[Date].&[""" + dateFormat(past) + """]:... ) DIMENSION PROPERTIES MEMBER_CAPTION ON ROWS FROM [ProductInsight] ) """) sql1().iterator.foreach(row => { // process row val event = row[String]("event") // .... })

Projects > Mobile iOS > Actual MDX SELECT { [Measures].[Counter], [Measures].[PreviousPeriodCounter] } ON COLUMNS, NON EMPTY CROSSJOIN ( exists([Date].[Date].[Date].allmembers, [Date].[Date].&[2012-05-19T00:00:00]:[Date]. [Date].&[2012-06-02T00:00:00]), [Events].[Event].[Event].allmembers ) DIMENSION PROPERTIES MEMBER_CAPTION ON ROWS FROM [ProductInsight] WHERE ({[Projects].[Project].[mobile-ios]})

Drilling down to the Events >Query Hive using Excel 15

Drilling down to the Events > HiveQL QueryCREATE TABLE mobile-ios-details-20120530 asSELECT get_json_object(json_text,$.sid) as sid, get_json_object(json_text,$.inc) as inc, get_json_object(json_text,$.status) as status, event json_textFROM bi.event_logWHERE project="mobile-ios" AND dt=20120530 AND get_json_object(json_text,$.v)!=1.5 AND (event = api_error OR event = api_timeout)DISTRIBUTE BY get_json_object(json_text,$.sid)SORT BY get_json_object(json_text,$.sid) asc

Adhoc Analysis >Answering Questions on the Fly 17

Summary•  Leverage the best tool for the function or job•  Big Data != Business Intelligence•  Go open source wherever possible but use commercial software when needed 18

Any Questions? What’s next 19

http://marilson.pbworks.com	48
http://eventifier.co	15
http://tweetedtimes.com	4

Klout changing landscape of social media

by Hadoop_Summit

Accessibility

Categories

Upload Details

Usage Rights

3 Embeds 67

Statistics