Cassandra eu

Powering Social Business IntelligenceCassandra and Hadoop at the Dachis Group

Social Business Wha?Big Data meets Big Budgets• Brand marketers spend • $450 (~£270) billion annually on tradition media • $50 (~£30) billion annually on SEO/SEM• Starting to transition to social media

Effectiveness - TraditionalMeasure all the things! Measuring traditional marketing effectiveness

Effectiveness - SocialMeasure all the things! Measuring social marketing effectiveness

The Dachis GroupMeasure all the things!• Jeff Dachis amasses small army of social strategists• Funds team to create social analytics platform • Measure business outcomes of social media strategies • Track social media surrounding Forbes Global 2000 • Include all brands, all subsidiaries, all social media types

Architecture• Raw data in S3• Cassandra • Realtime queries to return raw data • Hadoop analytic integration for foundational measures • Horizontal scalability • Operationally simple• RDBMS • Time rollups of measures • Aggregates and composite measures • Arbitrary dimensional queries • Mini data warehouse

Pipeline Memcached AWS S3 Cassandra Postgres Raw Signal Signal Metrics Storage Repository Store Normalization Enrichment Analysis

Normalization• Parallel copy from S3 to HDFS• MapReduce to Cassandra from Raw to Normalized CF• Normalized data model • Decent investment to get right • Mostly for conceptual reasons rather than concerns about queries • Secondary indexes vs app maintained indexes

Enrichment• Enrich with • Unique company/brand information • Sentiment • Relationships • Conversations • Social graph information• Enter Pig• Enter Oozie

The Bleeding EdgePig • newlogicalplan in 0.8.0 • Debugging/tracing? • Incremental development • Working with Cassandra • Pygmalion - facilitating to and from Cassandra • Experience, unit test framework, UDFs, community slowly became

The Bleeding EdgeOozie• Learning curve and common errors • User impersonation • Logs, we haz them, lots of them • Web UI needs love• Specific to Cassandra • mapreduce.fileoutputcommitter.marksuccessfuljobs • See http://wiki.apache.org/cassandra/HadoopSupport#Oozie• Still very good DAG workflow crunching tool • Subworkflows, fork/join, regular scheduling, dataset detection • Extensible • Apache Incubator (@oozie on twitter, #oozie on freenode)

The Bleeding EdgeCassandra • Rack aware snitch and replication • Always rotate racks in order in topology • In EC2 this likely means rotate AZs • Dealing with scanning over column families • Project early • General tuning and unique workload • Mahout and other higher memory hadoop tasks • EC2 instance types • Visualization tool helped (OpsCenter, Acunu has Control Center) • Community++

Social Business IndexLaunches September 2011 • Global Ranking of Companies • Industry Rankings • Visualization of strategy

This might actually work! • Fall 2011, built up the team • Expertise in Pig, Lucene/Solr, machine learning, statistics, event prediction and analysis • Making everlasting gobstoppers

Social Performance MonitorThe measures behind the score

Topics topics topics• Black Friday • Science project! • Mallet, Pig • Custom analysis• Superbowl• Oscars

Productizing Topics• Ongoing automated topic detection• Lessons from one-off topic analysis• Represented by term distributions• Threads with detail like • Signal volume • Participants • Links • Sentiment gauge

Advocates• Auto-discovery of potential advocates• Curated set of known advocates• Example signal (from Cassandra)• Reports and other useful bits

Lessons learned• Emerging products are sometimes frustrating, but well worth the pain in their respective niche.• “Never underestimate the massive impact of small bugs in big data.” (@peteskomoroch at LinkedIn)• Community karma

A Note on Community• Community involvement • IRC, mailing lists, twitter, conferences, meetups • Newer projects have little or outdated docs • Some features may be • Deprecated • Not ready for primetime • Not a ﬁt for your use case• Community karma • Don’t just take • Be a bridge builder • Positive karma helps

Questions?• We’re hiring• Ping me @jeromatron (Twitter and IRC)

http://tweetedtimes.com	7
http://waffle-prod.herokuapp.com	2
http://us-w1.rockmelt.com	1
http://www.twylah.com	1

Cassandra eu

by Jeremy Hanna on Mar 28, 2012

Accessibility

Categories

Upload Details

Usage Rights

4 Embeds 11

Statistics

Cassandra eu Presentation Transcript