The document discusses Socialmetrix's evolution of their real-time social media analytics architecture over 4 iterations to meet growing customer and data demands. It describes how they moved from a monolithic to distributed setup using technologies like AWS, Spark, Kafka and Cassandra to improve scalability, costs and resilience while adding new data sources and features. Key lessons included automating deployments, monitoring systems, and using AWS services like S3, EMR and DynamoDB to enable rapid prototyping and reprocessing as needed to support real-time and batch analytics.

1. ARC202
Real-World Real-Time Analytics
Gustavo Arjones | @arjones
CTO, Socialmetrix
November 13, 2014 | Las Vegas, NV
Sebastian Montini | @sebamontini
Solutions Architect, Socialmetrix
© 2014 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified, or distributed in whole or in part without the express consent of Amazon.com, Inc.

2. • SaaS Company—since 2008
• Social media analytics track and measure activity
of brands and personality, providing information to
market research and brand comparison
• Multilanguage technology (English, Portuguese,
and Spanish)
• Leader in Latin America, with operations in 5
countries, customers in Latin America and US
• 1 out of 34 Twitter Certified Program worldwide

3. Our customers

6. Share of topics
Ranking Brand 1 Brand 2 Brand 3
Q2 Q3 Q2 Q3 Q2 Q3
1° Flavor Breakfast Flavor Flavor Advertising Flavor
2° Healthy Flavor Packaging Brand I love Flavor Breakfast
3° Components Components Healthy Packaging Healthy Healthy
4° Advertising Healthy Components Addiction Components Advertising
5° Enquires Desire Prices Consumption Prices Components
TOTAL 1.401 8.189 463 5.519 1.081 2.445
Which conversations are my brand and my competitors’ brands driving?

7. smx.io/reinvent #reinvent

8. Challenges

9. Challenges: Variety
• Different data sources
• Different API
• SLA
• Method (pull or push)
• Rate-limit, backoff strategy

10. Challenges: Velocity
• Updates every second
• Top users, top hashtags each
minute
Last TV
Debate
• After event analysis are made
with batch over complete
dataset
• Spikes of 20,000+ tweets per
minute
Results
Announced

11. Challenges: Meaning
•Disambiguation
•Data Enrichment
– Demographics
– Sentiment
– Influencers
•Human analysis
Oi Telecom Hi!
PAN
Orange Telecom

12. Challenges: Alert and report
• Clear and
understandable UI
• Slice-dice for business
(not BI experts)
• Real-time alerts for
anomalies

13. Architecture evolution

14. Drivers for architecture evolution
• More customers, bigger customers
• Add new features
• Keep costs under control

15. Architecture evolution
120
100
80
60
40
20
0
#1 #2 #3 #4
Active Customers

16. Architecture—1st iteration
What we needed:
• Complete data isolation
• Trying different solutions/offerings

17. Architecture—1st iteration
What we did:
• All-in-one approach
• Multi-instance architecture
• Simple vertical scalability
• MySQL performance tuning

18. Architecture—1st iteration
What we've learned:
• Multi-instance is harder to administrate, but
minimizes instability impact on customers
• Vertical scalability: poor resource management
• MySQL schema changes translate into downtime

19. Architecture—2nd iteration
What we needed:
• Separation of responsibilities (crawling, processing)
• Horizontal scalability
• Fast provisioning
• Cost reduction

20. Architecture—2nd iteration
What we changed:
• Migrated to AWS
• RabbitMQ (Single Node)
• Replace MySQL for
Amazon RDS
• AWS CloudFormation
• Auto Scaling groups

21. Architecture—2nd iteration
What we've learned:
• PIOPS 
• Tuning the Auto Scaling policies can be hard
• AWS CloudFormation: great for migration, not
enough for daily ops

22. Architecture—3rd iteration
What we needed:
• Deliver new features (NRT, more complex analytics)
• Scale fast
• Be resilient against failure
• Adding and improving data sources
• Keep costs under control (always)

23. Architecture—3rd iteration
What we changed:
• Apache Storm
• RabbitMQ HA
• Amazon Elastic MapReduce
(Hadoop/Hive)
• AWS CloudFormation + Chef
• Amazon Glacier + Amazon S3
lifecycles policies

24. Architecture—3rd iteration
What we've learned:
• Spot Instances + Reserved Instances
• Hive = SQL  SQL scripts are hard to test
• Bulk upserts on Amazon RDS can be expensive
(PIOPS)
• Amazon DynamoDB is great, but expensive (for
our use-case)

25. Dashboard

26. Architecture—4th iteration
What we needed:
• Monitor millions of social media profiles
• Make data accessible (exploration, PoC)
• Improve UI response times
• Testing our data pipelines
• Reprocessing (faster)

27. Architecture—4th iteration
What we changed:
• Cassandra (DSE)
• MongoDB MMS
• Apache Spark

28. Architecture—4th iteration
What we've learned:
• Leverage AWS ecosystem
• Datastax AMI + Opscenter integration
• MongoDB MMS: automation magic!
• Apache Spark unit testing + Amazon EC2
launch scripts
• Amazon EMR doesn’t have the latest stable
versions

30. Architecture evolution
160
140
120
100
80
60
40
20
-
120
100
80
60
40
20
0
#1 #2 #3 #4
Active Customers
Costs Customers

31. Lessons learned

32. Lessons learned
• Automate since Day 1 (CloudFormation + Chef)
• Monitor systems activity, understand your data
patterns, e.g. LogStash (ELK)
• Always have a Source of Truth (Amazon S3 +
Glacier)
• Make your Source of Truth searchable

33. Lessons Learned (II)
•Approximation is a good thing: HLL, CMS, Bloom
• Write your pipelines considering reprocessing
needs
• Avoid at all costs framework explosion
•AWS ecosystem allows rapid prototype

34. Socialmetrix NextGen
2015

35. Architecture evolution
120
100
80
60
40
20
0
#1 #2 #3 #4
Active Customers

36. Architecture nextgen
• Reduce moving parts
• Apache Spark as central processing framework
– Realtime (Micro-batch)
– Batch-processing
• Kafka (Message Broker)
• Cassandra (Time-series storage)
• ElasticSearch (Content Indexer)

37. To infinity …
and beyond! Architecture evolution
120
100
80
60
40
20
0
#1 #2 #3 #4 NextGen
Active Customers

38. Feedback and QandA
Gustavo Arjones, CTO
@arjones | gustavo@socialmetrix.com
Sebastian Montini, Solutions Architect
@sebamontini | sebastian@socialmetrix.com
Let’s talk at Venetian—Titian Hallway

39. Please give us your feedback on this
presentation
ARC202: Real-World
Real-Time Analytics
Thank you!
Join the conversation on Twitter with #reinvent
© 2014 Amazon.com, Inc. and its affiliates. All rights reserved. May not be copied, modified, or distributed in whole or in part without the express consent of Amazon.com, Inc.