Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

AWS Summit Berlin 2013 - Tadaa - HD Camera and Photo Community


Published on

tadaa is the camera app replacing your DSLR, coupled with a massive community of pro and amateur photographers. Let's talk about cost-efficient living, scaling and growing in the cloud.
Speaker: Friedemann Wachsmuth, CTO, menschmaschine Publishing GmbH

Published in: Technology
  • Be the first to comment

AWS Summit Berlin 2013 - Tadaa - HD Camera and Photo Community

  1. 1. Scaling tadaaPhoto Broadcastingin the Cloud
  2. 2. What is tadaa?•HD Camera & Photo Editor for iOS•Photosharing Community for Prosumers•3 Million App Users
  3. 3. tadaa is a RealtimeApplication...•Constant Relationship Changes•Reactions, Interactions & Notifications•Fan-outs, Deletions•Sync Notification Counts
  4. 4. ...with a Lot of DataUnderneath•250M Contact Hashes•Hundreds of Millions of Images on S3•~2500 Messages / Second
  5. 5. Architecture
  6. 6. Architecture•100% Hosted on AWS•Mostly Using AWS Services: EC2, EMR,CloudFront, S3, DynamoDB,ElastiCache, CloudWatch, IAM...
  7. 7. Apple PushNotifications Sent
  8. 8. Load can be Bursty•If a Single User has 10k Followers➡ One Photo can cause 10k Push Notifs!•With 25% Open Rate, they create➡ 2500 API calls within few Seconds•leading to Thousands of Likes andReactions... Within a few Seconds.
  9. 9. When InstagramChanged T&Cs...ALL YOUR BASEARE BELONG TOUS
  10. 10. Using CouchDB...Pros:•Simple REST API•Replicates•Schemaless•Map/Reduce•EverythingversionedCons:•Sharding is Hard•JSON = DataInflation•Very Disk-I/ODependent•Everythingversioned
  11. 11. Juggling 8TBDatabase Files onEBS isn‘t Fun.
  12. 12. Hello,DynamoDB!
  13. 13. Moving to DynamoDB•Predictable Performance•Infinite Table Size•Full Redundancy•8TB Became 150GB!
  14. 14. DynamoDB Scales•We Migrated the Live System with noDowntime within a Few Days•Query Latency is Really just 2-3ms•Worst Value ever seen was 8ms•Now You get Some Free BurstAllowance!
  15. 15. Cost Scales Up, too...„How Can weHandle BurstsBetter?“
  16. 16. Optimizing AccessPatterns•Level Your Reads and Writes•Vary Hashkeys•Throttle Non-Realtime Tasks•Mirror in ElastiCache Where Possible,Persist Lazily➡Reduce Required CUs by 75%
  17. 17. Leveling Reads/Writes•Avoid Bursts and Scans•Queue Your I/O through a MessagingSystem•If You Can, Separate Hot and Cold Data
  18. 18. Vary Your Hashkeys•Your Table is Partitioned•# of Partitions Grows with Size of Tableand Provisioned CUs•You don‘t know the # of ActualPartitions•You only get your Provisioned CUs whenusing all Partitions Equally!
  19. 19. Throttle Non-Realtime Tasks•Purge old Data through a ThrottledMessage Queue•Spool Data with expected Redundanciesand drain only the uniqued Changes(e.g. Facebook LiveAPI)•Scale your Worker Throughput based onCloudWatch Data
  20. 20. Use ElastiCache forhot Data•Write hot Data to ElastiCache and yourDAL•Alter Hot Data in the Cache (e.g. Like-Counts)•Benefit e.g. from Increments•Read latest Data from Cache once youpersist to DynamoDB
  21. 21. It is all well worth it.
  22. 22. Thank You!
  23. 23.