Kash flow Presentation from kbbs
Your SlideShare is downloading.
×
Check these out next
Recommended
More Related Content
Slideshows for you (20)
Viewers also liked (7)
Similar to Scaling Runa Inc Big Data e-commerce service with AWS (20)
More from Robert Berger (9)
Recently uploaded (20)
Scaling Runa Inc Big Data e-commerce service with AWS
- 1. Runa on AWS Big Data & Machine Intelligence for a SaaS Startup
- 2. Runa
- 3. a SaaS
- 4. converts Shoppers to Buyers
- 5. for Online Commerce Sites
- 6. by presenting Dynamic Personalized Promotions
- 7. on the Merchant’s Website
- 8. in Real-Time
- 9. in the Shopping Flow
- 10. Tech Challenges
- 11. Big Data
- 12. JavaScript client collects activity on every Merchant page for every Shopper
- 13. One or more Ajax call & Event Store to Runa per Merchant page view
- 14. Step function increase of calls and stores as each new Merchant added
- 15. We capture everything we can and store it forever
- 16. Expecting to grow to thousands of merchants
- 17. That’s a lot of Data
- 18. Processing Data with Machine Intelligence
- 19. Batch Processing for Statistical Analysis and Reports
- 20. Real-Time Rule based inserts of Promotions
- 21. Why AWS for Runa?
- 22. At First (a couple years ago)
- 23. Not Much Money in the Bank
- 24. Didn’t Know exactly what were making
- 25. Or exactly how we were going to do it
- 26. Prototyped with Ruby / Rails / MySQL
- 27. Then Prototype became Production
- 28. EC2 & AWS let us scale the prototype to Beta Production
- 29. Flexibility to incrementally refine service & infrastructure
- 30. Confidence we could scale as we added Merchants
- 31. More Recently Incrementally added next-gen Tech & Full Production
- 32. Goal: Everything Horizontally Scalable
- 33. Batch Processing & Infinite Storage Map / Reduce & BigTable via Hadoop & HBase
- 34. Flexible Real-Time parallel processing via Clojure / Swarmiji
- 35. Opscode Chef Management & Monitoring Consumers on Merchant Websites Internet Admin & Merchant Dashboard (Rails) Runtime Rules Merchant Info Merchants Internet AnalyticsReporting Monitor & Recovery Data Collectors Hadoop / HBase Map / Reduce Petabyte Store Load Balancer HTTP Shared Session Memory HTTP Dispatchers Redis Mem Cache Redis Mem Cache Redis Mem Cache Redis Mem Cache Redis Mem Cache HBase HBase HBase HBase HBase HBase HBase HBase HBase HBase HBase HBase HBase HBase HBase HBase HBase HBase HBase HBase HBase HBase HBase HBase HBase HBase HBase HBase HBase HBase HBase HBase HBase HBase HBase HBase Amazon S3 Data Backup 9+ Amazon EC2 Instances Amazon Elastic Load Balancer 3+ Amazon EC2 Instances Cheshire / Swarmiji Dynamic Runtime Queue
- 36. AWS Elastic Load Balancer Rails App Servers Nginx / Unicorn EC2 m1.xlarge MySQL Master / Slave EC2 m1.xlarge EBS Legacy Runtime Rails App Nginx/Unicorn MySQL Master/Slave EC2 m1.xlarge / EBS Merchant Dashboard EC2 m1.xlarge HBase / Hadoop EC2 m1.xlargeEBS RabbitMQ Cheshire / Swarmiji Redis EC2 m1.xlarge Clojure Based Runtime AWS Elastic Load Balancer EC2 m1.large Opscode Chef Monitoring EC2 m1.large All Deployed on
- 37. Deployment & Configuration Management via Opscode Chef
- 38. Good Things
- 39. Able to Start Small
- 40. Then GROW BIGGER
- 41. Having the flexibility to throw “Hardware” at our Prototype got us to market faster
- 42. Ability to launch test and staging environments almost at will
- 43. “Hardware” as “Software”
- 44. Living in “interesting” times
- 45. Managing Complexity lots of moving parts
- 46. Easy to launch a few instances
- 47. Impossible to manage horizontal stacks “by hand”
- 48. Must have tool like Opscode Chef
- 49. Chef automates deployment & puts it under Revision Control
- 50. There’s going to be some blood when using cutting edge tech
- 51. Lots of Learning Curves to climb
- 52. Useful Monitoring is hard but Critical
- 53. HBase on AWS may be dangerous because of Hadoop namenode SPOF
- 54. EC2 bill can surprise you if you cavalierly deploy multiple versions of horizontally scalable environments
- 55. Could not do our startup without AWS or lots more VC Funding
