Scaling Runa Inc Big Data e-commerce service with AWS

362
-1

Published on

Presentation given the first AWS Startup Event 4/14/2010. Describes how Runa was using AWS for its SaaS for e-commerce sites

Published in: Technology
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
362
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
8
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

Scaling Runa Inc Big Data e-commerce service with AWS

  1. 1. Runa on AWS Big Data & Machine Intelligence for a SaaS Startup
  2. 2. Runa
  3. 3. a SaaS
  4. 4. converts Shoppers to Buyers
  5. 5. for Online Commerce Sites
  6. 6. by presenting Dynamic Personalized Promotions
  7. 7. on the Merchant’s Website
  8. 8. in Real-Time
  9. 9. in the Shopping Flow
  10. 10. Tech Challenges
  11. 11. Big Data
  12. 12. JavaScript client collects activity on every Merchant page for every Shopper
  13. 13. One or more Ajax call & Event Store to Runa per Merchant page view
  14. 14. Step function increase of calls and stores as each new Merchant added
  15. 15. We capture everything we can and store it forever
  16. 16. Expecting to grow to thousands of merchants
  17. 17. That’s a lot of Data
  18. 18. Processing Data with Machine Intelligence
  19. 19. Batch Processing for Statistical Analysis and Reports
  20. 20. Real-Time Rule based inserts of Promotions
  21. 21. Why AWS for Runa?
  22. 22. At First (a couple years ago)
  23. 23. Not Much Money in the Bank
  24. 24. Didn’t Know exactly what were making
  25. 25. Or exactly how we were going to do it
  26. 26. Prototyped with Ruby / Rails / MySQL
  27. 27. Then Prototype became Production
  28. 28. EC2 & AWS let us scale the prototype to Beta Production
  29. 29. Flexibility to incrementally refine service & infrastructure
  30. 30. Confidence we could scale as we added Merchants
  31. 31. More Recently Incrementally added next-gen Tech & Full Production
  32. 32. Goal: Everything Horizontally Scalable
  33. 33. Batch Processing & Infinite Storage Map / Reduce & BigTable via Hadoop & HBase
  34. 34. Flexible Real-Time parallel processing via Clojure / Swarmiji
  35. 35. Opscode Chef Management & Monitoring Consumers on Merchant Websites Internet Admin & Merchant Dashboard (Rails) Runtime Rules Merchant Info Merchants Internet AnalyticsReporting Monitor & Recovery Data Collectors Hadoop / HBase Map / Reduce Petabyte Store Load Balancer HTTP Shared Session Memory HTTP Dispatchers Redis Mem Cache Redis Mem Cache Redis Mem Cache Redis Mem Cache Redis Mem Cache HBase HBase HBase HBase HBase HBase HBase HBase HBase HBase HBase HBase HBase HBase HBase HBase HBase HBase HBase HBase HBase HBase HBase HBase HBase HBase HBase HBase HBase HBase HBase HBase HBase HBase HBase HBase Amazon S3 Data Backup 9+ Amazon EC2 Instances Amazon Elastic Load Balancer 3+ Amazon EC2 Instances Cheshire / Swarmiji Dynamic Runtime Queue
  36. 36. AWS Elastic Load Balancer Rails App Servers Nginx / Unicorn EC2 m1.xlarge MySQL Master / Slave EC2 m1.xlarge EBS Legacy Runtime Rails App Nginx/Unicorn MySQL Master/Slave EC2 m1.xlarge / EBS Merchant Dashboard EC2 m1.xlarge HBase / Hadoop EC2 m1.xlargeEBS RabbitMQ Cheshire / Swarmiji Redis EC2 m1.xlarge Clojure Based Runtime AWS Elastic Load Balancer EC2 m1.large Opscode Chef Monitoring EC2 m1.large All Deployed on
  37. 37. Deployment & Configuration Management via Opscode Chef
  38. 38. Good Things
  39. 39. Able to Start Small
  40. 40. Then GROW BIGGER
  41. 41. Having the flexibility to throw “Hardware” at our Prototype got us to market faster
  42. 42. Ability to launch test and staging environments almost at will
  43. 43. “Hardware” as “Software”
  44. 44. Living in “interesting” times
  45. 45. Managing Complexity lots of moving parts
  46. 46. Easy to launch a few instances
  47. 47. Impossible to manage horizontal stacks “by hand”
  48. 48. Must have tool like Opscode Chef
  49. 49. Chef automates deployment & puts it under Revision Control
  50. 50. There’s going to be some blood when using cutting edge tech
  51. 51. Lots of Learning Curves to climb
  52. 52. Useful Monitoring is hard but Critical
  53. 53. HBase on AWS may be dangerous because of Hadoop namenode SPOF
  54. 54. EC2 bill can surprise you if you cavalierly deploy multiple versions of horizontally scalable environments
  55. 55. Could not do our startup without AWS or lots more VC Funding
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×