Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Extract The Traffic From The Db


Published on

A presentation I gave in the AlphaGeeks meetup regarding NoSQL and other methods to boost software system performance. We were talking about Sharding, NoSQL, Memcached, In Memory Databases, CouchDB, Hadoop and SimpleDB.
Feel free to contact me regarding the presentation. Moshe Kaplan.

Published in: Technology, Business
  • Be the first to comment

Extract The Traffic From The Db

  1. 1. Get the Traffic Out of the DB AlphaGeeks Meetup, Feb 17, 2010 [email_address]
  2. 2. Presentation Objectives <ul><li>Where is the bottleneck? </li></ul><ul><li>Assumptions and market trends </li></ul><ul><li>Sharding, In Memory Databases and NoSQL </li></ul><ul><li>Cast Studies </li></ul><ul><li>Summary </li></ul>
  3. 3. Assumptions <ul><li>Database is the bottleneck </li></ul><ul><li>Your only solution is scaling up </li></ul>
  4. 4. Why Do I Care? <ul><li>From 0 to 100 (US mass adaptation) </li></ul><ul><ul><li>Phone: 100 yrs </li></ul></ul><ul><ul><li>Radio: 40 yrs </li></ul></ul><ul><ul><li>TV: 30 yrs </li></ul></ul><ul><ul><li>Mobile: 20 yrs </li></ul></ul><ul><ul><li>Internet: 10 yrs </li></ul></ul><ul><ul><li>Facebook: 2 yrs </li></ul></ul>
  5. 5. 500K New Users/Week
  6. 6. The Network Effect
  7. 7. What Should I Do?
  8. 8. How Large Can a Server Be?
  9. 9. Assumptions…
  10. 10. Option A - Sharding
  11. 11. Option B – In Memory Databases
  12. 12. Case Study – Ad Network (Before) UPDATE … SET HIT=HIT+1 UPDATE … SET HIT=HIT+1 Validate Validate Banner 1 Hit Banner 2 Hit
  13. 13. Case Study – Ad Network (After) IMDB UPDATE … SET HIT=HIT+1 UPDATE … SET HIT=HIT+1 Validate Validate UPDATE SET HIT=HIT+41 UPDATE SET HIT=HIT+22 UPDATE SET HIT=HIT+87 Banner 1 Hit Banner 2 Hit
  14. 14. Downturns
  15. 15. IMDB - Implementation
  16. 16. MySQL Testbench
  17. 17. Lessons <ul><li>Cache: MySQL IMDB </li></ul><ul><li>HA: MySQL Replication </li></ul><ul><li>Persistence: Periodic 5min Batch </li></ul><ul><li>Scalability: Sharding </li></ul><ul><li>4 Bill Req/Day using 25 machines (2000/s) </li></ul>
  18. 18. Option C – NoSQL
  19. 19. NO SQL <ul><li>Transaction (ACID) </li></ul><ul><li>Group By </li></ul><ul><li>Joins </li></ul>
  20. 20. Put, Get, Remove, Sync
  21. 21. Memcached - Implementation
  22. 22. Numbers
  23. 23. Scaling Across Multiple DCs
  24. 24. The Families
  25. 25. CouchDB <ul><li>RESTful JSON API </li></ul><ul><li>Schema-free </li></ul><ul><li>Distributed </li></ul><ul><li>Replication with bi-directional conflict detection and managment </li></ul>
  26. 26. Best Practices <ul><li>Make a stress test, check the numbers! </li></ul><ul><li>MemcacheQ and MemcacheDB warning </li></ul><ul><li>Relational databases are 40 years old </li></ul><ul><li>NoSQL is only 5 years old… </li></ul><ul><li>Don’t forget to close the connections </li></ul>
  27. 27. The Bottom Line: Grow ∞
  28. 28. Startup your Engines Thank you [email_address] Our Methodology Performance problems are extremely complex and due to the diferent technologies deployed, each case is unique. A “typical” performance problem requires delving into databases, application servers, client technology, code in difering programming languages and system and software architectures. RockeTier implements a unique methodology in order to simplify the problem and evaluate each performance bottleneck, providing both an immediate efective relief and when necessary, design a gradual roadmap to speed up your software system and make it scalable and robust. Our 5 steps methodology: 1. Detect: Pinpoint your performance bottlenecks using various tools including load and stress tools, code profiling, database profiling, network sniffing and code review to detect performance bottlenecks in specific components. 2. Rate: Grade each bottleneck by importance and provide immediate practical recommendations and performance boost estimations. 3. Immediate effective relief: Provide immediate fixes and workarounds in a short time frame helping you meet your urgent business needs. 4. Roadmap Planning: When necessary, redesign next generation Solutions, using proven robust and scalable solutions such as grid and in memory databases. 5. Scale up and Scale out: In cases where redesign is necessary - RockeTier provides implementation or software design description (SDD), and guidance for in house programmers for the implementation of the next generation scalable system, which will meet your growing business needs. Your Value Business: Achieve your business performance requirements. GreenIT: Protect the environment and reduce CO2 emissions. Bottom Line: Reducing hardware and 3rd party software cost. The Performance Experts Success Stories The Finance Sector: An international insurance company managing over 20 Billion US dollars in assets was facing poor performance in its core life insurance policy software system. The RockeTier team detected bottlenecks originating from several software infrastructure modules. A practical solution was implemented. The customer’s success criteria was a 20% decrease in insurance policy creation run time, Our solution provided a 40% decrease in run time! Telecom: A VC backed start-up company was facing critical installation problems in the leading Israeli cellular operator. Knowing that existing system performance would not meet client requirements, the company asked RockeTier to help it boost its performance. RockeTier evaluated the system and implemented a workaround to the system database architecture, boosting the overall system performance by 30%. Following that. the RockeTier team designed the company’s next generation architecture, meeting a throughput of 3000Mbps by design. RockeTier at a Glance RockeTier is a software solutions company, which utilizes its knowledge and skills to help companies from both the enterprise sector and the start-up industry. RockeTier has numerous success stories in solving customers’ system performance bottlenecks and scale out limitations, providing immediate improvements and workarounds in a short time frame and, when necessary, redesign and implementation of the next generation solutions employing grid and/or in- memory databases in the Web 2.0, Telecom and finance markets. Web 2.0: a start-up company providing an innovative electronic advertising and billing system was facing its technological limits. The RockeTier team evaluated and redesigned its system architecture and is currently implementing a scale out grid mechanism and caching algorithms. The solution supports 20 times the original capacity using the same hardware. Moreover it supports semi-linear growth (by simple scale out) and high availability requirements. “ 20% reduction in transaction time within 3 months” “ Boost Performance by a factor of 200” “ 200 million events per day”