Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Aerospike: Comparing NoSQL Databases for Real-Time Bidding

6,173 views

Published on

Sergey Zhemzhitsky, CTO of CleverDATA (a division of LANIT, a leading system integrator in Russia), presented via live webinar how his team developed their in-house tech stack - which includes Hadoop, NoSQL, and a variety of Data Mining and Big Data Analytics tools.

Watch the video! http://pages.aerospike.com/32515----Webinar-Compare-RealTime-NoSQL_32515----Webinar-Compare-RealTime-NoSQL-Watch.html

Learn about:
• Architecture of the CleverDATA Platform including DMP capabilities
• Results of evaluations and performance testing on Aerospike, MongoDB, and Redis
• Critical requirements for operating DSP, SSP and DMP for high performance, low-latency scaling
• Do’s and Don’ts of applying this design pattern to your real-time application

About the Speaker:

Sergey Zhemzhitsky started in IT sphere 10 years ago as a developer and rose through the ranks to joined CleverDATA in 2014 as Chief Technical Officer.

About CleverDATA:

CleverDATA specializes in developing and providing an in-house platform 1DMP.RU as a cloud service. The platform enables customers to exchange, process and accumulate big amounts of various data. The company provides IT consulting services, develops private data management platform and implements solutions involving AdTech technologies, unstructured big data mining, digital intelligence and operational analytics, machine learning for marketing automation, risk management, real-time web analytics, RTB purposes and customer engagement and retention.

About Aerospike

Aerospike is the world’s fastest database – the system of engagement – powering a new class of real-time, context-driven applications that personalize the consumer experience across the Internet. Developers use Aerospike, an open-source, flash-optimized, in-memory NoSQL key-value store for caching, as a user profile or context store, and to simplify scaling with smaller clusters and the price/performance of flash. Recognized by industry analysts as a visionary and leader, Aerospike powers 13 of the top 26 real-time bidding platforms including AppNexus and developers are rapidly taking advantage of the Startup Special to scale their business. Download the open source Aerospike Community Edition at www.aerospike.com and follow @aerospikedb.

Published in: Data & Analytics
  • Link to Video for the Webinar: http://pages.aerospike.com/32515----Webinar-Compare-RealTime-NoSQL_32515----Webinar-Compare-RealTime-NoSQL-Watch.html
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here

Aerospike: Comparing NoSQL Databases for Real-Time Bidding

  1. 1. WEBINAR: Comparing NoSQL Databases for Real-time Bidding Presented by Sergey Zhemzhitsky, CTO of CleverDATA
  2. 2. cleverdata.ru | info@cleverdata.ru International market business development since 2012 One of three leading IT companies in Russia 43 branches in Russia and abroad +5500 employees 100K projects for 10K customers Data management innovative platform (Data Exchange Service) Cloud Service In-house development Internet advertising solutions Data Management Platforms Customers Base Management Web Analytics Marketing automation Big Data Data Mining Digital Intelligence Operational Intelligence Low Latency and NoSQL Cloud Computing
  3. 3. cleverdata.ru | info@cleverdata.ru Agenda • RTB intro; • Challenges; • Choice difficulties; • Results; • Do’s and Don’ts.
  4. 4. cleverdata.ru | info@cleverdata.ru RTB intro
  5. 5. cleverdata.ru | info@cleverdata.ru Real Time Bidding (RTB) publishers AD NETWORK AD NETWORK AD NETWORK AD NETWORK AD NETWORK AD NETWORK advertisers
  6. 6. cleverdata.ru | info@cleverdata.ru How RTB works publishers AD NETWORK AD NETWORK AD NETWORK AD NETWORK AD NETWORK AD NETWORK advertisers
  7. 7. cleverdata.ru | info@cleverdata.ru Demand Side Platform (DSP) advertiserspublishers AD NETWORK AD NETWORK AD NETWORK AD NETWORK AD NETWORK AD NETWORK D S P
  8. 8. cleverdata.ru | info@cleverdata.ru Supply Side Platform (SSP) publishers AD NETWORK AD NETWORK AD NETWORK AD NETWORK AD NETWORK AD NETWORK advertisers S S P
  9. 9. TRACKING DATA cleverdata.ru | info@cleverdata.ru Data Management Platform (DMP) publishers COOKIE SYNCs ACCESS LOGS PARTNERS’ DATA 3rd PARTY DATA CLICK STREAMS advertisers S S P D S P WHATEVER ELSE DATA
  10. 10. cleverdata.ru | info@cleverdata.ru RTB challenges visitors publishers advertisers100ms 20 ms 10.000+ rps
  11. 11. TRACKING DATA cleverdata.ru | info@cleverdata.ru publishers COOKIE SYNCs ACCESS LOGS PARTNER’S DATA 3rd PARTY DATA CLICK STREAMS advertisers S S P D S P DMP in the RTB ecosystem DMP
  12. 12. cleverdata.ru | info@cleverdata.ru Wishes…
  13. 13. cleverdata.ru | info@cleverdata.ru Choice criteria • Linear scalability; • Sharding “out of the box”; • Distributed by design; • Data redundancy & replication; • Low latency.
  14. 14. cleverdata.ru | info@cleverdata.ru Which NoSQL … again
  15. 15. cleverdata.ru | info@cleverdata.ru Aerospike Cassandra Mongo Redis Scalability Sharding Redundancy Latency SPoF Maintainability Monitoring tools Features
  16. 16. cleverdata.ru | info@cleverdata.ru Public results
  17. 17. cleverdata.ru | info@cleverdata.ru nginx 1.2.x ngx-http-redis lua-nginx-module lua-resty-mongol ngx-aerospike aerospike 2.x redis 2.6.x mongodb 2.4.x wrk iperf nmon • Intel® Core™ i7-920 Quad-Core • 48 GB RAM • 1 Gbit/s NICs (*) • 20 * 106 msgs • 512 bytes msg * 200 Mbit/s guaranteed Tools & Materials
  18. 18. cleverdata.ru | info@cleverdata.ru • No logging; • No plugins; • CPU Affinity. NGINX WRK Measuring Nginx
  19. 19. cleverdata.ru | info@cleverdata.ru Nginx results. Base line L., ms., 50% L., ms., 75% L., ms., 90% L., ms., 99% T., r/s 1.35 1.46 1.64 1.90 68851
  20. 20. cleverdata.ru | info@cleverdata.ru Measuring Redis NGINX WRK Node 1 Shard 1 Shard 2 Slave 3 Slave 4 Node 2 Shard 3 Shard 4 Slave 5 Slave 6 Node 3 Shard 5 Shard 6 Slave 2 Slave 1
  21. 21. cleverdata.ru | info@cleverdata.ru Redis results L., ms., 50% L., ms., 75% L., ms., 90% L., ms., 99% T., r/s 2.68 3.10 3.25 3.90 34769
  22. 22. cleverdata.ru | info@cleverdata.ru Redis results
  23. 23. cleverdata.ru | info@cleverdata.ru Measuring Mongo NGINX :: MONGOS WRK Node 1 mongod :: cfg mongod :: repl mongod :: repl mongod :: repl Node 2 mongod :: cfg mongod :: repl mongod :: repl mongod :: repl Node 3 mongod :: cfg mongod :: repl mongod :: repl mongod :: repl Shard 1 Replica Set 1 Shard 2 Replica Set 2 Shard 3 Replica Set 3
  24. 24. cleverdata.ru | info@cleverdata.ru Mongo results L., ms., 50% L., ms., 75% L., ms., 90% L., ms., 99% T., r/s 6.70 8.22 10.22 15.46 14220
  25. 25. cleverdata.ru | info@cleverdata.ru Mongo results
  26. 26. cleverdata.ru | info@cleverdata.ru Node 1 Chunk 1 Replica 2 Node 2 Chunk 2 Replica 3 Node 3 Chunk 3 Replica 1 NGINX WRK Measuring Aerospike
  27. 27. cleverdata.ru | info@cleverdata.ru L., ms., 50% L., ms., 75% L., ms., 90% L., ms., 99% T., r/s 8.93 14.99 26.83 106.48 3402 Aerospike results
  28. 28. cleverdata.ru | info@cleverdata.ru Aerospike results
  29. 29. cleverdata.ru | info@cleverdata.ru Wks L, 50% L, 75% L, 90% L, 99% T, r/s Ngx, CPU Ngx, p/s As, CPU As, p/s 4 8.22 10.22 15.46 97.3 3402 7 8 K 1 1 K 8 7.89 15.16 92.4 89.8 6028 10 13 K 1 2 K 12 4.76 8.74 103.1 121.6 10233 15 20 K 2 3 K 16 3.91 6.21 99.6 111.3 13178 22 26 K 3 6 K 24 2.13 2.87 4.68 76.81 25744 21 60 K 5 10 K 32 2.01 2.60 4.42 81.29 28925 25 70 K 6 11 K 64 2.54 3.66 112.4 118.1 26468 27 70 K 6 11 K … more Aerospike results
  30. 30. cleverdata.ru | info@cleverdata.ru nginx + blocking I/O How to shoot yourself in the foot…
  31. 31. cleverdata.ru | info@cleverdata.ru Libevent to the rescue (2nd attempt) Node 1 Chunk 1 Replica 2 Node 2 Chunk 2 Replica 3 Node 3 Chunk 3 Replica 1 LIBEVENT :: LIBEVHTP WRK
  32. 32. cleverdata.ru | info@cleverdata.ru Finally… L., ms., 50% L., ms., 75% L., ms., 90% L., ms., 99% T., r/s 2.64 3.09 3.27 3.95 35746
  33. 33. cleverdata.ru | info@cleverdata.ru Finally…
  34. 34. cleverdata.ru | info@cleverdata.ru Nginx Redis Mongo Aerospike, b Aerospike, nb Latency, 50% 1.35 2.68 6.70 8.93 2.64 Latency, 75% 1.46 3.10 8.22 14.99 3.09 Latency, 90% 1.64 3.25 10.22 26.83 3.27 Latency, 99% 1.90 3.90 15.46 106.48 3.95 Throughput, m/s 68851 34769 14220 3402 35746 CPU, Http, % 29 20 70 7 33 Network, Http, p/s, K 71 71 37 8 71 CPU, Db, % - 6 25 1 6 Network, Http, p/s, K - 12 21 1 12 Unofficial results
  35. 35. cleverdata.ru | info@cleverdata.ru Aerospike
  36. 36. cleverdata.ru | info@cleverdata.ru Let’s start building…
  37. 37. cleverdata.ru | info@cleverdata.ru • Do use non-blocking I/O; • Do computations locally; • Be lazy. Don’t do more than needed; • Don’t afraid of using other’s good ideas; • Don’t trust anyone. Do’s & Don’ts
  38. 38. cleverdata.ru | info@cleverdata.ru Thanks a lot for your questions!
  39. 39. info@cleverleaf.co.uk :: info@cleverdata.ru cleverleaf.co.uk :: cleverdata.ru 1dmp.io/en :: crawler.1dmp.io/en facebook.com/CleverData :: +7 (495) 967-66-50
  40. 40. To get started now with Aerospike, visit: aerospike.com/get-started Tweet us at @Aerospikedb

×