Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Extreme Client Side Tweaking on Couchbase at Criteo: Couchbase Connect 2015

2,234 views

Published on

Extreme Client Side Tweaking on Couchbase at Criteo
When you're running 600 Couchbase nodes on 6 datacenters worldwide, happily servicing peaks of 10M QPS and all you have is a .NET client, things start getting interesting in ten millions ways. This is the story of how we embraced other technologies to improve our solution, rewrote an open source .NET Couchbase client from scratch, and how to tackle the head-warping problems that arise at scale.

Published in: Software
  • Be the first to comment

Extreme Client Side Tweaking on Couchbase at Criteo: Couchbase Connect 2015

  1. 1. Copyright © 2015 Criteo Extreme Client Side Tweaking on Couchbase Sébastien Foutrier Lead Software Developer June 3rd 2015
  2. 2. Copyright © 2015 Criteo 2 Criteo • French start up since 2005 • Keeping the spirit • Machine Learning and Big Data oriented • Algorithms as root • Ad Tech business • 230 in R&D 1500 worldwide • R&D teams in Paris
  3. 3. Copyright © 2015 Criteo 3 Criteo – Personalized performance advertising at scale • Personalized • Per user per view • Real time product recommendation • Performance • Cost of sales model • Cross device • Scale • 994 millions unique users reach globally • 741 billions ads in 2014
  4. 4. Copyright © 2015 Criteo 4 Site Reliability Engineering at Criteo 50 DevOps engineers and growing with the company Software infrastructure Systems (CentOS, Windows, monitoring) Automation (Chef) Big Data infrastructure (Hadoop, Storm, Kafka, HBase) NoSQL infrastructure (Memcached, Couchbase, Graphite, ElasticSearch, MongoDB, Cassandra, Redis) Escalation Predictive monitoring Build & continuous integration
  5. 5. Copyright © 2015 Criteo 5 Hardware Infrastructure • 6 Datacenters • 10k+ server (50/50 windows/linux) • Private worldwide dedicated network up to 10Gb/s • One of the biggest Hadoop clusters in Europe (1k+ servers, 37 PB)
  6. 6. Copyright © 2015 Criteo 6 From SQL to some NoSQL • Criteo started with • User data in client side cookies • Client and campaign configurations in MS SQL server • First issues • RAM cache in IIS servers • Server to server calls • Added some Memcached …. big instances
  7. 7. Copyright © 2015 Criteo 7 Next steps • Real Time Bidding • 100 ms to respond • Server to servers calls => bye bye client side cookies • Additional challenge : multi datacenter => sync ! • Lots of instances • Need for durability • Couchbase comes in
  8. 8. Copyright © 2015 Criteo 8 Grown big ! • 550 Couchbase servers (from 1.8 to 3) • 24 clusters • 107 TB Total RAM and SSD space • 10M hits/sec worldwide • 2M hits/sec peak single cluster
  9. 9. Copyright © 2015 Criteo 9 Things get serious, issues also • 3.5K+ Clients • Slow cross datacenter links (70ms) • Blocking call • 9K TCP sockets reached ! • Client buffering blows memory/threads
  10. 10. Copyright © 2015 Criteo 10 Solutions ? • Moxi • Unstable • Heavy CPU load • Bandwidth x3 • Prototype ! • https://code.google.com/p/memcached/wiki/BinaryProtocolRevamped Couchbase Moxi Memcached Couchbase Moxi Memcached
  11. 11. Copyright © 2015 Criteo 11 Raw socket, from scratch • Send(Byte[]) • Synchronous call • request buffering • sending thread(s) • BeginSend(…) / EndSend(…) • Overhead • SendAsync(SocketAsyncEventArgs e) • Efficient when reusing ‘e’
  12. 12. Copyright © 2015 Criteo 12 The asynchronous API if (!client.Get("Hello", (Status s, byte[] v) => { if (s == Status.NoError && v != null) Console.WriteLine("Hello, " + Encoding.UTF8.GetString(v)); else Console.WriteLine("Get failed with status " + s); })) Console.WriteLine("Get failed ");
  13. 13. Copyright © 2015 Criteo 13 Asynchronous gain • Less context switches • Less threads • Easy to pipeline !
  14. 14. Copyright © 2015 Criteo 14 TCP socket Sequential model Web Server Couchbase
  15. 15. Copyright © 2015 Criteo 15 TCP socket Pipelining model Web Server Couchbase
  16. 16. Copyright © 2015 Criteo 16 TCP socket Socket pooling TCP socket Web Server Couchbase
  17. 17. Copyright © 2015 Criteo 17 TCP socket TCP Windows Web Server Couchbase
  18. 18. Copyright © 2015 Criteo 18 Couchbase Redundancy (Memcached bucket) Web Server Couchbase Couchbase Couchbase OKDrop Z z z z
  19. 19. Copyright © 2015 Criteo 19 Couchbase GetWithReplica (Couchbase bucket) Web Server Couchbase Couchbase Couchbase OKOK Z z z z
  20. 20. Copyright © 2015 Criteo 20 Benchmark • Couchbase 1.X using Enyim.Caching • Ping 277 ms (Paris-Tokyo) • 20 nodes • 20 socket per node • 40 client threads
  21. 21. Copyright © 2015 Criteo 21 Benchmark • Criteo.Memcache • Ping 277 ms (Paris-Tokyo) • 20 nodes • 1 socket per node • 4 client threads
  22. 22. Copyright © 2015 Criteo 22 What is supported? • Get/Set/Update/Add • GetAndTouch • Increment/Decrement • Ketama locator • Couchbase locator • GetWithReplica • Stats • The features you’ll add!
  23. 23. Copyright © 2015 Criteo 23 Conclusion • Open source (feel free to contribute): • https://github.com/criteo/memcache-driver • Specific use cases • Async in Couchbase SDK 2.1
  24. 24. Copyright © 2015 Criteo 24 Conclusion (bis) CRITEO IS HIRING
  25. 25. Copyright © 2015 Criteo Thanks ! Questions ?

×