Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

A Cassandra driver from and for the Lua community

250 views

Published on

Talk given at LuaConf 2017 in Rio de Janerio about the Cassandra driver for Lua and OpenResty.

Published in: Software
  • Be the first to comment

A Cassandra driver from and for the Lua community

  1. 1. A Cassandra driver from and for the Lua community LuaConf 2017 - Rio de Janeiro Thibault Charbonnier Kong lead engineer OpenResty contributor https://github.com/thibaultcha 1
  2. 2. Cassandra - History & Use Cases 2
  3. 3. Apache Cassandra - History - Free and open-source NoSQL database - Published by Facebook in 2008 - Joined the Apache Foundation in 2010 3
  4. 4. Apache Cassandra - Use cases - Decentralized - Multi datacenter replication - Horizontal scalability (linear) - “Key/Value meets column-oriented” - CQL A typical Cassandra cluster, often represented as a ring 4
  5. 5. Apache Cassandra - Use cases - Leans towards AP - Tunable Consistency 5
  6. 6. Kong - History & Use Cases 6
  7. 7. Kong - History & Use Cases - API Gateway/Microservices orchestration tool - Started in 2014 - Free and open-source (Community Edition) More on Kong during Thijs Schreijer’s lightning talk at 4:45 PM https://github.com/Mashape/kong 7
  8. 8. Kong - Use Case 8
  9. 9. Why Cassandra for Kong? Long story short: business requirements! - Multi data center support “out of the box” - Ease to scale (linear + multi DC support) - Counters! Great for rate-limiting APIs - Great for time series data (logging & monitoring) One drawback... We need a pure Lua implementation! 9
  10. 10. Enters: lua-resty-cassandra 10
  11. 11. lua-resty-cassandra - History - Written by Juarez Bochi (@jbochi) in 2014 - Pure Lua (uses OpenResty’s cosockets) - Used by globo.com for its video streaming platform - Released under MIT License - Support for binary protocol v2 (Cassandra 2.x) - Single node https://github.com/jbochi/lua-resty-cassandra 11
  12. 12. lua-resty-cassandra - First Improvements Better support for Lua/OpenResty interoperability “Hmmm this is going to be a problem…” oops 12
  13. 13. lua-resty-cassandra - First Improvements Batch support, yay! “Thanks Juarez!” Batch support is necessary to provide Atomicity when updating several partition keys at once. 13
  14. 14. lua-resty-cassandra - First Improvements Pagination Support! 14
  15. 15. lua-resty-cassandra - A fork? - No love for our binary protocol v3 contribution :’( - Lack of responses from @jbochi :’( - Business requirements! Cassandra 3.x Binary protocol v3 Load balancing policies Retry policies Cluster awareness Multi data center Client-to-node SSL Client authentication Schema consensus 15
  16. 16. Introducing: lua-cassandra 16
  17. 17. Introducing: lua-cassandra Datastax-like driver for Lua: - Support for cluster awareness - Load balancing/retry/reconnection policies - Support for SSL connections - Cassandra 3.x - binary protocol v3 - Good documentation with concrete examples - Interoperability between OpenResty/Lua for community usage Forked in May 2015 We keep the lua-resty-cassandra marshallers! 17
  18. 18. lua-cassandra - Two Modules Single node: Cluster aware: OpenResty only :’( - Cluster topology discovery - Auto-prepared statements - Schema consensus wait on DDL queries - Retries - Policies - Automatic binary protocol selection (v2/v3/v4) - Support for SSL connections - Support for authentication 18
  19. 19. lua-cassandra - CQL queries Example of a DDL query 19
  20. 20. lua-cassandra - CQL queries Example of a DML query 20
  21. 21. lua-cassandra - CQL queries Example of a DML query with Cluster module 21
  22. 22. lua-cassandra - Authentication Support for PasswordAuthenticator: 22
  23. 23. lua-cassandra - Policies Load balancing - round-robin or datacenter-aware round-robin Reconnection - simple or exponential 23
  24. 24. lua-cassandra - Policies 24
  25. 25. leafo/pgmoon has the original Lua/OpenResty interoperability module https://github.com/leafo/pgmoon lua-resty-socket: An improved version released as a separate library https://github.com/thibaultcha/lua-resty-socket - Fallback on LuaSocket when cosockets are not available - Fallback on LuaSec when SSL is enabled lua-cassandra - Interoperability 25
  26. 26. lua-cassandra - Testing LuaCheck - https://github.com/mpeterv/luacheck busted - https://github.com/Olivine-Labs/busted Test::Nginx - https://github.com/openresty/test-nginx LuaCov - https://github.com/keplerproject/luacov 26
  27. 27. lua-cassandra - Documentation Built with LDoc - https://github.com/stevedonovan/LDoc Hosted on GitHub - https://thibaultcha.github.io/lua-cassandra/ 27
  28. 28. Reflexions 28
  29. 29. Maintaining code for Lua/LuaJIT/OpenResty - Sometimes hard because of disparities: - Language: (table.unpack, “table.new”…) - Environment: cosockets vs LuaSocket (or other…) & embedding application - Requires a heavy test suite - Modularization really helped - Consider the needs for the clients of your library, to avoid fragmentation Dropping support for PUC-Lua? More modularization to support cluster-awareness in PUC-Lua 29
  30. 30. We are hiring! https://mashape.com/jobs And contributors are welcome! 30
  31. 31. Thank you! Questions time! 31

×