In this presentation, we share approaches to replacing RAM-only caching infrastructure while achieving high performance against a persistent datastore. Memcached has proven very popular, but it also requires its users to sacrifice reliability, scalability, redundancy, availability, and security. To address these issues, Sensaphone implemented a memcached replacement called ValuStor, an easy-to-use key-value database client layer written in C++ that works well with Scylla. ValuStor includes features like client-side write queues, multi-threading support, automatic adaptive consistency, and support for multiple data types (including JSON).

1. Introducing ValuStor:
Memcached Alternative for Scylla
Derek Ramsey
Software Engineering Manager, Sensaphone

2. Who Am I?
Software Engineering Manager at Sensaphone
Developer and maintainer of ValuStor
Added ~33,000 county/city articles to Wikipedia
Avid nature photographer
Father of 5 children

3. INTRO TO VALUSTOR

4. What is ValuStor?
Key-value and document store NoSQL database
Header-only C++ abstraction layer
Replacement for memcached
MIT License

5. History
Memcached as MySQL cache
▪ 100’s of simultaneous connections
• Batched writes bought time
▪ No redundancy or scalability
▪ Cold cache performance risk
Took three days from inception to production
▪ No critical bugs
Success! Interest! … Open source it!
(Usage Guide on github)

6. Memcached
Often installed on client machines
Memcached FAQ
▪ Not recommended for sessions
▪ Minimal Security: Only SASL authentication
▪ No failover, replication, or persistence
▪ Must use batching for performance

7. FEATURES

8. ValuStor Features: Client
▪ Ease of use
▪ Abstraction
▪ Multiple Data Types
• Integers (bool, 8-, 16-, 32-, 64-bit)
• Floating Point
• String
• JSON (nlohmann::json)
• Blob / Bytes
• UUID
▪ Client-side write queue (backlog)
▪ Automatic Adaptive Consistency

9. ValuStor Features: Driver
Thread safety
Multi-threading
Connection Control
Server Selection

10. ValuStor Features: Server
Architectural Advantages
▪ Performance
▪ Security
▪ Fault Tolerance
• Reliability
• Redundancy
• Availability
▪ Scalability

11. Performance
Can a persistent DB compete with
RAM-only caching?
Async I/O
▪ < 1ms latency possible
Caching
▪ Cold cache penalty
▪ Cache warming (heat-weighted load balancing)

12. Security
ValuStor and Scylla
▪ TLS
• Client certificate authentication
• Server cert verification
• Domain or IP
• Inter- and intra-DC encryption
▪ Password Authentication
Memcached
▪ Vulnerable by design
• CVE-2018-1000115
• DoS in UDP Protocol
• Solution: disable UDP by default
• CVE-2017-9951
• DoS overflow
• CVE-2016-8704/8705/8706
• Exec Code Overflow

13. Fault Tolerance: Reliability
The database is offline, panic!
Client-side write queue
▪ Put failed writes into a backlog queue
• Automatic retry
• Backlog clears when database returns
• Modes: Allow, Only Use, Don’t Use
▪ Producers continue producing
• Writes are delayed, not lost

14. Fault Tolerance: Redundancy
Custom data replication factor
▪ Configurable per keyspace (“database”)
▪ Can be changed on-the-fly
▪ Client aware
Multi-datacenter support
▪ Data replication factor per keyspace and per data center
▪ Client aware

15. Fault Tolerance: Availability
CAP Theorem
▪ Consistency, Availability, Partition tolerance
C and P
▪ Redis
▪ MongoDB
Quorum Problem
▪ Must have N/2 + 1 active nodes
▪ Fragile multi-datacenter setups

16. Quorum Problem: Example 1
Quorum requires three nodes.

17. Quorum Problem: Example 1

18. Quorum Problem: Example 2
Quorum now requires four nodes.

19. Quorum Problem: Example 2
The primary DC goes down…
...and a backup DC takes over.

20. Quorum Problem: Example 2
Both secondary DC are down...
...the primary loses quorum!

21. Fault Tolerance: Availability
Solution: Automatic Adaptive Consistency
▪ Eventual Consistency
▪ Tunable Consistency
▪ Adaptively downgrade on retry
• Likelihood of inconsistency can be low or limited
▪ Bonus: Hinted Handoff

22. Adaptive Consistency: Writes
Data is written to all nodes.
If two nodes go down…
...just keep going!
Data on the remaining node is still consistent*

23. Adaptive Consistency: Writes
Data is written to all nodes.
If two nodes go down…
...just keep going!
Data on the remaining node is still consistent*
* If you used consistency mode ALL

24. Adaptive Consistency: Writes
ALL: Writes confirmed on every node
QUORUM: Some recent writes can be lost
ANY: At least it went somewhere… (e.g. Domain Name System)

25. Adaptive Consistency: Reads
ALL: Always consistent
QUORUM: Might be inconsistent [after node failure]
ANY: At least you get something…
Automatically or manually fix inconsistencies

26. Scalability
Scylla architecture has (almost) ∞ scalability
▪ Sharding per core: just add more
▪ Driver is sharding-aware
Memcached
▪ Physically limited
▪ Must shard manually

27. USING VALUSTOR

28. ValuStor Configuration
// e.g. CREATE TABLE cache.values (key bigint, value text, PRIMARY KEY (k1))
ValuStor::ValuStor<std::string, int64_t> store({
{"table", "cache.values"},
{"key_field", "key"},
{"value_field", "value"},
{"hosts", "127.0.0.1"}
});

29. ValuStor Usage
Example
// e.g. CREATE TABLE cache.values (key_field bigint, value_field text, PRIMARY KEY (key_field))
//
// <value> <key>
ValuStor::ValuStor<std::string, int64_t> vs("example.conf");
auto store_result = vs.store(1234, "value");
if(store_result){
auto retrieve_result = vs.retrieve(1234);
if(retrieve_result){
std::cout << 1234 << " => " << result.data << std::endl;
}
}

30. COMPARISONS

31. Key-Value NoSQL Comparison
ValuStor Memcached Redis Riak KV* DynamoDB
TLS Support
Persistence
Multi-Master Replication
Multi-DC Replication
FOSS
Language C++ C C Erlang Cloud Based
Complexity Simple Simple Complex Complex Complex
*Open-source / community version

32. What about Redis?
▪ It’s more complex*
▪ May not perform as well
▪ Only master-slave replication
▪ Commons Clause modules
Replace with Pedis
▪ Uses SEASTAR for I/O
...or use ValuStor
* This may be desired

33. What about MongoDB or CouchDB?
Use Case: JSON document store
#include "nlohmann/json.hpp"
#include "ValuStor.hpp"
…
ValuStor::ValuStor<nlohmann::json, int64_t> docstore("example.conf");
std::string raw_json = "{"key":"value"}";
auto json = nlohmann::json::parse(raw_json);
docstore.store(document_id, document);
auto result = docstore.retrieve(document_id);
if(result){
raw_json = result.data.dump();
}

34. FUTURE

35. Future
Beyond C/C++
▪ SWIG Bindings (www.swig.org)
• Current: PHP, Python, Perl
• Future: C#, D, Go, Java, Javascript, Ruby
▪ Command line utilities
New Features
▪ Performance enhancement
• Futures
▪ Non-template version

36. Thank You
Any Questions ?
Please stay in touch
dramsey@sensaphone.com
@ThyRamMan
github.com/Sensaphone/ValuStor

37. ANSWERS TO QUESTIONS

38. What is ValuStor?
Client Driver Database
Ease of Use Multi-Threading Persistence
Abstraction Layer Thread Safety High Availability
Type Flexibility Server Selection Redundancy
Backlog Write Queue Connection Control Scalability
Adaptive Consistency Security Security
Performance

39. Why Scylla?
Eventually consistent persistent NoSQL database
Efficient, close-to-the-hardware design
Consistent, low-latency performance
▪ C++
▪ Asynchronous I/O
▪ Userspace I/O scheduler
▪ Shared per core

40. Caveats and Limitations
Atomicity
▪ Writes are atomic, but not ordered
• Requires minimum QUORUM consistency
▪ No read-and-write operations or ACID transactions
Cost
▪ RAM for memcached relatively cheap
▪ Additional Maintenance
Memcached evicts data, Scylla does not
▪ May need more RAM or effective use of TTL

41. ValuStor Installation
Dependencies
▪ C++11 compiler
▪ Cassandra C/C++ client driver
• “libssl” and “libuv” libraries
• https://github.com/datastax/cpp-driver/releases
▪ Scylla
• https://docs.scylladb.com/getting-started/

42. ValuStor Installation
Include the ValuStor header file.
#include "ValuStor.hpp"
Have “cassandra.h” in the include path.
Link with “libcassandra.so” and pthread.

43. ValuStor API
// TEMPLATE
template<typename Val_T, typename Key_T...> class ValuStor
// CONSTRUCTORS
ValuStor( std::string config_file )
ValuStor( std::map<std::string, std::string> configuration_kvp )
// API
Result store( Key_T... keys,
Val_T value,
uint32_t seconds_ttl = 0,
InsertMode_t insert_mode = ValuStor::DEFAULT_BACKLOG_MODE,
int64_t microseconds_since_epoch = 0 )
Result retrieve( Key_T... keys,
size_t key_count = 0)

44. ValuStor Configuration
ValuStor::ValuStor<std::string, int64_t> store({
...
{"username", "username"},
{"password", "password"},
{"server_trusted_cert", "/path/to/ca.pem"},
{"server_verify_mode", "3"},
{"client_ssl_cert", "/path/to/client.crt"},
{"client_ssl_key", "/path/to/client.key"},
{"client_key_password", "password"}
});

45. ValuStor Usage
Comparison: Reads
// libmemcached
std::string key = "key to find";
memcached_return rc;
size_t value_length = 0;
uint32_t flags = 0;
char* result = memcached_get(memc, key.c_str(), key.size(), &value_length, &flags, &rc);
if(rc == MEMCACHED_SUCCESS){ std::string value_found(result); }
else{ std::string error_message(memcached_last_error_message(memc)); }
// ValuStor
auto result = store.retrieve("key to find");
if(result){ std::string value_found = result.data; }
else{ std::string error_message = result.result_message; }

46. ValuStor Usage
Comparison: Multiple Reads
// libmemcached
// Won’t fit on the slide
// ValuStor
std::vector<std::string> keys = {"key1", "key2", "key3"};
for(auto& key : keys){
auto result = store.retrieve(key);
if(result){
std::string val = result.data;
...
}
}

47. ValuStor Usage
Comparison: Writes
// libmemcached
std::string key = "key to use";
std::string value = "value to write";
time_t expiration = time(nullptr) + 1000; // or '0' for no expiration
memcached_return rc = memcached_set(memc, key_to_use.c_str(), key_to_use.size(), value.c_str(), value.size(),
expiration_time, 0);
// ValuStor
auto result = store.store("key to use", "value to write", 1000); // or '0' for no expiration

48. Scylla Database Administration
Creating keyspaces
▪ CREATE KEYSPACE space WITH replication = {'class':'NetworkTopologyStrategy', 'DC1':3, 'DC2':1}
Creating tables
▪ CREATE TABLE space.mytable (mykey bigint PRIMARY KEY, myval text)
Configuration
▪ Password authentication
▪ TLS
Multiple nodes / data centers

49. Compound Keys
Partition Key
▪ Must be completely specified on queries
▪ Data is stored on disk in Partition Key chunks
Clustering Key
▪ May be conditionally specified on queries
▪ Allows querying of multiple records
Example:
▪ PRIMARY KEY ((key1, key2,...,keyN), keyB, keyC, …, keyZ)