This is my talk from the third Israeli Aerospike User Group meetup. It covers modeling counters, handling hot keys on counters, and implementing accurate counters with Aerospike's strong consistency mode.
We’ll start with a recap from the beginning of my previous talk
In an RDBMS we connect to a database. In that database we have tables containing rows. Each row will typically have a primary key that uniquely identifies it
Rows contain one or more columns of a supported data types
Most applications avoid entity-relationship purity for access speed, denormalizing into a single table, and accessed by primary-key lookup
Records are stored contiguously
Records are grouped together into blocks and written in a single write operation to disk
The block size is controlled by the write-block-size config parameter
This applies to both raw device and file-based storage
The records are first placed in a streaming write buffer – a block in-memory of the same size as the write block
The SWB is flushed to disk when the block is full or when flush-max-ms is hit. Or every time if commit-to-device true
The primary index is updated. For each record being written Aerospike notes the device/file, the block ID, byte offset
Therefore any record can be reached with a single read IO
Assume many other bins – address, bills, payment history, etc, this object is 2K roughly
If there are 400M users storage is over ten billion bytes, almost 10GiB
In hybrid-memory mode (primary index in process memory or shared memory) Aerospike uses 64B of DRAM per-record
Less reads so we get the result faster, only if we always fetch all the data
Since most operations will be updates and not reads, there's a benefit to splitting them. Also the updates operate on a much smaller object reducing disk IO
For 400M objects we're using 8192 sprigs, memory overhead of 848MiB evenly distributed over the number of nodes
8B storage for numeric (integer, double) data
Removes all bin related overhead
How much memory does this take in Aerospike 4.2 or higher? How much storage space? (as low as 48B, typically 64B with short set name)
Create the monthly data counters ahead of time
If RecordNotFound you can also set it to any explicit TTL such as 24h, rather than use default-TTL
This is the opposite of consolidating counters into a single record.
This is a more complex solution, separating into an in-memory or data-in-index namespace.
In some languages exception handling is very heavy (Java), in others you are encouraged to EAFP – try first, handle the exception.
If you search redislabs 'strong consistency' on Google the term comes up in the search results, but magically it doesn't appear in the page itself. That's 'solving' a technical gap with marketing and SEO.
Some competitors use 'strong eventual consistency'. That is a form of weak consistency.
As a metaphor you know Golden State Warriors? 1st place in Western Conference of the NBA. Do you know Santa Cruz Warriors? 1st place in the G-League with .707 win/total ratio (better than Golden State). However, it's 1st place in the G-League, not the NBA.
Write went to the master, master wrote to the slave, master dies. The state on the slave is now different, and it doesn't know the master died. The slave is promoted but the write failed. Any read now before the client acts (to write again or not write again) is an unexpected value. The client might try to repeat the write again and modify not the previous state but the current one.
A strongly consistent counter is modeled with a pair of bins – an integer counter and an unordered list of unique transaction IDs.
A strongly consistent counter is modeled with a pair of bins – an integer counter and an unordered list of unique transaction IDs.
A strongly consistent counter is modeled with a pair of bins – an integer counter and an unordered list of unique transaction IDs.
A strongly consistent counter is modeled with a pair of bins – an integer counter and an unordered list of unique transaction IDs.
Removing a cart is exactly like a rollback when there aren't enough seats.
More of these to come
New and renewing EE users, and anyone signed up to download CE gets academy access.