Exploring Modeling - Doing More with Lists

Exploring Data Modeling
Doing More with Lists

2 Proprietary & Confidential | All rights reserved. © 2018 Aerospike Inc.
 Hi again! I’m Ronen Botzer, and I’m a solutions architect at Aerospike.
 I’ve worked at Aerospike since June 2014, first on the Python and PHP clients.
 I’m active on StackOverflow and the Aerospike community forum.
 This is the second in a series of tech talks about data modeling.
 Check out the slides from the first Israeli ASUG meetup.
Welcome to Aerospike User Group – Tel Aviv!

 A Quick Overview of Aerospike Data Types
 The List API – Modify flags, return types for List operations
 Modeling with Lists
 Lists as queues
 Unique ordered lists
 Containers for time series data
 Leaderboards
 New Additions to the Map API
Agenda

Aerospike is a Primary Key Database
Objects stored in Aerospike are called records
A bin holds the value of a supported data type: integer, double, string, bytes, list, map,
geospatial
Every record is uniquely identified by the 3-tuple (namespace, set, primary-key)
A record contains one or more bins
(namespace, set, primary-key)
EXP – Expiration Timestamp
LUT – Last Update Time
GEN – Generation
RECORD
EXP LUT GEN BIN1 BIN2

 Aerospike is a row-oriented distributed database
 Rows (records) contain one or more columns (bins)
 Similar to an RDBMS with primary-key table lookups
 Single record transactions
 Namespaces can be configured for strong consistency
Aerospike Concepts
Aerospike RDBMS
Namespace Tablespace or Database
Set Table
Record Row
Bin Column
Bin type
Integer
Double
String
Bytes
List (Unordered, Ordered)
Map (Unordered,
K-Ordered, KV-Ordered)
GeoJSON

 Aerospike data types have powerful APIs.
 Atomic operations simplify application logic and reduce network load.
 Complex Data Types (List, Map) can be nested to many levels.
 Atomic operations can execute only at the top level of a List or Map.
 Native operations perform and scale better than UDFs.
Data Modeling in Aerospike

[ 1, 4, 7, 3, 9, 26, 11 ]
 The index is the (0 origin) position of an element in the list.
 The value at index 2 is 7. The value at index -2 is 26.
 The rank is the value order of the elements in the list.
 The lowest value element has rank 0. The highest value element has rank -1.
 The value of the element with rank 2 is 4. The value of the element with rank -2 is 11.
List Terms

 Starting with release 3.16, a List has a type, either unordered (default) or ordered.
Unordered [ 1, 4, 7, 3, 9, 26, 11 ]
Ordered [ 1, 3, 4, 7, 9, 11, 26 ]
 An unordered list maintains insertion order, as long as new elements are appended.
 An ordered list maintains rank order, and sorts itself every time new elements are added.
List Types

Ordered [ 1, 3, 4, 7, 9, 11, 26, 'a', 'c', [ 25, 'y' ] ]
 Elements of different data types are ordered by type.
 Nil < (Boolean) < Integer < String < List < Map < Bytes < Double < GeoJSON
 In an unordered list, getting elements by rank will obey this (just-in-time) ordering.
List Element Ordering

List operations supported by the server. Method names in the clients might be different.
set_type() (unordered, ordered)
append(), append_items(), insert(), insert_items(), set()
increment()
sort(), clear(), size()
remove_by_index(), remove_by_index_range()
remove_by_rank(), remove_by_rank_range()
remove_by_value(), remove_by_value_interval(), remove_all_by_value()
remove_all_by_value_list(), remove_by_value_rel_rank_range()
get_by_index(), get_by_index_range()
get_by_rank(), get_by_rank_range()
get_by_value(), get_by_value_interval(), get_all_by_value()
get_all_by_value_list(), get_by_value_rel_rank_range()
List Operations

Unordered [ 1, 4, 7, 3, 9, [ 25 ], 11 ]
 List increment() is atomic, and can only execute on integer or float values.
 You cannot increment the value at index 5 because its data type is a list.
 Atomic list operations only work against top level elements, not nested values.
Incrementing a List Value

opFlags include the result type flags and the INVERTED flag.
Return Types for List Operations
Result Type Description
None No results
Index Index order: 0 = smallest index, -1 = largest index
RevIndex Reverse index order: 0 = largest index
Rank Value order: 0 = smallest value, -1 = largest value
RevRank Reverse value order: 0 = largest value
Count Return number of items matching criteria
Value Value for single item operations, list of values for multi-ops

Unordered [ 0, 1, 1, 2, 3, 5, 8, 13, 21, 34, 55, 89, 144, 233 ]
Queue:
append_items([377, 610], UNORDERED)
Dequeue:
remove_by_index_range(VALUE, 0, 1)
Lists as Queues

 append(value[, createType, modifyFlag])
 append_items(items[, createType, modifyFlag])
 insert(index, value[, modifyFlag])
 insert_items(index, items[, modifyFlag])
Modify Flags for Operations that Add Elements
Flag Description
ADD_UNIQUE Only add elements that do not already exist in the list
INSERT_BOUNDED Do not insert past index N, where N is element count
NO_FAIL No-op instead of fail if policy violation occured, such as
ADD_UNIQUE or INSERT_BOUNDED
DO_PARTIAL When used with NO_FAIL, add elements that did not violate
policy

modifyFlags = ADD_UNIQUE | DO_PARTIAL| NO_FAIL
• append_items(ORDERED, [ 'michaelz', 'robert', 'minting', 'srini' ], modifyFlags)
• append_items(ORDERED, [ 'srini', 'psi', 'mark', 'tibor' ], modifyFlags)
• append_items(ORDERED, [ 'matt', 'heidi', 'claudia', 'brianb' ], modifyFlags)
• get_by_index_range(VALUE, 0)
['brianb', 'claudia', 'heidi', 'mark', 'matt', 'michaelz', 'minting', 'psi',
'robert', 'srini', 'tibor']
Unique Contacts using Ordered Lists

Starting Aerospike Database 4.3.1, lists are compared as follows
1. [1, 1] < [1, 3] – Compare the entries in order.
2. [1, 2] < [1, 2, 0, 1] – Length is the secondary criteria.
So [1, 1] < [1, 2] < [1, 2, 0, 1] < [1, 3]
This can be used by these List API methods
• get_by_value_interval( [v1, nil], [v2, nil] )
• get_all_by_value( [v1, *])
• get_all_by_value_list( [v1, *], [v2, *])
* In the C client * is AS_CMP_WILDCARD, in the Python client it’s Aerospike.CDTWildcard(),
etc
How List Values are Compared

[
[1523474230000, 39.04],
[1523474231001, 39.78],
[1523474236006, 40.07],
[1523474235005, 41.18],
[1523474233003, 40.89],
[1523474234004, 40.93]
]
get_all_by_value_interval(VALUE, [1523474231000, nil], [1523474234000, nil])
[
[1523474231001: 39.78],
[1523474233003: 40.89]
]
Lists as Containers for Time Series Data

[
[1523474230000, 39.04],
[1523474231001, 39.78],
[1523474236006, 40.07],
[1523474235005, 41.18],
[1523474233003, 40.89],
[1523474234004, 40.93]
]
Insert a list of [timestamp, price] tuples; remove all but the most recent 100 elements; return the size.
ops = [
append_items([ [1523474237007, 40.11], [1523474238008, 40.26], .. ], UNORDERED)
remove_by_rank_range(NONE | INVERTED, -100, 100)
size()
]
operate(ops)
Lists as Capped Collections

[
[10.03, "Jim Hines", "Sacramento, USA", "June 20, 1968"],
[10.02, "Charles Greene", "Mexico City, Mexico", "October 13, 1968"],
[9.95, "Jim Hines", "Mexico City, Mexico", "October 14, 1968"],
[9.93, "Calvin Smith", "Colorado Springs, USA", "July 3, 1983"],
[9.93, "Carl Lewis", "Rome, Italy", "August 30, 1987"],
[9.92, "Carl Lewis", "Seoul, South Korea", "September 24, 1988"]
]
get_by_value_rel_rank_range(VALUE, [10.0, nil], -1, 2)
[
[9.95, 'Jim Hines', 'Mexico City, Mexico', 'October 14, 1968'],
[10.02, 'Charles Greene', 'Mexico City, Mexico', 'October 13, 1968']
]
Lists as Leaderboards

Map operations supported by the server. Method names in the clients might be different.
set_type() (unordered, k-ordered or kv-ordered)
add(), add_items(), increment(), decrement(), size(), clear()
remove_by_key(), remove_by_index(), remove_by_rank()
remove_by_key_interval(), remove_by_index_range()
remove_by_value_interval(), remove_by_rank_range(), remove_all_by_value()
remove_by_key_rel_index_range(), remove_by_value_rel_rank_range()
remove_all_by_key_list(), remove_all_by_value_list()
get_by_key(), get_by_index(), get_by_rank()
get_by_key_interval(), get_by_index_range()
get_by_value_interval(), get_by_rank_range(), get_all_by_value()
get_by_key_rel_index_range(), get_by_value_rel_rank_range()
get_all_by_key_list(), get_all_by_value_list()
Map Operations – New Additions

{
1523474230000: ['fav', {'sku':1, 'b':2}],
1523474231001: ['comment', {'sku':2, 'b':22}],
1523474236006: ['viewed', {'foo':'bar', 'sku':3, 'zz':'top'}],
1523474235005: ['comment', {'sku':1, 'c':1234}],
1523474233003: ['viewed', {'sku':3, 'z':26}],
1523474234004: ['viewed', {'sku':1, 'ff':'hhhl'}]
}
get_key_rel_index_range(KEYVALUE, 1523474232000, -1, 2)
{
1523474231001: ['comment', {'b': 22, 'sku': 2}],
1523474233003: ['viewed', {'sku': 3, 'z': 26}]
}
Maps as Containers for Events – Timestamp Keys

{
'0edf5b73-535c-4be7-b653-c0513dc79fb4': [1523474230, "Billie Jean is not my lover", "MJ"],
'29342a0b-e20f-4676-9ecf-dfdf02ef6683': [1523474241, "She's just a girl who", "MJ"],
'31a8ba1b-8415-aab7-0ecc-56ee659f0a83': [1523474245, "claims that I am the one", "MJ"],
'9f54b4f8-992e-427f-9fb3-e63348cd6ac9': [1523474249, "...", "Tito"],
'1ae56b18-7a3c-4f64-adb7-2e845eb5094e': [1523474257, "But the kid is not my son", "MJ"],
'08785e96-eb1b-4a74-a767-7b56e8f13ea9': [1523474306, "ok...", "Tito"],
'319fa1a6-0640-4354-a426-10c4d3459f0a': [1523474316, "Hee-hee!", "MJ"]
}
get_by_value_rel_rank_range(VALUE, [1523474235, nil], -1, 2)
{
[1523474230, 'Billie Jean is not my lover', 'MJ'],
[1523474241, "She's just a girl who", 'MJ']
}
Maps as Containers for Events – Timestamp Values

 We talked about Aerospike’s data types, and how they can be used for modeling
 We looked more closely at the List API
 We modeled several collection types with lists
 Relative key-index range and relative value-rank methods are really useful!
What next?
 If you haven’t seen it, take a look at the slides from the first Israeli ASUG meetup
 Go to GitHub; clone the code samples repo; run it; read the code
 Read the Aerospike blog. Get familiar with all the database features
 Participate in the community forum (https://discuss.aerospike.com), StackOverflow’s
aerospike tag
Summary

List & Map API
 https://www.aerospike.com/docs/guide/cdt-list.html
 https://www.aerospike.com/docs/guide/cdt-ordering.html
 https://www.aerospike.com/docs/guide/cdt-list-ops.html
 https://aerospike-python-
client.readthedocs.io/en/latest/aerospike_helpers.operations.html
 https://www.aerospike.com/apidocs/java/com/aerospike/client/cdt/ListOperation.html
 https://www.aerospike.com/docs/guide/cdt-map.html
Code Samples
 https://github.com/rbotzer/aerospike-cdt-examples
Aerospike Training
 https://www.aerospike.com/training/
More material you can explore:

Thank You!
Any questions?
ronen@aerospike.com

Exploring Modeling - Doing More with Lists

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Exploring Modeling - Doing More with Lists

Similar to Exploring Modeling - Doing More with Lists (20)

Recently uploaded

Recently uploaded (20)

Exploring Modeling - Doing More with Lists

Editor's Notes