Доклад посвящен архитектуре Tarantool и ее эволюции. Кирилл рассказал, почему важно располагать базу данных и сервер приложений в одном адресном пространстве, почему Tarantool сделали однопоточным и зачем базе-в-памяти нужен механизм хранения данных на диске. Затем Кирилл рассказал про последние наработки команды создателей Tarantool: зачем мы добавили синтаксис SQL и как это может решить ваши задачи.

1. TarantoolTarantool
A no-SQL DBMS nowA no-SQL DBMS now
with SQLwith SQL
Kirill Yukhin
1

2. 0 Hello
Hello, my name is Kirill Yukhin and I am working on Tarantool database.
I totally understand, that this is last talk of second day and you all
are pretty much tired, so will try not to overload you!
Speaker notes

3. AgendaAgenda
What is Tarantool?
Performance
Storage engines
SQL
Plans
2

4. 1. Agenda
Today I am going to talk about Tarantool, what is it? Where it came from?
Its most notable features. And finally why we decided to implement support
of SQL in this no-SQL DBMS and where we're at the moment.
Speaker notes

5. HistoryHistory
Was born @ Mail.ru group
Used to store sessions/profiles of Ms of users
Web servers 4 instances
8 instances
3
load web-page
AJAX request
mobile API
> 1.000.000 requests per second
sessions
profiles

6. 2. Intro
Anybody heard about Tarantool? Please, rise your hands.
Tarantool was initially created as in-Memory database to store user profiles
at Mail.ru. This is one of biggest internet companies in Russia.
What is a DB storing user profiles? Well, not that complicated. It is actually
one huge table, with great number of sparse columns. Developers adding new
Speaker notes

7. Must-have and mustn't-haveMust-have and mustn't-have
No secondary keys, constraints etc.
Schema-less
Need a language. *QL is not must-have
High-speed in any sense!
Simple
Extensible
Stored functions
Transactions
Persistency
Once again: it must be fast, no excuses
4

8. This leads us to:
a. We do not need declarative language like SQL. Why? Beacuse data is straight
no FKs no relations, no nothing.
b. We need to have some language which is: must be ultra-fast (we like to have
a jit for free), very simple (web developers don't like to bother too much), and
easy to embed since we don't like to bother too much too :)
Speaker notes

9. Tarantool: Bird's Eye ViewTarantool: Bird's Eye View
No need for cache
It is in-memory
But still DBMS: persistency and transactions
It regards ACID
Single threaded
It is lock-free
Easy: imperative language is on board: Lua
It JITs
It's easy to program business
It scales
Replication and sharding
5

10. Variant 1.
What do we have at the end of the day?
- Well, if we want to be ultra-fast then we probably will live in RAM. So, we're probably sit in-memory
- We are must be responsive: I'll talk about this later. But if we don't use locking this probably will be good
- No *QL, but Lua w/ luajit. It is fast, extensible, easy to embed. And it is imperative: programming is easy. Even
sharding might be implemented in it and it was. Try to do that in PL/SQL!
Speaker notes

11. DBMS + Application Server
C, Lua, SQL, Python, PHP, Go, Java, C# ...
Queries
handling WAL Network
Process
Threads
Persistent in-memory and disk storage engines
Stored procedures in C, Lua, SQL
6

12. So, what is Tarantol?
Tarantool is DBMS and application server sitting in the
same process. It can be used like DBMS, create tables (which are called spaces),
connect to it using network, performing requests etc. But the true power is unleashed
when buiseness logic sits inside Tarantool. Same process! Same address space with
data!
Speaker notes

13. Coöperative multitasking
Multithreading
7
Fibers
Event-loop

14. Coöperative multitasking
Multithreading That is
a stall
Losses on caches coherency support
Losses on locks
Losses on long operations
7
Fibers
Event-loop

15. Coöperative multitasking
Multithreading That is
a stall
Losses on caches coherency support
Losses on locks
Losses on long operations
7
Fibers
Event-loop
Thread is always busy
Lock-free
Single core - no coherency issues at all

16. 6. (*12.) Few more words on performance
One of killer-features of Tarantool is cooperative multitasking. Anybody knows
what is that? This is mechanism which allows inside a single thread, which serves
clients requests, lots of co-routines to execute. Each has its own stack and you
can run thousands of such cotoutines. They're called fibers in Tarantool.
Again, this is alternative to multi-threading, but:
Speaker notes

17. VinylVinyl
In-memory is OK, but not always enough
Write-oriented: LSM tree
Same API as memtx
Transactions, secondary keys
tarantool> s = box.schema.space.create('space55',
engine='vinyl')
8

18. Variant 1.
7. Vinyl
This is yet another feature of Tarantool which I'll cover briefly. We'll submit a
talk on Percona Europe this fall, so just a couple of words.
Did you heard about Justin Biber's problem? This is pretty much common issue in
internet services. Suppose you host Justin Biber's FB account, and he just
Speaker notes

19. Why SQL?Why SQL?
SQL> SELECT DISTINCT(a)
FROM t1, t2
WHERE t1.id = t2.id
AND t2.y > 1;
CREATE TABLE t1 (id INTEGER PRIMARY KEY, a INTEGER, b INTEGER, c INTEGER)
CREATE TABLE t2 (id INTEGER PRIMARY KEY, x INTEGER, y INTEGER, z INTEGER)
9

20. So, after all that said. Why do we decided to implement SQL?
Here's motivating example. Let's create two spaces or table in SQL sense.
Then, having Lua language alone, let's try to re-implement this SQL snippet.
Those of you, who knows Lua starts to understand the issue I suppose.
Speaker notes

21. Why SQL?Why SQL?
CREATE TABLE t1 (id INTEGER PRIMARY KEY, a INTEGER, b INTEGER, c INTEGER)
CREATE TABLE t2 (id INTEGER PRIMARY KEY, x INTEGER, y INTEGER, z INTEGER)
function query()
local join = {}
for _, v1 in box.space.t1:pairs({}, {iterator='ALL'}) do
local v2 = box.space.t2:get(v1[1])
if v2[3] > 1 then
table.insert(join, {t1=v1, t2=v2})
end
end
local dist = {}
for _, v in pairs(join) do
if dist[v['t1'][2]] == nil then
dist[v['t1'][2]] = 1
end
end
local result = {}
for k, _ in pairs(dist) do
table.insert(result, k)
end
return result
end
query()
10

22. I've created a separate slide for Lua implementation since it is too big :)
Even if we skip issue that there're lots of lines actually error might hide
There're other pretty serious issues: like memory for `join` table.
Those of you who touched laujit probably know, that there're certain limitations
to memory available. I am not talking about GC which might possibly
step in during query processing.
Speaker notes

23. SQL FeaturesSQL Features
Trying to be subset of ANSI
Minimum overhead of query planner
ACID transactions, SAVEPOINTs
left/inner/natural JOIN, UNION/EXCEPT,
subqueries
HAVING, GROUP BY, ORDER BY
WITH RECURSIVE
Triggers
Views
Constraints
Collations
11

24. Here's a boring list of what's actually available in Tarantool's SQL implementation
Speaker notes

25. SQL v. Single-threaded TxSQL v. Single-threaded Tx
12

26. SQL v. Single-threaded TxSQL v. Single-threaded Tx
OLTP
12

27. SQL v. Single-threaded TxSQL v. Single-threaded Tx
OLTP
OLAP
12

28. SQL v. Single-threaded TxSQL v. Single-threaded Tx
OLTP
OLAP
M
12

29. SQL v. Single-threaded TxSQL v. Single-threaded Tx
OLTP
OLAP
M
S
12

30. SQL v. Single-threaded TxSQL v. Single-threaded Tx
OLTP
OLAP
M
S
12
OLTP

31. SQL v. Single-threaded TxSQL v. Single-threaded Tx
OLTP
OLAP
M
S
12
OLTP
OLAP

32. PerspectivesPerspectives
Onboard sharding
Synchronous replication
SQL: more types, JIT, query planner
13

33. Sharding
Replication
In-memory
Disk
Persistency
SQL
Stored procedures
Audit logging
Connectors to
DBMSes
Static build
GUI
Unprecedented
performance
Tarantool VShard
Synchronous/Asynchronous
memtx engine
vinyl engine, LSM-tree
Both engines
ANSI
Lua, C, SQL
Yes
MySQL, Oracle,
Memcached
for Linux
Cluster management
100.000 RPS per instance - easy!
14

34. Спасибо!Спасибо!
https://tarantool.io
https://github.com/tarantool/tarantool
15

35. ReplicationReplication
Lag of async replicas
slow
network
Lag
M
aster
16

36. ReplicationReplication
Lag of async replicas
slow
network
Lag
M
aster
Re-send of lost changes
Rejoin
16

37. ReplicationReplication
Lag of async replicas
slow
network
Lag
M
aster
Re-send of lost changes
Rejoin
Complex topologies
16

38. ReplicationReplication
Lag of async replicas
slow
network
Lag
M
aster
Re-send of lost changes
Rejoin
Complex topologies
Support of arbitrary
topologies
16

39. *6.
There're lots of issues and problems with replication. For example, async replica
may lag from master which sends updates if there were network problems. How to
keep up? What master should do with updates which it is unable to send? Should it
accept new updates? Tarantool solves that by storng all updates on master until
they are sent to replicas. If replica is not around for a long time then changes
Speaker notes

40. Multikey & JSON in TarantoolMultikey & JSON in Tarantool
17

41. ShardingSharding
18

42. JSON: A Rich Way to Store DataJSON: A Rich Way to Store Data
Storing data in the JSON format is also a natural way to
store data than in rows and columns
19