In 2014 we had to do a major overhaul of ArangoDB's database engine,because we wanted to introduce a write-ahead log. Since for a database this change is similar in nature to the proverbial open-heart surgery for humans, it was clear from day one that this would be a difficult endeavour with a lot of risk to break things. Rather fundamental changes were needed in nearly all places of the kernel code and it seemedimpossible to serialise the work to keep the system in a working state. As usual, time was at a premium, since the next major release had to go out of the door in 2 months time. In this talk I will tell the story of this overhaul, explain the role of unit tests and continuous integration and describe the challenges we faced and how finally overcame them.

1. Overhauling a database
engine in 2 months
Max Neunhöffer
Move Fast and Break Things, 12 March 2015
www.arangodb.com

2. Max Neunhöffer
I am a mathematician
“Earlier life”: Research in Computer Algebra
(Computational Group Theory)
Always juggled with big data
Now: working in database development, NoSQL, ArangoDB
I like:
research,
hacking,
teaching,
tickling the highest performance out of computer systems.
1

3. ArangoDB GmbH
triAGENS GmbH offers consulting services since 2004:
software architecture
project management
software development
business analysis
a lot of experience with specialised database systems.
have done NoSQL, before the term was coined at all
2011/2012, an idea emerged:
to build the database one had wished to have all those years!
development of ArangoDB as open source software since 2012
ArangoDB GmbH: spin-off to take care of ArangoDB (2014)
2

4. is a multi-model database (document store & graph database),
is open source and free (Apache 2 license),
offers convenient queries (via HTTP/REST and AQL),
including joins between different collections,
configurable consistency guarantees using transactions
is memory efficient by shape detection,
uses JavaScript throughout (Google’s V8 built into server),
API extensible by JS code in the Foxx Microservice Framework,
offers many drivers for a wide range of languages,
is easy to use with web front end and good documentation,
and enjoys good community as well as professional support.
3

5. Architecture
DB Engine
Transactions Sharding
Cluster
Infrastructure
V8
JavaScript
libev
zliblibICU
Unicode
Lib: TCP server, HTTP, JSON,
OS−dep., V8 helpers
API
CRUD
high level API in JavaScript Foxx
etcd
(Go)
Client
Scheduler
RequestsHTTP
Dispatcher
HTTP Responses
4

6. ArangoDB in numbers
DB engine written in C++
embeds Google’s V8 (∼ 130 000 lines of code)
mostly in memory, using memory mapped files
processes JSON data, schema-less but “shapes”
library: ∼ 128 000 lines (C++)
DB engine: ∼ 210 000 lines (C++, including 12 000 for utilities)
JavaScript layer: ∼ 1 232 000 lines of code
∼ 85 000 standard API implementation
∼ 592 000 Foxx apps (API extensions, web front end)
∼ 298 000 unit tests
∼ 327 000 node.js modules
further unit tests: ∼ 10 000 C++ and ∼ 24 000 Ruby for HTTP
plus documentation
and drivers (in other repositories)
5

7. The Task
It is March 2014, we have just released V2.0.
V2.1 is scheduled for end of May, V2.2 is scheduled for July
V2.1 is incremental, V2.2 is “Write-Ahead-Log”
work for V2.2 started in March
unfortunately, introducing a WAL is akin to open heart surgery,
→ essentially need to reengineer the database engine
do not have the capacity to assign 10 developers to the job
6

8. The old setup
New data:
{ name: "watch", price: 99 }
Data files:
Collection: products
Collection: sales
(append only)
(append only)
For transactions:
Locks and commit markers on all collections necessary.
7

9. The new setup
"Collector" (later)
Collection: sales
Collection: products (append only)
(append only)
Data files:
New data:
Write Ahead Log (WAL) (append only)
{ name: "watch", price: 99 }
For transactions:
Less locks and commit markers only in WAL.
8

10. Advantages of a WAL
have a single history of events
have a single place to note the commit of a transaction
easy asynchronous replication
efficient sync to disk
uncommitted stuff does not hit the data files at all
better support for transactions → much higher performance
better crash recovery
deterministic, well defined behaviour
9

11. Challenges
need fundamental change in the storage engine
the collector changes stuff that is potentially being read
need to be careful not to create a bottleneck
need to get locking right
crashes hard to test
if possible, users must not notice the change
(except better performance)
10

12. Our testing setup
We do continuous tests after every push to github and nightly.
We have separate test suites for single server and cluster.
Different types of test for good coverage:
low-level C++ library unit tests (10000 LOC C++)
JS tests (separately on server and JS shell, 290000 LOC)
AQL query engine (230000 LOC of the above)
HTTP interface (TCP and SSL, 24000 LOC Ruby)
dump/restore and bulk import
benchmarks (3000 LOC C++)
user interface (phantomjs, comparing screen shots)
run tests with valgrind
check coverage
11

13. Test methodologies
Tests can have different aims:
Unit tests:
ensure that individual components work according to
specifications
Integration tests:
ensure that multiple components work together correctly
Benchmark tests:
ensure performance
End to End tests:
ensure that complex systems as a whole do their job
User interface tests:
ensure that the user interface works and behaves as
documented/specified
12

14. Test characteristics needed for our task
Characteristics of our task
well-defined, deterministic behaviour
if possible, no observable change in functionality
changes relatively far down in the software stack
but reach wide
temporary breakage expected and accepted
=⇒ For our task, we needed:
unit tests,
integration tests and
benchmarks.
Fortunately, we had all these in place!
13

15. Approach — preparation phase
1. design work:
WAL, markers, collection, compaction, where lock what
2. implement infrastructure for WAL:
mmapped files, append op, marker format
3. add “write to WAL” to write operations
4. test filling of WAL
14

16. Approach — breaking phase
4. remove old write operations
5. implement collector thread and adjust compactor thread
6. repair by adjusting read operations for WAL/data file
15

17. Approach — repairing phase
7. fix transaction management
8. implement startup with non-empty WAL: crash recovery
9. fix/simplify replication
10. fix dump/restore
11. fix cluster
12. tune performance (legends)
13. Hurray, tests work again!
14. release beta version
15. fix, fix, fix and tune
16. publish V2.2
16