This document discusses Chronicle, an open source Java library for very fast embedded persistence designed for applications that require microsecond latency, such as high-frequency trading systems. Chronicle provides lock-free, garbage-collected logging to file or shared memory in a way that is throughput-efficient and allows the producer and consumer to operate independently without waiting on each other. It aims to offer persistence performance better than traditional databases or messaging systems for low-latency applications.

1. (c) Higher Frequency Trading
Who am I?

Australian living in UK. Three kids 5, 8 and 15

Five years designing, developing and supporting HFT
systems in Java

My blog, “Vanilla Java” gets 120K page views per month.

3rd
for Java on StackOverflow

Lead developer for OpenHFT which includes Chronicle
and Thread Affinity.

2. (c) Higher Frequency Trading
Introduction to Chronicle
Micro second latency persistence
Higher Frequency Trading
(c) Peter Lawrey

3. (c) Higher Frequency Trading
What is Chronicle?
Very fast embedded persistence for Java.
Functionality is simple and low level by design

4. (c) Higher Frequency Trading
Where does Chronicle come from

Low latency, high frequency trading
– Applications which are sub 100 micro-second
external to the system.

5. (c) Higher Frequency Trading
Where does Chronicle come from

High throughput trading systems
– Hundreds of thousand of events per second

6. (c) Higher Frequency Trading
Where does Chronicle come from

Modes of use
– GC free
– Lock-less
– Shared memory
– Text or binary
– Replicated over
TCP
– Supports thread
affinity

7. (c) Higher Frequency Trading
Is there a free version?

It is open source and free with an Apache 2.0
license.

You can pay for training and consulting

8. (c) Higher Frequency Trading
Use for Chronicle

Synchronous text logging

Synchronous binary data logging

9. (c) Higher Frequency Trading
Use for Chronicle

Messaging between processes
via shared memory

Messaging across systems

10. (c) Higher Frequency Trading
Use for Chronicle

Supports recording micro-second timestamps
across the systems

Replay for production data in test

11. (c) Higher Frequency Trading
Writing to Chronicle
IndexedChronicle ic = new IndexedChronicle(basePath);
Excerpt excerpt = ic.createExcerpt();
for (int i = 1; i <= runs; i++) {
excerpt.startExcerpt(17);
excerpt.writeUnsignedByte('M'); // message type
excerpt.writeLong(i); // e.g. time stamp
excerpt.writeDouble(i);
excerpt.finish();
}
ic.close();

12. (c) Higher Frequency Trading
Reading from Chronicle
IndexedChronicle ic = new IndexedChronicle(basePath);
ic.useUnsafe(true); // for benchmarks
Excerpt excerpt = ic.createExcerpt();
for (int i = 1; i <= runs; i++) {
while (!excerpt.nextIndex()) {
// busy wait
}
char ch = (char) excerpt.readUnsignedByte();
long l = excerpt.readLong();
double d = excerpt.readDouble();
assert ch == 'M';
assert l == i;
assert d == i;
excerpt.finish();
}
ic.close();

13. (c) Higher Frequency Trading
How does it perform
With one thread writing and another reading
* Chronicle 2.0
-XX:+AggressiveOpts -Xmx32m
Tiny
4 B
Small
16 B
Medium
64 B
Large
256 B
tmpfs 77 M/s 57 M/s 23 M/s 6.6 M/s
ext4 65 M/s 35 M/s 12 M/s 3.2 M/s

14. (c) Higher Frequency Trading
How does it recover?
Once finish()
returns, the OS will do
the rest.
If an excerpt is
incomplete, it will be
pruned.

15. (c) Higher Frequency Trading
Cache friendly
Data is laid out continuously, naturally packed.
You can compress some types. One entry
starts in the next byte to the previous one.

16. (c) Higher Frequency Trading
Consumer insensitive
No matter how slow the consumer is, the
producer never has to wait. It never needs to
clean messages before publishing (as a ring
buffer does)
You can start a consumer at the end of the day
e.g. for reporting. The consumer can be more
than the main memory size behind the
producer as a Chronicle is not limited by main
memory.

17. (c) Higher Frequency Trading
How does it collect garbage?
There is an assumption that your application has a daily
or weekly maintenance cycle.
This is implemented by
closing the files and
creating new ones.
i.e. the whole lot is moved,
compressed or deleted.
Anything which must be
retained can be copied
to the new Chronicle

18. (c) Higher Frequency Trading
Is there a lower level API?
Chronicle 2.0 is based on OpenHFT Java Lang
library which supports access to 64-bit native
memory.
●
Has long size and offsets.
●
Support serialization and deserialization
●
Thread safe access including locking

19. (c) Higher Frequency Trading
Is there a higher level API?
You can hide the low level details with an
interface.

20. (c) Higher Frequency Trading
Is there a higher level API?
There is a demo
program with a
simple interface.
This models a “hub”
process which take in
events, processes
them and publishes
results.

21. (c) Higher Frequency Trading
Is there a higher level API?
The interfaces look like this
public interface Gw2PeEvents {
public void small(MetaData metaData,
SmallCommand command);
}
public interface Pe2GwEvents {
public void report(MetaData metaData,
SmallReport smallReport);
}

22. (c) Higher Frequency Trading
What does Chronicle need
More documentation
More tutorials
More use case
examples

23. (c) Higher Frequency Trading
What is planned
Lower overhead per
message
80 ns → 15 ns.
Low latency XML and
FIX parser and
writer
Central Order Book
engine.

24. (c) Higher Frequency Trading
Q & A