Introduction to chronicle (low latency persistence)

4,928 views

Published on

Chronicle is a low latency persistence and IPC library.

Published in: Technology, Business

Introduction to chronicle (low latency persistence)

  1. 1. (c) Higher Frequency Trading Who am I?  Australian living in UK. Three kids 5, 8 and 15  Five years designing, developing and supporting HFT systems in Java  My blog, “Vanilla Java” gets 120K page views per month.  3rd for Java on StackOverflow  Lead developer for OpenHFT which includes Chronicle and Thread Affinity.
  2. 2. (c) Higher Frequency Trading Introduction to Chronicle Micro second latency persistence Higher Frequency Trading (c) Peter Lawrey
  3. 3. (c) Higher Frequency Trading What is Chronicle? Very fast embedded persistence for Java. Functionality is simple and low level by design
  4. 4. (c) Higher Frequency Trading Where does Chronicle come from  Low latency, high frequency trading – Applications which are sub 100 micro-second external to the system.
  5. 5. (c) Higher Frequency Trading Where does Chronicle come from  High throughput trading systems – Hundreds of thousand of events per second
  6. 6. (c) Higher Frequency Trading Where does Chronicle come from  Modes of use – GC free – Lock-less – Shared memory – Text or binary – Replicated over TCP – Supports thread affinity
  7. 7. (c) Higher Frequency Trading Is there a free version?  It is open source and free with an Apache 2.0 license.  You can pay for training and consulting
  8. 8. (c) Higher Frequency Trading Use for Chronicle  Synchronous text logging  Synchronous binary data logging
  9. 9. (c) Higher Frequency Trading Use for Chronicle  Messaging between processes via shared memory  Messaging across systems
  10. 10. (c) Higher Frequency Trading Use for Chronicle  Supports recording micro-second timestamps across the systems  Replay for production data in test
  11. 11. (c) Higher Frequency Trading Writing to Chronicle IndexedChronicle ic = new IndexedChronicle(basePath); Excerpt excerpt = ic.createExcerpt(); for (int i = 1; i <= runs; i++) { excerpt.startExcerpt(17); excerpt.writeUnsignedByte('M'); // message type excerpt.writeLong(i); // e.g. time stamp excerpt.writeDouble(i); excerpt.finish(); } ic.close();
  12. 12. (c) Higher Frequency Trading Reading from Chronicle IndexedChronicle ic = new IndexedChronicle(basePath); ic.useUnsafe(true); // for benchmarks Excerpt excerpt = ic.createExcerpt(); for (int i = 1; i <= runs; i++) { while (!excerpt.nextIndex()) { // busy wait } char ch = (char) excerpt.readUnsignedByte(); long l = excerpt.readLong(); double d = excerpt.readDouble(); assert ch == 'M'; assert l == i; assert d == i; excerpt.finish(); } ic.close();
  13. 13. (c) Higher Frequency Trading How does it perform With one thread writing and another reading * Chronicle 2.0 -XX:+AggressiveOpts -Xmx32m Tiny 4 B Small 16 B Medium 64 B Large 256 B tmpfs 77 M/s 57 M/s 23 M/s 6.6 M/s ext4 65 M/s 35 M/s 12 M/s 3.2 M/s
  14. 14. (c) Higher Frequency Trading How does it recover? Once finish() returns, the OS will do the rest. If an excerpt is incomplete, it will be pruned.
  15. 15. (c) Higher Frequency Trading Cache friendly Data is laid out continuously, naturally packed. You can compress some types. One entry starts in the next byte to the previous one.
  16. 16. (c) Higher Frequency Trading Consumer insensitive No matter how slow the consumer is, the producer never has to wait. It never needs to clean messages before publishing (as a ring buffer does) You can start a consumer at the end of the day e.g. for reporting. The consumer can be more than the main memory size behind the producer as a Chronicle is not limited by main memory.
  17. 17. (c) Higher Frequency Trading How does it collect garbage? There is an assumption that your application has a daily or weekly maintenance cycle. This is implemented by closing the files and creating new ones. i.e. the whole lot is moved, compressed or deleted. Anything which must be retained can be copied to the new Chronicle
  18. 18. (c) Higher Frequency Trading Is there a lower level API? Chronicle 2.0 is based on OpenHFT Java Lang library which supports access to 64-bit native memory. ● Has long size and offsets. ● Support serialization and deserialization ● Thread safe access including locking
  19. 19. (c) Higher Frequency Trading Is there a higher level API? You can hide the low level details with an interface.
  20. 20. (c) Higher Frequency Trading Is there a higher level API? There is a demo program with a simple interface. This models a “hub” process which take in events, processes them and publishes results.
  21. 21. (c) Higher Frequency Trading Is there a higher level API? The interfaces look like this public interface Gw2PeEvents { public void small(MetaData metaData, SmallCommand command); } public interface Pe2GwEvents { public void report(MetaData metaData, SmallReport smallReport); }
  22. 22. (c) Higher Frequency Trading What does Chronicle need More documentation More tutorials More use case examples
  23. 23. (c) Higher Frequency Trading What is planned Lower overhead per message 80 ns → 15 ns. Low latency XML and FIX parser and writer Central Order Book engine.
  24. 24. (c) Higher Frequency Trading Q & A

×