Calum McCrea, Software Engineer at Kx Systems speaking at Big Data, Paris v 6.0 on April 27, 2017 "Kx: How Wall Street Tech can Speed up the World"

1. Kx: How Wall Street Tech can
Speed up the World
Calum McCrea
April 2017

2. 2
Kx Technology
• Integrated columnar database & programming system
• Streaming, real time and historical data
• Built for massive data volumes
• In-database analytics
• Parallelism
• Compression
What is Kx Technology
Key Features
WHY Kx TECHNOLOGY?
HIGH PERFORMANCE, LOW LATENCY
• We are fast not only due to data architecture, our native
programming language, q, runs inside the database not in
separate processes with costly data passing.
POWER
• We are one of the few fully 64 bit databases and unique
in having time as a native type, with nanosecond
resolution and a full set of operations over time.
QUICK TO DELIVER
• Unlike many compile-link and run approaches q is
dynamic allowing much shorter development and
deployment cycles
• Python
• MATLAB
• R
• Java
• C/C++
• .NET
Use your preferred programming language
Kx is the world’s leading time series database. Constantly surpassing industry standards

3. 3
CLIENTS

4. 4
APL Example – Prime Numbers to R
(~T∊T∘.×T)/T←1↓⍳R
• Executed from right to left

5. 5
Arthur Whitney & The Development of q
• 1988: A+ Created
• 1993: k created and Kx Systems established
• 1998: kdb+ introduced
• 2003: q as we know it today

6. 6
• World’s fastest time-series database
• Runs on Linux, Windows, Solaris, and OS X
• Runs on commodity hardware
• Expressive query & programming language (q)
• In-memory compute engine for Complex Event Processing
• Multi-core / Multi-processor / Multi-thread / Multi-server
Kx Time-Series Database
Proven database technology used by hundreds of organizations
who require high performance time-series solutions

7. 7
Why
• Performance, light-weight, flexible, scalable
• Independent benchmarking
• Throughput/CEP
• Data Response Time SLA
• Maintenance
• Infrastructure Arbitrage
• Tiny footprint (~500kb)

8. 8
Performance Comparison
8
Tests were run on the NYSE taq dataset
(1.1. trillion rows)
Time to query for smallest day trades.
2003.09.10 35M quotes. 5M trades.
Machine: 16 Core 256GB. Date Partitioned
& Symbol Indexed.
Query 1 Query 1 Query 3 Query 4 Disk (TB) RAM (MB)
Kdb+ / q 18 12 36 13 25 200
big3rdbms 6,400 2,200 3,100 DNF 50 5,000
big3accel 4,200 1,600 2,300 DNF 26 3,400
postgres 6,800 5,200 4,800 DNF 150 100
cloudera 4,800 1,190 1,000 DNF 6 4,000
mongodb 8,900 1,700 5,800 N/A 245 9,000
spark 34,000 7,400 8,400 DNF 60 50,000
Performance Results …
Industry Speed Trial

9. 9
Analytics In One Place
9
• Kx can sit at any level of the tech stack
• Flexible to build on top of multiple types of
databases including Teradata, SAS, Oracle etc
• Designed to connect insight and bring analytics in to one
place
• Modular to build tools and user interfaces across any
data source
• Powerful predictive analytics at every level with easy
to interpret outputs
• Machine Learning capability to optimize models using
real-time data feeds
• Powerful data blending technology for internal and
external data
• Beautiful visualizations to bring data to life
• Can integrate with existing BI tools such as Tableau or
PowerBI

10. 10
Financial Markets – Key Functionality
• All asset classes supported
• Level 1 and Level 2 data
• Dynamic orderbook snapshots and rebuild /
replay
• Historical symbology management
• Symbology mapping (e.g. BBG ID->Ric)
• Corporate action adjustment to the tick level
• Condition code management and filtering
• Analytical (calculated) filtering on any column
• Dynamic bucketing, racking and filling at any
level
• Cancellation & corrections
• Timezone management
• Data permissioning per user to the instrument
level
• Nanosecond timestamps
• Seemless historical and realtime data access
• Streaming and snapshot analytics

11. 11
How does it work?
Client A Client B Client n
Gateway
Kdb+ Based Backend
Gateway
Feed
handler
Feed
handler
Feed
handler
Data
Vendor A
Data
Vendor B
Data
Vendor n
1
1
Standard client / vendor connectivity layer for data ingress
KX dev team to establish connectivity to each provider
Market data vendor stack provides exchange connectivity & data
Realtime data publication to core platform
Data consumption, persistence and client requests handled here
Asynchronous parallel processing of client requests
Secure single entry point for any and all client requests
Async TCP based connectivity for simplicity, security
& performance
System designed to scale horizontally to support any number
of clients and data vendors

12. 12
Similar Patterns but Faster….
TickerPlant
• Observer Pattern
• Pub/Sub Relationship

13. 13
Adding security
Gateway Structure
• Entitled users
• Limited Databases
• Separation of Frequently Queried
Database

14. 14
Live demo of Ticker Plant

15. 15
Live demo of Ticker Plant
The heat map shows the properties which are not responding. Then one can reviews todays latest
alerts to view a detailed description for why it is not reporting.

16. 16
Satellite Data – Dashboards

17. 17
Something a little different

18. 18
Company Overview
Group
People
1,700 FTEs
Office Locations
Ireland, London, New York,
Singapore, Palo Alto,
Sydney, Tokyo, Hong Kong,
Toronto, Zurich
History
Founded in 1996
Public listed Company (LSE)
Always Profitable
Division
Kx
Big Fast Data
Capital Markets Consulting
Domain experts
Technology experts
Prelytix
Digital marketing
Product
Kx
Streaming Analytics, In
Memory Computing, - the
market standard
Tooling for Kx
Develop, Deploy, Visualize
Industry Solutions
Powered by Kx

19. 19
OTHER VERTICALS
Health & Life Sciences
Automotive
Utilities
Space and Telco
Retail
Earth Observation
Geospatial Data Analytics
Anomaly Detection
Genomics Data Processing
Connected Health
Patient Record Analytics
Telemetry Analytics
Anomaly Detection
Performance Analytics
Sensor Analytics
Smart Meters Data
Predictive Analytics
Manufacturing
Retail Analytics
Marketing Optimization
Customer Journey
Edge Computing
Multivariate Analysis
Fault Detection

20. 20
Kx TECHNOLOGY FUND
Int. Office
Access
Up To
£10 Mil
Funding
Kx Tech
Access
Sales &
Marketing
Support
Logistical
Support
Learn more at www.kx.com

21. 21
RECRUITMENT
o Opportunities to travel & live
internationally
o Paid for city accommodation
o Competitive salary and expenses
package
o Health & travel insurance
programme
o Generous retirement savings
programme

22. 22
• http://code.kx.com/wiki/Reference  Wiki page
• https://groups.google.com/forum/#!forum/personal-kdbplus  forums
• http://stackoverflow.com/questions/tagged/kdb
• https://kx.com/download/  Free 32 bit version
Appendices

23. 23
Thank you!
Calum Mccrea
Software Engineer
cmccrea@kx.com
+44-7512561112
Contact and Closing Information