The document discusses the company's utilization of AWS services, particularly Amazon Kinesis, for real-time financial market data processing in Munich. It outlines the challenges faced with various database solutions and the strategic implementation of Kinesis for data ingestion and analytics. The presentation includes technical insights into their architecture and future plans to enhance user experience and performance.

1. AWS Pop Up Loft Munich
Analyzing and processing
Financial Market data on AWS
with Kinesis
Florian Benz
Engineering Team Lead
November 14, 2019

2. ■ Founded in 2014
■ 115 Employees
■ Managing 1.8 billion Euro
■ Developed in Munich
Our Company.

3. Our product.

4. International B2C + B2B Business.

5. 5
AWS - Going Serverless.

6. Real-Time Market Data.

7. Real-Time Market Data.
½ MiB/s
Show real time
24 GiB/day
Store all data

8. {
"isin":"DE0007480204",
"currency": "EUR",
"bid": 26.96,
"ask": 27.2,
"time":"2019-11-07T17:18:51Z"
...
}
Real-Time Market Data.

9. Our Journey.

10. Aurora TimescaleDB DynamoDB BigTableTimesteam

11. ● Looks like a good solution
● Data organized in time intervals
● Not available yet
Timesteam

12. ● PostgreSQL
● Handled the load w/ high resource usage
● Performance degraded over time
Aurora

13. ● Managed service available
● Better than pure PostgreSQL for time-series data
● High resource usage
TimescaleDB

14. ● Couldn’t handle the load
● Not easy to design for
● High costs for our use case
DynamoDB

15. ● Advertised for market data
● Overhead due to connecting Clouds
● Minimal production setup is rather big
BigTable

16. Separate Processing
and Storage.

17. Separate Processing
and Storage.
Streams?

18. Separate Processing
and Storage.
Streams!
Amazon Kinesis

19. Ingest data as
it’s generated
Process data
on the fly
Real-time
analytics
Amazon Kinesis.

20. Data Streams
Capture, process and
store data streams
Data Firehose
Load data streams
into data stores
Amazon Kinesis.

21. Producer
● 1 MiB/s/shard or
● 1,000 PUT records/s/shard
Consumer
● 2 MiB/s/shard
● 5 GetRecords/s/shard
Amazon Kinesis Data Streams.

22. Consumer - Enhanced Fan-Out
● Subscribe via HTTP/2
● No limitations on the consumers
● Low latency
Amazon Kinesis Data Streams.

23. Our Kinesis Setup.

25. Data Stream Firehose
Kinesis in Action.

26. Producer
● Kotlin w/ Spring Boot
● Kinesis Producer Library (KPL)
Processor
● Kotlin
Infrastructure
● Terraform
Choices around Kinesis.

27. KPL aggregates
● Daemon
● RecordMaxBufferedTime, default = 100ms
● Firehose deaggregates by default
● Lambda has to deaggregate
AWS SDK vs Kinesis Producer Library (KPL).

28. Issues.

29. ● Written in C++
● Child process of the main process
● Java release with precompiled binaries
Issue - KPL Language Support.

30. Issue - Processing Performance.

31. Issue - Terraform Support.

32. Evolution.

33. ● Stream data to all clients
● Use data to improve UX
Awesome UI Components.

34. ● Try to reduce latency further
● Reduce configuration options
Kinesis Enhanced Fan-Out.

35. ● Optimize data on S3 for Athena
● Employ Athena to get insights
Amazon Athena.

36. Questions ?
Andreas Schranzhofer, CTO
andreas@scalable.capital
@Schranzhofer