Apache Superset , apache druid , dataworks summit 19, Apache nifi , apache kafka, apache superset

1. Developing High Frequency
Indicators Using Real-Time Tick Data
on Apache Superset and Druid
CBRT Big Data Team
Emre Tokel, Kerem Başol, M. Yağmur Şahin
Zekeriya Besiroglu / Komtas Bilgi Yonetimi
21 March 2019 Barcelona

2. Agenda
WHO WE ARE
CBRT & Our Team
PROJECT DETAILS
Before, Test Cluster,
Phase 1-2-3, Prod
Migration
HIGH FREQUENCY
INDICATORS
Importance & Goals
CURRENT ARCHITECTURE
Apache Kafka, Spark,
Druid & Superset
WORK IN
PROGRESS
Further analyses
FUTURE PLANS
6
5
4
3
2
1

3. Who We Are
1

4. Our Solutions
Data Management
• Data Governance Solutions
• Next Generation Analytics
• 360 Engagement
• Data Security
Analytics
• Data Warehouse Solutions
• Customer Journey Analytics
• Advanced Marketing Analytics Solutions
• Industry-specific analytic use cases
• Online Customer Data Platform
• IoT Analytics
• Analytic Lab Solution
Big Data & AI
• Big Data & AI Advisory Services
• Big Data & AI Accelerators
• Data Lake Foundation
• EDW Optimization / Offloading
• Big Data Ingestion and Governance
• AI Implementation – Chatbot
• AI Implementation – Image Recognition
Security Analytics
• Security Analytic Advisory Services
• Integrated Law Enforcement Solutions
• Cyber Security Solutions
• Fraud Analytics Solutions
• Governance, Risk & Compliance Solutions

5. • +20 IT , +18 DB&DWH
• +7 BIG DATA
• Lead Archtitect &Big Data /Analytics
@KOMTAS
• Instructor&Consultant
• ITU,MEF,Şehir Uni. BigData Instr.
• Certified R programmer
• Certified Hadoop Administrator

6. Our Organization
§ The Central Bank of the Republic of Turkey is primarily responsible for steering the
monetary and exchange rate policies in Turkey.
o Price stability
o Financial stability
o Exchange rate regime
o The privilege of printing and issuing banknotes
o Payment systems

7. • Big Data Engineer• Big Data Engineer
M. Yağmur Şahin Emre Tokel Kerem Başol
• Big Data Team Leader

8. High Frequency
Indicators
2
1

9. Importance and Goals
§ To observe foreign exchange markets in real-time
o Are there any patterns regarding to specific time intervals during the day?
o Is there anything to observe before/after local working hours throughout the whole day?
o What does the difference between bid/ask prices tell us?
§ To be able to detect risks and take necessary policy measures in a timely manner
o Developing liquidity and risk indicators based real-time tick data
o Visualizing observations for decision makers in real-time
o Finally, discovering possible intraday seasonality
§ Wouldn’t it be great to be able to correlate with news flow as well?

10. Project Details 3
2
1

11. Development of High Frequency Indicators Using Real-Time Tick
Data on Apache Superset and Druid
Phase 1
Prod
migratio
n
Next
phases
Test
Cluster
Phase 2 Phase 3

12. Test Cluster
§ Our first studies on big data have started on very humble servers
o 5 servers with 32 GB RAM for each
o 3 TB storage
§ HDP 2.6.0.3 installed
o Not the latest version back then
§ Technical difficulties
o Performance problems
o Apache Druid indexing
o Apache Superset maturity

13. Development of High Frequency Indicators Using Real-Time Tick
Data on Apache Superset and Druid
Phase 1
Prod
migratio
n
Next
phases
Test
Cluster
Phase 2 Phase 3

14. TREP API
Apache
Kafka
Apache NiFi MongoDB
Apache
Zeppelin &
Power BI

15. Thomson Reuters Enterprise Platform (TREP)
§ Thomson Reuters provides its subscribers with an enterprise platform that they can
collect the market data as it is generated
§ Each financial instrument on TREP has a unique code called RIC
§ The event queue implemented by the platform can be consumed with the provided
Java SDK
§ We developed a Java application for consuming this event queue to collect tick-data
according to required RICs

16. TREP API
Apache
Kafka
Apache NiFi MongoDB
Apache
Zeppelin &
Power BI

17. Apache Kafka
§ The data flow is very fast and quite dense
o We published the messages containing tick data collected by our Java application to a message
queue
o Twofold analysis: Batch and real-time
§ We decided to use Apache Kafka residing on our test big data cluster
§ We created a topic for each RIC on Apache Kafka and published data to related topics

18. TREP API
Apache
Kafka
Apache NiFi MongoDB
Apache
Zeppelin &
Power BI

19. Apache NiFi
§ In order to manage the flow, we decided to use Apache NiFi
§ We used KafkaConsumer processor to consume messages from Kafka queues
§ The NiFi flow was designed to be persisted on MongoDB

20. Our NiFi
Flow

21. TREP API
Apache
Kafka
Apache NiFi MongoDB
Apache
Zeppelin &
Power BI

22. MongoDB
§ We had prepared data in JSON format with our Java application
§ Since we have MongoDB installed on our enterprise systems, we decided to persist
this data to MongoDB
§ Although MongoDB is not a part of HDP, it seemed as a good choice for our
researchers to use this data in their analyses

23. TREP API
Apache
Kafka
Apache NiFi MongoDB
Apache
Zeppelin &
Power BI

24. Apache Zeppelin
§ We provided our researchers with access to Apache Zeppelin and connection to
MongoDB via Python
§ By doing so, we offered an alternative to the tools on local computers and provided a
unified interface for financial analysis

25. Business Intelligence on Client Side
§ Our users had to download daily tick-data manually from their Thomson Reuters
Terminals and work on Excel
§ Users were then able to access tick-data using Power BI
o We also provided our users with a news timeline along with the tick-data

26. We needed more!
§ We had to visualize the data in real-time
o Analysis on persisted data using MongoDB, PowerBI and Apache Zeppelin was not enough

27. TREP API
Apache
Kafka
Apache NiFi MongoDB
Apache
Zeppelin &
Power BI

28. Development of High Frequency Indicators Using Real-Time Tick
Data on Apache Superset and Druid
Phase 1
Prod
migratio
n
Next
phases
Test
Cluster
Phase 2 Phase 3

29. TREP
API
Apache
Kafka
Apache
Druid
Apache
Superset

30. Apache Druid
§ We needed a database which was able to:
o Answer ad-hoc queries (slice/dice) for a limited window efficiently
o Store historic data and seamlessly integrate current and historic data
o Provide native integration with possible real-time visualization frameworks (preferably from
Apache stack)
o Provide native integration with Apache Kafka
§ Apache Druid addressed all the aforementioned requirements
§ Indexing task was achieved using Tranquility

31. TREP
API
Apache
Kafka
Apache
Druid
Apache
Superset

32. Apache Superset
§ Apache Superset was the obvious alternative for real-time visualization since tick-data
was stored on Apache Druid
o Native integration with Apache Druid
o Freely available on Hortonworks service stack
§ We prepared real-time dashboards including:
o Transaction Count
o Bid / Ask Prices
o Contributor Distribution
o Bid - Ask Spread

33. We needed more, again!
§ Reliability issues with Druid
§ Performance issues
§ Enterprise integration requirements

34. Development of High Frequency Indicators Using Real-Time Tick
Data on Apache Superset and Druid
Phase 1
Prod
migratio
n
Next
phases
Test
Cluster
Phase 2 Phase 3

35. Architecture
Internet Data
Enterprise Content
Social Media/Media
Micro Level Data
Commercial Data Vendors
Ingestion
Big Data Platform Data Science
GovernanceData Sources

36. Development of High Frequency Indicators Using Real-Time Tick
Data on Apache Superset and Druid
Phase 1
Prod
migratio
n
Next
phases
Test
Cluster
Phase 2 Phase 3

37. TREP API Apache Kafka
Apache
Hive + Druid
Integration
Apache Spark
Apache
Superset

38. Apache Hive + Druid Integration
§ After setting up our production environment (using HDP 3.0.1.0) and started to
feed data, we realized that data were scattered and we were missing the option to
co-utilize these different data sources
§ We then realized that Apache Hive was already providing Kafka & Druid indexing
service in the form of a simple table creation and querying facility for Druid from
Hive

39. TREP API Apache Kafka
Apache
Hive + Druid
Integration
Apache Spark
Apache
Superset

40. Apache Spark
§ Due to additional calculation requirements of our users, we decided to utilize Apache
Spark
§ With Apache Spark 2.4, we used Spark Streaming and Spark SQL contexts together in
the same application
§ In our Spark application
o For every 5 seconds, a 30-second window is created
o On each window, outlier boundaries are calculated
o Outlier data points are detected

42. Current Architecture
4
3
2
1

43. Current Architecture & Progress So Far
Java Application
Kafka Topic (real-time)
Kafka Topic (windowed)
TREP Event Queue
Consume Publish
Spark Application
Consume
Publish
Druid Datasource
(real-time)
Druid Datasource
(windowed)
Superset Dashboard
(tick data)
Superset Dashboard
(outlier)

44. TREP Data Flow

45. Windowed Spark Streaming

46. Tick-Data Dashboard

47. Outlier Dashboard

48. Work in Progress
5
4
3
2
1

49. Implementing…
§ Moving average calculation (20-day window)
§ Volatility Indicator
§ Average True Range Indicator (moving average)
o [ max(t) - min(t) ]
o [ max(t) - close(t-1) ]
o [ max(t) - close(t-1) ]

50. Future Plans
6
5
4
3
2
1

51. To-Do List
§ Matching data subscription
§ Bringing historical tick data into real-time analysis
§ Possible use of machine learning for intraday indicators

52. Thank you!
Q & A