Development a framework The framework allows streaming and visualization of historical (previous) and current currency prices in close to real time The framework benchmarks every monitored broker to decide whether he/she is trustworthy

1. Foreign Exchange Data Crawling and
Analysis for Knowledge Discovery
Leading to Informative Decision
Making
Data mining
Dr. Nasser Ghadiri
Mostafa Arjmand

2. The Presentation Include The Following :
• Introduction
• Data Coverage And Size
• Framework Structure And The Data Fetching Mechanism
• Experimental study
• Ranking And Classification
• Conclusions
2

3. Introduction
3

4. Introduction :
• Foreign exchange (Forex) market
• Development a framework
• The framework allows streaming and visualization of historical (previous) and current currency
prices in close to real time
• The framework benchmarks every monitored broker to decide whether he/she is trustworthy
4

5. Broker
• However, human decision making process is mostly subjective and emotional
• It is not necessary that every broker considers the right factors affecting exchange rate, or even gives the
right weight to each
5

6. CTS-Forex Performance Study
Forex monitoring system designed to :
• Automatically track and monitor brokers
• By fetching, visualizing and analyzing their
announced exchange rate
• Our current effort focuses on EUR/USD exchange
rate
• covers 24.1% of forex market
CTS FOREX PERFORMANCE STUDY MAIN FEATURES
6

7. Main Components Of the Proposed Framework :
Visualizing the Data Streams
CTS FOREX PERFORMANCE STUDY MAIN FEATURES
The ability to visualize previously captured data
as well as current data is essential for a expert
1) Single Broker Visualization
2) Multiple Brokers Visualization
3) Ranking and Classification Visualization
7

8. Main Components Of the Proposed Framework :
Analyzing Captured Data
CTS FOREX PERFORMANCE STUDY MAIN FEATURES
The ability to analyze and interpret the captured
data provides useful information for enhancing
the decision making process
Short
term
Medium
term
Long
term
8

9. DATA COVERAGE AND SIZE
9

10. Data Coverage And Size :
The Data Structure
Snapshot : The data for each broker
• Date of the snapshot
• Number of seconds recorded in the snapshot
Tick : Each piece of data is called a tick
• Date of the tick
• The ASK price of the currency
• The BID price of the currency
Recording Periods
Indeed data is the main source for knowledge discovery leading to informative decision making
10
4/5 are ignored

11. Data Coverage And Size :
Brokers Data Summary
• They monitored a total of 52 brokers
• All the collected data used is from the
MetaTrader 4 platform
BROKERS SUMMARY TABLE
11

12. Data Coverage And Size :
Daily Data Distribution (Monthly Stacked)
12

13. FRAMEWORK STRUCTURE AND
DATA FETCHING MECHANISM
13

14. Framework Structure And The Data Fetching Mechanism
System Block Diagram 14

15. Framework Features
• Main function of this framework is capturing and visualizing stream data
• The number of online brokers is increasing
• This rapid increase has to be handled by our framework by having the ability of easily add new
brokers
• Brokers use different platforms to provide traders with several features with minimum modifications
• The framework should provide runtime visualization as well as visualization of previously captured data
15

16. Framework Features
Framework should provide the ability to analyze data at three levels:
1) Short term analysis
2) Medium term analysis
3) Long term analysis
16

17. Available Resources
Meta Trader 4
• The software consists of two components:
• Client
• Server
• The server is used by brokers
• To provide a client component to their clients
• The client provides end users with the ability to :
• View live streaming prices
• Place orders
• Write their own scripts that automate the trading process
17

18. Client Applications Installation
• First mission is install the client applications (Meta Trader) to
start fetching the data
• The problem we faced was fitting all clients on the same
computer as they consume a lot of memory
• Distribute client applications
• save more memory for short term analysis
18

19. Local Data Centralization
Client-Server ModelDatabase
Insertion and Selection Model
The first model we came up with was a client/server model
The second model is a database insertion and selection model
Although the first showed better results
• Decided to stick to the second model
• It provides better integrity
19

20. Global Data Centralization
Global Data Centralization
We have all the fetched data in each computer centralized in its
fetcher application
• This allows us to run short term analysis and
• Some medium term analysis
• But it does not allow us to run long term analysis
• brokers ranking
• benchmarking
• Although centralizing data will have the drawback
20

21. Online/Local Visualization
One of the implemented features was a visualization system for online users
• Track their favorite brokers and compare them against others
• The first model was storing data into a MySQL database
• The second model was pre-compiling the data into JSON data files
We still had to find a way to pre-compile the data stored locally on our
computers and push it to the online server
Database model VS JSON model21

22. Manager Application
Online Visualization Structure
The manager application will be responsible for
• Running a lot of medium and long term analysis
• Ranking the brokers
• Benchmarking them
• Prioritizing the snapshots
22

23. EXPERIMENTAL STUDY
23

24. Experimental study
Windows 8 Pro with 32GB installed memory (RAM).
Intel Core i7-3770 CPU @ 3.40GHz and 3.40 GHz.
The first disk is a HDD second is a SSD.
DATA SET SUMMARY
Local centralization study
Visualization component study
24

25. Local Data Centralization Study
Local Data Centralization (Model 1 vs Model 2 HHD) Week 4
25

26. Visualization Study
WEEK 1 - VISUALIZATION THROUGHPUT RESULTS (MODEL 1 VS MODEL 2)
WEEK 2 - VISUALIZATION THROUGHPUT RESULTS (MODEL 1 VS MODEL 2)
Fetching data
from the
database
Fetching data
from pre-
compiled JSON
files
26

27. RANKING AND CLASSIFICATION
27

28. Ranking And Classification
Our main focus is to study performance of brokers
and find trustworthy ones
The two approaches are supplementary
28

29. Feed-price performance
Feed price performance key features Spread Consistency
29

30. Ranking And Classification
1) The Algorithm:
30

31. Ranking And Classification
• The ranking algorithm was just the first step for distinguishing good brokers
• We do not have any gold standard rules to specify ranges of good and bad brokers’ scores
• Therefore, we decided to use a clustering algorithm
• For each ranked snapshot, we cluster the brokers based on their final scores
• Thus we are clustering one-dimensional data.
31

32. Clustering
Hierarchical
Density Based
K-Means
Distribution of Brokers w.r.t. their Most Common Class
In our case, we want to find trustworthy brokers, gray, and bad ones
32

33. Ranking And Classification
2) Membership:
To determine whether a broker is good or not, we have to study their memberships in all classes during
their life cycle
33

34. CONCLUSIONS
34

35. CONCLUSIONS
• We Present a group of issues that exist in the current forex market
• Designed and implemented a full framework
• To monitor a list of brokers by fetching their data continuously
• Comparing them to each other and thus speed up and enhance domain experts’
decision making process
35

36. 36
THANK YOU FOR LISTENING
•‫مطالب‬‫تکراری‬
•‫شده‬ ‫ارائه‬ ‫کار‬ ‫با‬ ‫مقاله‬ ‫تیتر‬ ‫نداشتن‬ ‫تطابق‬
•‫شده‬ ‫ارائه‬ ‫پراکنده‬ ‫صورت‬ ‫به‬ ‫آینده‬ ‫کارهای‬
•‫و‬ ‫شکلها‬ ‫آوردن‬ ‫در‬ ‫سازماندهی‬ ‫و‬ ‫نظم‬ ‫داشتن‬ ‫عدم‬‫جداول‬
•‫غیر‬ ‫برای‬ ‫چارچوب‬‫از‬EUR-USD
•‫بندی‬ ‫خوشه‬ ‫مورد‬ ‫در‬ ‫کم‬ ‫توضیحات‬
•‫مطالب‬ ‫پراکندگی‬
•‫داده‬ ‫پنجم‬ ‫یک‬‫ها‬
•‫شده‬ ‫ارائه‬ ‫چارچوب‬ ‫با‬ ‫کار‬ ‫نحوه‬ ‫چگونگی‬