These are slides from Jay Chin's presentation at ElasticOnTour 2015 conference that brings together Elasticsearch, Logstash, and Kibana experts.

1. Jay Chin – jay.chin@excelian.com
Principal Consultant, Excelian
3 November 2015
0
Grid Computing and Trade Analytics
with Elastic

2. ExcelianTechnical Consulting –Who we are
1
 Financial Services specialists
 Distributed computing specialists since 2006
 Experts in niche and emerging technologies

3. Financial Services – Insatiable appetite for Compute
• Algorithms (Computers) that actually do the trading - Remember the
Flash Crash of May 6 2010?This is a result of HFT stopping and not trading
causing the Market to drop 6% in mere minutes.
• Financial modelling - Use complex mathematical models to deal with asset
prices, market movement, portfolio returns, etc.
• Huge amounts of data to process - e.g. connecting to one of the exchanges
FIX, there will be up to 100,000 messages to process per second.
2
Source: InformationWeek,Wall Street &Technology Source:TheTelegraph

4. What do compute grids look like ?
3
Typical Numbers For A Standard Grid
- 40k cores/engines
- 30m tasks
- 120 GB of Log metrics
- 60 – 80% Average Utilisation
- Data retention up to 6 Months
https://flic.kr/p/ydnEvw

5. Grid Maturity in Financial Services
4
HPC Maturity
Benchmark
2014
Tier I =Tier I banks
Tier II =Tier II banks
Point = point solutions
used only for a specific
use case
(e.g. behind a software
package, only for one
business line…)
MaturityLevel
It is fairly common for bank to have grids.
Larger banks tend to have at least 30,000 cores.

6. Case Study: ELK for Enterprise Grid Reporting Framework
Requirements:
• Enterprise Grid with 40,000 Cores across 4 Data centers
in 2 Countries
• Reporting Dashboard forGrid Metrics
• Scalable up to 100,000 cores and 200 million Grid tasks
per day
5
Goal: Architect an Enterprise Grid
and design a Grid metrics reporting framework
for a top-tier investment bank.

7. The Case for ELK
6
Features ElasticSearch
Intuitive Interface
Ease of Use
Security Integration
Scalability
Support
Pricing
Features
Integration with Grid Middleware
Elasticsearch met all the requirements except for the last one,
which required some work on our part.

8. Initial Architecture: Single cluster across 2 regions
7
curl -XPUT localhost:9200/GridA_metrics/_settings -d '{
"index.routing.allocation.include.tag" : “region_A" }'

9. Architecture After Consultation with Elastic Platinum Support
8

10. Challenges
9
• Bespoke deployment due to security restrictions in Bank’s
Datacentre
• https://github.com/Excelian/ansible_fs_elkstack
• Development of custom ETL to query Grid Metrics
database and load them into ElasticSearch
https://flic.kr/p/eqJHbr

11. More ELK Goodness
• Bank was very impressed with
the reporting capabilities
• Support team at Elastic was
also superior compared to
some of the big vendors we
were dealing with
10
AS A RESULT
1. We were tasked to do log centralization using
Logstash
2. Explore Watcher for monitoring Grid and
applications

12. Feedback from Investment Bank
• For the first time ever, developers
were able to view Grid metrics and
correlate them with logging
events from a single interface
• Application teams are
experimenting with Elastic
• Developers rethinking logging
11

13. KeyTakeaways
• Lots of opportunities and interest in ElasticSearch in Financial
services
• Single tool to do log analytics, alerting, events, searching, and
metrics
• Elastic ticks all the right boxes for financial services: Security,
scalability, support SLAs, etc.
• Elastic Platinum support has been fantastic
• Advanced Use Cases : Fraud Detection,Trade surveillance,
Market Sentiment Analysis
12

14. 13
Thank you!
If you have any feedback, please get in touch:
jay.chin@excelian.com
If you would like to join our community of technologists at Excelian
please have a look at our careers page for the latest vacancies:
www.excelian.com/careers/
@Excelian
@Excelian
@ExcelianLTD

15. www.elastic.co
14