1.
TRANSFORMING
YOUR APPLICATION WITH
BRIAN RITCHIE
CTO, XEOHEALTH
2018
@brian_ritchie
brian.ritchie@gmail.com
http://www.dotnetpowered.com

2.
TRANSFORMING YOUR APPLICATION WITH ELASTIC SEARCH
Brian Ritchie
CTO for XeoHealth Corporation
22 years of technology experience developing
software and managing technology teams in various
industries.
Author of RavenDB High Performance published by
PACKT in 2013.
Pursuing a Masters in Computer Science from
Georgia Tech.
@brian_ritchie
brian.ritchie@gmail.com
http://www.dotnetpowered.com

3.
TRANSFORMING YOUR APPLICATION WITH ELASTIC SEARCH
WELCOME
INTRODUCTION TO ELASTIC SEARCH
BRING YOUR DATA
SECURITY CONSIDERATIONS
PUTTING IT ALL TOGETHER

4.
TRANSFORMING YOUR APPLICATION WITH ELASTIC SEARCH
LET’S GET STARTED
With the story of your application…

5.
TRANSFORMING YOUR APPLICATION WITH ELASTIC SEARCH
IT WAS AMAZING…
It even had the new application smell.

6.
TRANSFORMING YOUR APPLICATION WITH ELASTIC SEARCH
BUT THEN CAME THE USERS, THE DATA, AND THE
NEW REQUIREMENTS….

7.
TRANSFORMING YOUR APPLICATION WITH ELASTIC SEARCH
NOW YOUR APPLICATION IS LEAVING YOU STRANDED
ON THE SIDE OF THE ROAD…

8.
TRANSFORMING YOUR APPLICATION WITH ELASTIC SEARCH
SO, WHY IS YOUR APPLICATION MELTING?
Too Many Reads,
Too Many Writes
= Contention

9.
TRANSFORMING YOUR APPLICATION WITH ELASTIC SEARCH
IS THIS THE ONLY FUTURE FOR YOUR APPLICATION?

10.
TRANSFORMING YOUR APPLICATION WITH ELASTIC SEARCH
NO WAY! IT’S TIME TO PIMP YOUR RIDE!

11.
TRANSFORMING YOUR APPLICATION WITH ELASTIC SEARCH
INTRODUCING

12.
TRANSFORMING YOUR APPLICATION WITH ELASTIC SEARCH
INTRODUCING
Elasticsearch is a search engine based
on Lucene. It provides a lightening fast,
distributed, multitenant-capable full-text
search engine with a HTTP RESTful
interface.
Elasticsearch is released as open source
under the terms of the Apache License.
It can be deployed locally or hosted in the
cloud.

13.
TRANSFORMING YOUR APPLICATION WITH ELASTIC SEARCH
INTRODUCING
Well known for
being part of the
Stack
The Elastic StackNote: this was rebranded
used to analyze log data
generated by your application

14.
TRANSFORMING YOUR APPLICATION WITH ELASTIC SEARCH
WHY
FAST
Document
Centric
Shard/
Partition
Index
Everything
Extremely Fast. Optimized for search. Flexible Schema.
Distributed
Cluster
FOR YOUR APPLICATION

15.
TRANSFORMING YOUR APPLICATION WITH ELASTIC SEARCH
WHY
SCALABLE “We can support clusters of massive scale. Well
into the 100s of terabytes. One of our largest
customers is currently at 750TB and should be in
excess of 1PB by the end of the year. “ - Scalefastr
“Today there are 60+ Elasticsearch clusters and
2000+ nodes. The daily ingestion reaches 18 billion
documents, and daily search requests reach 3.5
billion. ” - eBay
FOR YOUR APPLICATION

16.
TRANSFORMING YOUR APPLICATION WITH ELASTIC SEARCH
YOUR USERS ON

17.
TRANSFORMING YOUR APPLICATION WITH ELASTIC SEARCH
That sounds awesome!!
But, how does Elastic Search fit
into my data infrastructure?
BRINGING YOUR DATA INTO

18.
TRANSFORMING YOUR APPLICATION WITH ELASTIC SEARCH
Utilizing a CQRS inspired pattern
BRINGING YOUR DATA INTO ELASTIC SEARCH
Command/Query Responsibility Segregation (CQRS) is the idea that
you can use a different model to update information than the model
you use to read information
Persistent View Model
Transactional Data

19.
TRANSFORMING YOUR APPLICATION WITH ELASTIC SEARCH
OPTION 1: Add Elastic search into your database pipeline
cluster
Kafka
ActiveMQ
RabbitMQ
….
BRINGING YOUR DATA INTO ELASTIC SEARCH

20.
TRANSFORMING YOUR APPLICATION WITH ELASTIC SEARCH
clusterKafka
ActiveMQ
RabbitMQ
….
OPTION 2: Replicate database updates to Elastic Search
BRINGING YOUR DATA INTO ELASTIC SEARCH

21.
TRANSFORMING YOUR APPLICATION WITH ELASTIC SEARCH
cluster
OPTION 3: Batch your database updates on a schedule
BRINGING YOUR DATA INTO ELASTIC SEARCH

22.
TRANSFORMING YOUR APPLICATION WITH ELASTIC SEARCH
Structuring your data for search
DATA MODELING
Conceptually, indexes are like databases.
• SQL => Databases => Tables => Columns/Rows
• Elasticsearch => Indices => Types => Documents with Properties
Loans Payments Inventory

23.
TRANSFORMING YOUR APPLICATION WITH ELASTIC SEARCH
Defining indexes
DATA MODELING:
Like databases, but more powerful. Queries can easily span indexes bringing
data together as needed.
Use indexes for:
• Security – use indexes to support
multi-tenant system
loan_hud loan_fdic
• Partitioning – time or other
data
log_20180501 log_20180502

24.
TRANSFORMING YOUR APPLICATION WITH ELASTIC SEARCH
Defining documents
DATA MODELING:
{
"FirstName": "Jonathan",
"Address": "15 Wanamassa Point Road ",
"City": "Jacksonville",
"State": "FL",
"Children": [
{ "Name": "Michael", "Age": 10 },
{ "Name": "Jennifer", "Age": 8 },
{ "Name": "Samantha", "Age": 5 },
{ "Name": "Elena", "Age": 2 }
]
}
Instead of decomposing
data into separate entities:
Design documents based on what needs to
be searched and displayed together:

25.
TRANSFORMING YOUR APPLICATION WITH ELASTIC SEARCH
Teach Elastic Search about specific types
DATA MODELING:
PUT _template/file_name_mapping
{
"template": ”loan_*",
"settings": { "number_of_shards": 5 },
"mappings": {
"_default_": {
"dynamic_templates": [ {
"filename_string_not_analyzed": {
"match": "edi_filename",
"mapping": {
"index": "not_analyzed",
"type": "string” }
} } ]
} }
}

26.
TRANSFORMING YOUR APPLICATION WITH ELASTIC SEARCH
Giving the right users access to the right data
SECURITY CONSIDERATIONS
cluster
Search Guard
Custom
.NET Core / Java / etc.
Elastic Stack
Security

27.
TRANSFORMING YOUR APPLICATION WITH ELASTIC SEARCH
PUTTING IT ALL TOGETHER
The Home Mortgage Disclosure Act (HMDA) requires many financial institutions to
maintain, report, and publicly disclose loan-level information about mortgages.
Searching CFPB loan records w/ Elastic Search

28.
TRANSFORMING YOUR APPLICATION WITH ELASTIC SEARCH
PUTTING IT ALL TOGETHER
107 million
loan records
var record = csv.GetRecord<dynamic>();
var loan = Loan.FromFileLoan(record);
var index_name = string.Format("loan_{0}",
loan.agency_abbr.ToLower());
var response = client.Index<Loan>(
new IndexRequest<Loan>(loan, index_name));
Loading CFPB loan records into Elastic Search
{
"county": "Maricopa County",
"respondent_id": "0000451965",
"lien_status": "Not applicable",
"sequence_number": "0945293",
"applicant_income": 132000,
"preapproval": "Not applicable",
"applicant_race": "White",
"msamd": "Phoenix, Mesa, Scottsdale - AZ",
….

29.
TRANSFORMING YOUR APPLICATION WITH ELASTIC SEARCH
Using Kibana to search and visualize your data
PUTTING IT ALL TOGETHER

30.
TRANSFORMING YOUR APPLICATION WITH ELASTIC SEARCH
Using Elastic Search to optimize your user experience
PUTTING IT ALL TOGETHER
cluster
Single Page
Application
(SPA)
Angular /
React / etc.
JWT
Query
Request
REST API
.NET Core /
Java / etc.
Add security,
logging, etc.

31.
TRANSFORMING YOUR APPLICATION WITH ELASTIC SEARCH
Custom application using Angular, PrimeNG, .NET Core, and Elastic Search
PUTTING IT ALL TOGETHER

32.
BRIAN RITCHIE
CTO, XEOHEALTH
2018
@brian_ritchie
brian.ritchie@gmail.com
http://www.dotnetpowered.com
QUESTIONS?
TRANSFORMING
YOUR APPLICATIONS WITH