SlideShare uses cookies to improve functionality and performance, and to provide you with relevant advertising. If you continue browsing the site, you agree to the use of cookies on this website. See our User Agreement and Privacy Policy.
SlideShare uses cookies to improve functionality and performance, and to provide you with relevant advertising. If you continue browsing the site, you agree to the use of cookies on this website. See our Privacy Policy and User Agreement for details.
Successfully reported this slideshow.
Activate your 14 day free trial to unlock unlimited reading.
Have an existing application that needs search super powers? Or are ad hoc searches melting your SQL Server? Either way, this is the talk for you. We will explore search enabling an existing application - from data modeling to retrieval. In this talk, we will cover data modeling for search, connecting ElasticSearch to your data pipeline, building a search API and connecting to an Angular web site. Presented at Code on the Beach 2018
Have an existing application that needs search super powers? Or are ad hoc searches melting your SQL Server? Either way, this is the talk for you. We will explore search enabling an existing application - from data modeling to retrieval. In this talk, we will cover data modeling for search, connecting ElasticSearch to your data pipeline, building a search API and connecting to an Angular web site. Presented at Code on the Beach 2018
1.
TRANSFORMING
YOUR APPLICATION WITH
BRIAN RITCHIE
CTO, XEOHEALTH
2018
@brian_ritchie
brian.ritchie@gmail.com
http://www.dotnetpowered.com
2.
TRANSFORMING YOUR APPLICATION WITH ELASTIC SEARCH
Brian Ritchie
CTO for XeoHealth Corporation
22 years of technology experience developing
software and managing technology teams in various
industries.
Author of RavenDB High Performance published by
PACKT in 2013.
Pursuing a Masters in Computer Science from
Georgia Tech.
@brian_ritchie
brian.ritchie@gmail.com
http://www.dotnetpowered.com
3.
TRANSFORMING YOUR APPLICATION WITH ELASTIC SEARCH
WELCOME
INTRODUCTION TO ELASTIC SEARCH
BRING YOUR DATA
SECURITY CONSIDERATIONS
PUTTING IT ALL TOGETHER
4.
TRANSFORMING YOUR APPLICATION WITH ELASTIC SEARCH
LET’S GET STARTED
With the story of your application…
5.
TRANSFORMING YOUR APPLICATION WITH ELASTIC SEARCH
IT WAS AMAZING…
It even had the new application smell.
6.
TRANSFORMING YOUR APPLICATION WITH ELASTIC SEARCH
BUT THEN CAME THE USERS, THE DATA, AND THE
NEW REQUIREMENTS….
7.
TRANSFORMING YOUR APPLICATION WITH ELASTIC SEARCH
NOW YOUR APPLICATION IS LEAVING YOU STRANDED
ON THE SIDE OF THE ROAD…
8.
TRANSFORMING YOUR APPLICATION WITH ELASTIC SEARCH
SO, WHY IS YOUR APPLICATION MELTING?
Too Many Reads,
Too Many Writes
= Contention
9.
TRANSFORMING YOUR APPLICATION WITH ELASTIC SEARCH
IS THIS THE ONLY FUTURE FOR YOUR APPLICATION?
10.
TRANSFORMING YOUR APPLICATION WITH ELASTIC SEARCH
NO WAY! IT’S TIME TO PIMP YOUR RIDE!
11.
TRANSFORMING YOUR APPLICATION WITH ELASTIC SEARCH
INTRODUCING
12.
TRANSFORMING YOUR APPLICATION WITH ELASTIC SEARCH
INTRODUCING
Elasticsearch is a search engine based
on Lucene. It provides a lightening fast,
distributed, multitenant-capable full-text
search engine with a HTTP RESTful
interface.
Elasticsearch is released as open source
under the terms of the Apache License.
It can be deployed locally or hosted in the
cloud.
13.
TRANSFORMING YOUR APPLICATION WITH ELASTIC SEARCH
INTRODUCING
Well known for
being part of the
Stack
The Elastic StackNote: this was rebranded
used to analyze log data
generated by your application
14.
TRANSFORMING YOUR APPLICATION WITH ELASTIC SEARCH
WHY
FAST
Document
Centric
Shard/
Partition
Index
Everything
Extremely Fast. Optimized for search. Flexible Schema.
Distributed
Cluster
FOR YOUR APPLICATION
15.
TRANSFORMING YOUR APPLICATION WITH ELASTIC SEARCH
WHY
SCALABLE “We can support clusters of massive scale. Well
into the 100s of terabytes. One of our largest
customers is currently at 750TB and should be in
excess of 1PB by the end of the year. “ - Scalefastr
“Today there are 60+ Elasticsearch clusters and
2000+ nodes. The daily ingestion reaches 18 billion
documents, and daily search requests reach 3.5
billion. ” - eBay
FOR YOUR APPLICATION
16.
TRANSFORMING YOUR APPLICATION WITH ELASTIC SEARCH
YOUR USERS ON
17.
TRANSFORMING YOUR APPLICATION WITH ELASTIC SEARCH
That sounds awesome!!
But, how does Elastic Search fit
into my data infrastructure?
BRINGING YOUR DATA INTO
18.
TRANSFORMING YOUR APPLICATION WITH ELASTIC SEARCH
Utilizing a CQRS inspired pattern
BRINGING YOUR DATA INTO ELASTIC SEARCH
Command/Query Responsibility Segregation (CQRS) is the idea that
you can use a different model to update information than the model
you use to read information
Persistent View Model
Transactional Data
19.
TRANSFORMING YOUR APPLICATION WITH ELASTIC SEARCH
OPTION 1: Add Elastic search into your database pipeline
cluster
Kafka
ActiveMQ
RabbitMQ
….
BRINGING YOUR DATA INTO ELASTIC SEARCH
20.
TRANSFORMING YOUR APPLICATION WITH ELASTIC SEARCH
clusterKafka
ActiveMQ
RabbitMQ
….
OPTION 2: Replicate database updates to Elastic Search
BRINGING YOUR DATA INTO ELASTIC SEARCH
21.
TRANSFORMING YOUR APPLICATION WITH ELASTIC SEARCH
cluster
OPTION 3: Batch your database updates on a schedule
BRINGING YOUR DATA INTO ELASTIC SEARCH
22.
TRANSFORMING YOUR APPLICATION WITH ELASTIC SEARCH
Structuring your data for search
DATA MODELING
Conceptually, indexes are like databases.
• SQL => Databases => Tables => Columns/Rows
• Elasticsearch => Indices => Types => Documents with Properties
Loans Payments Inventory
23.
TRANSFORMING YOUR APPLICATION WITH ELASTIC SEARCH
Defining indexes
DATA MODELING:
Like databases, but more powerful. Queries can easily span indexes bringing
data together as needed.
Use indexes for:
• Security – use indexes to support
multi-tenant system
loan_hud loan_fdic
• Partitioning – time or other
data
log_20180501 log_20180502
24.
TRANSFORMING YOUR APPLICATION WITH ELASTIC SEARCH
Defining documents
DATA MODELING:
{
"FirstName": "Jonathan",
"Address": "15 Wanamassa Point Road ",
"City": "Jacksonville",
"State": "FL",
"Children": [
{ "Name": "Michael", "Age": 10 },
{ "Name": "Jennifer", "Age": 8 },
{ "Name": "Samantha", "Age": 5 },
{ "Name": "Elena", "Age": 2 }
]
}
Instead of decomposing
data into separate entities:
Design documents based on what needs to
be searched and displayed together:
25.
TRANSFORMING YOUR APPLICATION WITH ELASTIC SEARCH
Teach Elastic Search about specific types
DATA MODELING:
PUT _template/file_name_mapping
{
"template": ”loan_*",
"settings": { "number_of_shards": 5 },
"mappings": {
"_default_": {
"dynamic_templates": [ {
"filename_string_not_analyzed": {
"match": "edi_filename",
"mapping": {
"index": "not_analyzed",
"type": "string” }
} } ]
} }
}
26.
TRANSFORMING YOUR APPLICATION WITH ELASTIC SEARCH
Giving the right users access to the right data
SECURITY CONSIDERATIONS
cluster
Search Guard
Custom
.NET Core / Java / etc.
Elastic Stack
Security
27.
TRANSFORMING YOUR APPLICATION WITH ELASTIC SEARCH
PUTTING IT ALL TOGETHER
The Home Mortgage Disclosure Act (HMDA) requires many financial institutions to
maintain, report, and publicly disclose loan-level information about mortgages.
Searching CFPB loan records w/ Elastic Search
28.
TRANSFORMING YOUR APPLICATION WITH ELASTIC SEARCH
PUTTING IT ALL TOGETHER
107 million
loan records
var record = csv.GetRecord<dynamic>();
var loan = Loan.FromFileLoan(record);
var index_name = string.Format("loan_{0}",
loan.agency_abbr.ToLower());
var response = client.Index<Loan>(
new IndexRequest<Loan>(loan, index_name));
Loading CFPB loan records into Elastic Search
{
"county": "Maricopa County",
"respondent_id": "0000451965",
"lien_status": "Not applicable",
"sequence_number": "0945293",
"applicant_income": 132000,
"preapproval": "Not applicable",
"applicant_race": "White",
"msamd": "Phoenix, Mesa, Scottsdale - AZ",
….
29.
TRANSFORMING YOUR APPLICATION WITH ELASTIC SEARCH
Using Kibana to search and visualize your data
PUTTING IT ALL TOGETHER
30.
TRANSFORMING YOUR APPLICATION WITH ELASTIC SEARCH
Using Elastic Search to optimize your user experience
PUTTING IT ALL TOGETHER
cluster
Single Page
Application
(SPA)
Angular /
React / etc.
JWT
Query
Request
REST API
.NET Core /
Java / etc.
Add security,
logging, etc.
31.
TRANSFORMING YOUR APPLICATION WITH ELASTIC SEARCH
Custom application using Angular, PrimeNG, .NET Core, and Elastic Search
PUTTING IT ALL TOGETHER
32.
BRIAN RITCHIE
CTO, XEOHEALTH
2018
@brian_ritchie
brian.ritchie@gmail.com
http://www.dotnetpowered.com
QUESTIONS?
TRANSFORMING
YOUR APPLICATIONS WITH