Transforming your application with Elasticsearch

Brian Ritchie
Brian RitchieCTO at XeoHealth Corporation
TRANSFORMING
YOUR APPLICATION WITH
BRIAN RITCHIE
CTO, XEOHEALTH
2018
@brian_ritchie
brian.ritchie@gmail.com
http://www.dotnetpowered.com
TRANSFORMING YOUR APPLICATION WITH ELASTIC SEARCH
Brian Ritchie
CTO for XeoHealth Corporation
22 years of technology experience developing
software and managing technology teams in various
industries.
Author of RavenDB High Performance published by
PACKT in 2013.
Pursuing a Masters in Computer Science from
Georgia Tech.
@brian_ritchie
brian.ritchie@gmail.com
http://www.dotnetpowered.com
TRANSFORMING YOUR APPLICATION WITH ELASTIC SEARCH
WELCOME
INTRODUCTION TO ELASTIC SEARCH
BRING YOUR DATA
SECURITY CONSIDERATIONS
PUTTING IT ALL TOGETHER
TRANSFORMING YOUR APPLICATION WITH ELASTIC SEARCH
LET’S GET STARTED
With the story of your application…
TRANSFORMING YOUR APPLICATION WITH ELASTIC SEARCH
IT WAS AMAZING…
It even had the new application smell.
TRANSFORMING YOUR APPLICATION WITH ELASTIC SEARCH
BUT THEN CAME THE USERS, THE DATA, AND THE
NEW REQUIREMENTS….
TRANSFORMING YOUR APPLICATION WITH ELASTIC SEARCH
NOW YOUR APPLICATION IS LEAVING YOU STRANDED
ON THE SIDE OF THE ROAD…
TRANSFORMING YOUR APPLICATION WITH ELASTIC SEARCH
SO, WHY IS YOUR APPLICATION MELTING?
Too Many Reads,
Too Many Writes
= Contention
TRANSFORMING YOUR APPLICATION WITH ELASTIC SEARCH
IS THIS THE ONLY FUTURE FOR YOUR APPLICATION?
TRANSFORMING YOUR APPLICATION WITH ELASTIC SEARCH
NO WAY! IT’S TIME TO PIMP YOUR RIDE!
TRANSFORMING YOUR APPLICATION WITH ELASTIC SEARCH
INTRODUCING
TRANSFORMING YOUR APPLICATION WITH ELASTIC SEARCH
INTRODUCING
Elasticsearch is a search engine based
on Lucene. It provides a lightening fast,
distributed, multitenant-capable full-text
search engine with a HTTP RESTful
interface.
Elasticsearch is released as open source
under the terms of the Apache License.
It can be deployed locally or hosted in the
cloud.
TRANSFORMING YOUR APPLICATION WITH ELASTIC SEARCH
INTRODUCING
Well known for
being part of the
Stack
The Elastic StackNote: this was rebranded
used to analyze log data
generated by your application
TRANSFORMING YOUR APPLICATION WITH ELASTIC SEARCH
WHY
FAST
Document
Centric
Shard/
Partition
Index
Everything
Extremely Fast. Optimized for search. Flexible Schema.
Distributed
Cluster
FOR YOUR APPLICATION
TRANSFORMING YOUR APPLICATION WITH ELASTIC SEARCH
WHY
SCALABLE “We can support clusters of massive scale. Well
into the 100s of terabytes. One of our largest
customers is currently at 750TB and should be in
excess of 1PB by the end of the year. “ - Scalefastr
“Today there are 60+ Elasticsearch clusters and
2000+ nodes. The daily ingestion reaches 18 billion
documents, and daily search requests reach 3.5
billion. ” - eBay
FOR YOUR APPLICATION
TRANSFORMING YOUR APPLICATION WITH ELASTIC SEARCH
YOUR USERS ON
TRANSFORMING YOUR APPLICATION WITH ELASTIC SEARCH
That sounds awesome!!
But, how does Elastic Search fit
into my data infrastructure?
BRINGING YOUR DATA INTO
TRANSFORMING YOUR APPLICATION WITH ELASTIC SEARCH
Utilizing a CQRS inspired pattern
BRINGING YOUR DATA INTO ELASTIC SEARCH
Command/Query Responsibility Segregation (CQRS) is the idea that
you can use a different model to update information than the model
you use to read information
Persistent View Model
Transactional Data
TRANSFORMING YOUR APPLICATION WITH ELASTIC SEARCH
OPTION 1: Add Elastic search into your database pipeline
cluster
Kafka
ActiveMQ
RabbitMQ
….
BRINGING YOUR DATA INTO ELASTIC SEARCH
TRANSFORMING YOUR APPLICATION WITH ELASTIC SEARCH
clusterKafka
ActiveMQ
RabbitMQ
….
OPTION 2: Replicate database updates to Elastic Search
BRINGING YOUR DATA INTO ELASTIC SEARCH
TRANSFORMING YOUR APPLICATION WITH ELASTIC SEARCH
cluster
OPTION 3: Batch your database updates on a schedule
BRINGING YOUR DATA INTO ELASTIC SEARCH
TRANSFORMING YOUR APPLICATION WITH ELASTIC SEARCH
Structuring your data for search
DATA MODELING
Conceptually, indexes are like databases.
• SQL => Databases => Tables => Columns/Rows
• Elasticsearch => Indices => Types => Documents with Properties
Loans Payments Inventory
TRANSFORMING YOUR APPLICATION WITH ELASTIC SEARCH
Defining indexes
DATA MODELING:
Like databases, but more powerful. Queries can easily span indexes bringing
data together as needed.
Use indexes for:
• Security – use indexes to support
multi-tenant system
loan_hud loan_fdic
• Partitioning – time or other
data
log_20180501 log_20180502
TRANSFORMING YOUR APPLICATION WITH ELASTIC SEARCH
Defining documents
DATA MODELING:
{
"FirstName": "Jonathan",
"Address": "15 Wanamassa Point Road ",
"City": "Jacksonville",
"State": "FL",
"Children": [
{ "Name": "Michael", "Age": 10 },
{ "Name": "Jennifer", "Age": 8 },
{ "Name": "Samantha", "Age": 5 },
{ "Name": "Elena", "Age": 2 }
]
}
Instead of decomposing
data into separate entities:
Design documents based on what needs to
be searched and displayed together:
TRANSFORMING YOUR APPLICATION WITH ELASTIC SEARCH
Teach Elastic Search about specific types
DATA MODELING:
PUT _template/file_name_mapping
{
"template": ”loan_*",
"settings": { "number_of_shards": 5 },
"mappings": {
"_default_": {
"dynamic_templates": [ {
"filename_string_not_analyzed": {
"match": "edi_filename",
"mapping": {
"index": "not_analyzed",
"type": "string” }
} } ]
} }
}
TRANSFORMING YOUR APPLICATION WITH ELASTIC SEARCH
Giving the right users access to the right data
SECURITY CONSIDERATIONS
cluster
Search Guard
Custom
.NET Core / Java / etc.
Elastic Stack
Security
TRANSFORMING YOUR APPLICATION WITH ELASTIC SEARCH
PUTTING IT ALL TOGETHER
The Home Mortgage Disclosure Act (HMDA) requires many financial institutions to
maintain, report, and publicly disclose loan-level information about mortgages.
Searching CFPB loan records w/ Elastic Search
TRANSFORMING YOUR APPLICATION WITH ELASTIC SEARCH
PUTTING IT ALL TOGETHER
107 million
loan records
var record = csv.GetRecord<dynamic>();
var loan = Loan.FromFileLoan(record);
var index_name = string.Format("loan_{0}",
loan.agency_abbr.ToLower());
var response = client.Index<Loan>(
new IndexRequest<Loan>(loan, index_name));
Loading CFPB loan records into Elastic Search
{
"county": "Maricopa County",
"respondent_id": "0000451965",
"lien_status": "Not applicable",
"sequence_number": "0945293",
"applicant_income": 132000,
"preapproval": "Not applicable",
"applicant_race": "White",
"msamd": "Phoenix, Mesa, Scottsdale - AZ",
….
TRANSFORMING YOUR APPLICATION WITH ELASTIC SEARCH
Using Kibana to search and visualize your data
PUTTING IT ALL TOGETHER
TRANSFORMING YOUR APPLICATION WITH ELASTIC SEARCH
Using Elastic Search to optimize your user experience
PUTTING IT ALL TOGETHER
cluster
Single Page
Application
(SPA)
Angular /
React / etc.
JWT
Query
Request
REST API
.NET Core /
Java / etc.
Add security,
logging, etc.
TRANSFORMING YOUR APPLICATION WITH ELASTIC SEARCH
Custom application using Angular, PrimeNG, .NET Core, and Elastic Search
PUTTING IT ALL TOGETHER
BRIAN RITCHIE
CTO, XEOHEALTH
2018
@brian_ritchie
brian.ritchie@gmail.com
http://www.dotnetpowered.com
QUESTIONS?
TRANSFORMING
YOUR APPLICATIONS WITH
1 of 32

Recommended

schema.org, Linked Data's Gateway Drug by
schema.org, Linked Data's Gateway Drugschema.org, Linked Data's Gateway Drug
schema.org, Linked Data's Gateway DrugConnected Data World
387 views41 slides
Online MongoDB Training by Easylearning.guru by
Online MongoDB Training by Easylearning.guruOnline MongoDB Training by Easylearning.guru
Online MongoDB Training by Easylearning.guruKCC Software Ltd. & Easylearning.guru
432 views49 slides
Building materialised views for linked data systems using microservices by
Building materialised views for linked data systems using microservicesBuilding materialised views for linked data systems using microservices
Building materialised views for linked data systems using microservicesConnected Data World
721 views37 slides
Practical Applications of Semantic Web in Retail -- Semtech 2014 by
Practical Applications of Semantic Web in Retail -- Semtech 2014 Practical Applications of Semantic Web in Retail -- Semtech 2014
Practical Applications of Semantic Web in Retail -- Semtech 2014 Jay Myers
3.4K views19 slides
Fried data summit data quality data analytics together by
Fried data summit data quality data analytics togetherFried data summit data quality data analytics together
Fried data summit data quality data analytics togetherJeff Fried
296 views33 slides
Linked Data Presentation at TDWI Mpls by
Linked Data Presentation at TDWI MplsLinked Data Presentation at TDWI Mpls
Linked Data Presentation at TDWI MplsJay Myers
764 views45 slides

More Related Content

What's hot

Building, and communicating, a knowledge graph in Zalando by
Building, and communicating, a knowledge graph in ZalandoBuilding, and communicating, a knowledge graph in Zalando
Building, and communicating, a knowledge graph in ZalandoConnected Data World
1.6K views31 slides
Webinar: Introducing the MongoDB Connector for BI 2.0 with Tableau by
Webinar: Introducing the MongoDB Connector for BI 2.0 with TableauWebinar: Introducing the MongoDB Connector for BI 2.0 with Tableau
Webinar: Introducing the MongoDB Connector for BI 2.0 with TableauMongoDB
3.5K views39 slides
Beyond the Basics 3: Introduction to the MongoDB BI Connector by
Beyond the Basics 3: Introduction to the MongoDB BI ConnectorBeyond the Basics 3: Introduction to the MongoDB BI Connector
Beyond the Basics 3: Introduction to the MongoDB BI ConnectorMongoDB
2.3K views15 slides
RDF Data Quality Assessment - connecting the pieces by
RDF Data Quality Assessment - connecting the piecesRDF Data Quality Assessment - connecting the pieces
RDF Data Quality Assessment - connecting the piecesConnected Data World
576 views33 slides
Using NoSQL and Enterprise Shared Services (ESS) to Achieve a More Efficient ... by
Using NoSQL and Enterprise Shared Services (ESS) to Achieve a More Efficient ...Using NoSQL and Enterprise Shared Services (ESS) to Achieve a More Efficient ...
Using NoSQL and Enterprise Shared Services (ESS) to Achieve a More Efficient ...MongoDB
2.4K views7 slides
Search Data center Dallas | ColoCompare Data Center Marketplace by
Search Data center Dallas | ColoCompare Data Center MarketplaceSearch Data center Dallas | ColoCompare Data Center Marketplace
Search Data center Dallas | ColoCompare Data Center MarketplaceColo Compare
30 views7 slides

What's hot(20)

Building, and communicating, a knowledge graph in Zalando by Connected Data World
Building, and communicating, a knowledge graph in ZalandoBuilding, and communicating, a knowledge graph in Zalando
Building, and communicating, a knowledge graph in Zalando
Webinar: Introducing the MongoDB Connector for BI 2.0 with Tableau by MongoDB
Webinar: Introducing the MongoDB Connector for BI 2.0 with TableauWebinar: Introducing the MongoDB Connector for BI 2.0 with Tableau
Webinar: Introducing the MongoDB Connector for BI 2.0 with Tableau
MongoDB3.5K views
Beyond the Basics 3: Introduction to the MongoDB BI Connector by MongoDB
Beyond the Basics 3: Introduction to the MongoDB BI ConnectorBeyond the Basics 3: Introduction to the MongoDB BI Connector
Beyond the Basics 3: Introduction to the MongoDB BI Connector
MongoDB2.3K views
Using NoSQL and Enterprise Shared Services (ESS) to Achieve a More Efficient ... by MongoDB
Using NoSQL and Enterprise Shared Services (ESS) to Achieve a More Efficient ...Using NoSQL and Enterprise Shared Services (ESS) to Achieve a More Efficient ...
Using NoSQL and Enterprise Shared Services (ESS) to Achieve a More Efficient ...
MongoDB2.4K views
Search Data center Dallas | ColoCompare Data Center Marketplace by Colo Compare
Search Data center Dallas | ColoCompare Data Center MarketplaceSearch Data center Dallas | ColoCompare Data Center Marketplace
Search Data center Dallas | ColoCompare Data Center Marketplace
Colo Compare30 views
MongoDB .local Chicago 2019: Using MongoDB Transactions to Implement Cryptogr... by MongoDB
MongoDB .local Chicago 2019: Using MongoDB Transactions to Implement Cryptogr...MongoDB .local Chicago 2019: Using MongoDB Transactions to Implement Cryptogr...
MongoDB .local Chicago 2019: Using MongoDB Transactions to Implement Cryptogr...
MongoDB384 views
CData Data Today: A Developer's Dilemma by Jerod Johnson
CData Data Today: A Developer's DilemmaCData Data Today: A Developer's Dilemma
CData Data Today: A Developer's Dilemma
Jerod Johnson553 views
MongoDB.local Seattle 2019: Building Your First MongoDB App Using Atlas & Stitch by MongoDB
MongoDB.local Seattle 2019: Building Your First MongoDB App Using Atlas & StitchMongoDB.local Seattle 2019: Building Your First MongoDB App Using Atlas & Stitch
MongoDB.local Seattle 2019: Building Your First MongoDB App Using Atlas & Stitch
MongoDB177 views
Using a Semantic and Graph-based Data Catalog in a Modern Data Fabric by Cambridge Semantics
Using a Semantic and Graph-based Data Catalog in a Modern Data FabricUsing a Semantic and Graph-based Data Catalog in a Modern Data Fabric
Using a Semantic and Graph-based Data Catalog in a Modern Data Fabric
Webinar: Live Data Visualisation with Tableau and MongoDB by MongoDB
Webinar: Live Data Visualisation with Tableau and MongoDBWebinar: Live Data Visualisation with Tableau and MongoDB
Webinar: Live Data Visualisation with Tableau and MongoDB
MongoDB10.2K views
Data Center, Data Center Comparison by Colo Compare
Data Center, Data Center ComparisonData Center, Data Center Comparison
Data Center, Data Center Comparison
Colo Compare97 views
SlamData - How MongoDB Is Powering a Revolution in Visual Analytics by John De Goes
SlamData - How MongoDB Is Powering a Revolution in Visual AnalyticsSlamData - How MongoDB Is Powering a Revolution in Visual Analytics
SlamData - How MongoDB Is Powering a Revolution in Visual Analytics
John De Goes3.7K views
APIs, Web Services, and Mashups: What they are and how they can be used by snackeru
APIs, Web Services, and Mashups: What they are and how they can be usedAPIs, Web Services, and Mashups: What they are and how they can be used
APIs, Web Services, and Mashups: What they are and how they can be used
snackeru2.5K views
MongoDB .local Munich 2019: A Complete Methodology to Data Modeling for MongoDB by MongoDB
MongoDB .local Munich 2019: A Complete Methodology to Data Modeling for MongoDBMongoDB .local Munich 2019: A Complete Methodology to Data Modeling for MongoDB
MongoDB .local Munich 2019: A Complete Methodology to Data Modeling for MongoDB
MongoDB331 views
Jumpstart: Introduction to MongoDB by MongoDB
Jumpstart: Introduction to MongoDBJumpstart: Introduction to MongoDB
Jumpstart: Introduction to MongoDB
MongoDB509 views
Tlantic @ ElasticSearch POA Meetup by Vinicius Linck
Tlantic @ ElasticSearch POA MeetupTlantic @ ElasticSearch POA Meetup
Tlantic @ ElasticSearch POA Meetup
Vinicius Linck926 views
The Right (and Wrong) Use Cases for MongoDB by MongoDB
The Right (and Wrong) Use Cases for MongoDBThe Right (and Wrong) Use Cases for MongoDB
The Right (and Wrong) Use Cases for MongoDB
MongoDB13.4K views
MongoDB Evenings Minneapolis: MongoDB is Cool But When Should I Use It? by MongoDB
MongoDB Evenings Minneapolis: MongoDB is Cool But When Should I Use It?MongoDB Evenings Minneapolis: MongoDB is Cool But When Should I Use It?
MongoDB Evenings Minneapolis: MongoDB is Cool But When Should I Use It?
MongoDB1.2K views

Similar to Transforming your application with Elasticsearch

Docker Summit MongoDB - Data Democratization by
Docker Summit MongoDB - Data Democratization Docker Summit MongoDB - Data Democratization
Docker Summit MongoDB - Data Democratization Chris Grabosky
91 views37 slides
The State of the Data Warehouse in 2017 and Beyond by
The State of the Data Warehouse in 2017 and BeyondThe State of the Data Warehouse in 2017 and Beyond
The State of the Data Warehouse in 2017 and BeyondSingleStore
1.1K views42 slides
APIs and Linked Data: A match made in Heaven by
APIs and Linked Data: A match made in HeavenAPIs and Linked Data: A match made in Heaven
APIs and Linked Data: A match made in HeavenMichael Petychakis
1.7K views29 slides
Stephen Buxton | Data Integration - a Multi-Model Approach - Documents and Tr... by
Stephen Buxton | Data Integration - a Multi-Model Approach - Documents and Tr...Stephen Buxton | Data Integration - a Multi-Model Approach - Documents and Tr...
Stephen Buxton | Data Integration - a Multi-Model Approach - Documents and Tr...semanticsconference
643 views22 slides
Introduction to question answering for linked data & big data by
Introduction to question answering for linked data & big dataIntroduction to question answering for linked data & big data
Introduction to question answering for linked data & big dataAndre Freitas
16.6K views193 slides
The Power of Data by
The Power of DataThe Power of Data
The Power of DataDataWorks Summit
131 views31 slides

Similar to Transforming your application with Elasticsearch(20)

Docker Summit MongoDB - Data Democratization by Chris Grabosky
Docker Summit MongoDB - Data Democratization Docker Summit MongoDB - Data Democratization
Docker Summit MongoDB - Data Democratization
Chris Grabosky91 views
The State of the Data Warehouse in 2017 and Beyond by SingleStore
The State of the Data Warehouse in 2017 and BeyondThe State of the Data Warehouse in 2017 and Beyond
The State of the Data Warehouse in 2017 and Beyond
SingleStore1.1K views
APIs and Linked Data: A match made in Heaven by Michael Petychakis
APIs and Linked Data: A match made in HeavenAPIs and Linked Data: A match made in Heaven
APIs and Linked Data: A match made in Heaven
Michael Petychakis1.7K views
Stephen Buxton | Data Integration - a Multi-Model Approach - Documents and Tr... by semanticsconference
Stephen Buxton | Data Integration - a Multi-Model Approach - Documents and Tr...Stephen Buxton | Data Integration - a Multi-Model Approach - Documents and Tr...
Stephen Buxton | Data Integration - a Multi-Model Approach - Documents and Tr...
Introduction to question answering for linked data & big data by Andre Freitas
Introduction to question answering for linked data & big dataIntroduction to question answering for linked data & big data
Introduction to question answering for linked data & big data
Andre Freitas16.6K views
Making Sense of Schema on Read by Kent Graziano
Making Sense of Schema on ReadMaking Sense of Schema on Read
Making Sense of Schema on Read
Kent Graziano10.4K views
Business Intelligence Solution Using Search Engine by ankur881120
Business Intelligence Solution Using Search EngineBusiness Intelligence Solution Using Search Engine
Business Intelligence Solution Using Search Engine
ankur8811201.5K views
Job Data Analysis Reveals Key Skills Required for Data Scientists by JobsPikr
Job Data Analysis Reveals Key Skills Required for Data ScientistsJob Data Analysis Reveals Key Skills Required for Data Scientists
Job Data Analysis Reveals Key Skills Required for Data Scientists
JobsPikr624 views
Neo4j Aura on AWS: The Customer Choice for Graph Databases by Neo4j
Neo4j Aura on AWS: The Customer Choice for Graph DatabasesNeo4j Aura on AWS: The Customer Choice for Graph Databases
Neo4j Aura on AWS: The Customer Choice for Graph Databases
Neo4j256 views
Real-time big data analytics based on product recommendations case study by deep.bi
Real-time big data analytics based on product recommendations case studyReal-time big data analytics based on product recommendations case study
Real-time big data analytics based on product recommendations case study
deep.bi2.3K views
Analytical Innovation: How to Build the Next Generation Data Platform by VMware Tanzu
Analytical Innovation: How to Build the Next Generation Data PlatformAnalytical Innovation: How to Build the Next Generation Data Platform
Analytical Innovation: How to Build the Next Generation Data Platform
VMware Tanzu1K views
Test Trend Analysis : Towards robust, reliable and timely tests by Hugh McCamphill
Test Trend Analysis : Towards robust, reliable and timely testsTest Trend Analysis : Towards robust, reliable and timely tests
Test Trend Analysis : Towards robust, reliable and timely tests
Hugh McCamphill231 views
Koneksys - Offering Services to Connect Data using the Data Web by Koneksys
Koneksys - Offering Services to Connect Data using the Data WebKoneksys - Offering Services to Connect Data using the Data Web
Koneksys - Offering Services to Connect Data using the Data Web
Koneksys3.9K views
API Management and OAuth for Web, Mobile and the Cloud: Scott Morrison's Pres... by CA API Management
API Management and OAuth for Web, Mobile and the Cloud: Scott Morrison's Pres...API Management and OAuth for Web, Mobile and the Cloud: Scott Morrison's Pres...
API Management and OAuth for Web, Mobile and the Cloud: Scott Morrison's Pres...
CA API Management666 views
A search engine in a world of events and microservices - SF Pot @Meetic by meeticTech
A search engine in a world of events and microservices - SF Pot @MeeticA search engine in a world of events and microservices - SF Pot @Meetic
A search engine in a world of events and microservices - SF Pot @Meetic
meeticTech3.2K views
Introduction: Relational to Graphs by Neo4j
Introduction: Relational to GraphsIntroduction: Relational to Graphs
Introduction: Relational to Graphs
Neo4j2.2K views
Lies you have been told about REST by darrelmiller71
Lies you have been told about RESTLies you have been told about REST
Lies you have been told about REST
darrelmiller71339 views
Accelerate Big Data Application Development with Cascading by Cascading
Accelerate Big Data Application Development with CascadingAccelerate Big Data Application Development with Cascading
Accelerate Big Data Application Development with Cascading
Cascading921 views
Big Data Paris v 9.0 I 'The Search Engine for Structured Data' - Emmanuel Dau... by Dataconomy Media
Big Data Paris v 9.0 I 'The Search Engine for Structured Data' - Emmanuel Dau...Big Data Paris v 9.0 I 'The Search Engine for Structured Data' - Emmanuel Dau...
Big Data Paris v 9.0 I 'The Search Engine for Structured Data' - Emmanuel Dau...
Dataconomy Media91 views

More from Brian Ritchie

Building Event-Driven Systems with Apache Kafka by
Building Event-Driven Systems with Apache KafkaBuilding Event-Driven Systems with Apache Kafka
Building Event-Driven Systems with Apache KafkaBrian Ritchie
8.7K views33 slides
From Dev to Ops:Delivering an API to Production with Splunk by
From Dev to Ops:Delivering an API to Production with SplunkFrom Dev to Ops:Delivering an API to Production with Splunk
From Dev to Ops:Delivering an API to Production with SplunkBrian Ritchie
866 views26 slides
Extending the Enterprise with MEF by
Extending the Enterprise with MEFExtending the Enterprise with MEF
Extending the Enterprise with MEFBrian Ritchie
1.9K views24 slides
CQRS: Command/Query Responsibility Segregation by
CQRS: Command/Query Responsibility SegregationCQRS: Command/Query Responsibility Segregation
CQRS: Command/Query Responsibility SegregationBrian Ritchie
15.1K views41 slides
IIS Always-On Services by
IIS Always-On ServicesIIS Always-On Services
IIS Always-On ServicesBrian Ritchie
5.3K views26 slides
Scaling Out .NET by
Scaling Out .NETScaling Out .NET
Scaling Out .NETBrian Ritchie
4.1K views26 slides

More from Brian Ritchie(7)

Building Event-Driven Systems with Apache Kafka by Brian Ritchie
Building Event-Driven Systems with Apache KafkaBuilding Event-Driven Systems with Apache Kafka
Building Event-Driven Systems with Apache Kafka
Brian Ritchie8.7K views
From Dev to Ops:Delivering an API to Production with Splunk by Brian Ritchie
From Dev to Ops:Delivering an API to Production with SplunkFrom Dev to Ops:Delivering an API to Production with Splunk
From Dev to Ops:Delivering an API to Production with Splunk
Brian Ritchie866 views
Extending the Enterprise with MEF by Brian Ritchie
Extending the Enterprise with MEFExtending the Enterprise with MEF
Extending the Enterprise with MEF
Brian Ritchie1.9K views
CQRS: Command/Query Responsibility Segregation by Brian Ritchie
CQRS: Command/Query Responsibility SegregationCQRS: Command/Query Responsibility Segregation
CQRS: Command/Query Responsibility Segregation
Brian Ritchie15.1K views
IIS Always-On Services by Brian Ritchie
IIS Always-On ServicesIIS Always-On Services
IIS Always-On Services
Brian Ritchie5.3K views
Document Databases & RavenDB by Brian Ritchie
Document Databases & RavenDBDocument Databases & RavenDB
Document Databases & RavenDB
Brian Ritchie4.7K views

Recently uploaded

Keynote Talk: Open Source is Not Dead - Charles Schulz - Vates by
Keynote Talk: Open Source is Not Dead - Charles Schulz - VatesKeynote Talk: Open Source is Not Dead - Charles Schulz - Vates
Keynote Talk: Open Source is Not Dead - Charles Schulz - VatesShapeBlue
252 views15 slides
Webinar : Desperately Seeking Transformation - Part 2: Insights from leading... by
Webinar : Desperately Seeking Transformation - Part 2:  Insights from leading...Webinar : Desperately Seeking Transformation - Part 2:  Insights from leading...
Webinar : Desperately Seeking Transformation - Part 2: Insights from leading...The Digital Insurer
90 views52 slides
Updates on the LINSTOR Driver for CloudStack - Rene Peinthor - LINBIT by
Updates on the LINSTOR Driver for CloudStack - Rene Peinthor - LINBITUpdates on the LINSTOR Driver for CloudStack - Rene Peinthor - LINBIT
Updates on the LINSTOR Driver for CloudStack - Rene Peinthor - LINBITShapeBlue
206 views8 slides
CloudStack and GitOps at Enterprise Scale - Alex Dometrius, Rene Glover - AT&T by
CloudStack and GitOps at Enterprise Scale - Alex Dometrius, Rene Glover - AT&TCloudStack and GitOps at Enterprise Scale - Alex Dometrius, Rene Glover - AT&T
CloudStack and GitOps at Enterprise Scale - Alex Dometrius, Rene Glover - AT&TShapeBlue
152 views34 slides
Ransomware is Knocking your Door_Final.pdf by
Ransomware is Knocking your Door_Final.pdfRansomware is Knocking your Door_Final.pdf
Ransomware is Knocking your Door_Final.pdfSecurity Bootcamp
96 views46 slides
Declarative Kubernetes Cluster Deployment with Cloudstack and Cluster API - O... by
Declarative Kubernetes Cluster Deployment with Cloudstack and Cluster API - O...Declarative Kubernetes Cluster Deployment with Cloudstack and Cluster API - O...
Declarative Kubernetes Cluster Deployment with Cloudstack and Cluster API - O...ShapeBlue
132 views13 slides

Recently uploaded(20)

Keynote Talk: Open Source is Not Dead - Charles Schulz - Vates by ShapeBlue
Keynote Talk: Open Source is Not Dead - Charles Schulz - VatesKeynote Talk: Open Source is Not Dead - Charles Schulz - Vates
Keynote Talk: Open Source is Not Dead - Charles Schulz - Vates
ShapeBlue252 views
Webinar : Desperately Seeking Transformation - Part 2: Insights from leading... by The Digital Insurer
Webinar : Desperately Seeking Transformation - Part 2:  Insights from leading...Webinar : Desperately Seeking Transformation - Part 2:  Insights from leading...
Webinar : Desperately Seeking Transformation - Part 2: Insights from leading...
Updates on the LINSTOR Driver for CloudStack - Rene Peinthor - LINBIT by ShapeBlue
Updates on the LINSTOR Driver for CloudStack - Rene Peinthor - LINBITUpdates on the LINSTOR Driver for CloudStack - Rene Peinthor - LINBIT
Updates on the LINSTOR Driver for CloudStack - Rene Peinthor - LINBIT
ShapeBlue206 views
CloudStack and GitOps at Enterprise Scale - Alex Dometrius, Rene Glover - AT&T by ShapeBlue
CloudStack and GitOps at Enterprise Scale - Alex Dometrius, Rene Glover - AT&TCloudStack and GitOps at Enterprise Scale - Alex Dometrius, Rene Glover - AT&T
CloudStack and GitOps at Enterprise Scale - Alex Dometrius, Rene Glover - AT&T
ShapeBlue152 views
Declarative Kubernetes Cluster Deployment with Cloudstack and Cluster API - O... by ShapeBlue
Declarative Kubernetes Cluster Deployment with Cloudstack and Cluster API - O...Declarative Kubernetes Cluster Deployment with Cloudstack and Cluster API - O...
Declarative Kubernetes Cluster Deployment with Cloudstack and Cluster API - O...
ShapeBlue132 views
Business Analyst Series 2023 - Week 4 Session 8 by DianaGray10
Business Analyst Series 2023 -  Week 4 Session 8Business Analyst Series 2023 -  Week 4 Session 8
Business Analyst Series 2023 - Week 4 Session 8
DianaGray10123 views
Elevating Privacy and Security in CloudStack - Boris Stoyanov - ShapeBlue by ShapeBlue
Elevating Privacy and Security in CloudStack - Boris Stoyanov - ShapeBlueElevating Privacy and Security in CloudStack - Boris Stoyanov - ShapeBlue
Elevating Privacy and Security in CloudStack - Boris Stoyanov - ShapeBlue
ShapeBlue222 views
iSAQB Software Architecture Gathering 2023: How Process Orchestration Increas... by Bernd Ruecker
iSAQB Software Architecture Gathering 2023: How Process Orchestration Increas...iSAQB Software Architecture Gathering 2023: How Process Orchestration Increas...
iSAQB Software Architecture Gathering 2023: How Process Orchestration Increas...
Bernd Ruecker54 views
2FA and OAuth2 in CloudStack - Andrija Panić - ShapeBlue by ShapeBlue
2FA and OAuth2 in CloudStack - Andrija Panić - ShapeBlue2FA and OAuth2 in CloudStack - Andrija Panić - ShapeBlue
2FA and OAuth2 in CloudStack - Andrija Panić - ShapeBlue
ShapeBlue147 views
Centralized Logging Feature in CloudStack using ELK and Grafana - Kiran Chava... by ShapeBlue
Centralized Logging Feature in CloudStack using ELK and Grafana - Kiran Chava...Centralized Logging Feature in CloudStack using ELK and Grafana - Kiran Chava...
Centralized Logging Feature in CloudStack using ELK and Grafana - Kiran Chava...
ShapeBlue145 views
What’s New in CloudStack 4.19 - Abhishek Kumar - ShapeBlue by ShapeBlue
What’s New in CloudStack 4.19 - Abhishek Kumar - ShapeBlueWhat’s New in CloudStack 4.19 - Abhishek Kumar - ShapeBlue
What’s New in CloudStack 4.19 - Abhishek Kumar - ShapeBlue
ShapeBlue263 views
TrustArc Webinar - Managing Online Tracking Technology Vendors_ A Checklist f... by TrustArc
TrustArc Webinar - Managing Online Tracking Technology Vendors_ A Checklist f...TrustArc Webinar - Managing Online Tracking Technology Vendors_ A Checklist f...
TrustArc Webinar - Managing Online Tracking Technology Vendors_ A Checklist f...
TrustArc170 views
CloudStack Object Storage - An Introduction - Vladimir Petrov - ShapeBlue by ShapeBlue
CloudStack Object Storage - An Introduction - Vladimir Petrov - ShapeBlueCloudStack Object Storage - An Introduction - Vladimir Petrov - ShapeBlue
CloudStack Object Storage - An Introduction - Vladimir Petrov - ShapeBlue
ShapeBlue138 views
Live Demo Showcase: Unveiling Dell PowerFlex’s IaaS Capabilities with Apache ... by ShapeBlue
Live Demo Showcase: Unveiling Dell PowerFlex’s IaaS Capabilities with Apache ...Live Demo Showcase: Unveiling Dell PowerFlex’s IaaS Capabilities with Apache ...
Live Demo Showcase: Unveiling Dell PowerFlex’s IaaS Capabilities with Apache ...
ShapeBlue126 views
Business Analyst Series 2023 - Week 4 Session 7 by DianaGray10
Business Analyst Series 2023 -  Week 4 Session 7Business Analyst Series 2023 -  Week 4 Session 7
Business Analyst Series 2023 - Week 4 Session 7
DianaGray10139 views
Migrating VMware Infra to KVM Using CloudStack - Nicolas Vazquez - ShapeBlue by ShapeBlue
Migrating VMware Infra to KVM Using CloudStack - Nicolas Vazquez - ShapeBlueMigrating VMware Infra to KVM Using CloudStack - Nicolas Vazquez - ShapeBlue
Migrating VMware Infra to KVM Using CloudStack - Nicolas Vazquez - ShapeBlue
ShapeBlue218 views
KVM Security Groups Under the Hood - Wido den Hollander - Your.Online by ShapeBlue
KVM Security Groups Under the Hood - Wido den Hollander - Your.OnlineKVM Security Groups Under the Hood - Wido den Hollander - Your.Online
KVM Security Groups Under the Hood - Wido den Hollander - Your.Online
ShapeBlue221 views
Mitigating Common CloudStack Instance Deployment Failures - Jithin Raju - Sha... by ShapeBlue
Mitigating Common CloudStack Instance Deployment Failures - Jithin Raju - Sha...Mitigating Common CloudStack Instance Deployment Failures - Jithin Raju - Sha...
Mitigating Common CloudStack Instance Deployment Failures - Jithin Raju - Sha...
ShapeBlue180 views

Transforming your application with Elasticsearch

  • 1. TRANSFORMING YOUR APPLICATION WITH BRIAN RITCHIE CTO, XEOHEALTH 2018 @brian_ritchie brian.ritchie@gmail.com http://www.dotnetpowered.com
  • 2. TRANSFORMING YOUR APPLICATION WITH ELASTIC SEARCH Brian Ritchie CTO for XeoHealth Corporation 22 years of technology experience developing software and managing technology teams in various industries. Author of RavenDB High Performance published by PACKT in 2013. Pursuing a Masters in Computer Science from Georgia Tech. @brian_ritchie brian.ritchie@gmail.com http://www.dotnetpowered.com
  • 3. TRANSFORMING YOUR APPLICATION WITH ELASTIC SEARCH WELCOME INTRODUCTION TO ELASTIC SEARCH BRING YOUR DATA SECURITY CONSIDERATIONS PUTTING IT ALL TOGETHER
  • 4. TRANSFORMING YOUR APPLICATION WITH ELASTIC SEARCH LET’S GET STARTED With the story of your application…
  • 5. TRANSFORMING YOUR APPLICATION WITH ELASTIC SEARCH IT WAS AMAZING… It even had the new application smell.
  • 6. TRANSFORMING YOUR APPLICATION WITH ELASTIC SEARCH BUT THEN CAME THE USERS, THE DATA, AND THE NEW REQUIREMENTS….
  • 7. TRANSFORMING YOUR APPLICATION WITH ELASTIC SEARCH NOW YOUR APPLICATION IS LEAVING YOU STRANDED ON THE SIDE OF THE ROAD…
  • 8. TRANSFORMING YOUR APPLICATION WITH ELASTIC SEARCH SO, WHY IS YOUR APPLICATION MELTING? Too Many Reads, Too Many Writes = Contention
  • 9. TRANSFORMING YOUR APPLICATION WITH ELASTIC SEARCH IS THIS THE ONLY FUTURE FOR YOUR APPLICATION?
  • 10. TRANSFORMING YOUR APPLICATION WITH ELASTIC SEARCH NO WAY! IT’S TIME TO PIMP YOUR RIDE!
  • 11. TRANSFORMING YOUR APPLICATION WITH ELASTIC SEARCH INTRODUCING
  • 12. TRANSFORMING YOUR APPLICATION WITH ELASTIC SEARCH INTRODUCING Elasticsearch is a search engine based on Lucene. It provides a lightening fast, distributed, multitenant-capable full-text search engine with a HTTP RESTful interface. Elasticsearch is released as open source under the terms of the Apache License. It can be deployed locally or hosted in the cloud.
  • 13. TRANSFORMING YOUR APPLICATION WITH ELASTIC SEARCH INTRODUCING Well known for being part of the Stack The Elastic StackNote: this was rebranded used to analyze log data generated by your application
  • 14. TRANSFORMING YOUR APPLICATION WITH ELASTIC SEARCH WHY FAST Document Centric Shard/ Partition Index Everything Extremely Fast. Optimized for search. Flexible Schema. Distributed Cluster FOR YOUR APPLICATION
  • 15. TRANSFORMING YOUR APPLICATION WITH ELASTIC SEARCH WHY SCALABLE “We can support clusters of massive scale. Well into the 100s of terabytes. One of our largest customers is currently at 750TB and should be in excess of 1PB by the end of the year. “ - Scalefastr “Today there are 60+ Elasticsearch clusters and 2000+ nodes. The daily ingestion reaches 18 billion documents, and daily search requests reach 3.5 billion. ” - eBay FOR YOUR APPLICATION
  • 16. TRANSFORMING YOUR APPLICATION WITH ELASTIC SEARCH YOUR USERS ON
  • 17. TRANSFORMING YOUR APPLICATION WITH ELASTIC SEARCH That sounds awesome!! But, how does Elastic Search fit into my data infrastructure? BRINGING YOUR DATA INTO
  • 18. TRANSFORMING YOUR APPLICATION WITH ELASTIC SEARCH Utilizing a CQRS inspired pattern BRINGING YOUR DATA INTO ELASTIC SEARCH Command/Query Responsibility Segregation (CQRS) is the idea that you can use a different model to update information than the model you use to read information Persistent View Model Transactional Data
  • 19. TRANSFORMING YOUR APPLICATION WITH ELASTIC SEARCH OPTION 1: Add Elastic search into your database pipeline cluster Kafka ActiveMQ RabbitMQ …. BRINGING YOUR DATA INTO ELASTIC SEARCH
  • 20. TRANSFORMING YOUR APPLICATION WITH ELASTIC SEARCH clusterKafka ActiveMQ RabbitMQ …. OPTION 2: Replicate database updates to Elastic Search BRINGING YOUR DATA INTO ELASTIC SEARCH
  • 21. TRANSFORMING YOUR APPLICATION WITH ELASTIC SEARCH cluster OPTION 3: Batch your database updates on a schedule BRINGING YOUR DATA INTO ELASTIC SEARCH
  • 22. TRANSFORMING YOUR APPLICATION WITH ELASTIC SEARCH Structuring your data for search DATA MODELING Conceptually, indexes are like databases. • SQL => Databases => Tables => Columns/Rows • Elasticsearch => Indices => Types => Documents with Properties Loans Payments Inventory
  • 23. TRANSFORMING YOUR APPLICATION WITH ELASTIC SEARCH Defining indexes DATA MODELING: Like databases, but more powerful. Queries can easily span indexes bringing data together as needed. Use indexes for: • Security – use indexes to support multi-tenant system loan_hud loan_fdic • Partitioning – time or other data log_20180501 log_20180502
  • 24. TRANSFORMING YOUR APPLICATION WITH ELASTIC SEARCH Defining documents DATA MODELING: { "FirstName": "Jonathan", "Address": "15 Wanamassa Point Road ", "City": "Jacksonville", "State": "FL", "Children": [ { "Name": "Michael", "Age": 10 }, { "Name": "Jennifer", "Age": 8 }, { "Name": "Samantha", "Age": 5 }, { "Name": "Elena", "Age": 2 } ] } Instead of decomposing data into separate entities: Design documents based on what needs to be searched and displayed together:
  • 25. TRANSFORMING YOUR APPLICATION WITH ELASTIC SEARCH Teach Elastic Search about specific types DATA MODELING: PUT _template/file_name_mapping { "template": ”loan_*", "settings": { "number_of_shards": 5 }, "mappings": { "_default_": { "dynamic_templates": [ { "filename_string_not_analyzed": { "match": "edi_filename", "mapping": { "index": "not_analyzed", "type": "string” } } } ] } } }
  • 26. TRANSFORMING YOUR APPLICATION WITH ELASTIC SEARCH Giving the right users access to the right data SECURITY CONSIDERATIONS cluster Search Guard Custom .NET Core / Java / etc. Elastic Stack Security
  • 27. TRANSFORMING YOUR APPLICATION WITH ELASTIC SEARCH PUTTING IT ALL TOGETHER The Home Mortgage Disclosure Act (HMDA) requires many financial institutions to maintain, report, and publicly disclose loan-level information about mortgages. Searching CFPB loan records w/ Elastic Search
  • 28. TRANSFORMING YOUR APPLICATION WITH ELASTIC SEARCH PUTTING IT ALL TOGETHER 107 million loan records var record = csv.GetRecord<dynamic>(); var loan = Loan.FromFileLoan(record); var index_name = string.Format("loan_{0}", loan.agency_abbr.ToLower()); var response = client.Index<Loan>( new IndexRequest<Loan>(loan, index_name)); Loading CFPB loan records into Elastic Search { "county": "Maricopa County", "respondent_id": "0000451965", "lien_status": "Not applicable", "sequence_number": "0945293", "applicant_income": 132000, "preapproval": "Not applicable", "applicant_race": "White", "msamd": "Phoenix, Mesa, Scottsdale - AZ", ….
  • 29. TRANSFORMING YOUR APPLICATION WITH ELASTIC SEARCH Using Kibana to search and visualize your data PUTTING IT ALL TOGETHER
  • 30. TRANSFORMING YOUR APPLICATION WITH ELASTIC SEARCH Using Elastic Search to optimize your user experience PUTTING IT ALL TOGETHER cluster Single Page Application (SPA) Angular / React / etc. JWT Query Request REST API .NET Core / Java / etc. Add security, logging, etc.
  • 31. TRANSFORMING YOUR APPLICATION WITH ELASTIC SEARCH Custom application using Angular, PrimeNG, .NET Core, and Elastic Search PUTTING IT ALL TOGETHER