Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
TRANSFORMING
YOUR APPLICATION WITH
BRIAN RITCHIE
CTO, XEOHEALTH
2018
@brian_ritchie
brian.ritchie@gmail.com
http://www.dot...
TRANSFORMING YOUR APPLICATION WITH ELASTIC SEARCH
Brian Ritchie
CTO for XeoHealth Corporation
22 years of technology exper...
TRANSFORMING YOUR APPLICATION WITH ELASTIC SEARCH
WELCOME
INTRODUCTION TO ELASTIC SEARCH
BRING YOUR DATA
SECURITY CONSIDER...
TRANSFORMING YOUR APPLICATION WITH ELASTIC SEARCH
LET’S GET STARTED
With the story of your application…
TRANSFORMING YOUR APPLICATION WITH ELASTIC SEARCH
IT WAS AMAZING…
It even had the new application smell.
TRANSFORMING YOUR APPLICATION WITH ELASTIC SEARCH
BUT THEN CAME THE USERS, THE DATA, AND THE
NEW REQUIREMENTS….
TRANSFORMING YOUR APPLICATION WITH ELASTIC SEARCH
NOW YOUR APPLICATION IS LEAVING YOU STRANDED
ON THE SIDE OF THE ROAD…
TRANSFORMING YOUR APPLICATION WITH ELASTIC SEARCH
SO, WHY IS YOUR APPLICATION MELTING?
Too Many Reads,
Too Many Writes
= C...
TRANSFORMING YOUR APPLICATION WITH ELASTIC SEARCH
IS THIS THE ONLY FUTURE FOR YOUR APPLICATION?
TRANSFORMING YOUR APPLICATION WITH ELASTIC SEARCH
NO WAY! IT’S TIME TO PIMP YOUR RIDE!
TRANSFORMING YOUR APPLICATION WITH ELASTIC SEARCH
INTRODUCING
TRANSFORMING YOUR APPLICATION WITH ELASTIC SEARCH
INTRODUCING
Elasticsearch is a search engine based
on Lucene. It provide...
TRANSFORMING YOUR APPLICATION WITH ELASTIC SEARCH
INTRODUCING
Well known for
being part of the
Stack
The Elastic StackNote...
TRANSFORMING YOUR APPLICATION WITH ELASTIC SEARCH
WHY
FAST
Document
Centric
Shard/
Partition
Index
Everything
Extremely Fa...
TRANSFORMING YOUR APPLICATION WITH ELASTIC SEARCH
WHY
SCALABLE “We can support clusters of massive scale. Well
into the 10...
TRANSFORMING YOUR APPLICATION WITH ELASTIC SEARCH
YOUR USERS ON
TRANSFORMING YOUR APPLICATION WITH ELASTIC SEARCH
That sounds awesome!!
But, how does Elastic Search fit
into my data infr...
TRANSFORMING YOUR APPLICATION WITH ELASTIC SEARCH
Utilizing a CQRS inspired pattern
BRINGING YOUR DATA INTO ELASTIC SEARCH...
TRANSFORMING YOUR APPLICATION WITH ELASTIC SEARCH
OPTION 1: Add Elastic search into your database pipeline
cluster
Kafka
A...
TRANSFORMING YOUR APPLICATION WITH ELASTIC SEARCH
clusterKafka
ActiveMQ
RabbitMQ
….
OPTION 2: Replicate database updates t...
TRANSFORMING YOUR APPLICATION WITH ELASTIC SEARCH
cluster
OPTION 3: Batch your database updates on a schedule
BRINGING YOU...
TRANSFORMING YOUR APPLICATION WITH ELASTIC SEARCH
Structuring your data for search
DATA MODELING
Conceptually, indexes are...
TRANSFORMING YOUR APPLICATION WITH ELASTIC SEARCH
Defining indexes
DATA MODELING:
Like databases, but more powerful. Queri...
TRANSFORMING YOUR APPLICATION WITH ELASTIC SEARCH
Defining documents
DATA MODELING:
{
"FirstName": "Jonathan",
"Address": ...
TRANSFORMING YOUR APPLICATION WITH ELASTIC SEARCH
Teach Elastic Search about specific types
DATA MODELING:
PUT _template/f...
TRANSFORMING YOUR APPLICATION WITH ELASTIC SEARCH
Giving the right users access to the right data
SECURITY CONSIDERATIONS
...
TRANSFORMING YOUR APPLICATION WITH ELASTIC SEARCH
PUTTING IT ALL TOGETHER
The Home Mortgage Disclosure Act (HMDA) requires...
TRANSFORMING YOUR APPLICATION WITH ELASTIC SEARCH
PUTTING IT ALL TOGETHER
107 million
loan records
var record = csv.GetRec...
TRANSFORMING YOUR APPLICATION WITH ELASTIC SEARCH
Using Kibana to search and visualize your data
PUTTING IT ALL TOGETHER
TRANSFORMING YOUR APPLICATION WITH ELASTIC SEARCH
Using Elastic Search to optimize your user experience
PUTTING IT ALL TOG...
TRANSFORMING YOUR APPLICATION WITH ELASTIC SEARCH
Custom application using Angular, PrimeNG, .NET Core, and Elastic Search...
BRIAN RITCHIE
CTO, XEOHEALTH
2018
@brian_ritchie
brian.ritchie@gmail.com
http://www.dotnetpowered.com
QUESTIONS?
TRANSFORM...
Upcoming SlideShare
Loading in …5
×

Transforming your application with Elasticsearch

259 views

Published on

Have an existing application that needs search super powers? Or are ad hoc searches melting your SQL Server? Either way, this is the talk for you. We will explore search enabling an existing application - from data modeling to retrieval. In this talk, we will cover data modeling for search, connecting ElasticSearch to your data pipeline, building a search API and connecting to an Angular web site. Presented at Code on the Beach 2018

Published in: Technology
  • Be the first to comment

Transforming your application with Elasticsearch

  1. 1. TRANSFORMING YOUR APPLICATION WITH BRIAN RITCHIE CTO, XEOHEALTH 2018 @brian_ritchie brian.ritchie@gmail.com http://www.dotnetpowered.com
  2. 2. TRANSFORMING YOUR APPLICATION WITH ELASTIC SEARCH Brian Ritchie CTO for XeoHealth Corporation 22 years of technology experience developing software and managing technology teams in various industries. Author of RavenDB High Performance published by PACKT in 2013. Pursuing a Masters in Computer Science from Georgia Tech. @brian_ritchie brian.ritchie@gmail.com http://www.dotnetpowered.com
  3. 3. TRANSFORMING YOUR APPLICATION WITH ELASTIC SEARCH WELCOME INTRODUCTION TO ELASTIC SEARCH BRING YOUR DATA SECURITY CONSIDERATIONS PUTTING IT ALL TOGETHER
  4. 4. TRANSFORMING YOUR APPLICATION WITH ELASTIC SEARCH LET’S GET STARTED With the story of your application…
  5. 5. TRANSFORMING YOUR APPLICATION WITH ELASTIC SEARCH IT WAS AMAZING… It even had the new application smell.
  6. 6. TRANSFORMING YOUR APPLICATION WITH ELASTIC SEARCH BUT THEN CAME THE USERS, THE DATA, AND THE NEW REQUIREMENTS….
  7. 7. TRANSFORMING YOUR APPLICATION WITH ELASTIC SEARCH NOW YOUR APPLICATION IS LEAVING YOU STRANDED ON THE SIDE OF THE ROAD…
  8. 8. TRANSFORMING YOUR APPLICATION WITH ELASTIC SEARCH SO, WHY IS YOUR APPLICATION MELTING? Too Many Reads, Too Many Writes = Contention
  9. 9. TRANSFORMING YOUR APPLICATION WITH ELASTIC SEARCH IS THIS THE ONLY FUTURE FOR YOUR APPLICATION?
  10. 10. TRANSFORMING YOUR APPLICATION WITH ELASTIC SEARCH NO WAY! IT’S TIME TO PIMP YOUR RIDE!
  11. 11. TRANSFORMING YOUR APPLICATION WITH ELASTIC SEARCH INTRODUCING
  12. 12. TRANSFORMING YOUR APPLICATION WITH ELASTIC SEARCH INTRODUCING Elasticsearch is a search engine based on Lucene. It provides a lightening fast, distributed, multitenant-capable full-text search engine with a HTTP RESTful interface. Elasticsearch is released as open source under the terms of the Apache License. It can be deployed locally or hosted in the cloud.
  13. 13. TRANSFORMING YOUR APPLICATION WITH ELASTIC SEARCH INTRODUCING Well known for being part of the Stack The Elastic StackNote: this was rebranded used to analyze log data generated by your application
  14. 14. TRANSFORMING YOUR APPLICATION WITH ELASTIC SEARCH WHY FAST Document Centric Shard/ Partition Index Everything Extremely Fast. Optimized for search. Flexible Schema. Distributed Cluster FOR YOUR APPLICATION
  15. 15. TRANSFORMING YOUR APPLICATION WITH ELASTIC SEARCH WHY SCALABLE “We can support clusters of massive scale. Well into the 100s of terabytes. One of our largest customers is currently at 750TB and should be in excess of 1PB by the end of the year. “ - Scalefastr “Today there are 60+ Elasticsearch clusters and 2000+ nodes. The daily ingestion reaches 18 billion documents, and daily search requests reach 3.5 billion. ” - eBay FOR YOUR APPLICATION
  16. 16. TRANSFORMING YOUR APPLICATION WITH ELASTIC SEARCH YOUR USERS ON
  17. 17. TRANSFORMING YOUR APPLICATION WITH ELASTIC SEARCH That sounds awesome!! But, how does Elastic Search fit into my data infrastructure? BRINGING YOUR DATA INTO
  18. 18. TRANSFORMING YOUR APPLICATION WITH ELASTIC SEARCH Utilizing a CQRS inspired pattern BRINGING YOUR DATA INTO ELASTIC SEARCH Command/Query Responsibility Segregation (CQRS) is the idea that you can use a different model to update information than the model you use to read information Persistent View Model Transactional Data
  19. 19. TRANSFORMING YOUR APPLICATION WITH ELASTIC SEARCH OPTION 1: Add Elastic search into your database pipeline cluster Kafka ActiveMQ RabbitMQ …. BRINGING YOUR DATA INTO ELASTIC SEARCH
  20. 20. TRANSFORMING YOUR APPLICATION WITH ELASTIC SEARCH clusterKafka ActiveMQ RabbitMQ …. OPTION 2: Replicate database updates to Elastic Search BRINGING YOUR DATA INTO ELASTIC SEARCH
  21. 21. TRANSFORMING YOUR APPLICATION WITH ELASTIC SEARCH cluster OPTION 3: Batch your database updates on a schedule BRINGING YOUR DATA INTO ELASTIC SEARCH
  22. 22. TRANSFORMING YOUR APPLICATION WITH ELASTIC SEARCH Structuring your data for search DATA MODELING Conceptually, indexes are like databases. • SQL => Databases => Tables => Columns/Rows • Elasticsearch => Indices => Types => Documents with Properties Loans Payments Inventory
  23. 23. TRANSFORMING YOUR APPLICATION WITH ELASTIC SEARCH Defining indexes DATA MODELING: Like databases, but more powerful. Queries can easily span indexes bringing data together as needed. Use indexes for: • Security – use indexes to support multi-tenant system loan_hud loan_fdic • Partitioning – time or other data log_20180501 log_20180502
  24. 24. TRANSFORMING YOUR APPLICATION WITH ELASTIC SEARCH Defining documents DATA MODELING: { "FirstName": "Jonathan", "Address": "15 Wanamassa Point Road ", "City": "Jacksonville", "State": "FL", "Children": [ { "Name": "Michael", "Age": 10 }, { "Name": "Jennifer", "Age": 8 }, { "Name": "Samantha", "Age": 5 }, { "Name": "Elena", "Age": 2 } ] } Instead of decomposing data into separate entities: Design documents based on what needs to be searched and displayed together:
  25. 25. TRANSFORMING YOUR APPLICATION WITH ELASTIC SEARCH Teach Elastic Search about specific types DATA MODELING: PUT _template/file_name_mapping { "template": ”loan_*", "settings": { "number_of_shards": 5 }, "mappings": { "_default_": { "dynamic_templates": [ { "filename_string_not_analyzed": { "match": "edi_filename", "mapping": { "index": "not_analyzed", "type": "string” } } } ] } } }
  26. 26. TRANSFORMING YOUR APPLICATION WITH ELASTIC SEARCH Giving the right users access to the right data SECURITY CONSIDERATIONS cluster Search Guard Custom .NET Core / Java / etc. Elastic Stack Security
  27. 27. TRANSFORMING YOUR APPLICATION WITH ELASTIC SEARCH PUTTING IT ALL TOGETHER The Home Mortgage Disclosure Act (HMDA) requires many financial institutions to maintain, report, and publicly disclose loan-level information about mortgages. Searching CFPB loan records w/ Elastic Search
  28. 28. TRANSFORMING YOUR APPLICATION WITH ELASTIC SEARCH PUTTING IT ALL TOGETHER 107 million loan records var record = csv.GetRecord<dynamic>(); var loan = Loan.FromFileLoan(record); var index_name = string.Format("loan_{0}", loan.agency_abbr.ToLower()); var response = client.Index<Loan>( new IndexRequest<Loan>(loan, index_name)); Loading CFPB loan records into Elastic Search { "county": "Maricopa County", "respondent_id": "0000451965", "lien_status": "Not applicable", "sequence_number": "0945293", "applicant_income": 132000, "preapproval": "Not applicable", "applicant_race": "White", "msamd": "Phoenix, Mesa, Scottsdale - AZ", ….
  29. 29. TRANSFORMING YOUR APPLICATION WITH ELASTIC SEARCH Using Kibana to search and visualize your data PUTTING IT ALL TOGETHER
  30. 30. TRANSFORMING YOUR APPLICATION WITH ELASTIC SEARCH Using Elastic Search to optimize your user experience PUTTING IT ALL TOGETHER cluster Single Page Application (SPA) Angular / React / etc. JWT Query Request REST API .NET Core / Java / etc. Add security, logging, etc.
  31. 31. TRANSFORMING YOUR APPLICATION WITH ELASTIC SEARCH Custom application using Angular, PrimeNG, .NET Core, and Elastic Search PUTTING IT ALL TOGETHER
  32. 32. BRIAN RITCHIE CTO, XEOHEALTH 2018 @brian_ritchie brian.ritchie@gmail.com http://www.dotnetpowered.com QUESTIONS? TRANSFORMING YOUR APPLICATIONS WITH

×