Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Transforming your application with Elasticsearch

125 views

Published on

Have an existing application that needs search super powers? Or are ad hoc searches melting your SQL Server? Either way, this is the talk for you. We will explore search enabling an existing application - from data modeling to retrieval. In this talk, we will cover data modeling for search, connecting ElasticSearch to your data pipeline, building a search API and connecting to an Angular web site. Presented at Code on the Beach 2018

Published in: Technology
  • Be the first to comment

Transforming your application with Elasticsearch

  1. 1. TRANSFORMING YOUR APPLICATION WITH BRIAN RITCHIE CTO, XEOHEALTH 2018 @brian_ritchie brian.ritchie@gmail.com http://www.dotnetpowered.com
  2. 2. TRANSFORMING YOUR APPLICATION WITH ELASTIC SEARCH Brian Ritchie CTO for XeoHealth Corporation 22 years of technology experience developing software and managing technology teams in various industries. Author of RavenDB High Performance published by PACKT in 2013. Pursuing a Masters in Computer Science from Georgia Tech. @brian_ritchie brian.ritchie@gmail.com http://www.dotnetpowered.com
  3. 3. TRANSFORMING YOUR APPLICATION WITH ELASTIC SEARCH WELCOME INTRODUCTION TO ELASTIC SEARCH BRING YOUR DATA SECURITY CONSIDERATIONS PUTTING IT ALL TOGETHER
  4. 4. TRANSFORMING YOUR APPLICATION WITH ELASTIC SEARCH LET’S GET STARTED With the story of your application…
  5. 5. TRANSFORMING YOUR APPLICATION WITH ELASTIC SEARCH IT WAS AMAZING… It even had the new application smell.
  6. 6. TRANSFORMING YOUR APPLICATION WITH ELASTIC SEARCH BUT THEN CAME THE USERS, THE DATA, AND THE NEW REQUIREMENTS….
  7. 7. TRANSFORMING YOUR APPLICATION WITH ELASTIC SEARCH NOW YOUR APPLICATION IS LEAVING YOU STRANDED ON THE SIDE OF THE ROAD…
  8. 8. TRANSFORMING YOUR APPLICATION WITH ELASTIC SEARCH SO, WHY IS YOUR APPLICATION MELTING? Too Many Reads, Too Many Writes = Contention
  9. 9. TRANSFORMING YOUR APPLICATION WITH ELASTIC SEARCH IS THIS THE ONLY FUTURE FOR YOUR APPLICATION?
  10. 10. TRANSFORMING YOUR APPLICATION WITH ELASTIC SEARCH NO WAY! IT’S TIME TO PIMP YOUR RIDE!
  11. 11. TRANSFORMING YOUR APPLICATION WITH ELASTIC SEARCH INTRODUCING
  12. 12. TRANSFORMING YOUR APPLICATION WITH ELASTIC SEARCH INTRODUCING Elasticsearch is a search engine based on Lucene. It provides a lightening fast, distributed, multitenant-capable full-text search engine with a HTTP RESTful interface. Elasticsearch is released as open source under the terms of the Apache License. It can be deployed locally or hosted in the cloud.
  13. 13. TRANSFORMING YOUR APPLICATION WITH ELASTIC SEARCH INTRODUCING Well known for being part of the Stack The Elastic StackNote: this was rebranded used to analyze log data generated by your application
  14. 14. TRANSFORMING YOUR APPLICATION WITH ELASTIC SEARCH WHY FAST Document Centric Shard/ Partition Index Everything Extremely Fast. Optimized for search. Flexible Schema. Distributed Cluster FOR YOUR APPLICATION
  15. 15. TRANSFORMING YOUR APPLICATION WITH ELASTIC SEARCH WHY SCALABLE “We can support clusters of massive scale. Well into the 100s of terabytes. One of our largest customers is currently at 750TB and should be in excess of 1PB by the end of the year. “ - Scalefastr “Today there are 60+ Elasticsearch clusters and 2000+ nodes. The daily ingestion reaches 18 billion documents, and daily search requests reach 3.5 billion. ” - eBay FOR YOUR APPLICATION
  16. 16. TRANSFORMING YOUR APPLICATION WITH ELASTIC SEARCH YOUR USERS ON
  17. 17. TRANSFORMING YOUR APPLICATION WITH ELASTIC SEARCH That sounds awesome!! But, how does Elastic Search fit into my data infrastructure? BRINGING YOUR DATA INTO
  18. 18. TRANSFORMING YOUR APPLICATION WITH ELASTIC SEARCH Utilizing a CQRS inspired pattern BRINGING YOUR DATA INTO ELASTIC SEARCH Command/Query Responsibility Segregation (CQRS) is the idea that you can use a different model to update information than the model you use to read information Persistent View Model Transactional Data
  19. 19. TRANSFORMING YOUR APPLICATION WITH ELASTIC SEARCH OPTION 1: Add Elastic search into your database pipeline cluster Kafka ActiveMQ RabbitMQ …. BRINGING YOUR DATA INTO ELASTIC SEARCH
  20. 20. TRANSFORMING YOUR APPLICATION WITH ELASTIC SEARCH clusterKafka ActiveMQ RabbitMQ …. OPTION 2: Replicate database updates to Elastic Search BRINGING YOUR DATA INTO ELASTIC SEARCH
  21. 21. TRANSFORMING YOUR APPLICATION WITH ELASTIC SEARCH cluster OPTION 3: Batch your database updates on a schedule BRINGING YOUR DATA INTO ELASTIC SEARCH
  22. 22. TRANSFORMING YOUR APPLICATION WITH ELASTIC SEARCH Structuring your data for search DATA MODELING Conceptually, indexes are like databases. • SQL => Databases => Tables => Columns/Rows • Elasticsearch => Indices => Types => Documents with Properties Loans Payments Inventory
  23. 23. TRANSFORMING YOUR APPLICATION WITH ELASTIC SEARCH Defining indexes DATA MODELING: Like databases, but more powerful. Queries can easily span indexes bringing data together as needed. Use indexes for: • Security – use indexes to support multi-tenant system loan_hud loan_fdic • Partitioning – time or other data log_20180501 log_20180502
  24. 24. TRANSFORMING YOUR APPLICATION WITH ELASTIC SEARCH Defining documents DATA MODELING: { "FirstName": "Jonathan", "Address": "15 Wanamassa Point Road ", "City": "Jacksonville", "State": "FL", "Children": [ { "Name": "Michael", "Age": 10 }, { "Name": "Jennifer", "Age": 8 }, { "Name": "Samantha", "Age": 5 }, { "Name": "Elena", "Age": 2 } ] } Instead of decomposing data into separate entities: Design documents based on what needs to be searched and displayed together:
  25. 25. TRANSFORMING YOUR APPLICATION WITH ELASTIC SEARCH Teach Elastic Search about specific types DATA MODELING: PUT _template/file_name_mapping { "template": ”loan_*", "settings": { "number_of_shards": 5 }, "mappings": { "_default_": { "dynamic_templates": [ { "filename_string_not_analyzed": { "match": "edi_filename", "mapping": { "index": "not_analyzed", "type": "string” } } } ] } } }
  26. 26. TRANSFORMING YOUR APPLICATION WITH ELASTIC SEARCH Giving the right users access to the right data SECURITY CONSIDERATIONS cluster Search Guard Custom .NET Core / Java / etc. Elastic Stack Security
  27. 27. TRANSFORMING YOUR APPLICATION WITH ELASTIC SEARCH PUTTING IT ALL TOGETHER The Home Mortgage Disclosure Act (HMDA) requires many financial institutions to maintain, report, and publicly disclose loan-level information about mortgages. Searching CFPB loan records w/ Elastic Search
  28. 28. TRANSFORMING YOUR APPLICATION WITH ELASTIC SEARCH PUTTING IT ALL TOGETHER 107 million loan records var record = csv.GetRecord<dynamic>(); var loan = Loan.FromFileLoan(record); var index_name = string.Format("loan_{0}", loan.agency_abbr.ToLower()); var response = client.Index<Loan>( new IndexRequest<Loan>(loan, index_name)); Loading CFPB loan records into Elastic Search { "county": "Maricopa County", "respondent_id": "0000451965", "lien_status": "Not applicable", "sequence_number": "0945293", "applicant_income": 132000, "preapproval": "Not applicable", "applicant_race": "White", "msamd": "Phoenix, Mesa, Scottsdale - AZ", ….
  29. 29. TRANSFORMING YOUR APPLICATION WITH ELASTIC SEARCH Using Kibana to search and visualize your data PUTTING IT ALL TOGETHER
  30. 30. TRANSFORMING YOUR APPLICATION WITH ELASTIC SEARCH Using Elastic Search to optimize your user experience PUTTING IT ALL TOGETHER cluster Single Page Application (SPA) Angular / React / etc. JWT Query Request REST API .NET Core / Java / etc. Add security, logging, etc.
  31. 31. TRANSFORMING YOUR APPLICATION WITH ELASTIC SEARCH Custom application using Angular, PrimeNG, .NET Core, and Elastic Search PUTTING IT ALL TOGETHER
  32. 32. BRIAN RITCHIE CTO, XEOHEALTH 2018 @brian_ritchie brian.ritchie@gmail.com http://www.dotnetpowered.com QUESTIONS? TRANSFORMING YOUR APPLICATIONS WITH

×