Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

[DevDay2018] PLAY & LEARN with Elasticsearch Workshop - By: Thang Luong and Trung Do, Developer at Axon Active Viet Nam

336 views

Published on

n 2013, Dell – a computer giant, struggled to support e-commerce search for 60+ countries in 21+ languages. Their search engine was not responsive and did not support multi-tenancy or cloud readiness. A year later, they started testing out Elasticsearch, a modern open-source search & analytics engine, and began enjoying success in building linguistic processing, query understanding and result relevancy. From then on, they received good feedback and saw even better results.

What is Elasticsearch? What is its power to help them build a super search engine? How can you begin and apply Elasticsearch to your software development? You will find out these answers in our sessions.

This workshop includes 2 sessions:
– Session 1: Elasticseach in theory (duration: 1 hour)
– Session 2: Practical workshop (duration: 1.5 hour)

The workshop is intended for all levels of experience including beginners. If you develop software and would like to learn about a technology that complements conventional databases to solve hard problems, you will find it useful.

Published in: Technology
  • Be the first to comment

[DevDay2018] PLAY & LEARN with Elasticsearch Workshop - By: Thang Luong and Trung Do, Developer at Axon Active Viet Nam

  1. 1. DEVDAY 2018 Da Nang - Apr 2018
  2. 2. Elasticsearch Learn & Play
  3. 3. www.axon.vnfb.com/AxonActiveVietNam Who we are? Lương Văn Thắng
  4. 4. www.axon.vnfb.com/AxonActiveVietNam Who we are? Đỗ Việt Trung
  5. 5. www.axon.vnfb.com/AxonActiveVietNam 1. LEARN a. OVERVIEW b. Searching c. SCALE d. SECURITY e. MIGRATION 2. PLAY - workshop with exercises Agenda
  6. 6. www.axon.vnfb.com/AxonActiveVietNam OVERVIEW
  7. 7. www.axon.vnfb.com/AxonActiveVietNam What is Elasticsearch? ❖ Search engine based on Lucene ❖ Real-time distributed, full-text search engine ❖ RESTful API ❖ Schema-free ❖ First public release in Feb 2010 Wikipedia
  8. 8. www.axon.vnfb.com/AxonActiveVietNam WHO are using ELASTICSEARCH?
  9. 9. www.axon.vnfb.com/AxonActiveVietNam WhY Elasticsearch? ● NoSQL DB for indexing JSON contents ● Schema-free ● Distributed ● High performance ● REST semantics ● Graph capabilities ● Great documentation ● Open source!
  10. 10. www.axon.vnfb.com/AxonActiveVietNam WhY Elasticsearch?
  11. 11. www.axon.vnfb.com/AxonActiveVietNam WORK WITH Big data
  12. 12. www.axon.vnfb.com/AxonActiveVietNam Searching – faster and smarter
  13. 13. www.axon.vnfb.com/AxonActiveVietNam Searching – poor performance
  14. 14. www.axon.vnfb.com/AxonActiveVietNam Schema changes frequently
  15. 15. www.axon.vnfb.com/AxonActiveVietNam Scaling database
  16. 16. www.axon.vnfb.com/AxonActiveVietNam Our solution
  17. 17. www.axon.vnfb.com/AxonActiveVietNam Search In Action
  18. 18. www.axon.vnfb.com/AxonActiveVietNam Get started
  19. 19. www.axon.vnfb.com/AxonActiveVietNam ❖Node – a started instance of Elasticsearch ❖Cluster - collection of connected nodes of Elasticsearch Cluster and Nodes
  20. 20. www.axon.vnfb.com/AxonActiveVietNam Cluster and Nodes
  21. 21. www.axon.vnfb.com/AxonActiveVietNam ❖Similar to database ❖A collection of types Indexes
  22. 22. www.axon.vnfb.com/AxonActiveVietNam ❖Similar to table ❖Collection of documents ❖Has schema (implicit or explicit) Types
  23. 23. www.axon.vnfb.com/AxonActiveVietNam ❖Similar to record ❖Self-contained data ❖Have id ❖Have schema Documents
  24. 24. www.axon.vnfb.com/AxonActiveVietNam ❖Similar to column ❖Documents are structured in fields ❖Special fields: _id, _uid, _index, _type, _all, _source, … Fields
  25. 25. www.axon.vnfb.com/AxonActiveVietNam ❖ text, keyword ❖ long, integer, short, double, float, byte ❖ date ❖ boolean ❖ binary ❖ geo ❖ object ❖ nested ❖ ip ❖ arrays ❖ … Data types
  26. 26. www.axon.vnfb.com/AxonActiveVietNam Mappings and settings
  27. 27. www.axon.vnfb.com/AxonActiveVietNam Mappings and settings Mappings – schema of documents + dynamic mappings + explicit mappings Settings – configured for each index + good enough when using default settings + index analysis – expert users
  28. 28. www.axon.vnfb.com/AxonActiveVietNam Download and setup https://www.elastic.co/guide/en/elasticsearch/reference/5.4/setup.html
  29. 29. www.axon.vnfb.com/AxonActiveVietNam Config/elasticsearch.yml
  30. 30. www.axon.vnfb.com/AxonActiveVietNam Elasticsearch
  31. 31. www.axon.vnfb.com/AxonActiveVietNam { "from": 0, "size": 20, "min_score": 5, "query": { "bool": { "should": [], "must": [], "must_not": [], "filter": {} } }, "sort": [], "aggs": {} } Query structure
  32. 32. www.axon.vnfb.com/AxonActiveVietNam { "from": 0, "size": 20, "min_score": 5, "query": { "bool": { "should": [], "must": [], "must_not": [], "filter": {} } }, "sort": [], "aggs": {} } Query structure // do pagination
  33. 33. www.axon.vnfb.com/AxonActiveVietNam { "from": 0, "size": 20, "min_score": 5, "query": { "bool": { "should": [], "must": [], "must_not": [], "filter": {} } }, "sort": [], "aggs": {} } Query structure // limit hitlist return
  34. 34. www.axon.vnfb.com/AxonActiveVietNam { "from": 0, "size": 20, "min_score": 5, "query": { "bool": { "should": [], "must": [], "must_not": [], "filter": {} } }, "sort": [], "aggs": {} } Query structure // statistic
  35. 35. www.axon.vnfb.com/AxonActiveVietNam { "from": 0, "size": 20, "min_score": 5, "query": { "bool": { "should": [{..}, {..},..], "must": [{..}, {..},..], "must_not": [{..}, {..},..], "filter": {} // not impact to total score } }, "sort": [], "aggs": {} } Query structure // and // or // not
  36. 36. www.axon.vnfb.com/AxonActiveVietNam { "bool": { "must": [ {“strong”}, {“is man”} ] } } Exercise 1
  37. 37. www.axon.vnfb.com/AxonActiveVietNam { "bool": { "must": [ {“strong”}, {“is man”}] } } Exercise 1
  38. 38. www.axon.vnfb.com/AxonActiveVietNam { "bool": { "must": [{“has red on suite”}], “should“: [{“strong”},{“is man”}] } } Exercise 2
  39. 39. www.axon.vnfb.com/AxonActiveVietNam { "bool": { "must": [{“has red on suite”}], "should":[{“strong”},{“is man”}] } } Exercise 2
  40. 40. www.axon.vnfb.com/AxonActiveVietNam Exercise 3 { "bool": { "must": [{“strong”},{“has cape”}], “must_not“: [{“be seen the face”} ] } }
  41. 41. www.axon.vnfb.com/AxonActiveVietNam { "bool": { "must": [{“strong”}, {“has cape”}], “must_not“: [{“be seen the face”}] } } Exercise 3
  42. 42. www.axon.vnfb.com/AxonActiveVietNam building a query
  43. 43. www.axon.vnfb.com/AxonActiveVietNam Synonym
  44. 44. www.axon.vnfb.com/AxonActiveVietNam Synonym token filter
  45. 45. www.axon.vnfb.com/AxonActiveVietNam Synonym token filter
  46. 46. www.axon.vnfb.com/AxonActiveVietNam SCALing
  47. 47. www.axon.vnfb.com/AxonActiveVietNam What – Why?
  48. 48. www.axon.vnfb.com/AxonActiveVietNam How to Scale
  49. 49. www.axon.vnfb.com/AxonActiveVietNam How to Scale
  50. 50. www.axon.vnfb.com/AxonActiveVietNam SECURITY
  51. 51. www.axon.vnfb.com/AxonActiveVietNam WHY Elasticsearch has no concept of a user. Essentially, anyone that can send arbitrary requests to your cluster is a “super user”.
  52. 52. www.axon.vnfb.com/AxonActiveVietNam WHAT ❖ Authentication & authorization ❖ Access control ❖ Encryption ❖ Auditing
  53. 53. www.axon.vnfb.com/AxonActiveVietNam
  54. 54. www.axon.vnfb.com/AxonActiveVietNam Authentication & Authorization Role-based access control Encryption Auditing License PaidPaid/UnpaidPaid/Unpaid Paid/Unpaid
  55. 55. www.axon.vnfb.com/AxonActiveVietNam Prevent unauthorized access Keep Data Integrity what happen on system
  56. 56. www.axon.vnfb.com/AxonActiveVietNam Installing….
  57. 57. www.axon.vnfb.com/AxonActiveVietNam User Authentication
  58. 58. www.axon.vnfb.com/AxonActiveVietNam Identify who send the request
  59. 59. www.axon.vnfb.com/AxonActiveVietNam built-in users elastic kibana logstash_system
  60. 60. www.axon.vnfb.com/AxonActiveVietNam ADD More User PUT localhost:9200/_xpack/security/user/trungdo { "full_name" : "Trung Do", "email" : "trung.do@axonactive.com", "password" : "axonvn", "roles" : [ "admin", "superior", "kibana_user" ], "metadata" : { "workingYear" : 5 }, "enabled": true }
  61. 61. www.axon.vnfb.com/AxonActiveVietNam How authentication works Native (Basic Auth) LDAP file (Basic Auth) Active Directory PKI Custom Realms
  62. 62. www.axon.vnfb.com/AxonActiveVietNam Realms chain Native file LDAP1 LDAP2
  63. 63. www.axon.vnfb.com/AxonActiveVietNam xpack.security.authc: realms: file: //id of realm type: file //type of realm order: 0 //order in chain native: type: native order: 1 ldap1: type: ldap order: 2 enabled: false url: 'url_to_ldap1' … ldap2: type: ldap order: 3 enabled: false url: 'url_to_ldap2' Configure realms chain elasticsearch.yml
  64. 64. www.axon.vnfb.com/AxonActiveVietNam Authorization
  65. 65. www.axon.vnfb.com/AxonActiveVietNam ❖ Identify permission to execute request ❖ Support by Role Based Access Control (RBAC)
  66. 66. www.axon.vnfb.com/AxonActiveVietNam Secured Resource RBAC Role Privilege Permissions User
  67. 67. www.axon.vnfb.com/AxonActiveVietNam Secured Resource Object need to be restricted accessity Indices Document Field User Cluster
  68. 68. www.axon.vnfb.com/AxonActiveVietNam Privilege ❖ One or group of actions user can execute ❖ Two types: Cluster and Indice all read create index delete See full privileges: https://www.elastic.co/guide/en/x-pack/5.4/security-privileges.html all monitor manage_watcher manage_security Indice privilege Cluster privilege
  69. 69. www.axon.vnfb.com/AxonActiveVietNam Permission Set of privileges on a secured resource read on kibana indices monitor on cluster write on kibana indices kibana system permission
  70. 70. www.axon.vnfb.com/AxonActiveVietNam Built-in Role superuser kibana_user kibana_system logstash_system monitoring_user reporting_user watcher_admin watcher_user machine_learning_user See more at: https://www.elastic.co/guide/en/x-pack/5.4/built-in-roles.html
  71. 71. www.axon.vnfb.com/AxonActiveVietNam ADD ROLE POST localhost:9200/_xpack/security/role/scrum_admin { "run_as": [ ... ], "cluster": [ ... ], "indices": [ ... ] } //submit request on behalf of other user //privilege on cluster //permission entry on indices
  72. 72. www.axon.vnfb.com/AxonActiveVietNam ADD ROLE POST localhost:9200/_xpack/security/role/scrum_admin { "run_as": [ "khangvu" ], "cluster": [ "monitor" ], "indices": [ { "names": [ "scrum*" ], "privileges": [ "read" ], "field_security" : { "grant" : [ "category", "@timestamp", "message" ] }, "query": "{"match": {"event.type": "technical"}}" } ] //scrum_admin can submit request for khangvu user //can monitor cluster //name of targeted-indices: all indexes started with scrum //privilege on targeted-indices //allowed field to access //list of targeted-document
  73. 73. www.axon.vnfb.com/AxonActiveVietNam Auditing ❖ Audit activities/events that occur in the system ❖ Output to logfile or index ❖ Enable this feature by xpack.security.audit.enabled to true in elasticsearch.yml. See more at https://www.elastic.co/guide/en/x-pack/5.4/auditing.html
  74. 74. www.axon.vnfb.com/AxonActiveVietNam
  75. 75. www.axon.vnfb.com/AxonActiveVietNam SECURITY
  76. 76. www.axon.vnfb.com/AxonActiveVietNam
  77. 77. www.axon.vnfb.com/AxonActiveVietNam migration with logstash
  78. 78. www.axon.vnfb.com/AxonActiveVietNam Logstash is data collector pipeline
  79. 79. www.axon.vnfb.com/AxonActiveVietNam How ? ❖ Event processing pipeline ❖ Has three stages: ➢ Input ➢ Filter ➢ Output ❖ Has bunch of plugins for you to play with
  80. 80. www.axon.vnfb.com/AxonActiveVietNam usecase - Collect rdbms to es server Database Logstash filter ElasticSearch Store Read Kibana
  81. 81. www.axon.vnfb.com/AxonActiveVietNam
  82. 82. www.axon.vnfb.com/AxonActiveVietNam Exercise 1 (3 points): - Install ElasticSearch, Kibana - Import ~5000 films to Elastic - Install & configure User and Role with X-Pack Exercise 2 (4 points): - Query films by some criterias on Kibana Exercise 3 (5 points): - Install and configure Logstash - Filter films by Logstash - Import CSV File to ElasticSearch - Query new films by some criteria
  83. 83. www.axon.vnfb.com/AxonActiveVietNam How to run the exercises ● Register with speaker ● Divide by group of 4 ● Time for each group: 45 minutes ● Reach the checkpoint to get score ● Gifts: ○ First finish/highest score team - T-shirt ○ Runner-up - Bag ○ Rest - Paper & Pen
  84. 84. www.axon.vnfb.com/AxonActiveVietNam

×