SlideShare a Scribd company logo
“ Terms of Endearment” The ElasticSearch query language explained Clinton Gormley, YAPC::EU 2011 DRTECH @clintongormley
search for : “ DELETE QUERY ”  We can
search for : “ DELETE QUERY ”  and find : “ deleteByQuery ” We can
but you can only find  what is stored in the database
Normalise values  “ deleteByQuery” 'delete' 'by' 'query' 'deletebyquery'
Normalise values  and search terms “ deleteByQuery” “ DELETE QUERY” ' delete ' 'by' ' query ' 'deletebyquery'
Normalise  values  and search terms “ deleteByQuery” “ DELETE QUERY” ' delete ' 'by' ' query ' 'deletebyquery'
Analyse  values  and search terms “ deleteByQuery” “ DELETE QUERY” ' delete ' 'by' ' query ' 'deletebyquery'
What is stored in ElasticSearch?
{ tweet  => "Perl is GREAT!", posted => "2011-08-15", user  => { name  => "Clinton Gormley", email => "drtech@cpan.org", }, tags  => [" perl" ,"opinion"],  posts  => 2, } Document:
{ tweet   => "Perl is GREAT!", posted  => "2011-08-15", user   => { name   => "Clinton Gormley", email  => "drtech@cpan.org", }, tags   => [" perl" ,"opinion"],  posts   => 2, } Fields:
{ tweet  =>  "Perl is GREAT!", posted =>  "2011-08-15", user  =>  { name  =>  "Clinton Gormley", email =>  "drtech@cpan.org", }, tags   =>  [" perl" ,"opinion"],  posts  =>  2, } Values:
{ tweet  => "Perl is GREAT!", posted => "2011-08-15", user  => { name  => "Clinton Gormley", email => "drtech@cpan.org" }, tags  => [" perl" ,"opinion"],  posts  => 2, } Field types: # object # string # date # nested object # string # string # array of enums # integer
{ tweet  => "Perl is GREAT!", posted => "2011-08-15", user   => { name  => "Clinton Gormley", email  => "drtech@cpan.org", }, tags  => [" perl" ,"opinion"],  posts  => 2, } Nested objects flattened:
{ tweet  => "Perl is GREAT!", posted  => "2011-08-15", user.name  => "Clinton Gormley", user.email  => "drtech@cpan.org", tags  => [" perl" ,"opinion"],  posts  => 2, } Nested objects flattened
{ tweet  =>  "Perl is GREAT!", posted  =>  "2011-08-15", user.name  =>  "Clinton Gormley", user.email =>  "drtech@cpan.org", tags  =>  [" perl" ,"opinion"],  posts  =>  2, } Values analyzed into terms
{ tweet  =>  ['perl','great'], posted  =>  [Date(2011-08-15)], user.name  =>  ['clinton','gormley'], user.email =>  ['drtech','cpan.org'], tags  =>  [' perl' ,'opinion'],  posts  =>  [2], } Values analyzed into terms
database table row ⇒  many tables ⇒  many rows ⇒  one schema ⇒  many columns In MySQL
index type document ⇒  many types ⇒  many documents ⇒  one mapping ⇒  many fields In ElasticSearch
Create index with mappings $es-> create_index ( index  => 'twitter', mappings   => { tweet   => { properties  => { title  => { type => 'string' }, created => { type => 'date'  } }  } } );
Add a mapping $es-> put_mapping (  index => 'twitter', type  => ' user ', mapping   => { properties  => { name  => { type => 'string' }, created => { type => 'date'  }, }  } );
Can add to existing mapping
Can add to existing mapping Cannot change mapping for field
Core field types { type  => 'string', }
Core field types { type  => 'string', # byte|short|integer|long|double|float # date, ip addr, geolocation # boolean # binary (as base 64) }
Core field types { type  => 'string', index  => ' analyzed ', # 'Foo Bar'  ⇒  [ 'foo', 'bar' ] }
Core field types { type  => 'string', index  => ' not_analyzed ', # 'Foo Bar'  ⇒  [ 'Foo Bar' ] }
Core field types { type  => 'string', index  => ' no ', # 'Foo Bar'  ⇒  [ ] }
Core field types { type  => 'string', index  => 'analyzed', analyzer  => 'default', }
Core field types { type  => 'string', index  => 'analyzed', index_ analyzer  => 'default', search_ analyzer => 'default', }
Core field types { type  => 'string', index  => 'analyzed', analyzer  => 'default', boost  => 2, }
Core field types { type  => 'string', index  => 'analyzed', analyzer  => 'default', boost  => 2, include_in_all  => 1 |0 }
Standard
Simple
Whitespace
Stop
Keyword Built in analyzers Pattern
Language
Snowball
Custom
The Brown-Cow's Part_No.  #A.BC123-456 joe@bloggs.com keyword: The Brown-Cow's Part_No. #A.BC123-456 joe@bloggs.com whitespace: The, Brown-Cow's, Part_No., #A.BC123-456, joe@bloggs.com simple: the, brown, cow, s, part, no, a, bc, joe, bloggs, com standard: brown, cow's, part_no, a.bc123, 456, joe, bloggs.com snowball (English): brown, cow, part_no, a.bc123, 456, joe, bloggs.com
Token filters Standard
ASCII Folding
Length
Lowercase
NGram
Edge NGram
Porter Stem
Shingle
Stop
Word Delimiter Stemmer
KStem
Snowball
Phonetic
Synonym
Compound Word
Reverse
Elision
Truncate
Unique
Custom Analyzer $c->create_index( index  => 'twitter', settings  => { analysis => { analyzer => { ascii_html => { type  => 'custom', tokenizer  => 'standard', filter  => [ qw( standard lowercase asciifolding stop ) ], char_filter => ['html_strip'] } } }} );
Searching $result = $es->search( index  => 'twitter', type  => 'tweet',  );
Searching $result = $es->search( index  =>  ['twitter','facebook'] , type  =>  ['tweet','post'] ,  );
Searching $result = $es->search( #  all indices #  all types );
Searching $result = $es->search( index  => 'twitter', type  => 'tweet',  query  => { text => { _all => 'foo' }}, );
Searching $result = $es->search( index  => 'twitter', type  => 'tweet', query b   =>  'foo' , #  b == ElasticSearch::SearchBuilder );
Searching $result = $es->search( index  => 'twitter', type  => 'tweet', query  => { text => { _all => 'foo' }}, sort  => [{ '_score': 'desc' }] );
Searching $result = $es->search( index  => 'twitter', type  => 'tweet', query  => { text => { _all => 'foo' }}, sort  => [{ '_score': 'desc' }] from  => 0, size  => 10, );
Query DSL
Queries   vs  Filters
Queries   vs  Filters  full text & terms terms only
Queries   vs  Filters  full text & terms
relevance scoring terms only
no scoring
Queries   vs  Filters  full text & terms
relevance scoring
slower terms only
no scoring
faster
Queries   vs  Filters  full text & terms
relevance scoring
slower
no caching terms only
no scoring
faster
cacheable
Queries   vs  Filters  full text & terms
relevance scoring
slower
no caching terms only
no scoring
faster
cacheable  Use filters for anything that doesn't affect the relevance score!
Query only Query DSL: $es->search(  query => {  text => { title => 'perl' }  } ); SearchBuilder: $es->search(  query b  => {  title => 'perl'  } );
Filter only Query DSL: $es->search( query => { constant_score => { filter => { term => { tag => 'perl } } } }); SearchBuilder: $es->search( query b  => { -filter => {  tag => 'perl'  } });
Query and filter Query DSL: $es->search( query => { filtered  => { query => {  text => { title => 'perl' } }, filter =>{  term => { tag => 'perl'  } } } }); SearchBuilder: $es->search( query b  => { title  => 'perl', -filter => {  tag => 'perl'  }  });

More Related Content

What's hot

Html
HtmlHtml
Python and MongoDB
Python and MongoDB Python and MongoDB
Python and MongoDB
Norberto Leite
 
MongoDB Aggregation Performance
MongoDB Aggregation PerformanceMongoDB Aggregation Performance
MongoDB Aggregation Performance
MongoDB
 
Quick flask an intro to flask
Quick flask   an intro to flaskQuick flask   an intro to flask
Quick flask an intro to flask
juzten
 
Flask Basics
Flask BasicsFlask Basics
Flask Basics
Eueung Mulyana
 
Introduction to Python programming
Introduction to Python programmingIntroduction to Python programming
Introduction to Python programming
Damian T. Gordon
 
Django for Beginners
Django for BeginnersDjango for Beginners
Django for Beginners
Jason Davies
 
JavaScript - Chapter 10 - Strings and Arrays
 JavaScript - Chapter 10 - Strings and Arrays JavaScript - Chapter 10 - Strings and Arrays
JavaScript - Chapter 10 - Strings and Arrays
WebStackAcademy
 
Working with JSON
Working with JSONWorking with JSON
Django Framework and Application Structure
Django Framework and Application StructureDjango Framework and Application Structure
Django Framework and Application Structure
SEONGTAEK OH
 
Python/Flask Presentation
Python/Flask PresentationPython/Flask Presentation
Python/Flask Presentation
Parag Mujumdar
 
Spring 3.x - Spring MVC - Advanced topics
Spring 3.x - Spring MVC - Advanced topicsSpring 3.x - Spring MVC - Advanced topics
Spring 3.x - Spring MVC - Advanced topics
Guy Nir
 
Angular 10 course_content
Angular 10 course_contentAngular 10 course_content
Angular 10 course_content
NAVEENSAGGAM1
 
TypeScript Presentation
TypeScript PresentationTypeScript Presentation
TypeScript Presentation
Patrick John Pacaña
 
Spring Core
Spring CoreSpring Core
Spring Core
Pushan Bhattacharya
 
Nest.js Introduction
Nest.js IntroductionNest.js Introduction
Nest.js Introduction
Takuya Tejima
 
Eye catching HTML BASICS tips: Learn easily
Eye catching HTML BASICS tips: Learn easilyEye catching HTML BASICS tips: Learn easily
Eye catching HTML BASICS tips: Learn easily
shabab shihan
 
Modern Java web applications with Spring Boot and Thymeleaf
Modern Java web applications with Spring Boot and ThymeleafModern Java web applications with Spring Boot and Thymeleaf
Modern Java web applications with Spring Boot and Thymeleaf
LAY Leangsros
 
jQuery
jQueryjQuery
Tự học JavaScript
Tự học JavaScriptTự học JavaScript
Tự học JavaScript
Ánh Nguyễn
 

What's hot (20)

Html
HtmlHtml
Html
 
Python and MongoDB
Python and MongoDB Python and MongoDB
Python and MongoDB
 
MongoDB Aggregation Performance
MongoDB Aggregation PerformanceMongoDB Aggregation Performance
MongoDB Aggregation Performance
 
Quick flask an intro to flask
Quick flask   an intro to flaskQuick flask   an intro to flask
Quick flask an intro to flask
 
Flask Basics
Flask BasicsFlask Basics
Flask Basics
 
Introduction to Python programming
Introduction to Python programmingIntroduction to Python programming
Introduction to Python programming
 
Django for Beginners
Django for BeginnersDjango for Beginners
Django for Beginners
 
JavaScript - Chapter 10 - Strings and Arrays
 JavaScript - Chapter 10 - Strings and Arrays JavaScript - Chapter 10 - Strings and Arrays
JavaScript - Chapter 10 - Strings and Arrays
 
Working with JSON
Working with JSONWorking with JSON
Working with JSON
 
Django Framework and Application Structure
Django Framework and Application StructureDjango Framework and Application Structure
Django Framework and Application Structure
 
Python/Flask Presentation
Python/Flask PresentationPython/Flask Presentation
Python/Flask Presentation
 
Spring 3.x - Spring MVC - Advanced topics
Spring 3.x - Spring MVC - Advanced topicsSpring 3.x - Spring MVC - Advanced topics
Spring 3.x - Spring MVC - Advanced topics
 
Angular 10 course_content
Angular 10 course_contentAngular 10 course_content
Angular 10 course_content
 
TypeScript Presentation
TypeScript PresentationTypeScript Presentation
TypeScript Presentation
 
Spring Core
Spring CoreSpring Core
Spring Core
 
Nest.js Introduction
Nest.js IntroductionNest.js Introduction
Nest.js Introduction
 
Eye catching HTML BASICS tips: Learn easily
Eye catching HTML BASICS tips: Learn easilyEye catching HTML BASICS tips: Learn easily
Eye catching HTML BASICS tips: Learn easily
 
Modern Java web applications with Spring Boot and Thymeleaf
Modern Java web applications with Spring Boot and ThymeleafModern Java web applications with Spring Boot and Thymeleaf
Modern Java web applications with Spring Boot and Thymeleaf
 
jQuery
jQueryjQuery
jQuery
 
Tự học JavaScript
Tự học JavaScriptTự học JavaScript
Tự học JavaScript
 

Similar to Terms of endearment - the ElasticSearch Query DSL explained

Php Basic Security
Php Basic SecurityPhp Basic Security
Php Basic Security
mussawir20
 
High-level Web Testing
High-level Web TestingHigh-level Web Testing
High-level Web Testing
petersergeant
 
Exploiting Php With Php
Exploiting Php With PhpExploiting Php With Php
Exploiting Php With Php
Jeremy Coates
 
PHP 102: Out with the Bad, In with the Good
PHP 102: Out with the Bad, In with the GoodPHP 102: Out with the Bad, In with the Good
PHP 102: Out with the Bad, In with the Good
Jeremy Kendall
 
HTML5 Web Forms
HTML5 Web FormsHTML5 Web Forms
HTML5 Web Forms
Estelle Weyl
 
Intro python
Intro pythonIntro python
Intro python
kamzilla
 
JSP Custom Tags
JSP Custom TagsJSP Custom Tags
JSP Custom Tags
BG Java EE Course
 
Sencha Touch Intro
Sencha Touch IntroSencha Touch Intro
Sencha Touch Intro
Shea Frederick
 
JQuery 101
JQuery 101JQuery 101
Drupal Lightning FAPI Jumpstart
Drupal Lightning FAPI JumpstartDrupal Lightning FAPI Jumpstart
Drupal Lightning FAPI Jumpstart
guestfd47e4c7
 
JQuery Basics
JQuery BasicsJQuery Basics
JQuery Basics
Alin Taranu
 
03 Php Array String Functions
03 Php Array String Functions03 Php Array String Functions
03 Php Array String Functions
Geshan Manandhar
 
Schema design with MongoDB (Dwight Merriman)
Schema design with MongoDB (Dwight Merriman)Schema design with MongoDB (Dwight Merriman)
Schema design with MongoDB (Dwight Merriman)
MongoSF
 
Forum Presentation
Forum PresentationForum Presentation
Forum Presentation
Angus Pratt
 
Intro to #memtech PHP 2011-12-05
Intro to #memtech PHP   2011-12-05Intro to #memtech PHP   2011-12-05
Intro to #memtech PHP 2011-12-05
Jeremy Kendall
 
Cool bonsai cool - an introduction to ElasticSearch
Cool bonsai cool - an introduction to ElasticSearchCool bonsai cool - an introduction to ElasticSearch
Cool bonsai cool - an introduction to ElasticSearch
clintongormley
 
Haml & Sass presentation
Haml & Sass presentationHaml & Sass presentation
Haml & Sass presentation
bryanbibat
 
Ods Markup And Tagsets: A Tutorial
Ods Markup And Tagsets: A TutorialOds Markup And Tagsets: A Tutorial
Ods Markup And Tagsets: A Tutorial
simienc
 
Introduction into Struts2 jQuery Grid Tags
Introduction into Struts2 jQuery Grid TagsIntroduction into Struts2 jQuery Grid Tags
Introduction into Struts2 jQuery Grid Tags
Johannes Geppert
 
Mojolicious on Steroids
Mojolicious on SteroidsMojolicious on Steroids
Mojolicious on Steroids
Tudor Constantin
 

Similar to Terms of endearment - the ElasticSearch Query DSL explained (20)

Php Basic Security
Php Basic SecurityPhp Basic Security
Php Basic Security
 
High-level Web Testing
High-level Web TestingHigh-level Web Testing
High-level Web Testing
 
Exploiting Php With Php
Exploiting Php With PhpExploiting Php With Php
Exploiting Php With Php
 
PHP 102: Out with the Bad, In with the Good
PHP 102: Out with the Bad, In with the GoodPHP 102: Out with the Bad, In with the Good
PHP 102: Out with the Bad, In with the Good
 
HTML5 Web Forms
HTML5 Web FormsHTML5 Web Forms
HTML5 Web Forms
 
Intro python
Intro pythonIntro python
Intro python
 
JSP Custom Tags
JSP Custom TagsJSP Custom Tags
JSP Custom Tags
 
Sencha Touch Intro
Sencha Touch IntroSencha Touch Intro
Sencha Touch Intro
 
JQuery 101
JQuery 101JQuery 101
JQuery 101
 
Drupal Lightning FAPI Jumpstart
Drupal Lightning FAPI JumpstartDrupal Lightning FAPI Jumpstart
Drupal Lightning FAPI Jumpstart
 
JQuery Basics
JQuery BasicsJQuery Basics
JQuery Basics
 
03 Php Array String Functions
03 Php Array String Functions03 Php Array String Functions
03 Php Array String Functions
 
Schema design with MongoDB (Dwight Merriman)
Schema design with MongoDB (Dwight Merriman)Schema design with MongoDB (Dwight Merriman)
Schema design with MongoDB (Dwight Merriman)
 
Forum Presentation
Forum PresentationForum Presentation
Forum Presentation
 
Intro to #memtech PHP 2011-12-05
Intro to #memtech PHP   2011-12-05Intro to #memtech PHP   2011-12-05
Intro to #memtech PHP 2011-12-05
 
Cool bonsai cool - an introduction to ElasticSearch
Cool bonsai cool - an introduction to ElasticSearchCool bonsai cool - an introduction to ElasticSearch
Cool bonsai cool - an introduction to ElasticSearch
 
Haml & Sass presentation
Haml & Sass presentationHaml & Sass presentation
Haml & Sass presentation
 
Ods Markup And Tagsets: A Tutorial
Ods Markup And Tagsets: A TutorialOds Markup And Tagsets: A Tutorial
Ods Markup And Tagsets: A Tutorial
 
Introduction into Struts2 jQuery Grid Tags
Introduction into Struts2 jQuery Grid TagsIntroduction into Struts2 jQuery Grid Tags
Introduction into Struts2 jQuery Grid Tags
 
Mojolicious on Steroids
Mojolicious on SteroidsMojolicious on Steroids
Mojolicious on Steroids
 

Recently uploaded

Vertex AI Agent Builder - GDG Alicante - Julio 2024
Vertex AI Agent Builder - GDG Alicante - Julio 2024Vertex AI Agent Builder - GDG Alicante - Julio 2024
Vertex AI Agent Builder - GDG Alicante - Julio 2024
Nicolás Lopéz
 
Acumatica vs. Sage Intacct vs. NetSuite _ NOW CFO.pdf
Acumatica vs. Sage Intacct vs. NetSuite _ NOW CFO.pdfAcumatica vs. Sage Intacct vs. NetSuite _ NOW CFO.pdf
Acumatica vs. Sage Intacct vs. NetSuite _ NOW CFO.pdf
BrainSell Technologies
 
Vulnerability Management: A Comprehensive Overview
Vulnerability Management: A Comprehensive OverviewVulnerability Management: A Comprehensive Overview
Vulnerability Management: A Comprehensive Overview
Steven Carlson
 
BLOCKCHAIN TECHNOLOGY - Advantages and Disadvantages
BLOCKCHAIN TECHNOLOGY - Advantages and DisadvantagesBLOCKCHAIN TECHNOLOGY - Advantages and Disadvantages
BLOCKCHAIN TECHNOLOGY - Advantages and Disadvantages
SAI KAILASH R
 
Sonkoloniya documentation - ONEprojukti.pdf
Sonkoloniya documentation - ONEprojukti.pdfSonkoloniya documentation - ONEprojukti.pdf
Sonkoloniya documentation - ONEprojukti.pdf
SubhamMandal40
 
Evolution of iPaaS - simplify IT workloads to provide a unified view of data...
Evolution of iPaaS - simplify IT workloads to provide a unified view of  data...Evolution of iPaaS - simplify IT workloads to provide a unified view of  data...
Evolution of iPaaS - simplify IT workloads to provide a unified view of data...
Torry Harris
 
leewayhertz.com-AI agents for healthcare Applications benefits and implementa...
leewayhertz.com-AI agents for healthcare Applications benefits and implementa...leewayhertz.com-AI agents for healthcare Applications benefits and implementa...
leewayhertz.com-AI agents for healthcare Applications benefits and implementa...
alexjohnson7307
 
Best Practices for Effectively Running dbt in Airflow.pdf
Best Practices for Effectively Running dbt in Airflow.pdfBest Practices for Effectively Running dbt in Airflow.pdf
Best Practices for Effectively Running dbt in Airflow.pdf
Tatiana Al-Chueyr
 
Connector Corner: Leveraging Snowflake Integration for Smarter Decision Making
Connector Corner: Leveraging Snowflake Integration for Smarter Decision MakingConnector Corner: Leveraging Snowflake Integration for Smarter Decision Making
Connector Corner: Leveraging Snowflake Integration for Smarter Decision Making
DianaGray10
 
Mastering OnlyFans Clone App Development: Key Strategies for Success
Mastering OnlyFans Clone App Development: Key Strategies for SuccessMastering OnlyFans Clone App Development: Key Strategies for Success
Mastering OnlyFans Clone App Development: Key Strategies for Success
David Wilson
 
Introduction-to-the-IAM-Platform-Implementation-Plan.pptx
Introduction-to-the-IAM-Platform-Implementation-Plan.pptxIntroduction-to-the-IAM-Platform-Implementation-Plan.pptx
Introduction-to-the-IAM-Platform-Implementation-Plan.pptx
313mohammedarshad
 
Dublin_mulesoft_meetup_Mulesoft_Salesforce_Integration (1).pptx
Dublin_mulesoft_meetup_Mulesoft_Salesforce_Integration (1).pptxDublin_mulesoft_meetup_Mulesoft_Salesforce_Integration (1).pptx
Dublin_mulesoft_meetup_Mulesoft_Salesforce_Integration (1).pptx
Kunal Gupta
 
Acumatica vs. Sage Intacct _Construction_July (1).pptx
Acumatica vs. Sage Intacct _Construction_July (1).pptxAcumatica vs. Sage Intacct _Construction_July (1).pptx
Acumatica vs. Sage Intacct _Construction_July (1).pptx
BrainSell Technologies
 
Uncharted Together- Navigating AI's New Frontiers in Libraries
Uncharted Together- Navigating AI's New Frontiers in LibrariesUncharted Together- Navigating AI's New Frontiers in Libraries
Uncharted Together- Navigating AI's New Frontiers in Libraries
Brian Pichman
 
Semantic-Aware Code Model: Elevating the Future of Software Development
Semantic-Aware Code Model: Elevating the Future of Software DevelopmentSemantic-Aware Code Model: Elevating the Future of Software Development
Semantic-Aware Code Model: Elevating the Future of Software Development
Baishakhi Ray
 
(CISOPlatform Summit & SACON 2024) Digital Personal Data Protection Act.pdf
(CISOPlatform Summit & SACON 2024) Digital Personal Data Protection Act.pdf(CISOPlatform Summit & SACON 2024) Digital Personal Data Protection Act.pdf
(CISOPlatform Summit & SACON 2024) Digital Personal Data Protection Act.pdf
Priyanka Aash
 
IPLOOK Remote-Sensing Satellite Solution
IPLOOK Remote-Sensing Satellite SolutionIPLOOK Remote-Sensing Satellite Solution
IPLOOK Remote-Sensing Satellite Solution
IPLOOK Networks
 
Computer HARDWARE presenattion by CWD students class 10
Computer HARDWARE presenattion by CWD students class 10Computer HARDWARE presenattion by CWD students class 10
Computer HARDWARE presenattion by CWD students class 10
ankush9927
 
Russian Girls Call Navi Mumbai 🎈🔥9920725232 🔥💋🎈 Provide Best And Top Girl Ser...
Russian Girls Call Navi Mumbai 🎈🔥9920725232 🔥💋🎈 Provide Best And Top Girl Ser...Russian Girls Call Navi Mumbai 🎈🔥9920725232 🔥💋🎈 Provide Best And Top Girl Ser...
Russian Girls Call Navi Mumbai 🎈🔥9920725232 🔥💋🎈 Provide Best And Top Girl Ser...
bellared2
 
Litestack talk at Brighton 2024 (Unleashing the power of SQLite for Ruby apps)
Litestack talk at Brighton 2024 (Unleashing the power of SQLite for Ruby apps)Litestack talk at Brighton 2024 (Unleashing the power of SQLite for Ruby apps)
Litestack talk at Brighton 2024 (Unleashing the power of SQLite for Ruby apps)
Muhammad Ali
 

Recently uploaded (20)

Vertex AI Agent Builder - GDG Alicante - Julio 2024
Vertex AI Agent Builder - GDG Alicante - Julio 2024Vertex AI Agent Builder - GDG Alicante - Julio 2024
Vertex AI Agent Builder - GDG Alicante - Julio 2024
 
Acumatica vs. Sage Intacct vs. NetSuite _ NOW CFO.pdf
Acumatica vs. Sage Intacct vs. NetSuite _ NOW CFO.pdfAcumatica vs. Sage Intacct vs. NetSuite _ NOW CFO.pdf
Acumatica vs. Sage Intacct vs. NetSuite _ NOW CFO.pdf
 
Vulnerability Management: A Comprehensive Overview
Vulnerability Management: A Comprehensive OverviewVulnerability Management: A Comprehensive Overview
Vulnerability Management: A Comprehensive Overview
 
BLOCKCHAIN TECHNOLOGY - Advantages and Disadvantages
BLOCKCHAIN TECHNOLOGY - Advantages and DisadvantagesBLOCKCHAIN TECHNOLOGY - Advantages and Disadvantages
BLOCKCHAIN TECHNOLOGY - Advantages and Disadvantages
 
Sonkoloniya documentation - ONEprojukti.pdf
Sonkoloniya documentation - ONEprojukti.pdfSonkoloniya documentation - ONEprojukti.pdf
Sonkoloniya documentation - ONEprojukti.pdf
 
Evolution of iPaaS - simplify IT workloads to provide a unified view of data...
Evolution of iPaaS - simplify IT workloads to provide a unified view of  data...Evolution of iPaaS - simplify IT workloads to provide a unified view of  data...
Evolution of iPaaS - simplify IT workloads to provide a unified view of data...
 
leewayhertz.com-AI agents for healthcare Applications benefits and implementa...
leewayhertz.com-AI agents for healthcare Applications benefits and implementa...leewayhertz.com-AI agents for healthcare Applications benefits and implementa...
leewayhertz.com-AI agents for healthcare Applications benefits and implementa...
 
Best Practices for Effectively Running dbt in Airflow.pdf
Best Practices for Effectively Running dbt in Airflow.pdfBest Practices for Effectively Running dbt in Airflow.pdf
Best Practices for Effectively Running dbt in Airflow.pdf
 
Connector Corner: Leveraging Snowflake Integration for Smarter Decision Making
Connector Corner: Leveraging Snowflake Integration for Smarter Decision MakingConnector Corner: Leveraging Snowflake Integration for Smarter Decision Making
Connector Corner: Leveraging Snowflake Integration for Smarter Decision Making
 
Mastering OnlyFans Clone App Development: Key Strategies for Success
Mastering OnlyFans Clone App Development: Key Strategies for SuccessMastering OnlyFans Clone App Development: Key Strategies for Success
Mastering OnlyFans Clone App Development: Key Strategies for Success
 
Introduction-to-the-IAM-Platform-Implementation-Plan.pptx
Introduction-to-the-IAM-Platform-Implementation-Plan.pptxIntroduction-to-the-IAM-Platform-Implementation-Plan.pptx
Introduction-to-the-IAM-Platform-Implementation-Plan.pptx
 
Dublin_mulesoft_meetup_Mulesoft_Salesforce_Integration (1).pptx
Dublin_mulesoft_meetup_Mulesoft_Salesforce_Integration (1).pptxDublin_mulesoft_meetup_Mulesoft_Salesforce_Integration (1).pptx
Dublin_mulesoft_meetup_Mulesoft_Salesforce_Integration (1).pptx
 
Acumatica vs. Sage Intacct _Construction_July (1).pptx
Acumatica vs. Sage Intacct _Construction_July (1).pptxAcumatica vs. Sage Intacct _Construction_July (1).pptx
Acumatica vs. Sage Intacct _Construction_July (1).pptx
 
Uncharted Together- Navigating AI's New Frontiers in Libraries
Uncharted Together- Navigating AI's New Frontiers in LibrariesUncharted Together- Navigating AI's New Frontiers in Libraries
Uncharted Together- Navigating AI's New Frontiers in Libraries
 
Semantic-Aware Code Model: Elevating the Future of Software Development
Semantic-Aware Code Model: Elevating the Future of Software DevelopmentSemantic-Aware Code Model: Elevating the Future of Software Development
Semantic-Aware Code Model: Elevating the Future of Software Development
 
(CISOPlatform Summit & SACON 2024) Digital Personal Data Protection Act.pdf
(CISOPlatform Summit & SACON 2024) Digital Personal Data Protection Act.pdf(CISOPlatform Summit & SACON 2024) Digital Personal Data Protection Act.pdf
(CISOPlatform Summit & SACON 2024) Digital Personal Data Protection Act.pdf
 
IPLOOK Remote-Sensing Satellite Solution
IPLOOK Remote-Sensing Satellite SolutionIPLOOK Remote-Sensing Satellite Solution
IPLOOK Remote-Sensing Satellite Solution
 
Computer HARDWARE presenattion by CWD students class 10
Computer HARDWARE presenattion by CWD students class 10Computer HARDWARE presenattion by CWD students class 10
Computer HARDWARE presenattion by CWD students class 10
 
Russian Girls Call Navi Mumbai 🎈🔥9920725232 🔥💋🎈 Provide Best And Top Girl Ser...
Russian Girls Call Navi Mumbai 🎈🔥9920725232 🔥💋🎈 Provide Best And Top Girl Ser...Russian Girls Call Navi Mumbai 🎈🔥9920725232 🔥💋🎈 Provide Best And Top Girl Ser...
Russian Girls Call Navi Mumbai 🎈🔥9920725232 🔥💋🎈 Provide Best And Top Girl Ser...
 
Litestack talk at Brighton 2024 (Unleashing the power of SQLite for Ruby apps)
Litestack talk at Brighton 2024 (Unleashing the power of SQLite for Ruby apps)Litestack talk at Brighton 2024 (Unleashing the power of SQLite for Ruby apps)
Litestack talk at Brighton 2024 (Unleashing the power of SQLite for Ruby apps)
 

Terms of endearment - the ElasticSearch Query DSL explained

  • 1. “ Terms of Endearment” The ElasticSearch query language explained Clinton Gormley, YAPC::EU 2011 DRTECH @clintongormley
  • 2. search for : “ DELETE QUERY ” We can
  • 3. search for : “ DELETE QUERY ” and find : “ deleteByQuery ” We can
  • 4. but you can only find what is stored in the database
  • 5. Normalise values “ deleteByQuery” 'delete' 'by' 'query' 'deletebyquery'
  • 6. Normalise values and search terms “ deleteByQuery” “ DELETE QUERY” ' delete ' 'by' ' query ' 'deletebyquery'
  • 7. Normalise values and search terms “ deleteByQuery” “ DELETE QUERY” ' delete ' 'by' ' query ' 'deletebyquery'
  • 8. Analyse values and search terms “ deleteByQuery” “ DELETE QUERY” ' delete ' 'by' ' query ' 'deletebyquery'
  • 9. What is stored in ElasticSearch?
  • 10. { tweet => "Perl is GREAT!", posted => "2011-08-15", user => { name => "Clinton Gormley", email => "drtech@cpan.org", }, tags => [" perl" ,"opinion"], posts => 2, } Document:
  • 11. { tweet => "Perl is GREAT!", posted => "2011-08-15", user => { name => "Clinton Gormley", email => "drtech@cpan.org", }, tags => [" perl" ,"opinion"], posts => 2, } Fields:
  • 12. { tweet => "Perl is GREAT!", posted => "2011-08-15", user => { name => "Clinton Gormley", email => "drtech@cpan.org", }, tags => [" perl" ,"opinion"], posts => 2, } Values:
  • 13. { tweet => "Perl is GREAT!", posted => "2011-08-15", user => { name => "Clinton Gormley", email => "drtech@cpan.org" }, tags => [" perl" ,"opinion"], posts => 2, } Field types: # object # string # date # nested object # string # string # array of enums # integer
  • 14. { tweet => "Perl is GREAT!", posted => "2011-08-15", user => { name => "Clinton Gormley", email => "drtech@cpan.org", }, tags => [" perl" ,"opinion"], posts => 2, } Nested objects flattened:
  • 15. { tweet => "Perl is GREAT!", posted => "2011-08-15", user.name => "Clinton Gormley", user.email => "drtech@cpan.org", tags => [" perl" ,"opinion"], posts => 2, } Nested objects flattened
  • 16. { tweet => "Perl is GREAT!", posted => "2011-08-15", user.name => "Clinton Gormley", user.email => "drtech@cpan.org", tags => [" perl" ,"opinion"], posts => 2, } Values analyzed into terms
  • 17. { tweet => ['perl','great'], posted => [Date(2011-08-15)], user.name => ['clinton','gormley'], user.email => ['drtech','cpan.org'], tags => [' perl' ,'opinion'], posts => [2], } Values analyzed into terms
  • 18. database table row ⇒ many tables ⇒ many rows ⇒ one schema ⇒ many columns In MySQL
  • 19. index type document ⇒ many types ⇒ many documents ⇒ one mapping ⇒ many fields In ElasticSearch
  • 20. Create index with mappings $es-> create_index ( index => 'twitter', mappings => { tweet => { properties => { title => { type => 'string' }, created => { type => 'date' } } } } );
  • 21. Add a mapping $es-> put_mapping ( index => 'twitter', type => ' user ', mapping => { properties => { name => { type => 'string' }, created => { type => 'date' }, } } );
  • 22. Can add to existing mapping
  • 23. Can add to existing mapping Cannot change mapping for field
  • 24. Core field types { type => 'string', }
  • 25. Core field types { type => 'string', # byte|short|integer|long|double|float # date, ip addr, geolocation # boolean # binary (as base 64) }
  • 26. Core field types { type => 'string', index => ' analyzed ', # 'Foo Bar' ⇒ [ 'foo', 'bar' ] }
  • 27. Core field types { type => 'string', index => ' not_analyzed ', # 'Foo Bar' ⇒ [ 'Foo Bar' ] }
  • 28. Core field types { type => 'string', index => ' no ', # 'Foo Bar' ⇒ [ ] }
  • 29. Core field types { type => 'string', index => 'analyzed', analyzer => 'default', }
  • 30. Core field types { type => 'string', index => 'analyzed', index_ analyzer => 'default', search_ analyzer => 'default', }
  • 31. Core field types { type => 'string', index => 'analyzed', analyzer => 'default', boost => 2, }
  • 32. Core field types { type => 'string', index => 'analyzed', analyzer => 'default', boost => 2, include_in_all => 1 |0 }
  • 36. Stop
  • 37. Keyword Built in analyzers Pattern
  • 41. The Brown-Cow's Part_No. #A.BC123-456 joe@bloggs.com keyword: The Brown-Cow's Part_No. #A.BC123-456 joe@bloggs.com whitespace: The, Brown-Cow's, Part_No., #A.BC123-456, joe@bloggs.com simple: the, brown, cow, s, part, no, a, bc, joe, bloggs, com standard: brown, cow's, part_no, a.bc123, 456, joe, bloggs.com snowball (English): brown, cow, part_no, a.bc123, 456, joe, bloggs.com
  • 46. NGram
  • 50. Stop
  • 52. KStem
  • 61. Custom Analyzer $c->create_index( index => 'twitter', settings => { analysis => { analyzer => { ascii_html => { type => 'custom', tokenizer => 'standard', filter => [ qw( standard lowercase asciifolding stop ) ], char_filter => ['html_strip'] } } }} );
  • 62. Searching $result = $es->search( index => 'twitter', type => 'tweet', );
  • 63. Searching $result = $es->search( index => ['twitter','facebook'] , type => ['tweet','post'] , );
  • 64. Searching $result = $es->search( # all indices # all types );
  • 65. Searching $result = $es->search( index => 'twitter', type => 'tweet', query => { text => { _all => 'foo' }}, );
  • 66. Searching $result = $es->search( index => 'twitter', type => 'tweet', query b => 'foo' , # b == ElasticSearch::SearchBuilder );
  • 67. Searching $result = $es->search( index => 'twitter', type => 'tweet', query => { text => { _all => 'foo' }}, sort => [{ '_score': 'desc' }] );
  • 68. Searching $result = $es->search( index => 'twitter', type => 'tweet', query => { text => { _all => 'foo' }}, sort => [{ '_score': 'desc' }] from => 0, size => 10, );
  • 70. Queries vs Filters
  • 71. Queries vs Filters full text & terms terms only
  • 72. Queries vs Filters full text & terms
  • 75. Queries vs Filters full text & terms
  • 80. Queries vs Filters full text & terms
  • 87. Queries vs Filters full text & terms
  • 93. cacheable Use filters for anything that doesn't affect the relevance score!
  • 94. Query only Query DSL: $es->search( query => { text => { title => 'perl' } } ); SearchBuilder: $es->search( query b => { title => 'perl' } );
  • 95. Filter only Query DSL: $es->search( query => { constant_score => { filter => { term => { tag => 'perl } } } }); SearchBuilder: $es->search( query b => { -filter => { tag => 'perl' } });
  • 96. Query and filter Query DSL: $es->search( query => { filtered => { query => { text => { title => 'perl' } }, filter =>{ term => { tag => 'perl' } } } }); SearchBuilder: $es->search( query b => { title => 'perl', -filter => { tag => 'perl' } });
  • 98. Filters : equality Query DSL: { term => { tags => 'perl' }} { terms => { tags => ['perl','ruby'] }} SearchBuilder: { tags => 'perl' } { tags => ['perl','ruby'] }
  • 99. Filters : range Query DSL: { range => { date => { gte => '2010-11-01', lt => '2010-12-01' }} SearchBuilder: { date => { gte => '2010-11-01', lt => '2011-12-01' }}
  • 100. Filters : range (many values) Query DSL: { numeric_range => { date => { gte => '2010-11-01', lt => '2010-12-01 }} SearchBuilder: { date => { ' >= ' => '2010-11-01', ' < ' => '2011-12-01' }}
  • 101. Filters : and | or | not Query DSL: { and => [ {term=>{X=>1}}, {term=>{Y=>2}} ]} { or => [ {term=>{X=>1}}, {term=>{Y=>2}} ]} { not => { or => [ {term=>{X=>1}}, {term=>{Y=>2}} ] }} SearchBuilder: { X => 1, Y => 2 } [ X => 1, Y => 2 ] { -not => { X => 1, Y => 2 } } # and { -not => [ X => 1, Y => 2 ] } # or
  • 102. Filters : exists | missing Query DSL: { exists => { field => 'title' }} { missing => { field => 'title' }} SearchBuilder: { -exists => 'title' } { -missing => 'title' }
  • 103. Filter example SearchBuilder: { -filter => [ featured => 1, { created_at => { gt => '2011-08-01' }, status => { '!=' => 'pending' }, }, ] }
  • 104. Filter example Query DSL: { constant_score => { filter => { or => [ { term => { featured => 1 }}, { and => [ { not => { term => { status => 'pending' }}, { range => { created_at => { gt => '2011-08-01' }}}, ] } ] } } }
  • 105. Filters : others script
  • 106. nested
  • 108. query
  • 110. prefix
  • 112. type
  • 120. mlt / mlt_field Term / Not analyzed: term / terms
  • 121. range
  • 122. prefix
  • 123. fuzzy
  • 125. ids
  • 137. mlt / mlt_field Term / Not analyzed: term / terms
  • 138. range
  • 139. prefix
  • 140. fuzzy
  • 142. ids
  • 153. Text/Analyzed Queries analyzed ⇒ text query using search_analyzer
  • 154. Text-Query Family Query DSL: { text => { title => 'great perl' }} Search Builder: { title => 'great perl' }
  • 155. Text-Query Family Query DSL: { text => { title => { query => 'great perl' }}} Search Builder: { title => { '=' => { query => 'great perl' }}}
  • 156. Text-Query Family Query DSL: { text => { title => { query => 'great perl' , operator => 'and' }}} Search Builder: { title => { '=' => { query => 'great perl', operator => 'and' }}}
  • 157. Text-Query Family Query DSL: { text => { title => { query => 'great perl' , fuzziness => 0.5 }}} Search Builder: { title => { '=' => { query => 'great perl', fuzziness => 0.5 }}}
  • 158. Text-Query Family Query DSL: { text => { title => { query => 'great perl', type => 'phrase' }}} Search Builder: { title => { '==' => { query => 'great perl', }}}
  • 159. Text-Query Family Query DSL: { text => { title => { query => ' great perl ', type => 'phrase' }}} Search Builder: { title => { '==' => { query => ' great perl ', }}}
  • 160. Text-Query Family Query DSL: { text => { title => { query => ' perl is great ', type => 'phrase' }}} Search Builder: { title => { '==' => { query => ' perl is great ', }}}
  • 161. Text-Query Family Query DSL: { text => { title => { query => ' perl great ', type => 'phrase', slop => 3 }}} Search Builder: { title => { '==' => { query => ' perl great ', slop => 3 }}}
  • 162. Text-Query Family Query DSL: { text => { title => { query => ' perl is gr ', type => ' phrase_prefix ', }}} Search Builder: { title => { '^' => { query => ' perl is gr ', }}}
  • 163. Query string / Field Lucene Query Syntax aware “ perl is great”~5 AND author:clint* -deleted
  • 164. Query string / Field Syntax errors: AND perl is great ” author : clint* -
  • 165. Query string / Field Syntax errors: AND perl is great ” author : clint* - ElasticSearch::QueryParser
  • 166. Combining: Bool Query DSL: { bool => { must => [ { term => { foo => 1}}, ... ], must_not => [ { term => { bar => 1}}, ... ], should => [ { term => { X => 2}}, { term => { Y => 2}},... ], minimum_number_should_match => 1, }}
  • 167. Combining: Bool SearchBuilder: { foo => 1, bar => { '!=' => 1}, -or => [ X => 2, Y => 2], } { -bool => { must => { foo => 1 }, must_not => { bar => 1 }, should => [{ X => 2}, { Y => 2 }], minimum_number_should_match => 1, }}
  • 168. Combining: DisMax Query DSL: { dis_max => { queries => [ { term => { foo => 1}}, { term => { bar => 1}}, ] }} SearchBuilder: { -dis_max => [ { term => { foo => 1}}, { term => { bar => 1}}, ], }
  • 169. Bool: combines scores DisMax: uses highest score from all matching clauses
  • 172. Boosting: at index time { properties => { content => { type => “string” }, title => { type => “string” }, }
  • 173. Boosting: at index time { properties => { content => { type => “string” }, title => { type => “string”, boost => 2, }, }, }
  • 174. Boosting: at index time { properties => { content => { type => “string” }, title => { type => “string”, boost => 2, }, rank => { type => “integer” }, }, _boost => { name => 'rank', null_value => 1.0 }, }
  • 175. Boosting: at search time Query DSL: { bool => { should => [ { text => { content => 'perl' }}, { text => { title => 'perl' }}, ] }} SearchBuilder: { content => 'perl', title => 'perl' }
  • 176. Boosting: at search time Query DSL: { bool => { should => [ { text => { content => 'perl' }}, { text => { title => { query => 'perl', }}, ] }} SearchBuilder: { content => 'perl', title => { '=' => { query => 'perl' }} }
  • 177. Boosting: at search time Query DSL: { bool => { should => [ { text => { content => 'perl' }}, { text => { title => { query => 'perl', boost => 2 }}, ] }} SearchBuilder: { content => 'perl', title => { '=' => { query => 'perl', boost=> 2 }} }
  • 178. Boosting: custom_score Query DSL: { custom_score => { query => { text => { title => 'perl' }}, script => “_score * foo /doc['rank'].value”, }} SearchBuilder: { -custom_score => { query => { title => 'perl' }, script => “_score * foo /doc['rank'].value”, }}
  • 179. Query example SearchBuilder: { -or => [ title => { '=' => { query => 'custom score', boost => 2 }}, content => 'custom score', ], -filter => { repo => 'elasticsearch/elasticsearch', created_at => { '>=' => '2011-07-01', '<' => '2011-08-01'}, -or => [ creator_id => 123, assignee_id => 123, ], labels => ['bug','breaking'] } }
  • 180. Query example Query DSL: { query => { filtered => { query => { bool => { should => [ { text => { content => &quot;custom score&quot; } }, { text => { title => { boost => 2, query => &quot;custom score&quot; } } }, ], }, }, filter => { and => [ { or => [ { term => { creator_id => 123 } }, { term => { assignee_id => 123 } }, ]}, { terms => { labels => [&quot;bug&quot;, &quot;breaking&quot;] } }, { term => { repo => &quot;elasticsearch/elasticsearch&quot; } }, { numeric_range => { created_at => { gte => &quot;2011-07-01&quot;, lt => &quot;2011-08-01&quot; }}}, ]}, }}
  • 181.