SlideShare a Scribd company logo
1 of 51
Download to read offline
Practical
full-text search
in PostgreSQL
Bill Karwin
PostgreSQL Conference West 09 • 2009/10/17
Me
• 20+ years experience
  •   Application/SDK developer
  •   Support, Training, Proj Mgmt
  •   C, Java, Perl, PHP

• SQL maven
  •   MySQL, PostgreSQL, InterBase
  •   Zend Framework
  •   Oracle, SQL Server, IBM DB2, SQLite

• Community contributor
Full Text Search
Text search



• Web applications demand speed
• Let’s compare 5 solutions for text search
Sample data


• StackOverflow.com Posts
  •   Data dump exported September 2009

  •   1.2 million tuples

  •   ~850 MB
StackOverflow ER diagram
Naive Searching
Some people, when confronted with a problem,
  think “I know, I’ll use regular expressions.”
        Now they have two problems.
                                     — Jamie Zawinsky
Performance issue

• LIKE with wildcards:             time: 91 sec
  SELECT * FROM Posts
  WHERE body LIKE ‘%postgresql%’
• POSIX regular expressions:
  SELECT * FROM Posts
  WHERE body ~ ‘postgresql’        time: 105 sec
Why so slow?

CREATE TABLE telephone_book (

 full_name
 
 VARCHAR(50)
);
CREATE INDEX name_idx ON telephone_book

 (full_name);
INSERT INTO telephone_book VALUES

 (‘Riddle, Thomas’),

 (‘Thomas, Dean’);
Why so slow?


• Search for all with last name “Thomas”
                                  uses
  SELECT * FROM telephone_book      index
  WHERE full_name LIKE ‘Thomas%’

• Search for all with first name “Thomas”
  SELECT * FROM telephone_book
  WHERE full_name LIKE ‘%Thomas’
                                    doesn’t
                                   use index
Indexes don’t help
searching for substrings
Accuracy issue

• Irrelevant or false matching words
  ‘one’, ‘money’, ‘prone’, etc.:
  body LIKE ‘%one%’
• Regular expressions in PostgreSQL
  support escapes for word boundaries:
  body ~ ‘yoney’
Solutions

• Full-Text Indexing in the RDBMS
• Sphinx Search
• Apache Lucene
• Inverted Index
• Search Engine Service
PostgreSQL
Text-Search
PostgreSQL Text-Search


• Since PostgreSQL 8.3
• TSVECTOR to represent text data
• TSQUERY to represent search predicates
• Special indexes
PostgreSQL Text-Search:

            Basic Querying



SELECT * FROM Posts
WHERE to_tsvector(title || ‘ ’ || body || ‘ ’ || tags)

 @@ to_tsquery(‘postgresql & performance’);

                   text-search
                    matching
                    operator
PostgreSQL Text-Search:

            Basic Querying



SELECT * FROM Posts
WHERE title || ‘ ’ || body || ‘ ’ || tags

 @@ ‘postgresql & performance’;

              time with no index:
                 8 min 2 sec
PostgreSQL Text-Search:

   Add TSVECTOR column


ALTER TABLE Posts ADD COLUMN

 PostText TSVECTOR;
UPDATE Posts SET PostText =

 to_tsvector(‘english’, title || ‘ ’ || body || ‘ ’ || tags);
Special index types



• GIN (generalized inverted index)
• GiST (generalized search tree)
PostgreSQL Text-Search:

             Indexing



CREATE INDEX PostText_GIN ON Posts

 USING GIN(PostText);


        time: 39 min 36 sec
PostgreSQL Text-Search:

               Querying



SELECT * FROM Posts
WHERE PostText @@ ‘postgresql & performance’;


           time with index:
           20 milliseconds
PostgreSQL Text-Search:

  Keep TSVECTOR in sync


CREATE TRIGGER TS_PostText

 BEFORE INSERT OR UPDATE ON Posts
FOR EACH ROW
EXECUTE PROCEDURE

 tsvector_update_trigger(
 ostText,
                               P

 
 ‘english’, title, body, tags);
Lucene
Lucene

• Full-text indexing and search engine
• Apache Project since 2001
• Apache License
• Java implementation
• Ports exist for C, Perl, Ruby, Python, PHP,
  etc.
Lucene:

            How to use


1. Add documents to index
2. Parse query
3. Execute query
Lucene:

         Creating an index



• Programmatic solution in Java...
            time: 8 minutes 55 seconds
Lucene:

                               Indexing
String url = "jdbc:postgresql:stackoverflow";
Properties props = new Properties();
props.setProperty("user", "postgres");
                                                              run any SQL query
Class.forName("org.postgresql.Driver");
Connection con = DriverManager.getConnection(url, props);

Statement stmt = con.createStatement();
String sql = "SELECT PostId, Title, Body, Tags FROM Posts";
ResultSet rs = stmt.executeQuery(sql);
                                                                open Lucene
Date start = new Date();                                        index writer
IndexWriter writer = new IndexWriter(FSDirectory.open(INDEX_DIR),

 new StandardAnalyzer(Version.LUCENE_CURRENT),

 true, IndexWriter.MaxFieldLength.LIMITED);
Lucene:

                                    Indexing
       loop over SQL result

while (rs.next()) {
 Document doc = new Document();

    doc.add(new Field("PostId", rs.getString("PostId"), Field.Store.YES, Field.Index.NO));
    doc.add(new Field("Title", rs.getString("Title"), Field.Store.YES, Field.Index.ANALYZED));
    doc.add(new Field("Body", rs.getString("Body"), Field.Store.YES, Field.Index.ANALYZED));
    doc.add(new Field("Tags", rs.getString("Tags"), Field.Store.YES, Field.Index.ANALYZED));

    writer.addDocument(doc);           each row is
}
                                      a Document
writer.optimize();
writer.close();
                                     with four Fields


                finish and
               close index
Lucene:

                            Querying

• Parse a Lucene query                                         define fields
  String[] fields = new String[3];
  fields[0] = “title”; fields[1] = “body”; fields[2] = “tags”;

  Query q = new MultiFieldQueryParser(fields,
  
  new StandardAnalyzer()).parse(‘performance’);


• Execute the query                                           parse search
                                                                 query
  Searcher s = new IndexSearcher(indexName);

  Hits h = s.search(q);
                                                    time: 80 milliseconds
Sphinx Search
Sphinx Search


• Embedded full-text search engine
• Started in 2001
• GPLv2 license
• Good database integration
Sphinx Search:

            How to use


1. Edit configuration file
2. Index the data
3. Query the index
4. Issues
Sphinx Search:

                sphinx.conf

source stackoverflowsrc
{

 type = pgsql

 sql_host = localhost

 sql_user = postgres

 sql_pass = xxxx

 sql_db = stackoverflow

 sql_query = SELECT PostId, Title, Body, Tags FROM Posts

 sql_query_info = SELECT * FROM Posts WHERE PostId=$id
}
Sphinx Search:

                 sphinx.conf


index stackoverflow
{

 source = stackoverflowsrc

 path = /opt/local/var/db/sphinx/stackoverflow
}
Sphinx Search:

               Building index


indexer -c sphinx.conf stackoverflow
collected 1242365 docs, 720.5 MB
sorted 88.3 Mhits, 100.0% done
total 1242365 docs, 720452944 bytes
total 357.647 sec, 2014423.75 bytes/sec, 3473.72 docs/sec



                   time: 5 min 57 sec
Sphinx Search:

         Querying index



search -c sphinx.conf -i stackoverflow

 -b “sql & performance”


           time: 8 milliseconds
Sphinx Search:

                        Issues

• Index updates are as expensive as
  rebuilding the index from scratch
  •   Maintain “main” index plus “delta” index for
      recent changes

  •   Merge indexes periodically

  •   Not all data fits into this model
Inverted Index
Inverted index

                             searchable words




Posts           Tags                 TagTypes



           intersection of
            words / Posts
Inverted index:

Updated ER Diagram
Inverted index:

               Data definition
CREATE TABLE TagTypes (

  TagId
 
     SERIAL PRIMARY KEY,

  Tag
 
  
    VARCHAR(50) NOT NULL
);

CREATE UNIQUE INDEX TagTypes_Tag_index ON TagTypes(Tag);

CREATE TABLE Tags (

  PostId
 
    INT NOT NULL,

  TagId
 
     INT NOT NULL,

  PRIMARY KEY (PostId, TagId),

  FOREIGN KEY (PostId) REFERENCES Posts (PostId),

  FOREIGN KEY (TagId) REFERENCES TagTypes (TagId)
);

CREATE INDEX Tags_PostId_index ON Tags(PostId);
CREATE INDEX Tags_TagId_index ON Tags(TagId);
Inverted index:

               Indexing


INSERT INTO Tags (PostId, TagId)

 SELECT p.PostId, t.TagId

 FROM Posts p JOIN TagTypes t

 ON (p.Tags LIKE ‘%<’ || t.Tag || ‘>%’);

                90 seconds
                 per tag!!
Inverted index:

             Querying


SELECT p.* FROM Posts p
JOIN Tags t USING (PostId)
JOIN TagTypes tt USING (TagId)
WHERE tt.Tag = ‘performance’;


               40 milliseconds
Search Engine Services
Search engine services:

Google Custom Search Engine

• http://www.google.com/cse/



• DEMO ➪    http://www.karwin.com/demo/gcse-demo.html


                                            even big web sites
                                             use this solution
Search engine services:

         Is it right for you?


• Your site is public and allows external index
• Search is a non-critical feature for you
• Search results are satisfactory
• You need to offload search processing
Comparison: Time to Build Index
LIKE predicate      none

PostgreSQL / GIN   40 min

Sphinx Search       6 min

Apache Lucene       9 min

Inverted index       high

Google / Yahoo!     offline
Comparison: Index Storage
LIKE predicate        none

PostgreSQL / GIN     532 MB

Sphinx Search        533 MB

Apache Lucene        1071 MB

Inverted index       101 MB

Google / Yahoo!       offline
Comparison: Query Speed
LIKE predicate      90+ sec

PostgreSQL / GIN    20 ms

Sphinx Search        8 ms

Apache Lucene       80 ms

Inverted index      40 ms

Google / Yahoo!        *
Comparison: Bottom-Line
                   indexing   storage    query     solution

LIKE predicate     none       none      11,250x     SQL

PostgreSQL / GIN     7x       5.3x       2.5x     RDBMS

Sphinx Search       1x *      5.3x        1x      3rd party

Apache Lucene       1.5x       10x       10x      3rd party

Inverted index      high       1x         5x        SQL

Google / Yahoo!    offline     offline       *       Service
Copyright 2009 Bill Karwin
        www.slideshare.net/billkarwin
              Released under a Creative Commons 3.0 License:
              http://creativecommons.org/licenses/by-nc-nd/3.0/

                You are free to share - to copy, distribute and
             transmit this work, under the following conditions:

   Attribution.                Noncommercial.          No Derivative Works.
You must attribute this    You may not use this work       You may not alter,
 work to Bill Karwin.       for commercial purposes.      transform, or build
                                                            upon this work.

More Related Content

What's hot

Typed Properties and more: What's coming in PHP 7.4?
Typed Properties and more: What's coming in PHP 7.4?Typed Properties and more: What's coming in PHP 7.4?
Typed Properties and more: What's coming in PHP 7.4?Nikita Popov
 
Workshop Spring - Session 1 - L'offre Spring et les bases
Workshop Spring  - Session 1 - L'offre Spring et les basesWorkshop Spring  - Session 1 - L'offre Spring et les bases
Workshop Spring - Session 1 - L'offre Spring et les basesAntoine Rey
 
Node.js Express
Node.js  ExpressNode.js  Express
Node.js ExpressEyal Vardi
 
[APJ] Common Table Expressions (CTEs) in SQL
[APJ] Common Table Expressions (CTEs) in SQL[APJ] Common Table Expressions (CTEs) in SQL
[APJ] Common Table Expressions (CTEs) in SQLEDB
 
This keyword in java
This keyword in javaThis keyword in java
This keyword in javaHitesh Kumar
 
C10k and beyond - Uri Shamay, Akamai
C10k and beyond - Uri Shamay, AkamaiC10k and beyond - Uri Shamay, Akamai
C10k and beyond - Uri Shamay, AkamaiCodemotion Tel Aviv
 
More mastering the art of indexing
More mastering the art of indexingMore mastering the art of indexing
More mastering the art of indexingYoshinori Matsunobu
 
Federated Engine 실무적용사례
Federated Engine 실무적용사례Federated Engine 실무적용사례
Federated Engine 실무적용사례I Goo Lee
 
Java Generics Introduction - Syntax Advantages and Pitfalls
Java Generics Introduction - Syntax Advantages and PitfallsJava Generics Introduction - Syntax Advantages and Pitfalls
Java Generics Introduction - Syntax Advantages and PitfallsRakesh Waghela
 
Mysql Explain Explained
Mysql Explain ExplainedMysql Explain Explained
Mysql Explain ExplainedJeremy Coates
 

What's hot (20)

Typed Properties and more: What's coming in PHP 7.4?
Typed Properties and more: What's coming in PHP 7.4?Typed Properties and more: What's coming in PHP 7.4?
Typed Properties and more: What's coming in PHP 7.4?
 
Workshop Spring - Session 1 - L'offre Spring et les bases
Workshop Spring  - Session 1 - L'offre Spring et les basesWorkshop Spring  - Session 1 - L'offre Spring et les bases
Workshop Spring - Session 1 - L'offre Spring et les bases
 
MYSQL - PHP Database Connectivity
MYSQL - PHP Database ConnectivityMYSQL - PHP Database Connectivity
MYSQL - PHP Database Connectivity
 
Writing clean code
Writing clean codeWriting clean code
Writing clean code
 
Major Java 8 features
Major Java 8 featuresMajor Java 8 features
Major Java 8 features
 
Node.js Express
Node.js  ExpressNode.js  Express
Node.js Express
 
Java 8 Workshop
Java 8 WorkshopJava 8 Workshop
Java 8 Workshop
 
[APJ] Common Table Expressions (CTEs) in SQL
[APJ] Common Table Expressions (CTEs) in SQL[APJ] Common Table Expressions (CTEs) in SQL
[APJ] Common Table Expressions (CTEs) in SQL
 
This keyword in java
This keyword in javaThis keyword in java
This keyword in java
 
Arrays in java
Arrays in javaArrays in java
Arrays in java
 
Sql Antipatterns Strike Back
Sql Antipatterns Strike BackSql Antipatterns Strike Back
Sql Antipatterns Strike Back
 
Java 8 Lambda and Streams
Java 8 Lambda and StreamsJava 8 Lambda and Streams
Java 8 Lambda and Streams
 
C10k and beyond - Uri Shamay, Akamai
C10k and beyond - Uri Shamay, AkamaiC10k and beyond - Uri Shamay, Akamai
C10k and beyond - Uri Shamay, Akamai
 
How to Design Indexes, Really
How to Design Indexes, ReallyHow to Design Indexes, Really
How to Design Indexes, Really
 
Clean code
Clean codeClean code
Clean code
 
More mastering the art of indexing
More mastering the art of indexingMore mastering the art of indexing
More mastering the art of indexing
 
Federated Engine 실무적용사례
Federated Engine 실무적용사례Federated Engine 실무적용사례
Federated Engine 실무적용사례
 
Modern JS with ES6
Modern JS with ES6Modern JS with ES6
Modern JS with ES6
 
Java Generics Introduction - Syntax Advantages and Pitfalls
Java Generics Introduction - Syntax Advantages and PitfallsJava Generics Introduction - Syntax Advantages and Pitfalls
Java Generics Introduction - Syntax Advantages and Pitfalls
 
Mysql Explain Explained
Mysql Explain ExplainedMysql Explain Explained
Mysql Explain Explained
 

Similar to Full Text Search In PostgreSQL

Examiness hints and tips from the trenches
Examiness hints and tips from the trenchesExaminess hints and tips from the trenches
Examiness hints and tips from the trenchesIsmail Mayat
 
Full Text Search with Lucene
Full Text Search with LuceneFull Text Search with Lucene
Full Text Search with LuceneWO Community
 
Quick Introduction to Sphinx and Thinking Sphinx
Quick Introduction to Sphinx and Thinking SphinxQuick Introduction to Sphinx and Thinking Sphinx
Quick Introduction to Sphinx and Thinking Sphinxhayesdavis
 
Simon Elliston Ball – When to NoSQL and When to Know SQL - NoSQL matters Barc...
Simon Elliston Ball – When to NoSQL and When to Know SQL - NoSQL matters Barc...Simon Elliston Ball – When to NoSQL and When to Know SQL - NoSQL matters Barc...
Simon Elliston Ball – When to NoSQL and When to Know SQL - NoSQL matters Barc...NoSQLmatters
 
How to use the new Domino Query Language
How to use the new Domino Query LanguageHow to use the new Domino Query Language
How to use the new Domino Query LanguageTim Davis
 
Infinispan,Lucene,Hibername OGM
Infinispan,Lucene,Hibername OGMInfinispan,Lucene,Hibername OGM
Infinispan,Lucene,Hibername OGMJBug Italy
 
Using Thinking Sphinx with rails
Using Thinking Sphinx with railsUsing Thinking Sphinx with rails
Using Thinking Sphinx with railsRishav Dixit
 
Advanced full text searching techniques using Lucene
Advanced full text searching techniques using LuceneAdvanced full text searching techniques using Lucene
Advanced full text searching techniques using LuceneAsad Abbas
 
Lucene Introduction
Lucene IntroductionLucene Introduction
Lucene Introductionotisg
 
What is the best full text search engine for Python?
What is the best full text search engine for Python?What is the best full text search engine for Python?
What is the best full text search engine for Python?Andrii Soldatenko
 
ElasticSearch AJUG 2013
ElasticSearch AJUG 2013ElasticSearch AJUG 2013
ElasticSearch AJUG 2013Roy Russo
 
dotNet Miami - June 21, 2012: Richie Rump: Entity Framework: Code First and M...
dotNet Miami - June 21, 2012: Richie Rump: Entity Framework: Code First and M...dotNet Miami - June 21, 2012: Richie Rump: Entity Framework: Code First and M...
dotNet Miami - June 21, 2012: Richie Rump: Entity Framework: Code First and M...dotNet Miami
 
Entity Framework: Code First and Magic Unicorns
Entity Framework: Code First and Magic UnicornsEntity Framework: Code First and Magic Unicorns
Entity Framework: Code First and Magic UnicornsRichie Rump
 
10 Reasons to Start Your Analytics Project with PostgreSQL
10 Reasons to Start Your Analytics Project with PostgreSQL10 Reasons to Start Your Analytics Project with PostgreSQL
10 Reasons to Start Your Analytics Project with PostgreSQLSatoshi Nagayasu
 
ElasticSearch for .NET Developers
ElasticSearch for .NET DevelopersElasticSearch for .NET Developers
ElasticSearch for .NET DevelopersBen van Mol
 
An Introduction to Elastic Search.
An Introduction to Elastic Search.An Introduction to Elastic Search.
An Introduction to Elastic Search.Jurriaan Persyn
 
Полнотекстовый поиск в PostgreSQL за миллисекунды (Олег Бартунов, Александр К...
Полнотекстовый поиск в PostgreSQL за миллисекунды (Олег Бартунов, Александр К...Полнотекстовый поиск в PostgreSQL за миллисекунды (Олег Бартунов, Александр К...
Полнотекстовый поиск в PostgreSQL за миллисекунды (Олег Бартунов, Александр К...Ontico
 
About elasticsearch
About elasticsearchAbout elasticsearch
About elasticsearchMinsoo Jun
 

Similar to Full Text Search In PostgreSQL (20)

Examiness hints and tips from the trenches
Examiness hints and tips from the trenchesExaminess hints and tips from the trenches
Examiness hints and tips from the trenches
 
Full Text Search with Lucene
Full Text Search with LuceneFull Text Search with Lucene
Full Text Search with Lucene
 
Quick Introduction to Sphinx and Thinking Sphinx
Quick Introduction to Sphinx and Thinking SphinxQuick Introduction to Sphinx and Thinking Sphinx
Quick Introduction to Sphinx and Thinking Sphinx
 
Simon Elliston Ball – When to NoSQL and When to Know SQL - NoSQL matters Barc...
Simon Elliston Ball – When to NoSQL and When to Know SQL - NoSQL matters Barc...Simon Elliston Ball – When to NoSQL and When to Know SQL - NoSQL matters Barc...
Simon Elliston Ball – When to NoSQL and When to Know SQL - NoSQL matters Barc...
 
How to use the new Domino Query Language
How to use the new Domino Query LanguageHow to use the new Domino Query Language
How to use the new Domino Query Language
 
Infinispan,Lucene,Hibername OGM
Infinispan,Lucene,Hibername OGMInfinispan,Lucene,Hibername OGM
Infinispan,Lucene,Hibername OGM
 
Using Thinking Sphinx with rails
Using Thinking Sphinx with railsUsing Thinking Sphinx with rails
Using Thinking Sphinx with rails
 
PostgreSQL
PostgreSQLPostgreSQL
PostgreSQL
 
Advanced full text searching techniques using Lucene
Advanced full text searching techniques using LuceneAdvanced full text searching techniques using Lucene
Advanced full text searching techniques using Lucene
 
Lucene Introduction
Lucene IntroductionLucene Introduction
Lucene Introduction
 
What is the best full text search engine for Python?
What is the best full text search engine for Python?What is the best full text search engine for Python?
What is the best full text search engine for Python?
 
ElasticSearch AJUG 2013
ElasticSearch AJUG 2013ElasticSearch AJUG 2013
ElasticSearch AJUG 2013
 
dotNet Miami - June 21, 2012: Richie Rump: Entity Framework: Code First and M...
dotNet Miami - June 21, 2012: Richie Rump: Entity Framework: Code First and M...dotNet Miami - June 21, 2012: Richie Rump: Entity Framework: Code First and M...
dotNet Miami - June 21, 2012: Richie Rump: Entity Framework: Code First and M...
 
Entity Framework: Code First and Magic Unicorns
Entity Framework: Code First and Magic UnicornsEntity Framework: Code First and Magic Unicorns
Entity Framework: Code First and Magic Unicorns
 
Lucene in Action
Lucene in ActionLucene in Action
Lucene in Action
 
10 Reasons to Start Your Analytics Project with PostgreSQL
10 Reasons to Start Your Analytics Project with PostgreSQL10 Reasons to Start Your Analytics Project with PostgreSQL
10 Reasons to Start Your Analytics Project with PostgreSQL
 
ElasticSearch for .NET Developers
ElasticSearch for .NET DevelopersElasticSearch for .NET Developers
ElasticSearch for .NET Developers
 
An Introduction to Elastic Search.
An Introduction to Elastic Search.An Introduction to Elastic Search.
An Introduction to Elastic Search.
 
Полнотекстовый поиск в PostgreSQL за миллисекунды (Олег Бартунов, Александр К...
Полнотекстовый поиск в PostgreSQL за миллисекунды (Олег Бартунов, Александр К...Полнотекстовый поиск в PostgreSQL за миллисекунды (Олег Бартунов, Александр К...
Полнотекстовый поиск в PostgreSQL за миллисекунды (Олег Бартунов, Александр К...
 
About elasticsearch
About elasticsearchAbout elasticsearch
About elasticsearch
 

More from Karwin Software Solutions LLC (15)

Recursive Query Throwdown
Recursive Query ThrowdownRecursive Query Throwdown
Recursive Query Throwdown
 
Load Data Fast!
Load Data Fast!Load Data Fast!
Load Data Fast!
 
InnoDB Locking Explained with Stick Figures
InnoDB Locking Explained with Stick FiguresInnoDB Locking Explained with Stick Figures
InnoDB Locking Explained with Stick Figures
 
SQL Outer Joins for Fun and Profit
SQL Outer Joins for Fun and ProfitSQL Outer Joins for Fun and Profit
SQL Outer Joins for Fun and Profit
 
Extensible Data Modeling
Extensible Data ModelingExtensible Data Modeling
Extensible Data Modeling
 
Sql query patterns, optimized
Sql query patterns, optimizedSql query patterns, optimized
Sql query patterns, optimized
 
Survey of Percona Toolkit
Survey of Percona ToolkitSurvey of Percona Toolkit
Survey of Percona Toolkit
 
Schemadoc
SchemadocSchemadoc
Schemadoc
 
Percona toolkit
Percona toolkitPercona toolkit
Percona toolkit
 
MySQL 5.5 Guide to InnoDB Status
MySQL 5.5 Guide to InnoDB StatusMySQL 5.5 Guide to InnoDB Status
MySQL 5.5 Guide to InnoDB Status
 
Requirements the Last Bottleneck
Requirements the Last BottleneckRequirements the Last Bottleneck
Requirements the Last Bottleneck
 
Mentor Your Indexes
Mentor Your IndexesMentor Your Indexes
Mentor Your Indexes
 
Models for hierarchical data
Models for hierarchical dataModels for hierarchical data
Models for hierarchical data
 
Sql Injection Myths and Fallacies
Sql Injection Myths and FallaciesSql Injection Myths and Fallacies
Sql Injection Myths and Fallacies
 
Practical Object Oriented Models In Sql
Practical Object Oriented Models In SqlPractical Object Oriented Models In Sql
Practical Object Oriented Models In Sql
 

Recently uploaded

Navigating the Large Language Model choices_Ravi Daparthi
Navigating the Large Language Model choices_Ravi DaparthiNavigating the Large Language Model choices_Ravi Daparthi
Navigating the Large Language Model choices_Ravi DaparthiRaviKumarDaparthi
 
Six Myths about Ontologies: The Basics of Formal Ontology
Six Myths about Ontologies: The Basics of Formal OntologySix Myths about Ontologies: The Basics of Formal Ontology
Six Myths about Ontologies: The Basics of Formal Ontologyjohnbeverley2021
 
AI mind or machine power point presentation
AI mind or machine power point presentationAI mind or machine power point presentation
AI mind or machine power point presentationyogeshlabana357357
 
TrustArc Webinar - Unified Trust Center for Privacy, Security, Compliance, an...
TrustArc Webinar - Unified Trust Center for Privacy, Security, Compliance, an...TrustArc Webinar - Unified Trust Center for Privacy, Security, Compliance, an...
TrustArc Webinar - Unified Trust Center for Privacy, Security, Compliance, an...TrustArc
 
AI in Action: Real World Use Cases by Anitaraj
AI in Action: Real World Use Cases by AnitarajAI in Action: Real World Use Cases by Anitaraj
AI in Action: Real World Use Cases by AnitarajAnitaRaj43
 
Continuing Bonds Through AI: A Hermeneutic Reflection on Thanabots
Continuing Bonds Through AI: A Hermeneutic Reflection on ThanabotsContinuing Bonds Through AI: A Hermeneutic Reflection on Thanabots
Continuing Bonds Through AI: A Hermeneutic Reflection on ThanabotsLeah Henrickson
 
ChatGPT and Beyond - Elevating DevOps Productivity
ChatGPT and Beyond - Elevating DevOps ProductivityChatGPT and Beyond - Elevating DevOps Productivity
ChatGPT and Beyond - Elevating DevOps ProductivityVictorSzoltysek
 
UiPath manufacturing technology benefits and AI overview
UiPath manufacturing technology benefits and AI overviewUiPath manufacturing technology benefits and AI overview
UiPath manufacturing technology benefits and AI overviewDianaGray10
 
Portal Kombat : extension du réseau de propagande russe
Portal Kombat : extension du réseau de propagande russePortal Kombat : extension du réseau de propagande russe
Portal Kombat : extension du réseau de propagande russe中 央社
 
The Ultimate Prompt Engineering Guide for Generative AI: Get the Most Out of ...
The Ultimate Prompt Engineering Guide for Generative AI: Get the Most Out of ...The Ultimate Prompt Engineering Guide for Generative AI: Get the Most Out of ...
The Ultimate Prompt Engineering Guide for Generative AI: Get the Most Out of ...SOFTTECHHUB
 
Introduction to use of FHIR Documents in ABDM
Introduction to use of FHIR Documents in ABDMIntroduction to use of FHIR Documents in ABDM
Introduction to use of FHIR Documents in ABDMKumar Satyam
 
Tales from a Passkey Provider Progress from Awareness to Implementation.pptx
Tales from a Passkey Provider  Progress from Awareness to Implementation.pptxTales from a Passkey Provider  Progress from Awareness to Implementation.pptx
Tales from a Passkey Provider Progress from Awareness to Implementation.pptxFIDO Alliance
 
Finding Java's Hidden Performance Traps @ DevoxxUK 2024
Finding Java's Hidden Performance Traps @ DevoxxUK 2024Finding Java's Hidden Performance Traps @ DevoxxUK 2024
Finding Java's Hidden Performance Traps @ DevoxxUK 2024Victor Rentea
 
Introduction to FIDO Authentication and Passkeys.pptx
Introduction to FIDO Authentication and Passkeys.pptxIntroduction to FIDO Authentication and Passkeys.pptx
Introduction to FIDO Authentication and Passkeys.pptxFIDO Alliance
 
AI+A11Y 11MAY2024 HYDERBAD GAAD 2024 - HelloA11Y (11 May 2024)
AI+A11Y 11MAY2024 HYDERBAD GAAD 2024 - HelloA11Y (11 May 2024)AI+A11Y 11MAY2024 HYDERBAD GAAD 2024 - HelloA11Y (11 May 2024)
AI+A11Y 11MAY2024 HYDERBAD GAAD 2024 - HelloA11Y (11 May 2024)Samir Dash
 
WebRTC and SIP not just audio and video @ OpenSIPS 2024
WebRTC and SIP not just audio and video @ OpenSIPS 2024WebRTC and SIP not just audio and video @ OpenSIPS 2024
WebRTC and SIP not just audio and video @ OpenSIPS 2024Lorenzo Miniero
 
Generative AI Use Cases and Applications.pdf
Generative AI Use Cases and Applications.pdfGenerative AI Use Cases and Applications.pdf
Generative AI Use Cases and Applications.pdfalexjohnson7307
 
CORS (Kitworks Team Study 양다윗 발표자료 240510)
CORS (Kitworks Team Study 양다윗 발표자료 240510)CORS (Kitworks Team Study 양다윗 발표자료 240510)
CORS (Kitworks Team Study 양다윗 발표자료 240510)Wonjun Hwang
 
Design and Development of a Provenance Capture Platform for Data Science
Design and Development of a Provenance Capture Platform for Data ScienceDesign and Development of a Provenance Capture Platform for Data Science
Design and Development of a Provenance Capture Platform for Data SciencePaolo Missier
 
Event-Driven Architecture Masterclass: Engineering a Robust, High-performance...
Event-Driven Architecture Masterclass: Engineering a Robust, High-performance...Event-Driven Architecture Masterclass: Engineering a Robust, High-performance...
Event-Driven Architecture Masterclass: Engineering a Robust, High-performance...ScyllaDB
 

Recently uploaded (20)

Navigating the Large Language Model choices_Ravi Daparthi
Navigating the Large Language Model choices_Ravi DaparthiNavigating the Large Language Model choices_Ravi Daparthi
Navigating the Large Language Model choices_Ravi Daparthi
 
Six Myths about Ontologies: The Basics of Formal Ontology
Six Myths about Ontologies: The Basics of Formal OntologySix Myths about Ontologies: The Basics of Formal Ontology
Six Myths about Ontologies: The Basics of Formal Ontology
 
AI mind or machine power point presentation
AI mind or machine power point presentationAI mind or machine power point presentation
AI mind or machine power point presentation
 
TrustArc Webinar - Unified Trust Center for Privacy, Security, Compliance, an...
TrustArc Webinar - Unified Trust Center for Privacy, Security, Compliance, an...TrustArc Webinar - Unified Trust Center for Privacy, Security, Compliance, an...
TrustArc Webinar - Unified Trust Center for Privacy, Security, Compliance, an...
 
AI in Action: Real World Use Cases by Anitaraj
AI in Action: Real World Use Cases by AnitarajAI in Action: Real World Use Cases by Anitaraj
AI in Action: Real World Use Cases by Anitaraj
 
Continuing Bonds Through AI: A Hermeneutic Reflection on Thanabots
Continuing Bonds Through AI: A Hermeneutic Reflection on ThanabotsContinuing Bonds Through AI: A Hermeneutic Reflection on Thanabots
Continuing Bonds Through AI: A Hermeneutic Reflection on Thanabots
 
ChatGPT and Beyond - Elevating DevOps Productivity
ChatGPT and Beyond - Elevating DevOps ProductivityChatGPT and Beyond - Elevating DevOps Productivity
ChatGPT and Beyond - Elevating DevOps Productivity
 
UiPath manufacturing technology benefits and AI overview
UiPath manufacturing technology benefits and AI overviewUiPath manufacturing technology benefits and AI overview
UiPath manufacturing technology benefits and AI overview
 
Portal Kombat : extension du réseau de propagande russe
Portal Kombat : extension du réseau de propagande russePortal Kombat : extension du réseau de propagande russe
Portal Kombat : extension du réseau de propagande russe
 
The Ultimate Prompt Engineering Guide for Generative AI: Get the Most Out of ...
The Ultimate Prompt Engineering Guide for Generative AI: Get the Most Out of ...The Ultimate Prompt Engineering Guide for Generative AI: Get the Most Out of ...
The Ultimate Prompt Engineering Guide for Generative AI: Get the Most Out of ...
 
Introduction to use of FHIR Documents in ABDM
Introduction to use of FHIR Documents in ABDMIntroduction to use of FHIR Documents in ABDM
Introduction to use of FHIR Documents in ABDM
 
Tales from a Passkey Provider Progress from Awareness to Implementation.pptx
Tales from a Passkey Provider  Progress from Awareness to Implementation.pptxTales from a Passkey Provider  Progress from Awareness to Implementation.pptx
Tales from a Passkey Provider Progress from Awareness to Implementation.pptx
 
Finding Java's Hidden Performance Traps @ DevoxxUK 2024
Finding Java's Hidden Performance Traps @ DevoxxUK 2024Finding Java's Hidden Performance Traps @ DevoxxUK 2024
Finding Java's Hidden Performance Traps @ DevoxxUK 2024
 
Introduction to FIDO Authentication and Passkeys.pptx
Introduction to FIDO Authentication and Passkeys.pptxIntroduction to FIDO Authentication and Passkeys.pptx
Introduction to FIDO Authentication and Passkeys.pptx
 
AI+A11Y 11MAY2024 HYDERBAD GAAD 2024 - HelloA11Y (11 May 2024)
AI+A11Y 11MAY2024 HYDERBAD GAAD 2024 - HelloA11Y (11 May 2024)AI+A11Y 11MAY2024 HYDERBAD GAAD 2024 - HelloA11Y (11 May 2024)
AI+A11Y 11MAY2024 HYDERBAD GAAD 2024 - HelloA11Y (11 May 2024)
 
WebRTC and SIP not just audio and video @ OpenSIPS 2024
WebRTC and SIP not just audio and video @ OpenSIPS 2024WebRTC and SIP not just audio and video @ OpenSIPS 2024
WebRTC and SIP not just audio and video @ OpenSIPS 2024
 
Generative AI Use Cases and Applications.pdf
Generative AI Use Cases and Applications.pdfGenerative AI Use Cases and Applications.pdf
Generative AI Use Cases and Applications.pdf
 
CORS (Kitworks Team Study 양다윗 발표자료 240510)
CORS (Kitworks Team Study 양다윗 발표자료 240510)CORS (Kitworks Team Study 양다윗 발표자료 240510)
CORS (Kitworks Team Study 양다윗 발표자료 240510)
 
Design and Development of a Provenance Capture Platform for Data Science
Design and Development of a Provenance Capture Platform for Data ScienceDesign and Development of a Provenance Capture Platform for Data Science
Design and Development of a Provenance Capture Platform for Data Science
 
Event-Driven Architecture Masterclass: Engineering a Robust, High-performance...
Event-Driven Architecture Masterclass: Engineering a Robust, High-performance...Event-Driven Architecture Masterclass: Engineering a Robust, High-performance...
Event-Driven Architecture Masterclass: Engineering a Robust, High-performance...
 

Full Text Search In PostgreSQL

  • 1. Practical full-text search in PostgreSQL Bill Karwin PostgreSQL Conference West 09 • 2009/10/17
  • 2. Me • 20+ years experience • Application/SDK developer • Support, Training, Proj Mgmt • C, Java, Perl, PHP • SQL maven • MySQL, PostgreSQL, InterBase • Zend Framework • Oracle, SQL Server, IBM DB2, SQLite • Community contributor
  • 4. Text search • Web applications demand speed • Let’s compare 5 solutions for text search
  • 5. Sample data • StackOverflow.com Posts • Data dump exported September 2009 • 1.2 million tuples • ~850 MB
  • 7. Naive Searching Some people, when confronted with a problem, think “I know, I’ll use regular expressions.” Now they have two problems. — Jamie Zawinsky
  • 8. Performance issue • LIKE with wildcards: time: 91 sec SELECT * FROM Posts WHERE body LIKE ‘%postgresql%’ • POSIX regular expressions: SELECT * FROM Posts WHERE body ~ ‘postgresql’ time: 105 sec
  • 9. Why so slow? CREATE TABLE telephone_book ( full_name VARCHAR(50) ); CREATE INDEX name_idx ON telephone_book (full_name); INSERT INTO telephone_book VALUES (‘Riddle, Thomas’), (‘Thomas, Dean’);
  • 10. Why so slow? • Search for all with last name “Thomas” uses SELECT * FROM telephone_book index WHERE full_name LIKE ‘Thomas%’ • Search for all with first name “Thomas” SELECT * FROM telephone_book WHERE full_name LIKE ‘%Thomas’ doesn’t use index
  • 12. Accuracy issue • Irrelevant or false matching words ‘one’, ‘money’, ‘prone’, etc.: body LIKE ‘%one%’ • Regular expressions in PostgreSQL support escapes for word boundaries: body ~ ‘yoney’
  • 13. Solutions • Full-Text Indexing in the RDBMS • Sphinx Search • Apache Lucene • Inverted Index • Search Engine Service
  • 15. PostgreSQL Text-Search • Since PostgreSQL 8.3 • TSVECTOR to represent text data • TSQUERY to represent search predicates • Special indexes
  • 16. PostgreSQL Text-Search: Basic Querying SELECT * FROM Posts WHERE to_tsvector(title || ‘ ’ || body || ‘ ’ || tags) @@ to_tsquery(‘postgresql & performance’); text-search matching operator
  • 17. PostgreSQL Text-Search: Basic Querying SELECT * FROM Posts WHERE title || ‘ ’ || body || ‘ ’ || tags @@ ‘postgresql & performance’; time with no index: 8 min 2 sec
  • 18. PostgreSQL Text-Search: Add TSVECTOR column ALTER TABLE Posts ADD COLUMN PostText TSVECTOR; UPDATE Posts SET PostText = to_tsvector(‘english’, title || ‘ ’ || body || ‘ ’ || tags);
  • 19. Special index types • GIN (generalized inverted index) • GiST (generalized search tree)
  • 20. PostgreSQL Text-Search: Indexing CREATE INDEX PostText_GIN ON Posts USING GIN(PostText); time: 39 min 36 sec
  • 21. PostgreSQL Text-Search: Querying SELECT * FROM Posts WHERE PostText @@ ‘postgresql & performance’; time with index: 20 milliseconds
  • 22. PostgreSQL Text-Search: Keep TSVECTOR in sync CREATE TRIGGER TS_PostText BEFORE INSERT OR UPDATE ON Posts FOR EACH ROW EXECUTE PROCEDURE tsvector_update_trigger( ostText, P ‘english’, title, body, tags);
  • 24. Lucene • Full-text indexing and search engine • Apache Project since 2001 • Apache License • Java implementation • Ports exist for C, Perl, Ruby, Python, PHP, etc.
  • 25. Lucene: How to use 1. Add documents to index 2. Parse query 3. Execute query
  • 26. Lucene: Creating an index • Programmatic solution in Java... time: 8 minutes 55 seconds
  • 27. Lucene: Indexing String url = "jdbc:postgresql:stackoverflow"; Properties props = new Properties(); props.setProperty("user", "postgres"); run any SQL query Class.forName("org.postgresql.Driver"); Connection con = DriverManager.getConnection(url, props); Statement stmt = con.createStatement(); String sql = "SELECT PostId, Title, Body, Tags FROM Posts"; ResultSet rs = stmt.executeQuery(sql); open Lucene Date start = new Date(); index writer IndexWriter writer = new IndexWriter(FSDirectory.open(INDEX_DIR), new StandardAnalyzer(Version.LUCENE_CURRENT), true, IndexWriter.MaxFieldLength.LIMITED);
  • 28. Lucene: Indexing loop over SQL result while (rs.next()) { Document doc = new Document(); doc.add(new Field("PostId", rs.getString("PostId"), Field.Store.YES, Field.Index.NO)); doc.add(new Field("Title", rs.getString("Title"), Field.Store.YES, Field.Index.ANALYZED)); doc.add(new Field("Body", rs.getString("Body"), Field.Store.YES, Field.Index.ANALYZED)); doc.add(new Field("Tags", rs.getString("Tags"), Field.Store.YES, Field.Index.ANALYZED)); writer.addDocument(doc); each row is } a Document writer.optimize(); writer.close(); with four Fields finish and close index
  • 29. Lucene: Querying • Parse a Lucene query define fields String[] fields = new String[3]; fields[0] = “title”; fields[1] = “body”; fields[2] = “tags”; Query q = new MultiFieldQueryParser(fields, new StandardAnalyzer()).parse(‘performance’); • Execute the query parse search query Searcher s = new IndexSearcher(indexName); Hits h = s.search(q); time: 80 milliseconds
  • 31. Sphinx Search • Embedded full-text search engine • Started in 2001 • GPLv2 license • Good database integration
  • 32. Sphinx Search: How to use 1. Edit configuration file 2. Index the data 3. Query the index 4. Issues
  • 33. Sphinx Search: sphinx.conf source stackoverflowsrc { type = pgsql sql_host = localhost sql_user = postgres sql_pass = xxxx sql_db = stackoverflow sql_query = SELECT PostId, Title, Body, Tags FROM Posts sql_query_info = SELECT * FROM Posts WHERE PostId=$id }
  • 34. Sphinx Search: sphinx.conf index stackoverflow { source = stackoverflowsrc path = /opt/local/var/db/sphinx/stackoverflow }
  • 35. Sphinx Search: Building index indexer -c sphinx.conf stackoverflow collected 1242365 docs, 720.5 MB sorted 88.3 Mhits, 100.0% done total 1242365 docs, 720452944 bytes total 357.647 sec, 2014423.75 bytes/sec, 3473.72 docs/sec time: 5 min 57 sec
  • 36. Sphinx Search: Querying index search -c sphinx.conf -i stackoverflow -b “sql & performance” time: 8 milliseconds
  • 37. Sphinx Search: Issues • Index updates are as expensive as rebuilding the index from scratch • Maintain “main” index plus “delta” index for recent changes • Merge indexes periodically • Not all data fits into this model
  • 39. Inverted index searchable words Posts Tags TagTypes intersection of words / Posts
  • 41. Inverted index: Data definition CREATE TABLE TagTypes ( TagId SERIAL PRIMARY KEY, Tag VARCHAR(50) NOT NULL ); CREATE UNIQUE INDEX TagTypes_Tag_index ON TagTypes(Tag); CREATE TABLE Tags ( PostId INT NOT NULL, TagId INT NOT NULL, PRIMARY KEY (PostId, TagId), FOREIGN KEY (PostId) REFERENCES Posts (PostId), FOREIGN KEY (TagId) REFERENCES TagTypes (TagId) ); CREATE INDEX Tags_PostId_index ON Tags(PostId); CREATE INDEX Tags_TagId_index ON Tags(TagId);
  • 42. Inverted index: Indexing INSERT INTO Tags (PostId, TagId) SELECT p.PostId, t.TagId FROM Posts p JOIN TagTypes t ON (p.Tags LIKE ‘%<’ || t.Tag || ‘>%’); 90 seconds per tag!!
  • 43. Inverted index: Querying SELECT p.* FROM Posts p JOIN Tags t USING (PostId) JOIN TagTypes tt USING (TagId) WHERE tt.Tag = ‘performance’; 40 milliseconds
  • 45. Search engine services: Google Custom Search Engine • http://www.google.com/cse/ • DEMO ➪ http://www.karwin.com/demo/gcse-demo.html even big web sites use this solution
  • 46. Search engine services: Is it right for you? • Your site is public and allows external index • Search is a non-critical feature for you • Search results are satisfactory • You need to offload search processing
  • 47. Comparison: Time to Build Index LIKE predicate none PostgreSQL / GIN 40 min Sphinx Search 6 min Apache Lucene 9 min Inverted index high Google / Yahoo! offline
  • 48. Comparison: Index Storage LIKE predicate none PostgreSQL / GIN 532 MB Sphinx Search 533 MB Apache Lucene 1071 MB Inverted index 101 MB Google / Yahoo! offline
  • 49. Comparison: Query Speed LIKE predicate 90+ sec PostgreSQL / GIN 20 ms Sphinx Search 8 ms Apache Lucene 80 ms Inverted index 40 ms Google / Yahoo! *
  • 50. Comparison: Bottom-Line indexing storage query solution LIKE predicate none none 11,250x SQL PostgreSQL / GIN 7x 5.3x 2.5x RDBMS Sphinx Search 1x * 5.3x 1x 3rd party Apache Lucene 1.5x 10x 10x 3rd party Inverted index high 1x 5x SQL Google / Yahoo! offline offline * Service
  • 51. Copyright 2009 Bill Karwin www.slideshare.net/billkarwin Released under a Creative Commons 3.0 License: http://creativecommons.org/licenses/by-nc-nd/3.0/ You are free to share - to copy, distribute and transmit this work, under the following conditions: Attribution. Noncommercial. No Derivative Works. You must attribute this You may not use this work You may not alter, work to Bill Karwin. for commercial purposes. transform, or build upon this work.