How to navigate the rich but confusing field of (Full) Text Search in PostgreSQL. A short introduction will explain the concepts involved, followed by a discussion of functions, operators, indexes and collation support in Postgres in relevance to searching for text. Examples of usage will be provided, along with some stats demonstrating the differences.
Full text search in PostgreSQL is a flexible and powerful facility to search collection of documents using natural language queries. We will discuss several new improvements of FTS in PostgreSQL 9.6 release, such as phrase search, better dictionaries support and tsvector editing functions. Also, we will present new features currently in development - RUM index support, which enables acceleration of some important kinds of full text queries, new and better ranking function for relevance search, loading dictionaries into shared memory and support for search multilingual content.
QuestDB: The building blocks of a fast open-source time-series databasejavier ramirez
(talk delivered at OSA CON 23)
Traditionally, databases have treated timestamps just as another data type. However, when performing real-time analytics, timestamps should be first class citizens and we need rich time semantics to get the most out of our data. We also need to deal with ever growing datasets while keeping performant, which is as fun as it sounds.
It is no wonder time-series databases are now more popular than ever before. Join me in this session to learn about the internal architecture and building blocks of QuestDB, an open source time-series database designed for speed.
We will learn how it deals with data ingestion, and which SQL extensions it implements for working with time-series efficiently.
We will also review a history of some of the changes we have gone over the past two years to deal with late and unordered data, non-blocking writes, read-replicas, or data deduplication.
Full text search in PostgreSQL is a flexible and powerful facility to search collection of documents using natural language queries. We will discuss several new improvements of FTS in PostgreSQL 9.6 release, such as phrase search, better dictionaries support and tsvector editing functions. Also, we will present new features currently in development - RUM index support, which enables acceleration of some important kinds of full text queries, new and better ranking function for relevance search, loading dictionaries into shared memory and support for search multilingual content.
QuestDB: The building blocks of a fast open-source time-series databasejavier ramirez
(talk delivered at OSA CON 23)
Traditionally, databases have treated timestamps just as another data type. However, when performing real-time analytics, timestamps should be first class citizens and we need rich time semantics to get the most out of our data. We also need to deal with ever growing datasets while keeping performant, which is as fun as it sounds.
It is no wonder time-series databases are now more popular than ever before. Join me in this session to learn about the internal architecture and building blocks of QuestDB, an open source time-series database designed for speed.
We will learn how it deals with data ingestion, and which SQL extensions it implements for working with time-series efficiently.
We will also review a history of some of the changes we have gone over the past two years to deal with late and unordered data, non-blocking writes, read-replicas, or data deduplication.
Новые возможности полнотекстового поиска в PostgreSQL / Олег Бартунов (Postgr...Ontico
Я расскажу про новые возможности полнотекстового поиска, которые вошли в последний релиз PostgreSQL - поддержку фразового поиска и набор функций для манипулирования полнотекстовым типом данных (tsvector). Помимо этого, мы улучшили поддержку морфологических словарей, что привело к значительному увеличению числа поддерживаемых языков, оптимизировали работу со словарями, разработали новый индексный метод доступа RUM, который значительно ускорил выполнение ряда запросов с полнотекстовыми операторами.
Your Timestamps Deserve Better than a Generic Databasejavier ramirez
If you are storing records with a timestamp in your database, it is very likely a time series database can make your life easier.
However, time series databases are still the great unknown for a large part of the tech community.
In this talk, I will show you what use cases they are good for, what they give you that you cannot get from a traditional database, and when it is a good idea (and when it is not) to use them.
For the demos, we will be using QuestDB, the fastest open-source time series database.
String Comparison Surprises: Did Postgres lose my data?Jeremy Schneider
Comparisons are fundamental to computing - and comparing strings is not nearly as straightforward as you might think. Come learn about the history, nuance and surprises of “putting words in order” that you never knew existed in computer science, and how that nuance impacts both general programming and SQL programming. Next, walk through a few actual scenarios and demonstrations using PostgreSQL as a user and administrator, which you can re-run yourself later for further study, including one way you could easily corrupt your self-managed PostgreSQL database if you aren't prepared. Finally we’ll dive into an explanation of the surprising behaviors we saw in PostgreSQL, and learn more about user and administrative features PostgreSQL provides related to localized string comparison.
Search is an important part of informative web-sites, but there are many different possible solutions to implement such a search. This session evaluates possible options for the integration of a search engine into your web-site, ranging from simple solutions as MySQL's full text to using an external engine to power search
This presentation by Bruce Momjian. Co-Founder of the Global PostgreSQL Development team and a Senior Architect at EDB. He demonstrates how to use arrays, geometry and JSON for NoSQL data types to overcome restrictions of relational storage to support new innovative applications, specifically by storing and indexing multiple values, even unrelated ones, in a single database field. Such storage allows for greater efficiency and access simplicity, and can also avoid the negatives of entity-attribute-value (eav) storage.
Postgres has always had strong support for relational storage. However, there are some cases where relational storage might be inefficient or overly restrictive.
Based on the legendary "Don't Do This" PostgreSQL wiki page, this talk explores some of the common pitfalls and misconceptions that Postgres users can face - and shows possible ways to undo them or workarounds.
Some of the things discussed:
- Bad SQL habits
- Correct types for data storage
- (Sub-)Partitioning (and how to get it wrong)
- Table inheritance (and how to undo it)
- Connections (number of, and properly handling)
- Security issues (unsafe configurations and usage)
Talk given at FOSDEM 2023
Slow things down to make them go faster [FOSDEM 2022]Jimmy Angelakos
Talk from FOSDEM 2022
It's easy to get misled into overconfidence based on the performance of powerful servers, given today's monster core counts and RAM sizes. However, the reality of high concurrency usage is often disappointing, with less throughput than one would expect. Because of its internals and its multi-process architecture, PostgreSQL is very particular about how it likes to deal with high concurrency and in some cases it can slow down to the point where it looks like it's not performing as it should. In this talk we'll take a look at potential pitfalls when you throw a lot of work at your database. Specifically, very high concurrency and resource contention can cause problems with lock waits in Postgres. Very high transaction rates can also cause problems of a different nature. Finally, we will be looking at ways to mitigate these by examining our queries and connection parameters, leveraging connection pooling and replication, or adapting the workload.
Topics:
1. Understand what we mean by high concurrency.
2. Understand ACID & MVCC in Postgres.
3. Understand how high concurrency affects Postgres performance.
4. Understand how locks/latches affect Postgres performance.
5. Understand how high transaction rates can affect Postgres.
6. Mitigation strategies for high concurrency scenarios.
More Related Content
Similar to The State of (Full) Text Search in PostgreSQL 12
Новые возможности полнотекстового поиска в PostgreSQL / Олег Бартунов (Postgr...Ontico
Я расскажу про новые возможности полнотекстового поиска, которые вошли в последний релиз PostgreSQL - поддержку фразового поиска и набор функций для манипулирования полнотекстовым типом данных (tsvector). Помимо этого, мы улучшили поддержку морфологических словарей, что привело к значительному увеличению числа поддерживаемых языков, оптимизировали работу со словарями, разработали новый индексный метод доступа RUM, который значительно ускорил выполнение ряда запросов с полнотекстовыми операторами.
Your Timestamps Deserve Better than a Generic Databasejavier ramirez
If you are storing records with a timestamp in your database, it is very likely a time series database can make your life easier.
However, time series databases are still the great unknown for a large part of the tech community.
In this talk, I will show you what use cases they are good for, what they give you that you cannot get from a traditional database, and when it is a good idea (and when it is not) to use them.
For the demos, we will be using QuestDB, the fastest open-source time series database.
String Comparison Surprises: Did Postgres lose my data?Jeremy Schneider
Comparisons are fundamental to computing - and comparing strings is not nearly as straightforward as you might think. Come learn about the history, nuance and surprises of “putting words in order” that you never knew existed in computer science, and how that nuance impacts both general programming and SQL programming. Next, walk through a few actual scenarios and demonstrations using PostgreSQL as a user and administrator, which you can re-run yourself later for further study, including one way you could easily corrupt your self-managed PostgreSQL database if you aren't prepared. Finally we’ll dive into an explanation of the surprising behaviors we saw in PostgreSQL, and learn more about user and administrative features PostgreSQL provides related to localized string comparison.
Search is an important part of informative web-sites, but there are many different possible solutions to implement such a search. This session evaluates possible options for the integration of a search engine into your web-site, ranging from simple solutions as MySQL's full text to using an external engine to power search
This presentation by Bruce Momjian. Co-Founder of the Global PostgreSQL Development team and a Senior Architect at EDB. He demonstrates how to use arrays, geometry and JSON for NoSQL data types to overcome restrictions of relational storage to support new innovative applications, specifically by storing and indexing multiple values, even unrelated ones, in a single database field. Such storage allows for greater efficiency and access simplicity, and can also avoid the negatives of entity-attribute-value (eav) storage.
Postgres has always had strong support for relational storage. However, there are some cases where relational storage might be inefficient or overly restrictive.
Based on the legendary "Don't Do This" PostgreSQL wiki page, this talk explores some of the common pitfalls and misconceptions that Postgres users can face - and shows possible ways to undo them or workarounds.
Some of the things discussed:
- Bad SQL habits
- Correct types for data storage
- (Sub-)Partitioning (and how to get it wrong)
- Table inheritance (and how to undo it)
- Connections (number of, and properly handling)
- Security issues (unsafe configurations and usage)
Talk given at FOSDEM 2023
Slow things down to make them go faster [FOSDEM 2022]Jimmy Angelakos
Talk from FOSDEM 2022
It's easy to get misled into overconfidence based on the performance of powerful servers, given today's monster core counts and RAM sizes. However, the reality of high concurrency usage is often disappointing, with less throughput than one would expect. Because of its internals and its multi-process architecture, PostgreSQL is very particular about how it likes to deal with high concurrency and in some cases it can slow down to the point where it looks like it's not performing as it should. In this talk we'll take a look at potential pitfalls when you throw a lot of work at your database. Specifically, very high concurrency and resource contention can cause problems with lock waits in Postgres. Very high transaction rates can also cause problems of a different nature. Finally, we will be looking at ways to mitigate these by examining our queries and connection parameters, leveraging connection pooling and replication, or adapting the workload.
Topics:
1. Understand what we mean by high concurrency.
2. Understand ACID & MVCC in Postgres.
3. Understand how high concurrency affects Postgres performance.
4. Understand how locks/latches affect Postgres performance.
5. Understand how high transaction rates can affect Postgres.
6. Mitigation strategies for high concurrency scenarios.
Practical Partitioning in Production with PostgresJimmy Angelakos
Has your table become too large to handle? Have you thought about chopping it up into smaller pieces that are easier to query and maintain? What if it's in constant use?
An introduction to the problems that can arise and how PostgreSQL's partitioning features can help, followed by a real-world scenario of partitioning an existing huge table on a live system.
Talk from Postgres Vision 2021
Changing your huge table's data types in productionJimmy Angelakos
You have a huge table, and it is necessary to change a column's data type, but your database has to keep running with no downtime. What do you do?
Here's one way to perform this change, in as unobtrusive a manner as possible while your table keeps serving users, by avoiding long DDL table locks and leveraging procedural transaction control.
Talk from the PostgreSQL devroom at FOSDEM 2021
A look at some of the ways available to deploy Postgres in a Kubernetes cloud environment, either in small scale using simple configurations, or in larger scale using tools such as Helm charts and the Crunchy PostgreSQL Operator. A short introduction to Kubernetes will be given to explain the concepts involved, followed by examples from each deployment method and observations on the key differences.
Bringing the Semantic Web closer to reality: PostgreSQL as RDF Graph DatabaseJimmy Angelakos
Presentation of an investigation into how Python's RDFLib and SQLAlchemy can be used to leverage PostgreSQL's capabilities to provide a persistent storage back-end for Graphs, and become the elusive practical RDF triple store for the Semantic Web (or simply help you export your data to someone who's expecting RDF)!
Talk presented at FOSDEM 2017 in Brussels on 04-05/02/2017. Practical & hands-on presentation with example code which is certainly not optimal ;)
Video:
MP4: http://video.fosdem.org/2017/H.1309/postgresql_semantic_web.mp4
WebM/VP8: http://ftp.osuosl.org/pub/fosdem/2017/H.1309/postgresql_semantic_web.vp8.webm
An investigation of how PostgreSQL and its latest capabilities (JSONB data type, GIN indices, Full Text Search) can be used to store, index and perform queries on structured Bibliographic Data such as MARC21/MARCXML, breaking the dependence on proprietary and arcane or obsolete software products.
Talk presented at FOSDEM 2016 in Brussels on 31/01/2016. This is a very practical & hands-on presentation with example code which is certainly not optimal ;)
Eισαγωγή στην PostgreSQL - Χρήση σε επιχειρησιακό περιβάλλονJimmy Angelakos
Παρουσίασή μου για τη βάση δεδομένων PostgreSQL και τη χρήση της σε επιχειρησιακό περιβάλλον, στα πλαίσια της εκδήλωσης "Προηγμένες Εφαρμογές της βάσης δεδομένων PostgreSQL" στις 26/6/2013 στο Εθνικό Ίδρυμα Ερευνών.
Παρουσίασή μου για τη βάση δεδομένων PostgreSQL και των δυνατοτήτων Data Replication που προσφέρει, στα πλαίσια του 6ου Συνεδρίου Κοινοτήτων Ανοιχτού Λογισμικού FOSSCOMM 2013 στις 21/4/2013 στο Χαροκόπειο Πανεπιστήμιο Αθηνών.
Listen to the keynote address and hear about the latest developments from Rachana Ananthakrishnan and Ian Foster who review the updates to the Globus Platform and Service, and the relevance of Globus to the scientific community as an automation platform to accelerate scientific discovery.
Enhancing Research Orchestration Capabilities at ORNL.pdfGlobus
Cross-facility research orchestration comes with ever-changing constraints regarding the availability and suitability of various compute and data resources. In short, a flexible data and processing fabric is needed to enable the dynamic redirection of data and compute tasks throughout the lifecycle of an experiment. In this talk, we illustrate how we easily leveraged Globus services to instrument the ACE research testbed at the Oak Ridge Leadership Computing Facility with flexible data and task orchestration capabilities.
Providing Globus Services to Users of JASMIN for Environmental Data AnalysisGlobus
JASMIN is the UK’s high-performance data analysis platform for environmental science, operated by STFC on behalf of the UK Natural Environment Research Council (NERC). In addition to its role in hosting the CEDA Archive (NERC’s long-term repository for climate, atmospheric science & Earth observation data in the UK), JASMIN provides a collaborative platform to a community of around 2,000 scientists in the UK and beyond, providing nearly 400 environmental science projects with working space, compute resources and tools to facilitate their work. High-performance data transfer into and out of JASMIN has always been a key feature, with many scientists bringing model outputs from supercomputers elsewhere in the UK, to analyse against observational or other model data in the CEDA Archive. A growing number of JASMIN users are now realising the benefits of using the Globus service to provide reliable and efficient data movement and other tasks in this and other contexts. Further use cases involve long-distance (intercontinental) transfers to and from JASMIN, and collecting results from a mobile atmospheric radar system, pushing data to JASMIN via a lightweight Globus deployment. We provide details of how Globus fits into our current infrastructure, our experience of the recent migration to GCSv5.4, and of our interest in developing use of the wider ecosystem of Globus services for the benefit of our user community.
Climate Science Flows: Enabling Petabyte-Scale Climate Analysis with the Eart...Globus
The Earth System Grid Federation (ESGF) is a global network of data servers that archives and distributes the planet’s largest collection of Earth system model output for thousands of climate and environmental scientists worldwide. Many of these petabyte-scale data archives are located in proximity to large high-performance computing (HPC) or cloud computing resources, but the primary workflow for data users consists of transferring data, and applying computations on a different system. As a part of the ESGF 2.0 US project (funded by the United States Department of Energy Office of Science), we developed pre-defined data workflows, which can be run on-demand, capable of applying many data reduction and data analysis to the large ESGF data archives, transferring only the resultant analysis (ex. visualizations, smaller data files). In this talk, we will showcase a few of these workflows, highlighting how Globus Flows can be used for petabyte-scale climate analysis.
Prosigns: Transforming Business with Tailored Technology SolutionsProsigns
Unlocking Business Potential: Tailored Technology Solutions by Prosigns
Discover how Prosigns, a leading technology solutions provider, partners with businesses to drive innovation and success. Our presentation showcases our comprehensive range of services, including custom software development, web and mobile app development, AI & ML solutions, blockchain integration, DevOps services, and Microsoft Dynamics 365 support.
Custom Software Development: Prosigns specializes in creating bespoke software solutions that cater to your unique business needs. Our team of experts works closely with you to understand your requirements and deliver tailor-made software that enhances efficiency and drives growth.
Web and Mobile App Development: From responsive websites to intuitive mobile applications, Prosigns develops cutting-edge solutions that engage users and deliver seamless experiences across devices.
AI & ML Solutions: Harnessing the power of Artificial Intelligence and Machine Learning, Prosigns provides smart solutions that automate processes, provide valuable insights, and drive informed decision-making.
Blockchain Integration: Prosigns offers comprehensive blockchain solutions, including development, integration, and consulting services, enabling businesses to leverage blockchain technology for enhanced security, transparency, and efficiency.
DevOps Services: Prosigns' DevOps services streamline development and operations processes, ensuring faster and more reliable software delivery through automation and continuous integration.
Microsoft Dynamics 365 Support: Prosigns provides comprehensive support and maintenance services for Microsoft Dynamics 365, ensuring your system is always up-to-date, secure, and running smoothly.
Learn how our collaborative approach and dedication to excellence help businesses achieve their goals and stay ahead in today's digital landscape. From concept to deployment, Prosigns is your trusted partner for transforming ideas into reality and unlocking the full potential of your business.
Join us on a journey of innovation and growth. Let's partner for success with Prosigns.
Enhancing Project Management Efficiency_ Leveraging AI Tools like ChatGPT.pdfJay Das
With the advent of artificial intelligence or AI tools, project management processes are undergoing a transformative shift. By using tools like ChatGPT, and Bard organizations can empower their leaders and managers to plan, execute, and monitor projects more effectively.
top nidhi software solution freedownloadvrstrong314
This presentation emphasizes the importance of data security and legal compliance for Nidhi companies in India. It highlights how online Nidhi software solutions, like Vector Nidhi Software, offer advanced features tailored to these needs. Key aspects include encryption, access controls, and audit trails to ensure data security. The software complies with regulatory guidelines from the MCA and RBI and adheres to Nidhi Rules, 2014. With customizable, user-friendly interfaces and real-time features, these Nidhi software solutions enhance efficiency, support growth, and provide exceptional member services. The presentation concludes with contact information for further inquiries.
Large Language Models and the End of ProgrammingMatt Welsh
Talk by Matt Welsh at Craft Conference 2024 on the impact that Large Language Models will have on the future of software development. In this talk, I discuss the ways in which LLMs will impact the software industry, from replacing human software developers with AI, to replacing conventional software with models that perform reasoning, computation, and problem-solving.
TROUBLESHOOTING 9 TYPES OF OUTOFMEMORYERRORTier1 app
Even though at surface level ‘java.lang.OutOfMemoryError’ appears as one single error; underlyingly there are 9 types of OutOfMemoryError. Each type of OutOfMemoryError has different causes, diagnosis approaches and solutions. This session equips you with the knowledge, tools, and techniques needed to troubleshoot and conquer OutOfMemoryError in all its forms, ensuring smoother, more efficient Java applications.
SOCRadar Research Team: Latest Activities of IntelBrokerSOCRadar
The European Union Agency for Law Enforcement Cooperation (Europol) has suffered an alleged data breach after a notorious threat actor claimed to have exfiltrated data from its systems. Infamous data leaker IntelBroker posted on the even more infamous BreachForums hacking forum, saying that Europol suffered a data breach this month.
The alleged breach affected Europol agencies CCSE, EC3, Europol Platform for Experts, Law Enforcement Forum, and SIRIUS. Infiltration of these entities can disrupt ongoing investigations and compromise sensitive intelligence shared among international law enforcement agencies.
However, this is neither the first nor the last activity of IntekBroker. We have compiled for you what happened in the last few days. To track such hacker activities on dark web sources like hacker forums, private Telegram channels, and other hidden platforms where cyber threats often originate, you can check SOCRadar’s Dark Web News.
Stay Informed on Threat Actors’ Activity on the Dark Web with SOCRadar!
Globus Compute wth IRI Workflows - GlobusWorld 2024Globus
As part of the DOE Integrated Research Infrastructure (IRI) program, NERSC at Lawrence Berkeley National Lab and ALCF at Argonne National Lab are working closely with General Atomics on accelerating the computing requirements of the DIII-D experiment. As part of the work the team is investigating ways to speedup the time to solution for many different parts of the DIII-D workflow including how they run jobs on HPC systems. One of these routes is looking at Globus Compute as a way to replace the current method for managing tasks and we describe a brief proof of concept showing how Globus Compute could help to schedule jobs and be a tool to connect compute at different facilities.
Globus Connect Server Deep Dive - GlobusWorld 2024Globus
We explore the Globus Connect Server (GCS) architecture and experiment with advanced configuration options and use cases. This content is targeted at system administrators who are familiar with GCS and currently operate—or are planning to operate—broader deployments at their institution.
Quarkus Hidden and Forbidden ExtensionsMax Andersen
Quarkus has a vast extension ecosystem and is known for its subsonic and subatomic feature set. Some of these features are not as well known, and some extensions are less talked about, but that does not make them less interesting - quite the opposite.
Come join this talk to see some tips and tricks for using Quarkus and some of the lesser known features, extensions and development techniques.
Top Features to Include in Your Winzo Clone App for Business Growth (4).pptxrickgrimesss22
Discover the essential features to incorporate in your Winzo clone app to boost business growth, enhance user engagement, and drive revenue. Learn how to create a compelling gaming experience that stands out in the competitive market.
Accelerate Enterprise Software Engineering with PlatformlessWSO2
Key takeaways:
Challenges of building platforms and the benefits of platformless.
Key principles of platformless, including API-first, cloud-native middleware, platform engineering, and developer experience.
How Choreo enables the platformless experience.
How key concepts like application architecture, domain-driven design, zero trust, and cell-based architecture are inherently a part of Choreo.
Demo of an end-to-end app built and deployed on Choreo.
Paketo Buildpacks : la meilleure façon de construire des images OCI? DevopsDa...Anthony Dahanne
Les Buildpacks existent depuis plus de 10 ans ! D’abord, ils étaient utilisés pour détecter et construire une application avant de la déployer sur certains PaaS. Ensuite, nous avons pu créer des images Docker (OCI) avec leur dernière génération, les Cloud Native Buildpacks (CNCF en incubation). Sont-ils une bonne alternative au Dockerfile ? Que sont les buildpacks Paketo ? Quelles communautés les soutiennent et comment ?
Venez le découvrir lors de cette session ignite
Developing Distributed High-performance Computing Capabilities of an Open Sci...Globus
COVID-19 had an unprecedented impact on scientific collaboration. The pandemic and its broad response from the scientific community has forged new relationships among public health practitioners, mathematical modelers, and scientific computing specialists, while revealing critical gaps in exploiting advanced computing systems to support urgent decision making. Informed by our team’s work in applying high-performance computing in support of public health decision makers during the COVID-19 pandemic, we present how Globus technologies are enabling the development of an open science platform for robust epidemic analysis, with the goal of collaborative, secure, distributed, on-demand, and fast time-to-solution analyses to support public health.
Developing Distributed High-performance Computing Capabilities of an Open Sci...
The State of (Full) Text Search in PostgreSQL 12
1. https://www.2ndQuadrant.com
Event / Conference name
Location, Date
The State of (Full) Text
Search in PostgreSQL 12
FOSDEM 2020
Jimmy Angelakos
Senior PostgreSQL Architect
Twitter: @vyruss 🏴 🇪🇺 🇬🇷
3. https://www.2ndQuadrant.com
FOSDEM
Brussels, 2020-02-02
Your attention please
● This presentation contains linguistics, NLP,
Markov chains, Levenshtein distances, and
various other confounding terms.
● These have been known to induce drowsiness
and inappropriate sleep onset in lecture theatres.
Allergy advice
4. https://www.2ndQuadrant.com
FOSDEM
Brussels, 2020-02-02
What is Text?
(Baby don’t hurt me)
●
PostgreSQL character types
– CHAR(n)
– VARCHAR(n)
– VARCHAR, TEXT
●
Trailing spaces: significant (e.g. for LIKE / regex)
●
Storage
– Character Set (e.g. UTF-8)
– 1+126 bytes 4+→ n bytes
– Compression, TOAST
5. https://www.2ndQuadrant.com
FOSDEM
Brussels, 2020-02-02
What is Text Search?
●
Information retrieval Text retrieval→
●
Search on metadata
– Descriptive, bibliographic, tags, etc.
– Discovery & identification
●
Search on parts of the text
– Matching
– Substring search
– Data extraction, cleaning, mining
6. https://www.2ndQuadrant.com
FOSDEM
Brussels, 2020-02-02
Text search operators in PostgreSQL
●
LIKE, ILIKE (~~, ~~*)
●
~, ~* (POSIX regex)
●
regexp_match(string text, pattern text)
●
But are SQL/regular expressions enough?
– No ranking of results
– No concept of language
– Cannot be indexed
●
Okay okay, can be somewhat indexed*
●
SIMILAR TO best forget about this one→
7. https://www.2ndQuadrant.com
FOSDEM
Brussels, 2020-02-02
What is Full Text Search (FTS)?
●
Information retrieval Text retrieval Document retrieval→ →
●
Search on words (on tokens) in a database (all documents)
●
No index Serial search (e.g.→ grep)
●
Indexing Avoid scanning whole documents→
●
Techniques for criteria-based matching
– Natural Language Processing (NLP)
●
Precision vs Recall
– Stop words
– Stemming
8. https://www.2ndQuadrant.com
FOSDEM
Brussels, 2020-02-02
Documents? Tokens?
●
Document: a chunk of text (a field in a row)
●
Parsing of documents into classes of tokens
– PostgreSQL parser (or write your own… in C)
●
Conversion of tokens into lexemes
– Normalisation of strings
●
Lexeme: an abstract lexical unit representing related
words (i.e. word root)
– SEARCH searched, searcher→
13. https://www.2ndQuadrant.com
FOSDEM
Brussels, 2020-02-02
Dictionaries in PostgreSQL
●
Programs!
●
Accept tokens as input
●
Improve search quality
– Eliminate stop words
– Normalise words into lexemes
●
Reduce size of tsvector
●
CREATE TEXT SEARCH DICTIONARY name
(TEMPLATE = simple, STOPWORDS = english);
●
Can be chained: most specific more general→
ALTER TEXT SEARCH CONFIGURATION name
ADD MAPPING FOR word WITH english_ispell, simple;
●
ispell, myspell, hunspell, etc.
14. https://www.2ndQuadrant.com
FOSDEM
Brussels, 2020-02-02
Text matching example (1)
fts=# SELECT to_tsvector('A nice day for a car ride')
fts-# @@ plainto_tsquery('I am riding');
?column?
----------
t
(1 row)
fts=# SELECT to_tsvector('A nice day for a car ride');
to_tsvector
-----------------------------------
'car':6 'day':3 'nice':2 'ride':7
(1 row)
fts=# SELECT plainto_tsquery('I am riding');
plainto_tsquery
-----------------
'ride'
(1 row)
15. https://www.2ndQuadrant.com
FOSDEM
Brussels, 2020-02-02
Text matching example (2)
fts=# SELECT to_tsvector('A nice day for a car ride')
fts-# @@ plainto_tsquery('I am riding a bike');
?column?
----------
f
(1 row)
fts=# SELECT to_tsvector('A nice day for a car ride');
to_tsvector
-----------------------------------
'car':6 'day':3 'nice':2 'ride':7
(1 row)
fts=# SELECT plainto_tsquery('I am riding a bike');
plainto_tsquery
-----------------
'ride' & 'bike'
(1 row)
17. https://www.2ndQuadrant.com
FOSDEM
Brussels, 2020-02-02
An example table
●
pgsql-hackers mailing list archive subset
fts=# d mail_messages
Table "public.mail_messages"
Column | Type | Collation | Nullable |
------------+-----------------------------+-----------+----------+-------------
id | integer | | not null | nextval('mai
parent_id | integer | | |
sent | timestamp without time zone | | |
subject | text | | |
author | text | | |
body_plain | text | | |
fts=# dt+ mail_messages
List of relations
Schema | Name | Type | Owner | Size | Description
--------+---------------+-------+----------+--------+-------------
public | mail_messages | table | postgres | 478 MB |
18. https://www.2ndQuadrant.com
FOSDEM
Brussels, 2020-02-02
Ranking results
ts_rank (and Cover Density variant ts_rank_cd)
fts=# SELECT subject, ts_rank(to_tsvector(coalesce(body_plain,'')),
fts(# to_tsquery('aggregate'), 32) AS rank
fts-# FROM mail_messages ORDER BY rank DESC LIMIT 5;
subject | rank
--------------------------------------------------------------+-------------
Re: Window functions patch v04 for the September commit fest | 0.08969686
Re: Window functions patch v04 for the September commit fest | 0.08940695
Re: [HACKERS] PoC: Grouped base relation | 0.08936066
Re: [HACKERS] PoC: Grouped base relation | 0.08931142
Re: [PERFORM] not using index for select min(...) | 0.08925897
19. https://www.2ndQuadrant.com
FOSDEM
Brussels, 2020-02-02
FTS Stats
ts_stat for verifying your TS configuration, identifying stop words
fts=# SELECT * FROM ts_stat(
fts(# 'SELECT to_tsvector(body_plain)
fts'# FROM mail_messages')
fts-# ORDER BY nentry DESC, ndoc DESC, word
fts-# LIMIT 5;
word | ndoc | nentry
-------+--------+--------
use | 173833 | 380951
wrote | 231174 | 350905
would | 157169 | 316416
think | 149858 | 256661
patch | 100991 | 226099
20. https://www.2ndQuadrant.com
FOSDEM
Brussels, 2020-02-02
Text indexing
Normal default:
●
B-Tree
– with B-Tree text_pattern_ops for left, right anchored text
– CREATE INDEX name ON table (column varchar_pattern_ops);
For FTS we have:
●
GIN
– Inverted index: one entry per lexeme
– Larger, slower to update Better on less dynamic data→
– On tsvector columns
●
GiST
– Lossy index, smaller but slower (to eliminate false positives)
– Better on fewer unique items
– On tsvector or tsquery columns
22. https://www.2ndQuadrant.com
FOSDEM
Brussels, 2020-02-02
FTS indexing
CREATE INDEX ON mail_messages USING GIN
(to_tsvector('english',
subject ||' '|| body_plain));
●
New in PG12: Generated columns (stored):
ALTER TABLE mail_messages
ADD COLUMN fts_col tsvector
GENERATED ALWAYS AS (to_tsvector('english',
coalesce(subject, '') ||' '||
coalesce(body_plain, ''))) STORED;
CREATE INDEX ON mail_messages USING GIN (fts_col);
23. https://www.2ndQuadrant.com
FOSDEM
Brussels, 2020-02-02
FTS, GiST indexed
fts=# EXPLAIN ANALYZE SELECT count(*) FROM mail_messages
fts-# WHERE to_tsvector('english',body_plain) @@ to_tsquery('aggregate');
QUERY PLAN
-------------------------------------------------------------------------------
Aggregate (cost=7210.61..7210.62 rows=1 width=8) (actual time=5630.167..5630.
-> Bitmap Heap Scan on mail_messages (cost=330.46..7206.16 rows=1781 width
Recheck Cond: (to_tsvector('english'::regconfig, body_plain) @@ to_tsq
Rows Removed by Index Recheck: 4267
Heap Blocks: exact=7883
-> Bitmap Index Scan on mail_messages_to_tsvector_idx (cost=0.00..33
Index Cond: (to_tsvector('english'::regconfig, body_plain) @@ to
Planning Time: 0.620 ms
Execution Time: 5630.249 ms
●
26.99 seconds 5.63 seconds! ~4.8x faster→ →
24. https://www.2ndQuadrant.com
FOSDEM
Brussels, 2020-02-02
FTS, GIN indexed
fts=# EXPLAIN ANALYZE SELECT count(*) FROM mail_messages
fts-# WHERE to_tsvector('english',body_plain) @@ to_tsquery('aggregate');
QUERY PLAN
-------------------------------------------------------------------------------
Aggregate (cost=6873.60..6873.61 rows=1 width=8) (actual time=6.133..6.134 ro
-> Bitmap Heap Scan on mail_messages (cost=33.96..6869.18 rows=1769 width=
Recheck Cond: (to_tsvector('english'::regconfig, body_plain) @@ to_tsq
Heap Blocks: exact=4630
-> Bitmap Index Scan on mail_messages_to_tsvector_idx (cost=0.00..33
Index Cond: (to_tsvector('english'::regconfig, body_plain) @@ to
Planning Time: 0.433 ms
Execution Time: 5.684 ms
●
26.99 seconds 5.684→ milliseconds! → ~4700x faster
28. https://www.2ndQuadrant.com
FOSDEM
Brussels, 2020-02-02
Free text but not natural?
●
One use case: identifying arbitrary strings
– e.g. keywords in device logs
●
Dictionaries not very helpful here
●
Arbitrary example: 10M * ~100 char “IoT device” log entries
– Some contain strings that are significant to user
(but we don’t know these keywords)
– Populate table with random hex codes but 1% of log entries
contains a keyword from /etc/dictionaries-common/words:
c4f2cede5da57f0ace6e669b51186cbaexcruciating9635d8a26a
efb2b4ee8b9845e89718577b3266f68dffa5ae12ebfebf1a508b21
29. https://www.2ndQuadrant.com
FOSDEM
Brussels, 2020-02-02
Free text but not natural?
fts=# SELECT message FROM logentries LIMIT 5 OFFSET 495;
message
--------------------------------------------------------------------------------------------------
da40c1006cd75105c1eb8ea70705828d195b264565f047c6d449e51cf99d01e901cf532f03018e793a394fdac9bb5d2a
aa88a5c43ec8b2a8578d44f924053e842584c0e6b8295b72230f7d19aa3ba2f2b9e1a4bffcf0f82e4d29344645b714ca
fe9731c39108a74714cad9fc8570b115howlingb9904fa4ad86544fb778ef5edfe362e02a94c66851c3c8d7fe47b26e5
b68430decf30085cc2e7810585c5d681source2b638d61c5972f25aa3fa5c35aa2be282f04843cfca007689cc6ecdbe3
5b7ba17108e416d04788dc9ac15121fad7625fa7c216666bf54c1b0ca21ab618829262dfd67a5cd40aefd66235cf9c7f
(5 rows)
fts=# dt+ logentries
List of relations
Schema | Name | Type | Owner | Size | Description
--------+------------+-------+----------+---------+-------------
public | logentries | table | postgres | 1421 MB |
(1 row)
fts=# SELECT * FROM logentries WHERE message LIKE '%source%';
30. https://www.2ndQuadrant.com
FOSDEM
Brussels, 2020-02-02
How long?
fts=# EXPLAIN ANALYZE SELECT * FROM logentries WHERE message LIKE '%source%';
QUERY PLAN
---------------------------------------------------------------------------------------------------------
Gather (cost=1000.00..235029.95 rows=1000 width=109) (actual time=143.010..9654.769 rows=16 loops=1)
Workers Planned: 2
Workers Launched: 2
-> Parallel Seq Scan on logentries (cost=0.00..233929.95 rows=417 width=109) (actual time=1017.442..
Filter: (message ~~ '%source%'::text)
Rows Removed by Filter: 3333594
Planning Time: 0.220 ms
JIT:
Functions: 6
Options: Inlining false, Optimization false, Expressions true, Deforming true
Timing: Generation 18.918 ms, Inlining 0.000 ms, Optimization 41.736 ms, Emission 121.955 ms, Total 18
Execution Time: 9673.582 ms
(12 rows)
●
9.6 seconds!
31. https://www.2ndQuadrant.com
FOSDEM
Brussels, 2020-02-02
Trigrams
●
n-gram model: probabilistic language model (Markov Chains)
●
3 characters trigrams→
●
Similarity of alphanumeric text number of shared trigrams→
●
CREATE EXTENSION pg_trgm;
●
fts=# SELECT show_trgm('source');
show_trgm
-------------------------------------
{" s"," so","ce ",our,rce,sou,urc}
●
fts=# CREATE INDEX ON logentries
fts-# USING GIN (message gin_trgm_ops);
32. https://www.2ndQuadrant.com
FOSDEM
Brussels, 2020-02-02
Did trigrams help?
fts=# EXPLAIN ANALYZE SELECT * FROM logentries WHERE message LIKE '%source%';
QUERY PLAN
---------------------------------------------------------------------------------------------------------
Bitmap Heap Scan on logentries (cost=87.75..3870.45 rows=1000 width=109) (actual time=0.152..0.206 rows
Recheck Cond: (message ~~ '%source%'::text)
Rows Removed by Index Recheck: 2
Heap Blocks: exact=18
-> Bitmap Index Scan on logentries_message_idx (cost=0.00..87.50 rows=1000 width=0) (actual time=0.1
Index Cond: (message ~~ '%source%'::text)
Planning Time: 0.222 ms
Execution Time: 0.258 ms
(8 rows)
●
0.258 milliseconds! → ~37000x faster
●
Also work with regex
33. https://www.2ndQuadrant.com
FOSDEM
Brussels, 2020-02-02
This comes at a cost
fts=# di+ logentries_message_idx
List of relations
Schema | Name | Type | Owner | Table | Size | Description
--------+------------------------+-------+----------+------------+---------+-------------
public | logentries_message_idx | index | postgres | logentries | 1601 MB |
(1 row)
34. https://www.2ndQuadrant.com
FOSDEM
Brussels, 2020-02-02
Other neat trigram tricks
●
similarity(text, text) real→
●
text <-> text → Distance (1-similarity)
●
text % text true→ if over similarity_threshold
●
Supported by indexes:
– GIN
– GiST is efficient: k-nearest neighbour (k-NN)
36. https://www.2ndQuadrant.com
FOSDEM
Brussels, 2020-02-02
Collation in PostgreSQL
●
Sort order and character classification
– Per-column: CREATE TABLE test1 (a text
COLLATE "de_DE" …
– Per-operation: SELECT a < b COLLATE "de_DE"
FROM test1;
– Not restricted by DB LC_COLLATE, LC_CTYPE
●
New in PG12: Nondeterministic collations (case-
insensitive, ignore accents)
37. https://www.2ndQuadrant.com
FOSDEM
Brussels, 2020-02-02
Other types of documents JSON→
●
Also a real world use case
●
JSONB supports indexing
(article ->> 'title' ||''||
article ->> 'author')::tsvector
●
jsonb_to_tsvector()
SELECT jsonb_to_tsvector('english', column,
'["numeric","key","string","boolean"]') FROM table;
●
New in PG12: SQL/JSON (SQL:2016) jsonpath expressions→
●
JsQuery: JSONB query language with GIN support
– Equivalent to tsquery, JSON query as a single value
– https://github.com/postgrespro/jsquery
38. https://www.2ndQuadrant.com
FOSDEM
Brussels, 2020-02-02
Finally, maintenance
●
VACUUM ANALYZE
– Keep your table statistics up-to-date
– Pending GIN entries
●
ALTER TABLE SET STATISTICS
– Keep your table statistics accurate
●
Number of distinct values
●
Correlated columns
●
EXPLAIN ANALYZE from time to time
– Your query works now – but a year from now?
●
maintenance_work_mem