SlideShare a Scribd company logo
1 of 48
Download to read offline
Postgres vs
Elasticsearch while
enriching data.
Vlad Somov @ Salt Edge Inc.
Unstructured Data
Enrichment
Incoming raw data
Structured identified data
Keyword1 Keyword2 Website Name
Tag
Keyword1 Keyword2 Website Name
Tag
Unstructured Data
Enrichment
Some Transaction Description Website
Incoming raw data
Keyword1 Keyword2 Website
Structured identified data
Name
Tag
Description
Keyword1
Tag
Basic Setup Performance
Min
Average
Max
Seconds
0 7.5 15 22.5 30
Postgres Elasticsearch
~4mln. Records
Basic Setup Performance
Min
Average
Max
Seconds
0 7.5 15 22.5 30
Postgres Elasticsearch
28.73
9.88
2.10
~4mln. Records
Basic Setup Performance
Min
Average
Max
Seconds
0 7.5 15 22.5 30
Postgres Elasticsearch
1.37
0.99
0.73
28.73
9.88
2.10
~4mln. Records
B-tree index structure
3 39 68
meta
39 42 55 68 89 943 15 28
3 9 15 21 29 32 39 42 42 48 55 68 68 77 89 93 94 98
39
B-tree index structure
3 68
meta
39 42 55 68 89 943 15 28
3 9 15 21 29 32 39 42 42 48 55 68 68 77 89 93 94 98
39
39
B-tree index structure
3 68
meta
42 55 68 89 943 15 28
3 9 15 21 29 32 39 42 42 48 55 68 68 77 89 93 94 98
39
39
B-tree index structure
3 68
meta
42 55 68 89 943 15 28
3 9 15 21 29 32 39 42 42 48 55 68 68 77 89 93 94 98
39
39
B-tree index structure
3 68
meta
42 55 68 89 943 15 28
3 9 15 21 29 32 39 42 42 48 55 68 68 77 89 93 94 98
39
39
B-tree index structure
3 68
meta
42 55 68 89 943 15 28
3 9 15 21 29 32 39 42 42 48 55 68 68 77 89 93 94 98
Why it is useful?
• b-tree index sort values inside each node.

• b-tree is balanced

• Same level nodes are connected using doubly linked list.
After multicolumn index on country_id
and merchant_type Performance
Min
Average
Max
Seconds
0 7.5 15 22.5 30
Postgres Elasticsearch
Postgres + multicolumn index
~4mln. Records
After multicolumn index on country_id
and merchant_type Performance
Min
Average
Max
Seconds
0 7.5 15 22.5 30
Postgres Elasticsearch
Postgres + multicolumn index
28.73
9.88
2.1
~4mln. Records
After multicolumn index on country_id
and merchant_type Performance
Min
Average
Max
Seconds
0 7.5 15 22.5 30
Postgres Elasticsearch
Postgres + multicolumn index
1.37
0.99
0.73
28.73
9.88
2.1
~4mln. Records
After multicolumn index on country_id
and merchant_type Performance
Min
Average
Max
Seconds
0 7.5 15 22.5 30
Postgres Elasticsearch
Postgres + multicolumn index
10.19
5.09
2.28
1.37
0.99
0.73
28.73
9.88
2.1
~4mln. Records
What is GiST
Generalized Search
Tree
• In GiST each leaf contains
logical expression and
pointer to TID, where
indexed data should
satisfy logical expression.

• Faster on insert, update
What is GIN
Generalized Inverted
Index
• It is b-tree with elements to
which is connected another
b-tree or plain list of TID's. 

• Faster and more accurate
on select.
Welcome to ruby meditation.

All of us love ruby.
Does everyone love meditation?
Everyone Of Welcome
All Does WelcomeRuby ToOfLove MeditationEveryone
0,1 0,10,1
2,1
1,51,5 2,1 2,1
Yellow rectangle are TID’s. First number is a page number and second is
position on a page
0,1
1,52,1
1,5
Welcome to ruby meditation.

All of us love ruby.
Does everyone love meditation?
Everyone Of Welcome
All Does WelcomeRuby ToOfLove MeditationEveryone
0,1
1,5
0,1 0,10,1
2,1
1,51,5 2,1 2,1
Yellow rectangle are TID’s. First number is a page number and second is
position on a page
2,1
1,5
ruby
rubylove
love
1,5
1,5
Welcome to ruby meditation.

All of us love ruby.
Does everyone love meditation?
Everyone Of Welcome
All Does WelcomeRuby ToOfLove MeditationEveryone
0,1 0,1 0,10,1
2,1
1,51,5 2,1 2,1
Yellow rectangle are TID’s. First number is a page number and second is
position on a page
2,1
ruby
rubylove
love
love ruby
gin_trgm_ops
A trigram is a group of three consecutive characters
taken from a string.
We can measure the similarity of two strings by counting the
number of trigrams they share.
Performance after gin index on websites
Min
Average
Max
Seconds
0 5 10 15 20
Postgres
Elasticsearch
Postgres + multicolumn index
Postgres + gin index with trgm_ops on websites
~4mln. Records
Performance after gin index on websites
Min
Average
Max
Seconds
0 5 10 15 20
Postgres
Elasticsearch
Postgres + multicolumn index
Postgres + gin index with trgm_ops on websites
28.73
9.88
2.1
~4mln. Records
Performance after gin index on websites
Min
Average
Max
Seconds
0 5 10 15 20
Postgres
Elasticsearch
Postgres + multicolumn index
Postgres + gin index with trgm_ops on websites
1.37
0.99
0.75
28.73
9.88
2.1
~4mln. Records
Performance after gin index on websites
Min
Average
Max
Seconds
0 5 10 15 20
Postgres
Elasticsearch
Postgres + multicolumn index
Postgres + gin index with trgm_ops on websites
10.19
5.09
2.28
1.37
0.99
0.75
28.73
9.88
2.1
~4mln. Records
Performance after gin index on websites
Min
Average
Max
Seconds
0 5 10 15 20
Postgres
Elasticsearch
Postgres + multicolumn index
Postgres + gin index with trgm_ops on websites
0.55
0.34
0.26
10.19
5.09
2.28
1.37
0.99
0.75
28.73
9.88
2.1
~4mln. Records
How elasticsearch works
• It uses analyzers for all incoming data. (it could be custom
or default one)

• Each analyzer has at least one tokenizer

• Zero or more TokenFilters

• Tokenizer may be preceded by one or more CharFilters
How analyzer works?
How analyzer works?
Input
How analyzer works?
Input Char Filter
String
How analyzer works?
Input Char Filter Tokenizer
String String
How analyzer works?
Input Char Filter Tokenizer
Token
Filter
String String Tokens
How analyzer works?
Input Char Filter Tokenizer
Token
Filter
Output
String String Tokens Tokens
Example
Example
The 2 QUICK <p>Brown-Foxes</p> jumped over the lazy dog's bone.
Example
The 2 QUICK <p>Brown-Foxes</p> jumped over the lazy dog's bone.
html_strip
The 2 QUICK Brown-Foxes jumped over the lazy dog's bone.
Example
The 2 QUICK <p>Brown-Foxes</p> jumped over the lazy dog's bone.
html_strip
The 2 QUICK Brown-Foxes jumped over the lazy dog's bone.
standart tokenizer
The 2 QUICK Brown jumpedFoxes over
the lazy dog’s bone
Example
The 2 QUICK <p>Brown-Foxes</p> jumped over the lazy dog's bone.
html_strip
The 2 QUICK Brown-Foxes jumped over the lazy dog's bone.
standart tokenizer
The 2 QUICK Brown jumpedFoxes over
the lazy dog’s bone
lowercase
the 2 quick brown jumpedfoxes over
the lazy dog’s bone
Example
The 2 QUICK <p>Brown-Foxes</p> jumped over the lazy dog's bone.
html_strip
The 2 QUICK Brown-Foxes jumped over the lazy dog's bone.
standart tokenizer
The 2 QUICK Brown jumpedFoxes over
the lazy dog’s bone
lowercase
the 2 quick brown jumpedfoxes over
the lazy dog’s bone
stop
2 quick brown jumpedfoxes over lazy dog’s bone
the
the
Example
The 2 QUICK <p>Brown-Foxes</p> jumped over the lazy dog's bone.
html_strip
The 2 QUICK Brown-Foxes jumped over the lazy dog's bone.
standart tokenizer
The 2 QUICK Brown jumpedFoxes over
the lazy dog’s bone
lowercase
the 2 quick brown jumpedfoxes over
the lazy dog’s bone
stop
2 quick brown jumpedfoxes over lazy dog’s bone
snowball
2 quick brown jumpfox over lazi dog bone
the
the
jump lazi dog
Postgres full search
implementation
• We can use tsvector type to achieve almost the same
functionality. By using to_tsvector function

• To imporve perfomance we could create separate tsvector
column with to_tsvector values.

• To create a request we should use to_tsquery. & | <->

• plainto_tsquery works with plain text so you don’t need to
insert any special symbols. Inserts &

• phraseto_tsquery also works with plain text but marks that
each token should be close to each other. Inserts <->
Rum access method
• Based on GIN access method code

• Solves slow ranking

• Solves slow phrase search (tsquery with <-> operator)

• Supports index on tsquery column
122
1
5
3
2
4
4
3
3
4211
Welcome to ruby meditation.

All of us love ruby.
Does everyone love meditation?
ruby, meditation, love
Everyone Of Welcome
All Does WelcomeRuby ToOfLove MeditationEveryone
0,1 0,10,1
2,1
1,51,5 2,1 2,1
The number in green rectangle is word position in the document.
0,1
1,52,1
1,5
8,4 8,4 8,4
122
1
5
3
2
4
4
3
3
4211
Welcome to ruby meditation.

All of us love ruby.
Does everyone love meditation?
ruby, meditation, love
Everyone Of Welcome
All Does WelcomeRuby ToOfLove MeditationEveryone
0,1
1,5
0,1 0,10,1
2,1
1,51,5 2,1 2,1
The number in green rectangle is word position in the document.
2,1
1,5
ruby
rubylove
love
love
ruby
8,4 8,4 8,4
122
1
5
3
2
4
4
3
3
4211
1,5
1,5
Welcome to ruby meditation.

All of us love ruby.
Does everyone love meditation?
ruby, meditation, love
Everyone Of Welcome
All Does WelcomeRuby ToOfLove MeditationEveryone
0,1 0,1 0,10,1
2,1
1,51,5 2,1 2,1
The number in green rectangle is word position in the document.
2,1
ruby
rubylove
love
love ruby
love
ruby
8,4 8,4 8,4
Conclusion
• Postgres can also be fast.

• Multicolumn indexes can improve performance if your search has multicolumn
constraints.

• For fast text search prefer using Gin when table doesn’t update occasionally,
otherwise use GiST

• Use gin with trgm_ops when using full text search. If full text search is still slow
try to use tsvector data type with gin index on it.

• When you have some kind ‘inverse full-text search’ problem. Add tsquery type in
your table as a query and incoming data treat as a document. Add rum access
method on query column with tsquery_ops for fast classification.

• Before moving to other instrument make analysis of current/new instrument and
verify is it worth moving or not.
email: vlad.somov@icloud.com
twitter: @vsomov93
Questions?

More Related Content

What's hot

mongodb와 mysql의 CRUD 연산의 성능 비교
mongodb와 mysql의 CRUD 연산의 성능 비교mongodb와 mysql의 CRUD 연산의 성능 비교
mongodb와 mysql의 CRUD 연산의 성능 비교
Woo Yeong Choi
 
Flink Forward San Francisco 2019: Massive Scale Data Processing at Netflix us...
Flink Forward San Francisco 2019: Massive Scale Data Processing at Netflix us...Flink Forward San Francisco 2019: Massive Scale Data Processing at Netflix us...
Flink Forward San Francisco 2019: Massive Scale Data Processing at Netflix us...
Flink Forward
 

What's hot (20)

mongodb와 mysql의 CRUD 연산의 성능 비교
mongodb와 mysql의 CRUD 연산의 성능 비교mongodb와 mysql의 CRUD 연산의 성능 비교
mongodb와 mysql의 CRUD 연산의 성능 비교
 
ELK Stack
ELK StackELK Stack
ELK Stack
 
Deep Dive into Stateful Stream Processing in Structured Streaming with Tathag...
Deep Dive into Stateful Stream Processing in Structured Streaming with Tathag...Deep Dive into Stateful Stream Processing in Structured Streaming with Tathag...
Deep Dive into Stateful Stream Processing in Structured Streaming with Tathag...
 
Elk - An introduction
Elk - An introductionElk - An introduction
Elk - An introduction
 
4. 대용량 아키텍쳐 설계 패턴
4. 대용량 아키텍쳐 설계 패턴4. 대용량 아키텍쳐 설계 패턴
4. 대용량 아키텍쳐 설계 패턴
 
Elastic stack Presentation
Elastic stack PresentationElastic stack Presentation
Elastic stack Presentation
 
Log analytics with ELK stack
Log analytics with ELK stackLog analytics with ELK stack
Log analytics with ELK stack
 
Grafana
GrafanaGrafana
Grafana
 
Scylla Summit 2022: How to Migrate a Counter Table for 68 Billion Records
Scylla Summit 2022: How to Migrate a Counter Table for 68 Billion RecordsScylla Summit 2022: How to Migrate a Counter Table for 68 Billion Records
Scylla Summit 2022: How to Migrate a Counter Table for 68 Billion Records
 
Kafka Retry and DLQ
Kafka Retry and DLQKafka Retry and DLQ
Kafka Retry and DLQ
 
Naver속도의, 속도에 의한, 속도를 위한 몽고DB (네이버 컨텐츠검색과 몽고DB) [Naver]
Naver속도의, 속도에 의한, 속도를 위한 몽고DB (네이버 컨텐츠검색과 몽고DB) [Naver]Naver속도의, 속도에 의한, 속도를 위한 몽고DB (네이버 컨텐츠검색과 몽고DB) [Naver]
Naver속도의, 속도에 의한, 속도를 위한 몽고DB (네이버 컨텐츠검색과 몽고DB) [Naver]
 
검색엔진이 데이터를 다루는 법 김종민
검색엔진이 데이터를 다루는 법 김종민검색엔진이 데이터를 다루는 법 김종민
검색엔진이 데이터를 다루는 법 김종민
 
Trino at linkedIn - 2021
Trino at linkedIn - 2021Trino at linkedIn - 2021
Trino at linkedIn - 2021
 
Flink Forward San Francisco 2019: Massive Scale Data Processing at Netflix us...
Flink Forward San Francisco 2019: Massive Scale Data Processing at Netflix us...Flink Forward San Francisco 2019: Massive Scale Data Processing at Netflix us...
Flink Forward San Francisco 2019: Massive Scale Data Processing at Netflix us...
 
PostgreSQL at 20TB and Beyond
PostgreSQL at 20TB and BeyondPostgreSQL at 20TB and Beyond
PostgreSQL at 20TB and Beyond
 
Data pipeline and data lake
Data pipeline and data lakeData pipeline and data lake
Data pipeline and data lake
 
Introduction of pg_statsinfo and pg_stats_reporter ~Statistics Reporting Tool...
Introduction of pg_statsinfo and pg_stats_reporter ~Statistics Reporting Tool...Introduction of pg_statsinfo and pg_stats_reporter ~Statistics Reporting Tool...
Introduction of pg_statsinfo and pg_stats_reporter ~Statistics Reporting Tool...
 
Beautiful Monitoring With Grafana and InfluxDB
Beautiful Monitoring With Grafana and InfluxDBBeautiful Monitoring With Grafana and InfluxDB
Beautiful Monitoring With Grafana and InfluxDB
 
Opentracing jaeger
Opentracing jaegerOpentracing jaeger
Opentracing jaeger
 
Message Redelivery: An Unexpected Journey - Pulsar Summit SF 2022
Message Redelivery: An Unexpected Journey - Pulsar Summit SF 2022Message Redelivery: An Unexpected Journey - Pulsar Summit SF 2022
Message Redelivery: An Unexpected Journey - Pulsar Summit SF 2022
 

Similar to Postgres vs Elasticsearch while enriching data - Vlad Somov | Ruby Meditaiton #23

Basics in algorithms and data structure
Basics in algorithms and data structure Basics in algorithms and data structure
Basics in algorithms and data structure
Eman magdy
 
ZFConf 2011: Что такое Sphinx, зачем он вообще нужен и как его использовать с...
ZFConf 2011: Что такое Sphinx, зачем он вообще нужен и как его использовать с...ZFConf 2011: Что такое Sphinx, зачем он вообще нужен и как его использовать с...
ZFConf 2011: Что такое Sphinx, зачем он вообще нужен и как его использовать с...
ZFConf Conference
 
The Challenges of Distributing Postgres: A Citus Story
The Challenges of Distributing Postgres: A Citus StoryThe Challenges of Distributing Postgres: A Citus Story
The Challenges of Distributing Postgres: A Citus Story
Hanna Kelman
 
The Challenges of Distributing Postgres: A Citus Story | DataEngConf NYC 2017...
The Challenges of Distributing Postgres: A Citus Story | DataEngConf NYC 2017...The Challenges of Distributing Postgres: A Citus Story | DataEngConf NYC 2017...
The Challenges of Distributing Postgres: A Citus Story | DataEngConf NYC 2017...
Citus Data
 
Performance Optimization of Rails Applications
Performance Optimization of Rails ApplicationsPerformance Optimization of Rails Applications
Performance Optimization of Rails Applications
Serge Smetana
 
London devops logging
London devops loggingLondon devops logging
London devops logging
Tomas Doran
 

Similar to Postgres vs Elasticsearch while enriching data - Vlad Somov | Ruby Meditaiton #23 (20)

Basics in algorithms and data structure
Basics in algorithms and data structure Basics in algorithms and data structure
Basics in algorithms and data structure
 
ZFConf 2011: Что такое Sphinx, зачем он вообще нужен и как его использовать с...
ZFConf 2011: Что такое Sphinx, зачем он вообще нужен и как его использовать с...ZFConf 2011: Что такое Sphinx, зачем он вообще нужен и как его использовать с...
ZFConf 2011: Что такое Sphinx, зачем он вообще нужен и как его использовать с...
 
Beyond php - it's not (just) about the code
Beyond php - it's not (just) about the codeBeyond php - it's not (just) about the code
Beyond php - it's not (just) about the code
 
About elasticsearch
About elasticsearchAbout elasticsearch
About elasticsearch
 
Beyond php - it's not (just) about the code
Beyond php - it's not (just) about the codeBeyond php - it's not (just) about the code
Beyond php - it's not (just) about the code
 
Unveiling etcd: Architecture and Source Code Deep Dive
Unveiling etcd: Architecture and Source Code Deep DiveUnveiling etcd: Architecture and Source Code Deep Dive
Unveiling etcd: Architecture and Source Code Deep Dive
 
The Challenges of Distributing Postgres: A Citus Story
The Challenges of Distributing Postgres: A Citus StoryThe Challenges of Distributing Postgres: A Citus Story
The Challenges of Distributing Postgres: A Citus Story
 
The Challenges of Distributing Postgres: A Citus Story | DataEngConf NYC 2017...
The Challenges of Distributing Postgres: A Citus Story | DataEngConf NYC 2017...The Challenges of Distributing Postgres: A Citus Story | DataEngConf NYC 2017...
The Challenges of Distributing Postgres: A Citus Story | DataEngConf NYC 2017...
 
Beyond php - it's not (just) about the code
Beyond php - it's not (just) about the codeBeyond php - it's not (just) about the code
Beyond php - it's not (just) about the code
 
Better Full Text Search in PostgreSQL
Better Full Text Search in PostgreSQLBetter Full Text Search in PostgreSQL
Better Full Text Search in PostgreSQL
 
SequoiaDB Distributed Relational Database
SequoiaDB Distributed Relational DatabaseSequoiaDB Distributed Relational Database
SequoiaDB Distributed Relational Database
 
Beyond PHP - it's not (just) about the code
Beyond PHP - it's not (just) about the codeBeyond PHP - it's not (just) about the code
Beyond PHP - it's not (just) about the code
 
Slash n near real time indexing
Slash n   near real time indexingSlash n   near real time indexing
Slash n near real time indexing
 
Elasticsearch at Dailymotion
Elasticsearch at DailymotionElasticsearch at Dailymotion
Elasticsearch at Dailymotion
 
SDPHP - Percona Toolkit (It's Basically Magic)
SDPHP - Percona Toolkit (It's Basically Magic)SDPHP - Percona Toolkit (It's Basically Magic)
SDPHP - Percona Toolkit (It's Basically Magic)
 
A Call for Sanity in NoSQL
A Call for Sanity in NoSQLA Call for Sanity in NoSQL
A Call for Sanity in NoSQL
 
Performance Optimization of Rails Applications
Performance Optimization of Rails ApplicationsPerformance Optimization of Rails Applications
Performance Optimization of Rails Applications
 
.NET Fest 2019. Łukasz Pyrzyk. Daily Performance Fuckups
.NET Fest 2019. Łukasz Pyrzyk. Daily Performance Fuckups.NET Fest 2019. Łukasz Pyrzyk. Daily Performance Fuckups
.NET Fest 2019. Łukasz Pyrzyk. Daily Performance Fuckups
 
Moving Toward Deep Learning Algorithms on HPCC Systems
Moving Toward Deep Learning Algorithms on HPCC SystemsMoving Toward Deep Learning Algorithms on HPCC Systems
Moving Toward Deep Learning Algorithms on HPCC Systems
 
London devops logging
London devops loggingLondon devops logging
London devops logging
 

More from Ruby Meditation

More from Ruby Meditation (20)

Is this Legacy or Revenant Code? - Sergey Sergyenko | Ruby Meditation 30
Is this Legacy or Revenant Code? - Sergey Sergyenko  | Ruby Meditation 30Is this Legacy or Revenant Code? - Sergey Sergyenko  | Ruby Meditation 30
Is this Legacy or Revenant Code? - Sergey Sergyenko | Ruby Meditation 30
 
Life with GraphQL API: good practices and unresolved issues - Roman Dubrovsky...
Life with GraphQL API: good practices and unresolved issues - Roman Dubrovsky...Life with GraphQL API: good practices and unresolved issues - Roman Dubrovsky...
Life with GraphQL API: good practices and unresolved issues - Roman Dubrovsky...
 
Where is your license, dude? - Viacheslav Miroshnychenko | Ruby Meditation 29
Where is your license, dude? - Viacheslav Miroshnychenko | Ruby Meditation 29Where is your license, dude? - Viacheslav Miroshnychenko | Ruby Meditation 29
Where is your license, dude? - Viacheslav Miroshnychenko | Ruby Meditation 29
 
Dry-validation update. Dry-validation vs Dry-schema 1.0 - Aleksandra Stolyar ...
Dry-validation update. Dry-validation vs Dry-schema 1.0 - Aleksandra Stolyar ...Dry-validation update. Dry-validation vs Dry-schema 1.0 - Aleksandra Stolyar ...
Dry-validation update. Dry-validation vs Dry-schema 1.0 - Aleksandra Stolyar ...
 
How to cook Rabbit on Production - Bohdan Parshentsev | Ruby Meditation 28
How to cook Rabbit on Production - Bohdan Parshentsev | Ruby Meditation 28 How to cook Rabbit on Production - Bohdan Parshentsev | Ruby Meditation 28
How to cook Rabbit on Production - Bohdan Parshentsev | Ruby Meditation 28
 
How to cook Rabbit on Production - Serhiy Nazarov | Ruby Meditation 28
How to cook Rabbit on Production - Serhiy Nazarov | Ruby Meditation 28How to cook Rabbit on Production - Serhiy Nazarov | Ruby Meditation 28
How to cook Rabbit on Production - Serhiy Nazarov | Ruby Meditation 28
 
Reinventing the wheel - why do it and how to feel good about it - Julik Tarkh...
Reinventing the wheel - why do it and how to feel good about it - Julik Tarkh...Reinventing the wheel - why do it and how to feel good about it - Julik Tarkh...
Reinventing the wheel - why do it and how to feel good about it - Julik Tarkh...
 
Performance Optimization 101 for Ruby developers - Nihad Abbasov (ENG) | Ruby...
Performance Optimization 101 for Ruby developers - Nihad Abbasov (ENG) | Ruby...Performance Optimization 101 for Ruby developers - Nihad Abbasov (ENG) | Ruby...
Performance Optimization 101 for Ruby developers - Nihad Abbasov (ENG) | Ruby...
 
Use cases for Serverless Technologies - Ruslan Tolstov (RUS) | Ruby Meditatio...
Use cases for Serverless Technologies - Ruslan Tolstov (RUS) | Ruby Meditatio...Use cases for Serverless Technologies - Ruslan Tolstov (RUS) | Ruby Meditatio...
Use cases for Serverless Technologies - Ruslan Tolstov (RUS) | Ruby Meditatio...
 
The Trailblazer Ride from the If Jungle into a Civilised Railway Station - Or...
The Trailblazer Ride from the If Jungle into a Civilised Railway Station - Or...The Trailblazer Ride from the If Jungle into a Civilised Railway Station - Or...
The Trailblazer Ride from the If Jungle into a Civilised Railway Station - Or...
 
What/How to do with GraphQL? - Valentyn Ostakh (ENG) | Ruby Meditation 27
What/How to do with GraphQL? - Valentyn Ostakh (ENG) | Ruby Meditation 27What/How to do with GraphQL? - Valentyn Ostakh (ENG) | Ruby Meditation 27
What/How to do with GraphQL? - Valentyn Ostakh (ENG) | Ruby Meditation 27
 
New features in Rails 6 - Nihad Abbasov (RUS) | Ruby Meditation 26
New features in Rails 6 -  Nihad Abbasov (RUS) | Ruby Meditation 26New features in Rails 6 -  Nihad Abbasov (RUS) | Ruby Meditation 26
New features in Rails 6 - Nihad Abbasov (RUS) | Ruby Meditation 26
 
Security Scanning Overview - Tetiana Chupryna (RUS) | Ruby Meditation 26
Security Scanning Overview - Tetiana Chupryna (RUS) | Ruby Meditation 26Security Scanning Overview - Tetiana Chupryna (RUS) | Ruby Meditation 26
Security Scanning Overview - Tetiana Chupryna (RUS) | Ruby Meditation 26
 
Teach your application eloquence. Logs, metrics, traces - Dmytro Shapovalov (...
Teach your application eloquence. Logs, metrics, traces - Dmytro Shapovalov (...Teach your application eloquence. Logs, metrics, traces - Dmytro Shapovalov (...
Teach your application eloquence. Logs, metrics, traces - Dmytro Shapovalov (...
 
Best practices. Exploring - Ike Kurghinyan (RUS) | Ruby Meditation 26
Best practices. Exploring - Ike Kurghinyan (RUS) | Ruby Meditation 26Best practices. Exploring - Ike Kurghinyan (RUS) | Ruby Meditation 26
Best practices. Exploring - Ike Kurghinyan (RUS) | Ruby Meditation 26
 
Road to A/B testing - Alexey Vasiliev (ENG) | Ruby Meditation 25
Road to A/B testing - Alexey Vasiliev (ENG) | Ruby Meditation 25Road to A/B testing - Alexey Vasiliev (ENG) | Ruby Meditation 25
Road to A/B testing - Alexey Vasiliev (ENG) | Ruby Meditation 25
 
Concurrency in production. Real life example - Dmytro Herasymuk | Ruby Medita...
Concurrency in production. Real life example - Dmytro Herasymuk | Ruby Medita...Concurrency in production. Real life example - Dmytro Herasymuk | Ruby Medita...
Concurrency in production. Real life example - Dmytro Herasymuk | Ruby Medita...
 
Data encryption for Ruby web applications - Dmytro Shapovalov (RUS) | Ruby Me...
Data encryption for Ruby web applications - Dmytro Shapovalov (RUS) | Ruby Me...Data encryption for Ruby web applications - Dmytro Shapovalov (RUS) | Ruby Me...
Data encryption for Ruby web applications - Dmytro Shapovalov (RUS) | Ruby Me...
 
Rails App performance at the limit - Bogdan Gusiev
Rails App performance at the limit - Bogdan GusievRails App performance at the limit - Bogdan Gusiev
Rails App performance at the limit - Bogdan Gusiev
 
GDPR. Next Y2K in 2018? - Anton Tkachov | Ruby Meditation #23
GDPR. Next Y2K in 2018? - Anton Tkachov | Ruby Meditation #23GDPR. Next Y2K in 2018? - Anton Tkachov | Ruby Meditation #23
GDPR. Next Y2K in 2018? - Anton Tkachov | Ruby Meditation #23
 

Recently uploaded

Future Visions: Predictions to Guide and Time Tech Innovation, Peter Udo Diehl
Future Visions: Predictions to Guide and Time Tech Innovation, Peter Udo DiehlFuture Visions: Predictions to Guide and Time Tech Innovation, Peter Udo Diehl
Future Visions: Predictions to Guide and Time Tech Innovation, Peter Udo Diehl
Peter Udo Diehl
 

Recently uploaded (20)

THE BEST IPTV in GERMANY for 2024: IPTVreel
THE BEST IPTV in  GERMANY for 2024: IPTVreelTHE BEST IPTV in  GERMANY for 2024: IPTVreel
THE BEST IPTV in GERMANY for 2024: IPTVreel
 
FDO for Camera, Sensor and Networking Device – Commercial Solutions from VinC...
FDO for Camera, Sensor and Networking Device – Commercial Solutions from VinC...FDO for Camera, Sensor and Networking Device – Commercial Solutions from VinC...
FDO for Camera, Sensor and Networking Device – Commercial Solutions from VinC...
 
Secure Zero Touch enabled Edge compute with Dell NativeEdge via FDO _ Brad at...
Secure Zero Touch enabled Edge compute with Dell NativeEdge via FDO _ Brad at...Secure Zero Touch enabled Edge compute with Dell NativeEdge via FDO _ Brad at...
Secure Zero Touch enabled Edge compute with Dell NativeEdge via FDO _ Brad at...
 
Free and Effective: Making Flows Publicly Accessible, Yumi Ibrahimzade
Free and Effective: Making Flows Publicly Accessible, Yumi IbrahimzadeFree and Effective: Making Flows Publicly Accessible, Yumi Ibrahimzade
Free and Effective: Making Flows Publicly Accessible, Yumi Ibrahimzade
 
How Red Hat Uses FDO in Device Lifecycle _ Costin and Vitaliy at Red Hat.pdf
How Red Hat Uses FDO in Device Lifecycle _ Costin and Vitaliy at Red Hat.pdfHow Red Hat Uses FDO in Device Lifecycle _ Costin and Vitaliy at Red Hat.pdf
How Red Hat Uses FDO in Device Lifecycle _ Costin and Vitaliy at Red Hat.pdf
 
TEST BANK For, Information Technology Project Management 9th Edition Kathy Sc...
TEST BANK For, Information Technology Project Management 9th Edition Kathy Sc...TEST BANK For, Information Technology Project Management 9th Edition Kathy Sc...
TEST BANK For, Information Technology Project Management 9th Edition Kathy Sc...
 
Demystifying gRPC in .Net by John Staveley
Demystifying gRPC in .Net by John StaveleyDemystifying gRPC in .Net by John Staveley
Demystifying gRPC in .Net by John Staveley
 
Enterprise Knowledge Graphs - Data Summit 2024
Enterprise Knowledge Graphs - Data Summit 2024Enterprise Knowledge Graphs - Data Summit 2024
Enterprise Knowledge Graphs - Data Summit 2024
 
IESVE for Early Stage Design and Planning
IESVE for Early Stage Design and PlanningIESVE for Early Stage Design and Planning
IESVE for Early Stage Design and Planning
 
SOQL 201 for Admins & Developers: Slice & Dice Your Org’s Data With Aggregate...
SOQL 201 for Admins & Developers: Slice & Dice Your Org’s Data With Aggregate...SOQL 201 for Admins & Developers: Slice & Dice Your Org’s Data With Aggregate...
SOQL 201 for Admins & Developers: Slice & Dice Your Org’s Data With Aggregate...
 
Future Visions: Predictions to Guide and Time Tech Innovation, Peter Udo Diehl
Future Visions: Predictions to Guide and Time Tech Innovation, Peter Udo DiehlFuture Visions: Predictions to Guide and Time Tech Innovation, Peter Udo Diehl
Future Visions: Predictions to Guide and Time Tech Innovation, Peter Udo Diehl
 
Salesforce Adoption – Metrics, Methods, and Motivation, Antone Kom
Salesforce Adoption – Metrics, Methods, and Motivation, Antone KomSalesforce Adoption – Metrics, Methods, and Motivation, Antone Kom
Salesforce Adoption – Metrics, Methods, and Motivation, Antone Kom
 
ECS 2024 Teams Premium - Pretty Secure
ECS 2024   Teams Premium - Pretty SecureECS 2024   Teams Premium - Pretty Secure
ECS 2024 Teams Premium - Pretty Secure
 
What's New in Teams Calling, Meetings and Devices April 2024
What's New in Teams Calling, Meetings and Devices April 2024What's New in Teams Calling, Meetings and Devices April 2024
What's New in Teams Calling, Meetings and Devices April 2024
 
Linux Foundation Edge _ Overview of FDO Software Components _ Randy at Intel.pdf
Linux Foundation Edge _ Overview of FDO Software Components _ Randy at Intel.pdfLinux Foundation Edge _ Overview of FDO Software Components _ Randy at Intel.pdf
Linux Foundation Edge _ Overview of FDO Software Components _ Randy at Intel.pdf
 
Extensible Python: Robustness through Addition - PyCon 2024
Extensible Python: Robustness through Addition - PyCon 2024Extensible Python: Robustness through Addition - PyCon 2024
Extensible Python: Robustness through Addition - PyCon 2024
 
Optimizing NoSQL Performance Through Observability
Optimizing NoSQL Performance Through ObservabilityOptimizing NoSQL Performance Through Observability
Optimizing NoSQL Performance Through Observability
 
The UX of Automation by AJ King, Senior UX Researcher, Ocado
The UX of Automation by AJ King, Senior UX Researcher, OcadoThe UX of Automation by AJ King, Senior UX Researcher, Ocado
The UX of Automation by AJ King, Senior UX Researcher, Ocado
 
AI presentation and introduction - Retrieval Augmented Generation RAG 101
AI presentation and introduction - Retrieval Augmented Generation RAG 101AI presentation and introduction - Retrieval Augmented Generation RAG 101
AI presentation and introduction - Retrieval Augmented Generation RAG 101
 
Simplified FDO Manufacturing Flow with TPMs _ Liam at Infineon.pdf
Simplified FDO Manufacturing Flow with TPMs _ Liam at Infineon.pdfSimplified FDO Manufacturing Flow with TPMs _ Liam at Infineon.pdf
Simplified FDO Manufacturing Flow with TPMs _ Liam at Infineon.pdf
 

Postgres vs Elasticsearch while enriching data - Vlad Somov | Ruby Meditaiton #23

  • 1. Postgres vs Elasticsearch while enriching data. Vlad Somov @ Salt Edge Inc.
  • 2. Unstructured Data Enrichment Incoming raw data Structured identified data
  • 3. Keyword1 Keyword2 Website Name Tag Keyword1 Keyword2 Website Name Tag Unstructured Data Enrichment Some Transaction Description Website Incoming raw data Keyword1 Keyword2 Website Structured identified data Name Tag Description Keyword1 Tag
  • 4. Basic Setup Performance Min Average Max Seconds 0 7.5 15 22.5 30 Postgres Elasticsearch ~4mln. Records
  • 5. Basic Setup Performance Min Average Max Seconds 0 7.5 15 22.5 30 Postgres Elasticsearch 28.73 9.88 2.10 ~4mln. Records
  • 6. Basic Setup Performance Min Average Max Seconds 0 7.5 15 22.5 30 Postgres Elasticsearch 1.37 0.99 0.73 28.73 9.88 2.10 ~4mln. Records
  • 7. B-tree index structure 3 39 68 meta 39 42 55 68 89 943 15 28 3 9 15 21 29 32 39 42 42 48 55 68 68 77 89 93 94 98
  • 8. 39 B-tree index structure 3 68 meta 39 42 55 68 89 943 15 28 3 9 15 21 29 32 39 42 42 48 55 68 68 77 89 93 94 98
  • 9. 39 39 B-tree index structure 3 68 meta 42 55 68 89 943 15 28 3 9 15 21 29 32 39 42 42 48 55 68 68 77 89 93 94 98
  • 10. 39 39 B-tree index structure 3 68 meta 42 55 68 89 943 15 28 3 9 15 21 29 32 39 42 42 48 55 68 68 77 89 93 94 98
  • 11. 39 39 B-tree index structure 3 68 meta 42 55 68 89 943 15 28 3 9 15 21 29 32 39 42 42 48 55 68 68 77 89 93 94 98
  • 12. 39 39 B-tree index structure 3 68 meta 42 55 68 89 943 15 28 3 9 15 21 29 32 39 42 42 48 55 68 68 77 89 93 94 98
  • 13. Why it is useful? • b-tree index sort values inside each node. • b-tree is balanced • Same level nodes are connected using doubly linked list.
  • 14. After multicolumn index on country_id and merchant_type Performance Min Average Max Seconds 0 7.5 15 22.5 30 Postgres Elasticsearch Postgres + multicolumn index ~4mln. Records
  • 15. After multicolumn index on country_id and merchant_type Performance Min Average Max Seconds 0 7.5 15 22.5 30 Postgres Elasticsearch Postgres + multicolumn index 28.73 9.88 2.1 ~4mln. Records
  • 16. After multicolumn index on country_id and merchant_type Performance Min Average Max Seconds 0 7.5 15 22.5 30 Postgres Elasticsearch Postgres + multicolumn index 1.37 0.99 0.73 28.73 9.88 2.1 ~4mln. Records
  • 17. After multicolumn index on country_id and merchant_type Performance Min Average Max Seconds 0 7.5 15 22.5 30 Postgres Elasticsearch Postgres + multicolumn index 10.19 5.09 2.28 1.37 0.99 0.73 28.73 9.88 2.1 ~4mln. Records
  • 18. What is GiST Generalized Search Tree • In GiST each leaf contains logical expression and pointer to TID, where indexed data should satisfy logical expression. • Faster on insert, update What is GIN Generalized Inverted Index • It is b-tree with elements to which is connected another b-tree or plain list of TID's. • Faster and more accurate on select.
  • 19. Welcome to ruby meditation.
 All of us love ruby. Does everyone love meditation? Everyone Of Welcome All Does WelcomeRuby ToOfLove MeditationEveryone 0,1 0,10,1 2,1 1,51,5 2,1 2,1 Yellow rectangle are TID’s. First number is a page number and second is position on a page 0,1 1,52,1 1,5
  • 20. Welcome to ruby meditation.
 All of us love ruby. Does everyone love meditation? Everyone Of Welcome All Does WelcomeRuby ToOfLove MeditationEveryone 0,1 1,5 0,1 0,10,1 2,1 1,51,5 2,1 2,1 Yellow rectangle are TID’s. First number is a page number and second is position on a page 2,1 1,5 ruby rubylove love
  • 21. 1,5 1,5 Welcome to ruby meditation.
 All of us love ruby. Does everyone love meditation? Everyone Of Welcome All Does WelcomeRuby ToOfLove MeditationEveryone 0,1 0,1 0,10,1 2,1 1,51,5 2,1 2,1 Yellow rectangle are TID’s. First number is a page number and second is position on a page 2,1 ruby rubylove love love ruby
  • 22. gin_trgm_ops A trigram is a group of three consecutive characters taken from a string. We can measure the similarity of two strings by counting the number of trigrams they share.
  • 23. Performance after gin index on websites Min Average Max Seconds 0 5 10 15 20 Postgres Elasticsearch Postgres + multicolumn index Postgres + gin index with trgm_ops on websites ~4mln. Records
  • 24. Performance after gin index on websites Min Average Max Seconds 0 5 10 15 20 Postgres Elasticsearch Postgres + multicolumn index Postgres + gin index with trgm_ops on websites 28.73 9.88 2.1 ~4mln. Records
  • 25. Performance after gin index on websites Min Average Max Seconds 0 5 10 15 20 Postgres Elasticsearch Postgres + multicolumn index Postgres + gin index with trgm_ops on websites 1.37 0.99 0.75 28.73 9.88 2.1 ~4mln. Records
  • 26. Performance after gin index on websites Min Average Max Seconds 0 5 10 15 20 Postgres Elasticsearch Postgres + multicolumn index Postgres + gin index with trgm_ops on websites 10.19 5.09 2.28 1.37 0.99 0.75 28.73 9.88 2.1 ~4mln. Records
  • 27. Performance after gin index on websites Min Average Max Seconds 0 5 10 15 20 Postgres Elasticsearch Postgres + multicolumn index Postgres + gin index with trgm_ops on websites 0.55 0.34 0.26 10.19 5.09 2.28 1.37 0.99 0.75 28.73 9.88 2.1 ~4mln. Records
  • 28. How elasticsearch works • It uses analyzers for all incoming data. (it could be custom or default one) • Each analyzer has at least one tokenizer • Zero or more TokenFilters • Tokenizer may be preceded by one or more CharFilters
  • 31. How analyzer works? Input Char Filter String
  • 32. How analyzer works? Input Char Filter Tokenizer String String
  • 33. How analyzer works? Input Char Filter Tokenizer Token Filter String String Tokens
  • 34. How analyzer works? Input Char Filter Tokenizer Token Filter Output String String Tokens Tokens
  • 36. Example The 2 QUICK <p>Brown-Foxes</p> jumped over the lazy dog's bone.
  • 37. Example The 2 QUICK <p>Brown-Foxes</p> jumped over the lazy dog's bone. html_strip The 2 QUICK Brown-Foxes jumped over the lazy dog's bone.
  • 38. Example The 2 QUICK <p>Brown-Foxes</p> jumped over the lazy dog's bone. html_strip The 2 QUICK Brown-Foxes jumped over the lazy dog's bone. standart tokenizer The 2 QUICK Brown jumpedFoxes over the lazy dog’s bone
  • 39. Example The 2 QUICK <p>Brown-Foxes</p> jumped over the lazy dog's bone. html_strip The 2 QUICK Brown-Foxes jumped over the lazy dog's bone. standart tokenizer The 2 QUICK Brown jumpedFoxes over the lazy dog’s bone lowercase the 2 quick brown jumpedfoxes over the lazy dog’s bone
  • 40. Example The 2 QUICK <p>Brown-Foxes</p> jumped over the lazy dog's bone. html_strip The 2 QUICK Brown-Foxes jumped over the lazy dog's bone. standart tokenizer The 2 QUICK Brown jumpedFoxes over the lazy dog’s bone lowercase the 2 quick brown jumpedfoxes over the lazy dog’s bone stop 2 quick brown jumpedfoxes over lazy dog’s bone the the
  • 41. Example The 2 QUICK <p>Brown-Foxes</p> jumped over the lazy dog's bone. html_strip The 2 QUICK Brown-Foxes jumped over the lazy dog's bone. standart tokenizer The 2 QUICK Brown jumpedFoxes over the lazy dog’s bone lowercase the 2 quick brown jumpedfoxes over the lazy dog’s bone stop 2 quick brown jumpedfoxes over lazy dog’s bone snowball 2 quick brown jumpfox over lazi dog bone the the jump lazi dog
  • 42. Postgres full search implementation • We can use tsvector type to achieve almost the same functionality. By using to_tsvector function • To imporve perfomance we could create separate tsvector column with to_tsvector values. • To create a request we should use to_tsquery. & | <-> • plainto_tsquery works with plain text so you don’t need to insert any special symbols. Inserts & • phraseto_tsquery also works with plain text but marks that each token should be close to each other. Inserts <->
  • 43. Rum access method • Based on GIN access method code • Solves slow ranking • Solves slow phrase search (tsquery with <-> operator) • Supports index on tsquery column
  • 44. 122 1 5 3 2 4 4 3 3 4211 Welcome to ruby meditation.
 All of us love ruby. Does everyone love meditation? ruby, meditation, love Everyone Of Welcome All Does WelcomeRuby ToOfLove MeditationEveryone 0,1 0,10,1 2,1 1,51,5 2,1 2,1 The number in green rectangle is word position in the document. 0,1 1,52,1 1,5 8,4 8,4 8,4
  • 45. 122 1 5 3 2 4 4 3 3 4211 Welcome to ruby meditation.
 All of us love ruby. Does everyone love meditation? ruby, meditation, love Everyone Of Welcome All Does WelcomeRuby ToOfLove MeditationEveryone 0,1 1,5 0,1 0,10,1 2,1 1,51,5 2,1 2,1 The number in green rectangle is word position in the document. 2,1 1,5 ruby rubylove love love ruby 8,4 8,4 8,4
  • 46. 122 1 5 3 2 4 4 3 3 4211 1,5 1,5 Welcome to ruby meditation.
 All of us love ruby. Does everyone love meditation? ruby, meditation, love Everyone Of Welcome All Does WelcomeRuby ToOfLove MeditationEveryone 0,1 0,1 0,10,1 2,1 1,51,5 2,1 2,1 The number in green rectangle is word position in the document. 2,1 ruby rubylove love love ruby love ruby 8,4 8,4 8,4
  • 47. Conclusion • Postgres can also be fast. • Multicolumn indexes can improve performance if your search has multicolumn constraints. • For fast text search prefer using Gin when table doesn’t update occasionally, otherwise use GiST • Use gin with trgm_ops when using full text search. If full text search is still slow try to use tsvector data type with gin index on it. • When you have some kind ‘inverse full-text search’ problem. Add tsquery type in your table as a query and incoming data treat as a document. Add rum access method on query column with tsquery_ops for fast classification. • Before moving to other instrument make analysis of current/new instrument and verify is it worth moving or not.