SlideShare a Scribd company logo
Postgres vs
Elasticsearch while
enriching data.
Vlad Somov @ Salt Edge Inc.
Unstructured Data
Enrichment
Incoming raw data
Structured identified data
Keyword1 Keyword2 Website Name
Tag
Keyword1 Keyword2 Website Name
Tag
Unstructured Data
Enrichment
Some Transaction Description Website
Incoming raw data
Keyword1 Keyword2 Website
Structured identified data
Name
Tag
Description
Keyword1
Tag
Basic Setup Performance
Min
Average
Max
Seconds
0 7.5 15 22.5 30
Postgres Elasticsearch
~4mln. Records
Basic Setup Performance
Min
Average
Max
Seconds
0 7.5 15 22.5 30
Postgres Elasticsearch
28.73
9.88
2.10
~4mln. Records
Basic Setup Performance
Min
Average
Max
Seconds
0 7.5 15 22.5 30
Postgres Elasticsearch
1.37
0.99
0.73
28.73
9.88
2.10
~4mln. Records
B-tree index structure
3 39 68
meta
39 42 55 68 89 943 15 28
3 9 15 21 29 32 39 42 42 48 55 68 68 77 89 93 94 98
39
B-tree index structure
3 68
meta
39 42 55 68 89 943 15 28
3 9 15 21 29 32 39 42 42 48 55 68 68 77 89 93 94 98
39
39
B-tree index structure
3 68
meta
42 55 68 89 943 15 28
3 9 15 21 29 32 39 42 42 48 55 68 68 77 89 93 94 98
39
39
B-tree index structure
3 68
meta
42 55 68 89 943 15 28
3 9 15 21 29 32 39 42 42 48 55 68 68 77 89 93 94 98
39
39
B-tree index structure
3 68
meta
42 55 68 89 943 15 28
3 9 15 21 29 32 39 42 42 48 55 68 68 77 89 93 94 98
39
39
B-tree index structure
3 68
meta
42 55 68 89 943 15 28
3 9 15 21 29 32 39 42 42 48 55 68 68 77 89 93 94 98
Why it is useful?
• b-tree index sort values inside each node.

• b-tree is balanced

• Same level nodes are connected using doubly linked list.
After multicolumn index on country_id
and merchant_type Performance
Min
Average
Max
Seconds
0 7.5 15 22.5 30
Postgres Elasticsearch
Postgres + multicolumn index
~4mln. Records
After multicolumn index on country_id
and merchant_type Performance
Min
Average
Max
Seconds
0 7.5 15 22.5 30
Postgres Elasticsearch
Postgres + multicolumn index
28.73
9.88
2.1
~4mln. Records
After multicolumn index on country_id
and merchant_type Performance
Min
Average
Max
Seconds
0 7.5 15 22.5 30
Postgres Elasticsearch
Postgres + multicolumn index
1.37
0.99
0.73
28.73
9.88
2.1
~4mln. Records
After multicolumn index on country_id
and merchant_type Performance
Min
Average
Max
Seconds
0 7.5 15 22.5 30
Postgres Elasticsearch
Postgres + multicolumn index
10.19
5.09
2.28
1.37
0.99
0.73
28.73
9.88
2.1
~4mln. Records
What is GiST
Generalized Search
Tree
• In GiST each leaf contains
logical expression and
pointer to TID, where
indexed data should
satisfy logical expression.

• Faster on insert, update
What is GIN
Generalized Inverted
Index
• It is b-tree with elements to
which is connected another
b-tree or plain list of TID's. 

• Faster and more accurate
on select.
Welcome to ruby meditation.

All of us love ruby.
Does everyone love meditation?
Everyone Of Welcome
All Does WelcomeRuby ToOfLove MeditationEveryone
0,1 0,10,1
2,1
1,51,5 2,1 2,1
Yellow rectangle are TID’s. First number is a page number and second is
position on a page
0,1
1,52,1
1,5
Welcome to ruby meditation.

All of us love ruby.
Does everyone love meditation?
Everyone Of Welcome
All Does WelcomeRuby ToOfLove MeditationEveryone
0,1
1,5
0,1 0,10,1
2,1
1,51,5 2,1 2,1
Yellow rectangle are TID’s. First number is a page number and second is
position on a page
2,1
1,5
ruby
rubylove
love
1,5
1,5
Welcome to ruby meditation.

All of us love ruby.
Does everyone love meditation?
Everyone Of Welcome
All Does WelcomeRuby ToOfLove MeditationEveryone
0,1 0,1 0,10,1
2,1
1,51,5 2,1 2,1
Yellow rectangle are TID’s. First number is a page number and second is
position on a page
2,1
ruby
rubylove
love
love ruby
gin_trgm_ops
A trigram is a group of three consecutive characters
taken from a string.
We can measure the similarity of two strings by counting the
number of trigrams they share.
Performance after gin index on websites
Min
Average
Max
Seconds
0 5 10 15 20
Postgres
Elasticsearch
Postgres + multicolumn index
Postgres + gin index with trgm_ops on websites
~4mln. Records
Performance after gin index on websites
Min
Average
Max
Seconds
0 5 10 15 20
Postgres
Elasticsearch
Postgres + multicolumn index
Postgres + gin index with trgm_ops on websites
28.73
9.88
2.1
~4mln. Records
Performance after gin index on websites
Min
Average
Max
Seconds
0 5 10 15 20
Postgres
Elasticsearch
Postgres + multicolumn index
Postgres + gin index with trgm_ops on websites
1.37
0.99
0.75
28.73
9.88
2.1
~4mln. Records
Performance after gin index on websites
Min
Average
Max
Seconds
0 5 10 15 20
Postgres
Elasticsearch
Postgres + multicolumn index
Postgres + gin index with trgm_ops on websites
10.19
5.09
2.28
1.37
0.99
0.75
28.73
9.88
2.1
~4mln. Records
Performance after gin index on websites
Min
Average
Max
Seconds
0 5 10 15 20
Postgres
Elasticsearch
Postgres + multicolumn index
Postgres + gin index with trgm_ops on websites
0.55
0.34
0.26
10.19
5.09
2.28
1.37
0.99
0.75
28.73
9.88
2.1
~4mln. Records
How elasticsearch works
• It uses analyzers for all incoming data. (it could be custom
or default one)

• Each analyzer has at least one tokenizer

• Zero or more TokenFilters

• Tokenizer may be preceded by one or more CharFilters
How analyzer works?
How analyzer works?
Input
How analyzer works?
Input Char Filter
String
How analyzer works?
Input Char Filter Tokenizer
String String
How analyzer works?
Input Char Filter Tokenizer
Token
Filter
String String Tokens
How analyzer works?
Input Char Filter Tokenizer
Token
Filter
Output
String String Tokens Tokens
Example
Example
The 2 QUICK <p>Brown-Foxes</p> jumped over the lazy dog's bone.
Example
The 2 QUICK <p>Brown-Foxes</p> jumped over the lazy dog's bone.
html_strip
The 2 QUICK Brown-Foxes jumped over the lazy dog's bone.
Example
The 2 QUICK <p>Brown-Foxes</p> jumped over the lazy dog's bone.
html_strip
The 2 QUICK Brown-Foxes jumped over the lazy dog's bone.
standart tokenizer
The 2 QUICK Brown jumpedFoxes over
the lazy dog’s bone
Example
The 2 QUICK <p>Brown-Foxes</p> jumped over the lazy dog's bone.
html_strip
The 2 QUICK Brown-Foxes jumped over the lazy dog's bone.
standart tokenizer
The 2 QUICK Brown jumpedFoxes over
the lazy dog’s bone
lowercase
the 2 quick brown jumpedfoxes over
the lazy dog’s bone
Example
The 2 QUICK <p>Brown-Foxes</p> jumped over the lazy dog's bone.
html_strip
The 2 QUICK Brown-Foxes jumped over the lazy dog's bone.
standart tokenizer
The 2 QUICK Brown jumpedFoxes over
the lazy dog’s bone
lowercase
the 2 quick brown jumpedfoxes over
the lazy dog’s bone
stop
2 quick brown jumpedfoxes over lazy dog’s bone
the
the
Example
The 2 QUICK <p>Brown-Foxes</p> jumped over the lazy dog's bone.
html_strip
The 2 QUICK Brown-Foxes jumped over the lazy dog's bone.
standart tokenizer
The 2 QUICK Brown jumpedFoxes over
the lazy dog’s bone
lowercase
the 2 quick brown jumpedfoxes over
the lazy dog’s bone
stop
2 quick brown jumpedfoxes over lazy dog’s bone
snowball
2 quick brown jumpfox over lazi dog bone
the
the
jump lazi dog
Postgres full search
implementation
• We can use tsvector type to achieve almost the same
functionality. By using to_tsvector function

• To imporve perfomance we could create separate tsvector
column with to_tsvector values.

• To create a request we should use to_tsquery. & | <->

• plainto_tsquery works with plain text so you don’t need to
insert any special symbols. Inserts &

• phraseto_tsquery also works with plain text but marks that
each token should be close to each other. Inserts <->
Rum access method
• Based on GIN access method code

• Solves slow ranking

• Solves slow phrase search (tsquery with <-> operator)

• Supports index on tsquery column
122
1
5
3
2
4
4
3
3
4211
Welcome to ruby meditation.

All of us love ruby.
Does everyone love meditation?
ruby, meditation, love
Everyone Of Welcome
All Does WelcomeRuby ToOfLove MeditationEveryone
0,1 0,10,1
2,1
1,51,5 2,1 2,1
The number in green rectangle is word position in the document.
0,1
1,52,1
1,5
8,4 8,4 8,4
122
1
5
3
2
4
4
3
3
4211
Welcome to ruby meditation.

All of us love ruby.
Does everyone love meditation?
ruby, meditation, love
Everyone Of Welcome
All Does WelcomeRuby ToOfLove MeditationEveryone
0,1
1,5
0,1 0,10,1
2,1
1,51,5 2,1 2,1
The number in green rectangle is word position in the document.
2,1
1,5
ruby
rubylove
love
love
ruby
8,4 8,4 8,4
122
1
5
3
2
4
4
3
3
4211
1,5
1,5
Welcome to ruby meditation.

All of us love ruby.
Does everyone love meditation?
ruby, meditation, love
Everyone Of Welcome
All Does WelcomeRuby ToOfLove MeditationEveryone
0,1 0,1 0,10,1
2,1
1,51,5 2,1 2,1
The number in green rectangle is word position in the document.
2,1
ruby
rubylove
love
love ruby
love
ruby
8,4 8,4 8,4
Conclusion
• Postgres can also be fast.

• Multicolumn indexes can improve performance if your search has multicolumn
constraints.

• For fast text search prefer using Gin when table doesn’t update occasionally,
otherwise use GiST

• Use gin with trgm_ops when using full text search. If full text search is still slow
try to use tsvector data type with gin index on it.

• When you have some kind ‘inverse full-text search’ problem. Add tsquery type in
your table as a query and incoming data treat as a document. Add rum access
method on query column with tsquery_ops for fast classification.

• Before moving to other instrument make analysis of current/new instrument and
verify is it worth moving or not.
email: vlad.somov@icloud.com
twitter: @vsomov93
Questions?

More Related Content

What's hot

Deep Dive into Building a Secure & Multi-tenant SaaS Solution with NATS
Deep Dive into Building a Secure & Multi-tenant SaaS Solution with NATSDeep Dive into Building a Secure & Multi-tenant SaaS Solution with NATS
Deep Dive into Building a Secure & Multi-tenant SaaS Solution with NATS
NATS
 
Introducing BinarySortedMultiMap - A new Flink state primitive to boost your ...
Introducing BinarySortedMultiMap - A new Flink state primitive to boost your ...Introducing BinarySortedMultiMap - A new Flink state primitive to boost your ...
Introducing BinarySortedMultiMap - A new Flink state primitive to boost your ...
Flink Forward
 
3. Relationships Matter: Using Connected Data for Better Machine Learning
3. Relationships Matter: Using Connected Data for Better Machine Learning3. Relationships Matter: Using Connected Data for Better Machine Learning
3. Relationships Matter: Using Connected Data for Better Machine Learning
Neo4j
 
Microservices Meetup San Francisco - August 2017 Talk on NATS
Microservices Meetup San Francisco - August 2017 Talk on NATSMicroservices Meetup San Francisco - August 2017 Talk on NATS
Microservices Meetup San Francisco - August 2017 Talk on NATS
NATS
 
Microservice - Up to 500k CCU
Microservice - Up to 500k CCUMicroservice - Up to 500k CCU
Microservice - Up to 500k CCU
Viet Tran
 
Graph Gurus Episode 26: Using Graph Algorithms for Advanced Analytics Part 1
Graph Gurus Episode 26: Using Graph Algorithms for Advanced Analytics Part 1Graph Gurus Episode 26: Using Graph Algorithms for Advanced Analytics Part 1
Graph Gurus Episode 26: Using Graph Algorithms for Advanced Analytics Part 1
TigerGraph
 
“Alexa, be quiet!”: End-to-end near-real time model building and evaluation i...
“Alexa, be quiet!”: End-to-end near-real time model building and evaluation i...“Alexa, be quiet!”: End-to-end near-real time model building and evaluation i...
“Alexa, be quiet!”: End-to-end near-real time model building and evaluation i...
Flink Forward
 
An Introduction to Druid
An Introduction to DruidAn Introduction to Druid
An Introduction to Druid
DataWorks Summit
 
Pub/Sub Messaging
Pub/Sub MessagingPub/Sub Messaging
Pub/Sub Messaging
Peter Hanzlik
 
Change data capture with MongoDB and Kafka.
Change data capture with MongoDB and Kafka.Change data capture with MongoDB and Kafka.
Change data capture with MongoDB and Kafka.
Dan Harvey
 
Neo4j GraphTour Santa Monica 2019 - Amundsen Presentation
Neo4j GraphTour Santa Monica 2019 - Amundsen PresentationNeo4j GraphTour Santa Monica 2019 - Amundsen Presentation
Neo4j GraphTour Santa Monica 2019 - Amundsen Presentation
TamikaTannis
 
Deep Dive Into Elasticsearch
Deep Dive Into ElasticsearchDeep Dive Into Elasticsearch
Deep Dive Into Elasticsearch
Knoldus Inc.
 
Approximate nearest neighbor methods and vector models – NYC ML meetup
Approximate nearest neighbor methods and vector models – NYC ML meetupApproximate nearest neighbor methods and vector models – NYC ML meetup
Approximate nearest neighbor methods and vector models – NYC ML meetup
Erik Bernhardsson
 
Developing Real-Time Data Pipelines with Apache Kafka
Developing Real-Time Data Pipelines with Apache KafkaDeveloping Real-Time Data Pipelines with Apache Kafka
Developing Real-Time Data Pipelines with Apache Kafka
Joe Stein
 
Numeric Range Queries in Lucene and Solr
Numeric Range Queries in Lucene and SolrNumeric Range Queries in Lucene and Solr
Numeric Range Queries in Lucene and Solr
Vadim Kirilchuk
 
NATS Streaming - an alternative to Apache Kafka?
NATS Streaming - an alternative to Apache Kafka?NATS Streaming - an alternative to Apache Kafka?
NATS Streaming - an alternative to Apache Kafka?
Anton Zadorozhniy
 
Introducing Saga Pattern in Microservices with Spring Statemachine
Introducing Saga Pattern in Microservices with Spring StatemachineIntroducing Saga Pattern in Microservices with Spring Statemachine
Introducing Saga Pattern in Microservices with Spring Statemachine
VMware Tanzu
 
Graph Gurus 15: Introducing TigerGraph 2.4
Graph Gurus 15: Introducing TigerGraph 2.4 Graph Gurus 15: Introducing TigerGraph 2.4
Graph Gurus 15: Introducing TigerGraph 2.4
TigerGraph
 
Dynatrace
DynatraceDynatrace
Dynatrace
Purnima Kurella
 
Dynamically Scaling Data Streams across Multiple Kafka Clusters with Zero Fli...
Dynamically Scaling Data Streams across Multiple Kafka Clusters with Zero Fli...Dynamically Scaling Data Streams across Multiple Kafka Clusters with Zero Fli...
Dynamically Scaling Data Streams across Multiple Kafka Clusters with Zero Fli...
Flink Forward
 

What's hot (20)

Deep Dive into Building a Secure & Multi-tenant SaaS Solution with NATS
Deep Dive into Building a Secure & Multi-tenant SaaS Solution with NATSDeep Dive into Building a Secure & Multi-tenant SaaS Solution with NATS
Deep Dive into Building a Secure & Multi-tenant SaaS Solution with NATS
 
Introducing BinarySortedMultiMap - A new Flink state primitive to boost your ...
Introducing BinarySortedMultiMap - A new Flink state primitive to boost your ...Introducing BinarySortedMultiMap - A new Flink state primitive to boost your ...
Introducing BinarySortedMultiMap - A new Flink state primitive to boost your ...
 
3. Relationships Matter: Using Connected Data for Better Machine Learning
3. Relationships Matter: Using Connected Data for Better Machine Learning3. Relationships Matter: Using Connected Data for Better Machine Learning
3. Relationships Matter: Using Connected Data for Better Machine Learning
 
Microservices Meetup San Francisco - August 2017 Talk on NATS
Microservices Meetup San Francisco - August 2017 Talk on NATSMicroservices Meetup San Francisco - August 2017 Talk on NATS
Microservices Meetup San Francisco - August 2017 Talk on NATS
 
Microservice - Up to 500k CCU
Microservice - Up to 500k CCUMicroservice - Up to 500k CCU
Microservice - Up to 500k CCU
 
Graph Gurus Episode 26: Using Graph Algorithms for Advanced Analytics Part 1
Graph Gurus Episode 26: Using Graph Algorithms for Advanced Analytics Part 1Graph Gurus Episode 26: Using Graph Algorithms for Advanced Analytics Part 1
Graph Gurus Episode 26: Using Graph Algorithms for Advanced Analytics Part 1
 
“Alexa, be quiet!”: End-to-end near-real time model building and evaluation i...
“Alexa, be quiet!”: End-to-end near-real time model building and evaluation i...“Alexa, be quiet!”: End-to-end near-real time model building and evaluation i...
“Alexa, be quiet!”: End-to-end near-real time model building and evaluation i...
 
An Introduction to Druid
An Introduction to DruidAn Introduction to Druid
An Introduction to Druid
 
Pub/Sub Messaging
Pub/Sub MessagingPub/Sub Messaging
Pub/Sub Messaging
 
Change data capture with MongoDB and Kafka.
Change data capture with MongoDB and Kafka.Change data capture with MongoDB and Kafka.
Change data capture with MongoDB and Kafka.
 
Neo4j GraphTour Santa Monica 2019 - Amundsen Presentation
Neo4j GraphTour Santa Monica 2019 - Amundsen PresentationNeo4j GraphTour Santa Monica 2019 - Amundsen Presentation
Neo4j GraphTour Santa Monica 2019 - Amundsen Presentation
 
Deep Dive Into Elasticsearch
Deep Dive Into ElasticsearchDeep Dive Into Elasticsearch
Deep Dive Into Elasticsearch
 
Approximate nearest neighbor methods and vector models – NYC ML meetup
Approximate nearest neighbor methods and vector models – NYC ML meetupApproximate nearest neighbor methods and vector models – NYC ML meetup
Approximate nearest neighbor methods and vector models – NYC ML meetup
 
Developing Real-Time Data Pipelines with Apache Kafka
Developing Real-Time Data Pipelines with Apache KafkaDeveloping Real-Time Data Pipelines with Apache Kafka
Developing Real-Time Data Pipelines with Apache Kafka
 
Numeric Range Queries in Lucene and Solr
Numeric Range Queries in Lucene and SolrNumeric Range Queries in Lucene and Solr
Numeric Range Queries in Lucene and Solr
 
NATS Streaming - an alternative to Apache Kafka?
NATS Streaming - an alternative to Apache Kafka?NATS Streaming - an alternative to Apache Kafka?
NATS Streaming - an alternative to Apache Kafka?
 
Introducing Saga Pattern in Microservices with Spring Statemachine
Introducing Saga Pattern in Microservices with Spring StatemachineIntroducing Saga Pattern in Microservices with Spring Statemachine
Introducing Saga Pattern in Microservices with Spring Statemachine
 
Graph Gurus 15: Introducing TigerGraph 2.4
Graph Gurus 15: Introducing TigerGraph 2.4 Graph Gurus 15: Introducing TigerGraph 2.4
Graph Gurus 15: Introducing TigerGraph 2.4
 
Dynatrace
DynatraceDynatrace
Dynatrace
 
Dynamically Scaling Data Streams across Multiple Kafka Clusters with Zero Fli...
Dynamically Scaling Data Streams across Multiple Kafka Clusters with Zero Fli...Dynamically Scaling Data Streams across Multiple Kafka Clusters with Zero Fli...
Dynamically Scaling Data Streams across Multiple Kafka Clusters with Zero Fli...
 

Similar to Postgres vs Elasticsearch while enriching data - Vlad Somov | Ruby Meditaiton #23

Basics in algorithms and data structure
Basics in algorithms and data structure Basics in algorithms and data structure
Basics in algorithms and data structure
Eman magdy
 
ZFConf 2011: Что такое Sphinx, зачем он вообще нужен и как его использовать с...
ZFConf 2011: Что такое Sphinx, зачем он вообще нужен и как его использовать с...ZFConf 2011: Что такое Sphinx, зачем он вообще нужен и как его использовать с...
ZFConf 2011: Что такое Sphinx, зачем он вообще нужен и как его использовать с...
ZFConf Conference
 
Beyond php - it's not (just) about the code
Beyond php - it's not (just) about the codeBeyond php - it's not (just) about the code
Beyond php - it's not (just) about the code
Wim Godden
 
About elasticsearch
About elasticsearchAbout elasticsearch
About elasticsearch
Minsoo Jun
 
Beyond php - it's not (just) about the code
Beyond php - it's not (just) about the codeBeyond php - it's not (just) about the code
Beyond php - it's not (just) about the code
Wim Godden
 
Unveiling etcd: Architecture and Source Code Deep Dive
Unveiling etcd: Architecture and Source Code Deep DiveUnveiling etcd: Architecture and Source Code Deep Dive
Unveiling etcd: Architecture and Source Code Deep Dive
Chieh (Jack) Yu
 
The Challenges of Distributing Postgres: A Citus Story
The Challenges of Distributing Postgres: A Citus StoryThe Challenges of Distributing Postgres: A Citus Story
The Challenges of Distributing Postgres: A Citus Story
Hanna Kelman
 
The Challenges of Distributing Postgres: A Citus Story | DataEngConf NYC 2017...
The Challenges of Distributing Postgres: A Citus Story | DataEngConf NYC 2017...The Challenges of Distributing Postgres: A Citus Story | DataEngConf NYC 2017...
The Challenges of Distributing Postgres: A Citus Story | DataEngConf NYC 2017...
Citus Data
 
Beyond php - it's not (just) about the code
Beyond php - it's not (just) about the codeBeyond php - it's not (just) about the code
Beyond php - it's not (just) about the code
Wim Godden
 
Better Full Text Search in PostgreSQL
Better Full Text Search in PostgreSQLBetter Full Text Search in PostgreSQL
Better Full Text Search in PostgreSQL
Artur Zakirov
 
SequoiaDB Distributed Relational Database
SequoiaDB Distributed Relational DatabaseSequoiaDB Distributed Relational Database
SequoiaDB Distributed Relational Database
wangzhonnew
 
Beyond PHP - it's not (just) about the code
Beyond PHP - it's not (just) about the codeBeyond PHP - it's not (just) about the code
Beyond PHP - it's not (just) about the code
Wim Godden
 
Slash n near real time indexing
Slash n   near real time indexingSlash n   near real time indexing
Slash n near real time indexing
Umesh Prasad
 
Elasticsearch at Dailymotion
Elasticsearch at DailymotionElasticsearch at Dailymotion
Elasticsearch at Dailymotion
Cédric Hourcade
 
SDPHP - Percona Toolkit (It's Basically Magic)
SDPHP - Percona Toolkit (It's Basically Magic)SDPHP - Percona Toolkit (It's Basically Magic)
SDPHP - Percona Toolkit (It's Basically Magic)
Robert Swisher
 
A Call for Sanity in NoSQL
A Call for Sanity in NoSQLA Call for Sanity in NoSQL
A Call for Sanity in NoSQL
C4Media
 
Performance Optimization of Rails Applications
Performance Optimization of Rails ApplicationsPerformance Optimization of Rails Applications
Performance Optimization of Rails Applications
Serge Smetana
 
.NET Fest 2019. Łukasz Pyrzyk. Daily Performance Fuckups
.NET Fest 2019. Łukasz Pyrzyk. Daily Performance Fuckups.NET Fest 2019. Łukasz Pyrzyk. Daily Performance Fuckups
.NET Fest 2019. Łukasz Pyrzyk. Daily Performance Fuckups
NETFest
 
Moving Toward Deep Learning Algorithms on HPCC Systems
Moving Toward Deep Learning Algorithms on HPCC SystemsMoving Toward Deep Learning Algorithms on HPCC Systems
Moving Toward Deep Learning Algorithms on HPCC Systems
HPCC Systems
 
London devops logging
London devops loggingLondon devops logging
London devops logging
Tomas Doran
 

Similar to Postgres vs Elasticsearch while enriching data - Vlad Somov | Ruby Meditaiton #23 (20)

Basics in algorithms and data structure
Basics in algorithms and data structure Basics in algorithms and data structure
Basics in algorithms and data structure
 
ZFConf 2011: Что такое Sphinx, зачем он вообще нужен и как его использовать с...
ZFConf 2011: Что такое Sphinx, зачем он вообще нужен и как его использовать с...ZFConf 2011: Что такое Sphinx, зачем он вообще нужен и как его использовать с...
ZFConf 2011: Что такое Sphinx, зачем он вообще нужен и как его использовать с...
 
Beyond php - it's not (just) about the code
Beyond php - it's not (just) about the codeBeyond php - it's not (just) about the code
Beyond php - it's not (just) about the code
 
About elasticsearch
About elasticsearchAbout elasticsearch
About elasticsearch
 
Beyond php - it's not (just) about the code
Beyond php - it's not (just) about the codeBeyond php - it's not (just) about the code
Beyond php - it's not (just) about the code
 
Unveiling etcd: Architecture and Source Code Deep Dive
Unveiling etcd: Architecture and Source Code Deep DiveUnveiling etcd: Architecture and Source Code Deep Dive
Unveiling etcd: Architecture and Source Code Deep Dive
 
The Challenges of Distributing Postgres: A Citus Story
The Challenges of Distributing Postgres: A Citus StoryThe Challenges of Distributing Postgres: A Citus Story
The Challenges of Distributing Postgres: A Citus Story
 
The Challenges of Distributing Postgres: A Citus Story | DataEngConf NYC 2017...
The Challenges of Distributing Postgres: A Citus Story | DataEngConf NYC 2017...The Challenges of Distributing Postgres: A Citus Story | DataEngConf NYC 2017...
The Challenges of Distributing Postgres: A Citus Story | DataEngConf NYC 2017...
 
Beyond php - it's not (just) about the code
Beyond php - it's not (just) about the codeBeyond php - it's not (just) about the code
Beyond php - it's not (just) about the code
 
Better Full Text Search in PostgreSQL
Better Full Text Search in PostgreSQLBetter Full Text Search in PostgreSQL
Better Full Text Search in PostgreSQL
 
SequoiaDB Distributed Relational Database
SequoiaDB Distributed Relational DatabaseSequoiaDB Distributed Relational Database
SequoiaDB Distributed Relational Database
 
Beyond PHP - it's not (just) about the code
Beyond PHP - it's not (just) about the codeBeyond PHP - it's not (just) about the code
Beyond PHP - it's not (just) about the code
 
Slash n near real time indexing
Slash n   near real time indexingSlash n   near real time indexing
Slash n near real time indexing
 
Elasticsearch at Dailymotion
Elasticsearch at DailymotionElasticsearch at Dailymotion
Elasticsearch at Dailymotion
 
SDPHP - Percona Toolkit (It's Basically Magic)
SDPHP - Percona Toolkit (It's Basically Magic)SDPHP - Percona Toolkit (It's Basically Magic)
SDPHP - Percona Toolkit (It's Basically Magic)
 
A Call for Sanity in NoSQL
A Call for Sanity in NoSQLA Call for Sanity in NoSQL
A Call for Sanity in NoSQL
 
Performance Optimization of Rails Applications
Performance Optimization of Rails ApplicationsPerformance Optimization of Rails Applications
Performance Optimization of Rails Applications
 
.NET Fest 2019. Łukasz Pyrzyk. Daily Performance Fuckups
.NET Fest 2019. Łukasz Pyrzyk. Daily Performance Fuckups.NET Fest 2019. Łukasz Pyrzyk. Daily Performance Fuckups
.NET Fest 2019. Łukasz Pyrzyk. Daily Performance Fuckups
 
Moving Toward Deep Learning Algorithms on HPCC Systems
Moving Toward Deep Learning Algorithms on HPCC SystemsMoving Toward Deep Learning Algorithms on HPCC Systems
Moving Toward Deep Learning Algorithms on HPCC Systems
 
London devops logging
London devops loggingLondon devops logging
London devops logging
 

More from Ruby Meditation

Is this Legacy or Revenant Code? - Sergey Sergyenko | Ruby Meditation 30
Is this Legacy or Revenant Code? - Sergey Sergyenko  | Ruby Meditation 30Is this Legacy or Revenant Code? - Sergey Sergyenko  | Ruby Meditation 30
Is this Legacy or Revenant Code? - Sergey Sergyenko | Ruby Meditation 30
Ruby Meditation
 
Life with GraphQL API: good practices and unresolved issues - Roman Dubrovsky...
Life with GraphQL API: good practices and unresolved issues - Roman Dubrovsky...Life with GraphQL API: good practices and unresolved issues - Roman Dubrovsky...
Life with GraphQL API: good practices and unresolved issues - Roman Dubrovsky...
Ruby Meditation
 
Where is your license, dude? - Viacheslav Miroshnychenko | Ruby Meditation 29
Where is your license, dude? - Viacheslav Miroshnychenko | Ruby Meditation 29Where is your license, dude? - Viacheslav Miroshnychenko | Ruby Meditation 29
Where is your license, dude? - Viacheslav Miroshnychenko | Ruby Meditation 29
Ruby Meditation
 
Dry-validation update. Dry-validation vs Dry-schema 1.0 - Aleksandra Stolyar ...
Dry-validation update. Dry-validation vs Dry-schema 1.0 - Aleksandra Stolyar ...Dry-validation update. Dry-validation vs Dry-schema 1.0 - Aleksandra Stolyar ...
Dry-validation update. Dry-validation vs Dry-schema 1.0 - Aleksandra Stolyar ...
Ruby Meditation
 
How to cook Rabbit on Production - Bohdan Parshentsev | Ruby Meditation 28
How to cook Rabbit on Production - Bohdan Parshentsev | Ruby Meditation 28 How to cook Rabbit on Production - Bohdan Parshentsev | Ruby Meditation 28
How to cook Rabbit on Production - Bohdan Parshentsev | Ruby Meditation 28
Ruby Meditation
 
How to cook Rabbit on Production - Serhiy Nazarov | Ruby Meditation 28
How to cook Rabbit on Production - Serhiy Nazarov | Ruby Meditation 28How to cook Rabbit on Production - Serhiy Nazarov | Ruby Meditation 28
How to cook Rabbit on Production - Serhiy Nazarov | Ruby Meditation 28
Ruby Meditation
 
Reinventing the wheel - why do it and how to feel good about it - Julik Tarkh...
Reinventing the wheel - why do it and how to feel good about it - Julik Tarkh...Reinventing the wheel - why do it and how to feel good about it - Julik Tarkh...
Reinventing the wheel - why do it and how to feel good about it - Julik Tarkh...
Ruby Meditation
 
Performance Optimization 101 for Ruby developers - Nihad Abbasov (ENG) | Ruby...
Performance Optimization 101 for Ruby developers - Nihad Abbasov (ENG) | Ruby...Performance Optimization 101 for Ruby developers - Nihad Abbasov (ENG) | Ruby...
Performance Optimization 101 for Ruby developers - Nihad Abbasov (ENG) | Ruby...
Ruby Meditation
 
Use cases for Serverless Technologies - Ruslan Tolstov (RUS) | Ruby Meditatio...
Use cases for Serverless Technologies - Ruslan Tolstov (RUS) | Ruby Meditatio...Use cases for Serverless Technologies - Ruslan Tolstov (RUS) | Ruby Meditatio...
Use cases for Serverless Technologies - Ruslan Tolstov (RUS) | Ruby Meditatio...
Ruby Meditation
 
The Trailblazer Ride from the If Jungle into a Civilised Railway Station - Or...
The Trailblazer Ride from the If Jungle into a Civilised Railway Station - Or...The Trailblazer Ride from the If Jungle into a Civilised Railway Station - Or...
The Trailblazer Ride from the If Jungle into a Civilised Railway Station - Or...
Ruby Meditation
 
What/How to do with GraphQL? - Valentyn Ostakh (ENG) | Ruby Meditation 27
What/How to do with GraphQL? - Valentyn Ostakh (ENG) | Ruby Meditation 27What/How to do with GraphQL? - Valentyn Ostakh (ENG) | Ruby Meditation 27
What/How to do with GraphQL? - Valentyn Ostakh (ENG) | Ruby Meditation 27
Ruby Meditation
 
New features in Rails 6 - Nihad Abbasov (RUS) | Ruby Meditation 26
New features in Rails 6 -  Nihad Abbasov (RUS) | Ruby Meditation 26New features in Rails 6 -  Nihad Abbasov (RUS) | Ruby Meditation 26
New features in Rails 6 - Nihad Abbasov (RUS) | Ruby Meditation 26
Ruby Meditation
 
Security Scanning Overview - Tetiana Chupryna (RUS) | Ruby Meditation 26
Security Scanning Overview - Tetiana Chupryna (RUS) | Ruby Meditation 26Security Scanning Overview - Tetiana Chupryna (RUS) | Ruby Meditation 26
Security Scanning Overview - Tetiana Chupryna (RUS) | Ruby Meditation 26
Ruby Meditation
 
Teach your application eloquence. Logs, metrics, traces - Dmytro Shapovalov (...
Teach your application eloquence. Logs, metrics, traces - Dmytro Shapovalov (...Teach your application eloquence. Logs, metrics, traces - Dmytro Shapovalov (...
Teach your application eloquence. Logs, metrics, traces - Dmytro Shapovalov (...
Ruby Meditation
 
Best practices. Exploring - Ike Kurghinyan (RUS) | Ruby Meditation 26
Best practices. Exploring - Ike Kurghinyan (RUS) | Ruby Meditation 26Best practices. Exploring - Ike Kurghinyan (RUS) | Ruby Meditation 26
Best practices. Exploring - Ike Kurghinyan (RUS) | Ruby Meditation 26
Ruby Meditation
 
Road to A/B testing - Alexey Vasiliev (ENG) | Ruby Meditation 25
Road to A/B testing - Alexey Vasiliev (ENG) | Ruby Meditation 25Road to A/B testing - Alexey Vasiliev (ENG) | Ruby Meditation 25
Road to A/B testing - Alexey Vasiliev (ENG) | Ruby Meditation 25
Ruby Meditation
 
Concurrency in production. Real life example - Dmytro Herasymuk | Ruby Medita...
Concurrency in production. Real life example - Dmytro Herasymuk | Ruby Medita...Concurrency in production. Real life example - Dmytro Herasymuk | Ruby Medita...
Concurrency in production. Real life example - Dmytro Herasymuk | Ruby Medita...
Ruby Meditation
 
Data encryption for Ruby web applications - Dmytro Shapovalov (RUS) | Ruby Me...
Data encryption for Ruby web applications - Dmytro Shapovalov (RUS) | Ruby Me...Data encryption for Ruby web applications - Dmytro Shapovalov (RUS) | Ruby Me...
Data encryption for Ruby web applications - Dmytro Shapovalov (RUS) | Ruby Me...
Ruby Meditation
 
Rails App performance at the limit - Bogdan Gusiev
Rails App performance at the limit - Bogdan GusievRails App performance at the limit - Bogdan Gusiev
Rails App performance at the limit - Bogdan Gusiev
Ruby Meditation
 
GDPR. Next Y2K in 2018? - Anton Tkachov | Ruby Meditation #23
GDPR. Next Y2K in 2018? - Anton Tkachov | Ruby Meditation #23GDPR. Next Y2K in 2018? - Anton Tkachov | Ruby Meditation #23
GDPR. Next Y2K in 2018? - Anton Tkachov | Ruby Meditation #23
Ruby Meditation
 

More from Ruby Meditation (20)

Is this Legacy or Revenant Code? - Sergey Sergyenko | Ruby Meditation 30
Is this Legacy or Revenant Code? - Sergey Sergyenko  | Ruby Meditation 30Is this Legacy or Revenant Code? - Sergey Sergyenko  | Ruby Meditation 30
Is this Legacy or Revenant Code? - Sergey Sergyenko | Ruby Meditation 30
 
Life with GraphQL API: good practices and unresolved issues - Roman Dubrovsky...
Life with GraphQL API: good practices and unresolved issues - Roman Dubrovsky...Life with GraphQL API: good practices and unresolved issues - Roman Dubrovsky...
Life with GraphQL API: good practices and unresolved issues - Roman Dubrovsky...
 
Where is your license, dude? - Viacheslav Miroshnychenko | Ruby Meditation 29
Where is your license, dude? - Viacheslav Miroshnychenko | Ruby Meditation 29Where is your license, dude? - Viacheslav Miroshnychenko | Ruby Meditation 29
Where is your license, dude? - Viacheslav Miroshnychenko | Ruby Meditation 29
 
Dry-validation update. Dry-validation vs Dry-schema 1.0 - Aleksandra Stolyar ...
Dry-validation update. Dry-validation vs Dry-schema 1.0 - Aleksandra Stolyar ...Dry-validation update. Dry-validation vs Dry-schema 1.0 - Aleksandra Stolyar ...
Dry-validation update. Dry-validation vs Dry-schema 1.0 - Aleksandra Stolyar ...
 
How to cook Rabbit on Production - Bohdan Parshentsev | Ruby Meditation 28
How to cook Rabbit on Production - Bohdan Parshentsev | Ruby Meditation 28 How to cook Rabbit on Production - Bohdan Parshentsev | Ruby Meditation 28
How to cook Rabbit on Production - Bohdan Parshentsev | Ruby Meditation 28
 
How to cook Rabbit on Production - Serhiy Nazarov | Ruby Meditation 28
How to cook Rabbit on Production - Serhiy Nazarov | Ruby Meditation 28How to cook Rabbit on Production - Serhiy Nazarov | Ruby Meditation 28
How to cook Rabbit on Production - Serhiy Nazarov | Ruby Meditation 28
 
Reinventing the wheel - why do it and how to feel good about it - Julik Tarkh...
Reinventing the wheel - why do it and how to feel good about it - Julik Tarkh...Reinventing the wheel - why do it and how to feel good about it - Julik Tarkh...
Reinventing the wheel - why do it and how to feel good about it - Julik Tarkh...
 
Performance Optimization 101 for Ruby developers - Nihad Abbasov (ENG) | Ruby...
Performance Optimization 101 for Ruby developers - Nihad Abbasov (ENG) | Ruby...Performance Optimization 101 for Ruby developers - Nihad Abbasov (ENG) | Ruby...
Performance Optimization 101 for Ruby developers - Nihad Abbasov (ENG) | Ruby...
 
Use cases for Serverless Technologies - Ruslan Tolstov (RUS) | Ruby Meditatio...
Use cases for Serverless Technologies - Ruslan Tolstov (RUS) | Ruby Meditatio...Use cases for Serverless Technologies - Ruslan Tolstov (RUS) | Ruby Meditatio...
Use cases for Serverless Technologies - Ruslan Tolstov (RUS) | Ruby Meditatio...
 
The Trailblazer Ride from the If Jungle into a Civilised Railway Station - Or...
The Trailblazer Ride from the If Jungle into a Civilised Railway Station - Or...The Trailblazer Ride from the If Jungle into a Civilised Railway Station - Or...
The Trailblazer Ride from the If Jungle into a Civilised Railway Station - Or...
 
What/How to do with GraphQL? - Valentyn Ostakh (ENG) | Ruby Meditation 27
What/How to do with GraphQL? - Valentyn Ostakh (ENG) | Ruby Meditation 27What/How to do with GraphQL? - Valentyn Ostakh (ENG) | Ruby Meditation 27
What/How to do with GraphQL? - Valentyn Ostakh (ENG) | Ruby Meditation 27
 
New features in Rails 6 - Nihad Abbasov (RUS) | Ruby Meditation 26
New features in Rails 6 -  Nihad Abbasov (RUS) | Ruby Meditation 26New features in Rails 6 -  Nihad Abbasov (RUS) | Ruby Meditation 26
New features in Rails 6 - Nihad Abbasov (RUS) | Ruby Meditation 26
 
Security Scanning Overview - Tetiana Chupryna (RUS) | Ruby Meditation 26
Security Scanning Overview - Tetiana Chupryna (RUS) | Ruby Meditation 26Security Scanning Overview - Tetiana Chupryna (RUS) | Ruby Meditation 26
Security Scanning Overview - Tetiana Chupryna (RUS) | Ruby Meditation 26
 
Teach your application eloquence. Logs, metrics, traces - Dmytro Shapovalov (...
Teach your application eloquence. Logs, metrics, traces - Dmytro Shapovalov (...Teach your application eloquence. Logs, metrics, traces - Dmytro Shapovalov (...
Teach your application eloquence. Logs, metrics, traces - Dmytro Shapovalov (...
 
Best practices. Exploring - Ike Kurghinyan (RUS) | Ruby Meditation 26
Best practices. Exploring - Ike Kurghinyan (RUS) | Ruby Meditation 26Best practices. Exploring - Ike Kurghinyan (RUS) | Ruby Meditation 26
Best practices. Exploring - Ike Kurghinyan (RUS) | Ruby Meditation 26
 
Road to A/B testing - Alexey Vasiliev (ENG) | Ruby Meditation 25
Road to A/B testing - Alexey Vasiliev (ENG) | Ruby Meditation 25Road to A/B testing - Alexey Vasiliev (ENG) | Ruby Meditation 25
Road to A/B testing - Alexey Vasiliev (ENG) | Ruby Meditation 25
 
Concurrency in production. Real life example - Dmytro Herasymuk | Ruby Medita...
Concurrency in production. Real life example - Dmytro Herasymuk | Ruby Medita...Concurrency in production. Real life example - Dmytro Herasymuk | Ruby Medita...
Concurrency in production. Real life example - Dmytro Herasymuk | Ruby Medita...
 
Data encryption for Ruby web applications - Dmytro Shapovalov (RUS) | Ruby Me...
Data encryption for Ruby web applications - Dmytro Shapovalov (RUS) | Ruby Me...Data encryption for Ruby web applications - Dmytro Shapovalov (RUS) | Ruby Me...
Data encryption for Ruby web applications - Dmytro Shapovalov (RUS) | Ruby Me...
 
Rails App performance at the limit - Bogdan Gusiev
Rails App performance at the limit - Bogdan GusievRails App performance at the limit - Bogdan Gusiev
Rails App performance at the limit - Bogdan Gusiev
 
GDPR. Next Y2K in 2018? - Anton Tkachov | Ruby Meditation #23
GDPR. Next Y2K in 2018? - Anton Tkachov | Ruby Meditation #23GDPR. Next Y2K in 2018? - Anton Tkachov | Ruby Meditation #23
GDPR. Next Y2K in 2018? - Anton Tkachov | Ruby Meditation #23
 

Recently uploaded

Nordic Marketo Engage User Group_June 13_ 2024.pptx
Nordic Marketo Engage User Group_June 13_ 2024.pptxNordic Marketo Engage User Group_June 13_ 2024.pptx
Nordic Marketo Engage User Group_June 13_ 2024.pptx
MichaelKnudsen27
 
Your One-Stop Shop for Python Success: Top 10 US Python Development Providers
Your One-Stop Shop for Python Success: Top 10 US Python Development ProvidersYour One-Stop Shop for Python Success: Top 10 US Python Development Providers
Your One-Stop Shop for Python Success: Top 10 US Python Development Providers
akankshawande
 
Building Production Ready Search Pipelines with Spark and Milvus
Building Production Ready Search Pipelines with Spark and MilvusBuilding Production Ready Search Pipelines with Spark and Milvus
Building Production Ready Search Pipelines with Spark and Milvus
Zilliz
 
TrustArc Webinar - 2024 Global Privacy Survey
TrustArc Webinar - 2024 Global Privacy SurveyTrustArc Webinar - 2024 Global Privacy Survey
TrustArc Webinar - 2024 Global Privacy Survey
TrustArc
 
Fueling AI with Great Data with Airbyte Webinar
Fueling AI with Great Data with Airbyte WebinarFueling AI with Great Data with Airbyte Webinar
Fueling AI with Great Data with Airbyte Webinar
Zilliz
 
How to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdf
How to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdfHow to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdf
How to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdf
Chart Kalyan
 
Public CyberSecurity Awareness Presentation 2024.pptx
Public CyberSecurity Awareness Presentation 2024.pptxPublic CyberSecurity Awareness Presentation 2024.pptx
Public CyberSecurity Awareness Presentation 2024.pptx
marufrahmanstratejm
 
Columbus Data & Analytics Wednesdays - June 2024
Columbus Data & Analytics Wednesdays - June 2024Columbus Data & Analytics Wednesdays - June 2024
Columbus Data & Analytics Wednesdays - June 2024
Jason Packer
 
Apps Break Data
Apps Break DataApps Break Data
Apps Break Data
Ivo Velitchkov
 
Astute Business Solutions | Oracle Cloud Partner |
Astute Business Solutions | Oracle Cloud Partner |Astute Business Solutions | Oracle Cloud Partner |
Astute Business Solutions | Oracle Cloud Partner |
AstuteBusiness
 
Connector Corner: Seamlessly power UiPath Apps, GenAI with prebuilt connectors
Connector Corner: Seamlessly power UiPath Apps, GenAI with prebuilt connectorsConnector Corner: Seamlessly power UiPath Apps, GenAI with prebuilt connectors
Connector Corner: Seamlessly power UiPath Apps, GenAI with prebuilt connectors
DianaGray10
 
The Microsoft 365 Migration Tutorial For Beginner.pptx
The Microsoft 365 Migration Tutorial For Beginner.pptxThe Microsoft 365 Migration Tutorial For Beginner.pptx
The Microsoft 365 Migration Tutorial For Beginner.pptx
operationspcvita
 
HCL Notes und Domino Lizenzkostenreduzierung in der Welt von DLAU
HCL Notes und Domino Lizenzkostenreduzierung in der Welt von DLAUHCL Notes und Domino Lizenzkostenreduzierung in der Welt von DLAU
HCL Notes und Domino Lizenzkostenreduzierung in der Welt von DLAU
panagenda
 
Programming Foundation Models with DSPy - Meetup Slides
Programming Foundation Models with DSPy - Meetup SlidesProgramming Foundation Models with DSPy - Meetup Slides
Programming Foundation Models with DSPy - Meetup Slides
Zilliz
 
5th LF Energy Power Grid Model Meet-up Slides
5th LF Energy Power Grid Model Meet-up Slides5th LF Energy Power Grid Model Meet-up Slides
5th LF Energy Power Grid Model Meet-up Slides
DanBrown980551
 
Biomedical Knowledge Graphs for Data Scientists and Bioinformaticians
Biomedical Knowledge Graphs for Data Scientists and BioinformaticiansBiomedical Knowledge Graphs for Data Scientists and Bioinformaticians
Biomedical Knowledge Graphs for Data Scientists and Bioinformaticians
Neo4j
 
Taking AI to the Next Level in Manufacturing.pdf
Taking AI to the Next Level in Manufacturing.pdfTaking AI to the Next Level in Manufacturing.pdf
Taking AI to the Next Level in Manufacturing.pdf
ssuserfac0301
 
Digital Banking in the Cloud: How Citizens Bank Unlocked Their Mainframe
Digital Banking in the Cloud: How Citizens Bank Unlocked Their MainframeDigital Banking in the Cloud: How Citizens Bank Unlocked Their Mainframe
Digital Banking in the Cloud: How Citizens Bank Unlocked Their Mainframe
Precisely
 
Introduction of Cybersecurity with OSS at Code Europe 2024
Introduction of Cybersecurity with OSS  at Code Europe 2024Introduction of Cybersecurity with OSS  at Code Europe 2024
Introduction of Cybersecurity with OSS at Code Europe 2024
Hiroshi SHIBATA
 
Deep Dive: AI-Powered Marketing to Get More Leads and Customers with HyperGro...
Deep Dive: AI-Powered Marketing to Get More Leads and Customers with HyperGro...Deep Dive: AI-Powered Marketing to Get More Leads and Customers with HyperGro...
Deep Dive: AI-Powered Marketing to Get More Leads and Customers with HyperGro...
saastr
 

Recently uploaded (20)

Nordic Marketo Engage User Group_June 13_ 2024.pptx
Nordic Marketo Engage User Group_June 13_ 2024.pptxNordic Marketo Engage User Group_June 13_ 2024.pptx
Nordic Marketo Engage User Group_June 13_ 2024.pptx
 
Your One-Stop Shop for Python Success: Top 10 US Python Development Providers
Your One-Stop Shop for Python Success: Top 10 US Python Development ProvidersYour One-Stop Shop for Python Success: Top 10 US Python Development Providers
Your One-Stop Shop for Python Success: Top 10 US Python Development Providers
 
Building Production Ready Search Pipelines with Spark and Milvus
Building Production Ready Search Pipelines with Spark and MilvusBuilding Production Ready Search Pipelines with Spark and Milvus
Building Production Ready Search Pipelines with Spark and Milvus
 
TrustArc Webinar - 2024 Global Privacy Survey
TrustArc Webinar - 2024 Global Privacy SurveyTrustArc Webinar - 2024 Global Privacy Survey
TrustArc Webinar - 2024 Global Privacy Survey
 
Fueling AI with Great Data with Airbyte Webinar
Fueling AI with Great Data with Airbyte WebinarFueling AI with Great Data with Airbyte Webinar
Fueling AI with Great Data with Airbyte Webinar
 
How to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdf
How to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdfHow to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdf
How to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdf
 
Public CyberSecurity Awareness Presentation 2024.pptx
Public CyberSecurity Awareness Presentation 2024.pptxPublic CyberSecurity Awareness Presentation 2024.pptx
Public CyberSecurity Awareness Presentation 2024.pptx
 
Columbus Data & Analytics Wednesdays - June 2024
Columbus Data & Analytics Wednesdays - June 2024Columbus Data & Analytics Wednesdays - June 2024
Columbus Data & Analytics Wednesdays - June 2024
 
Apps Break Data
Apps Break DataApps Break Data
Apps Break Data
 
Astute Business Solutions | Oracle Cloud Partner |
Astute Business Solutions | Oracle Cloud Partner |Astute Business Solutions | Oracle Cloud Partner |
Astute Business Solutions | Oracle Cloud Partner |
 
Connector Corner: Seamlessly power UiPath Apps, GenAI with prebuilt connectors
Connector Corner: Seamlessly power UiPath Apps, GenAI with prebuilt connectorsConnector Corner: Seamlessly power UiPath Apps, GenAI with prebuilt connectors
Connector Corner: Seamlessly power UiPath Apps, GenAI with prebuilt connectors
 
The Microsoft 365 Migration Tutorial For Beginner.pptx
The Microsoft 365 Migration Tutorial For Beginner.pptxThe Microsoft 365 Migration Tutorial For Beginner.pptx
The Microsoft 365 Migration Tutorial For Beginner.pptx
 
HCL Notes und Domino Lizenzkostenreduzierung in der Welt von DLAU
HCL Notes und Domino Lizenzkostenreduzierung in der Welt von DLAUHCL Notes und Domino Lizenzkostenreduzierung in der Welt von DLAU
HCL Notes und Domino Lizenzkostenreduzierung in der Welt von DLAU
 
Programming Foundation Models with DSPy - Meetup Slides
Programming Foundation Models with DSPy - Meetup SlidesProgramming Foundation Models with DSPy - Meetup Slides
Programming Foundation Models with DSPy - Meetup Slides
 
5th LF Energy Power Grid Model Meet-up Slides
5th LF Energy Power Grid Model Meet-up Slides5th LF Energy Power Grid Model Meet-up Slides
5th LF Energy Power Grid Model Meet-up Slides
 
Biomedical Knowledge Graphs for Data Scientists and Bioinformaticians
Biomedical Knowledge Graphs for Data Scientists and BioinformaticiansBiomedical Knowledge Graphs for Data Scientists and Bioinformaticians
Biomedical Knowledge Graphs for Data Scientists and Bioinformaticians
 
Taking AI to the Next Level in Manufacturing.pdf
Taking AI to the Next Level in Manufacturing.pdfTaking AI to the Next Level in Manufacturing.pdf
Taking AI to the Next Level in Manufacturing.pdf
 
Digital Banking in the Cloud: How Citizens Bank Unlocked Their Mainframe
Digital Banking in the Cloud: How Citizens Bank Unlocked Their MainframeDigital Banking in the Cloud: How Citizens Bank Unlocked Their Mainframe
Digital Banking in the Cloud: How Citizens Bank Unlocked Their Mainframe
 
Introduction of Cybersecurity with OSS at Code Europe 2024
Introduction of Cybersecurity with OSS  at Code Europe 2024Introduction of Cybersecurity with OSS  at Code Europe 2024
Introduction of Cybersecurity with OSS at Code Europe 2024
 
Deep Dive: AI-Powered Marketing to Get More Leads and Customers with HyperGro...
Deep Dive: AI-Powered Marketing to Get More Leads and Customers with HyperGro...Deep Dive: AI-Powered Marketing to Get More Leads and Customers with HyperGro...
Deep Dive: AI-Powered Marketing to Get More Leads and Customers with HyperGro...
 

Postgres vs Elasticsearch while enriching data - Vlad Somov | Ruby Meditaiton #23

  • 1. Postgres vs Elasticsearch while enriching data. Vlad Somov @ Salt Edge Inc.
  • 2. Unstructured Data Enrichment Incoming raw data Structured identified data
  • 3. Keyword1 Keyword2 Website Name Tag Keyword1 Keyword2 Website Name Tag Unstructured Data Enrichment Some Transaction Description Website Incoming raw data Keyword1 Keyword2 Website Structured identified data Name Tag Description Keyword1 Tag
  • 4. Basic Setup Performance Min Average Max Seconds 0 7.5 15 22.5 30 Postgres Elasticsearch ~4mln. Records
  • 5. Basic Setup Performance Min Average Max Seconds 0 7.5 15 22.5 30 Postgres Elasticsearch 28.73 9.88 2.10 ~4mln. Records
  • 6. Basic Setup Performance Min Average Max Seconds 0 7.5 15 22.5 30 Postgres Elasticsearch 1.37 0.99 0.73 28.73 9.88 2.10 ~4mln. Records
  • 7. B-tree index structure 3 39 68 meta 39 42 55 68 89 943 15 28 3 9 15 21 29 32 39 42 42 48 55 68 68 77 89 93 94 98
  • 8. 39 B-tree index structure 3 68 meta 39 42 55 68 89 943 15 28 3 9 15 21 29 32 39 42 42 48 55 68 68 77 89 93 94 98
  • 9. 39 39 B-tree index structure 3 68 meta 42 55 68 89 943 15 28 3 9 15 21 29 32 39 42 42 48 55 68 68 77 89 93 94 98
  • 10. 39 39 B-tree index structure 3 68 meta 42 55 68 89 943 15 28 3 9 15 21 29 32 39 42 42 48 55 68 68 77 89 93 94 98
  • 11. 39 39 B-tree index structure 3 68 meta 42 55 68 89 943 15 28 3 9 15 21 29 32 39 42 42 48 55 68 68 77 89 93 94 98
  • 12. 39 39 B-tree index structure 3 68 meta 42 55 68 89 943 15 28 3 9 15 21 29 32 39 42 42 48 55 68 68 77 89 93 94 98
  • 13. Why it is useful? • b-tree index sort values inside each node. • b-tree is balanced • Same level nodes are connected using doubly linked list.
  • 14. After multicolumn index on country_id and merchant_type Performance Min Average Max Seconds 0 7.5 15 22.5 30 Postgres Elasticsearch Postgres + multicolumn index ~4mln. Records
  • 15. After multicolumn index on country_id and merchant_type Performance Min Average Max Seconds 0 7.5 15 22.5 30 Postgres Elasticsearch Postgres + multicolumn index 28.73 9.88 2.1 ~4mln. Records
  • 16. After multicolumn index on country_id and merchant_type Performance Min Average Max Seconds 0 7.5 15 22.5 30 Postgres Elasticsearch Postgres + multicolumn index 1.37 0.99 0.73 28.73 9.88 2.1 ~4mln. Records
  • 17. After multicolumn index on country_id and merchant_type Performance Min Average Max Seconds 0 7.5 15 22.5 30 Postgres Elasticsearch Postgres + multicolumn index 10.19 5.09 2.28 1.37 0.99 0.73 28.73 9.88 2.1 ~4mln. Records
  • 18. What is GiST Generalized Search Tree • In GiST each leaf contains logical expression and pointer to TID, where indexed data should satisfy logical expression. • Faster on insert, update What is GIN Generalized Inverted Index • It is b-tree with elements to which is connected another b-tree or plain list of TID's. • Faster and more accurate on select.
  • 19. Welcome to ruby meditation.
 All of us love ruby. Does everyone love meditation? Everyone Of Welcome All Does WelcomeRuby ToOfLove MeditationEveryone 0,1 0,10,1 2,1 1,51,5 2,1 2,1 Yellow rectangle are TID’s. First number is a page number and second is position on a page 0,1 1,52,1 1,5
  • 20. Welcome to ruby meditation.
 All of us love ruby. Does everyone love meditation? Everyone Of Welcome All Does WelcomeRuby ToOfLove MeditationEveryone 0,1 1,5 0,1 0,10,1 2,1 1,51,5 2,1 2,1 Yellow rectangle are TID’s. First number is a page number and second is position on a page 2,1 1,5 ruby rubylove love
  • 21. 1,5 1,5 Welcome to ruby meditation.
 All of us love ruby. Does everyone love meditation? Everyone Of Welcome All Does WelcomeRuby ToOfLove MeditationEveryone 0,1 0,1 0,10,1 2,1 1,51,5 2,1 2,1 Yellow rectangle are TID’s. First number is a page number and second is position on a page 2,1 ruby rubylove love love ruby
  • 22. gin_trgm_ops A trigram is a group of three consecutive characters taken from a string. We can measure the similarity of two strings by counting the number of trigrams they share.
  • 23. Performance after gin index on websites Min Average Max Seconds 0 5 10 15 20 Postgres Elasticsearch Postgres + multicolumn index Postgres + gin index with trgm_ops on websites ~4mln. Records
  • 24. Performance after gin index on websites Min Average Max Seconds 0 5 10 15 20 Postgres Elasticsearch Postgres + multicolumn index Postgres + gin index with trgm_ops on websites 28.73 9.88 2.1 ~4mln. Records
  • 25. Performance after gin index on websites Min Average Max Seconds 0 5 10 15 20 Postgres Elasticsearch Postgres + multicolumn index Postgres + gin index with trgm_ops on websites 1.37 0.99 0.75 28.73 9.88 2.1 ~4mln. Records
  • 26. Performance after gin index on websites Min Average Max Seconds 0 5 10 15 20 Postgres Elasticsearch Postgres + multicolumn index Postgres + gin index with trgm_ops on websites 10.19 5.09 2.28 1.37 0.99 0.75 28.73 9.88 2.1 ~4mln. Records
  • 27. Performance after gin index on websites Min Average Max Seconds 0 5 10 15 20 Postgres Elasticsearch Postgres + multicolumn index Postgres + gin index with trgm_ops on websites 0.55 0.34 0.26 10.19 5.09 2.28 1.37 0.99 0.75 28.73 9.88 2.1 ~4mln. Records
  • 28. How elasticsearch works • It uses analyzers for all incoming data. (it could be custom or default one) • Each analyzer has at least one tokenizer • Zero or more TokenFilters • Tokenizer may be preceded by one or more CharFilters
  • 31. How analyzer works? Input Char Filter String
  • 32. How analyzer works? Input Char Filter Tokenizer String String
  • 33. How analyzer works? Input Char Filter Tokenizer Token Filter String String Tokens
  • 34. How analyzer works? Input Char Filter Tokenizer Token Filter Output String String Tokens Tokens
  • 36. Example The 2 QUICK <p>Brown-Foxes</p> jumped over the lazy dog's bone.
  • 37. Example The 2 QUICK <p>Brown-Foxes</p> jumped over the lazy dog's bone. html_strip The 2 QUICK Brown-Foxes jumped over the lazy dog's bone.
  • 38. Example The 2 QUICK <p>Brown-Foxes</p> jumped over the lazy dog's bone. html_strip The 2 QUICK Brown-Foxes jumped over the lazy dog's bone. standart tokenizer The 2 QUICK Brown jumpedFoxes over the lazy dog’s bone
  • 39. Example The 2 QUICK <p>Brown-Foxes</p> jumped over the lazy dog's bone. html_strip The 2 QUICK Brown-Foxes jumped over the lazy dog's bone. standart tokenizer The 2 QUICK Brown jumpedFoxes over the lazy dog’s bone lowercase the 2 quick brown jumpedfoxes over the lazy dog’s bone
  • 40. Example The 2 QUICK <p>Brown-Foxes</p> jumped over the lazy dog's bone. html_strip The 2 QUICK Brown-Foxes jumped over the lazy dog's bone. standart tokenizer The 2 QUICK Brown jumpedFoxes over the lazy dog’s bone lowercase the 2 quick brown jumpedfoxes over the lazy dog’s bone stop 2 quick brown jumpedfoxes over lazy dog’s bone the the
  • 41. Example The 2 QUICK <p>Brown-Foxes</p> jumped over the lazy dog's bone. html_strip The 2 QUICK Brown-Foxes jumped over the lazy dog's bone. standart tokenizer The 2 QUICK Brown jumpedFoxes over the lazy dog’s bone lowercase the 2 quick brown jumpedfoxes over the lazy dog’s bone stop 2 quick brown jumpedfoxes over lazy dog’s bone snowball 2 quick brown jumpfox over lazi dog bone the the jump lazi dog
  • 42. Postgres full search implementation • We can use tsvector type to achieve almost the same functionality. By using to_tsvector function • To imporve perfomance we could create separate tsvector column with to_tsvector values. • To create a request we should use to_tsquery. & | <-> • plainto_tsquery works with plain text so you don’t need to insert any special symbols. Inserts & • phraseto_tsquery also works with plain text but marks that each token should be close to each other. Inserts <->
  • 43. Rum access method • Based on GIN access method code • Solves slow ranking • Solves slow phrase search (tsquery with <-> operator) • Supports index on tsquery column
  • 44. 122 1 5 3 2 4 4 3 3 4211 Welcome to ruby meditation.
 All of us love ruby. Does everyone love meditation? ruby, meditation, love Everyone Of Welcome All Does WelcomeRuby ToOfLove MeditationEveryone 0,1 0,10,1 2,1 1,51,5 2,1 2,1 The number in green rectangle is word position in the document. 0,1 1,52,1 1,5 8,4 8,4 8,4
  • 45. 122 1 5 3 2 4 4 3 3 4211 Welcome to ruby meditation.
 All of us love ruby. Does everyone love meditation? ruby, meditation, love Everyone Of Welcome All Does WelcomeRuby ToOfLove MeditationEveryone 0,1 1,5 0,1 0,10,1 2,1 1,51,5 2,1 2,1 The number in green rectangle is word position in the document. 2,1 1,5 ruby rubylove love love ruby 8,4 8,4 8,4
  • 46. 122 1 5 3 2 4 4 3 3 4211 1,5 1,5 Welcome to ruby meditation.
 All of us love ruby. Does everyone love meditation? ruby, meditation, love Everyone Of Welcome All Does WelcomeRuby ToOfLove MeditationEveryone 0,1 0,1 0,10,1 2,1 1,51,5 2,1 2,1 The number in green rectangle is word position in the document. 2,1 ruby rubylove love love ruby love ruby 8,4 8,4 8,4
  • 47. Conclusion • Postgres can also be fast. • Multicolumn indexes can improve performance if your search has multicolumn constraints. • For fast text search prefer using Gin when table doesn’t update occasionally, otherwise use GiST • Use gin with trgm_ops when using full text search. If full text search is still slow try to use tsvector data type with gin index on it. • When you have some kind ‘inverse full-text search’ problem. Add tsquery type in your table as a query and incoming data treat as a document. Add rum access method on query column with tsquery_ops for fast classification. • Before moving to other instrument make analysis of current/new instrument and verify is it worth moving or not.