SlideShare a Scribd company logo
1 of 38
Download to read offline
Implementing your first
PostgreSQL extension:
From Coding to
Distribution
Burak Yücesoy Önder Kalacı
Who are we?
• Two engineers from Microsoft.
• Used to work for Citus Data (now acquired by
Microsoft).
• Earn our lives by developing PostgreSQL
extensions.
Who are we?
Outline
What is PostgreSQL Extension?
Why PostgreSQL is extendable?
What is extendable in PostgreSQL?
•Developing
•Debugging
•Testing
•Packaging
•Releasing
Life cycle of PostgreSQL extension development
What is PostgreSQL Extension?
• A PostgreSQL extension is a piece of software that adds functionality to
Postgres. Each extension bundles related objects together.
• Postgres 9.1 started providing official APIs to override or extend any
database module’s behavior.
• “CREATE EXTENSION extension_name;” dynamically loads these objects into
Postgres’ address space.
Example Extension: pg_cron
• A task scheduler inside the database; it allows to perform periodic jobs on
the database.
• Every Saturday at 03:30; delete old data
 psql> SELECT cron.schedule('30 3 * * 6', $$DELETE FROM events WHERE
event_time < now() - interval '1 week'$$);
• Every day at 10:00, run VACUUM
 psql> SELECT cron.schedule('0 10 * * *', $$VACUUM$$);
Why PostgreSQL is Extendable?
• Every decade brings new workloads for databases.
• The last decade was about capturing more data, in more shapes and form.
• Postgres has been forked by dozens of commercial databases for new
workloads. When you fork, your database diverges from the community.
• What if you could leverage the database ecosystem and grow with it?
What is Extendable in PostgreSQL?
You can override, cooperate with, or extend any combination of the following
database modules:
 Type system and operators
 User defined functions and aggregates
 Storage system and indexes
 Write ahead logging and replication
 Transaction engine
 Background worker processes
 Query planner and query executor
 Configuration and database metadata
Type system and operators
• PostgreSQL already has lots of different data types;
 bigint, text, timestampz, jsonb, …
• If you need a data type which doesn’t exist in PostgreSQL;
 You can define new type with CREATE TYPE command.
 You can add the types created by other people using extensions
• Some additional data types;
 ip-address, e-mail
Type system and operators
Type system and operators
• It is also possible to define (or overwrite) operators for the types you created.
• For example > or < operators can be meaningful for ip address data type.
• Or you can come up with completely new operator such as;
 Distance function for points; Point <-> Point
 Membership of point in Sphere; Point <.> Sphere
User defined functions and
aggregates
• You can create new function or aggregate using CREATE FUNCTION
command.
• If you are performing some operations frequently it may make sense to
implement them as function.
• Also if you defined new type, you can also create the functions to perform
specific things on the type you create.
User defined functions and
aggregates
What is Extendable in PostgreSQL?
You can override, cooperate with, or extend any combination of the following
database modules:
 Type system and operators
 User defined functions and aggregates
 Storage system and indexes
 Write ahead logging and replication
 Transaction engine
 Background worker processes
 Query planner and query executor
 Configuration and database metadata
Can be done in SQL
Needs to be done in lower lever like C
PostgreSQL
Extension
Development
Life Cycle
Developing
Debugging
Testing
Packaging
Releasing
Disclaimer
“Every problem is a
gift. Without them we
wouldn’t grow”
- Tony Robbins
Today’s Schedule for Extension
Development
• Today we will follow several steps to create a working prototype to represent
color information in database;
 Primitive approach with using text
 Composite type
 UDFs and operators for our type
 C level implementation
 Creation of custom nodes for our type
 Modifications to executor and planner
• You can follow the development from this repo
PostgreSQL Internals: Datums
• Datums are PostgreSQL’s way of representing single value.
• Values are passed to or from PostgreSQL as Datums.
• It encapsulates the actual value and provides generic interface for all kinds
of value types.
• The code using the Datum has to know which type it is, since the Datum
itself doesn't contain that information.
• Conversion to and from Datum is made by helper functions;
 Int32GetDatum(int): Converts int value to Datum
 DatumGet32Int(Datum): Gets the int value stored in Datum
PostgreSQL Internals: Tuple
• Tuples have many different use-cases but most importantly are
representation of rows in the database.
• They are made up from Datums.
• Interactions with tuples are made by macros defined in PostgreSQL
codebase
PostgreSQL Internals: Memory
Context
• All memory allocations is handled by various memory contexts.
• You need to allocate memory by palloc() function instead of standard
malloc().
• There are multiple memory contexts with different lifetimes;
 TopMemoryContext
 CacheMemoryContext
 MessageMemoryContext
• More information at:
https://github.com/postgres/postgres/blob/master/src/backend/utils/mmgr/README
PostgreSQL Internals: Error
Reporting
• elog and ereport functions are used for error reporting.
• They are used to print user visible error messages, but more importantly;
 They rollback open transaction
 They release any allocated memory for the transaction/queries in related memory
contexts.
• It is even possible to extend the way error messages are handled in
PostgreSQL
• More information at:
https://github.com/postgres/postgres/blob/master/src/backend/utils/error/elog.
c#L3
PostgreSQL Internals: Node
• PostgreSQL creates a query text to a query tree.
• Query tree is made up from nodes.
• Each node has a type and related data in it.
• It is possible to create your own node types.
• More information:
https://github.com/postgres/postgres/blob/master/src/backend/nodes/READM
E
Let’s Implement An Equality Function
Datum
color_eq(PG_FUNCTION_ARGS)
{
}
Let’s Implement An Equality Function
Datum
color_eq(PG_FUNCTION_ARGS)
{
color *c1 = PG_GETARG_COLOR(0);
}
Let’s Implement An Equality Function
Datum
color_eq(PG_FUNCTION_ARGS)
{
color *c1 = PG_GETARG_COLOR(0);
color *c2 = PG_GETARG_COLOR(1);
}
Let’s Implement An Equality Function
Datum
color_eq(PG_FUNCTION_ARGS)
{
color *c1 = PG_GETARG_COLOR(0);
color *c2 = PG_GETARG_COLOR(1);
return c1->r == c2->r && c1->g == c2->g && c1->b == c2->b;
}
Let’s Implement An Equality Function V2
static bool
EqualPgColorExtendedNode(
const struct ExtensibleNode *target_node,
const struct ExtensibleNode *source_node)
{
}
Let’s Implement An Equality Function V2
static bool
EqualPgColorExtendedNode(
const struct ExtensibleNode *target_node,
const struct ExtensibleNode *source_node)
{
PgColorExtendedNode *targetPlan = (PgColorExtendedNode *) target_node;
}
Let’s Implement An Equality Function V2
static bool
EqualPgColorExtendedNode(
const struct ExtensibleNode *target_node,
const struct ExtensibleNode *source_node)
{
PgColorExtendedNode *targetPlan = (PgColorExtendedNode *) target_node;
PgColorExtendedNode *sourcePlan = (PgColorExtendedNode *) source_node;
}
Let’s Implement An Equality Function V2
static bool
EqualPgColorExtendedNode(
const struct ExtensibleNode *target_node,
const struct ExtensibleNode *source_node)
{
PgColorExtendedNode *targetPlan = (PgColorExtendedNode *) target_node;
PgColorExtendedNode *sourcePlan = (PgColorExtendedNode *) source_node;
return targetPlan->interceptedColor->r == sourcePlan->interceptedColor->r &&
targetPlan->interceptedColor->g == sourcePlan->interceptedColor->g &&
targetPlan->interceptedColor->b == sourcePlan->interceptedColor->b;
}
Testing
• Extensions can use PostgreSQL's own testing suite
• Make check and make installcheck
• Runs queries against database and compare the output
Packaging
• PostgreSQL is commonly used in RedHat and Debian bases operation
systems.
• For each operating system you want to run your extension on, you need
generate binaries in that particular system
• Docker is life saver
• Our open source packaging tools; https://github.com/citusdata/packaging
Packaging
Minimum requirements;
• debian/pgversions
• debian/control.in
• debian/changelog
• debian/copyright
• debian/rules
• debian/compat
Packaging
• > pg_buildext updatecontrol
• > debuild -uc -us -B --lintian-opts --profile debian --allow-root
Releasing
• PostgreSQL community software repositories
• PGXN
• Your own package repository;
 You can install your package repository to a server and respond install requests
from that server.
 Managed services; packagecloud.io
Thank you
&
Questions

More Related Content

What's hot

Demystifying DataFrame and Dataset
Demystifying DataFrame and DatasetDemystifying DataFrame and Dataset
Demystifying DataFrame and DatasetKazuaki Ishizaki
 
Jake Mannix, Lead Data Engineer, Lucidworks at MLconf SEA - 5/20/16
Jake Mannix, Lead Data Engineer, Lucidworks at MLconf SEA - 5/20/16Jake Mannix, Lead Data Engineer, Lucidworks at MLconf SEA - 5/20/16
Jake Mannix, Lead Data Engineer, Lucidworks at MLconf SEA - 5/20/16MLconf
 
Strata NYC 2015: Sketching Big Data with Spark: randomized algorithms for lar...
Strata NYC 2015: Sketching Big Data with Spark: randomized algorithms for lar...Strata NYC 2015: Sketching Big Data with Spark: randomized algorithms for lar...
Strata NYC 2015: Sketching Big Data with Spark: randomized algorithms for lar...Databricks
 
Feature Hashing for Scalable Machine Learning: Spark Summit East talk by Nick...
Feature Hashing for Scalable Machine Learning: Spark Summit East talk by Nick...Feature Hashing for Scalable Machine Learning: Spark Summit East talk by Nick...
Feature Hashing for Scalable Machine Learning: Spark Summit East talk by Nick...Spark Summit
 
Time-evolving Graph Processing on Commodity Clusters: Spark Summit East talk ...
Time-evolving Graph Processing on Commodity Clusters: Spark Summit East talk ...Time-evolving Graph Processing on Commodity Clusters: Spark Summit East talk ...
Time-evolving Graph Processing on Commodity Clusters: Spark Summit East talk ...Spark Summit
 
Exploiting GPU's for Columnar DataFrrames by Kiran Lonikar
Exploiting GPU's for Columnar DataFrrames by Kiran LonikarExploiting GPU's for Columnar DataFrrames by Kiran Lonikar
Exploiting GPU's for Columnar DataFrrames by Kiran LonikarSpark Summit
 
Elasticsearch And Apache Lucene For Apache Spark And MLlib
Elasticsearch And Apache Lucene For Apache Spark And MLlibElasticsearch And Apache Lucene For Apache Spark And MLlib
Elasticsearch And Apache Lucene For Apache Spark And MLlibJen Aman
 
Apache Flink - Hadoop MapReduce Compatibility
Apache Flink - Hadoop MapReduce CompatibilityApache Flink - Hadoop MapReduce Compatibility
Apache Flink - Hadoop MapReduce CompatibilityFabian Hueske
 
Introduction to Apache Flink - Fast and reliable big data processing
Introduction to Apache Flink - Fast and reliable big data processingIntroduction to Apache Flink - Fast and reliable big data processing
Introduction to Apache Flink - Fast and reliable big data processingTill Rohrmann
 
Automated Machine Learning Using Spark Mllib to Improve Customer Experience-(...
Automated Machine Learning Using Spark Mllib to Improve Customer Experience-(...Automated Machine Learning Using Spark Mllib to Improve Customer Experience-(...
Automated Machine Learning Using Spark Mllib to Improve Customer Experience-(...Spark Summit
 
Spark Summit EU talk by Qifan Pu
Spark Summit EU talk by Qifan PuSpark Summit EU talk by Qifan Pu
Spark Summit EU talk by Qifan PuSpark Summit
 
Extending Spark's Ingestion: Build Your Own Java Data Source with Jean George...
Extending Spark's Ingestion: Build Your Own Java Data Source with Jean George...Extending Spark's Ingestion: Build Your Own Java Data Source with Jean George...
Extending Spark's Ingestion: Build Your Own Java Data Source with Jean George...Databricks
 
GraphLab Conference 2014 Yucheng Low - Scalable Data Structures: SFrame & SGr...
GraphLab Conference 2014 Yucheng Low - Scalable Data Structures: SFrame & SGr...GraphLab Conference 2014 Yucheng Low - Scalable Data Structures: SFrame & SGr...
GraphLab Conference 2014 Yucheng Low - Scalable Data Structures: SFrame & SGr...Turi, Inc.
 
Mathias Brandewinder, Software Engineer & Data Scientist, Clear Lines Consult...
Mathias Brandewinder, Software Engineer & Data Scientist, Clear Lines Consult...Mathias Brandewinder, Software Engineer & Data Scientist, Clear Lines Consult...
Mathias Brandewinder, Software Engineer & Data Scientist, Clear Lines Consult...MLconf
 
Spark Workflow Management
Spark Workflow ManagementSpark Workflow Management
Spark Workflow ManagementRomi Kuntsman
 
Scalable data structures for data science
Scalable data structures for data scienceScalable data structures for data science
Scalable data structures for data scienceTuri, Inc.
 
Gradoop: Scalable Graph Analytics with Apache Flink @ Flink Forward 2015
Gradoop: Scalable Graph Analytics with Apache Flink @ Flink Forward 2015Gradoop: Scalable Graph Analytics with Apache Flink @ Flink Forward 2015
Gradoop: Scalable Graph Analytics with Apache Flink @ Flink Forward 2015Martin Junghanns
 
VEGAS: The Missing Matplotlib for Scala/Apache Spark with DB Tsai and Roger M...
VEGAS: The Missing Matplotlib for Scala/Apache Spark with DB Tsai and Roger M...VEGAS: The Missing Matplotlib for Scala/Apache Spark with DB Tsai and Roger M...
VEGAS: The Missing Matplotlib for Scala/Apache Spark with DB Tsai and Roger M...Spark Summit
 
Seattle Scalability Mahout
Seattle Scalability MahoutSeattle Scalability Mahout
Seattle Scalability MahoutJake Mannix
 

What's hot (20)

Demystifying DataFrame and Dataset
Demystifying DataFrame and DatasetDemystifying DataFrame and Dataset
Demystifying DataFrame and Dataset
 
Jake Mannix, Lead Data Engineer, Lucidworks at MLconf SEA - 5/20/16
Jake Mannix, Lead Data Engineer, Lucidworks at MLconf SEA - 5/20/16Jake Mannix, Lead Data Engineer, Lucidworks at MLconf SEA - 5/20/16
Jake Mannix, Lead Data Engineer, Lucidworks at MLconf SEA - 5/20/16
 
Strata NYC 2015: Sketching Big Data with Spark: randomized algorithms for lar...
Strata NYC 2015: Sketching Big Data with Spark: randomized algorithms for lar...Strata NYC 2015: Sketching Big Data with Spark: randomized algorithms for lar...
Strata NYC 2015: Sketching Big Data with Spark: randomized algorithms for lar...
 
Feature Hashing for Scalable Machine Learning: Spark Summit East talk by Nick...
Feature Hashing for Scalable Machine Learning: Spark Summit East talk by Nick...Feature Hashing for Scalable Machine Learning: Spark Summit East talk by Nick...
Feature Hashing for Scalable Machine Learning: Spark Summit East talk by Nick...
 
Time-evolving Graph Processing on Commodity Clusters: Spark Summit East talk ...
Time-evolving Graph Processing on Commodity Clusters: Spark Summit East talk ...Time-evolving Graph Processing on Commodity Clusters: Spark Summit East talk ...
Time-evolving Graph Processing on Commodity Clusters: Spark Summit East talk ...
 
Exploiting GPU's for Columnar DataFrrames by Kiran Lonikar
Exploiting GPU's for Columnar DataFrrames by Kiran LonikarExploiting GPU's for Columnar DataFrrames by Kiran Lonikar
Exploiting GPU's for Columnar DataFrrames by Kiran Lonikar
 
Elasticsearch And Apache Lucene For Apache Spark And MLlib
Elasticsearch And Apache Lucene For Apache Spark And MLlibElasticsearch And Apache Lucene For Apache Spark And MLlib
Elasticsearch And Apache Lucene For Apache Spark And MLlib
 
Apache Flink - Hadoop MapReduce Compatibility
Apache Flink - Hadoop MapReduce CompatibilityApache Flink - Hadoop MapReduce Compatibility
Apache Flink - Hadoop MapReduce Compatibility
 
Introduction to Apache Flink - Fast and reliable big data processing
Introduction to Apache Flink - Fast and reliable big data processingIntroduction to Apache Flink - Fast and reliable big data processing
Introduction to Apache Flink - Fast and reliable big data processing
 
Automated Machine Learning Using Spark Mllib to Improve Customer Experience-(...
Automated Machine Learning Using Spark Mllib to Improve Customer Experience-(...Automated Machine Learning Using Spark Mllib to Improve Customer Experience-(...
Automated Machine Learning Using Spark Mllib to Improve Customer Experience-(...
 
Spark Summit EU talk by Qifan Pu
Spark Summit EU talk by Qifan PuSpark Summit EU talk by Qifan Pu
Spark Summit EU talk by Qifan Pu
 
Extending Spark's Ingestion: Build Your Own Java Data Source with Jean George...
Extending Spark's Ingestion: Build Your Own Java Data Source with Jean George...Extending Spark's Ingestion: Build Your Own Java Data Source with Jean George...
Extending Spark's Ingestion: Build Your Own Java Data Source with Jean George...
 
GraphLab Conference 2014 Yucheng Low - Scalable Data Structures: SFrame & SGr...
GraphLab Conference 2014 Yucheng Low - Scalable Data Structures: SFrame & SGr...GraphLab Conference 2014 Yucheng Low - Scalable Data Structures: SFrame & SGr...
GraphLab Conference 2014 Yucheng Low - Scalable Data Structures: SFrame & SGr...
 
Mathias Brandewinder, Software Engineer & Data Scientist, Clear Lines Consult...
Mathias Brandewinder, Software Engineer & Data Scientist, Clear Lines Consult...Mathias Brandewinder, Software Engineer & Data Scientist, Clear Lines Consult...
Mathias Brandewinder, Software Engineer & Data Scientist, Clear Lines Consult...
 
Spark Workflow Management
Spark Workflow ManagementSpark Workflow Management
Spark Workflow Management
 
Scalable data structures for data science
Scalable data structures for data scienceScalable data structures for data science
Scalable data structures for data science
 
Gradoop: Scalable Graph Analytics with Apache Flink @ Flink Forward 2015
Gradoop: Scalable Graph Analytics with Apache Flink @ Flink Forward 2015Gradoop: Scalable Graph Analytics with Apache Flink @ Flink Forward 2015
Gradoop: Scalable Graph Analytics with Apache Flink @ Flink Forward 2015
 
VEGAS: The Missing Matplotlib for Scala/Apache Spark with DB Tsai and Roger M...
VEGAS: The Missing Matplotlib for Scala/Apache Spark with DB Tsai and Roger M...VEGAS: The Missing Matplotlib for Scala/Apache Spark with DB Tsai and Roger M...
VEGAS: The Missing Matplotlib for Scala/Apache Spark with DB Tsai and Roger M...
 
SFrame
SFrameSFrame
SFrame
 
Seattle Scalability Mahout
Seattle Scalability MahoutSeattle Scalability Mahout
Seattle Scalability Mahout
 

Similar to Tutorial: Implementing your first Postgres extension | PGConf EU 2019 | Burak Yucesoy, Onder Kalaci

Useful PostgreSQL Extensions
Useful PostgreSQL ExtensionsUseful PostgreSQL Extensions
Useful PostgreSQL ExtensionsEDB
 
PostgreSQL Performance Problems: Monitoring and Alerting
PostgreSQL Performance Problems: Monitoring and AlertingPostgreSQL Performance Problems: Monitoring and Alerting
PostgreSQL Performance Problems: Monitoring and AlertingGrant Fritchey
 
PostgreSQL Extension APIs are Changing the Face of Relational Databases | PGC...
PostgreSQL Extension APIs are Changing the Face of Relational Databases | PGC...PostgreSQL Extension APIs are Changing the Face of Relational Databases | PGC...
PostgreSQL Extension APIs are Changing the Face of Relational Databases | PGC...Teresa Giacomini
 
Machine Learning with ML.NET and Azure - Andy Cross
Machine Learning with ML.NET and Azure - Andy CrossMachine Learning with ML.NET and Azure - Andy Cross
Machine Learning with ML.NET and Azure - Andy CrossAndrew Flatters
 
AI與大數據數據處理 Spark實戰(20171216)
AI與大數據數據處理 Spark實戰(20171216)AI與大數據數據處理 Spark實戰(20171216)
AI與大數據數據處理 Spark實戰(20171216)Paul Chao
 
Agile Oracle to PostgreSQL migrations (PGConf.EU 2013)
Agile Oracle to PostgreSQL migrations (PGConf.EU 2013)Agile Oracle to PostgreSQL migrations (PGConf.EU 2013)
Agile Oracle to PostgreSQL migrations (PGConf.EU 2013)Gabriele Bartolini
 
Beyond Wordcount with spark datasets (and scalaing) - Nide PDX Jan 2018
Beyond Wordcount  with spark datasets (and scalaing) - Nide PDX Jan 2018Beyond Wordcount  with spark datasets (and scalaing) - Nide PDX Jan 2018
Beyond Wordcount with spark datasets (and scalaing) - Nide PDX Jan 2018Holden Karau
 
5 hs mpostcustomizationrenefonseca
5 hs mpostcustomizationrenefonseca5 hs mpostcustomizationrenefonseca
5 hs mpostcustomizationrenefonsecassuserfadb24
 
FluentMigrator - Dayton .NET - July 2023
FluentMigrator - Dayton .NET - July 2023FluentMigrator - Dayton .NET - July 2023
FluentMigrator - Dayton .NET - July 2023Matthew Groves
 
Migration From Oracle to PostgreSQL
Migration From Oracle to PostgreSQLMigration From Oracle to PostgreSQL
Migration From Oracle to PostgreSQLPGConf APAC
 
Introduction to Spark Datasets - Functional and relational together at last
Introduction to Spark Datasets - Functional and relational together at lastIntroduction to Spark Datasets - Functional and relational together at last
Introduction to Spark Datasets - Functional and relational together at lastHolden Karau
 
Big Data processing with Apache Spark
Big Data processing with Apache SparkBig Data processing with Apache Spark
Big Data processing with Apache SparkLucian Neghina
 
Anatomy of Data Frame API : A deep dive into Spark Data Frame API
Anatomy of Data Frame API :  A deep dive into Spark Data Frame APIAnatomy of Data Frame API :  A deep dive into Spark Data Frame API
Anatomy of Data Frame API : A deep dive into Spark Data Frame APIdatamantra
 
dotNet Miami - June 21, 2012: Richie Rump: Entity Framework: Code First and M...
dotNet Miami - June 21, 2012: Richie Rump: Entity Framework: Code First and M...dotNet Miami - June 21, 2012: Richie Rump: Entity Framework: Code First and M...
dotNet Miami - June 21, 2012: Richie Rump: Entity Framework: Code First and M...dotNet Miami
 
Entity Framework: Code First and Magic Unicorns
Entity Framework: Code First and Magic UnicornsEntity Framework: Code First and Magic Unicorns
Entity Framework: Code First and Magic UnicornsRichie Rump
 

Similar to Tutorial: Implementing your first Postgres extension | PGConf EU 2019 | Burak Yucesoy, Onder Kalaci (20)

Useful PostgreSQL Extensions
Useful PostgreSQL ExtensionsUseful PostgreSQL Extensions
Useful PostgreSQL Extensions
 
PostgreSQL Performance Problems: Monitoring and Alerting
PostgreSQL Performance Problems: Monitoring and AlertingPostgreSQL Performance Problems: Monitoring and Alerting
PostgreSQL Performance Problems: Monitoring and Alerting
 
PostgreSQL Extension APIs are Changing the Face of Relational Databases | PGC...
PostgreSQL Extension APIs are Changing the Face of Relational Databases | PGC...PostgreSQL Extension APIs are Changing the Face of Relational Databases | PGC...
PostgreSQL Extension APIs are Changing the Face of Relational Databases | PGC...
 
Machine Learning with ML.NET and Azure - Andy Cross
Machine Learning with ML.NET and Azure - Andy CrossMachine Learning with ML.NET and Azure - Andy Cross
Machine Learning with ML.NET and Azure - Andy Cross
 
An Introduction to Postgresql
An Introduction to PostgresqlAn Introduction to Postgresql
An Introduction to Postgresql
 
AI與大數據數據處理 Spark實戰(20171216)
AI與大數據數據處理 Spark實戰(20171216)AI與大數據數據處理 Spark實戰(20171216)
AI與大數據數據處理 Spark實戰(20171216)
 
Agile Oracle to PostgreSQL migrations (PGConf.EU 2013)
Agile Oracle to PostgreSQL migrations (PGConf.EU 2013)Agile Oracle to PostgreSQL migrations (PGConf.EU 2013)
Agile Oracle to PostgreSQL migrations (PGConf.EU 2013)
 
Beyond Wordcount with spark datasets (and scalaing) - Nide PDX Jan 2018
Beyond Wordcount  with spark datasets (and scalaing) - Nide PDX Jan 2018Beyond Wordcount  with spark datasets (and scalaing) - Nide PDX Jan 2018
Beyond Wordcount with spark datasets (and scalaing) - Nide PDX Jan 2018
 
dev_int_96
dev_int_96dev_int_96
dev_int_96
 
5 hs mpostcustomizationrenefonseca
5 hs mpostcustomizationrenefonseca5 hs mpostcustomizationrenefonseca
5 hs mpostcustomizationrenefonseca
 
FluentMigrator - Dayton .NET - July 2023
FluentMigrator - Dayton .NET - July 2023FluentMigrator - Dayton .NET - July 2023
FluentMigrator - Dayton .NET - July 2023
 
Migration From Oracle to PostgreSQL
Migration From Oracle to PostgreSQLMigration From Oracle to PostgreSQL
Migration From Oracle to PostgreSQL
 
Introduction to Spark Datasets - Functional and relational together at last
Introduction to Spark Datasets - Functional and relational together at lastIntroduction to Spark Datasets - Functional and relational together at last
Introduction to Spark Datasets - Functional and relational together at last
 
Big Data processing with Apache Spark
Big Data processing with Apache SparkBig Data processing with Apache Spark
Big Data processing with Apache Spark
 
Database programming
Database programmingDatabase programming
Database programming
 
Anatomy of Data Frame API : A deep dive into Spark Data Frame API
Anatomy of Data Frame API :  A deep dive into Spark Data Frame APIAnatomy of Data Frame API :  A deep dive into Spark Data Frame API
Anatomy of Data Frame API : A deep dive into Spark Data Frame API
 
dotNet Miami - June 21, 2012: Richie Rump: Entity Framework: Code First and M...
dotNet Miami - June 21, 2012: Richie Rump: Entity Framework: Code First and M...dotNet Miami - June 21, 2012: Richie Rump: Entity Framework: Code First and M...
dotNet Miami - June 21, 2012: Richie Rump: Entity Framework: Code First and M...
 
Oops lecture 1
Oops lecture 1Oops lecture 1
Oops lecture 1
 
Entity Framework: Code First and Magic Unicorns
Entity Framework: Code First and Magic UnicornsEntity Framework: Code First and Magic Unicorns
Entity Framework: Code First and Magic Unicorns
 
ProgrammingPrimerAndOOPS
ProgrammingPrimerAndOOPSProgrammingPrimerAndOOPS
ProgrammingPrimerAndOOPS
 

More from Citus Data

Architecting peta-byte-scale analytics by scaling out Postgres on Azure with ...
Architecting peta-byte-scale analytics by scaling out Postgres on Azure with ...Architecting peta-byte-scale analytics by scaling out Postgres on Azure with ...
Architecting peta-byte-scale analytics by scaling out Postgres on Azure with ...Citus Data
 
JSONB Tricks: Operators, Indexes, and When (Not) to Use It | PostgresOpen 201...
JSONB Tricks: Operators, Indexes, and When (Not) to Use It | PostgresOpen 201...JSONB Tricks: Operators, Indexes, and When (Not) to Use It | PostgresOpen 201...
JSONB Tricks: Operators, Indexes, and When (Not) to Use It | PostgresOpen 201...Citus Data
 
Whats wrong with postgres | PGConf EU 2019 | Craig Kerstiens
Whats wrong with postgres | PGConf EU 2019 | Craig KerstiensWhats wrong with postgres | PGConf EU 2019 | Craig Kerstiens
Whats wrong with postgres | PGConf EU 2019 | Craig KerstiensCitus Data
 
When it all goes wrong | PGConf EU 2019 | Will Leinweber
When it all goes wrong | PGConf EU 2019 | Will LeinweberWhen it all goes wrong | PGConf EU 2019 | Will Leinweber
When it all goes wrong | PGConf EU 2019 | Will LeinweberCitus Data
 
Amazing SQL your ORM can (or can't) do | PGConf EU 2019 | Louise Grandjonc
Amazing SQL your ORM can (or can't) do | PGConf EU 2019 | Louise GrandjoncAmazing SQL your ORM can (or can't) do | PGConf EU 2019 | Louise Grandjonc
Amazing SQL your ORM can (or can't) do | PGConf EU 2019 | Louise GrandjoncCitus Data
 
What Microsoft is doing with Postgres & the Citus Data acquisition | PGConf E...
What Microsoft is doing with Postgres & the Citus Data acquisition | PGConf E...What Microsoft is doing with Postgres & the Citus Data acquisition | PGConf E...
What Microsoft is doing with Postgres & the Citus Data acquisition | PGConf E...Citus Data
 
Why Postgres Why This Database Why Now | SF Bay Area Postgres Meetup | Claire...
Why Postgres Why This Database Why Now | SF Bay Area Postgres Meetup | Claire...Why Postgres Why This Database Why Now | SF Bay Area Postgres Meetup | Claire...
Why Postgres Why This Database Why Now | SF Bay Area Postgres Meetup | Claire...Citus Data
 
A story on Postgres index types | PostgresLondon 2019 | Louise Grandjonc
A story on Postgres index types | PostgresLondon 2019 | Louise GrandjoncA story on Postgres index types | PostgresLondon 2019 | Louise Grandjonc
A story on Postgres index types | PostgresLondon 2019 | Louise GrandjoncCitus Data
 
Why developers need marketing now more than ever | GlueCon 2019 | Claire Gior...
Why developers need marketing now more than ever | GlueCon 2019 | Claire Gior...Why developers need marketing now more than ever | GlueCon 2019 | Claire Gior...
Why developers need marketing now more than ever | GlueCon 2019 | Claire Gior...Citus Data
 
The Art of PostgreSQL | PostgreSQL Ukraine | Dimitri Fontaine
The Art of PostgreSQL | PostgreSQL Ukraine | Dimitri FontaineThe Art of PostgreSQL | PostgreSQL Ukraine | Dimitri Fontaine
The Art of PostgreSQL | PostgreSQL Ukraine | Dimitri FontaineCitus Data
 
When it all goes wrong (with Postgres) | RailsConf 2019 | Will Leinweber
When it all goes wrong (with Postgres) | RailsConf 2019 | Will LeinweberWhen it all goes wrong (with Postgres) | RailsConf 2019 | Will Leinweber
When it all goes wrong (with Postgres) | RailsConf 2019 | Will LeinweberCitus Data
 
The Art of PostgreSQL | PostgreSQL Ukraine Meetup | Dimitri Fontaine
The Art of PostgreSQL | PostgreSQL Ukraine Meetup | Dimitri FontaineThe Art of PostgreSQL | PostgreSQL Ukraine Meetup | Dimitri Fontaine
The Art of PostgreSQL | PostgreSQL Ukraine Meetup | Dimitri FontaineCitus Data
 
Using Postgres and Citus for Lightning Fast Analytics, also ft. Rollups | Liv...
Using Postgres and Citus for Lightning Fast Analytics, also ft. Rollups | Liv...Using Postgres and Citus for Lightning Fast Analytics, also ft. Rollups | Liv...
Using Postgres and Citus for Lightning Fast Analytics, also ft. Rollups | Liv...Citus Data
 
How to write SQL queries | pgDay Paris 2019 | Dimitri Fontaine
How to write SQL queries | pgDay Paris 2019 | Dimitri FontaineHow to write SQL queries | pgDay Paris 2019 | Dimitri Fontaine
How to write SQL queries | pgDay Paris 2019 | Dimitri FontaineCitus Data
 
When it all Goes Wrong |Nordic PGDay 2019 | Will Leinweber
When it all Goes Wrong |Nordic PGDay 2019 | Will LeinweberWhen it all Goes Wrong |Nordic PGDay 2019 | Will Leinweber
When it all Goes Wrong |Nordic PGDay 2019 | Will LeinweberCitus Data
 
Why PostgreSQL Why This Database Why Now | Nordic PGDay 2019 | Claire Giordano
Why PostgreSQL Why This Database Why Now | Nordic PGDay 2019 | Claire GiordanoWhy PostgreSQL Why This Database Why Now | Nordic PGDay 2019 | Claire Giordano
Why PostgreSQL Why This Database Why Now | Nordic PGDay 2019 | Claire GiordanoCitus Data
 
Scaling Multi-Tenant Applications Using the Django ORM & Postgres | PyCaribbe...
Scaling Multi-Tenant Applications Using the Django ORM & Postgres | PyCaribbe...Scaling Multi-Tenant Applications Using the Django ORM & Postgres | PyCaribbe...
Scaling Multi-Tenant Applications Using the Django ORM & Postgres | PyCaribbe...Citus Data
 
Data Modeling, Normalization, and Denormalisation | FOSDEM '19 | Dimitri Font...
Data Modeling, Normalization, and Denormalisation | FOSDEM '19 | Dimitri Font...Data Modeling, Normalization, and Denormalisation | FOSDEM '19 | Dimitri Font...
Data Modeling, Normalization, and Denormalisation | FOSDEM '19 | Dimitri Font...Citus Data
 
Five data models for sharding and which is right | PGConf.ASIA 2018 | Craig K...
Five data models for sharding and which is right | PGConf.ASIA 2018 | Craig K...Five data models for sharding and which is right | PGConf.ASIA 2018 | Craig K...
Five data models for sharding and which is right | PGConf.ASIA 2018 | Craig K...Citus Data
 
Monitoring Postgres at Scale | PGConf.ASIA 2018 | Lukas Fittl
Monitoring Postgres at Scale | PGConf.ASIA 2018 | Lukas FittlMonitoring Postgres at Scale | PGConf.ASIA 2018 | Lukas Fittl
Monitoring Postgres at Scale | PGConf.ASIA 2018 | Lukas FittlCitus Data
 

More from Citus Data (20)

Architecting peta-byte-scale analytics by scaling out Postgres on Azure with ...
Architecting peta-byte-scale analytics by scaling out Postgres on Azure with ...Architecting peta-byte-scale analytics by scaling out Postgres on Azure with ...
Architecting peta-byte-scale analytics by scaling out Postgres on Azure with ...
 
JSONB Tricks: Operators, Indexes, and When (Not) to Use It | PostgresOpen 201...
JSONB Tricks: Operators, Indexes, and When (Not) to Use It | PostgresOpen 201...JSONB Tricks: Operators, Indexes, and When (Not) to Use It | PostgresOpen 201...
JSONB Tricks: Operators, Indexes, and When (Not) to Use It | PostgresOpen 201...
 
Whats wrong with postgres | PGConf EU 2019 | Craig Kerstiens
Whats wrong with postgres | PGConf EU 2019 | Craig KerstiensWhats wrong with postgres | PGConf EU 2019 | Craig Kerstiens
Whats wrong with postgres | PGConf EU 2019 | Craig Kerstiens
 
When it all goes wrong | PGConf EU 2019 | Will Leinweber
When it all goes wrong | PGConf EU 2019 | Will LeinweberWhen it all goes wrong | PGConf EU 2019 | Will Leinweber
When it all goes wrong | PGConf EU 2019 | Will Leinweber
 
Amazing SQL your ORM can (or can't) do | PGConf EU 2019 | Louise Grandjonc
Amazing SQL your ORM can (or can't) do | PGConf EU 2019 | Louise GrandjoncAmazing SQL your ORM can (or can't) do | PGConf EU 2019 | Louise Grandjonc
Amazing SQL your ORM can (or can't) do | PGConf EU 2019 | Louise Grandjonc
 
What Microsoft is doing with Postgres & the Citus Data acquisition | PGConf E...
What Microsoft is doing with Postgres & the Citus Data acquisition | PGConf E...What Microsoft is doing with Postgres & the Citus Data acquisition | PGConf E...
What Microsoft is doing with Postgres & the Citus Data acquisition | PGConf E...
 
Why Postgres Why This Database Why Now | SF Bay Area Postgres Meetup | Claire...
Why Postgres Why This Database Why Now | SF Bay Area Postgres Meetup | Claire...Why Postgres Why This Database Why Now | SF Bay Area Postgres Meetup | Claire...
Why Postgres Why This Database Why Now | SF Bay Area Postgres Meetup | Claire...
 
A story on Postgres index types | PostgresLondon 2019 | Louise Grandjonc
A story on Postgres index types | PostgresLondon 2019 | Louise GrandjoncA story on Postgres index types | PostgresLondon 2019 | Louise Grandjonc
A story on Postgres index types | PostgresLondon 2019 | Louise Grandjonc
 
Why developers need marketing now more than ever | GlueCon 2019 | Claire Gior...
Why developers need marketing now more than ever | GlueCon 2019 | Claire Gior...Why developers need marketing now more than ever | GlueCon 2019 | Claire Gior...
Why developers need marketing now more than ever | GlueCon 2019 | Claire Gior...
 
The Art of PostgreSQL | PostgreSQL Ukraine | Dimitri Fontaine
The Art of PostgreSQL | PostgreSQL Ukraine | Dimitri FontaineThe Art of PostgreSQL | PostgreSQL Ukraine | Dimitri Fontaine
The Art of PostgreSQL | PostgreSQL Ukraine | Dimitri Fontaine
 
When it all goes wrong (with Postgres) | RailsConf 2019 | Will Leinweber
When it all goes wrong (with Postgres) | RailsConf 2019 | Will LeinweberWhen it all goes wrong (with Postgres) | RailsConf 2019 | Will Leinweber
When it all goes wrong (with Postgres) | RailsConf 2019 | Will Leinweber
 
The Art of PostgreSQL | PostgreSQL Ukraine Meetup | Dimitri Fontaine
The Art of PostgreSQL | PostgreSQL Ukraine Meetup | Dimitri FontaineThe Art of PostgreSQL | PostgreSQL Ukraine Meetup | Dimitri Fontaine
The Art of PostgreSQL | PostgreSQL Ukraine Meetup | Dimitri Fontaine
 
Using Postgres and Citus for Lightning Fast Analytics, also ft. Rollups | Liv...
Using Postgres and Citus for Lightning Fast Analytics, also ft. Rollups | Liv...Using Postgres and Citus for Lightning Fast Analytics, also ft. Rollups | Liv...
Using Postgres and Citus for Lightning Fast Analytics, also ft. Rollups | Liv...
 
How to write SQL queries | pgDay Paris 2019 | Dimitri Fontaine
How to write SQL queries | pgDay Paris 2019 | Dimitri FontaineHow to write SQL queries | pgDay Paris 2019 | Dimitri Fontaine
How to write SQL queries | pgDay Paris 2019 | Dimitri Fontaine
 
When it all Goes Wrong |Nordic PGDay 2019 | Will Leinweber
When it all Goes Wrong |Nordic PGDay 2019 | Will LeinweberWhen it all Goes Wrong |Nordic PGDay 2019 | Will Leinweber
When it all Goes Wrong |Nordic PGDay 2019 | Will Leinweber
 
Why PostgreSQL Why This Database Why Now | Nordic PGDay 2019 | Claire Giordano
Why PostgreSQL Why This Database Why Now | Nordic PGDay 2019 | Claire GiordanoWhy PostgreSQL Why This Database Why Now | Nordic PGDay 2019 | Claire Giordano
Why PostgreSQL Why This Database Why Now | Nordic PGDay 2019 | Claire Giordano
 
Scaling Multi-Tenant Applications Using the Django ORM & Postgres | PyCaribbe...
Scaling Multi-Tenant Applications Using the Django ORM & Postgres | PyCaribbe...Scaling Multi-Tenant Applications Using the Django ORM & Postgres | PyCaribbe...
Scaling Multi-Tenant Applications Using the Django ORM & Postgres | PyCaribbe...
 
Data Modeling, Normalization, and Denormalisation | FOSDEM '19 | Dimitri Font...
Data Modeling, Normalization, and Denormalisation | FOSDEM '19 | Dimitri Font...Data Modeling, Normalization, and Denormalisation | FOSDEM '19 | Dimitri Font...
Data Modeling, Normalization, and Denormalisation | FOSDEM '19 | Dimitri Font...
 
Five data models for sharding and which is right | PGConf.ASIA 2018 | Craig K...
Five data models for sharding and which is right | PGConf.ASIA 2018 | Craig K...Five data models for sharding and which is right | PGConf.ASIA 2018 | Craig K...
Five data models for sharding and which is right | PGConf.ASIA 2018 | Craig K...
 
Monitoring Postgres at Scale | PGConf.ASIA 2018 | Lukas Fittl
Monitoring Postgres at Scale | PGConf.ASIA 2018 | Lukas FittlMonitoring Postgres at Scale | PGConf.ASIA 2018 | Lukas Fittl
Monitoring Postgres at Scale | PGConf.ASIA 2018 | Lukas Fittl
 

Recently uploaded

Moving Beyond Passwords: FIDO Paris Seminar.pdf
Moving Beyond Passwords: FIDO Paris Seminar.pdfMoving Beyond Passwords: FIDO Paris Seminar.pdf
Moving Beyond Passwords: FIDO Paris Seminar.pdfLoriGlavin3
 
The State of Passkeys with FIDO Alliance.pptx
The State of Passkeys with FIDO Alliance.pptxThe State of Passkeys with FIDO Alliance.pptx
The State of Passkeys with FIDO Alliance.pptxLoriGlavin3
 
SIP trunking in Janus @ Kamailio World 2024
SIP trunking in Janus @ Kamailio World 2024SIP trunking in Janus @ Kamailio World 2024
SIP trunking in Janus @ Kamailio World 2024Lorenzo Miniero
 
Gen AI in Business - Global Trends Report 2024.pdf
Gen AI in Business - Global Trends Report 2024.pdfGen AI in Business - Global Trends Report 2024.pdf
Gen AI in Business - Global Trends Report 2024.pdfAddepto
 
Dev Dives: Streamline document processing with UiPath Studio Web
Dev Dives: Streamline document processing with UiPath Studio WebDev Dives: Streamline document processing with UiPath Studio Web
Dev Dives: Streamline document processing with UiPath Studio WebUiPathCommunity
 
Transcript: New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024
Transcript: New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024Transcript: New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024
Transcript: New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024BookNet Canada
 
From Family Reminiscence to Scholarly Archive .
From Family Reminiscence to Scholarly Archive .From Family Reminiscence to Scholarly Archive .
From Family Reminiscence to Scholarly Archive .Alan Dix
 
The Role of FIDO in a Cyber Secure Netherlands: FIDO Paris Seminar.pptx
The Role of FIDO in a Cyber Secure Netherlands: FIDO Paris Seminar.pptxThe Role of FIDO in a Cyber Secure Netherlands: FIDO Paris Seminar.pptx
The Role of FIDO in a Cyber Secure Netherlands: FIDO Paris Seminar.pptxLoriGlavin3
 
Digital Identity is Under Attack: FIDO Paris Seminar.pptx
Digital Identity is Under Attack: FIDO Paris Seminar.pptxDigital Identity is Under Attack: FIDO Paris Seminar.pptx
Digital Identity is Under Attack: FIDO Paris Seminar.pptxLoriGlavin3
 
Ensuring Technical Readiness For Copilot in Microsoft 365
Ensuring Technical Readiness For Copilot in Microsoft 365Ensuring Technical Readiness For Copilot in Microsoft 365
Ensuring Technical Readiness For Copilot in Microsoft 3652toLead Limited
 
What is DBT - The Ultimate Data Build Tool.pdf
What is DBT - The Ultimate Data Build Tool.pdfWhat is DBT - The Ultimate Data Build Tool.pdf
What is DBT - The Ultimate Data Build Tool.pdfMounikaPolabathina
 
The Ultimate Guide to Choosing WordPress Pros and Cons
The Ultimate Guide to Choosing WordPress Pros and ConsThe Ultimate Guide to Choosing WordPress Pros and Cons
The Ultimate Guide to Choosing WordPress Pros and ConsPixlogix Infotech
 
What's New in Teams Calling, Meetings and Devices March 2024
What's New in Teams Calling, Meetings and Devices March 2024What's New in Teams Calling, Meetings and Devices March 2024
What's New in Teams Calling, Meetings and Devices March 2024Stephanie Beckett
 
Time Series Foundation Models - current state and future directions
Time Series Foundation Models - current state and future directionsTime Series Foundation Models - current state and future directions
Time Series Foundation Models - current state and future directionsNathaniel Shimoni
 
New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024
New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024
New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024BookNet Canada
 
Rise of the Machines: Known As Drones...
Rise of the Machines: Known As Drones...Rise of the Machines: Known As Drones...
Rise of the Machines: Known As Drones...Rick Flair
 
WordPress Websites for Engineers: Elevate Your Brand
WordPress Websites for Engineers: Elevate Your BrandWordPress Websites for Engineers: Elevate Your Brand
WordPress Websites for Engineers: Elevate Your Brandgvaughan
 
"ML in Production",Oleksandr Bagan
"ML in Production",Oleksandr Bagan"ML in Production",Oleksandr Bagan
"ML in Production",Oleksandr BaganFwdays
 
The Fit for Passkeys for Employee and Consumer Sign-ins: FIDO Paris Seminar.pptx
The Fit for Passkeys for Employee and Consumer Sign-ins: FIDO Paris Seminar.pptxThe Fit for Passkeys for Employee and Consumer Sign-ins: FIDO Paris Seminar.pptx
The Fit for Passkeys for Employee and Consumer Sign-ins: FIDO Paris Seminar.pptxLoriGlavin3
 
Scale your database traffic with Read & Write split using MySQL Router
Scale your database traffic with Read & Write split using MySQL RouterScale your database traffic with Read & Write split using MySQL Router
Scale your database traffic with Read & Write split using MySQL RouterMydbops
 

Recently uploaded (20)

Moving Beyond Passwords: FIDO Paris Seminar.pdf
Moving Beyond Passwords: FIDO Paris Seminar.pdfMoving Beyond Passwords: FIDO Paris Seminar.pdf
Moving Beyond Passwords: FIDO Paris Seminar.pdf
 
The State of Passkeys with FIDO Alliance.pptx
The State of Passkeys with FIDO Alliance.pptxThe State of Passkeys with FIDO Alliance.pptx
The State of Passkeys with FIDO Alliance.pptx
 
SIP trunking in Janus @ Kamailio World 2024
SIP trunking in Janus @ Kamailio World 2024SIP trunking in Janus @ Kamailio World 2024
SIP trunking in Janus @ Kamailio World 2024
 
Gen AI in Business - Global Trends Report 2024.pdf
Gen AI in Business - Global Trends Report 2024.pdfGen AI in Business - Global Trends Report 2024.pdf
Gen AI in Business - Global Trends Report 2024.pdf
 
Dev Dives: Streamline document processing with UiPath Studio Web
Dev Dives: Streamline document processing with UiPath Studio WebDev Dives: Streamline document processing with UiPath Studio Web
Dev Dives: Streamline document processing with UiPath Studio Web
 
Transcript: New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024
Transcript: New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024Transcript: New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024
Transcript: New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024
 
From Family Reminiscence to Scholarly Archive .
From Family Reminiscence to Scholarly Archive .From Family Reminiscence to Scholarly Archive .
From Family Reminiscence to Scholarly Archive .
 
The Role of FIDO in a Cyber Secure Netherlands: FIDO Paris Seminar.pptx
The Role of FIDO in a Cyber Secure Netherlands: FIDO Paris Seminar.pptxThe Role of FIDO in a Cyber Secure Netherlands: FIDO Paris Seminar.pptx
The Role of FIDO in a Cyber Secure Netherlands: FIDO Paris Seminar.pptx
 
Digital Identity is Under Attack: FIDO Paris Seminar.pptx
Digital Identity is Under Attack: FIDO Paris Seminar.pptxDigital Identity is Under Attack: FIDO Paris Seminar.pptx
Digital Identity is Under Attack: FIDO Paris Seminar.pptx
 
Ensuring Technical Readiness For Copilot in Microsoft 365
Ensuring Technical Readiness For Copilot in Microsoft 365Ensuring Technical Readiness For Copilot in Microsoft 365
Ensuring Technical Readiness For Copilot in Microsoft 365
 
What is DBT - The Ultimate Data Build Tool.pdf
What is DBT - The Ultimate Data Build Tool.pdfWhat is DBT - The Ultimate Data Build Tool.pdf
What is DBT - The Ultimate Data Build Tool.pdf
 
The Ultimate Guide to Choosing WordPress Pros and Cons
The Ultimate Guide to Choosing WordPress Pros and ConsThe Ultimate Guide to Choosing WordPress Pros and Cons
The Ultimate Guide to Choosing WordPress Pros and Cons
 
What's New in Teams Calling, Meetings and Devices March 2024
What's New in Teams Calling, Meetings and Devices March 2024What's New in Teams Calling, Meetings and Devices March 2024
What's New in Teams Calling, Meetings and Devices March 2024
 
Time Series Foundation Models - current state and future directions
Time Series Foundation Models - current state and future directionsTime Series Foundation Models - current state and future directions
Time Series Foundation Models - current state and future directions
 
New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024
New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024
New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024
 
Rise of the Machines: Known As Drones...
Rise of the Machines: Known As Drones...Rise of the Machines: Known As Drones...
Rise of the Machines: Known As Drones...
 
WordPress Websites for Engineers: Elevate Your Brand
WordPress Websites for Engineers: Elevate Your BrandWordPress Websites for Engineers: Elevate Your Brand
WordPress Websites for Engineers: Elevate Your Brand
 
"ML in Production",Oleksandr Bagan
"ML in Production",Oleksandr Bagan"ML in Production",Oleksandr Bagan
"ML in Production",Oleksandr Bagan
 
The Fit for Passkeys for Employee and Consumer Sign-ins: FIDO Paris Seminar.pptx
The Fit for Passkeys for Employee and Consumer Sign-ins: FIDO Paris Seminar.pptxThe Fit for Passkeys for Employee and Consumer Sign-ins: FIDO Paris Seminar.pptx
The Fit for Passkeys for Employee and Consumer Sign-ins: FIDO Paris Seminar.pptx
 
Scale your database traffic with Read & Write split using MySQL Router
Scale your database traffic with Read & Write split using MySQL RouterScale your database traffic with Read & Write split using MySQL Router
Scale your database traffic with Read & Write split using MySQL Router
 

Tutorial: Implementing your first Postgres extension | PGConf EU 2019 | Burak Yucesoy, Onder Kalaci

  • 1. Implementing your first PostgreSQL extension: From Coding to Distribution Burak Yücesoy Önder Kalacı
  • 2. Who are we? • Two engineers from Microsoft. • Used to work for Citus Data (now acquired by Microsoft). • Earn our lives by developing PostgreSQL extensions.
  • 4. Outline What is PostgreSQL Extension? Why PostgreSQL is extendable? What is extendable in PostgreSQL? •Developing •Debugging •Testing •Packaging •Releasing Life cycle of PostgreSQL extension development
  • 5. What is PostgreSQL Extension? • A PostgreSQL extension is a piece of software that adds functionality to Postgres. Each extension bundles related objects together. • Postgres 9.1 started providing official APIs to override or extend any database module’s behavior. • “CREATE EXTENSION extension_name;” dynamically loads these objects into Postgres’ address space.
  • 6. Example Extension: pg_cron • A task scheduler inside the database; it allows to perform periodic jobs on the database. • Every Saturday at 03:30; delete old data  psql> SELECT cron.schedule('30 3 * * 6', $$DELETE FROM events WHERE event_time < now() - interval '1 week'$$); • Every day at 10:00, run VACUUM  psql> SELECT cron.schedule('0 10 * * *', $$VACUUM$$);
  • 7. Why PostgreSQL is Extendable? • Every decade brings new workloads for databases. • The last decade was about capturing more data, in more shapes and form. • Postgres has been forked by dozens of commercial databases for new workloads. When you fork, your database diverges from the community. • What if you could leverage the database ecosystem and grow with it?
  • 8. What is Extendable in PostgreSQL? You can override, cooperate with, or extend any combination of the following database modules:  Type system and operators  User defined functions and aggregates  Storage system and indexes  Write ahead logging and replication  Transaction engine  Background worker processes  Query planner and query executor  Configuration and database metadata
  • 9. Type system and operators • PostgreSQL already has lots of different data types;  bigint, text, timestampz, jsonb, … • If you need a data type which doesn’t exist in PostgreSQL;  You can define new type with CREATE TYPE command.  You can add the types created by other people using extensions • Some additional data types;  ip-address, e-mail
  • 10. Type system and operators
  • 11. Type system and operators • It is also possible to define (or overwrite) operators for the types you created. • For example > or < operators can be meaningful for ip address data type. • Or you can come up with completely new operator such as;  Distance function for points; Point <-> Point  Membership of point in Sphere; Point <.> Sphere
  • 12. User defined functions and aggregates • You can create new function or aggregate using CREATE FUNCTION command. • If you are performing some operations frequently it may make sense to implement them as function. • Also if you defined new type, you can also create the functions to perform specific things on the type you create.
  • 13. User defined functions and aggregates
  • 14. What is Extendable in PostgreSQL? You can override, cooperate with, or extend any combination of the following database modules:  Type system and operators  User defined functions and aggregates  Storage system and indexes  Write ahead logging and replication  Transaction engine  Background worker processes  Query planner and query executor  Configuration and database metadata Can be done in SQL Needs to be done in lower lever like C
  • 17. “Every problem is a gift. Without them we wouldn’t grow” - Tony Robbins
  • 18.
  • 19. Today’s Schedule for Extension Development • Today we will follow several steps to create a working prototype to represent color information in database;  Primitive approach with using text  Composite type  UDFs and operators for our type  C level implementation  Creation of custom nodes for our type  Modifications to executor and planner • You can follow the development from this repo
  • 20. PostgreSQL Internals: Datums • Datums are PostgreSQL’s way of representing single value. • Values are passed to or from PostgreSQL as Datums. • It encapsulates the actual value and provides generic interface for all kinds of value types. • The code using the Datum has to know which type it is, since the Datum itself doesn't contain that information. • Conversion to and from Datum is made by helper functions;  Int32GetDatum(int): Converts int value to Datum  DatumGet32Int(Datum): Gets the int value stored in Datum
  • 21. PostgreSQL Internals: Tuple • Tuples have many different use-cases but most importantly are representation of rows in the database. • They are made up from Datums. • Interactions with tuples are made by macros defined in PostgreSQL codebase
  • 22. PostgreSQL Internals: Memory Context • All memory allocations is handled by various memory contexts. • You need to allocate memory by palloc() function instead of standard malloc(). • There are multiple memory contexts with different lifetimes;  TopMemoryContext  CacheMemoryContext  MessageMemoryContext • More information at: https://github.com/postgres/postgres/blob/master/src/backend/utils/mmgr/README
  • 23. PostgreSQL Internals: Error Reporting • elog and ereport functions are used for error reporting. • They are used to print user visible error messages, but more importantly;  They rollback open transaction  They release any allocated memory for the transaction/queries in related memory contexts. • It is even possible to extend the way error messages are handled in PostgreSQL • More information at: https://github.com/postgres/postgres/blob/master/src/backend/utils/error/elog. c#L3
  • 24. PostgreSQL Internals: Node • PostgreSQL creates a query text to a query tree. • Query tree is made up from nodes. • Each node has a type and related data in it. • It is possible to create your own node types. • More information: https://github.com/postgres/postgres/blob/master/src/backend/nodes/READM E
  • 25. Let’s Implement An Equality Function Datum color_eq(PG_FUNCTION_ARGS) { }
  • 26. Let’s Implement An Equality Function Datum color_eq(PG_FUNCTION_ARGS) { color *c1 = PG_GETARG_COLOR(0); }
  • 27. Let’s Implement An Equality Function Datum color_eq(PG_FUNCTION_ARGS) { color *c1 = PG_GETARG_COLOR(0); color *c2 = PG_GETARG_COLOR(1); }
  • 28. Let’s Implement An Equality Function Datum color_eq(PG_FUNCTION_ARGS) { color *c1 = PG_GETARG_COLOR(0); color *c2 = PG_GETARG_COLOR(1); return c1->r == c2->r && c1->g == c2->g && c1->b == c2->b; }
  • 29. Let’s Implement An Equality Function V2 static bool EqualPgColorExtendedNode( const struct ExtensibleNode *target_node, const struct ExtensibleNode *source_node) { }
  • 30. Let’s Implement An Equality Function V2 static bool EqualPgColorExtendedNode( const struct ExtensibleNode *target_node, const struct ExtensibleNode *source_node) { PgColorExtendedNode *targetPlan = (PgColorExtendedNode *) target_node; }
  • 31. Let’s Implement An Equality Function V2 static bool EqualPgColorExtendedNode( const struct ExtensibleNode *target_node, const struct ExtensibleNode *source_node) { PgColorExtendedNode *targetPlan = (PgColorExtendedNode *) target_node; PgColorExtendedNode *sourcePlan = (PgColorExtendedNode *) source_node; }
  • 32. Let’s Implement An Equality Function V2 static bool EqualPgColorExtendedNode( const struct ExtensibleNode *target_node, const struct ExtensibleNode *source_node) { PgColorExtendedNode *targetPlan = (PgColorExtendedNode *) target_node; PgColorExtendedNode *sourcePlan = (PgColorExtendedNode *) source_node; return targetPlan->interceptedColor->r == sourcePlan->interceptedColor->r && targetPlan->interceptedColor->g == sourcePlan->interceptedColor->g && targetPlan->interceptedColor->b == sourcePlan->interceptedColor->b; }
  • 33. Testing • Extensions can use PostgreSQL's own testing suite • Make check and make installcheck • Runs queries against database and compare the output
  • 34. Packaging • PostgreSQL is commonly used in RedHat and Debian bases operation systems. • For each operating system you want to run your extension on, you need generate binaries in that particular system • Docker is life saver • Our open source packaging tools; https://github.com/citusdata/packaging
  • 35. Packaging Minimum requirements; • debian/pgversions • debian/control.in • debian/changelog • debian/copyright • debian/rules • debian/compat
  • 36. Packaging • > pg_buildext updatecontrol • > debuild -uc -us -B --lintian-opts --profile debian --allow-root
  • 37. Releasing • PostgreSQL community software repositories • PGXN • Your own package repository;  You can install your package repository to a server and respond install requests from that server.  Managed services; packagecloud.io