The document discusses database tools and techniques used by Skype to optimize database performance. It describes:
1) Skype's use of stored procedures to access databases for security and optimization reasons.
2) Tools like pgBouncer and PL/Proxy that minimize connections and allow remote queries to optimize resource usage and enable high availability.
3) The SkyTools framework for batch jobs that handles common tasks to simplify application development.
4) The PgQ queue system for asynchronous processing that provides transactional, efficient message handling between producers and consumers.
Connection Pooling in PostgreSQL using pgbouncer Sameer Kumar
The presentation was presented at 5th Postgres User Group, Singapore.
It explain how to setup pgbouncer and also shows a few demonstration graphs comparing the advantages/gains in performance when using pgbouncer instead of direct connections to PostgreSQL database.
This document discusses scaling PostgreSQL databases using open-source tools from Skype including pgBouncer, PGQ, Londiste, and PL/Proxy. It describes how these tools can be used to improve concurrency, add read replicas, replicate data changes, and shard queries across multiple databases. Configuration files and SQL functions are shown as examples of setting up and using these tools to horizontally scale a PostgreSQL deployment.
The document provides an overview of PostgreSQL performance tuning. It discusses caching, query processing internals, and optimization of storage and memory usage. Specific topics covered include the PostgreSQL configuration parameters for tuning shared buffers, work memory, and free space map settings.
This document provides instructions for setting up hot standby replication between a primary and secondary PostgreSQL database. It describes configuring WAL archiving on the primary, taking a backup of the primary to initialize the secondary, creating a recovery.conf file on the secondary, and testing replication. It also explains how to trigger a failover, switchover, and rebuild the primary database after a failover.
Building tungsten-clusters-with-postgre sql-hot-standby-and-streaming-replica...Command Prompt., Inc
Alex Alexander & Linas Virbalas
Hot standby and streaming replication will move the needle forward for high availability and scaling for a wide number of applications. Tungsten already supports clustering using warm standby. In this talk we will describe how to build clusters using the new PostgreSQL features and give our report from the trenches.
This talk will cover how hot standby and streaming replication work from a user perspective, then dive into a description of how to use them, taking Tungsten as an example. We'll cover the following issues:
* Configuration of warm standby and streaming replication
* Provisioning new standby instances
* Strategies for balancing reads across primary and standby database
* Managing failover
* Troubleshooting and gotchas
Please join us for an enlightening discussion a set of PostgreSQL features that are interesting to a wide range of PostgreSQL users.
Connection Pooling in PostgreSQL using pgbouncer Sameer Kumar
The presentation was presented at 5th Postgres User Group, Singapore.
It explain how to setup pgbouncer and also shows a few demonstration graphs comparing the advantages/gains in performance when using pgbouncer instead of direct connections to PostgreSQL database.
This document discusses scaling PostgreSQL databases using open-source tools from Skype including pgBouncer, PGQ, Londiste, and PL/Proxy. It describes how these tools can be used to improve concurrency, add read replicas, replicate data changes, and shard queries across multiple databases. Configuration files and SQL functions are shown as examples of setting up and using these tools to horizontally scale a PostgreSQL deployment.
The document provides an overview of PostgreSQL performance tuning. It discusses caching, query processing internals, and optimization of storage and memory usage. Specific topics covered include the PostgreSQL configuration parameters for tuning shared buffers, work memory, and free space map settings.
This document provides instructions for setting up hot standby replication between a primary and secondary PostgreSQL database. It describes configuring WAL archiving on the primary, taking a backup of the primary to initialize the secondary, creating a recovery.conf file on the secondary, and testing replication. It also explains how to trigger a failover, switchover, and rebuild the primary database after a failover.
Building tungsten-clusters-with-postgre sql-hot-standby-and-streaming-replica...Command Prompt., Inc
Alex Alexander & Linas Virbalas
Hot standby and streaming replication will move the needle forward for high availability and scaling for a wide number of applications. Tungsten already supports clustering using warm standby. In this talk we will describe how to build clusters using the new PostgreSQL features and give our report from the trenches.
This talk will cover how hot standby and streaming replication work from a user perspective, then dive into a description of how to use them, taking Tungsten as an example. We'll cover the following issues:
* Configuration of warm standby and streaming replication
* Provisioning new standby instances
* Strategies for balancing reads across primary and standby database
* Managing failover
* Troubleshooting and gotchas
Please join us for an enlightening discussion a set of PostgreSQL features that are interesting to a wide range of PostgreSQL users.
The document discusses various techniques for performance tuning and cluster administration in HBase, including garbage collection tuning, use of memstore-local allocation buffers (MSLAB), enabling compression, optimizing splits and compactions through pre-splitting regions, and addressing hotspotting through manual splits. It provides guidance on configuring garbage collection, compression codecs, and approaches for managing splits and compactions to reduce disk I/O loads.
Out of the box replication in postgres 9.4(pg confus)Denish Patel
This document contains notes from a presentation on PostgreSQL replication. It discusses write-ahead logs (WAL), replication history in PostgreSQL from versions 7.0 to 9.4, how to set up basic replication, tools for backups and monitoring replication, and demonstrates setting up replication without third party tools using pg_basebackup, replication slots, and pg_receivexlog. It also includes contact information for the presenter and an invitation to join the PostgreSQL Slack channel.
Streaming Replication Made Easy in v9.3Sameer Kumar
This document discusses setting up streaming replication in PostgreSQL v9.3 to enable high availability. It covers preparing primary and standby servers, configuring wal_level and max_wal_senders on the primary, taking a backup and restoring on the standby, creating a recovery.conf file, starting the servers to test replication, triggering failover by promoting the standby, handling multiple replicas without rebuilding, and rebuilding the original primary as a new standby. Monitoring replication status is also addressed using views like pg_stat_replication.
PostgreSQL Hangout Replication Features v9.4Ashnikbiz
See the new enhancements in v9.4 which takes away the pain of guessing right wal_keep_segment
See the new time lagging replication capability in v9.4
Short intro to logical replication introduced in v9.4
Background Tasks in Node - Evan Tahler, TaskRabbitRedis Labs
The part of the talk discusses various strategies for handling background tasks in Node.js, including:
1. Foreground tasks, which process tasks inline but can slow down response times.
2. Parallel tasks, which ignore callbacks to try and speed up response times but lose error handling.
3. Local messages using clustering, which uses inter-process communication to pass tasks to worker processes and provides error handling but is limited to one server.
4. Remote messages using Redis pub/sub, which allows distributing tasks across multiple servers but limits server types to one.
5. Remote queue using Node-Resque, which treats tasks as jobs in a Redis queue for scheduling across worker processes, providing
Basics of Logical Replication,Streaming replication vs Logical Replication ,U...Rajni Baliyan
This document provides an overview of logical replication in PostgreSQL 10. It discusses the history of PostgreSQL replication, the key concepts of logical replication including publications, subscriptions, and replication slots. It presents several use cases for logical replication such as cross-platform, cross-version, and write operations at the subscriber. The document also covers configuration settings, a quick setup demonstration, monitoring, resolving conflicts, and limitations of logical replication.
A meticulous presentation on Authorization, Encryption & Authentication of the security features in MySQL 8.0 by Vignesh Prabhu, Database reliability engineer, Mydbops.
Out of the box replication in postgres 9.4Denish Patel
This document provides an overview of setting up out of the box replication in PostgreSQL 9.4 without third party tools. It discusses write-ahead logs (WAL), replication slots, pg_basebackup, and pg_receivexlog. The document then demonstrates setting up replication on VMs with pg_basebackup to initialize a standby server, configuration of primary and standby servers, and monitoring of replication.
The document provides configuration instructions and guidelines for setting up streaming replication between a PostgreSQL master and standby server, including setting parameter values for wal_level, max_wal_senders, wal_keep_segments, creating a dedicated replication role, using pg_basebackup to initialize the standby, and various recovery target options to control the standby's behavior. It also discusses synchronous replication using replication slots and monitoring the replication process on both the master and standby servers.
Denish Patel deployed PostgreSQL on Amazon EC2 for a startup that had its entire IT architecture on the Amazon cloud. The initial deployment involved setting up master-slave configurations across two EC2 environments with issues like weekly instance failures and lack of monitoring. Patel then consolidated the environments, configured high availability using replication across availability zones and regions, implemented automation using Puppet, and added monitoring and backups to improve the stability and management of the PostgreSQL deployment.
This document discusses using NGINX to deliver high performance applications through efficient caching. It explains that NGINX can be used as a web server, load balancer, and high availability content cache to provide low latency, scalability, availability and reduced costs. Specific NGINX caching configurations like proxy_cache, proxy_cache_valid and proxy_cache_background_update are described. Microcaching optimizations with NGINX are also covered, showing significant performance improvements over Apache+WordPress and a reverse proxy only setup.
Percona XtraBackup is an open-source tool for performing backups of MySQL or MariaDB databases. It can create full, incremental, compressed, encrypted, and streaming backups. For full backups, it copies data files and redo logs then runs crash recovery to make the data consistent. Incremental backups only copy changed pages by tracking log sequence numbers. The backups can be prepared then restored using copy-back or move-back options.
Secure our data is a complex topic. We can build a very strong protection around our data, but nothing will prevent the one WHO could potentially access it to compromise the data integrity or to expose it.
This because we either under estimate the control we can or should impose, or because we think to do not have the tools to perform such control.
Nowadays to be able to control and manage what can access our data is a must, while how to do with standard tools it is a nightmare.
The presentation will guide you in a journey, there you will discover how implementing a quite robust protection, more than what you thought was possible.
Even more, it is possible and your performances will even improve. Cool right?
We will discuss:
- Access using not standard port
- Implement selective query access
- Define accessibility by location/ip/id
- Reduce to minimum cost of filtering
- Automate the query discovery
This document provides an introduction to MySQL master-slave replication. It defines replication as copying data from a master database server to one or more slave servers. The key concepts of master and slave are explained. Advantages of replication include scaling out solutions, increasing data security, enabling analytics, and distributing data over long distances. Event-based and GTID-based replication methods are described. The document also demonstrates setting up master-slave replication between two AWS MySQL instances and provides guidance on when replication is best used, such as for systems with many reads and few writes.
Enterprise grade deployment and security with PostgreSQLHimanchali -
This document summarizes an enterprise deployment of PostgreSQL at a large mobile advertising company. It discusses:
1) The company uses PostgreSQL for transaction processing and reporting across 60 servers storing 10GB to 20TB of data each, with the critical databases averaging 80GB and handling 18,000 queries per second.
2) To achieve 99.99% uptime, the company employs disaster recovery planning including cascaded replication across data centers and regular disaster drills to test backups.
3) The company uses Puppet for configuration management, stored procedures to optimize queries, monitoring tools, and access control and encryption to secure user access and network traffic.
Ilya Kosmodemiansky - An ultimate guide to upgrading your PostgreSQL installa...PostgreSQL-Consulting
Even an experienced PostgreSQL DBA can not always say that upgrading between major versions of Postgres is an easy task, especially if there are some special requirements, such as downtime limitations or if something goes wrong. For less experienced DBAs anything more complex than dump/restore can be frustrating.
In this talk I will describe why we need a special procedure to upgrade between major versions, how that can be achieved and what sort of problems can occur. I will review all possible ways to upgrade your cluster from classical pg_upgrade to old-school slony or modern methods like logical replication. For all approaches, I will give a brief explanation how it works (limited by the scope of this talk of course), examples how to perform upgrade and some advice on potentially problematic steps. Besides I will touch upon such topics as integration of upgrade tools and procedures with other software — connection brokers, operating system package managers, automation tools, etc. This talk would not be complete if I do not cover cases when something goes wrong and how to deal with such cases.
This document discusses MySQL data replication. It explains that replication asynchronously copies data from a master database server to slave servers, allowing the slaves to handle read operations and serve as backups. It provides configuration steps for setting up replication by enabling binary logging on the master, setting server IDs, and specifying replication users and hosts. Code examples demonstrate how to configure the master and slave servers and check the slave's replication status. Finally, it briefly mentions alternative replication topologies and tools.
A tutorial on MySQL High Availability on the Pacemaker stack. Covers both MySQL on DRBD, and MySQL with MySQL replication.
Presented by Florian Haas and Yves Trudeau at the Percona Live MySQL Conference & Expo, 2012
This document provides an overview and instructions for installing and configuring ProxySQL. It discusses:
1. What ProxySQL is and its functions like load balancing and query caching
2. How to install ProxySQL on CentOS and configure the /etc/proxysql.cnf file
3. How to set up the ProxySQL schema to define servers, users, variables and other settings needed for operation
4. How to test ProxySQL functions like server status changes and benchmark performance
Spilo is a tool that provides high availability for PostgreSQL databases running on AWS. It uses Patroni and ETCD to handle replication, failover, and cluster state management. Teams at Zalando use Spilo to run over 150 PostgreSQL databases in a self-managed way on AWS, with each team responsible for their own databases. Spilo provides automation for deploying, replicating, and failing over PostgreSQL clusters on AWS, allowing for increased agility compared to managed database services.
LAMP was originally Linux, Apache, MySQL, PHP. While the L & A have parts have become more flexible, most still use MySQL. With the recent acquisition by Oracle there's no better time to demystify PostgreSQL. For years PostgreSQL has had a reputation of being difficult, but this is the furthest from the truth.
This presentation by Asher Snyder will cover installation, basic queries, stored procedures, triggers, and full-text search.
The document discusses various techniques for performance tuning and cluster administration in HBase, including garbage collection tuning, use of memstore-local allocation buffers (MSLAB), enabling compression, optimizing splits and compactions through pre-splitting regions, and addressing hotspotting through manual splits. It provides guidance on configuring garbage collection, compression codecs, and approaches for managing splits and compactions to reduce disk I/O loads.
Out of the box replication in postgres 9.4(pg confus)Denish Patel
This document contains notes from a presentation on PostgreSQL replication. It discusses write-ahead logs (WAL), replication history in PostgreSQL from versions 7.0 to 9.4, how to set up basic replication, tools for backups and monitoring replication, and demonstrates setting up replication without third party tools using pg_basebackup, replication slots, and pg_receivexlog. It also includes contact information for the presenter and an invitation to join the PostgreSQL Slack channel.
Streaming Replication Made Easy in v9.3Sameer Kumar
This document discusses setting up streaming replication in PostgreSQL v9.3 to enable high availability. It covers preparing primary and standby servers, configuring wal_level and max_wal_senders on the primary, taking a backup and restoring on the standby, creating a recovery.conf file, starting the servers to test replication, triggering failover by promoting the standby, handling multiple replicas without rebuilding, and rebuilding the original primary as a new standby. Monitoring replication status is also addressed using views like pg_stat_replication.
PostgreSQL Hangout Replication Features v9.4Ashnikbiz
See the new enhancements in v9.4 which takes away the pain of guessing right wal_keep_segment
See the new time lagging replication capability in v9.4
Short intro to logical replication introduced in v9.4
Background Tasks in Node - Evan Tahler, TaskRabbitRedis Labs
The part of the talk discusses various strategies for handling background tasks in Node.js, including:
1. Foreground tasks, which process tasks inline but can slow down response times.
2. Parallel tasks, which ignore callbacks to try and speed up response times but lose error handling.
3. Local messages using clustering, which uses inter-process communication to pass tasks to worker processes and provides error handling but is limited to one server.
4. Remote messages using Redis pub/sub, which allows distributing tasks across multiple servers but limits server types to one.
5. Remote queue using Node-Resque, which treats tasks as jobs in a Redis queue for scheduling across worker processes, providing
Basics of Logical Replication,Streaming replication vs Logical Replication ,U...Rajni Baliyan
This document provides an overview of logical replication in PostgreSQL 10. It discusses the history of PostgreSQL replication, the key concepts of logical replication including publications, subscriptions, and replication slots. It presents several use cases for logical replication such as cross-platform, cross-version, and write operations at the subscriber. The document also covers configuration settings, a quick setup demonstration, monitoring, resolving conflicts, and limitations of logical replication.
A meticulous presentation on Authorization, Encryption & Authentication of the security features in MySQL 8.0 by Vignesh Prabhu, Database reliability engineer, Mydbops.
Out of the box replication in postgres 9.4Denish Patel
This document provides an overview of setting up out of the box replication in PostgreSQL 9.4 without third party tools. It discusses write-ahead logs (WAL), replication slots, pg_basebackup, and pg_receivexlog. The document then demonstrates setting up replication on VMs with pg_basebackup to initialize a standby server, configuration of primary and standby servers, and monitoring of replication.
The document provides configuration instructions and guidelines for setting up streaming replication between a PostgreSQL master and standby server, including setting parameter values for wal_level, max_wal_senders, wal_keep_segments, creating a dedicated replication role, using pg_basebackup to initialize the standby, and various recovery target options to control the standby's behavior. It also discusses synchronous replication using replication slots and monitoring the replication process on both the master and standby servers.
Denish Patel deployed PostgreSQL on Amazon EC2 for a startup that had its entire IT architecture on the Amazon cloud. The initial deployment involved setting up master-slave configurations across two EC2 environments with issues like weekly instance failures and lack of monitoring. Patel then consolidated the environments, configured high availability using replication across availability zones and regions, implemented automation using Puppet, and added monitoring and backups to improve the stability and management of the PostgreSQL deployment.
This document discusses using NGINX to deliver high performance applications through efficient caching. It explains that NGINX can be used as a web server, load balancer, and high availability content cache to provide low latency, scalability, availability and reduced costs. Specific NGINX caching configurations like proxy_cache, proxy_cache_valid and proxy_cache_background_update are described. Microcaching optimizations with NGINX are also covered, showing significant performance improvements over Apache+WordPress and a reverse proxy only setup.
Percona XtraBackup is an open-source tool for performing backups of MySQL or MariaDB databases. It can create full, incremental, compressed, encrypted, and streaming backups. For full backups, it copies data files and redo logs then runs crash recovery to make the data consistent. Incremental backups only copy changed pages by tracking log sequence numbers. The backups can be prepared then restored using copy-back or move-back options.
Secure our data is a complex topic. We can build a very strong protection around our data, but nothing will prevent the one WHO could potentially access it to compromise the data integrity or to expose it.
This because we either under estimate the control we can or should impose, or because we think to do not have the tools to perform such control.
Nowadays to be able to control and manage what can access our data is a must, while how to do with standard tools it is a nightmare.
The presentation will guide you in a journey, there you will discover how implementing a quite robust protection, more than what you thought was possible.
Even more, it is possible and your performances will even improve. Cool right?
We will discuss:
- Access using not standard port
- Implement selective query access
- Define accessibility by location/ip/id
- Reduce to minimum cost of filtering
- Automate the query discovery
This document provides an introduction to MySQL master-slave replication. It defines replication as copying data from a master database server to one or more slave servers. The key concepts of master and slave are explained. Advantages of replication include scaling out solutions, increasing data security, enabling analytics, and distributing data over long distances. Event-based and GTID-based replication methods are described. The document also demonstrates setting up master-slave replication between two AWS MySQL instances and provides guidance on when replication is best used, such as for systems with many reads and few writes.
Enterprise grade deployment and security with PostgreSQLHimanchali -
This document summarizes an enterprise deployment of PostgreSQL at a large mobile advertising company. It discusses:
1) The company uses PostgreSQL for transaction processing and reporting across 60 servers storing 10GB to 20TB of data each, with the critical databases averaging 80GB and handling 18,000 queries per second.
2) To achieve 99.99% uptime, the company employs disaster recovery planning including cascaded replication across data centers and regular disaster drills to test backups.
3) The company uses Puppet for configuration management, stored procedures to optimize queries, monitoring tools, and access control and encryption to secure user access and network traffic.
Ilya Kosmodemiansky - An ultimate guide to upgrading your PostgreSQL installa...PostgreSQL-Consulting
Even an experienced PostgreSQL DBA can not always say that upgrading between major versions of Postgres is an easy task, especially if there are some special requirements, such as downtime limitations or if something goes wrong. For less experienced DBAs anything more complex than dump/restore can be frustrating.
In this talk I will describe why we need a special procedure to upgrade between major versions, how that can be achieved and what sort of problems can occur. I will review all possible ways to upgrade your cluster from classical pg_upgrade to old-school slony or modern methods like logical replication. For all approaches, I will give a brief explanation how it works (limited by the scope of this talk of course), examples how to perform upgrade and some advice on potentially problematic steps. Besides I will touch upon such topics as integration of upgrade tools and procedures with other software — connection brokers, operating system package managers, automation tools, etc. This talk would not be complete if I do not cover cases when something goes wrong and how to deal with such cases.
This document discusses MySQL data replication. It explains that replication asynchronously copies data from a master database server to slave servers, allowing the slaves to handle read operations and serve as backups. It provides configuration steps for setting up replication by enabling binary logging on the master, setting server IDs, and specifying replication users and hosts. Code examples demonstrate how to configure the master and slave servers and check the slave's replication status. Finally, it briefly mentions alternative replication topologies and tools.
A tutorial on MySQL High Availability on the Pacemaker stack. Covers both MySQL on DRBD, and MySQL with MySQL replication.
Presented by Florian Haas and Yves Trudeau at the Percona Live MySQL Conference & Expo, 2012
This document provides an overview and instructions for installing and configuring ProxySQL. It discusses:
1. What ProxySQL is and its functions like load balancing and query caching
2. How to install ProxySQL on CentOS and configure the /etc/proxysql.cnf file
3. How to set up the ProxySQL schema to define servers, users, variables and other settings needed for operation
4. How to test ProxySQL functions like server status changes and benchmark performance
Spilo is a tool that provides high availability for PostgreSQL databases running on AWS. It uses Patroni and ETCD to handle replication, failover, and cluster state management. Teams at Zalando use Spilo to run over 150 PostgreSQL databases in a self-managed way on AWS, with each team responsible for their own databases. Spilo provides automation for deploying, replicating, and failing over PostgreSQL clusters on AWS, allowing for increased agility compared to managed database services.
LAMP was originally Linux, Apache, MySQL, PHP. While the L & A have parts have become more flexible, most still use MySQL. With the recent acquisition by Oracle there's no better time to demystify PostgreSQL. For years PostgreSQL has had a reputation of being difficult, but this is the furthest from the truth.
This presentation by Asher Snyder will cover installation, basic queries, stored procedures, triggers, and full-text search.
The document discusses using plProxy and pgBouncer to split a PostgreSQL database horizontally and vertically to improve scalability. It describes how plProxy allows functions to make remote calls to other databases and how pgBouncer can be used for connection pooling. The RUN ON clause of plProxy is also summarized, which allows queries to execute on all partitions or on a specific partition.
Managing Databases In A DevOps Environment 2016Robert Treat
Given at #pgdayphilly2016, this talk covers how configuration management, monitoring, and rapid deployments are impacting how we think about database management.
This talk discusses Linux profiling using perf_events (also called "perf") based on Netflix's use of it. It covers how to use perf to get CPU profiling working and overcome common issues. The speaker will give a tour of perf_events features and show how Netflix uses it to analyze performance across their massive Amazon EC2 Linux cloud. They rely on tools like perf for customer satisfaction, cost optimization, and developing open source tools like NetflixOSS. Key aspects covered include why profiling is needed, a crash course on perf, CPU profiling workflows, and common "gotchas" to address like missing stacks, symbols, or profiling certain languages and events.
Skype uses PostgreSQL for over 100 database servers and 200 databases to support its online services. It partitions data across multiple databases for scalability and uses tools like pgBouncer, PL/Proxy, SkyTools, PgQ, and Londiste to connect, load balance, and replicate data between databases. This complex database architecture has allowed Skype to grow significantly while remaining easy to manage.
Skype uses PostgreSQL databases that are split both vertically and horizontally to handle their large load. They connect databases using stored procedures and PL/Proxy for remote calls. Key components include pgBouncer for connection pooling, PgQ for queueing between databases, and SkyTools which contains many reusable database scripts and tools. While complex, this architecture has allowed Skype to scale their database infrastructure significantly while remaining manageable.
Skype uses PostgreSQL for its databases and has over 100 database servers handling over 10,000 transactions per second across 200 databases storing billions of records. The databases are split both vertically and horizontally to scale with load. All database access is through stored procedures for security and flexibility. Replication and remote procedure calls connect the databases. Tools like pgBouncer, PL/Proxy, PgQ, and Londiste help partition, connect, and load balance the databases to form a large and complex but manageable distributed database architecture.
Apache Pulsar Development 101 with PythonTimothy Spann
Apache Pulsar Development 101 with Python PS2022_Ecosystem_v0.0
There is always the fear a speaker cannot make it. So just in case, since I was the MC for the ecosystem track I put together a talk just in case.
Here it is. Never seen or presented.
What’s the Best PostgreSQL High Availability Framework? PAF vs. repmgr vs. Pa...ScaleGrid.io
Compare top PostgreSQL high availability frameworks - PostgreSQL Automatic Failover (PAF), Replication Manager (repmgr) and Patroni to improve your app uptime. ScaleGrid blog - https://scalegrid.io/blog/whats-the-best-postgresql-high-availability-framework-paf-vs-repmgr-vs-patroni-infographic/
Container orchestration from theory to practiceDocker, Inc.
"Join Laura Frank and Stephen Day as they explain and examine technical concepts behind container orchestration systems, like distributed consensus, object models, and node topology. These concepts build the foundation of every modern orchestration system, and each technical explanation will be illustrated using SwarmKit and Kubernetes as a real-world example. Gain a deeper understanding of how orchestration systems work in practice and walk away with more insights into your production applications."
Kubernetes is an open-source system for automating deployment, scaling, and management of containerized applications. It groups containers that make up an application into logical units for easy management and discovery called Pods. ReplicaSets ensure that a specified number of pod replicas are running at any given time. Key components include Pods, Services for enabling network access to applications, and Deployments to update Pods and manage releases.
Big mountain data and dev conference apache pulsar with mqtt for edge compu...Timothy Spann
This document provides an overview and summary of Apache Pulsar with MQTT for edge computing. It discusses how Pulsar is an open-source, cloud-native distributed messaging and streaming platform that supports MQTT and other protocols. It also summarizes Pulsar's key capabilities like data durability, scalability, geo-replication, and unified messaging model. The document includes diagrams showcasing Pulsar's publish-subscribe model and different subscription modes. It demonstrates how Pulsar can be used with edge devices via protocols like MQTT and how streams of data from edge can be processed using connectors, functions and SQL.
Airflow Best Practises & Roadmap to Airflow 2.0Kaxil Naik
This document provides an overview of new features in Airflow 1.10.8/1.10.9 and best practices for writing DAGs and configuring Airflow for production. It also outlines the roadmap for Airflow 2.0, including dag serialization, a revamped real-time UI, developing a production-grade modern API, releasing official Docker/Helm support, and improving the scheduler. The document aims to help users understand recent Airflow updates and plan their migration to version 2.0.
Www Kitebird Com Articles Pydbapi Html Toc 1AkramWaseem
The document discusses using Python and the DB-API module to write scripts that interact with MySQL databases. It provides an example of a short script that connects to a MySQL database, issues a query to get the server version, and prints the result. It then discusses a more extensive script that handles errors, issues different statement types like INSERT and SELECT, and retrieves result rows using fetchone() and fetchall() methods. The document provides information on installing prerequisites like Python and MySQLdb, and executing the example scripts.
Replication - Nick Carboni - ManageIQ Design Summit 2016ManageIQ
This document discusses database replication in ManageIQ. It describes two replication technologies: rubyrep, which ManageIQ previously used but is now outdated, and pglogical, which is an extension of PostgreSQL that ManageIQ is exploring using instead. The document compares the performance of rubyrep and pglogical in replicating different amounts of data with and without added artificial network latency.
Logging for Production Systems in The Container Era discusses how to effectively collect and analyze logs and metrics in microservices-based container environments. It introduces Fluentd as a centralized log collection service that supports pluggable input/output, buffering, and aggregation. Fluentd allows collecting logs from containers and routing them to storage systems like Kafka, HDFS and Elasticsearch. It also supports parsing, filtering and enriching log data through plugins.
This short document discusses the Puppet configuration management tool. It begins with a quote about controlling the spice and universe from Dune and includes the author's name and date. The document then lists various Puppet features like scalability, flexibility, simple installation, and modules. It provides examples of large sites using Puppet like Google and describes Puppet's client-server and masterless models. In closing, it emphasizes Puppet's flexibility, community support, and over 1200 pre-built configurations.
Apache Pulsar with MQTT for Edge Computing - Pulsar Summit Asia 2021StreamNative
This document discusses using Apache Pulsar with MQTT for edge computing. It provides an overview of Apache Pulsar and how it enables message queuing and data streaming with features like pub-sub, geo-replication, and multi-protocol support including MQTT. It also discusses edge computing characteristics and challenges, and how running Apache Pulsar on edge devices can address these by extending data processing to the edge and integrating with sensors using the MQTT protocol. Examples are provided of ingesting IoT data into Pulsar from Python and using NVIDIA Jetson devices with Pulsar.
Webinar | Building Apps with the Cassandra Python DriverDataStax Academy
With the new Python driver for Cassandra it is easy to build integrations and apps that use Cassandra seamlessly as a back in. This session will explore what it takes to build the app and the features available with the new Python drivers.
This document provides an overview of using R and high performance computers (HPC). It discusses why HPC is useful when data becomes too large for a local machine, and strategies like moving to more powerful hardware, using parallel packages, or rewriting code. It also covers topics like accessing HPC resources through batch jobs, setting up the R environment, profiling code, and using packages like purrr and foreach to parallelize workflows. The overall message is that HPC can scale up R analyses, but developers must adapt their code for parallel and distributed processing.
PgQ is a queue implementation for PostgreSQL that allows events to be produced asynchronously by producers and processed by consumers, providing reliable messaging, batch processing, and replication capabilities. It stores events transactionally in PostgreSQL tables and uses a ticker process to group events into batches for efficient processing by multiple consumer applications running in different languages. PgQ guarantees that each event is seen at least once by consumers but does not prevent duplicate processing, requiring external tracking by consumers.
ClojureScript allows developers to use the Clojure programming language to build applications that compile to JavaScript. This enables Clojure code to run in environments where JavaScript is supported, like web browsers and mobile apps. ClojureScript leverages the Google Closure compiler and library to provide whole program optimization of Clojure code compiling to JavaScript.
Why you should be excited about ClojureScriptelliando dias
ClojureScript allows Clojure code to compile to JavaScript. Created by Rich Hickey and friends, it provides optimizations for performance while maintaining readability and abstraction. As a Lisp for JavaScript, ClojureScript controls complexity on the web and benefits from JavaScript's status as a compilation target for many languages.
Functional Programming with Immutable Data Structureselliando dias
1. The document discusses the advantages of functional programming with immutable data structures for multi-threaded environments. It argues that shared mutable data and variables are fundamentally flawed concepts that can lead to bugs, while immutable data avoids these issues.
2. It presents Clojure as a functional programming language that uses immutable persistent data structures and software transactional memory to allow for safe, lock-free concurrency. This approach allows readers and writers to operate concurrently without blocking each other.
3. The document makes the case that Lisp parentheses in function calls uniquely define the tree structure of computations and enable powerful macro systems, homoiconicity, and structural editing of code.
O documento lista e descreve as principais partes de um contêiner de carga seco, incluindo o painel frontal, laterais, traseira, teto, piso e estrutura inferior. Muitos componentes como painéis laterais, travessas do teto e fundo são numerados de acordo com sua localização. As portas traseiras contêm quadros, painéis, dobradiças e barras de fechamento.
O documento discute a história da geometria projetiva, desde Euclides até seu uso em computação gráfica. Aborda figuras-chave como Pascal, que foi pioneiro na área, e como a perspectiva foi aplicada nas artes ao longo dos séculos.
Polyglot and Poly-paradigm Programming for Better Agilityelliando dias
This document discusses the benefits of polyglot and poly-paradigm programming approaches for building more agile applications. It describes how using multiple languages and programming paradigms can optimize both performance and developer productivity. Specifically, it suggests that statically-typed compiled languages be used for core application components while dynamically-typed scripting languages connect and customize these components. This approach allows optimizing areas that require speed/efficiency separately from those requiring flexibility. The document also advocates aspects and functional programming to address cross-cutting concerns and concurrency challenges that arise in modern applications.
This document discusses JavaScript libraries and frameworks. It provides an overview of some popular options like jQuery, Prototype, Dojo, MooTools, and YUI. It explains why developers use libraries, such as for faster development, cross-browser compatibility, and animation capabilities. The document also discusses how libraries resemble CSS and use selector syntax. Basic examples are provided to demonstrate common tasks like hover effects and row striping. Factors for choosing a library are outlined like maturity, documentation, community, and licensing. The document concludes by explaining how to obtain library code from project websites or Google's AJAX Libraries API.
How to Make an Eight Bit Computer and Save the World!elliando dias
This document summarizes a talk given to introduce an open source 8-bit computer project called the Humane Reader. The talk outlines the goals of providing a cheap e-book reader and computing platform using open source tools. It describes the hardware design which uses an AVR microcontroller and interfaces like video output, SD card, and USB. The talk also covers using open source tools for development and sourcing low-cost fabrication and assembly. The overall goals are to create an inexpensive device that can provide educational resources in developing areas.
Ragel is a parser generator that compiles to various host languages including Ruby. It is useful for parsing protocols and data formats and provides faster parsing than regular expressions or full LALR parsers. Several Ruby projects like Mongrel and Hpricot use Ragel for tasks like HTTP request parsing and HTML parsing. When using Ragel with Ruby, it can be compiled to Ruby code directly, which is slow, or a C extension can be written for better performance. The C extension extracts the parsed data from Ragel and makes it available to Ruby.
A Practical Guide to Connecting Hardware to the Webelliando dias
This document provides an overview of connecting hardware devices to the web using the Arduino platform. It discusses trends in electronics and computing that make this easier, describes the Arduino hardware and software, and covers various connection methods including directly to a computer, via wireless modems, Ethernet shields, and services like Pachube that allow sharing sensor data over the internet. The document aims to demonstrate how Arduinos can communicate with other devices and be used to build interactive systems.
O documento introduz o Arduino, uma plataforma de desenvolvimento open-source. Discute as características e componentes do Arduino, incluindo microcontroladores, software e exemplos de código. Também fornece instruções básicas sobre como programar o Arduino usando linguagem C.
O documento apresenta um mini-curso introdutório sobre Arduino, abordando o que é a plataforma Arduino, como é estruturado seu hardware, como programá-lo, exemplos básicos de código e aplicações possíveis como controle residencial e robótica.
The document discusses various functions for working with datasets in the Incanter library for Clojure. It describes how to create, read, save, select rows and columns from, and sort datasets. Functions are presented for building datasets from sequences, reading datasets from files and URLs, saving datasets to files and databases, selecting single or multiple columns, and filtering rows based on conditions. The document also provides an overview of the Incanter library and its various namespaces for statistics, charts, and other functionality.
Rango is a lightweight Ruby web framework built on Rack that aims to be more robust than Sinatra but smaller than Rails or Merb. It is inspired by Django and Merb, uses Ruby 1.9, and supports features like code reloading, Bundler, routing, rendering, and HTTP error handling. The documentation provides examples and details on using Rango.
Fab.in.a.box - Fab Academy: Machine Designelliando dias
This document describes the design of a multifab machine called MTM. It includes descriptions of the XY stage and Z axis drive mechanisms, as well as the tool heads and network used to control the machine. Key aspects of the design addressed include the stepper motor selection, drive electronics, motion control firmware, and use of a virtual machine environment and circular buffer to enable distributed control of the machine. Strengths of the design include low inertia enabling high acceleration, while weaknesses include low basic resolution and stiffness unsuitable for heavy milling.
The Digital Revolution: Machines that makeselliando dias
The document discusses the results of a study on the effects of a new drug on memory and cognitive function in older adults. The double-blind study involved giving either the new drug or a placebo to 100 volunteers aged 65-80 over a 6 month period. Testing showed those receiving the drug experienced statistically significant improvements in short-term memory retention and processing speed compared to the placebo group.
The document discusses using Clojure for Hadoop programming. Clojure is a dynamic functional programming language that runs on the Java Virtual Machine. The document provides an overview of Clojure and how its features like immutability and concurrency make it well-suited for Hadoop. It then shows examples of implementing Hadoop MapReduce jobs using Clojure by defining mapper and reducer functions.
This document provides an overview of Hadoop, including:
1) Hadoop solves the problems of analyzing massively large datasets by distributing data storage and analysis across multiple machines to tolerate node failure.
2) Hadoop uses HDFS for distributed data storage, which shards massive files across data nodes with replication for fault tolerance, and MapReduce for distributed data analysis by sending code to the data.
3) The document demonstrates MapReduce concepts like map, reduce, and their composition with an example job.
Hadoop and Hive Development at Facebookelliando dias
Facebook generates large amounts of user data daily from activities like status updates, photo uploads, and shared content. This data is stored in Hadoop using Hive for analytics. Some key facts:
- Facebook adds 4TB of new compressed data daily to its Hadoop cluster.
- The cluster has 4800 cores and 5.5PB of storage across 12TB nodes.
- Hive is used for over 7500 jobs daily and by around 200 engineers/analysts monthly.
- Performance improvements to Hive include lazy deserialization, map-side aggregation, and joins.
Multi-core Parallelization in Clojure - a Case Studyelliando dias
The document describes a case study on using Clojure for multi-core parallelization of the K-means clustering algorithm. It provides background on parallel programming concepts, an introduction to Clojure, and details on how the authors implemented a parallel K-means algorithm in Clojure using agents and software transactional memory. They present results showing speedups from parallelization and accuracy comparable to R's implementation on both synthetic and real-world datasets.
How to Get CNIC Information System with Paksim Ga.pptxdanishmna97
Pakdata Cf is a groundbreaking system designed to streamline and facilitate access to CNIC information. This innovative platform leverages advanced technology to provide users with efficient and secure access to their CNIC details.
Epistemic Interaction - tuning interfaces to provide information for AI supportAlan Dix
Paper presented at SYNERGY workshop at AVI 2024, Genoa, Italy. 3rd June 2024
https://alandix.com/academic/papers/synergy2024-epistemic/
As machine learning integrates deeper into human-computer interactions, the concept of epistemic interaction emerges, aiming to refine these interactions to enhance system adaptability. This approach encourages minor, intentional adjustments in user behaviour to enrich the data available for system learning. This paper introduces epistemic interaction within the context of human-system communication, illustrating how deliberate interaction design can improve system understanding and adaptation. Through concrete examples, we demonstrate the potential of epistemic interaction to significantly advance human-computer interaction by leveraging intuitive human communication strategies to inform system design and functionality, offering a novel pathway for enriching user-system engagements.
Goodbye Windows 11: Make Way for Nitrux Linux 3.5.0!SOFTTECHHUB
As the digital landscape continually evolves, operating systems play a critical role in shaping user experiences and productivity. The launch of Nitrux Linux 3.5.0 marks a significant milestone, offering a robust alternative to traditional systems such as Windows 11. This article delves into the essence of Nitrux Linux 3.5.0, exploring its unique features, advantages, and how it stands as a compelling choice for both casual users and tech enthusiasts.
Why You Should Replace Windows 11 with Nitrux Linux 3.5.0 for enhanced perfor...SOFTTECHHUB
The choice of an operating system plays a pivotal role in shaping our computing experience. For decades, Microsoft's Windows has dominated the market, offering a familiar and widely adopted platform for personal and professional use. However, as technological advancements continue to push the boundaries of innovation, alternative operating systems have emerged, challenging the status quo and offering users a fresh perspective on computing.
One such alternative that has garnered significant attention and acclaim is Nitrux Linux 3.5.0, a sleek, powerful, and user-friendly Linux distribution that promises to redefine the way we interact with our devices. With its focus on performance, security, and customization, Nitrux Linux presents a compelling case for those seeking to break free from the constraints of proprietary software and embrace the freedom and flexibility of open-source computing.
Generative AI Deep Dive: Advancing from Proof of Concept to ProductionAggregage
Join Maher Hanafi, VP of Engineering at Betterworks, in this new session where he'll share a practical framework to transform Gen AI prototypes into impactful products! He'll delve into the complexities of data collection and management, model selection and optimization, and ensuring security, scalability, and responsible use.
DevOps and Testing slides at DASA ConnectKari Kakkonen
My and Rik Marselis slides at 30.5.2024 DASA Connect conference. We discuss about what is testing, then what is agile testing and finally what is Testing in DevOps. Finally we had lovely workshop with the participants trying to find out different ways to think about quality and testing in different parts of the DevOps infinity loop.
Maruthi Prithivirajan, Head of ASEAN & IN Solution Architecture, Neo4j
Get an inside look at the latest Neo4j innovations that enable relationship-driven intelligence at scale. Learn more about the newest cloud integrations and product enhancements that make Neo4j an essential choice for developers building apps with interconnected data and generative AI.
UiPath Test Automation using UiPath Test Suite series, part 6DianaGray10
Welcome to UiPath Test Automation using UiPath Test Suite series part 6. In this session, we will cover Test Automation with generative AI and Open AI.
UiPath Test Automation with generative AI and Open AI webinar offers an in-depth exploration of leveraging cutting-edge technologies for test automation within the UiPath platform. Attendees will delve into the integration of generative AI, a test automation solution, with Open AI advanced natural language processing capabilities.
Throughout the session, participants will discover how this synergy empowers testers to automate repetitive tasks, enhance testing accuracy, and expedite the software testing life cycle. Topics covered include the seamless integration process, practical use cases, and the benefits of harnessing AI-driven automation for UiPath testing initiatives. By attending this webinar, testers, and automation professionals can gain valuable insights into harnessing the power of AI to optimize their test automation workflows within the UiPath ecosystem, ultimately driving efficiency and quality in software development processes.
What will you get from this session?
1. Insights into integrating generative AI.
2. Understanding how this integration enhances test automation within the UiPath platform
3. Practical demonstrations
4. Exploration of real-world use cases illustrating the benefits of AI-driven test automation for UiPath
Topics covered:
What is generative AI
Test Automation with generative AI and Open AI.
UiPath integration with generative AI
Speaker:
Deepak Rai, Automation Practice Lead, Boundaryless Group and UiPath MVP
“An Outlook of the Ongoing and Future Relationship between Blockchain Technologies and Process-aware Information Systems.” Invited talk at the joint workshop on Blockchain for Information Systems (BC4IS) and Blockchain for Trusted Data Sharing (B4TDS), co-located with with the 36th International Conference on Advanced Information Systems Engineering (CAiSE), 3 June 2024, Limassol, Cyprus.
GraphSummit Singapore | The Art of the Possible with Graph - Q2 2024Neo4j
Neha Bajwa, Vice President of Product Marketing, Neo4j
Join us as we explore breakthrough innovations enabled by interconnected data and AI. Discover firsthand how organizations use relationships in data to uncover contextual insights and solve our most pressing challenges – from optimizing supply chains, detecting fraud, and improving customer experiences to accelerating drug discoveries.
Observability Concepts EVERY Developer Should Know -- DeveloperWeek Europe.pdfPaige Cruz
Monitoring and observability aren’t traditionally found in software curriculums and many of us cobble this knowledge together from whatever vendor or ecosystem we were first introduced to and whatever is a part of your current company’s observability stack.
While the dev and ops silo continues to crumble….many organizations still relegate monitoring & observability as the purview of ops, infra and SRE teams. This is a mistake - achieving a highly observable system requires collaboration up and down the stack.
I, a former op, would like to extend an invitation to all application developers to join the observability party will share these foundational concepts to build on:
Communications Mining Series - Zero to Hero - Session 1DianaGray10
This session provides introduction to UiPath Communication Mining, importance and platform overview. You will acquire a good understand of the phases in Communication Mining as we go over the platform with you. Topics covered:
• Communication Mining Overview
• Why is it important?
• How can it help today’s business and the benefits
• Phases in Communication Mining
• Demo on Platform overview
• Q/A
Climate Impact of Software Testing at Nordic Testing DaysKari Kakkonen
My slides at Nordic Testing Days 6.6.2024
Climate impact / sustainability of software testing discussed on the talk. ICT and testing must carry their part of global responsibility to help with the climat warming. We can minimize the carbon footprint but we can also have a carbon handprint, a positive impact on the climate. Quality characteristics can be added with sustainability, and then measured continuously. Test environments can be used less, and in smaller scale and on demand. Test techniques can be used in optimizing or minimizing number of tests. Test automation can be used to speed up testing.
GridMate - End to end testing is a critical piece to ensure quality and avoid...ThomasParaiso2
End to end testing is a critical piece to ensure quality and avoid regressions. In this session, we share our journey building an E2E testing pipeline for GridMate components (LWC and Aura) using Cypress, JSForce, FakerJS…
2. Stored Procedure API
Skype has had from the beginning the requirement that all database access must be
implemented through stored procedures.
Using function-based database access has more general good points:
It's good to have SQL statements that operate on data near to tables. That
makes life of DBA's easier.
It makes it possible to optimize and reorganize tables transparently to the
application.
Enables DBA's to move parts of database into another database without
changing application interface.
Easier to manage security if you don't have to do it on table level. In most cases
you need to control what user can do and on which data not on what tables.
All transactions can be made in autocommit mode. That means absolutely
minimal amount of roundtrips (1) for each query and also each transaction
takes shortest possible amount of time on server - remember that various locks
that transactions acquire are release on COMMIT.
3. Keeping Online Databases Slim and Fit
Rule 1: Minimize number of connections. Channel all incoming queries through
optimal number of database connections using pgBouncer
Rule 2: Minimize number of indexes constraints that use up performance and
resources. For example backoffice applications need quite often more and different
indexes than are needed for online queries.
Rule 3: Keep as little data as possible in online database. Move all data that is not
need into second or third tier databases. Use remote calls to access it if needed.
Rule 4: Keep transactions short. Using functions is the best way to do this as you
can use functions in autocommit mode and still do complex SQL operations.
Rule 5: Keep external dependancies to the minimum then you have less things that
can get broken and affect online database. For examples run batch jobs against
second and third tier databases.
5. pgBouncer – PostgreSQL Connection pooler
pgBouncer is lightweight and robust
connection pooler for PostgreSQL. FrontEnd WebServer
Reduces thousands of incoming connections
to only tens of connections in database.
Low number of connections is important
because each connection uses computer
resources and each new connection is quite pgBouncer
expensive as prepared plans have to be
created each time from scratch.
We are not using pgBouncer for load userdb
balancing.
Can be used to redirect database calls
(database aliases).
6. plProxy – Remote Call Language
PL/Proxy is compact language for remote
calls between PostgreSQL databases.
With PL/Proxy user can create proxy
functions that have same signature as
userdb
remote functions to be called. The function
body describes how the remote connection
should be acquired.
userdb_ro
CREATE FUNCTION get_user_email(username text) RETURNS text AS $$
CONNECT 'dbname=userdb_ro';
$$ LANGUAGE plproxy;
7. SkyTools: Batch Job Framework
Written in python and contain most everything we have found
useful in our everyday work with databases and PostgreSQL.
Framework provides database connectivity, logging, stats
management, encoding, decoding etc for batch jobs. Developers
need only to take care of business logic in batch jobs all the rest database
is handled by batch jobs.
SkyTools contains 10s of reusable generic scripts for doing
various data related tasks.
PgQ that adds event queues to PostgreSQL.
Londiste replication. batch
database
job
Walmgr for wal based log shipping.
database
8. PgQ: PostgreSQL Queues
PgQ is PostgreSQL based event processing system. It is part of SkyTools package
that contains several useful implementations on this engine.
Event - atomic piece of data created by Producers. In PgQ event is one record in
one of tables that services that queue. PgQ guarantees that each event is seen at
least once but it is up to consumer to make sure that event is processed no more
than once if that is needed.
Batch - PgQ is designed for efficiency and high throughput so events are grouped
into batches for bulk processing.
Queue - Event are stored in queue tables i.e queues. Several producers can write
into same queue and several consumers can read from the queue. Events are kept
in queue until all the consumers have seen them. Queue can contain any number of
event types it is up to Producer and Consumer to agree on what types of events are
passed and how they are encoded
Producer - applications that pushes event into queue. Producer can be written in
any language that is able to run stored procedures in PostgreSQL.
Consumer - application that reads events from queue. Consumers can be written in
any language that can interact with PostgreSQL.
9. PgQ: PostgreSQL Queues Illustration
Database that keeps track of user status (online, offline, busy, etc.
Producer can be anybody who can call stored procedure
The same goes about consumers
java
presence_cache1
c++:
presence_listener PresenceDB java
presence_cache2
function:
set_user_presence
php:
presence_listener c++
table:
presence_poller
user_status
queue:
user_status python:
presence_updater
10. PgQ: Features
Transactional. Event insertion is committed or rolled back together with the other
things that transaction changes.
Efficient. Usually database based queue implementations are not efficient but we
managed to solve it because PostgreSQL makes transaction state visible to users.
Fast. Events are processed in batches which gives low per event overhead.
Flexible. Each queue can have several producers and consumers and any number
of event types handled in it.
Robust. PgQ guarantees that each consumers sees event at least once. There
several methods how the processed events can be tracked depending on business
needs.
Ideally suited for all kinds of batch processing.
12. plProxy: Installation
Download PL/Proxy from http://pgfoundry.org/projects/plproxy and extract.
Build PL/Proxy by running make and make install inside of the plproxy directory. If
your having problems make sure that pg_config from the postgresql bin directory is
in your path.
To install PL/Proxy in a database execute the commands in the plproxy.sql file. For
example psql -f $SHAREDIR/contrib/plproxy.sql mydatabase
Steps 1 and 2 can be skipped if your installed pl/proxy from a packaging system
such as RPM.
Create a test function to validate that plProxy is working as expected.
CREATE FUNCTION get_user_email(username text)
RETURNS text AS $$
CONNECT 'dbname=userdb';
$$ LANGUAGE plproxy;
13. plProxy Language
The language is similar to plpgsql - string quoting, comments, semicolon at the
statements end.It contains only 4 statements: CONNECT, CLUSTER, RUN and
SELECT.
Each function needs to have either CONNECT or pair of CLUSTER + RUN
statements to specify where to run the function.
CONNECT 'libpq connstr'; -- Specifies exact location where to connect and
execute the query. If several functions have same connstr, they will use same
connection.
CLUSTER 'cluster_name'; -- Specifies exact cluster name to be run on. The cluster
name will be passed to plproxy.get_cluster_* functions.
CLUSTER cluster_func(..); -- Cluster name can be dynamically decided upon
proxy function arguments. cluster_func should return text value of final cluster
name.
14. plProxy Language RUN ON ...
RUN ON ALL; -- Query will be run on all partitions in cluster in parallel.
RUN ON ANY; -- Query will be run on random partition.
RUN ON <NR>; -- Run on partition number <NR>.
RUN ON partition_func(..); -- Run partition_func() which should return one or more
hash values. (int4) Query will be run on tagged partitions. If more than one partition
was tagged, query will be sent in parallel to them.
CREATE FUNCTION get_user_email(text)
RETURNS text AS $$
CLUSTER 'userdb';
RUN ON get_hash($1);
$$ LANGUAGE plproxy;
Partition selection is done by taking lower bits from hash value.
hash & (n-1), where n is power of 2.
15. plProxy Configuration
Schema plproxy and three functions are needed for plProxy
plproxy.get_cluster_partitions(cluster_name text) – initializes
plProxy connect strings to remote databases
plproxy.get_cluster_version(cluster_name text) – used by plProxy to
determine if configuration has changed and should be read again. Should be as fast
as possible because it is called for every function call that goes through plProxy.
plproxy.get_cluster_config(in cluster_name text, out key text,
out val text) – can be used to change plProxy parameters like connection
lifetime.
CREATE FUNCTION plproxy.get_cluster_version(i_cluster text)
RETURNS integer AS $$
SELECT 1;
$$ LANGUAGE sql;
CREATE FUNCTION plproxy.get_cluster_config(cluster_name text,
OUT key text, OUT val text)
RETURNS SETOF record AS $$
SELECT 'connection_lifetime'::text as key, text( 30*60 ) as val;
$$ LANGUAGE sql;
16. plProxy: Configuration Functions
CREATE FUNCTION plproxy.get_cluster_partitions(cluster_name text)
RETURNS SETOF text AS $$
begin
if cluster_name = 'userdb' then
return next 'port=9000 dbname=userdb_p00 user=proxy';
return next 'port=9000 dbname=userdb_p01 user=proxy';
return;
end if;
raise exception 'no such cluster: %', cluster_name;
end; $$ LANGUAGE plpgsql SECURITY DEFINER;
CREATE FUNCTION plproxy.get_cluster_partitions(i_cluster_name text)
RETURNS SETOF text AS $$
declare r record;
begin
for r in
select connect_string from plproxy.conf
where cluster_name = i_cluster_name
order by part_nr
loop
return next r.connect_string;
end loop;
if not found then
raise exception 'no such cluster: %', i_cluster_name;
end if;
return;
end; $$ LANGUAGE plpgsql SECURITY DEFINER;
17. plProxy: Read Only Remote Calls
We use remote calls mostly for read only queries in cases where it is not reasonable
to replicate data needed to calling database.
For example balance data is changing very often but whenever doing decisions
based on balance we must use the latest balance so we use remote call to get user
balance.
plProxy remote calls are not suitable for data modification because there is no
guarantee that both transactions will be committed or rolled back (No 2phase
commit). Instead of using remote calls we try to use PgQ queues and various
consumers on them that provide our 2phase commit.
userDB
shopDb
balanceDB
18. plProxy: Remote Calls to Archive
Closed records from large online tables (invoices, payments, call detail records, etc)
are moved to archive database on monthly basis after month is closed and data is
locked against changes.
Normally users work only on online table but in some cases they need to see data
from all the history.
plProxy can used to also query data from archive database when user requests data
for longer period than online database holds.
onlineDB archiveDB
19. plProxy: Remote Calls for Autonomous Transactions
PostgreSQL does not have autonomous transactions but plProxy calls can be used
for mimicking them.
Autonomous transactions can be useful for logging failed procedure calls.
If used extensively then usage of pgBouncer should be considered to reduce
number of additional connections needed and reducing connection creation
overhead.
20. plProxy: Proxy Databases as Interface
Additional layer between application and databases.
Keep applications database connectivity simpler giving DBA's and developer's more
flexibility for moving data and functionality around.
It gets really useful when number of databases gets bigger then connectivity
management outside database layer gets too complicated :)
BackOffice
Application
shopDb backofficeDb
(proxy) (proxy)
shopDb userDB internalDb
21. plProxy: Proxy Databases for Security
Security layer. By giving access to proxy database DBA's can be sure that user has
no way of accessing tables by accident or by any other means as only functions
published in proxy database are visible to user.
Such proxy databases may be located in servers visible from outside network while
databases containing data are usually not.
BackOffice
Application
manualfixDb ccinterfaceDb
(proxy) (proxy)
shopDb creditcardDb
22. plProxy: Run On All for Data
Also usable when exact partition where data resides is not known. Then
function may be run on all partitions and only the one that has data does
something. CREATE FUNCTION get_balances(
i_users text[],
CREATE FUNCTION balance.get_balances( OUT user text,
i_users text[], OUT currency text,
OUT username text, OUT balance numeric)
OUT currency text, RETURNS SETOF record AS $$
OUT balance numeric) CLUSTER 'userdb'; RUN ON ALL;
RETURNS SETOF record AS $$ $$ LANGUAGE plproxy;
BEGIN
FOR i IN COALESCE(array_lower(i_users,1),0) ..
COALESCE(array_upper(i_users,1),-1)
LOOP
IF partconf.valid_hash(i_users[i]) THEN
SELECT i_users[i], u.currency, u.balance
INTO username, currency, balance userdb
FROM balances
WHERE username = i_users[i];
RETURN NEXT;
END IF;
END LOOP;
RETURN;
END;
$$ LANGUAGE plpgsql SECURITY DEFINER; userdb_p0 userdb_p1 userdb_p2 userdb_p3
23. plProxy: Run On All for for Stats
Gather statistics from all the partitions CREATE FUNCTION stats._get_stats(
OUT stat_name text,
and return summaries to the caller OUT stat_value bigint)
CREATE FUNCTION stats._get_stats( RETURNS SETOF record AS $$
OUT stat_name text, declare
OUT stat_value bigint) seq record;
begin
RETURNS SETOF record AS $$ for seq in
cluster 'userdb'; select c.relname,
run on all; 'select last_value from stats.'
$$ LANGUAGE plproxy; || c.relname as sql
from pg_class c, pg_namespace n
CREATE FUNCTION stats.get_stats( where n.nspname = 'stats'
OUT stat_name text, and c.relnamespace = n.oid
OUT stat_value bigint) and c.relkind = 'S'
RETURNS SETOF record AS $$ order by 1
select stat_name, loop
(sum(stat_value))::bigint stat_name := seq.relname;
from stats._get_stats() execute seq.sql into stat_value;
group by stat_name return next;
order by stat_name; end loop;
$$ LANGUAGE sql; return;
end;
$$ LANGUAGE plpgsql;
24. plProxy: Geographical Partitioning
plProxy can be used to split database into partitions based on country code.
Example database is split into 'us' and 'row' (rest of the world)
Each function call caused by online users has country code as one of the
parameters
All data is replicated into backend database for use by backoffice applications and
batch jobs. That also reduces number of indexes needed in online databases.
CREATE FUNCTION get_cluster(
i_key_cc text)
onlinedb
RETURNS text AS $$ (proxy)
BEGIN
IF i_key_cc = 'ru' THEN
RETURN 'oltp_ru';
ELSE
RETURN 'oltp_row'; onlinedb_RU onlinedb_ROW backenddb
END IF;
END;
$$ LANGUAGE plpgsql;
25. plProxy: Application Driven Partitioning
Sometimes same function can be run on read only replica and on online database
but the right partition is known only on runtime depending on context.
In that situation get_cluster can be used to allow calling application to choose where
this function will be executed.
CREATE FUNCTION get_cluster(i_refresh boolean)
RETURNS text AS $$
BEGIN
IF i_refresh THEN
return 'shopdb';
ELSE shopdb
return 'shopdb_ro'; (proxy)
END IF;
END; $$ LANGUAGE plpgsql;
shopdb_RW shopdb_RO shopdb_RO
26. plProxy: Horizontal Partitioning
We have partitioned most of our database by username using PostgreSQL hashtext
function to get equal distribution between partitions.
As proxy databases are stateless we can have several exact duplicates for load
balancing and high availability.
CREATE FUNCTION public.get_user_email(text)
RETURNS text AS $$
cluster 'userdb';
run on get_hash($1);
$$ LANGUAGE plproxy;
CREATE FUNCTION public.get_hash(
i_key_user text) userdb userdb
RETURNS integer AS $$
BEGIN
return hashtext(lower(i_key_user));
END;
$$ LANGUAGE plpgsql;
userdb_p0 userdb_p1 userdb_p2 userdb_p3
27. plProxy: Partitioning Partition Functions
check_hash used to validate incoming calls
partconf functionality to handle sequences
validate_hash function in triggers to exclude data during split
CREATE FUNCTION public.get_user_email( i_username text)
RETURNS text AS $$
DECLARE
retval text;
BEGIN
PERFORM partconf.check_hash(i_username);
SELECT email FROM users
WHERE username = lower(i_username) INTO retval;
RETURN retval;
END;
$$ LANGUAGE plpgsql;
28. plProxy: Horizontal Split
When splitting databases we usually prepare new partitions in other servers and
then switch all traffic at once to keep our life simple.
Steps how to increase partition count by 2x:
Create 2 partitions for each old one.
Replicate partition data to corresponding new partitions
Delete unnecessary data from new partitions and cluster and analyze tables
Prepare new plProxy configuration
Switch over to them
Take down ip
Up plproxy version
Wait replica to catch up
Set ip up
You can have several partitions in one server. Moving them out later will be easy.
userdb_a0 userdb_a1 userdb_b0 userdb_b1 userdb_b2 userdb_b3
29. plProxy: Partconf
Partconf is useful when you have split database into several partitions and there are
also failover replicas of same databases.
Some mechanism for handling sequences id's is needed
Some mechanism for protecting from misconfiguration makes life safer
CREATE FUNCTION partconf.set_conf(
i_part_nr integer ,
i_max_part integer ,
i_db_code integer )
Used when partconf is installed to initialize it. db_code is supposed to be unique
over all the databases.
CREATE FUNCTION partconf.check_hash(key_user text) RETURNS boolean
used internally from other functions to validate that function call was directed to
correct partition. Raises error if not right partition for that username.
CREATE FUNCTION global_id() RETURNS bigint
used to get globally unique id's
30. plProxy: Summary
plProxy adds very little overhead when used together with pgBouncer.
On the other hand plProxy adds complexity to development and
maintenace so it must be used with care but that is true for most
everything.
32. pgBouncer
pgBouncer is lightweight and robust connection pooler for Postgres.
Low memory requirements (2k per connection by default). This is due to the fact that
PgBouncer does not need to see full packet at once.
It is not tied to one backend server, the destination databases can reside on
different hosts.
Supports online reconfiguration for most of the settings.
Supports online restart/upgrade without dropping client connections.
Supports protocol V3 only, so backend version must be >= 7.4.
Does not parse SQL so is very fast and uses little CPU time.
33. pgBouncer Pooling Modes
Session pooling - Most polite method. When client connects, a server connection
will be assigned to it for the whole duration it stays connected. When client
disconnects, the server connection will be put back into pool. Should be used with
legacy applications that won't work with more efficient pooling modes.
Transaction pooling - Server connection is assigned to client only during a
transaction. When PgBouncer notices that transaction is over, the server will be put
back into pool. This is a hack as it breaks application expectations of backend
connection. You can use it only when application cooperates with such usage by not
using features that can break.
Statement pooling - Most aggressive method. This is transaction pooling with a
twist - multi-statement transactions are disallowed. This is meant to enforce
"autocommit" mode on client, mostly targeted for PL/Proxy.
35. pgBouncer console
Connect to database 'pgbouncer':
$ psql -d pgbouncer -U admin -p 6432
SHOW HELP;
SHOW CONFIG: SHOW DATABASES;
SET default_pool_size = 50;
RELOAD;
SHOW POOLS; SHOW STATS;
PAUSE; PAUSE <db>;
RESUME; RESUME <db>;
36. pgBouncer Use-cases
Decreasing the number of connections hitting real databases
Taking the logging-in load from real databases
Central redirection point for database connections. Function s like
ServiceIP. Allows pausing, redirecting of users.
Backend pooler for PL/Proxy
Database aliases. You can have different external names connected to
one DSN. Makes it possible to pause them separately and merge
databases.
38. SkyTools and PgQ
We keep SkyTools like pgBouncer and plProxy in our standard database server
image so each new server has it installed from the start.
Framework provides database connectivity, logging, stats management, encoding,
decoding etc for batch jobs. Developers need only to take care of business logic in
batch jobs all the rest is handled by batch jobs.
SkyTools contains many reusable generic scripts.
39. PgQ: PostgreSQL Queuing
Ideally suited for all kinds of batch processing. onlinedb
SkyTools
Efficient. Creating events is adds very little
overhead and queue has beautiful interface that
allows events to be created using standard SQL
insert into syntax. queuedb
Fast. Events are processed in batches that usually SkyTools
means higher speed when processing.
Flexible. Each queue can have several producers
and consumers and any number of event types
handled in it. batchdb
Robust. PgQ guarantees that each consumers
sees event at least once. There several methods
how the processed events can be tracked
depending on business needs.
batch
job
40. PgQ - Setup
Basic PgQ setup and usage can be illustrated by the following steps:
1. create the database
2. edit a PgQ ticker configuration file, say ticker.ini
3. install PgQ internal tables
$ pgqadm.py ticker.ini install
4. launch the PgQ ticker on database machine as daemon
$ pgqadm.py -d ticker.ini ticker
5. create queue
$ pgqadm.py ticker.ini create <queue>
6. register or run consumer to register it automatically
$ pgqadm.py ticker.ini register <queue> <consumer>
7. start producing events
41. PgQ – Configuration Ticker reads event id sequence for
each queue.
[pgqadm] If new events have appeared, then
job_name = pgqadm_somedb inserts tick if:
Configurable amount of events
db = dbname=somedb have appeared
ticker_max_count (500)
# how often to run maintenance Configurable amount of time has
maint_delay = 600 passed from last tick
ticker_max_lag (3 sec)
# how often to check for activity If no events in the queue, creates tick
loop_delay = 0.1 if some time has passed.
ticker_idle_period (60
logfile = ~/log/%(job_name)s.log sec)
pidfile = ~/pid/%(job_name)s.pid Configuring from command line:
pgqadm.py ticker.ini
config my_queue
ticker_max_count=100
42. PgQ event structure
CREATE TABLE pgq.event (
ev_id int8 NOT NULL,
ev_txid int8 NOT NULL DEFAULT txid_current(),
ev_time timestamptz NOT NULL DEFAULT now(),
-- rest are user fields --
ev_type text, -- what to expect from ev_data
ev_data text, -- main data, urlenc, xml, json
ev_extra1 text, -- metadata
ev_extra2 text, -- metadata
ev_extra3 text, -- metadata
ev_extra4 text -- metadata
);
CREATE INDEX txid_idx ON pgq.event (ev_txid);
43. PgQ: Producer - API Event Insertion
Single event insertion:
pgq.insert_event(queue, ev_type, ev_data): int8
Bulk insertion, in single transaction:
pgq.current_event_table(queue): text
Inserting with triggers:
pgq.sqltriga(queue, ...) - partial SQL format
pgq.logutriga(queue, ...) - urlencoded format
44. PgQ API: insert complex event with pure SQL
Create interface table for queue with logutriga for encoding inserted events
Type safety, default values, sequences, constraints!
Several tables can insert into same queue.
Encoding decoding is totally hidden from developers they just work with tables.
CREATE TABLE queue.mailq (
mk_template bigint NOT NULL,
username text NOT NULL,
email text,
mk_language text,
payload text
);
CREATE TRIGGER send_mailq_trg
BEFORE INSERT ON queue.mailq
FOR EACH ROW
EXECUTE PROCEDURE pgq.logutriga('mail_events', 'SKIP');
-- send e-mail
INSERT INTO queue.mailq (mk_template, username, email)
VALUES(73, _username, _email);
46. PgQ: Tracking Events
Per-event overhead
Need to avoid accumulating
pgq_ext solution
pgq_ext.is_event_done(consumer, batch_id, ev_id)
pgq_ext.set_event_done(consumer, batch_id, ev_id)
If batch changes, deletes old events
Eg. email sender, plProxy.
47. PgQ: Tracking Batches
Minimal per-event overhead
Requires that all batch is processed in one TX
pgq_ext.is_batch_done(consumer, batch_id)
pgq_ext.set_batch_done(consumer, batch_id)
Eg. replication, most of the SkyTools partitioning scripts like queue_mover,
queue_splitter, table_dispatcher, cube_dispatcher.
48. PgQ: Queue Mover
Moves data from source queue in one database to another queue in other database.
Used to move events from online databases to other databases for processing or
just storage.
Consolidates events if there are several producers as in case of partitioned
databases.
[queue_mover]
job_name = eventlog_to_target_mover
src_db = dbname=oltpdb
dst_db = dbname=batchdb
userdb_p1 queue
pgq_queue_name = batch_events mover
dst_queue_name = batch_events logdb
queue
pidfile = log/%(job_name)s.pid userdb_p2
logfile = pid/%(job_name)s.log mover
49. PgQ: Queue Mover Code
import pgq
class QueueMover(pgq.RemoteConsumer):
def process_remote_batch(self, db, batch_id, ev_list, dst_db):
# prepare data
rows = []
for ev in ev_list:
rows.append([ev.type, ev.data, ev.time])
ev.tag_done()
# insert data
fields = ['ev_type', 'ev_data', 'ev_time']
curs = dst_db.cursor()
dst_queue = self.cf.get('dst_queue_name')
pgq.bulk_insert_events(curs, rows, fields, dst_queue)
script = QueueMover('queue_mover', 'src_db', 'dst_db', sys.argv[1:])
script.start()
50. PgQ: Queue Splitter
Moves data from source queue in one database to one or more
queue's in target database based on producer. That is another version
of queue_mover but it has it's own benefits.
Used to move events from online databases to queue databases.
Reduces number of dependencies of online databases.
database: promotional
database:
userdb_pN mailer
mailqdb
producer: queue;
welcome_emails welcome_emails
producer: queue:
password_emails password_emails
transactional
queue: mailer
batch_events queue
splitter
51. PgQ: Simple Consumer
Simplest consumer. Basic idea is to read url encoded events from queue and
execute SQL using fields from those events as parameters.
Very convenient when you have events generated in one database and want to get
background processing or play them into another database.
Does not guarantee that each event is processed only once so it's up to the called
function to guarantee that duplicate calls will be ignored.
[simple_consumer]
job_name = job_to_be_done
src_db = dbname=sourcedb
dst_db = dbname=targetdb
pgq_queue_name = event_queue
dst_query =
SELECT * FROM function_to_be_run(%%(username)s, %%(email)s);
logfile = ~/log/%(job_name)s.log
pidfile = ~/pid/%(job_name)s.pid
52. PgQ: Simple Serial Consumer
Similar to simpler consumer but uses pgq_ext to guarantee that each event will be
processed only once.
import sys, pgq, skytools
class SimpleSerialConsumer(pgq.SerialConsumer):
def __init__(self, args):
pgq.SerialConsumer.__init__(self,"simple_serial_consumer","src_db","dst_db", args)
self.dst_query = self.cf.get("dst_query")
def process_remote_batch(self, batch_id, ev_list, dst_db):
curs = dst_db.cursor()
for ev in ev_list:
payload = skytools.db_urldecode(ev.payload)
curs.execute(self.dst_query, payload)
res = curs.dictfetchone()
ev.tag_done()
if __name__ == '__main__':
script = SimpleSerialConsumer(sys.argv[1:])
script.start()
53. PgQ: Table Dispatcher
Has url encoded events as data source and writes them into table on target
database.
Used to partiton data. For example change log's that need to kept online only shortly
can be written to daily tables and then dropped as they become irrelevant.
Also allows to select which columns have to be written into target database
Creates target tables according to configuration file as needed
table table:history
dispatcher history_2008_09
history_2008_10
...
table
queue: dispatcher table:cr
call_records cr_2008_09_01
cr_2008_09_02
...
54. PgQ: Cube Dispatcher
Has url encoded events as data source and writes them into partitoned tables in
target database. Logutriga is used to create events.
Used to provide batches of data for business intelligence and data cubes.
Only one instance of each record is stored. For example if record is created and
then updated twice only latest version of record stays in that days table.
Does not support deletes.
table:invoices
invoices_2008_09_01
producer: invoices_2008_09_02
invoices ...
producer: table:payments
payments cube
dispatcher payments_2008_09_01
payments_2008_09_02
queue: ...
cube_events
56. Londiste: Replication
We use replication
to transfer online data into internal databases
to create failover databases for online databases.
to switch between PostgreSQL versions
to distribute internally produced data into online servers
to create read only replicas for load balancing
Londiste is implemented using PgQ as transport layer.
It has DBA friendly command line interface.
shopdb analysisdb
(online) (internal)
shopdb_ro shopdb
(read only) (failover)
57. Londiste: Setup
1. create the subscriber database, with tables to replicate
2. edit a londiste configuration file, say conf.ini, and a PgQ ticker
configuration file, say ticker.ini
3. install londiste on the provider and subscriber nodes. This step
requires admin privileges on both provider and subscriber sides,
and both install commands can be run remotely:
$ londiste.py conf.ini provider install
$ londiste.py conf.ini subscriber install
4. launch the PgQ ticker on the provider machine:
$ pgqadm.py -d ticker.ini ticker
5. launch the londiste replay process:
$ londiste.py -d conf.ini replay
6. add tables to replicate from the provider database:
$ londiste.py conf.ini provider add table1 table2 ...
7. add tables to replicate to the subscriber database:
$ londiste.py conf.ini subscriber add table1 table2 ...
58. Londiste: Configuration
[londiste]
job_name = londiste_providerdb_to_subscriberdb
provider_db = dbname=provider port=5432 host=127.0.0.1
subscriber_db = dbname=subscriber port=6432 host=127.0.0.1
# it will be used as sql ident so no dots/spaces
pgq_queue_name = londiste.replika
logfile = /tmp/%(job_name)s.log
pidfile = /tmp/%(job_name)s.pid
59. Londiste: Usage
Tables can be added one-by-one into set.
Initial COPY for one table does not block event replay for other tables.
Can compare tables on both sides.
Supports sequences.
There can be several subscribers listening to one provider queue.
Each subscriber can listen to any number of tables provided by
provider all the other events are ignored.
61. SkyTools: WalMgr
Used to creat standby servers inside same colocation.
Also provides point in time recovery.
Requires more network bandwidth than londiste but has very little
management overhead.
Most common usage is when servers starts showing signs of braking
down then we move database into gresh server until other one gets
fixed.
Also upgrade on better hardware is quite common.
62. SODI Framework
Rapid application development Application layer
Cheap changes - java / php ...
- UI logic
Most of the design is in metadata.
. mostly generated
Application design stays in sync with application over - SODI framework
time.
AppServer layer:
Minimizes number of database roundtrips. - java / php( ...
Can send multiple recordsets to one function call. - user authentication
- roles rights
Can receive several recordsets from function call. - no business logic
Database layer
- business logic
- plPython
- PostgreSQL
- SkyTools
63. Multiple Recordsets in Function Resultset
CREATE FUNCTION meta.get_iotype(i_context text, i_params text)
RETURNS SETOF public.ret_multiset AS $$
import dbservice2
dbs = dbservice2.ServiceContext( i_context, GD )
params = dbs.get_record( i_params )
sql = """
select id_iotype, mod_code, schema || '.' || iotype_name as iotype
, comments ,version
from meta.iotype
where id_iotype = {id_iotype}
"""
dbs.return_next_sql( sql, params, 'iotype' )
sql = """
select id_iotype_attr, key_iotype ,attr_name, mk_datatype
, comments, label, version, order_by, br_codes
from meta.iotype_attr
where key_iotype = {id_iotype}
order by order_by
"""
dbs.return_next_sql( sql, params, 'ioattr')
return dbs.retval()
$$ LANGUAGE plpythonu;
64. Multiple Recordsets as Function Parameters
CREATE FUNCTION dba.set_role_grants(
i_context text,
i_params text,
i_roles text,
i_grants text)
RETURNS SETOF public.ret_multiset AS $$
import dbservice2
dbs = dbservice2.ServiceContext( i_context, GD )
params = dbs.get_record( i_params )
t_roles = dbs.get_record_list( i_roles )
t_grants = dbs.get_record_list( i_grants )
...
for r in t_roles:
key_role = roles.get(r.role_name)
r.id_host = params.id_host
r.key_role = key_role
if key_role is None:
dbs.run_query( """
insert into dba.role (
key_host, role_name, system_privs, role_options, password_hash
, connection_limit, can_login, granted_roles
) values (
{id_host},{role_name},{system_privs},{role_options},{password_hash}
, {connection_limit},{can_login},string_to_array({granted_roles},','))
""", r)
dbs.tell_user(dbs.INFO, 'dbs1213', 'Role added: %s' % r.role_name)
...