This document provides instructions for minimizing downtime when performing a major version upgrade of PostgreSQL using logical replication with Slony. It discusses various methods for performing the upgrade, including dump/restore, pg_upgrade, and logical replication with Slony. It then provides a step-by-step guide to setting up logical replication between two PostgreSQL nodes using Slony, including initializing the cluster and nodes, creating replication sets, subscribing nodes, and monitoring the initial synchronization process. The document demonstrates how Slony allows performing a graceful switchover and switchback between nodes when upgrading PostgreSQL versions.
Spencer Christensen
There are many aspects to managing an RDBMS. Some of these are handled by an experienced DBA, but there are a good many things that any sys admin should be able to take care of if they know what to look for.
This presentation will cover basics of managing Postgres, including creating database clusters, overview of configuration, and logging. We will also look at tools to help monitor Postgres and keep an eye on what is going on. Some of the tools we will review are:
* pgtop
* pg_top
* pgfouine
* check_postgres.pl.
Check_postgres.pl is a great tool that can plug into your Nagios or Cacti monitoring systems, giving you even better visibility into your databases.
PgCenter is a tool for monitoring and troubleshooting PostgreSQL. It provides a graphical interface to view key performance metrics and statuses. Some of its main features include displaying server health, load, memory and disk usage, statement performance, replication status and more. It aims to help PostgreSQL administrators quickly check the health of their databases and identify potential problems.
This document discusses using PostgreSQL statistics to optimize performance. It describes various statistics sources like pg_stat_database, pg_stat_bgwriter, and pg_stat_replication that provide information on operations, caching, and replication lag. It also provides examples of using these sources to identify issues like long transactions, temporary file growth, and replication delays.
The document discusses strategic autovacuum configuration and monitoring in PostgreSQL. It begins by explaining the ACID properties and how MVCC and transactions work. It then discusses how to monitor workloads for heavily updated tables, adjust per-table autovacuum thresholds to prioritize those tables, monitor autovacuum behavior over time using logs and queries, and tune the autovacuum throttle settings based on that monitoring to optimize autovacuum performance. The key steps are to start with defaults, monitor workload changes, adjust settings for busy tables, continue monitoring, and refine settings as needed.
The document provides configuration instructions and guidelines for setting up streaming replication between a PostgreSQL master and standby server, including setting parameter values for wal_level, max_wal_senders, wal_keep_segments, creating a dedicated replication role, using pg_basebackup to initialize the standby, and various recovery target options to control the standby's behavior. It also discusses synchronous replication using replication slots and monitoring the replication process on both the master and standby servers.
Non-Relational Postgres / Bruce Momjian (EnterpriseDB)Ontico
Postgres has always had strong support for relational storage. However, there are many cases where relational storage is either inefficient or overly restrictive. This talk shows the many ways that Postgres has expanded to support non-relational storage, specifically the ability to store and index multiple values, even unrelated ones, in a single database field. Such storage allows for greater efficiency and access simplicity, and can also avoid the negatives of entity-attribute-value (eav) storage. The talk will cover many examples of multiple-value-per-field storage, including arrays, range types, geometry, full text search, xml, json, and records.
This document discusses advanced Postgres monitoring. It begins with an introduction of the speaker and an agenda for the discussion. It then covers selection criteria for monitoring solutions, compares open source and SAAS monitoring options, and provides examples of collecting specific Postgres metrics using CollectD. It also discusses alerting, handling monitoring changes, and being prepared to respond to incidents outside of normal hours.
Spencer Christensen
There are many aspects to managing an RDBMS. Some of these are handled by an experienced DBA, but there are a good many things that any sys admin should be able to take care of if they know what to look for.
This presentation will cover basics of managing Postgres, including creating database clusters, overview of configuration, and logging. We will also look at tools to help monitor Postgres and keep an eye on what is going on. Some of the tools we will review are:
* pgtop
* pg_top
* pgfouine
* check_postgres.pl.
Check_postgres.pl is a great tool that can plug into your Nagios or Cacti monitoring systems, giving you even better visibility into your databases.
PgCenter is a tool for monitoring and troubleshooting PostgreSQL. It provides a graphical interface to view key performance metrics and statuses. Some of its main features include displaying server health, load, memory and disk usage, statement performance, replication status and more. It aims to help PostgreSQL administrators quickly check the health of their databases and identify potential problems.
This document discusses using PostgreSQL statistics to optimize performance. It describes various statistics sources like pg_stat_database, pg_stat_bgwriter, and pg_stat_replication that provide information on operations, caching, and replication lag. It also provides examples of using these sources to identify issues like long transactions, temporary file growth, and replication delays.
The document discusses strategic autovacuum configuration and monitoring in PostgreSQL. It begins by explaining the ACID properties and how MVCC and transactions work. It then discusses how to monitor workloads for heavily updated tables, adjust per-table autovacuum thresholds to prioritize those tables, monitor autovacuum behavior over time using logs and queries, and tune the autovacuum throttle settings based on that monitoring to optimize autovacuum performance. The key steps are to start with defaults, monitor workload changes, adjust settings for busy tables, continue monitoring, and refine settings as needed.
The document provides configuration instructions and guidelines for setting up streaming replication between a PostgreSQL master and standby server, including setting parameter values for wal_level, max_wal_senders, wal_keep_segments, creating a dedicated replication role, using pg_basebackup to initialize the standby, and various recovery target options to control the standby's behavior. It also discusses synchronous replication using replication slots and monitoring the replication process on both the master and standby servers.
Non-Relational Postgres / Bruce Momjian (EnterpriseDB)Ontico
Postgres has always had strong support for relational storage. However, there are many cases where relational storage is either inefficient or overly restrictive. This talk shows the many ways that Postgres has expanded to support non-relational storage, specifically the ability to store and index multiple values, even unrelated ones, in a single database field. Such storage allows for greater efficiency and access simplicity, and can also avoid the negatives of entity-attribute-value (eav) storage. The talk will cover many examples of multiple-value-per-field storage, including arrays, range types, geometry, full text search, xml, json, and records.
This document discusses advanced Postgres monitoring. It begins with an introduction of the speaker and an agenda for the discussion. It then covers selection criteria for monitoring solutions, compares open source and SAAS monitoring options, and provides examples of collecting specific Postgres metrics using CollectD. It also discusses alerting, handling monitoring changes, and being prepared to respond to incidents outside of normal hours.
This document provides an overview of PostgreSQL and instructions for installing and configuring it. It discusses using the initdb command to initialize a PostgreSQL database cluster and create the template1 and postgres databases. It also explains that the template1 database serves as a template that is copied whenever new databases are created.
This document summarizes PL/Java, which allows writing server-side functions in Java for PostgreSQL. It discusses how to define and deploy Java functions, configure PL/Java, handle parameters and return types, use JDBC from functions, and write triggers in Java. While compatible with Oracle's SQL/JRT standard, PL/Java has some limitations around memory usage and performance. It works best on Linux and is a stable option for adding Java code to PostgreSQL databases.
The document discusses the glance-replicator tool in OpenStack. Glance-replicator allows replication of images between two glance servers. It can replicate images and also import and export images. The document provides examples of using glance-replicator commands like compare, livecopy to replicate images between two devstack all-in-one OpenStack environments. It demonstrates the initial state with only one environment having images and after replication both environments having the same set of images.
Postgresql 12 streaming replication holVijay Kumar N
This is a step by step hands on lab for PostgreSQL 12 , setup of replication, replication slot, failover (promoting) to standby as new master cluster and also covering the scenario where old master has to be reinstated using the utility "pg_rewind"
PGDay.Amsterdam 2018 - Stefan Fercot - Save your data with pgBackRestPGDay.Amsterdam
Ever heard of Point-in-time recovery? pgBackRest is an awsome tool to handle backups, restores and even helps you build streaming replication ! This talk will introduce the tool, its basic features and how to use it.
This presentation covers all aspects of PostgreSQL administration, including installation, security, file structure, configuration, reporting, backup, daily maintenance, monitoring activity, disk space computations, and disaster recovery. It shows how to control host connectivity, configure the server, find the query being run by each session, and find the disk space used by each database.
This document provides an overview of pgCenter, a tool for managing and monitoring PostgreSQL databases. It describes pgCenter's interface which displays system metrics, PostgreSQL statistics and additional information. The interface shows values for items like CPU and memory usage, database connections, autovacuum operations, and query information. PgCenter provides a quick way to view real-time PostgreSQL and server performance metrics.
Top 10 Mistakes When Migrating From Oracle to PostgreSQLJim Mlodgenski
As more and more people are moving to PostgreSQL from Oracle, a pattern of mistakes is emerging. They can be caused by the tools being used or just not understanding how PostgreSQL is different than Oracle. In this talk we will discuss the top mistakes people generally make when moving to PostgreSQL from Oracle and what the correct course of action.
1. A PostgreSQL database outage occurred at GitLab on January 31st due to a combination of factors including an increase in load, replication lag, and the deletion of the database directory.
2. Lessons learned include monitoring replication, using tools like pg_basebackup properly, and having backups and disaster recovery processes in place.
3. Recommended preventative measures include setting sane configuration values, automated testing of backups, assigning an owner for data durability, and improving documentation.
The document discusses sessionization with Spark streaming to analyze user sessions from a constant stream of page visit data. Key points include:
- Streaming page visit data presents challenges like joining new visits to ongoing sessions and handling variable data volumes and long user sessions.
- The proposed solution uses Spark streaming to join a checkpoint of incomplete sessions with new visit data to calculate session metrics in real-time.
- Important aspects are controlling data ingress size and partitioning to optimize performance of operations like joins and using custom formats to handle output to multiple sinks.
This document discusses setting up MySQL auditing using the Percona Audit Plugin and ELK (Elasticsearch, Logstash, Kibana) stack to retrieve and analyze MySQL logs. Key steps include installing the Percona Audit Plugin on MySQL servers, configuring it to log to syslog, installing and configuring rsyslog/syslog-ng on database and ELK servers to forward logs, and installing and configuring the ELK stack including Elasticsearch, Logstash, and Kibana to index and visualize the logs. Examples are provided of creating searches, graphs, and dashboards in Kibana for analyzing the MySQL audit logs.
pg_proctab: Accessing System Stats in PostgreSQLMark Wong
pg_proctab is a collection of PostgreSQL stored functions that provide access to the operating system process table using SQL. We'll show you which functions are available and where they collect the data, and give examples of their use to collect processor and I/O statistics on SQL queries.
PostgreSQL is one of the most advanced relational databases. It offers superb replication capabilities. The most important features are: Streaming replication, Point-In-Time-Recovery, advanced monitoring, etc.
Oracle goldengate 11g schema replication from standby databaseuzzal basak
GoldenGate can replicate database schemas between an Oracle source and target database. It was configured to replicate the SCOTT schema from a source Oracle 11gR2 database in standby mode to a target Oracle 11gR2 database. The key steps included enabling supplemental logging on the source, setting up the GoldenGate user and processes on both databases, and defining the extract, pump and replicate processes to copy data and DDL changes from the source to the target schema.
This document describes setting up and testing ProxySQL for query routing and high availability with Percona XtraDB Cluster (PXC). It includes instructions for installing and configuring ProxySQL, adding backend PXC servers, creating query rules for routing, and testing read/write splitting and failover through sysbench tests. Failover is demonstrated by stopping one PXC node, and ProxySQL is shown routing queries to the remaining nodes and marking the failed node as offline in its status.
MySQL exposes a collection of tunable parameters and indicators that is frankly intimidating. But a poorly tuned MySQL server is a bottleneck for your PHP application scalability. This session shows how to do InnoDB tuning and read the InnoDB status report in MySQL 5.5.
PostgreSQL Procedural Languages: Tips, Tricks and GotchasJim Mlodgenski
One of the most powerful features of PostgreSQL is its diversity of procedural languages, but with that diversity comes a lot of options.
Did you ever wonder:
- What all of those options are on the CREATE FUNCTION statement?
- How do they affect my application?
- Does my choice of procedural language affect the performance of my statements?
- Should I create a single trigger with IF statements or several simple triggers?
- How do I debug my code?
- Can I tell which line in my function is taking all of the time?
This document discusses different types of enqueue waits caused by locks in an Oracle database. It covers common lock names like TX, TM, and UL; lock modes like exclusive and share; and how to identify the waiter, blocker, lock type, and object being blocked using views like v$lock, v$session, and v$active_session_history. The key pieces of information needed to resolve lock waits are the session ID of the waiter, the lock mode, and the object being blocked. User locks like TX and TM locks result from transactions and table modifications, while internal locks govern objects like redo and reusable objects.
Setup oracle golden gate 11g replicationKanwar Batra
How to setup Oracle Goldengate Replication between 11gR2 RAC or Single node instances. For RAC setup the GoldenGate custom cluster service . Not part of this document
[Pgday.Seoul 2019] Citus를 이용한 분산 데이터베이스PgDay.Seoul
This document summarizes how to set up and use Citus, an open-source PostgreSQL-based distributed database. It explains how to install Citus, add worker nodes, create distributed tables, and use features like reference tables to perform distributed queries across the cluster.
This document provides an overview of PostgreSQL and instructions for installing and configuring it. It discusses using the initdb command to initialize a PostgreSQL database cluster and create the template1 and postgres databases. It also explains that the template1 database serves as a template that is copied whenever new databases are created.
This document summarizes PL/Java, which allows writing server-side functions in Java for PostgreSQL. It discusses how to define and deploy Java functions, configure PL/Java, handle parameters and return types, use JDBC from functions, and write triggers in Java. While compatible with Oracle's SQL/JRT standard, PL/Java has some limitations around memory usage and performance. It works best on Linux and is a stable option for adding Java code to PostgreSQL databases.
The document discusses the glance-replicator tool in OpenStack. Glance-replicator allows replication of images between two glance servers. It can replicate images and also import and export images. The document provides examples of using glance-replicator commands like compare, livecopy to replicate images between two devstack all-in-one OpenStack environments. It demonstrates the initial state with only one environment having images and after replication both environments having the same set of images.
Postgresql 12 streaming replication holVijay Kumar N
This is a step by step hands on lab for PostgreSQL 12 , setup of replication, replication slot, failover (promoting) to standby as new master cluster and also covering the scenario where old master has to be reinstated using the utility "pg_rewind"
PGDay.Amsterdam 2018 - Stefan Fercot - Save your data with pgBackRestPGDay.Amsterdam
Ever heard of Point-in-time recovery? pgBackRest is an awsome tool to handle backups, restores and even helps you build streaming replication ! This talk will introduce the tool, its basic features and how to use it.
This presentation covers all aspects of PostgreSQL administration, including installation, security, file structure, configuration, reporting, backup, daily maintenance, monitoring activity, disk space computations, and disaster recovery. It shows how to control host connectivity, configure the server, find the query being run by each session, and find the disk space used by each database.
This document provides an overview of pgCenter, a tool for managing and monitoring PostgreSQL databases. It describes pgCenter's interface which displays system metrics, PostgreSQL statistics and additional information. The interface shows values for items like CPU and memory usage, database connections, autovacuum operations, and query information. PgCenter provides a quick way to view real-time PostgreSQL and server performance metrics.
Top 10 Mistakes When Migrating From Oracle to PostgreSQLJim Mlodgenski
As more and more people are moving to PostgreSQL from Oracle, a pattern of mistakes is emerging. They can be caused by the tools being used or just not understanding how PostgreSQL is different than Oracle. In this talk we will discuss the top mistakes people generally make when moving to PostgreSQL from Oracle and what the correct course of action.
1. A PostgreSQL database outage occurred at GitLab on January 31st due to a combination of factors including an increase in load, replication lag, and the deletion of the database directory.
2. Lessons learned include monitoring replication, using tools like pg_basebackup properly, and having backups and disaster recovery processes in place.
3. Recommended preventative measures include setting sane configuration values, automated testing of backups, assigning an owner for data durability, and improving documentation.
The document discusses sessionization with Spark streaming to analyze user sessions from a constant stream of page visit data. Key points include:
- Streaming page visit data presents challenges like joining new visits to ongoing sessions and handling variable data volumes and long user sessions.
- The proposed solution uses Spark streaming to join a checkpoint of incomplete sessions with new visit data to calculate session metrics in real-time.
- Important aspects are controlling data ingress size and partitioning to optimize performance of operations like joins and using custom formats to handle output to multiple sinks.
This document discusses setting up MySQL auditing using the Percona Audit Plugin and ELK (Elasticsearch, Logstash, Kibana) stack to retrieve and analyze MySQL logs. Key steps include installing the Percona Audit Plugin on MySQL servers, configuring it to log to syslog, installing and configuring rsyslog/syslog-ng on database and ELK servers to forward logs, and installing and configuring the ELK stack including Elasticsearch, Logstash, and Kibana to index and visualize the logs. Examples are provided of creating searches, graphs, and dashboards in Kibana for analyzing the MySQL audit logs.
pg_proctab: Accessing System Stats in PostgreSQLMark Wong
pg_proctab is a collection of PostgreSQL stored functions that provide access to the operating system process table using SQL. We'll show you which functions are available and where they collect the data, and give examples of their use to collect processor and I/O statistics on SQL queries.
PostgreSQL is one of the most advanced relational databases. It offers superb replication capabilities. The most important features are: Streaming replication, Point-In-Time-Recovery, advanced monitoring, etc.
Oracle goldengate 11g schema replication from standby databaseuzzal basak
GoldenGate can replicate database schemas between an Oracle source and target database. It was configured to replicate the SCOTT schema from a source Oracle 11gR2 database in standby mode to a target Oracle 11gR2 database. The key steps included enabling supplemental logging on the source, setting up the GoldenGate user and processes on both databases, and defining the extract, pump and replicate processes to copy data and DDL changes from the source to the target schema.
This document describes setting up and testing ProxySQL for query routing and high availability with Percona XtraDB Cluster (PXC). It includes instructions for installing and configuring ProxySQL, adding backend PXC servers, creating query rules for routing, and testing read/write splitting and failover through sysbench tests. Failover is demonstrated by stopping one PXC node, and ProxySQL is shown routing queries to the remaining nodes and marking the failed node as offline in its status.
MySQL exposes a collection of tunable parameters and indicators that is frankly intimidating. But a poorly tuned MySQL server is a bottleneck for your PHP application scalability. This session shows how to do InnoDB tuning and read the InnoDB status report in MySQL 5.5.
PostgreSQL Procedural Languages: Tips, Tricks and GotchasJim Mlodgenski
One of the most powerful features of PostgreSQL is its diversity of procedural languages, but with that diversity comes a lot of options.
Did you ever wonder:
- What all of those options are on the CREATE FUNCTION statement?
- How do they affect my application?
- Does my choice of procedural language affect the performance of my statements?
- Should I create a single trigger with IF statements or several simple triggers?
- How do I debug my code?
- Can I tell which line in my function is taking all of the time?
This document discusses different types of enqueue waits caused by locks in an Oracle database. It covers common lock names like TX, TM, and UL; lock modes like exclusive and share; and how to identify the waiter, blocker, lock type, and object being blocked using views like v$lock, v$session, and v$active_session_history. The key pieces of information needed to resolve lock waits are the session ID of the waiter, the lock mode, and the object being blocked. User locks like TX and TM locks result from transactions and table modifications, while internal locks govern objects like redo and reusable objects.
Setup oracle golden gate 11g replicationKanwar Batra
How to setup Oracle Goldengate Replication between 11gR2 RAC or Single node instances. For RAC setup the GoldenGate custom cluster service . Not part of this document
[Pgday.Seoul 2019] Citus를 이용한 분산 데이터베이스PgDay.Seoul
This document summarizes how to set up and use Citus, an open-source PostgreSQL-based distributed database. It explains how to install Citus, add worker nodes, create distributed tables, and use features like reference tables to perform distributed queries across the cluster.
Internationalizing CakePHP ApplicationsPierre MARTIN
The document discusses internationalization in CakePHP, including:
- Using methods like __() and __n() to translate text strings, and Configure::write() to set the application language.
- The Translate behavior, which allows translating database records into multiple languages and automatically filtering by the current language.
- Generating translation files using the i18n extractor, editing them with POEDIT, and caching translated elements.
The document discusses Pgtap, a unit testing framework for PostgreSQL. It begins with an introduction to the speaker and an agenda that includes an introduction to Pgtap, why to use it, best practices, and Q&A. The rest of the document delves into details of Pgtap, including how to add it to a database, how to write tests using its functions for assertions and object validation, and how to run tests from the command line using pg_prove. It emphasizes that Pgtap allows testing all aspects of a PostgreSQL database and provides stable and comprehensive testing.
This document discusses Linux kernel debugging. It provides an overview of debugging techniques including collecting system information, handling failures, and using printk(), KGDB, and debuggers. Key points covered are the components of a debugger, how KGDB can be used with gdb to debug interactively, analyzing crash data, and some debugging tricks and print functions.
The document provides an introduction to using DotCloud and Go. It discusses deploying a Perl application on DotCloud, including defining services in dotcloud.yml, connecting services to each other using environment variables, and pushing code to DotCloud. It also covers troubleshooting applications running on DotCloud using commands like dotcloud logs and dotcloud run.
This document provides an overview of Cuckoo sandbox and tips for using and customizing it. It discusses supported platforms and hypervisors, how to retrieve analysis results using signatures, different ways to write hooks, and examples of analyzing malware like Andromeda and Locky. The document also shares some "goodies" like redirecting SMTP traffic and injecting emulator headers to trigger behaviors.
Backdooring the web is the cheapest and most hidden way to achieve
persistence on a compromised network, both if you're looking at
privileges on the webapp itself or at executing command to underlying
system.
During the talk, we will discuss the context of a web backdoor: the
environment where she can born and grow up will be defined.
Each environmental aspect will be thoroughly analyzed: where is the best
point of injection, why we choose a specific function or trick, what
permissions are needed, how to trigger the backdoor in a safe, hidden
and reproducible way, and of course what to inject.
The talk will thus present several ways to inject obfuscated and hard to
spot vulnerabilities in PHP code. Shown examples will backdoor CMS
plugins as well as custom code, altering the code and polluting the
webapp ecosystem (read: DBMS and webservers).
This document provides an overview of PL/Proxy, a database partitioning system implemented as a PostgreSQL procedural language extension. PL/Proxy allows applications to perform database operations like inserts, updates, deletes and queries across multiple PostgreSQL database partitions in a transparent manner. It works by routing operations to the appropriate partition based on the value of a partitioning key. The document discusses PL/Proxy concepts, areas of application, example usage, installation, backend and frontend functions, configuration options and more.
seccomp is a computer security facility in the Linux kernel, pledge is a similar security facility in the OpenBSD kernel. In this presentation Giovanni Bechis will review the development story and progress of both kernel interfaces and will analyze the main differences. There will be some examples of implementations of security patches made for some important open source projects.
Стажировка 2016-07-27 02 Денис Нелюбин. PostgreSQL и jsonbSmartTools
The document discusses using PostgreSQL and JSONB data. It covers installing PostgreSQL, connecting to a database, configuring network access and authentication, creating a database and user, inserting and querying JSONB data using operators like ->> and ->, updating and deleting rows, and creating a functional index to query on fields within the JSONB data.
This document summarizes the key features and changes in PostgreSQL 9.0 beta release. It highlights major new features like replication, permissions, and anonymous code blocks. It also briefly outlines many other enhancements, including performance improvements, monitoring tools, JSON/XML output for EXPLAIN, and mobile app contest. The presentation aims to excite developers about trying the new beta version.
Building a DSL with GraalVM (VoxxedDays Luxembourg)Maarten Mulders
The document discusses implementing the Brainfuck programming language on the JVM using GraalVM and provides a link for conference organizers to rate a talk on the topic; it requests that organizers not forget to provide feedback on the talk by rating it through the provided link.
This document discusses six Python packages that are useful to know:
1. First - A utility for selecting the first successful result from a sequence of functions.
2. Parse - A library for parsing Python format strings and extracting values.
3. Filecmp - A module for comparing files and directories.
4. Bitrot - A tool for detecting silent data corruption in files.
5. Docopt - A tool for generating command-line interfaces from a docstring.
6. Six - A library for writing code that is compatible with both Python 2 and Python 3.
This document discusses codifying PostgreSQL database schemas using Terraform. It begins by explaining how to bootstrap a database by hand and then introduces Terraform as a way to automate and version the database schema. Key concepts covered include using Terraform providers and resources to define database schemas, importing existing databases into Terraform, and iterating on schema designs in a declarative way. The document aims to help users avoid issues with Terraform by following best practices.
- Drizzle is an open source database that is compatible with MySQL and aims to be faster and more scalable.
- The document discusses how to install Drizzle, differences from MySQL like unsupported features and data types, and how to migrate data from MySQL to Drizzle using the drizzledump tool which converts the schema and data.
- Issues that may occur during migration like character set problems are addressed, and future plans like replication from MySQL to Drizzle are mentioned.
PostgreSQL is an open source object-relational database system that has been in development since 1982. It supports Linux, Windows, Mac OS X, and Solaris and can be installed using package managers or installers. PostgreSQL provides many features including procedural languages, functions, indexes, triggers, multi-version concurrency control, and point-in-time recovery. It also has various administration and development tools.
Finding and fixing bugs is a major chunk of any developers time. This talk describes the basic rules for effective debugging in any language, but shows how the tools available in PHP can be used to find and fix even the most elusive error
This document describes a scalable, versioned document store built within PostgreSQL. It discusses the motivation for moving from multiple data stores and repositories to a single PostgreSQL database. It then covers the design of storing immutable content as Merkle DAG nodes linked by cryptographic hashes, with references and tags allowing different versions. It also explains how the system was implemented using PostgreSQL functions to generate hashes, insert nodes, and handle migrations from the original data model.
Similar to SCALE 15x Minimizing PostgreSQL Major Version Upgrade Downtime (20)
Building Production Ready Search Pipelines with Spark and MilvusZilliz
Spark is the widely used ETL tool for processing, indexing and ingesting data to serving stack for search. Milvus is the production-ready open-source vector database. In this talk we will show how to use Spark to process unstructured data to extract vector representations, and push the vectors to Milvus vector database for search serving.
Observability Concepts EVERY Developer Should Know -- DeveloperWeek Europe.pdfPaige Cruz
Monitoring and observability aren’t traditionally found in software curriculums and many of us cobble this knowledge together from whatever vendor or ecosystem we were first introduced to and whatever is a part of your current company’s observability stack.
While the dev and ops silo continues to crumble….many organizations still relegate monitoring & observability as the purview of ops, infra and SRE teams. This is a mistake - achieving a highly observable system requires collaboration up and down the stack.
I, a former op, would like to extend an invitation to all application developers to join the observability party will share these foundational concepts to build on:
How to Get CNIC Information System with Paksim Ga.pptxdanishmna97
Pakdata Cf is a groundbreaking system designed to streamline and facilitate access to CNIC information. This innovative platform leverages advanced technology to provide users with efficient and secure access to their CNIC details.
GraphRAG for Life Science to increase LLM accuracyTomaz Bratanic
GraphRAG for life science domain, where you retriever information from biomedical knowledge graphs using LLMs to increase the accuracy and performance of generated answers
Unlocking Productivity: Leveraging the Potential of Copilot in Microsoft 365, a presentation by Christoforos Vlachos, Senior Solutions Manager – Modern Workplace, Uni Systems
GraphSummit Singapore | The Art of the Possible with Graph - Q2 2024Neo4j
Neha Bajwa, Vice President of Product Marketing, Neo4j
Join us as we explore breakthrough innovations enabled by interconnected data and AI. Discover firsthand how organizations use relationships in data to uncover contextual insights and solve our most pressing challenges – from optimizing supply chains, detecting fraud, and improving customer experiences to accelerating drug discoveries.
Maruthi Prithivirajan, Head of ASEAN & IN Solution Architecture, Neo4j
Get an inside look at the latest Neo4j innovations that enable relationship-driven intelligence at scale. Learn more about the newest cloud integrations and product enhancements that make Neo4j an essential choice for developers building apps with interconnected data and generative AI.
Programming Foundation Models with DSPy - Meetup SlidesZilliz
Prompting language models is hard, while programming language models is easy. In this talk, I will discuss the state-of-the-art framework DSPy for programming foundation models with its powerful optimizers and runtime constraint system.
GraphSummit Singapore | The Future of Agility: Supercharging Digital Transfor...Neo4j
Leonard Jayamohan, Partner & Generative AI Lead, Deloitte
This keynote will reveal how Deloitte leverages Neo4j’s graph power for groundbreaking digital twin solutions, achieving a staggering 100x performance boost. Discover the essential role knowledge graphs play in successful generative AI implementations. Plus, get an exclusive look at an innovative Neo4j + Generative AI solution Deloitte is developing in-house.
Cosa hanno in comune un mattoncino Lego e la backdoor XZ?Speck&Tech
ABSTRACT: A prima vista, un mattoncino Lego e la backdoor XZ potrebbero avere in comune il fatto di essere entrambi blocchi di costruzione, o dipendenze di progetti creativi e software. La realtà è che un mattoncino Lego e il caso della backdoor XZ hanno molto di più di tutto ciò in comune.
Partecipate alla presentazione per immergervi in una storia di interoperabilità, standard e formati aperti, per poi discutere del ruolo importante che i contributori hanno in una comunità open source sostenibile.
BIO: Sostenitrice del software libero e dei formati standard e aperti. È stata un membro attivo dei progetti Fedora e openSUSE e ha co-fondato l'Associazione LibreItalia dove è stata coinvolta in diversi eventi, migrazioni e formazione relativi a LibreOffice. In precedenza ha lavorato a migrazioni e corsi di formazione su LibreOffice per diverse amministrazioni pubbliche e privati. Da gennaio 2020 lavora in SUSE come Software Release Engineer per Uyuni e SUSE Manager e quando non segue la sua passione per i computer e per Geeko coltiva la sua curiosità per l'astronomia (da cui deriva il suo nickname deneb_alpha).
TrustArc Webinar - 2024 Global Privacy SurveyTrustArc
How does your privacy program stack up against your peers? What challenges are privacy teams tackling and prioritizing in 2024?
In the fifth annual Global Privacy Benchmarks Survey, we asked over 1,800 global privacy professionals and business executives to share their perspectives on the current state of privacy inside and outside of their organizations. This year’s report focused on emerging areas of importance for privacy and compliance professionals, including considerations and implications of Artificial Intelligence (AI) technologies, building brand trust, and different approaches for achieving higher privacy competence scores.
See how organizational priorities and strategic approaches to data security and privacy are evolving around the globe.
This webinar will review:
- The top 10 privacy insights from the fifth annual Global Privacy Benchmarks Survey
- The top challenges for privacy leaders, practitioners, and organizations in 2024
- Key themes to consider in developing and maintaining your privacy program
Sudheer Mechineni, Head of Application Frameworks, Standard Chartered Bank
Discover how Standard Chartered Bank harnessed the power of Neo4j to transform complex data access challenges into a dynamic, scalable graph database solution. This keynote will cover their journey from initial adoption to deploying a fully automated, enterprise-grade causal cluster, highlighting key strategies for modelling organisational changes and ensuring robust disaster recovery. Learn how these innovations have not only enhanced Standard Chartered Bank’s data infrastructure but also positioned them as pioneers in the banking sector’s adoption of graph technology.
Pushing the limits of ePRTC: 100ns holdover for 100 daysAdtran
At WSTS 2024, Alon Stern explored the topic of parametric holdover and explained how recent research findings can be implemented in real-world PNT networks to achieve 100 nanoseconds of accuracy for up to 100 days.
Communications Mining Series - Zero to Hero - Session 1DianaGray10
This session provides introduction to UiPath Communication Mining, importance and platform overview. You will acquire a good understand of the phases in Communication Mining as we go over the platform with you. Topics covered:
• Communication Mining Overview
• Why is it important?
• How can it help today’s business and the benefits
• Phases in Communication Mining
• Demo on Platform overview
• Q/A
Removing Uninteresting Bytes in Software FuzzingAftab Hussain
Imagine a world where software fuzzing, the process of mutating bytes in test seeds to uncover hidden and erroneous program behaviors, becomes faster and more effective. A lot depends on the initial seeds, which can significantly dictate the trajectory of a fuzzing campaign, particularly in terms of how long it takes to uncover interesting behaviour in your code. We introduce DIAR, a technique designed to speedup fuzzing campaigns by pinpointing and eliminating those uninteresting bytes in the seeds. Picture this: instead of wasting valuable resources on meaningless mutations in large, bloated seeds, DIAR removes the unnecessary bytes, streamlining the entire process.
In this work, we equipped AFL, a popular fuzzer, with DIAR and examined two critical Linux libraries -- Libxml's xmllint, a tool for parsing xml documents, and Binutil's readelf, an essential debugging and security analysis command-line tool used to display detailed information about ELF (Executable and Linkable Format). Our preliminary results show that AFL+DIAR does not only discover new paths more quickly but also achieves higher coverage overall. This work thus showcases how starting with lean and optimized seeds can lead to faster, more comprehensive fuzzing campaigns -- and DIAR helps you find such seeds.
- These are slides of the talk given at IEEE International Conference on Software Testing Verification and Validation Workshop, ICSTW 2022.
5. + A good option if you need to do the
upgrade in place
+ A good option if you are missing
primary keys (gasp!) on larger tables
+ It’s a one way trip! (You tested the
new PostgreSQL version with your
workload, right?)
5
pg_upgrade
8. + Trigger based logical replication
+ Requires Primary Keys on all replicated
tables
+ Kicks off an initial sync
+ Triggers store data modification
statements in log tables for later replay
+ Slony Trivia: Slony is Russian for a
Group of Elephants
8
Slony High Level
10. + “A named set of PostgreSQL database
instances”
+ cluster name = migration
+ _migration schema created in
PostgreSQL DBs that are part of the
cluster
10
Slony Cluster
11. + A database that is part of a cluster
+ Ultimately defined by the CONNINFO
string
+ 'dbname=mydb host=myserver user=slony'
+ 'dbname=mydb host=mynewserver user=slony'
+ 'dbname=mydb host=myserver user=slony port
= 5433'
11
Slony Node
12. + “A set of tables and sequences that
are to be replicated”
+ You can have multiple sets in a
cluster
+ We’re not going to do that today
12
Slony Set
13. + Origin is the read/write master
+ Origin is also the first Provider
+ Subscriber nodes receive their data
from Providers
+ For the purpose of this tutorial, we
will have an Origin node which is the
only Provider node
13
Slony Origin/Provider/Subscriber
16. + Don’t make any schema changes while
you’ve got slony running
16
One item of Note!
17. + Make a schema-only copy of the DB
+ Our first “slonik” script
+ Preamble
+ Cluster Initialization
+ Node Path Info
+ Set Creation
+ Table Addition
+ Sequence Addition
+ Subscribe
+ Kick off replication!
17
Let’s get started!
21. + Slonik is the Slony command processor
+ You call it just like any other
scripting language with a shebang at
the top:
+ #!/usr/bin/slonik
+ Trivia: Slonik means “little
elephant” in Russian
21
Our First Slonik Script!
27. CREATE SET (id = 1, origin = 1, comment
= 'all tables and sequences');
27
Create the Set
28. CREATE SET (id = 1, origin = 1, comment
= 'all tables and sequences');
ID of the Origin node.
28
Create the Set
29. Got Primary Keys on all your tables?
SET ADD TABLE (SET id = 1, origin = 1,
TABLES='public.*');
SET ADD TABLE (SET id = 1, origin = 1,
TABLES='mgd.*');
29
Add Tables to the Set!
30. Don’t do this:
SET ADD TABLE (SET id = 1, origin = 1,
TABLES='*');
30
Add Tables to the Set!
31. Don’t have primary keys on all your tables:
SET ADD TABLE (SET id = 1, origin = 1, FULL QUALIFIED NAME =
'mgd.acc_accession', comment='mgd.acc_accession TABLE');
SET ADD TABLE (SET id = 1, origin = 1, FULL QUALIFIED NAME =
'mgd.acc_accessionmax', comment='mgd.acc_accessionmax TABLE');
SET ADD TABLE (SET id = 1, origin = 1, FULL QUALIFIED NAME =
'mgd.acc_accessionreference', comment='mgd.acc_accessionreference
TABLE');
……
31
Add Tables to the Set!
32. SQL to the Rescue:
SELECT 'SET ADD TABLE (SET id = 1, origin = 1,
FULL QUALIFIED NAME = ''' || nspname || '.' ||
relname || ''', comment=''' || nspname || '.'
|| relname || ' TABLE'');' FROM pg_class JOIN
pg_namespace ON relnamespace = pg_namespace.oid
WHERE relkind = 'r' AND relhaspkey AND nspname
NOT IN ('information_schema', 'pg_catalog');
32
Add Tables to the Set!
33. What about the tables that don’t have pkeys?
+Add primary keys if you can
+ If not, dump/restore just those tables
during the maintenance window
33
Add Tables to the Set!
34. SET ADD SEQUENCE (SET id = 1, origin =
1, SEQUENCES = 'public.*');
SET ADD SEQUENCE (SET id = 1, origin =
1, SEQUENCES = 'mgd.*');
34
Don’t Forget the Sequences!
35. Or the old school way:
SET ADD SEQUENCE (SET id = 1, origin = 1, FULL
QUALIFIED NAME = 'mgd.pwi_report_id_seq',
comment='mgd.pwi_report_id_seq SEQUENCE');
SET ADD SEQUENCE (SET id = 1, origin = 1, FULL
QUALIFIED NAME = 'mgd.pwi_report_label_id_seq',
comment='mgd.pwi_report_label_id_seq SEQUENCE');
35
Add Sequences to the Set!
36. SUBSCRIBE SET (id = 1, provider = 1,
receiver = 2, forward = yes);
36
Subscribe the Set!
37. #!/usr/bin/slonik
CLUSTER NAME = migration;
NODE 1 ADMIN CONNINFO='host=db1.jefftest dbname=mgd user=slony port=5432';
NODE 2 ADMIN CONNINFO='host=db2.jefftest dbname=mgd user=slony port=5432';
INIT CLUSTER (id = 1, comment = 'db1.jefftest');
STORE NODE (id = 2, comment = 'db2.jefftest', event node = 1);
STORE PATH (server = 1, client = 2, conninfo = 'host=db1.jefftest dbname=mgd user=slony');
STORE PATH (server = 2, client = 1, conninfo = 'host=db2.jefftest dbname=mgd user=slony');
CREATE SET (id = 1, origin = 1, comment = 'all tables and sequences');
SET ADD TABLE (SET id = 1, origin = 1, TABLES='public.*');
SET ADD TABLE (SET id = 1, origin = 1, TABLES='mgd.*');
SET ADD SEQUENCE (SET id = 1, origin = 1, SEQUENCES = 'public.*');
SET ADD SEQUENCE (SET id = 1, origin = 1, SEQUENCES = 'mgd.*');
SUBSCRIBE SET (id = 1, provider = 1, receiver = 2, forward = yes);
37
Here’s the entire (unreadable on a slide?) script
40. 40
Add lock_timeout if possible
+ Added in 9.3
+ Abort any statement that waits longer than this
for a lock.
+ We only need it for trigger addition, so we just
add the ENV variable before we call our slonik
script:
PGOPTIONS="-c lock_timeout=5000" ./subscribe.slonik
41. 41
Add lock_timeout if possible
jfrost@db1.jefftest: ~$ PGOPTIONS="-c lock_timeout=5000" ./subscribe.slonik
./subscribe.slonik:11: Possible unsupported PostgreSQL version (90601) 9.6,
defaulting to 8.4 support
./subscribe.slonik:20: PGRES_FATAL_ERROR lock table
"_migration".sl_config_lock;select "_migration".setAddTable(1, 1,
'mgd.acc_accession', 'acc_accession_pkey', 'replicated table'); - ERROR: canceling
statement due to lock timeout
CONTEXT: SQL statement "lock table "mgd"."acc_accession" in access exclusive mode"
PL/pgSQL function _migration.altertableaddtriggers(integer) line 48 at EXECUTE
statement
SQL statement "SELECT "_migration".alterTableAddTriggers(p_tab_id)"
PL/pgSQL function setaddtable_int(integer,integer,text,name,text) line 104 at PERFORM
SQL statement "SELECT "_migration".setAddTable_int(p_set_id, p_tab_id, p_fqname,
p_tab_idxname, p_tab_comment)"
PL/pgSQL function setaddtable(integer,integer,text,name,text) line 33 at PERFORM
42. +Slon is the Slony daemon which manages
replication.
+ You need one for each node.
+ Trivia: slon is Russian for “elephant”
42
Introducing Slon
45. jfrost@db2.jefftest: ~$ tail -f slony.log
2017-02-07 00:43:07 UTC CONFIG remoteWorkerThread_1: prepare to copy table "mgd"."wks_rosetta"
2017-02-07 00:43:07 UTC CONFIG remoteWorkerThread_1: all tables for set 1 found on subscriber
2017-02-07 00:43:07 UTC CONFIG remoteWorkerThread_1: copy table "mgd"."acc_accession"
2017-02-07 00:43:07 UTC CONFIG remoteWorkerThread_1: Begin COPY of table "mgd"."acc_accession"
NOTICE: truncate of "mgd"."acc_accession" failed - doing delete
2017-02-07 00:44:45 UTC CONFIG remoteWorkerThread_1: 2935201458 bytes copied for table
“mgd"."acc_accession"
2017-02-07 00:49:17 UTC CONFIG remoteWorkerThread_1: 369.339 seconds to copy table
"mgd"."acc_accession"
2017-02-07 00:49:17 UTC CONFIG remoteWorkerThread_1: copy table "mgd"."acc_accessionmax"
2017-02-07 00:49:17 UTC CONFIG remoteWorkerThread_1: Begin COPY of table "mgd"."acc_accessionmax"
NOTICE: truncate of "mgd"."acc_accessionmax" succeeded
2017-02-07 00:49:17 UTC CONFIG remoteWorkerThread_1: 119 bytes copied for table
"mgd"."acc_accessionmax"
2017-02-07 00:49:17 UTC CONFIG remoteWorkerThread_1: 0.088 seconds to copy table
"mgd"."acc_accessionmax"
2017-02-07 00:49:17 UTC CONFIG remoteWorkerThread_1: copy table "mgd"."acc_accessionreference"
2017-02-07 00:49:17 UTC CONFIG remoteWorkerThread_1: Begin COPY of table
"mgd"."acc_accessionreference"
NOTICE: truncate of "mgd"."acc_accessionreference" succeeded
2017-02-07 00:49:37 UTC CONFIG remoteWorkerThread_1: 538589206 bytes copied for table
"mgd"."acc_accessionreference" 45
Watch the Logs (and Exercise Patience!)
47. 2017-02-07 02:11:30 UTC CONFIG remoteWorkerThread_1: Begin COPY of
table "mgd"."wks_rosetta"
NOTICE: truncate of "mgd"."wks_rosetta" succeeded
2017-02-07 02:11:30 UTC CONFIG remoteWorkerThread_1: 5302 bytes
copied for table "mgd"."wks_rosetta"
2017-02-07 02:11:30 UTC CONFIG remoteWorkerThread_1: 0.060 seconds to
copy table "mgd"."wks_rosetta"
2017-02-07 02:11:30 UTC INFO remoteWorkerThread_1: copy_set SYNC
found, use event seqno 5000000205.
2017-02-07 02:11:30 UTC INFO remoteWorkerThread_1: 0.016 seconds to
build initial setsync status
2017-02-07 02:11:30 UTC INFO copy_set 1 done in 1837.853 seconds
2017-02-07 02:11:30 UTC CONFIG enableSubscription: sub_set=1
47
Initial Sync is done!
53. +That’s the best part about Slony!
+We can switch back!
CLUSTER NAME = migration;
NODE 1 ADMIN CONNINFO='host=db1.jefftest dbname=mgd user=slony port=5432';
NODE 2 ADMIN CONNINFO='host=db2.jefftest dbname=mgd user=slony port=5432';
LOCK SET ( ID = 1, ORIGIN = 2);
MOVE SET ( ID = 1, OLD ORIGIN = 2, NEW ORIGIN = 1);
53
What if we find a regression on Monday?
55. +Let’s rip it out!
+ Can be as simple as:
+killall slon
+DROP SCHEMA _migration CASCADE;
+ Watch out for locking!
55
What if we didn’t find a regression?
Slony = plural for (many) elephants
Slonik = little elephant (in a cute way)
Slon = 1 elephant
The old fashioned, tried/true method. It always works!
Depending on your maintenance window requirements and disk subsystem
But, you might end up like this poor fella…..
What do I mean by this?
In slony a switchover reverses the direction of the subscription.
It’s going to be very different than what you’re used to and it can break replication.
So just put a freeze on DDL changes till the migration is complete.
Interprets the slony confguration and command scripting language
Define the cluster name
Admin Conninfo is how the slonik interpreter will connect to the nodes.
Creating the _migration slony schema in the primary DB
Prefer to make the “comment” the name of the DB server
Might not make sense if you’re replicating to a DB on the same server
In that case, maybe use something like db1.jefftest.old and db1.jefftest.new
This is how the slon daemons will connect to each node. This is usually the same as the ADMIN CONNINFO, but not necessarily.
It’ll subscribe the slony schema as well and chaos will ensue after the initial sync.
* Make sure you script up *and test* that dump / restore to minimize the downtime and also don’t forget to script up dump/restoring them the opposite direction in case you need to revert.
Might have to break up your slonik script into multiple SET ADD TABLE scripts
Outside the scope of this talk, but you can talk to me later if you’re interested
* Which is about 30 minutes for our 40GB mgd database
You probably did this off hours or on a weekend!
Wait until you’ve had at least a day or two running on the new PostgreSQL version before you tear it down.