Speaker: Ian Barwick
PostgreSQL and reliability go hand-in-hand - but your data is only truly safe with a solid and trusted backup system in place, and no matter how good your application is, it's useless if it can't talk to your database.
In this talk we'll demonstrate how to set up a reliable replication
cluster using open source tools closely associated with the PostgreSQL project. The talk will cover following areas:
- how to set up and manage a replication cluster with `repmgr`
- how to set up and manage reliable backups with `Barman`
- how to manage failover and application connections with `repmgr` and `PgBouncer`
Ian Barwick has worked for 2ndQuadrant since 2014, and as well as making various contributions to PostgreSQL itself, is lead `repmgr` developer. He lives in Tokyo, Japan.
PGConf APAC 2018 - High performance json postgre-sql vs. mongodbPGConf APAC
Speakers: Dominic Dwyer & Wei Shan Ang
This talk was presented in Percona Live Europe 2017. However, we did not have enough time to test against more scenario. We will be giving an updated talk with a more comprehensive tests and numbers. We hope to run it against citusDB and MongoRocks as well to provide a comprehensive comparison.
https://www.percona.com/live/e17/sessions/high-performance-json-postgresql-vs-mongodb
While the physical replication in PostgreSQL is quite robust, however, it doesn’t fit well in the picture when:
- You need partial replication only
- You want to replicate between different major versions of PostgreSQL
- You need to replicate multiple databases to the same target
- Transformation of the data is needed
- You want to replicate in order to upgrade without downtime
The answer to these use cases is logical replication
This talk will discuss and cover these use cases followed by a logical replication demo.
Speaker: Alexander Kukushkin
Kubernetes is a solid leader among different cloud orchestration engines and its adoption rate is growing on a daily basis. Naturally people want to run both their applications and databases on the same infrastructure.
There are a lot of ways to deploy and run PostgreSQL on Kubernetes, but most of them are not cloud-native. Around one year ago Zalando started to run HA setup of PostgreSQL on Kubernetes managed by Patroni. Those experiments were quite successful and produced a Helm chart for Patroni. That chart was useful, albeit a single problem: Patroni depended on Etcd, ZooKeeper or Consul.
Few people look forward to deploy two applications instead of one and support them later on. In this talk I would like to introduce Kubernetes-native Patroni. I will explain how Patroni uses Kubernetes API to run a leader election and store the cluster state. I’m going to live-demo a deployment of HA PostgreSQL cluster on Minikube and share our own experience of running more than 130 clusters on Kubernetes.
Patroni is a Python open-source project developed by Zalando in cooperation with other contributors on GitHub: https://github.com/zalando/patroni
PGConf APAC 2018 - Monitoring PostgreSQL at ScalePGConf APAC
Speaker: Lukas Fittl
Your PostgreSQL database is one of the most important pieces of your architecture - yet the level of introspection available in Postgres is often hard to work with. Its easy to get very detailed information, but what should you really watch out for, send reports on and alert on?
In this talk we'll discuss how query performance statistics can be made accessible to application developers, critical entries one should monitor in the PostgreSQL log files, how to collect EXPLAIN plans at scale, how to watch over autovacuum and VACUUM operations, and how to flag issues based on schema statistics.
We'll also talk a bit about monitoring multi-server setups, first going into high availability and read standbys, logical replication, and then reviewing how monitoring looks like for sharded databases like Citus.
The talk will primarily describe free/open-source tools and statistics views readily available from within Postgres.
Kubernetes is a solid leader among different cloud orchestration engines and its adoption rate is growing on a daily basis. Naturally people want to run both their applications and databases on the same infrastructure.
There are a lot of ways to deploy and run PostgreSQL on Kubernetes, but most of them are not cloud-native. Around one year ago Zalando started to run HA setup of PostgreSQL on Kubernetes managed by Patroni. Those experiments were quite successful and produced a Helm chart for Patroni. That chart was useful, albeit a single problem: Patroni depended on Etcd, ZooKeeper or Consul.
Few people look forward to deploy two applications instead of one and support them later on. In this talk I would like to introduce Kubernetes-native Patroni. I will explain how Patroni uses Kubernetes API to run a leader election and store the cluster state. I’m going to live-demo a deployment of HA PostgreSQL cluster on Minikube and share our own experience of running more than 130 clusters on Kubernetes.
Patroni is a Python open-source project developed by Zalando in cooperation with other contributors on GitHub: https://github.com/zalando/patroni
How to do a LIVE-demo with minikube:
1. git clone https://github.com/zalando/patroni
2. cd patroni
3. git checkout feature/demo
4. cd kubernetes
5. open demo.sh and edit line #4 (specify the minikube context )
6. docker build -t patroni .
7. may be docker push patroni
8. may be edit patroni_k8s.yaml line #22 and put the name of patroni image you build there
9. install tmux
10. run tmux in one terminal
11. run bash demo.sh in another terminal and press Enter from time to time
PGConf.ASIA 2019 Bali - Setup a High-Availability and Load Balancing PostgreS...Equnix Business Solutions
PGConf.ASIA 2019 Bali - 10 September 2019
Speaker: Bo Peng
Room: SQL
Title: Setup a High-Availability and Load Balancing PostgreSQL Cluster - New Features of Pgpool-II 4.1
PGConf APAC 2018 - High performance json postgre-sql vs. mongodbPGConf APAC
Speakers: Dominic Dwyer & Wei Shan Ang
This talk was presented in Percona Live Europe 2017. However, we did not have enough time to test against more scenario. We will be giving an updated talk with a more comprehensive tests and numbers. We hope to run it against citusDB and MongoRocks as well to provide a comprehensive comparison.
https://www.percona.com/live/e17/sessions/high-performance-json-postgresql-vs-mongodb
While the physical replication in PostgreSQL is quite robust, however, it doesn’t fit well in the picture when:
- You need partial replication only
- You want to replicate between different major versions of PostgreSQL
- You need to replicate multiple databases to the same target
- Transformation of the data is needed
- You want to replicate in order to upgrade without downtime
The answer to these use cases is logical replication
This talk will discuss and cover these use cases followed by a logical replication demo.
Speaker: Alexander Kukushkin
Kubernetes is a solid leader among different cloud orchestration engines and its adoption rate is growing on a daily basis. Naturally people want to run both their applications and databases on the same infrastructure.
There are a lot of ways to deploy and run PostgreSQL on Kubernetes, but most of them are not cloud-native. Around one year ago Zalando started to run HA setup of PostgreSQL on Kubernetes managed by Patroni. Those experiments were quite successful and produced a Helm chart for Patroni. That chart was useful, albeit a single problem: Patroni depended on Etcd, ZooKeeper or Consul.
Few people look forward to deploy two applications instead of one and support them later on. In this talk I would like to introduce Kubernetes-native Patroni. I will explain how Patroni uses Kubernetes API to run a leader election and store the cluster state. I’m going to live-demo a deployment of HA PostgreSQL cluster on Minikube and share our own experience of running more than 130 clusters on Kubernetes.
Patroni is a Python open-source project developed by Zalando in cooperation with other contributors on GitHub: https://github.com/zalando/patroni
PGConf APAC 2018 - Monitoring PostgreSQL at ScalePGConf APAC
Speaker: Lukas Fittl
Your PostgreSQL database is one of the most important pieces of your architecture - yet the level of introspection available in Postgres is often hard to work with. Its easy to get very detailed information, but what should you really watch out for, send reports on and alert on?
In this talk we'll discuss how query performance statistics can be made accessible to application developers, critical entries one should monitor in the PostgreSQL log files, how to collect EXPLAIN plans at scale, how to watch over autovacuum and VACUUM operations, and how to flag issues based on schema statistics.
We'll also talk a bit about monitoring multi-server setups, first going into high availability and read standbys, logical replication, and then reviewing how monitoring looks like for sharded databases like Citus.
The talk will primarily describe free/open-source tools and statistics views readily available from within Postgres.
Kubernetes is a solid leader among different cloud orchestration engines and its adoption rate is growing on a daily basis. Naturally people want to run both their applications and databases on the same infrastructure.
There are a lot of ways to deploy and run PostgreSQL on Kubernetes, but most of them are not cloud-native. Around one year ago Zalando started to run HA setup of PostgreSQL on Kubernetes managed by Patroni. Those experiments were quite successful and produced a Helm chart for Patroni. That chart was useful, albeit a single problem: Patroni depended on Etcd, ZooKeeper or Consul.
Few people look forward to deploy two applications instead of one and support them later on. In this talk I would like to introduce Kubernetes-native Patroni. I will explain how Patroni uses Kubernetes API to run a leader election and store the cluster state. I’m going to live-demo a deployment of HA PostgreSQL cluster on Minikube and share our own experience of running more than 130 clusters on Kubernetes.
Patroni is a Python open-source project developed by Zalando in cooperation with other contributors on GitHub: https://github.com/zalando/patroni
How to do a LIVE-demo with minikube:
1. git clone https://github.com/zalando/patroni
2. cd patroni
3. git checkout feature/demo
4. cd kubernetes
5. open demo.sh and edit line #4 (specify the minikube context )
6. docker build -t patroni .
7. may be docker push patroni
8. may be edit patroni_k8s.yaml line #22 and put the name of patroni image you build there
9. install tmux
10. run tmux in one terminal
11. run bash demo.sh in another terminal and press Enter from time to time
PGConf.ASIA 2019 Bali - Setup a High-Availability and Load Balancing PostgreS...Equnix Business Solutions
PGConf.ASIA 2019 Bali - 10 September 2019
Speaker: Bo Peng
Room: SQL
Title: Setup a High-Availability and Load Balancing PostgreSQL Cluster - New Features of Pgpool-II 4.1
20180503 kube con eu kubernetes metrics deep diveBob Cotton
Kubernetes generates a wealth of metrics. Some explicitly within the Kubernetes API server, the Kublet, and cAdvisor or implicitly by observing events such as the kube-state-metrics project. A subset of these metrics are used within Kubernetes itself to make scheduling decisions, however, other metrics can be used to determine the overall health of the system or for capacity planning purposes.
Kubernetes exposes metrics from several places, some available internally, others through add-on projects. In this session you will learn about:
- Node level metrics, as exposed from the node_exporter
- Kublet metrics
- API server metrics
- etcd metrics
- cAdvisor metrics
- Metrics exposed from kube-state-metrics
PGConf.ASIA 2019 Bali - How did PostgreSQL Write Load Balancing of Queries Us...Equnix Business Solutions
PGConf.ASIA 2019 Bali - 10 September 2019
Speaker: Atsushi Mitani
Room: WAL
Title: How did PostgreSQL Write Load Balancing of Queries Using Transactions?
This talk is from Distributed Data Summit SF 2018 - http://distributeddatasummit.com/2018-sf/sessions#chella
Audit logging is one of the most critical features in an enterprise-ready database in terms of security compliance. Furthermore, live traffic troubleshooting is critical for operators to troubleshoot production issues quickly. While past versions have lacked these critical features, the Cassandra team understood the need for better solutions and in the upcoming release of Cassandra both of these features now come out of the box which makes Cassandra even more awesome to work with. Cassandra now supports Audit logging and query logging as part of C* itself. As part of this talk, audience will learn about how to enable, configure, and tune audit logging for their C* clusters and how to log live traffic/queries for serverel needs including troubleshooting or even live traffic reply
SF Big Analytics 2019112: Uncovering performance regressions in the TCP SACK...Chester Chen
Uncovering performance regressions in the TCP SACKs vulnerability fixes
In early July 2019, Databricks noticed some Apache Spark workloads regressing by as much as 6x. In this talk, we'll discuss how we traced these regressions back to the Linux kernel and the fixes for the TCP SACKs vulnerabilities. We will explain the symptoms we were seeing, walk through how we debugged the TCP connections, and dive into the Linux source to uncover the root cause.
Speaker: Chris Stevens (Databricks)
Chris Stevens is a software engineer at Databricks where he works on the reliability, scalability, and security of Apache Spark clusters. His work focuses on auto-scaling compute, auto-scaling storage, node initialization performance, and node health monitoring. Prior to Databricks, Chris founded the Minoca OS project, where he built a POSIX compliant, general purpose OS - from scratch - to run on resource constrained device. He got his start at Microsoft working on the Windows kernel team, porting the Windows boot environment from BIOS to UEFI.
Agreement in a distributed system is complicated but required. Scylla gained lightweight transactions through Paxos but the latter has a cost of 3X roundtrips. Raft can allow consistent transactions without the performance penalty. Beyond LWT, we plan to integrate Raft with most aspects of Scylla making a leap forward in manageability and consistency
This is a talk I gave to the late crew at the DevOps KC meetup outlining why/what/how of setting up a Graphite server using Python end-to-end for getting stats.
Swapping Pacemaker Corosync with repmgrPGConf APAC
These slides were used by Wei Shan from GMO GlobalSign while presenting at pgDay Asia 2016. He discussed about challenges with the maintenance of Pacemaker/Corosync HA Clusters and how he migrated over to repmgr. He also did a short demo
20180503 kube con eu kubernetes metrics deep diveBob Cotton
Kubernetes generates a wealth of metrics. Some explicitly within the Kubernetes API server, the Kublet, and cAdvisor or implicitly by observing events such as the kube-state-metrics project. A subset of these metrics are used within Kubernetes itself to make scheduling decisions, however, other metrics can be used to determine the overall health of the system or for capacity planning purposes.
Kubernetes exposes metrics from several places, some available internally, others through add-on projects. In this session you will learn about:
- Node level metrics, as exposed from the node_exporter
- Kublet metrics
- API server metrics
- etcd metrics
- cAdvisor metrics
- Metrics exposed from kube-state-metrics
PGConf.ASIA 2019 Bali - How did PostgreSQL Write Load Balancing of Queries Us...Equnix Business Solutions
PGConf.ASIA 2019 Bali - 10 September 2019
Speaker: Atsushi Mitani
Room: WAL
Title: How did PostgreSQL Write Load Balancing of Queries Using Transactions?
This talk is from Distributed Data Summit SF 2018 - http://distributeddatasummit.com/2018-sf/sessions#chella
Audit logging is one of the most critical features in an enterprise-ready database in terms of security compliance. Furthermore, live traffic troubleshooting is critical for operators to troubleshoot production issues quickly. While past versions have lacked these critical features, the Cassandra team understood the need for better solutions and in the upcoming release of Cassandra both of these features now come out of the box which makes Cassandra even more awesome to work with. Cassandra now supports Audit logging and query logging as part of C* itself. As part of this talk, audience will learn about how to enable, configure, and tune audit logging for their C* clusters and how to log live traffic/queries for serverel needs including troubleshooting or even live traffic reply
SF Big Analytics 2019112: Uncovering performance regressions in the TCP SACK...Chester Chen
Uncovering performance regressions in the TCP SACKs vulnerability fixes
In early July 2019, Databricks noticed some Apache Spark workloads regressing by as much as 6x. In this talk, we'll discuss how we traced these regressions back to the Linux kernel and the fixes for the TCP SACKs vulnerabilities. We will explain the symptoms we were seeing, walk through how we debugged the TCP connections, and dive into the Linux source to uncover the root cause.
Speaker: Chris Stevens (Databricks)
Chris Stevens is a software engineer at Databricks where he works on the reliability, scalability, and security of Apache Spark clusters. His work focuses on auto-scaling compute, auto-scaling storage, node initialization performance, and node health monitoring. Prior to Databricks, Chris founded the Minoca OS project, where he built a POSIX compliant, general purpose OS - from scratch - to run on resource constrained device. He got his start at Microsoft working on the Windows kernel team, porting the Windows boot environment from BIOS to UEFI.
Agreement in a distributed system is complicated but required. Scylla gained lightweight transactions through Paxos but the latter has a cost of 3X roundtrips. Raft can allow consistent transactions without the performance penalty. Beyond LWT, we plan to integrate Raft with most aspects of Scylla making a leap forward in manageability and consistency
This is a talk I gave to the late crew at the DevOps KC meetup outlining why/what/how of setting up a Graphite server using Python end-to-end for getting stats.
Swapping Pacemaker Corosync with repmgrPGConf APAC
These slides were used by Wei Shan from GMO GlobalSign while presenting at pgDay Asia 2016. He discussed about challenges with the maintenance of Pacemaker/Corosync HA Clusters and how he migrated over to repmgr. He also did a short demo
Building tungsten-clusters-with-postgre sql-hot-standby-and-streaming-replica...Command Prompt., Inc
Alex Alexander & Linas Virbalas
Hot standby and streaming replication will move the needle forward for high availability and scaling for a wide number of applications. Tungsten already supports clustering using warm standby. In this talk we will describe how to build clusters using the new PostgreSQL features and give our report from the trenches.
This talk will cover how hot standby and streaming replication work from a user perspective, then dive into a description of how to use them, taking Tungsten as an example. We'll cover the following issues:
* Configuration of warm standby and streaming replication
* Provisioning new standby instances
* Strategies for balancing reads across primary and standby database
* Managing failover
* Troubleshooting and gotchas
Please join us for an enlightening discussion a set of PostgreSQL features that are interesting to a wide range of PostgreSQL users.
Percona Live 2022 - PBM - The Backup Open Source Tool for MongoDBJean Da Silva
Backup and restore are two of the most important things for databases. We don't often use the backup, but during a disaster situation, it is crucial to work.
In this session, we will discuss Percona Backup for MongoDB (PBM short).
We will walk through the process of taking backups and executing restores. We will also introduce the newest backup method that PBM offers, the physical backup in addition to the logical backup. After the introduction of the backup methods, we will evaluate the backup and restore times, and how to store the backup on remote backup storage.
This talk cover various advanced topics in the area of backups:
- incremental backups;
- archive management;
- backup validation;
- retention policies;
etc.
Based on these features, we'll compare various backup/recovery solutions for PostgreSQL.
This information will help you to choose the most appropriate tool for your system.
Automate Oracle database patches and upgrades using Fleet Provisioning and Pa...Nelson Calero
Each new version of the Oracle database includes improvements in the upgrade and patching utilities, forcing us to update our procedures to incorporate these changes.
The Fleet Provisioning & Patching (FPP, formerly RHP) utility, together with the change in its licensing announced at OOW 2019 that makes it free in RAC, now makes it possible to centrally manage the software life cycle.
This presentation shows examples of how to use FPP and different configuration options.
Joshua D. Drake
Are you tired of not having a real solution for PITR? Enter PITRTools, a single and secure solution for using Point In Time Recovery for PostgreSQL.
A Step-By-Step Disaster Recovery Blueprint & Best Practices for Your NetBacku...Symantec
In this technical session we will share a few customer tested blueprints for implementing DR strategies with NetBackup appliances showing support for onsite and offsite disaster recovery. This includes the architecture design with Symantec best practices, down to execution of the wizards and command lines needed to implement the solution.
Watch the recording of this Google+ Hangout: http://bit.ly/13oTjvp
Troubleshooting common oslo.messaging and RabbitMQ issuesMichael Klishin
This talk focuses on troubleshooting of common oslo.messaging and RabbitMQ issues in OpenStack environments. Co-presented at the OpenStack Summit Austin in April 2016.
Intro to Apache Kafka I gave at the Big Data Meetup in Geneva in June 2016. Covers the basics and gets into some more advanced topics. Includes demo and source code to write clients and unit tests in Java (GitHub repo on the last slides).
PGConf APAC 2018: Sponsored Talk by Fujitsu - The growing mandatory requireme...PGConf APAC
Speaker: Rajni Baliyan
As the volume of data of a personal nature and commodification of information collected and analysed increases; so is the focus on privacy and data security. Many countries are examining international and domestic laws in order to protect consumers and organisations alike.
The Australian Senate has recently passed a bill containing mandatory requirements to notify the privacy commissioner and consumers when data is at risk of causing serious harm in the case of a data breach occurring.
Europe has also announced new laws that allow consumers more control over their data. These laws allow consumers to tell companies to erase any data held about them.
These new laws will have a significant impact on organisations that store personal information.
This talk will examine some of these legislative changes and how specific PostgreSQL features can assist organisations in meeting their obligations and avoid heavy fines associated with breaching them.
PGConf APAC 2018 - A PostgreSQL DBAs Toolbelt for 2018PGConf APAC
There's no need to re-invent the wheel! Dozens of people have already tried...and succeeded. This talk is a categorized and illustrated overview on most popular and/or useful PostgreSQL specific scripts, utilities and whole toolsets that DBAs should be aware of for solving daily tasks. Inlcuding - performance monitoring, logs management/analyzis, identifying/fixing most common adminstration problems around areas of general performance metrics, tuning, locking, indexing, bloat, leaving out high-availability topics. Covered are venerable oldies from wiki.postgresql.org as well as my newer favourites from Github.
PGConf APAC 2018 - Where's Waldo - Text Search and Pattern in PostgreSQLPGConf APAC
Speaker: Joe Conway
There are many use cases for text search and pattern matching, and there are also a wide variety of techniques available in PostgreSQL to perform text search and pattern matching. Figuring out the best "match" between use case and technique can be confusing. This talk will review the possibilities and provide guidance regarding when to use what method, and especially how to properly deal with the related index methods to ensure speedy searches. This talk covers:
* The primary available search methods
* Examples illustrating when to use each
* Extensive discussion of index use
* Timing comparisons using realistic examples
PGConf APAC 2018 - PostgreSQL HA with Pgpool-II and whats been happening in P...PGConf APAC
Speaker: Muhammad Usama
Pgpool-II has been around to complement PostgreSQL over a decade and provides many features like connection pooling, failover, query caching, load balancing, and HA. High Availability (HA) is very critical to most enterprise application, the clients needs the ability to automatically reconnect with a secondary node when the master nodes goes down.
This is where Pgpool-II watchdog feature comes in, the core feature of Pgpool-II provides HA by eliminating the SPOF is the Watchdog. This watchdog feature has been around for a while but it went through major overhauling and enhancements in recent releases. This talk aims to explain the watchdog feature, the recent enhancements went into the watchdog and describe how it can be used to provide PostgreSQL HA and automatic failover.
Their is rising trend of enterprise deployment shifting to cloud based environment, Pgpool II can be used in the cloud without any issues. In this talk we will give some ideas how Pgpool-II is used to provide PostgreSQL HA in cloud environment.
Finally we will summarise the major features that have been added in the recent major release of Pgpool II and whats in the pipeline for the next major release.
PGConf APAC 2018 - PostgreSQL performance comparison in various cloudsPGConf APAC
Speaker: Oskari Saarenmaa
Aiven PostgreSQL is available in five different public cloud providers' infrastructure in more than 60 regions around the world, including 18 in APAC. This has given us a unique opportunity to benchmark and compare performance of similar configurations in different environments.
We'll share our benchmark methods and results, comparing various PostgreSQL configurations and workloads across different clouds.
About a year ago I was caught up in line-of-fire when a production system started behaving abruptly
- A batch process which would finish in 15minutes started taking 1.5 hours
- We started facing OLTP read queries on standby being cancelled
- We faced a sudden slowness on the Primary server and we were forced to do a forceful switch to standby.
We were able to figure out that some peculiarities of the application code and batch process were responsible for this. But we could not fix the application code (as it is packaged application).
In this talk I would like to share more details of how we debugged, what was the problem we were facing and how we applied a work around for it. We also learnt that a query returning in 10minutes may not be as dangerous as a query returning in 10sec but executed 100s of times in an hour.
I will share in detail-
- How to map the process/top stats from OS with pg_stat_activity
- How to get and read explain plan
- How to judge if a query is costly
- What tools helped us
- A peculiar autovacuum/vacuum Vs Replication conflict we ran into
- Various parameters to tune autvacuum and auto-analyze process
- What we have done to work-around the problem
- What we have put in place for better monitoring and information gathering
This presentation was used by Blair during his talk on Aurora and PostgreSQl compatibility for Aurora at pgDay Asia 2017. The talk was part of dedicated PostgreSQL track at FOSSASIA 2017
PostgreSQL is one of the most loved databases and that is why AWS could not hold back from offering PostgreSQL as RDS. There are some really nice features in RDS which can be good for DBA and inspiring for Enterprises to build resilient solution with PostgreSQL.
This ppt was used by Devrim at pgDay Asia 2017. He talked about some important facts about WAL - Transaction Logs or xlogs in PostgreSQL. Some of these can really come handy on a bad day
Lessons PostgreSQL learned from commercial databases, and didn’tPGConf APAC
This is the ppt used by Illay for his presentation at pgDay Asia 2016 - "Lessons PostgreSQL learned from commercial
databases, and didn’t". The talk takes you through some of the really good things that PostgreSQL has done really well and somethings that PostgreSQL can learn from other databases
Query Parallelism in PostgreSQL: What's coming next?PGConf APAC
This presentation was presented by Dilip Kumar (a PostgreSQL contributor) at pgDay Asia 2017. The presentation talks about Prallel query features released in v9.6, the infrastructure for the prallel query feature which was built in previous versions and what is the roadmap for prallel query.
Why we love pgpool-II and why we hate it!PGConf APAC
This talk was presented at pgDay Asia 2017. This details some of the great features of pgpool and some practical challenges faced by the speaker. It concludes with some tips while using pgpool and when not to use pgpool
These slides were used by Bruce Momjian for the keynote opening at pgDay Asia. He spoke about how the PostgreSQL project and the database software itself has shaped over last few years. Bruce is a core-community members and has been involved with PostgreSQL for about 20 years. He works at EnterpriseDB.
Elevating Tactical DDD Patterns Through Object CalisthenicsDorra BARTAGUIZ
After immersing yourself in the blue book and its red counterpart, attending DDD-focused conferences, and applying tactical patterns, you're left with a crucial question: How do I ensure my design is effective? Tactical patterns within Domain-Driven Design (DDD) serve as guiding principles for creating clear and manageable domain models. However, achieving success with these patterns requires additional guidance. Interestingly, we've observed that a set of constraints initially designed for training purposes remarkably aligns with effective pattern implementation, offering a more ‘mechanical’ approach. Let's explore together how Object Calisthenics can elevate the design of your tactical DDD patterns, offering concrete help for those venturing into DDD for the first time!
Encryption in Microsoft 365 - ExpertsLive Netherlands 2024Albert Hoitingh
In this session I delve into the encryption technology used in Microsoft 365 and Microsoft Purview. Including the concepts of Customer Key and Double Key Encryption.
Securing your Kubernetes cluster_ a step-by-step guide to success !KatiaHIMEUR1
Today, after several years of existence, an extremely active community and an ultra-dynamic ecosystem, Kubernetes has established itself as the de facto standard in container orchestration. Thanks to a wide range of managed services, it has never been so easy to set up a ready-to-use Kubernetes cluster.
However, this ease of use means that the subject of security in Kubernetes is often left for later, or even neglected. This exposes companies to significant risks.
In this talk, I'll show you step-by-step how to secure your Kubernetes cluster for greater peace of mind and reliability.
Generating a custom Ruby SDK for your web service or Rails API using Smithyg2nightmarescribd
Have you ever wanted a Ruby client API to communicate with your web service? Smithy is a protocol-agnostic language for defining services and SDKs. Smithy Ruby is an implementation of Smithy that generates a Ruby SDK using a Smithy model. In this talk, we will explore Smithy and Smithy Ruby to learn how to generate custom feature-rich SDKs that can communicate with any web service, such as a Rails JSON API.
GraphRAG is All You need? LLM & Knowledge GraphGuy Korland
Guy Korland, CEO and Co-founder of FalkorDB, will review two articles on the integration of language models with knowledge graphs.
1. Unifying Large Language Models and Knowledge Graphs: A Roadmap.
https://arxiv.org/abs/2306.08302
2. Microsoft Research's GraphRAG paper and a review paper on various uses of knowledge graphs:
https://www.microsoft.com/en-us/research/blog/graphrag-unlocking-llm-discovery-on-narrative-private-data/
State of ICS and IoT Cyber Threat Landscape Report 2024 previewPrayukth K V
The IoT and OT threat landscape report has been prepared by the Threat Research Team at Sectrio using data from Sectrio, cyber threat intelligence farming facilities spread across over 85 cities around the world. In addition, Sectrio also runs AI-based advanced threat and payload engagement facilities that serve as sinks to attract and engage sophisticated threat actors, and newer malware including new variants and latent threats that are at an earlier stage of development.
The latest edition of the OT/ICS and IoT security Threat Landscape Report 2024 also covers:
State of global ICS asset and network exposure
Sectoral targets and attacks as well as the cost of ransom
Global APT activity, AI usage, actor and tactic profiles, and implications
Rise in volumes of AI-powered cyberattacks
Major cyber events in 2024
Malware and malicious payload trends
Cyberattack types and targets
Vulnerability exploit attempts on CVEs
Attacks on counties – USA
Expansion of bot farms – how, where, and why
In-depth analysis of the cyber threat landscape across North America, South America, Europe, APAC, and the Middle East
Why are attacks on smart factories rising?
Cyber risk predictions
Axis of attacks – Europe
Systemic attacks in the Middle East
Download the full report from here:
https://sectrio.com/resources/ot-threat-landscape-reports/sectrio-releases-ot-ics-and-iot-security-threat-landscape-report-2024/
Transcript: Selling digital books in 2024: Insights from industry leaders - T...BookNet Canada
The publishing industry has been selling digital audiobooks and ebooks for over a decade and has found its groove. What’s changed? What has stayed the same? Where do we go from here? Join a group of leading sales peers from across the industry for a conversation about the lessons learned since the popularization of digital books, best practices, digital book supply chain management, and more.
Link to video recording: https://bnctechforum.ca/sessions/selling-digital-books-in-2024-insights-from-industry-leaders/
Presented by BookNet Canada on May 28, 2024, with support from the Department of Canadian Heritage.
Essentials of Automations: Optimizing FME Workflows with ParametersSafe Software
Are you looking to streamline your workflows and boost your projects’ efficiency? Do you find yourself searching for ways to add flexibility and control over your FME workflows? If so, you’re in the right place.
Join us for an insightful dive into the world of FME parameters, a critical element in optimizing workflow efficiency. This webinar marks the beginning of our three-part “Essentials of Automation” series. This first webinar is designed to equip you with the knowledge and skills to utilize parameters effectively: enhancing the flexibility, maintainability, and user control of your FME projects.
Here’s what you’ll gain:
- Essentials of FME Parameters: Understand the pivotal role of parameters, including Reader/Writer, Transformer, User, and FME Flow categories. Discover how they are the key to unlocking automation and optimization within your workflows.
- Practical Applications in FME Form: Delve into key user parameter types including choice, connections, and file URLs. Allow users to control how a workflow runs, making your workflows more reusable. Learn to import values and deliver the best user experience for your workflows while enhancing accuracy.
- Optimization Strategies in FME Flow: Explore the creation and strategic deployment of parameters in FME Flow, including the use of deployment and geometry parameters, to maximize workflow efficiency.
- Pro Tips for Success: Gain insights on parameterizing connections and leveraging new features like Conditional Visibility for clarity and simplicity.
We’ll wrap up with a glimpse into future webinars, followed by a Q&A session to address your specific questions surrounding this topic.
Don’t miss this opportunity to elevate your FME expertise and drive your projects to new heights of efficiency.
UiPath Test Automation using UiPath Test Suite series, part 3DianaGray10
Welcome to UiPath Test Automation using UiPath Test Suite series part 3. In this session, we will cover desktop automation along with UI automation.
Topics covered:
UI automation Introduction,
UI automation Sample
Desktop automation flow
Pradeep Chinnala, Senior Consultant Automation Developer @WonderBotz and UiPath MVP
Deepak Rai, Automation Practice Lead, Boundaryless Group and UiPath MVP
Smart TV Buyer Insights Survey 2024 by 91mobiles.pdf91mobiles
91mobiles recently conducted a Smart TV Buyer Insights Survey in which we asked over 3,000 respondents about the TV they own, aspects they look at on a new TV, and their TV buying preferences.
GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using Deplo...James Anderson
Effective Application Security in Software Delivery lifecycle using Deployment Firewall and DBOM
The modern software delivery process (or the CI/CD process) includes many tools, distributed teams, open-source code, and cloud platforms. Constant focus on speed to release software to market, along with the traditional slow and manual security checks has caused gaps in continuous security as an important piece in the software supply chain. Today organizations feel more susceptible to external and internal cyber threats due to the vast attack surface in their applications supply chain and the lack of end-to-end governance and risk management.
The software team must secure its software delivery process to avoid vulnerability and security breaches. This needs to be achieved with existing tool chains and without extensive rework of the delivery processes. This talk will present strategies and techniques for providing visibility into the true risk of the existing vulnerabilities, preventing the introduction of security issues in the software, resolving vulnerabilities in production environments quickly, and capturing the deployment bill of materials (DBOM).
Speakers:
Bob Boule
Robert Boule is a technology enthusiast with PASSION for technology and making things work along with a knack for helping others understand how things work. He comes with around 20 years of solution engineering experience in application security, software continuous delivery, and SaaS platforms. He is known for his dynamic presentations in CI/CD and application security integrated in software delivery lifecycle.
Gopinath Rebala
Gopinath Rebala is the CTO of OpsMx, where he has overall responsibility for the machine learning and data processing architectures for Secure Software Delivery. Gopi also has a strong connection with our customers, leading design and architecture for strategic implementations. Gopi is a frequent speaker and well-known leader in continuous delivery and integrating security into software delivery.
4. https://www.2ndQuadrant.com
PGConf APAC 2018
Singapore, March 22nd
What is HA?
●
Recovery Point Objective (RPO)
●
Recovery Time Objective (RTO)
Both as close to zero as possible/feasible
●
HA needs:
●
planning
●
documentation
●
practice
●
monitoring
●
HA involves tradeoffs
●
Many ways of implementing it
●
Keep systems up-to-date!!!
9. https://www.2ndQuadrant.com
PGConf APAC 2018
Singapore, March 22nd
repmgr
● replication manager
● two main roles
– set up and manage a replication cluster
– automatic failover and monitoring
● License: GPL
● current version: 4.0.4
● https://repmgr.org
15. https://www.2ndQuadrant.com
PGConf APAC 2018
Singapore, March 22nd
pgbouncer
● lightweight connection pooler
● useful functionality
– lowers performance impact of PostgreSQL connections
– can divert connections to other servers
● License: ISC (BSD-ish)
● current version: 1.8.1 (Dec 2018)
● https://pgbouncer.github.io/
16. https://www.2ndQuadrant.com
PGConf APAC 2018
Singapore, March 22nd
barman - a quick demo (1)
● sample configuration file (Barman server only)
[barman]
barman_home = /home/barman
barman_user = barman
log_file = /var/log/barman/barman.log
compression = gzip
reuse_backup = link
minimum_redundancy = 2
retention_policy = RECOVERY WINDOW OF 4 WEEKS
streaming_archiver = on
[test_cluster]
description = "Repmgr Test Cluster"
ssh_command = ssh -q localhost
conninfo = host=127.0.0.1 user=postgres port=5501
17. https://www.2ndQuadrant.com
PGConf APAC 2018
Singapore, March 22nd
barman - a quick demo (2)
● take a backup:
$ barman backup test_cluster
Starting backup using rsync-exclusive method for server test in /home/barman/test_cluster/base/
20161028T151425
Backup start at xlog location: 0/5000028 (000000010000000000000005, 00000028)
Copying files.
Copy done.
Asking PostgreSQL server to finalize the backup.
Backup size: 29.2 MiB. Actual size on disk: 158.4 KiB (-99.47% deduplication ratio).
Backup end at xlog location: 0/5000130 (000000010000000000000005, 00000130)
Backup completed
Processing xlog segments from file archival for test
000000010000000000000004
000000010000000000000005
000000010000000000000005.00000028.backup
18. https://www.2ndQuadrant.com
PGConf APAC 2018
Singapore, March 22nd
barman - a quick demo (3)
● list backup(s)
$ barman list-backup test_cluster
test_cluster 20180301T124344 - Thu Mar 1 12:43:47 2018 - Size: 28.5 MiB - WAL Size: 0 B
test_cluster 20180301T121110 - Thu Mar 1 12:11:21 2018 - Size: 28.4 MiB - WAL Size: 54.8 KiB
19. https://www.2ndQuadrant.com
PGConf APAC 2018
Singapore, March 22nd
barman - a quick demo (4)
● backup detail
$ barman show-backup test 20180301T124344
Backup 20180301T124344:
Server Name : test
Status : DONE
PostgreSQL Version : 90605
PGDATA directory : /tmp/repmgr-test/node_1/data
Base backup information:
Disk usage : 28.4 MiB (28.5 MiB with WALs)
Incremental size : 142.3 KiB (-99.51%)
Timeline : 1
Begin WAL : 000000010000000000000005
End WAL : 000000010000000000000005
WAL number : 1
WAL compression ratio: 99.84%
Begin time : 2018-03-01 12:43:44.721340+09:00
End time : 2018-03-01 12:43:47.314113+09:00
Begin Offset : 40
End Offset : 248
Begin XLOG : 0/5000028
End XLOG : 0/50000F8
WAL information:
No of files : 0
Disk usage : 0 B
Last available : 000000010000000000000005
Catalog information:
Retention Policy : VALID
Previous Backup : 20180301T121110
Next Backup : - (this is the latest base backup)
20. https://www.2ndQuadrant.com
PGConf APAC 2018
Singapore, March 22nd
barman - a quick demo (5)
● restore from backup
● various kinds of PITR recovery also available
$ barman recover --remote-ssh-command "ssh postgres@remotehost" test_cluster last
/var/lib/postgresql/data
Starting remote restore for server test using backup 20180301T124344
Destination directory: /var/lib/postgresql/data
Copying the base backup.
Copying required WAL segments.
Generating archive status files
Identify dangerous settings in destination directory.
WARNING
You are required to review the following options as potentially dangerous
postgresql.conf line 643: include = 'postgresql.replication.conf'
postgresql.conf line 644: include = 'postgresql.local.conf'
Your PostgreSQL server has been successfully prepared for recovery!
22. https://www.2ndQuadrant.com
PGConf APAC 2018
Singapore, March 22nd
repmgr - a quick demo (2)
● clone a standby... from the Barman backup!
● recovery.conf looks like this:
$ repmgr -D /tmp/repmgr-test/node_2/data
-f /tmp/repmgr-test/node_2/repmgr.conf
-h localhost -p 5501 -d repmgr -U repmgr --verbose -LINFO
standby clone
[2016-11-01 12:12:25] [NOTICE] using configuration file "/tmp/repmgr-test/node_2/repmgr.conf"
[2016-11-01 12:12:25] [NOTICE] destination directory '/tmp/repmgr-test/node_2/data' provided
[2016-11-01 12:12:25] [INFO] Connecting to Barman server to verify backup for test_cluster
Warning: Permanently added 'localhost' (ECDSA) to the list of known hosts.
[2016-11-01 12:12:25] [INFO] creating directory "/tmp/repmgr-test/node_2/data/repmgr"...
[2016-11-01 12:12:25] [INFO] Connecting to Barman server to fetch server parameters
Warning: Permanently added 'localhost' (ECDSA) to the list of known hosts.
[2016-11-01 12:12:26] [INFO] connecting to upstream node
[2016-11-01 12:12:26] [INFO] connected to upstream node, checking its state
[2016-11-01 12:12:26] [INFO] Successfully connected to upstream node. Current installation size is 28 MB
[2016-11-01 12:12:26] [NOTICE] getting backup from Barman...
Warning: Permanently added 'localhost' (ECDSA) to the list of known hosts.
Warning: Permanently added 'localhost' (ECDSA) to the list of known hosts.
Warning: Permanently added 'localhost' (ECDSA) to the list of known hosts.
[2016-11-01 12:12:28] [NOTICE] standby clone (from Barman) complete
[2016-11-01 12:12:28] [NOTICE] you can now start your PostgreSQL server
[2016-11-01 12:12:28] [HINT] for example : pg_ctl -D /tmp/repmgr-test/node_2/data start
[2016-11-01 12:12:28] [HINT] After starting the server, you need to register this standby with
"repmgr standby register"
23. https://www.2ndQuadrant.com
PGConf APAC 2018
Singapore, March 22nd
repmgr - a quick demo (3)
● recovery.conf looks like this:
● barman-wal-restore script does the “ heavy lifting” of fetching WAL
– from barman 2.0 part of the barman-cli package
standby_mode = 'on'
primary_conninfo = 'user=repmgr port=5501 host=localhost application_name=node2'
recovery_target_timeline = 'latest'
restore_command = 'barman-wal-restore barman.local test_cluster %f %p'
24. https://www.2ndQuadrant.com
PGConf APAC 2018
Singapore, March 22nd
back to barman - keep the WAL flowing
● WAL retention management - tricky
● use barman in the restore command
● removes need to manage:
– wal_keep_segments
– replication slots
– archive_cleanup_command
26. https://www.2ndQuadrant.com
PGConf APAC 2018
Singapore, March 22nd
repmgrd demo (2)
● Time to say goodbye to the primary...
● Standby promotes itself
pg_ctl -D /var/lib/pgsql/data/ -m immediate stop
[2018-03-20 13:54:02] [INFO] node "node2" (node ID: 2) monitoring upstream node "node1" (node ID: 1) in normal
state
[2018-03-20 13:54:14] [WARNING] unable to connect to upstream node "node1" (node ID: 1)
[2018-03-20 13:54:14] [INFO] checking state of node 1, 1 of 5 attempts
[2018-03-20 13:54:14] [INFO] sleeping 1 seconds until next reconnection attempt
[2018-03-20 13:54:15] [INFO] checking state of node 1, 2 of 5 attempts
[2018-03-20 13:54:15] [INFO] sleeping 1 seconds until next reconnection attempt
[2018-03-20 13:54:16] [INFO] checking state of node 1, 3 of 5 attempts
[2018-03-20 13:54:16] [INFO] sleeping 1 seconds until next reconnection attempt
[2018-03-20 13:54:17] [INFO] checking state of node 1, 4 of 5 attempts
[2018-03-20 13:54:17] [INFO] sleeping 1 seconds until next reconnection attempt
[2018-03-20 13:54:18] [INFO] checking state of node 1, 5 of 5 attempts
[2018-03-20 13:54:18] [WARNING] unable to reconnect to node 1 after 5 attempts
[2018-03-20 13:54:18] [NOTICE] this node is the only available candidate and will now promote itself
NOTICE: promoting standby to primary
DETAIL: promoting server "node2" (ID: 2) using "/home/ibarwick/devel/builds/94/bin/pg_ctl -l
/tmp/postgres.5502.log -m fast -w -D '/space/sda1/ibarwick/repmgr-test/node_2/data' promote"
NOTICE: STANDBY PROMOTE successful
DETAIL: server "node2" (ID: 2) was successfully promoted to primary
[2018-03-20 13:54:19] [NOTICE] 0 followers to notify
[2018-03-20 13:54:19] [INFO] switching to primary monitoring mode
27. https://www.2ndQuadrant.com
PGConf APAC 2018
Singapore, March 22nd
repmgrd failover mechanism
● quorum vote
● individual standbys can have different priorities
● witness server to establish qualified majority
● concept of “locations”
28. https://www.2ndQuadrant.com
PGConf APAC 2018
Singapore, March 22nd
repmgrd event notifications
● repmgr/repmgrd generate “event notifications”
● “after” trigger for certain events
● recorded in the repmgr metadatabase
● can be used to execute custom scripts
● following events can generated when repmgrd is running:
– repmgrd_start
– repmgrd_shutdown
– repmgrd_failover_promote
– repmgrd_failover_follow
– standby_promote
– standby_follow
– standby_disconnect_manual
29. https://www.2ndQuadrant.com
PGConf APAC 2018
Singapore, March 22nd
repmgr event log
● Logs info about events of note
$ repmgr cluster event --terse
Node ID | Name | Event | OK | Timestamp
---------+-------+--------------------------+----+---------------------
2 | node2 | repmgrd_reload | t | 2018-03-20 13:54:19
2 | node2 | repmgrd_failover_promote | t | 2018-03-20 13:54:19
2 | node2 | standby_promote | t | 2018-03-20 13:54:19
2 | node2 | repmgrd_start | t | 2018-03-20 13:53:30
1 | node1 | repmgrd_start | t | 2018-03-20 13:53:27
2 | node2 | standby_register_sync | t | 2018-03-20 13:53:24
2 | node2 | standby_register | t | 2018-03-20 13:53:24
2 | node2 | standby_clone | t | 2018-03-20 13:53:23
1 | node1 | primary_register | t | 2018-03-20 13:53:11
1 | node1 | cluster_created | t | 2018-03-20 13:53:11
30. https://www.2ndQuadrant.com
PGConf APAC 2018
Singapore, March 22nd
PgBouncer - concepts
● PgBouncer intercepts incoming PostgreSQL connections
● Re-routes these to local or remote databases
– acts as both pooler and proxy
● Provides a “virtual database” for issuing commands
32. https://www.2ndQuadrant.com
PGConf APAC 2018
Singapore, March 22nd
pgbouncer console
$ psql 'host=localhost user=postgres port=6501 dbname=pgbouncer'
psql (9.6.0, server 1.7.2/bouncer)
Type "help" for help.
pgbouncer=# show help;
NOTICE: Console usage
DETAIL:
SHOW HELP|CONFIG|DATABASES|POOLS|CLIENTS|SERVERS|VERSION
SHOW STATS|FDS|SOCKETS|ACTIVE_SOCKETS|LISTS|MEM
SHOW DNS_HOSTS|DNS_ZONES
SET key = arg
RELOAD
PAUSE [<db>]
RESUME [<db>]
DISABLE <db>
ENABLE <db>
KILL <db>
SUSPEND
SHUTDOWN
SHOW
Time: 0.140 ms
33. https://www.2ndQuadrant.com
PGConf APAC 2018
Singapore, March 22nd
pgbouncer connections
● connect to the primary:
● connect to the standby:
$ psql 'host=node2 user=repmgr port=6432 dbname=repmgr-rw'
psql (9.6.5)
Type "help" for help.
repmgr-rw=# CREATE TABLE foo (id INT);
CREATE TABLE
Time: 4.636 ms
$ psql 'host=node2 user=repmgr port=6432 dbname=repmgr-ro'
psql (9.6.5)
Type "help" for help.
repmgr-ro=# CREATE TABLE foo (id INT);
ERROR: cannot execute CREATE TABLE in a read-only transaction
Time: 0.867 ms
34. https://www.2ndQuadrant.com
PGConf APAC 2018
Singapore, March 22nd
Putting it all together
● Custom promote_command
● Performs following steps:
– pauses PgBouncer
– promotes the standby
– dynamically rewrites PgBouncer config file
– reloads PgBouncer config
– resumes PgBouncer
● barman reconfiguration not implemented
35. https://www.2ndQuadrant.com
PGConf APAC 2018
Singapore, March 22nd
PgBouncer – use include file
● [database] section as include file:
Note: %include directive available from PgBouncer 1.6
[pgbouncer]
listen_addr = *
listen_port = 6432
%include /etc/pgbouncer.database.ini
36. https://www.2ndQuadrant.com
PGConf APAC 2018
Singapore, March 22nd
Script (1) - variables
● Assuming PgBouncer running on DB node:
#!/usr/bin/env bash
set -u
set -e
# Configurable items
PGBOUNCER_HOSTS="node1 node2 node3"
PGBOUNCER_DATABASE_INI="/etc/pgbouncer.database.ini"
PGBOUNCER_DATABASE="appdb"
PGBOUNCER_PORT=6432
REPMGR_DB="repmgr"
REPMGR_USER="repmgr"
37. https://www.2ndQuadrant.com
PGConf APAC 2018
Singapore, March 22nd
Script (2) – pause/promote
● PgBouncer paused while standby promoted
# 1. Pause running pgbouncer instances
for HOST in $PGBOUNCER_HOSTS
do
psql -t -c "pause" -h $HOST -p $PGBOUNCER_PORT
-U postgres pgbouncer
done
# 2. Promote this node from standby to primary
repmgr standby promote -f /etc/repmgr.conf
38. https://www.2ndQuadrant.com
PGConf APAC 2018
Singapore, March 22nd
Script (3) – reconfigure
● Rewrite [databases] section
# 3. Reconfigure pgbouncer instances
PGBOUNCER_DATABASE_INI_NEW="/tmp/pgbouncer.database.ini"
for HOST in $PGBOUNCER_HOSTS
do
# Recreate the pgbouncer config file
echo -e "[databases]n" > $PGBOUNCER_DATABASE_INI_NEW
psql -d $REPMGR_DB -U $REPMGR_USER -t -A
-c "SELECT '${PGBOUNCER_DATABASE}-rw= ' || conninfo || '
application_name=pgbouncer_${HOST}'
FROM repmgr.nodes
WHERE active = TRUE AND type='primary'"
>> $PGBOUNCER_DATABASE_INI_NEW
psql -d $REPMGR_DB -U $REPMGR_USER -t -A
-c "SELECT '${PGBOUNCER_DATABASE}-ro= ' || conninfo ||
' application_name=pgbouncer_${HOST}'
FROM repmgr.nodes
WHERE node_name='${HOST}'"
>> $PGBOUNCER_DATABASE_INI_NEW
rsync $PGBOUNCER_DATABASE_INI_NEW $HOST:$PGBOUNCER_DATABASE_INI
41. https://www.2ndQuadrant.com
PGConf APAC 2018
Singapore, March 22nd
Advantages/disadvantages
● Advantages
– can be set up on an existing cluster
– application level solution
● Disadvantages
– depends on SSH being available