Spencer Christensen
There are many aspects to managing an RDBMS. Some of these are handled by an experienced DBA, but there are a good many things that any sys admin should be able to take care of if they know what to look for.
This presentation will cover basics of managing Postgres, including creating database clusters, overview of configuration, and logging. We will also look at tools to help monitor Postgres and keep an eye on what is going on. Some of the tools we will review are:
* pgtop
* pg_top
* pgfouine
* check_postgres.pl.
Check_postgres.pl is a great tool that can plug into your Nagios or Cacti monitoring systems, giving you even better visibility into your databases.
The paperback version is available on lulu.com there http://goo.gl/fraa8o
This is the first volume of the postgresql database administration book. The book covers the steps for installing, configuring and administering a PostgreSQL 9.3 on Linux debian. The book covers the logical and physical aspect of PostgreSQL. Two chapters are dedicated to the backup/restore topic.
Josh Berkus
You've heard that PostgreSQL is the highest-performance transactional open source database, but you're not seeing it on YOUR server. In fact, your PostgreSQL application is kind of poky. What should you do? While doing advanced performance engineering for really high-end systems takes years to learn, you can learn the basics to solve performance issues for 80% of PostgreSQL installations in less than an hour. In this session, you will learn: -- The parts of database application performance -- The performance setup procedure -- Basic troubleshooting tools -- The 13 postgresql.conf settings you need to know -- Where to look for more information.
This presentation covers all aspects of PostgreSQL administration, including installation, security, file structure, configuration, reporting, backup, daily maintenance, monitoring activity, disk space computations, and disaster recovery. It shows how to control host connectivity, configure the server, find the query being run by each session, and find the disk space used by each database.
This ppt was used by Devrim at pgDay Asia 2017. He talked about some important facts about WAL - Transaction Logs or xlogs in PostgreSQL. Some of these can really come handy on a bad day
En savoir plus sur www.opensourceschool.fr
Ce support est diffusé sous licence Creative Commons (CC BY-SA 3.0 FR) Attribution - Partage dans les Mêmes Conditions 3.0 France
Plan :
1. Introduction
2. Installation
3. The psql client
4. Authentication and privileges
5. Backup and restoration
6. Internal Architecture
7. Performance optimization
8. Stats and monitoring
9. Logs
10. Replication
Spencer Christensen
There are many aspects to managing an RDBMS. Some of these are handled by an experienced DBA, but there are a good many things that any sys admin should be able to take care of if they know what to look for.
This presentation will cover basics of managing Postgres, including creating database clusters, overview of configuration, and logging. We will also look at tools to help monitor Postgres and keep an eye on what is going on. Some of the tools we will review are:
* pgtop
* pg_top
* pgfouine
* check_postgres.pl.
Check_postgres.pl is a great tool that can plug into your Nagios or Cacti monitoring systems, giving you even better visibility into your databases.
The paperback version is available on lulu.com there http://goo.gl/fraa8o
This is the first volume of the postgresql database administration book. The book covers the steps for installing, configuring and administering a PostgreSQL 9.3 on Linux debian. The book covers the logical and physical aspect of PostgreSQL. Two chapters are dedicated to the backup/restore topic.
Josh Berkus
You've heard that PostgreSQL is the highest-performance transactional open source database, but you're not seeing it on YOUR server. In fact, your PostgreSQL application is kind of poky. What should you do? While doing advanced performance engineering for really high-end systems takes years to learn, you can learn the basics to solve performance issues for 80% of PostgreSQL installations in less than an hour. In this session, you will learn: -- The parts of database application performance -- The performance setup procedure -- Basic troubleshooting tools -- The 13 postgresql.conf settings you need to know -- Where to look for more information.
This presentation covers all aspects of PostgreSQL administration, including installation, security, file structure, configuration, reporting, backup, daily maintenance, monitoring activity, disk space computations, and disaster recovery. It shows how to control host connectivity, configure the server, find the query being run by each session, and find the disk space used by each database.
This ppt was used by Devrim at pgDay Asia 2017. He talked about some important facts about WAL - Transaction Logs or xlogs in PostgreSQL. Some of these can really come handy on a bad day
En savoir plus sur www.opensourceschool.fr
Ce support est diffusé sous licence Creative Commons (CC BY-SA 3.0 FR) Attribution - Partage dans les Mêmes Conditions 3.0 France
Plan :
1. Introduction
2. Installation
3. The psql client
4. Authentication and privileges
5. Backup and restoration
6. Internal Architecture
7. Performance optimization
8. Stats and monitoring
9. Logs
10. Replication
Presentation that I gave as a guest lecture for a summer intensive development course at nod coworking in Dallas, TX. The presentation targets beginning web developers with little, to no experience in databases, SQL, or PostgreSQL. I cover the creation of a database, creating records, reading/querying records, updating records, destroying records, joining tables, and a brief introduction to transactions.
PostgreSQL (or Postgres) began its life in 1986 as POSTGRES, a research project of the University of California at Berkeley.
PostgreSQL isn't just relational, it's object-relational.it's object-relational. This gives it some advantages over other open source SQL databases like MySQL, MariaDB and Firebird.
Connection Pooling in PostgreSQL using pgbouncer Sameer Kumar
The presentation was presented at 5th Postgres User Group, Singapore.
It explain how to setup pgbouncer and also shows a few demonstration graphs comparing the advantages/gains in performance when using pgbouncer instead of direct connections to PostgreSQL database.
This is a introduction to PostgreSQL that provides a brief overview of PostgreSQL's architecture, features and ecosystem. It was delivered at NYLUG on Nov 24, 2014.
http://www.meetup.com/nylug-meetings/events/180533472/
PostgreSQL is designed to be easily extensible. For this reason, extensions loaded into the database can function just like features that are built in. In this session, we will learn more about PostgreSQL extension framework, how are they built, look at some popular extensions, management of these extensions in your deployments.
PostgreSQL is one of the most advanced relational databases. It offers superb replication capabilities. The most important features are: Streaming replication, Point-In-Time-Recovery, advanced monitoring, etc.
by Mahesh Pakal, AWS
PostgreSQL is a powerful, enterprise class open source object-relational database system with an emphasis on extensibility and standards-compliance. PostgreSQL boasts many sophisticated features and runs stored procedures in more than a dozen programming languages. We’ll explore the advantages and limitations of PostgreSQL, examples of where it is best suited for use, and examples of who is using PostgreSQL to power their applications.
This technical presentation by EDB Dave Thomas, Systems Engineer provides an overview of:
1) BGWriter/Writer Process
2) Wall Writer Process
3) Stats Collector Process
4) Autovacuum Launch Process
5) Syslogger Process/Logger process
6) Archiver Process
7) WAL Send/Receive Processes
Presentation that I gave as a guest lecture for a summer intensive development course at nod coworking in Dallas, TX. The presentation targets beginning web developers with little, to no experience in databases, SQL, or PostgreSQL. I cover the creation of a database, creating records, reading/querying records, updating records, destroying records, joining tables, and a brief introduction to transactions.
PostgreSQL (or Postgres) began its life in 1986 as POSTGRES, a research project of the University of California at Berkeley.
PostgreSQL isn't just relational, it's object-relational.it's object-relational. This gives it some advantages over other open source SQL databases like MySQL, MariaDB and Firebird.
Connection Pooling in PostgreSQL using pgbouncer Sameer Kumar
The presentation was presented at 5th Postgres User Group, Singapore.
It explain how to setup pgbouncer and also shows a few demonstration graphs comparing the advantages/gains in performance when using pgbouncer instead of direct connections to PostgreSQL database.
This is a introduction to PostgreSQL that provides a brief overview of PostgreSQL's architecture, features and ecosystem. It was delivered at NYLUG on Nov 24, 2014.
http://www.meetup.com/nylug-meetings/events/180533472/
PostgreSQL is designed to be easily extensible. For this reason, extensions loaded into the database can function just like features that are built in. In this session, we will learn more about PostgreSQL extension framework, how are they built, look at some popular extensions, management of these extensions in your deployments.
PostgreSQL is one of the most advanced relational databases. It offers superb replication capabilities. The most important features are: Streaming replication, Point-In-Time-Recovery, advanced monitoring, etc.
by Mahesh Pakal, AWS
PostgreSQL is a powerful, enterprise class open source object-relational database system with an emphasis on extensibility and standards-compliance. PostgreSQL boasts many sophisticated features and runs stored procedures in more than a dozen programming languages. We’ll explore the advantages and limitations of PostgreSQL, examples of where it is best suited for use, and examples of who is using PostgreSQL to power their applications.
This technical presentation by EDB Dave Thomas, Systems Engineer provides an overview of:
1) BGWriter/Writer Process
2) Wall Writer Process
3) Stats Collector Process
4) Autovacuum Launch Process
5) Syslogger Process/Logger process
6) Archiver Process
7) WAL Send/Receive Processes
Operating System
Topic Memory Management
for Btech/Bsc (C.S)/BCA...
Memory management is the functionality of an operating system which handles or manages primary memory. Memory management keeps track of each and every memory location either it is allocated to some process or it is free. It checks how much memory is to be allocated to processes. It decides which process will get memory at what time. It tracks whenever some memory gets freed or unallocated and correspondingly it updates the status.
Investigate SQL Server Memory Like Sherlock HolmesRichard Douglas
Memory is one part of the holy trinity of resources consumed by SQL Server, the others being CPU and disk. Most people know how to look at disk latency and throughput and then take remedial measures to fix those issues. But what about memory issues?
In this session, you will learn how SQL Server uses memory and various caches, how to gauge memory pressure, and how to address the significant problems it can cause.
You will leave with a much clearer understanding of how to monitor and manage memory consumption within SQL Server using native Dynamic Management Objects.
Optimizing elastic search on google compute engineBhuvaneshwaran R
If you are running the elastic search clusters on the GCE, then we need to take a look at the Capacity planning, OS level and Elasticsearch level optimization. I have presented this at GDG Delhi on Feb 22,2020.
Training Webinar: Enterprise application performance with distributed cachingOutSystems
2nd Session - Distributed Caching:
- What is Distributed Caching
- Performance hurdles solved by Distributed Caching
- When to use Distributed Caching
- Patterns to Populate a Distributed Cache
- How to use Distributed Caching in OutSystems
Free Online training: https://www.outsystems.com/learn/courses/
Follow us on Twitter http://www.twitter.com/OutSystemsDev
Like us on Facebook http://www.Facebook.com/OutSystemsDev
Similar to Postgresql Database Administration Basic - Day1 (20)
Covered Database Maintenance & Performance and Concurrency :
1. PostgreSQL Tuning and Performance
2. Find and Tune Slow Running Queries
3. Collecting regular statistics from pg_stat* views
4. Finding out what makes SQL slow
5. Speeding up queries without rewriting them
6. Discovering why a query is not using an index
7. Forcing a query to use an index
8. EXPLAIN and SQL Execution
9. Workload Analysis
Covered:
1. Databases and Schemas
2. Tablespaces
3. Data Type
4. Exploring Databases
5. Locating the database server's message log
6. Locating the database's system identifier
7. Listing databases on this database server
8. How much disk space does a table use?
9. Which are my biggest tables?
10. How many rows are there in a table?
11. Quickly estimating the number of rows in a table
12. Understanding object dependencies
The Art of the Pitch: WordPress Relationships and SalesLaura Byrne
Clients don’t know what they don’t know. What web solutions are right for them? How does WordPress come into the picture? How do you make sure you understand scope and timeline? What do you do if sometime changes?
All these questions and more will be explored as we talk about matching clients’ needs with what your agency offers without pulling teeth or pulling your hair out. Practical tips, and strategies for successful relationship building that leads to closing the deal.
Elevating Tactical DDD Patterns Through Object CalisthenicsDorra BARTAGUIZ
After immersing yourself in the blue book and its red counterpart, attending DDD-focused conferences, and applying tactical patterns, you're left with a crucial question: How do I ensure my design is effective? Tactical patterns within Domain-Driven Design (DDD) serve as guiding principles for creating clear and manageable domain models. However, achieving success with these patterns requires additional guidance. Interestingly, we've observed that a set of constraints initially designed for training purposes remarkably aligns with effective pattern implementation, offering a more ‘mechanical’ approach. Let's explore together how Object Calisthenics can elevate the design of your tactical DDD patterns, offering concrete help for those venturing into DDD for the first time!
UiPath Test Automation using UiPath Test Suite series, part 4DianaGray10
Welcome to UiPath Test Automation using UiPath Test Suite series part 4. In this session, we will cover Test Manager overview along with SAP heatmap.
The UiPath Test Manager overview with SAP heatmap webinar offers a concise yet comprehensive exploration of the role of a Test Manager within SAP environments, coupled with the utilization of heatmaps for effective testing strategies.
Participants will gain insights into the responsibilities, challenges, and best practices associated with test management in SAP projects. Additionally, the webinar delves into the significance of heatmaps as a visual aid for identifying testing priorities, areas of risk, and resource allocation within SAP landscapes. Through this session, attendees can expect to enhance their understanding of test management principles while learning practical approaches to optimize testing processes in SAP environments using heatmap visualization techniques
What will you get from this session?
1. Insights into SAP testing best practices
2. Heatmap utilization for testing
3. Optimization of testing processes
4. Demo
Topics covered:
Execution from the test manager
Orchestrator execution result
Defect reporting
SAP heatmap example with demo
Speaker:
Deepak Rai, Automation Practice Lead, Boundaryless Group and UiPath MVP
Smart TV Buyer Insights Survey 2024 by 91mobiles.pdf91mobiles
91mobiles recently conducted a Smart TV Buyer Insights Survey in which we asked over 3,000 respondents about the TV they own, aspects they look at on a new TV, and their TV buying preferences.
LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...DanBrown980551
Do you want to learn how to model and simulate an electrical network from scratch in under an hour?
Then welcome to this PowSyBl workshop, hosted by Rte, the French Transmission System Operator (TSO)!
During the webinar, you will discover the PowSyBl ecosystem as well as handle and study an electrical network through an interactive Python notebook.
PowSyBl is an open source project hosted by LF Energy, which offers a comprehensive set of features for electrical grid modelling and simulation. Among other advanced features, PowSyBl provides:
- A fully editable and extendable library for grid component modelling;
- Visualization tools to display your network;
- Grid simulation tools, such as power flows, security analyses (with or without remedial actions) and sensitivity analyses;
The framework is mostly written in Java, with a Python binding so that Python developers can access PowSyBl functionalities as well.
What you will learn during the webinar:
- For beginners: discover PowSyBl's functionalities through a quick general presentation and the notebook, without needing any expert coding skills;
- For advanced developers: master the skills to efficiently apply PowSyBl functionalities to your real-world scenarios.
Observability Concepts EVERY Developer Should Know -- DeveloperWeek Europe.pdfPaige Cruz
Monitoring and observability aren’t traditionally found in software curriculums and many of us cobble this knowledge together from whatever vendor or ecosystem we were first introduced to and whatever is a part of your current company’s observability stack.
While the dev and ops silo continues to crumble….many organizations still relegate monitoring & observability as the purview of ops, infra and SRE teams. This is a mistake - achieving a highly observable system requires collaboration up and down the stack.
I, a former op, would like to extend an invitation to all application developers to join the observability party will share these foundational concepts to build on:
Builder.ai Founder Sachin Dev Duggal's Strategic Approach to Create an Innova...Ramesh Iyer
In today's fast-changing business world, Companies that adapt and embrace new ideas often need help to keep up with the competition. However, fostering a culture of innovation takes much work. It takes vision, leadership and willingness to take risks in the right proportion. Sachin Dev Duggal, co-founder of Builder.ai, has perfected the art of this balance, creating a company culture where creativity and growth are nurtured at each stage.
Accelerate your Kubernetes clusters with Varnish CachingThijs Feryn
A presentation about the usage and availability of Varnish on Kubernetes. This talk explores the capabilities of Varnish caching and shows how to use the Varnish Helm chart to deploy it to Kubernetes.
This presentation was delivered at K8SUG Singapore. See https://feryn.eu/presentations/accelerate-your-kubernetes-clusters-with-varnish-caching-k8sug-singapore-28-2024 for more details.
PHP Frameworks: I want to break free (IPC Berlin 2024)Ralf Eggert
In this presentation, we examine the challenges and limitations of relying too heavily on PHP frameworks in web development. We discuss the history of PHP and its frameworks to understand how this dependence has evolved. The focus will be on providing concrete tips and strategies to reduce reliance on these frameworks, based on real-world examples and practical considerations. The goal is to equip developers with the skills and knowledge to create more flexible and future-proof web applications. We'll explore the importance of maintaining autonomy in a rapidly changing tech landscape and how to make informed decisions in PHP development.
This talk is aimed at encouraging a more independent approach to using PHP frameworks, moving towards a more flexible and future-proof approach to PHP development.
Welocme to ViralQR, your best QR code generator.ViralQR
Welcome to ViralQR, your best QR code generator available on the market!
At ViralQR, we design static and dynamic QR codes. Our mission is to make business operations easier and customer engagement more powerful through the use of QR technology. Be it a small-scale business or a huge enterprise, our easy-to-use platform provides multiple choices that can be tailored according to your company's branding and marketing strategies.
Our Vision
We are here to make the process of creating QR codes easy and smooth, thus enhancing customer interaction and making business more fluid. We very strongly believe in the ability of QR codes to change the world for businesses in their interaction with customers and are set on making that technology accessible and usable far and wide.
Our Achievements
Ever since its inception, we have successfully served many clients by offering QR codes in their marketing, service delivery, and collection of feedback across various industries. Our platform has been recognized for its ease of use and amazing features, which helped a business to make QR codes.
Our Services
At ViralQR, here is a comprehensive suite of services that caters to your very needs:
Static QR Codes: Create free static QR codes. These QR codes are able to store significant information such as URLs, vCards, plain text, emails and SMS, Wi-Fi credentials, and Bitcoin addresses.
Dynamic QR codes: These also have all the advanced features but are subscription-based. They can directly link to PDF files, images, micro-landing pages, social accounts, review forms, business pages, and applications. In addition, they can be branded with CTAs, frames, patterns, colors, and logos to enhance your branding.
Pricing and Packages
Additionally, there is a 14-day free offer to ViralQR, which is an exceptional opportunity for new users to take a feel of this platform. One can easily subscribe from there and experience the full dynamic of using QR codes. The subscription plans are not only meant for business; they are priced very flexibly so that literally every business could afford to benefit from our service.
Why choose us?
ViralQR will provide services for marketing, advertising, catering, retail, and the like. The QR codes can be posted on fliers, packaging, merchandise, and banners, as well as to substitute for cash and cards in a restaurant or coffee shop. With QR codes integrated into your business, improve customer engagement and streamline operations.
Comprehensive Analytics
Subscribers of ViralQR receive detailed analytics and tracking tools in light of having a view of the core values of QR code performance. Our analytics dashboard shows aggregate views and unique views, as well as detailed information about each impression, including time, device, browser, and estimated location by city and country.
So, thank you for choosing ViralQR; we have an offer of nothing but the best in terms of QR code services to meet business diversity!
UiPath Test Automation using UiPath Test Suite series, part 3DianaGray10
Welcome to UiPath Test Automation using UiPath Test Suite series part 3. In this session, we will cover desktop automation along with UI automation.
Topics covered:
UI automation Introduction,
UI automation Sample
Desktop automation flow
Pradeep Chinnala, Senior Consultant Automation Developer @WonderBotz and UiPath MVP
Deepak Rai, Automation Practice Lead, Boundaryless Group and UiPath MVP
2. PostgreSQL
Architecture,
Installation &
Configuration
Part-1
• Postgres Architecture
• Process and Memory Architecture
• Postgres Server Process
• Backend Processes &Background Processes
• Buffer Manager Structure
• Write Ahead Logging
• PostgreSQL Installation
• Setting Environment Variables
3. Architecture of PostgreSQL
• The physical structure of
PostgreSQL is very simple, it
consists of the following
components:
Shared Memory
Background processes
Data directory structure /
Data files
4. Data Files / Data Directory Structure
• PostgreSQL consist of multiple
databases this is called a database
cluster. When we initialize
PostgreSQL database template0,
template1 and Postgres databases
are created.
• Template0 and template1 are
template databases for new
database creation of user it contains
the system catalog tables.
• The user database will be created
by cloning the template1 database.
5. Process Architecture
• PostgreSQL is a client/server type relational database
management system with the multi-process architecture and
runs on a single host.
• A collection of multiple processes cooperatively managing
one database cluster is usually referred to as a 'PostgreSQL
server', and it contains the following types of processes:
• A postgres server process is a parent of all processes
related to a database cluster management.
• Each backend process handles all queries and statements
issued by a connected client.
• Various background processes perform processes of each
feature (e.g., VACUUM and CHECKPOINT processes) for
database management.
• In the replication associated processes, they perform the
streaming replication.
• In the background worker process supported from
version 9.3
6. Postgres Server Process
• a postgres server process is a parent of all in a PostgreSQL server. In the earlier versions, it
was called ‘postmaster’.
• By executing the pg_ctl utility with start option, a postgres server process starts up. Then,
it allocates a shared memory area in memory, starts various background processes, starts
replication associated processes and background worker processes if necessary, and waits
for connection requests from clients. Whenever receiving a connection request from a
client, it starts a backend process. (And then, the started backend process handles all
queries issued by the connected client.)
• A postgres server process listens to one network port, the default port is 5432. Although
more than one PostgreSQL server can be run on the same host, each server should be set
to listen to different port number in each other, e.g., 5432, 5433, etc.
7. Backend Processes
• A backend process, which is also called postgres, is started by the postgres server process and handles all queries issued
by one connected client. It communicates with the client by a single TCP connection, and terminates when the client
gets disconnected.
• As it is allowed to operate only one database, you have to specify a database you want to use explicitly when connecting
to a PostgreSQL server.
• PostgreSQL allows multiple clients to connect simultaneously; the configuration parameter max_connections controls
the maximum number of the clients (default is 100).
• The maximum number of backend processes is set by the max_connections parameter, and the default value is 100.
• The backend process performs the query request of the user process and then transmits the result. Some memory
structures are required for query execution, which is called local memory. The main parameters associated with local
memory are:
• work_mem Space used for sorting, bitmap operations, hash joins, and merge joins. The default setting is 4 MB.
• Maintenance_work_mem Space used for Vacuum and CREATE INDEX . The default setting is 64 MB.
• Temp_buffers Space used for temporary tables. The default setting is 8 MB.
8. Background Processes
process description
background writer • In this process, dirty pages on the shared buffer pool are written to a persistent storage (e.g., HDD, SSD) on a regular basis gradually.
• In other word this process writes and flushes periodically the WAL data on the WAL buffer to persistent storage.
checkpointer • The actual work of this process is when a checkpoint occurs it will write dirty buffer into a file.
• Checkpointer will write all dirty pages from memory to disk and clean shared buffers area. If PostgreSQL database is crashed, we can
measure data loss between last checkpoint time and PostgreSQL stopped time. The checkpoint command forces an immediate checkpoint
when the command is executed manually. Only database superuser can call checkpoint.
The checkpoint will occur in the following scenarios:
• The pages are dirty.
• Starting and restarting the DB server (pg_ctl STOP | RESTART).
• Issue of the commit.
• Starting the database backup (pg_start_backup).
• Stopping the database backup (pg_stop_backup).
• Creation of the database.
autovacuum launcher • The autovacuum-worker processes are invoked for vacuum process periodically. (More precisely, it requests to create the autovacuum
workers to the postgres server.)
• When autovacuum is enabled, this process has the responsibility of the autovacuum daemon to carry vacuum operations on bloated tables.
This process relies on the stats collector process for perfect table analysis.
WAL writer This process writes and flushes periodically the WAL data on the WAL buffer to persistent storage.
statistics collector In this process, statistics information such as for pg_stat_activity and for pg_stat_database, etc. is collected.
Logging Collector(logger) • This process also called a logger. It will write a WAL buffer to WAL file.
9. Background Processes
process description
Memory for Locks /
Lock Space
• In this process, dirty pages on the shared buffer pool are written to a persistent storage (e.g., HDD, SSD) on a regular basis gradually.
• In other word this process writes and flushes periodically the WAL data on the WAL buffer to persistent storage.
checkpointer • The actual work of this process is when a checkpoint occurs it will write dirty buffer into a file.
• Checkpointer will write all dirty pages from memory to disk and clean shared buffers area. If PostgreSQL database is crashed, we can
measure data loss between last checkpoint time and PostgreSQL stopped time. The checkpoint command forces an immediate checkpoint
when the command is executed manually. Only database superuser can call checkpoint.
The checkpoint will occur in the following scenarios:
• The pages are dirty.
• Starting and restarting the DB server (pg_ctl STOP | RESTART).
• Issue of the commit.
• Starting the database backup (pg_start_backup).
• Stopping the database backup (pg_stop_backup).
• Creation of the database.
autovacuum launcher • The autovacuum-worker processes are invoked for vacuum process periodically. (More precisely, it requests to create the autovacuum
workers to the postgres server.)
• When autovacuum is enabled, this process has the responsibility of the autovacuum daemon to carry vacuum operations on bloated tables.
This process relies on the stats collector process for perfect table analysis.
WAL writer This process writes and flushes periodically the WAL data on the WAL buffer to persistent storage.
statistics collector In this process, statistics information such as for pg_stat_activity and for pg_stat_database, etc. is collected.
10. Memory Architecture
• Memory architecture in
PostgreSQL can be classified into
two broad categories:
1. Local memory area – allocated
by each backend process for
its own use.
2. Shared memory area – used
by all processes of a
PostgreSQL server.
11. Local memory area
work_mem • Executor uses this area for sorting tuples by
ORDER BY and DISTINCT operations, and for
joining tables by merge-join and hash-join
operations.
• The default value of work memory in 9.3 and
the older version is 1 megabyte (1 MB) from
9.4 and later default value of work memory
is 4 megabytes ( default size 4 MB).
maintenance_
work_mem
• We need to specify the maximum amount of
memory for database maintenance
operations such as VACUUM, ANALYZE,
ALTER TABLE, CREATE INDEX, and ADD
FOREIGN KEY, etc.
• The default value of maintenance work
memory in 9.3 and the older version is 16
megabytes (16 MB) from 9.4 and later
default value of maintenance work memory
is 64 megabytes ( Default Size 64 MB).
• It is safe to set maintenance work memory is
large as compared to work memory. Larger
settings will improve the performance of
maintenance (VACUUM, ANALYZE, ALTER
TABLE, CREATE INDEX, and ADD FOREIGN
KEY, etc.) operations.
temp_buffers Executor uses this area for storing temporary
Shared memory area
shared
buffer pool
• PostgreSQL loads pages within tables and indexes from a
persistent storage to here and operates them directly.
• We need to set some amount of memory to a database
server for uses of shared buffers. The default value of
shared buffers in 9.2 and the older version is 32 megabytes
(32 MB) from 9.3 and the later default value of shared
buffers is 128 megabytes ( Default size 128 MB).
• If we have a dedicated server for PostgreSQL, reasonable
starting to set shared buffers value is 25% of total memory.
The purpose of shared buffers is to minimize server DISK
IO.
WAL buffer To ensure that no data has been lost by server failures,
PostgreSQL supports the WAL mechanism. WAL data (also
referred to as XLOG records) are transaction log in PostgreSQL;
and WAL buffer is a buffering area of the WAL data before
writing to a persistent storage.
WAL buffers temporarily store changes in the database, which
changes in the WAL buffers are written to the WAL file at a
predetermined time. At the time of backup and recovery, WAL
buffers and WAL files are very important to recover the data at
some peak of time.
The minimum value of shared buffers is 32 KB. If we set this
parameter as wal_buffers = -1 it will set based on
shared_buffers.
commit log • Commit Log(CLOG) keeps the states of all transactions
MEMORY ARCHITECTURE
12. Installing PostgreSQL on Linux/Unix
Follow the given steps to install PostgreSQL on your Linux machine.
Make sure you are logged in as root before you proceed for the
installation.
Pick the version number of PostgreSQL you want and, as exactly as
possible, the platform you want from EnterpriseDB
I downloaded postgresql-9.2.4-1-linux-x64.run for my 64 bit CentOS-6
machine. Now, let us execute it as follows −
[root@host]# chmod +x postgresql-9.2.4-1-linux-x64.run
[root@host]# ./postgresql-9.2.4-1-linux-x64.run
------------------------------------------------------------------------
Welcome to the PostgreSQL Setup Wizard.
------------------------------------------------------------------------
Please specify the directory where PostgreSQL will be installed.
Installation Directory [/opt/PostgreSQL/9.2]:
Once you launch the installer, it asks you a few basic questions like
location of the installation, password of the user who will use database,
port number, etc. So keep all of them at their default values except
password, which you can provide password as per your choice. It will
install PostgreSQL at your Linux machine and will display the following
message −
Please wait while Setup installs PostgreSQL on your computer.
Installing
0% ______________ 50% ______________ 100%
#########################################
-----------------------------------------------------------------------
Setup has finished installing PostgreSQL on your computer.
Follow the following post-installation steps to create your
database −
[root@host]# su - postgres
Password:
bash-4.1$ createdb testdb
bash-4.1$ psql testdb
psql (8.4.13, server 9.2.4)
test=#
You can start/restart postgres server in case it is not running using
the following command −
[root@host]# service postgresql restart
Stopping postgresql service: [ OK ]
Starting postgresql service: [ OK ]
If your installation was correct, you will have PotsgreSQL prompt
test=# as shown above.
13. Installing PostgreSQL on Windows
• Follow the given steps to install
PostgreSQL on your Windows machine.
Make sure you have turned Third Party
Antivirus off while installing.
• Pick the version number of PostgreSQL
you want and, as exactly as possible, the
platform you want from EnterpriseDB
• I downloaded postgresql-9.2.4-1-
windows.exe for my Windows PC
running in 32bit mode, so let us
run postgresql-9.2.4-1-windows.exe as
administrator to install PostgreSQL.
Select the location where you want to
install it. By default, it is installed within
Program Files folder.
14. PostgreSQL mostly tuned parameters
Listen_address
No doubt, you need to change it to let PostgreSQL know what IP address(es) to
listen on. If your postgres is not just used for localhost, add or change it
accordingly. Also, you need to setup access rules in pg_hba.conf.
listen_addresses = 'localhost,<dbserver>'
default value: "localhost"
Max_connections
max_connections = 2000
default: max_connections = 100 This parameter really depends on your application, I set it
to 2000 for most of connections are short lifetime SQL, and connection are reused
Buffer size
shared_buffers = 3GB
effective_cache_size = 16GB
Default:
shared_buffers = 32MB
effective_cache_size = 128MB
Work memory
work_mem = 32MB
maintenance_work_mem = 256MB
Default:
work_mem = 1MB and maintenance_work_mem = 16MB work_mem is for each
connection, while maintenance_work_mem is for maintenance tasks for example, vacuum,
create, index etc.. Set work_mem big is good for sorting types of query, but not for small
query. This has to be considered with max_connections
Check_segments
checkpoint_segments = 32
default:
checkpoint_segments=3 Maximum number of log file segments between
automatic WAL checkpoints (each segment is normally 16 megabytes)
Wal_level
wal_level = archiv
default:
wal_level = minimal wal_level determines how much information is written to the WAL. The
default value is minimal, which writes only the information needed to recover from a crash
or immediate shutdown. archive adds logging required for WAL archiving
ARCHIVE
Not mandatery for all cases. Here is my setting
archive_mode = on
archive_command = '/bin/cp -p %p /home/backups/archivelogs/%f
</dev/null'
AUTOVACUUM
autovacuum is a quite hot topic in Postgres, for most of time, global autovacuum doesn't
work well,
track_counts = on
autovacuum = on
autovacuum_max_workers
autovacuum_vacuum_threshold = 500
15. PostgreSQL table access statistics, index io statistics
#1 table access statistics
#select schemaname,relname,seq_scan,idx_scan,cast(idx_scan as numeric) /
(idx_scan + seq_scan)
as idx_scan_pct
from pg_stat_user_tables where (idx_scan +seq_scan) >0 order by idx_scan_pct;
Higher pct means more likely your postgreSQL is using index scan, which is good.
#2 table io statistics
#select relname,cast(heap_blks_hit as numeric) /(heap_blks_hit +heap_blks_read)
as hit_pct,heap_blks_hit,heap_blks_read from pg_statio_user_tables
where (heap_blks_hit + heap_blks_read) >0 order by hit_pct; Higher hit_pct means
more likely the data required is cached.
#3 index access statistics
this shows all of the disk i/o for every index on each table
#select relname,cast(idx_blks_hit as numeric) /(idx_blks_hit + idx_blks_read )
as hit_pct,idx_blks_hit,idx_blks_read from pg_statio_user_tables
where (idx_blks_hit +idx_blks_read) >0 order by hit_pct;
#4 index io statistics
#select indexrelname,cast(idx_blks_hit as numeric) /( idx_blks_hit + idx_blks_read)
as hit_pct,idx_blks_hit,idx_blks_read from pg_statio_user_indexes
where (idx_blks_hit +idx_blks_read)>0 order by hit_pct ;
#5 Less used indexes(from top to bottom)
#select
schemaname,relname,indexrelname,idx_scan,pg_size_pretty(pg_relation_size(i.in
dexrelid))
as index_size from pg_stat_user_indexes i join pg_index using (indexrelid)
where indisunique is false order by idx_scan,relname;
Note: The main thing that the counts in pg_stat_user_indexes are useful for is to
determining which indexes are actually being used by your application. Since
indexes add overhead to the system, but drop them with care.
To show current server configuration setting.
SHOW ALL;
SELECT name, setting, unit, context FROM pg_settings;
16. Write-Ahead Logging (WAL)- Parameter
Common Settings Checkpoints Archiving
#wal_level = minimal # minimal, archive, or hot_standby #
(change requires restart)
#fsync = on # turns forced synchronization on or off
#synchronous_commit = on # synchronization level; on, off, or local
#wal_sync_method = fsync # the default is the first option
#full_page_writes = on # recover from partial page writes
#wal_buffers = -1 # min 32kB, -1 sets based on shared_buffers
# (change requires restart)
#wal_writer_delay = 200ms # 1-10000 milliseconds
#commit_delay = 0 # range 0-100000, in microseconds
#commit_siblings = 5 # range 1-1000
#checkpoint_segments = 3 # in logfile segments, min 1, 16MB
each#archive_mode = off # allows archiving to be done #
(change requires restart)
#archive_command = '' # command to use to archive a logfile
segment
#archive_timeout = 0 # force a logfile segment switch after this #
number of seconds; 0 disables
#checkpoint_timeout = 5min # range 30s-1h
#checkpoint_completion_target = 0.5 # checkpoint target duration, 0.0
- 1.0
#checkpoint_warning = 30s # 0 disables
#archive_mode = off # allows archiving to be done # (change
requires restart)
#archive_command = '' # command to use to archive a logfile
segment
#archive_timeout = 0 # force a logfile segment switch after this #
number of seconds; 0 disables
Configuring the PostgreSQL Archive Log Directory
• Archive log files are stored in the Archive Log directory. Ensure to follow the below checkpoints before running the PostgreSQL File System backup.
• Specify the Archive log directory path in the postgresql.conf file prior to performing the PostgreSQL FS backup. Make sure that this path does not point to pg_log or log directories and pg_xlog or pg_wal directories.
archive_command = 'cp %p /opt/wal/%f' #UNIX
archive_command = 'copy "%p" "D:PostgreSQLwal%f"' #Windows
• The following configuration to turn on the archive_mode. This feature is not supported for PostgreSQL 8.2 and earlier versions.
archive_mode = on
• For PostgreSQL 9.x.x version, use the following configuration.
Set wal_level = archive instead of default wal_level = minimal
• From PostgreSQL 10.x.x version onwards, use the following configuration.
Set wal_level = replica
• Verify that the archive command provided in the postgresql.conf file is correct. You can test this by running the following commands and verifying that they successfully complete.
Select pg_start_backup(‘Testing’);
Select pg_stop_backup();
17. Write-Ahead Logging (WAL)- Parameter
•WAL Archive log In PostgreSQL database system, the actual database 'writes' to an addition file called write-ahead log (WAL) to disk.
•It contains a record of writes that made in the database system. In the case of Crash, database can be repaired/recovered from these
records.
•Normally, the write-ahead log logs at regular intervals (called Checkpoints) matched against the database and then deleted because it no
longer is required. You can also use the WAL as a backup because, there is a record of all writes made to the database.
WAL Archiving Concept :
In pg_xlog write ahead logs are stored. It is the log file, where all the logs are stored of committed and uncommitted transaction. It
contains max 6 logs, and last one overwrites. If archiver is on, it moves there.
•The write-ahead log is composed of each 16 MB large, which are called segments.
•The WALs reside under pg_xlog directory and it is the subdirectory of 'data directory'. The filenames will have numerical(0-9) and
character(a-z) named in ascending order by PostgreSQL Instance. To perform a backup on the basis of WAL, one needs a basic backup that
is, a complete backup of the data directory, and the WAL Segments between the base backup and the current date.
•PostgreSQL managing WAL files by removing or adding according to setting with wal_keep_segments, max_wal_size and min_wal_size.
20. Benefits of WAL
The first obvious benefit of using WAL is a significantly reduced number of disk writes, since only the log file needs to be
flushed to disk at the time of transaction commit; in multiuser environments, commits of many transactions may be
accomplished with a single fsync() of the log file. Furthermore, the log file is written sequentially, and so the cost of
syncing the log is much less than the cost of flushing the data pages.
The next benefit is consistency of the data pages. The truth is that, before WAL, PostgreSQL was never able to guarantee
consistency in the case of a crash. Before WAL, any crash during writing could result in:
1. index rows pointing to nonexistent table rows
2. index rows lost in split operations
3. totally corrupted table or index page content, because of partially written data pages
Problems with indexes (problems 1 and 2) could possibly have been fixed by additional fsync() calls, but it is not obvious
how to handle the last case without WAL; WAL saves the entire data page content in the log if that is required to ensure
page consistency for after-crash recovery.
WAL is significantly faster in most scenarios.
WAL provides more concurrency as readers do not block writers and a writer does not block readers. Reading and
writing can proceed concurrently.
Disk I/O operations tends to be more sequential using WAL.
WAL uses many fewer fsync() operations and is thus less vulnerable to problems on systems where the fsync() system
call is broken.
21. Postgres WAL Config
The postgres.conf file is set as follows for WAL archiving.
#------------------------------------------------------------------------------
# WRITE AHEAD LOG
#------------------------------------------------------------------------------
# - Settings -
#wal_level = minimal # minimal, archive, or hot_standby
wal_level = archive
# - Archiving -
archive_mode = on
#archive_mode = off # allows archiving to be done
# (change requires restart)
archive_command = 'cp %p /pgsql-backup/archive/postgres1/%f'
# command to use to archive a logfile segment
# archive_command = ''
# command to use to archive a logfile segment
# placeholders: %p = path of file to archive
# %f = file name only
# e.g. 'test ! -f /mnt/server/archivedir/%f && cp %p /mnt/server/archivedir/%f'
• #archive_timeout = 0 # force a logfile segment switch after this
• # number of seconds; 0 disables
22. How do I read a Wal file in PostgreSQL?
• First get the source for the version of Postgres that you wish to view
WAL data for. ./configure and make this, but no need to install.
• Then copy the xlogdump folder to the contrib folder (a git clone in
that folder works fine)
• Run make for xlogdump - it should find the parent postgres structure
and build the binary.