Input-output performance problems are on every day agenda for DBAs since the databases exist. Volume of data grows rapidly and you need to get your data fast from the disk and moreover - fast to the disk. For most databases there is a more or less easy to find checklist of recommended Linux settings to maximize IO throughput. In most cases that checklist is good enough. But it is always better to understand how it works, especially if you run into some corner-cases. This talk is about how IO in Linux works, how database pages travel from disk level to database own shared memory and back and what kind of mechanisms exist to control this. We will discuss memory structures, swap and page-out daemons, filesystems, schedullers and IO methods. Some fundamental differences in IO approaches between PostgreSQL, Oracle and MySQL will be covered.
This technical presentation by EDB Dave Thomas, Systems Engineer provides an overview of:
1) BGWriter/Writer Process
2) Wall Writer Process
3) Stats Collector Process
4) Autovacuum Launch Process
5) Syslogger Process/Logger process
6) Archiver Process
7) WAL Send/Receive Processes
Ilya Kosmodemiansky - An ultimate guide to upgrading your PostgreSQL installa...PostgreSQL-Consulting
Even an experienced PostgreSQL DBA can not always say that upgrading between major versions of Postgres is an easy task, especially if there are some special requirements, such as downtime limitations or if something goes wrong. For less experienced DBAs anything more complex than dump/restore can be frustrating.
In this talk I will describe why we need a special procedure to upgrade between major versions, how that can be achieved and what sort of problems can occur. I will review all possible ways to upgrade your cluster from classical pg_upgrade to old-school slony or modern methods like logical replication. For all approaches, I will give a brief explanation how it works (limited by the scope of this talk of course), examples how to perform upgrade and some advice on potentially problematic steps. Besides I will touch upon such topics as integration of upgrade tools and procedures with other software — connection brokers, operating system package managers, automation tools, etc. This talk would not be complete if I do not cover cases when something goes wrong and how to deal with such cases.
Tuning the Memory and Optimizer Parameters - This topic has come quite often in requests and suggestions from our regular viewers and followers of our Postgres Hangouts.
So we picked it up as the topic for this month's Postgres Hangout. This time we will try to explore and share on Memory and Memory .
Join us as we discuss what impacts your decision before you put a number against each of these important parameters!
PostgreSQL worst practices, version PGConf.US 2017 by Ilya KosmodemianskyPostgreSQL-Consulting
This talk is prepared as a bunch of slides, where each slide describes a really bad way people can screw up their PostgreSQL database and provides a weight - how frequently I saw that kind of problem. Right before the talk I will reshuffle the deck to draw twenty random slides and explain you why such practices are bad and how to avoid running into them.
Building tungsten-clusters-with-postgre sql-hot-standby-and-streaming-replica...Command Prompt., Inc
Alex Alexander & Linas Virbalas
Hot standby and streaming replication will move the needle forward for high availability and scaling for a wide number of applications. Tungsten already supports clustering using warm standby. In this talk we will describe how to build clusters using the new PostgreSQL features and give our report from the trenches.
This talk will cover how hot standby and streaming replication work from a user perspective, then dive into a description of how to use them, taking Tungsten as an example. We'll cover the following issues:
* Configuration of warm standby and streaming replication
* Provisioning new standby instances
* Strategies for balancing reads across primary and standby database
* Managing failover
* Troubleshooting and gotchas
Please join us for an enlightening discussion a set of PostgreSQL features that are interesting to a wide range of PostgreSQL users.
This technical presentation by EDB Dave Thomas, Systems Engineer provides an overview of:
1) BGWriter/Writer Process
2) Wall Writer Process
3) Stats Collector Process
4) Autovacuum Launch Process
5) Syslogger Process/Logger process
6) Archiver Process
7) WAL Send/Receive Processes
Ilya Kosmodemiansky - An ultimate guide to upgrading your PostgreSQL installa...PostgreSQL-Consulting
Even an experienced PostgreSQL DBA can not always say that upgrading between major versions of Postgres is an easy task, especially if there are some special requirements, such as downtime limitations or if something goes wrong. For less experienced DBAs anything more complex than dump/restore can be frustrating.
In this talk I will describe why we need a special procedure to upgrade between major versions, how that can be achieved and what sort of problems can occur. I will review all possible ways to upgrade your cluster from classical pg_upgrade to old-school slony or modern methods like logical replication. For all approaches, I will give a brief explanation how it works (limited by the scope of this talk of course), examples how to perform upgrade and some advice on potentially problematic steps. Besides I will touch upon such topics as integration of upgrade tools and procedures with other software — connection brokers, operating system package managers, automation tools, etc. This talk would not be complete if I do not cover cases when something goes wrong and how to deal with such cases.
Tuning the Memory and Optimizer Parameters - This topic has come quite often in requests and suggestions from our regular viewers and followers of our Postgres Hangouts.
So we picked it up as the topic for this month's Postgres Hangout. This time we will try to explore and share on Memory and Memory .
Join us as we discuss what impacts your decision before you put a number against each of these important parameters!
PostgreSQL worst practices, version PGConf.US 2017 by Ilya KosmodemianskyPostgreSQL-Consulting
This talk is prepared as a bunch of slides, where each slide describes a really bad way people can screw up their PostgreSQL database and provides a weight - how frequently I saw that kind of problem. Right before the talk I will reshuffle the deck to draw twenty random slides and explain you why such practices are bad and how to avoid running into them.
Building tungsten-clusters-with-postgre sql-hot-standby-and-streaming-replica...Command Prompt., Inc
Alex Alexander & Linas Virbalas
Hot standby and streaming replication will move the needle forward for high availability and scaling for a wide number of applications. Tungsten already supports clustering using warm standby. In this talk we will describe how to build clusters using the new PostgreSQL features and give our report from the trenches.
This talk will cover how hot standby and streaming replication work from a user perspective, then dive into a description of how to use them, taking Tungsten as an example. We'll cover the following issues:
* Configuration of warm standby and streaming replication
* Provisioning new standby instances
* Strategies for balancing reads across primary and standby database
* Managing failover
* Troubleshooting and gotchas
Please join us for an enlightening discussion a set of PostgreSQL features that are interesting to a wide range of PostgreSQL users.
OpenStack is rapidly gaining popularity with businesses as they realize the benefits of a private cloud architecture. This presentation was delivered by Dave Page, Chief Architect, Tools & Installers at EnterpriseDB & PostgreSQL Core Team member during PG Open 2014. He addressed some of the common components of OpenStack deployments, how they can affect Postgres servers, and how users might best utilize some of the features they offer when deploying Postgres, including:
• Different configurations for the Nova compute service
• Use of the Cinder block store
• Virtual networking options with Neutron
• WAL archiving with the Swift object store
Architecture for building scalable and highly available Postgres ClusterAshnikbiz
As PostgreSQL has made way into business critical applications, many customers who are using Oracle RAC for high availability and load balancing have asked for similar functionality for using PostgreSQL.
In this Hangout session we would discuss architecture and alternatives, based on real life experience, for achieving high availability and load balancing functionality when you deploy PostgreSQL. We will also present some of the key tools and how to deploy them for effectiveness of this architecture.
Right-Sizing your SQL Server Virtual Machineheraflux
Virtualizing your top-tier production SQL Servers is not as easy as P2V’ing it. Sometimes allocating more resources to the VM is the wrong approach, and getting it wrong will silently hurt performance. What is the most effective method for determining the ‘right’ amount of resources to allocate? What happens if the workload changes a month from now?
The methods for understanding the performance of your mission-critical SQL Servers gathered over the past ten years of SQL Server virtualization will be addressed, and valuable processes for performance statistic collection and analysis will be displayed. Come learn how to properly ‘right-size’ the resources allocated to a VM, improve the performance of your SQL Servers, and keep it maximized well into the future.
This ppt was used by Devrim at pgDay Asia 2017. He talked about some important facts about WAL - Transaction Logs or xlogs in PostgreSQL. Some of these can really come handy on a bad day
Countdown to PostgreSQL v9.5 - Foriegn Tables can be part of Inheritance Tree Ashnikbiz
Distributed databases and horizontal scale up is one of the key demands in today's date. PostgreSQL already had some vertical scaling features and horizontal scale-up by adding disks and table partitioning/child tables. With release of v9.5, PostgreSQL will get basic foundation for native sharing capability. From v9.5 Foreign Tables will be able to participate in Inheritance Tree as a child or parent table i.e. one can have table partitions residing on different system.
In our countdown to v9.5 series of hangouts, we will be covering some of the great features of PostgreSQL v9.5 and what is their real life applicability. In the first hangout in this series we will be talking about-
- The feature of foreign partitions/child tables
- Syntax and usage
- EXPLAIN plan demo
- Use cases and benefits
Join us for more and send us your queries on success@ashnik.com
PostgreSQL worst practices, version FOSDEM PGDay 2017 by Ilya KosmodemianskyPostgreSQL-Consulting
This talk is prepared as a bunch of slides, where each slide describes a really bad way people can screw up their PostgreSQL database and provides a weight - how frequently I saw that kind of problem. Right before the talk I will reshuffle the deck to draw ten random slides and explain you why such practices are bad and how to avoid running into them.
The revolt against SQL continues at a steady but considerably slower pace. Bespoke database software seems to crop up daily in the name of performance or functionality. This talk will examine the ever growing field of monitoring systems and their respective databases, and look in depth as to how Postgres can be used in a number of these places. Systems of this nature are typically tasked with collecting and storing metrics from your infrastructure, drawing pretty graphs, and nagging you when things break.
Forms of data stored by these systems are nothing to be afraid of - they often include:
- Time series metrics - the history of a measurement over time, e.g. temperatures
- Logs - unstructured text emitted by applications, operating systems and hardware
- Events - schema-less but well structured notifications
An assertion of this talk is that for a majority of use cases, Postgres is more than capable of storing all of this data. We will attempt to replace numerous well known pieces of software with just one Postgres database. Of course we are told to use the right tool for the job, but having to learn and operate a single tool is a huge operational advantage.
We’ll get quite technical in this talk, take a look the data models and access patterns required, and how this can be fitted into the general purpose environment of Postgres. Additionally, it is always constructive to look at what can be problematic, and not just focus on the positives, and why many turn to other bespoke solutions.
OpenStack is rapidly gaining popularity with businesses as they realize the benefits of a private cloud architecture. This presentation was delivered by Dave Page, Chief Architect, Tools & Installers at EnterpriseDB & PostgreSQL Core Team member during PG Open 2014. He addressed some of the common components of OpenStack deployments, how they can affect Postgres servers, and how users might best utilize some of the features they offer when deploying Postgres, including:
• Different configurations for the Nova compute service
• Use of the Cinder block store
• Virtual networking options with Neutron
• WAL archiving with the Swift object store
Architecture for building scalable and highly available Postgres ClusterAshnikbiz
As PostgreSQL has made way into business critical applications, many customers who are using Oracle RAC for high availability and load balancing have asked for similar functionality for using PostgreSQL.
In this Hangout session we would discuss architecture and alternatives, based on real life experience, for achieving high availability and load balancing functionality when you deploy PostgreSQL. We will also present some of the key tools and how to deploy them for effectiveness of this architecture.
Right-Sizing your SQL Server Virtual Machineheraflux
Virtualizing your top-tier production SQL Servers is not as easy as P2V’ing it. Sometimes allocating more resources to the VM is the wrong approach, and getting it wrong will silently hurt performance. What is the most effective method for determining the ‘right’ amount of resources to allocate? What happens if the workload changes a month from now?
The methods for understanding the performance of your mission-critical SQL Servers gathered over the past ten years of SQL Server virtualization will be addressed, and valuable processes for performance statistic collection and analysis will be displayed. Come learn how to properly ‘right-size’ the resources allocated to a VM, improve the performance of your SQL Servers, and keep it maximized well into the future.
This ppt was used by Devrim at pgDay Asia 2017. He talked about some important facts about WAL - Transaction Logs or xlogs in PostgreSQL. Some of these can really come handy on a bad day
Countdown to PostgreSQL v9.5 - Foriegn Tables can be part of Inheritance Tree Ashnikbiz
Distributed databases and horizontal scale up is one of the key demands in today's date. PostgreSQL already had some vertical scaling features and horizontal scale-up by adding disks and table partitioning/child tables. With release of v9.5, PostgreSQL will get basic foundation for native sharing capability. From v9.5 Foreign Tables will be able to participate in Inheritance Tree as a child or parent table i.e. one can have table partitions residing on different system.
In our countdown to v9.5 series of hangouts, we will be covering some of the great features of PostgreSQL v9.5 and what is their real life applicability. In the first hangout in this series we will be talking about-
- The feature of foreign partitions/child tables
- Syntax and usage
- EXPLAIN plan demo
- Use cases and benefits
Join us for more and send us your queries on success@ashnik.com
PostgreSQL worst practices, version FOSDEM PGDay 2017 by Ilya KosmodemianskyPostgreSQL-Consulting
This talk is prepared as a bunch of slides, where each slide describes a really bad way people can screw up their PostgreSQL database and provides a weight - how frequently I saw that kind of problem. Right before the talk I will reshuffle the deck to draw ten random slides and explain you why such practices are bad and how to avoid running into them.
The revolt against SQL continues at a steady but considerably slower pace. Bespoke database software seems to crop up daily in the name of performance or functionality. This talk will examine the ever growing field of monitoring systems and their respective databases, and look in depth as to how Postgres can be used in a number of these places. Systems of this nature are typically tasked with collecting and storing metrics from your infrastructure, drawing pretty graphs, and nagging you when things break.
Forms of data stored by these systems are nothing to be afraid of - they often include:
- Time series metrics - the history of a measurement over time, e.g. temperatures
- Logs - unstructured text emitted by applications, operating systems and hardware
- Events - schema-less but well structured notifications
An assertion of this talk is that for a majority of use cases, Postgres is more than capable of storing all of this data. We will attempt to replace numerous well known pieces of software with just one Postgres database. Of course we are told to use the right tool for the job, but having to learn and operate a single tool is a huge operational advantage.
We’ll get quite technical in this talk, take a look the data models and access patterns required, and how this can be fitted into the general purpose environment of Postgres. Additionally, it is always constructive to look at what can be problematic, and not just focus on the positives, and why many turn to other bespoke solutions.
Как в PostgreSQL устроено взаимодействие с диском, какие проблемы производительности при этом бывают и как их решать выбором подходящего hardware, настройками операционной системы и настройками PostgreSQL
#RuPostges в Yandex, эпизод 3. Что же нового в PostgreSQL 9.6Nikolay Samokhvalov
Первый релиз-кандидат версии 9.6 вышел 1 сентября, а это значит, что совсем скоро будет полноценный релиз. Все вокруг уже успели обсудить новинки, и теперь уже стыдно ничего не знать о таких вещах, как параллелизация выполнения запросов, pushdown для FDW, мониторинг waitlocks, полнотекстовый поиск по фразам или магический \gexec в psql. Чтобы никому не приходилось краснеть, мы быстро пройдёмся по всем основным и интересным моментам версии 9.6.
Linux IO internals for database administrators (SCaLE 2017 and PGDay Nordic 2...PostgreSQL-Consulting
Input-output performance problems are on every day agenda for DBAs since the databases exist. Volume of data grows rapidly and you need to get your data fast from the disk and moreover - fast to the disk. For most databases there is a more or less easy to find checklist of recommended Linux settings to maximize IO throughput. In most cases that checklist is good enough. But it is always better to understand how it works, especially if you run into some corner-cases. This talk is about how IO in Linux works, how database pages travel from disk level to database own shared memory and back and what kind of mechanisms exist to control this. We will discuss memory structures, swap and page-out daemons, filesystems, schedulers and IO methods. Some fundamental differences in IO approaches between PostgreSQL, Oracle and MySQL will be covered
Tuning Linux for your database FLOSSUK 2016Colin Charles
Some best practices about tuning Linux for your database workloads. The focus is not just on MySQL or MariaDB Server but also on understanding the OS from hardware/cloud, I/O, filesystems, memory, CPU, network, and resources.
NVDIMM is a standard for allowing non-volatile memory to be exposed to as normal RAM, which can be directly mapped to guests. This simple concept has the potential to dramatically change the way software is written; but also has a number of surprising problems to solve. Furthermore, this area is plagued with incomplete specifications and confusing terminoligy.
This talk will attempt to give an overview of NVDIMMs from an operating system perspective: What the terminology means, how they are discovered and partitioned, issues relating to filesystems, a brief description of the functionality available in Linux, and so on. It will then describe the various issues and design choices a Xen system has to make in order to allow Xen systems to use NVDIMMs effectively.
Caches are used in many layers of applications that we develop today, holding data inside or outside of your runtime environment, or even distributed across multiple platforms in data fabrics. However, considerable performance gains can often be realized by configuring the deployment platform/environment and coding your application to take advantage of the properties of CPU caches.
In this talk, we will explore what CPU caches are, how they work and how to measure your JVM-based application data usage to utilize them for maximum efficiency. We will discuss the future of CPU caches in a many-core world, as well as advancements that will soon arrive such as HP's Memristor.
NVDIMM is a standard for allowing non-volatile memory to be exposed to as normal RAM, which can be directly mapped to guests. This simple concept has the potential to dramatically change the way software is written; but also has a number of surprising problems to solve. Furthermore, this area is plagued with incomplete specifications and confusing terminology.
This talk will attempt to give an overview of NVDIMMs from an operating system perspective: What the terminology means, how they are discovered and partitioned, issues relating to filesystems, a brief description of the functionality available in Linux, and so on. It will then describe the various issues and design choices a Xen system has to make in order to allow Xen systems to use NVDIMMs effectively.
Accelerate your Kubernetes clusters with Varnish CachingThijs Feryn
A presentation about the usage and availability of Varnish on Kubernetes. This talk explores the capabilities of Varnish caching and shows how to use the Varnish Helm chart to deploy it to Kubernetes.
This presentation was delivered at K8SUG Singapore. See https://feryn.eu/presentations/accelerate-your-kubernetes-clusters-with-varnish-caching-k8sug-singapore-28-2024 for more details.
Epistemic Interaction - tuning interfaces to provide information for AI supportAlan Dix
Paper presented at SYNERGY workshop at AVI 2024, Genoa, Italy. 3rd June 2024
https://alandix.com/academic/papers/synergy2024-epistemic/
As machine learning integrates deeper into human-computer interactions, the concept of epistemic interaction emerges, aiming to refine these interactions to enhance system adaptability. This approach encourages minor, intentional adjustments in user behaviour to enrich the data available for system learning. This paper introduces epistemic interaction within the context of human-system communication, illustrating how deliberate interaction design can improve system understanding and adaptation. Through concrete examples, we demonstrate the potential of epistemic interaction to significantly advance human-computer interaction by leveraging intuitive human communication strategies to inform system design and functionality, offering a novel pathway for enriching user-system engagements.
GraphRAG is All You need? LLM & Knowledge GraphGuy Korland
Guy Korland, CEO and Co-founder of FalkorDB, will review two articles on the integration of language models with knowledge graphs.
1. Unifying Large Language Models and Knowledge Graphs: A Roadmap.
https://arxiv.org/abs/2306.08302
2. Microsoft Research's GraphRAG paper and a review paper on various uses of knowledge graphs:
https://www.microsoft.com/en-us/research/blog/graphrag-unlocking-llm-discovery-on-narrative-private-data/
Smart TV Buyer Insights Survey 2024 by 91mobiles.pdf91mobiles
91mobiles recently conducted a Smart TV Buyer Insights Survey in which we asked over 3,000 respondents about the TV they own, aspects they look at on a new TV, and their TV buying preferences.
LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...DanBrown980551
Do you want to learn how to model and simulate an electrical network from scratch in under an hour?
Then welcome to this PowSyBl workshop, hosted by Rte, the French Transmission System Operator (TSO)!
During the webinar, you will discover the PowSyBl ecosystem as well as handle and study an electrical network through an interactive Python notebook.
PowSyBl is an open source project hosted by LF Energy, which offers a comprehensive set of features for electrical grid modelling and simulation. Among other advanced features, PowSyBl provides:
- A fully editable and extendable library for grid component modelling;
- Visualization tools to display your network;
- Grid simulation tools, such as power flows, security analyses (with or without remedial actions) and sensitivity analyses;
The framework is mostly written in Java, with a Python binding so that Python developers can access PowSyBl functionalities as well.
What you will learn during the webinar:
- For beginners: discover PowSyBl's functionalities through a quick general presentation and the notebook, without needing any expert coding skills;
- For advanced developers: master the skills to efficiently apply PowSyBl functionalities to your real-world scenarios.
Search and Society: Reimagining Information Access for Radical FuturesBhaskar Mitra
The field of Information retrieval (IR) is currently undergoing a transformative shift, at least partly due to the emerging applications of generative AI to information access. In this talk, we will deliberate on the sociotechnical implications of generative AI for information access. We will argue that there is both a critical necessity and an exciting opportunity for the IR community to re-center our research agendas on societal needs while dismantling the artificial separation between the work on fairness, accountability, transparency, and ethics in IR and the rest of IR research. Instead of adopting a reactionary strategy of trying to mitigate potential social harms from emerging technologies, the community should aim to proactively set the research agenda for the kinds of systems we should build inspired by diverse explicitly stated sociotechnical imaginaries. The sociotechnical imaginaries that underpin the design and development of information access technologies needs to be explicitly articulated, and we need to develop theories of change in context of these diverse perspectives. Our guiding future imaginaries must be informed by other academic fields, such as democratic theory and critical theory, and should be co-developed with social science scholars, legal scholars, civil rights and social justice activists, and artists, among others.
UiPath Test Automation using UiPath Test Suite series, part 3DianaGray10
Welcome to UiPath Test Automation using UiPath Test Suite series part 3. In this session, we will cover desktop automation along with UI automation.
Topics covered:
UI automation Introduction,
UI automation Sample
Desktop automation flow
Pradeep Chinnala, Senior Consultant Automation Developer @WonderBotz and UiPath MVP
Deepak Rai, Automation Practice Lead, Boundaryless Group and UiPath MVP
Transcript: Selling digital books in 2024: Insights from industry leaders - T...BookNet Canada
The publishing industry has been selling digital audiobooks and ebooks for over a decade and has found its groove. What’s changed? What has stayed the same? Where do we go from here? Join a group of leading sales peers from across the industry for a conversation about the lessons learned since the popularization of digital books, best practices, digital book supply chain management, and more.
Link to video recording: https://bnctechforum.ca/sessions/selling-digital-books-in-2024-insights-from-industry-leaders/
Presented by BookNet Canada on May 28, 2024, with support from the Department of Canadian Heritage.
DevOps and Testing slides at DASA ConnectKari Kakkonen
My and Rik Marselis slides at 30.5.2024 DASA Connect conference. We discuss about what is testing, then what is agile testing and finally what is Testing in DevOps. Finally we had lovely workshop with the participants trying to find out different ways to think about quality and testing in different parts of the DevOps infinity loop.
"Impact of front-end architecture on development cost", Viktor TurskyiFwdays
I have heard many times that architecture is not important for the front-end. Also, many times I have seen how developers implement features on the front-end just following the standard rules for a framework and think that this is enough to successfully launch the project, and then the project fails. How to prevent this and what approach to choose? I have launched dozens of complex projects and during the talk we will analyze which approaches have worked for me and which have not.
Dev Dives: Train smarter, not harder – active learning and UiPath LLMs for do...UiPathCommunity
💥 Speed, accuracy, and scaling – discover the superpowers of GenAI in action with UiPath Document Understanding and Communications Mining™:
See how to accelerate model training and optimize model performance with active learning
Learn about the latest enhancements to out-of-the-box document processing – with little to no training required
Get an exclusive demo of the new family of UiPath LLMs – GenAI models specialized for processing different types of documents and messages
This is a hands-on session specifically designed for automation developers and AI enthusiasts seeking to enhance their knowledge in leveraging the latest intelligent document processing capabilities offered by UiPath.
Speakers:
👨🏫 Andras Palfi, Senior Product Manager, UiPath
👩🏫 Lenka Dulovicova, Product Program Manager, UiPath
Software Delivery At the Speed of AI: Inflectra Invests In AI-Powered QualityInflectra
In this insightful webinar, Inflectra explores how artificial intelligence (AI) is transforming software development and testing. Discover how AI-powered tools are revolutionizing every stage of the software development lifecycle (SDLC), from design and prototyping to testing, deployment, and monitoring.
Learn about:
• The Future of Testing: How AI is shifting testing towards verification, analysis, and higher-level skills, while reducing repetitive tasks.
• Test Automation: How AI-powered test case generation, optimization, and self-healing tests are making testing more efficient and effective.
• Visual Testing: Explore the emerging capabilities of AI in visual testing and how it's set to revolutionize UI verification.
• Inflectra's AI Solutions: See demonstrations of Inflectra's cutting-edge AI tools like the ChatGPT plugin and Azure Open AI platform, designed to streamline your testing process.
Whether you're a developer, tester, or QA professional, this webinar will give you valuable insights into how AI is shaping the future of software delivery.
Key Trends Shaping the Future of Infrastructure.pdfCheryl Hung
Keynote at DIGIT West Expo, Glasgow on 29 May 2024.
Cheryl Hung, ochery.com
Sr Director, Infrastructure Ecosystem, Arm.
The key trends across hardware, cloud and open-source; exploring how these areas are likely to mature and develop over the short and long-term, and then considering how organisations can position themselves to adapt and thrive.
Linux internals for Database administrators at Linux Piter 2016
1. Linux IO internals
for database administrators
Ilya Kosmodemiansky
ik@postgresql-consulting.com
2. Why this talk
• Linux is a most common OS for databases
• DBAs often run into IO problems
• Most of the information on topic is written by kernel
developers (for kernel developers) or is checklist-style
• Checklists are useful, but up to certain workload
3. The main IO problem for databases
• How to maximize page throughput between memory and
disks
• Things involved:
Disks
Memory
CPU
IO Schedulers
Filesystems
Database itself
• IO problems for databases are not always only about disks
5. Key things about such workload
• Shared memory segment can be very large
• Keeping in-memory pages synchronized with disk generates
huge IO
• WAL should be written fast and safe
• One and every layer of OS IO stack involved
6. Memory allocation and mapping
CPU L1
MMU TLB
Memory
L2 L3
page
table
Virtual addressing
Translation
Physical addressing
7. DBAs takeaways:
• Database with huge shared memory segment benefits from
huge pages
• But not from transparent huge pages
Databases operate large continuous shared memory segments
THP defragmentation can lead to severe performance
degradation in such cases
8. Freeing memory
Page cache
Swap
Free list Free list Free list
alloc()free()
page-out
vm.min_free_kbytes
reclaim
vm.swapiness
OOM-killer
vm.panic_on_oom
9. Page-out
• Page-out happens if:
someone calls fsync
30 sec timeout exceeded (vm.dirty_expire_centisecs)
Too many dirty pages (vm.dirty_background_ratio and
vm.dirty_ratio)
• It is reasonable to tune vm.dirty_* on rotating disks with
RAID controller, but helps a little on server class SSDs
10. DBAs takeaways:
• vm.overcommit_memory
0 - heuristic overcommit, reduces swap usage
1 - always overcommit
2 - do not overcommit (vm.overcommit_ratio = 50 by
default)
• vm.min_free_kbytes - reasonably high (can be easy 1000000
on a server with enough memory)
• vm.swappiness = 1
0 - swap disabled
60 - default
100 - swap preffered instead of other reaping mechanisms
• Your database would not like OOM-killer
11. OOM-killer: plague and cholera
• vm.panic_on_oom effectively disables OOM-killer, but that is
probably not the result you desire
• Or for a certain process: echo -17 > /proc/12465/oom_adj
but again
15. Elevators
• noop or none - then disks throughput is so high, that it can
not benefit from keen scheduling
PCIe SSDs
SAN disk arrays
• deadline - rotating disks
• CFQ - universal, default one
16. Thanks
to my collegues Alexey Lesovsky and Max Boguk for a lot of
research on this topic