The document discusses problems managing large Postgres installations containing terabytes of data. Key issues include: system resource exhaustion from running out of inodes or file descriptors; slow performance from huge catalogs, statistics collection, and backups that take many hours; and handling VACUUM problems like bloat and transaction wraparound. Solutions proposed include using connection poolers, moving statistics files to RAM, upgrading Postgres versions, and avoiding clusters with over 400,000 tables.
NVIDIA engineers will talk about the latest OpenGL API features for programmable GPUs. This session will cover the “hows, whys, and whens” of programming the fourth generation shader hardware. It will match detailed information on the API with practical examples. Topics covered will include geometry shaders, transformation feedback, instancing, and advanced texture formats.
NVIDIA engineers will talk about the latest OpenGL API features for programmable GPUs. This session will cover the “hows, whys, and whens” of programming the fourth generation shader hardware. It will match detailed information on the API with practical examples. Topics covered will include geometry shaders, transformation feedback, instancing, and advanced texture formats.
Ever hire someone and have it “just not work out”? How long did that take to figure out? Weeks? Days? Before your morning coffee?
Hiring, firing and retention are some of the most difficult tasks that a business faces. And we all make mistakes. The tech industry is currently in the middle of a competitive hiring bubble and it’s really hard to find good people. It’s even harder to retain them. So how do you find good people, and keep them?
We’ll show what mistakes we’ve made in our combined 30 years in open source and tech.
Postgres has long been a database that easily scales vertically. It has also supported durable and reliable binary replication.
In the last two years, work on HA, clustering and extensions to our MVCC system has produced clustering features that help Postgres efficiently scale across many smaller systems.
This talk will start with a demo of the latest Postgres replication features. Then, we'll walk through open source and commercially-supported stacks that supporting the most demanding applications backed by Postgres.
In version 9.1, Postgres added streaming and synchronous replication that can be configured in about 5 steps. In 9.2 (to be released in 2012), we're adding cascaded replication and optimizations for common developer requests.
These features set the stage for a fast, efficient and distributed SQL cluster -- one that has configurable durability, is easy to administer and supported by "cloud" and virtualization vendors.
Ever hire someone and have it “just not work out”? How long did that take to figure out? Weeks? Days? Before your morning coffee?
Hiring, firing and retention are some of the most difficult tasks that a business faces. And we all make mistakes. The tech industry is currently in the middle of a competitive hiring bubble and it’s really hard to find good people. It’s even harder to retain them. So how do you find good people, and keep them?
We’ll show what mistakes we’ve made in our combined 30 years in open source and tech.
Postgres has long been a database that easily scales vertically. It has also supported durable and reliable binary replication.
In the last two years, work on HA, clustering and extensions to our MVCC system has produced clustering features that help Postgres efficiently scale across many smaller systems.
This talk will start with a demo of the latest Postgres replication features. Then, we'll walk through open source and commercially-supported stacks that supporting the most demanding applications backed by Postgres.
In version 9.1, Postgres added streaming and synchronous replication that can be configured in about 5 steps. In 9.2 (to be released in 2012), we're adding cascaded replication and optimizations for common developer requests.
These features set the stage for a fast, efficient and distributed SQL cluster -- one that has configurable durability, is easy to administer and supported by "cloud" and virtualization vendors.
Have a project that just needs some cash to get off the ground? Need someone to fund beer and food for an event? Have a great idea and want to get paid for implementing it? Come find out how we did it.
Asking for money doesn’t have to be so hard. Whether you’re working on a non-profit, or a small side project. Or you’re bootstrapping a sweet new developer event. Or you want to find angel investing or venture capital… The same skills apply.
Come learn all about how to ask for money from Selena Deckelmann, J Chris Anderson and Teyo Tyree. Selena co-founded Open Source Bridge, raises money for the open source project PostgreSQL and has found funding for many small non- and for-profit projects. J Chris and Teyo both founded VC-backed startups – Couchbase and Puppet Labs – based on open source products.
They’ll lead you through their process, their successes and failures.
You’ll leave with proven strategies for developing relationships, asking the right questions and providing the right information to people who want to give you their money.
Size can creep up on you. Some day you may wake up to a multi-terabyte Postgres system handling over 3000 tps staring you down. Learn the best ways to manage these systems as they grow, and find out what new features in 9.0 have made life easier for administrators and application developers working with big data.
This talk will lead you through solutions to problems Postgres faces when it gets big: backups, transaction wraparound, bloat, huge catalogs and upgrades. You need to monitor the right things, find the gems in DBA-friendly database functions and catalog tables, and know the right places to look to spot problems early. We’ll also go over monitoring best practices and open source tools to get the job done.
Working with multiple versions of Postgres back to version 8.2 will be included, and as well as tips on making the most out of new features in 9.0. War stories will be taken from real-world work with Emma, an email marketing company with a few large databases.
Open Source software development starts with the code. But if you want a sustainable project, you need people too. This is a meditation on the idea of making friendship the core of your community's values.
Bucardo is a replication system for PostgreSQL. It supports both master-slave (to multiple slaves) and master-master replication, and does not require any modifications to PostgreSQL to run. Maintained by Greg Sabino Mullane, its management application (bucardo_ctl) has recently been significantly improved, making initial setup of replication very fast and easy.
Open Source Bridge 2009 Conference opening day slides. Includes some statistics about attendees, why they're here, goals for the conference, and a pointer about session/wiki tie-in.
Alt. GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using ...James Anderson
Effective Application Security in Software Delivery lifecycle using Deployment Firewall and DBOM
The modern software delivery process (or the CI/CD process) includes many tools, distributed teams, open-source code, and cloud platforms. Constant focus on speed to release software to market, along with the traditional slow and manual security checks has caused gaps in continuous security as an important piece in the software supply chain. Today organizations feel more susceptible to external and internal cyber threats due to the vast attack surface in their applications supply chain and the lack of end-to-end governance and risk management.
The software team must secure its software delivery process to avoid vulnerability and security breaches. This needs to be achieved with existing tool chains and without extensive rework of the delivery processes. This talk will present strategies and techniques for providing visibility into the true risk of the existing vulnerabilities, preventing the introduction of security issues in the software, resolving vulnerabilities in production environments quickly, and capturing the deployment bill of materials (DBOM).
Speakers:
Bob Boule
Robert Boule is a technology enthusiast with PASSION for technology and making things work along with a knack for helping others understand how things work. He comes with around 20 years of solution engineering experience in application security, software continuous delivery, and SaaS platforms. He is known for his dynamic presentations in CI/CD and application security integrated in software delivery lifecycle.
Gopinath Rebala
Gopinath Rebala is the CTO of OpsMx, where he has overall responsibility for the machine learning and data processing architectures for Secure Software Delivery. Gopi also has a strong connection with our customers, leading design and architecture for strategic implementations. Gopi is a frequent speaker and well-known leader in continuous delivery and integrating security into software delivery.
The Metaverse and AI: how can decision-makers harness the Metaverse for their...Jen Stirrup
The Metaverse is popularized in science fiction, and now it is becoming closer to being a part of our daily lives through the use of social media and shopping companies. How can businesses survive in a world where Artificial Intelligence is becoming the present as well as the future of technology, and how does the Metaverse fit into business strategy when futurist ideas are developing into reality at accelerated rates? How do we do this when our data isn't up to scratch? How can we move towards success with our data so we are set up for the Metaverse when it arrives?
How can you help your company evolve, adapt, and succeed using Artificial Intelligence and the Metaverse to stay ahead of the competition? What are the potential issues, complications, and benefits that these technologies could bring to us and our organizations? In this session, Jen Stirrup will explain how to start thinking about these technologies as an organisation.
PHP Frameworks: I want to break free (IPC Berlin 2024)Ralf Eggert
In this presentation, we examine the challenges and limitations of relying too heavily on PHP frameworks in web development. We discuss the history of PHP and its frameworks to understand how this dependence has evolved. The focus will be on providing concrete tips and strategies to reduce reliance on these frameworks, based on real-world examples and practical considerations. The goal is to equip developers with the skills and knowledge to create more flexible and future-proof web applications. We'll explore the importance of maintaining autonomy in a rapidly changing tech landscape and how to make informed decisions in PHP development.
This talk is aimed at encouraging a more independent approach to using PHP frameworks, moving towards a more flexible and future-proof approach to PHP development.
Elevating Tactical DDD Patterns Through Object CalisthenicsDorra BARTAGUIZ
After immersing yourself in the blue book and its red counterpart, attending DDD-focused conferences, and applying tactical patterns, you're left with a crucial question: How do I ensure my design is effective? Tactical patterns within Domain-Driven Design (DDD) serve as guiding principles for creating clear and manageable domain models. However, achieving success with these patterns requires additional guidance. Interestingly, we've observed that a set of constraints initially designed for training purposes remarkably aligns with effective pattern implementation, offering a more ‘mechanical’ approach. Let's explore together how Object Calisthenics can elevate the design of your tactical DDD patterns, offering concrete help for those venturing into DDD for the first time!
State of ICS and IoT Cyber Threat Landscape Report 2024 previewPrayukth K V
The IoT and OT threat landscape report has been prepared by the Threat Research Team at Sectrio using data from Sectrio, cyber threat intelligence farming facilities spread across over 85 cities around the world. In addition, Sectrio also runs AI-based advanced threat and payload engagement facilities that serve as sinks to attract and engage sophisticated threat actors, and newer malware including new variants and latent threats that are at an earlier stage of development.
The latest edition of the OT/ICS and IoT security Threat Landscape Report 2024 also covers:
State of global ICS asset and network exposure
Sectoral targets and attacks as well as the cost of ransom
Global APT activity, AI usage, actor and tactic profiles, and implications
Rise in volumes of AI-powered cyberattacks
Major cyber events in 2024
Malware and malicious payload trends
Cyberattack types and targets
Vulnerability exploit attempts on CVEs
Attacks on counties – USA
Expansion of bot farms – how, where, and why
In-depth analysis of the cyber threat landscape across North America, South America, Europe, APAC, and the Middle East
Why are attacks on smart factories rising?
Cyber risk predictions
Axis of attacks – Europe
Systemic attacks in the Middle East
Download the full report from here:
https://sectrio.com/resources/ot-threat-landscape-reports/sectrio-releases-ot-ics-and-iot-security-threat-landscape-report-2024/
Why You Should Replace Windows 11 with Nitrux Linux 3.5.0 for enhanced perfor...SOFTTECHHUB
The choice of an operating system plays a pivotal role in shaping our computing experience. For decades, Microsoft's Windows has dominated the market, offering a familiar and widely adopted platform for personal and professional use. However, as technological advancements continue to push the boundaries of innovation, alternative operating systems have emerged, challenging the status quo and offering users a fresh perspective on computing.
One such alternative that has garnered significant attention and acclaim is Nitrux Linux 3.5.0, a sleek, powerful, and user-friendly Linux distribution that promises to redefine the way we interact with our devices. With its focus on performance, security, and customization, Nitrux Linux presents a compelling case for those seeking to break free from the constraints of proprietary software and embrace the freedom and flexibility of open-source computing.
SAP Sapphire 2024 - ASUG301 building better apps with SAP Fiori.pdfPeter Spielvogel
Building better applications for business users with SAP Fiori.
• What is SAP Fiori and why it matters to you
• How a better user experience drives measurable business benefits
• How to get started with SAP Fiori today
• How SAP Fiori elements accelerates application development
• How SAP Build Code includes SAP Fiori tools and other generative artificial intelligence capabilities
• How SAP Fiori paves the way for using AI in SAP apps
UiPath Test Automation using UiPath Test Suite series, part 4DianaGray10
Welcome to UiPath Test Automation using UiPath Test Suite series part 4. In this session, we will cover Test Manager overview along with SAP heatmap.
The UiPath Test Manager overview with SAP heatmap webinar offers a concise yet comprehensive exploration of the role of a Test Manager within SAP environments, coupled with the utilization of heatmaps for effective testing strategies.
Participants will gain insights into the responsibilities, challenges, and best practices associated with test management in SAP projects. Additionally, the webinar delves into the significance of heatmaps as a visual aid for identifying testing priorities, areas of risk, and resource allocation within SAP landscapes. Through this session, attendees can expect to enhance their understanding of test management principles while learning practical approaches to optimize testing processes in SAP environments using heatmap visualization techniques
What will you get from this session?
1. Insights into SAP testing best practices
2. Heatmap utilization for testing
3. Optimization of testing processes
4. Demo
Topics covered:
Execution from the test manager
Orchestrator execution result
Defect reporting
SAP heatmap example with demo
Speaker:
Deepak Rai, Automation Practice Lead, Boundaryless Group and UiPath MVP
Epistemic Interaction - tuning interfaces to provide information for AI supportAlan Dix
Paper presented at SYNERGY workshop at AVI 2024, Genoa, Italy. 3rd June 2024
https://alandix.com/academic/papers/synergy2024-epistemic/
As machine learning integrates deeper into human-computer interactions, the concept of epistemic interaction emerges, aiming to refine these interactions to enhance system adaptability. This approach encourages minor, intentional adjustments in user behaviour to enrich the data available for system learning. This paper introduces epistemic interaction within the context of human-system communication, illustrating how deliberate interaction design can improve system understanding and adaptation. Through concrete examples, we demonstrate the potential of epistemic interaction to significantly advance human-computer interaction by leveraging intuitive human communication strategies to inform system design and functionality, offering a novel pathway for enriching user-system engagements.
DevOps and Testing slides at DASA ConnectKari Kakkonen
My and Rik Marselis slides at 30.5.2024 DASA Connect conference. We discuss about what is testing, then what is agile testing and finally what is Testing in DevOps. Finally we had lovely workshop with the participants trying to find out different ways to think about quality and testing in different parts of the DevOps infinity loop.
zkStudyClub - Reef: Fast Succinct Non-Interactive Zero-Knowledge Regex ProofsAlex Pruden
This paper presents Reef, a system for generating publicly verifiable succinct non-interactive zero-knowledge proofs that a committed document matches or does not match a regular expression. We describe applications such as proving the strength of passwords, the provenance of email despite redactions, the validity of oblivious DNS queries, and the existence of mutations in DNA. Reef supports the Perl Compatible Regular Expression syntax, including wildcards, alternation, ranges, capture groups, Kleene star, negations, and lookarounds. Reef introduces a new type of automata, Skipping Alternating Finite Automata (SAFA), that skips irrelevant parts of a document when producing proofs without undermining soundness, and instantiates SAFA with a lookup argument. Our experimental evaluation confirms that Reef can generate proofs for documents with 32M characters; the proofs are small and cheap to verify (under a second).
Paper: https://eprint.iacr.org/2023/1886
Securing your Kubernetes cluster_ a step-by-step guide to success !KatiaHIMEUR1
Today, after several years of existence, an extremely active community and an ultra-dynamic ecosystem, Kubernetes has established itself as the de facto standard in container orchestration. Thanks to a wide range of managed services, it has never been so easy to set up a ready-to-use Kubernetes cluster.
However, this ease of use means that the subject of security in Kubernetes is often left for later, or even neglected. This exposes companies to significant risks.
In this talk, I'll show you step-by-step how to secure your Kubernetes cluster for greater peace of mind and reliability.
3. The Environment
• 1.6 TB, 1 cluster,Version 8.2
• 1.1 TB, 1 cluster,Version 8.3
• 8.4/9.0 Dev systems
• Working toward 9.0 into prod (May 2011)
• pgpool, Redis, RabbitMQ, NFS
SP
ogm
Ceon
Cfo
.En
Uef2
re
3
0n
1c1e
4. Some stats
• daily peak: ~3000 commits per second
• average writes: 4 MBps
• average reads: 8 MBps
SP
ogm
Ceon
Cfo
.En
Uef2
re
4
0n
1c1e
5. What’s good
• Most queries are fast!
• Benchmarks say we’re pushing the limits of
the hardware
• Developers love working with Postgres
SP
ogm
Ceon
Cfo
.En
Uef2
re0n
1c1e
10. Running out of inodes
Problem: UFS on Solaris
“The only way to add more inodes to a UFS
filesystem is: 1. destroy the filesystem and create a
new filesystem with a higher inode density 2. enlarge
the filesystem - growfs man page”
SP
ogm
Ceon
Cfo
.En
Uef2
re
10
0n
1c1e
11. Running out of inodes
Solution 0: Delete files.
Solution 1: Sharding/bigger filesystem
Solution 2: xfs
SP
ogm
Ceon
Cfo
.En
Uef2
re
11
0n
1c1e
12. Running out of
file descriptors
Problem: Too many open files
by the database.
selena@lulu:~ #508 18:43 :)
sudo lsof -p 19121 | wc
40 355 4151
Solution: You need a connection
pooler.
SP
ogm
Ceon
Cfo
.En
Uef2
re
12
0n
1c1e
13. Running out of
file descriptors
Solution: You need a connection
pooler.
Recommended:
pgbouncer (threaded, online upgrade)
pgpool-II (failover)
SP
ogm
Ceon
Cfo
.En
Uef2
re
13
0n
1c1e
16. Maintenance problem
Minor mistake in parent table definitions:
not null default
nextval('important_sequence'::text)
vs
not null default
nextval('important_sequence'::regclass)
SP
ogm
Ceon
Cfo
.En
Uef2
re
16
0n
1c1e
17. Huge Catalogs
Problem: Slow scans of catalog data
Solution:
Upgrade to Postgres 8.4 or higher
But really: Avoid making a cluster with >400k
tables.
SP
ogm
Ceon
Cfo
.En
Uef2
re
17
0n
1c1e
18. Stats collection
9,019,868 total data points for table stats
4,550,770 total data points for index stats
Problem: This is slow to write.
(128 MB written every second or so)
SP
ogm
Ceon
Cfo
.En
Uef2
re
18
0n
1c1e
19. Stats collection
9,019,868 total data points for table stats
4,550,770 total data points for index stats
Soution: Move stats file to RAM.
stats_temp_directory (8.4 or higher)
There’s a trivial patch for earlier versions.
SP
ogm
Ceon
Cfo
.En
Uef2
re
19
0n
1c1e
20. Stats collection
9,019,868 total data points for table stats
4,550,770 total data points for index stats
Problem: This is slow to read.
SP
ogm
Ceon
Cfo
.En
Uef2
re
20
0n
1c1e
21. Stats collection
9,019,868 total data points for table stats
4,550,770 total data points for index stats
Solution:
Supposedly, this is better in 8.4 and higher.
(fewer writes per minute)
Still probably not fast.
SP
ogm
Ceon
Cfo
.En
Uef2
re
21
0n
1c1e
22. Backups
pg_dump takes longer and longer...
SP
ogm
Ceon
Cfo
.En
Uef2
re
22
0n
1c1e
24. Backups
Problem: pg_dump is too slow.
Solutions:
• patching pg_dump for SELECT ... LIMIT
• crank down shared_buffers
• Stop using pg_dump for backups
• 64-bit might help
SP
ogm
Ceon
Cfo
.En
Uef2
re
24
0n
1c1e
25. How not to migrate
to a 64-bit system
SP
ogm
Ceon
Cfo
.En
Uef2
re
25
0n
1c1e
26. Title Text
Install 32-bit Postgres and libraries on a 64-bit system.
Install 64-bit Postgres/libs of the same version.
Copy “hot backup” from 32-bit sys over to 64-bit sys.
Run pg_dump from 64-bit version on 32-bit Postgres.
SP
ogm
Ceon
Cfo
.En
Uef2
re
26
0n
1c1e
27. A single warm standby
is not a backup.
But lots of people use them that way!
SP
ogm
Ceon
Cfo
.En
Uef2
re
27
0n
1c1e
28. Ship WAL from Solaris x86 -> Linux
It did work!
SP
ogm
Ceon
Cfo
.En
Uef2
re
28
0n
1c1e
30. Bloat
Problem: Lots of dead tuples in tables.
• Frequent UPDATEs to long tables of log
data
• Frequent DELETEs without a VACUUM
• A terabyte of dead tuples
SP
ogm
Ceon
Cfo
.En
Uef2
re
30
0n
1c1e
31. Fixing bloat
Solution: Write custom scripts to clean
• VACUUM for small things
• CLUSTER for everything else
• Considered TRUNCATE
SP
ogm
Ceon
Cfo
.En
Uef2
re
31
0n
1c1e
32. Catalog Bloat
Application allowed users to initiate ALTER
TABLE.
Regular VACUUM couldn’t fix it.
VACUUM FULL of the catalog takes 2+ hours.
Use of NOTIFY/LISTEN can also cause bloat.
SP
ogm
Ceon
Cfo
.En
Uef2
re
32
0n
1c1e
33. Transaction
wraparound avoidance
Problem: autovacuum set off too
frequently
Watch age(datfrozenxid)
Solution:
Increase autovacuum_freeze_max_age
(default is 200 million, we increase to one
billion)
SP
ogm
Ceon
Cfo
.En
Uef2
re
33
0n
1c1e
35. Minor upgrades
Problem: Restarting Postgres causes bad
application performance.
• Require a start/stop of database
• Unexpected CHECKPOINT
• Cold cache
SP
ogm
Ceon
Cfo
.En
Uef2
re
35
0n
1c1e
36. Minor upgrades
Solutions:
• Plan for a CHECKPOINT before
shutdown
• Warm the cache (Queries that
exercise indexes, maybe table scans)
SP
ogm
Ceon
Cfo
.En
Uef2
re
36
0n
1c1e
37. Major Version upgrades
Problem: Major upgrades are a PITA.
• <8.2 - no pg_upgrade :(
• Time your restores.
• Document your SLAs.
SP
ogm
Ceon
Cfo
.En
Uef2
re
37
0n
1c1e
38. Major Version upgrades
Solutions: :(
• >=8.3 - pg_upgrade
• Time your restores.
• Document your SLAs.
SP
ogm
Ceon
Cfo
.En
Uef2
re
38
0n
1c1e
39. Major Version upgrades
Solutions: :(
• Write tools to migrate data
• Shard
• Trigger-based replication
SP
ogm
Ceon
Cfo
.En
Uef2
re
39
0n
1c1e
40. The Problems
1. System resource exhaustion
2. Everything is slow: Huge catalogs, Backups
3. Handling VACUUM problems: Bloat,
Transaction wraparound
4. Upgrades: Minor, Major
SP
ogm
Ceon
Cfo
.En
Uef2
re0n
1c1e
41. The Solutions
1. System resource exhaustion
Choose a better filesystem, Pooling
2. Everything is slow: Huge catalogs, Backups
Don’t do that, Monitor & Binary
backups
SP
ogm
Ceon
Cfo
.En
Uef2
re0n
1c1e