This presentation is primarily focused on how to use collectd (http://collectd.org/) to gather data from the Postgres statistics tables. Examples of how to use collectd with Postgres will be shown. There is some hackery involved to make collectd do a little more and collect more meaningful data from Postgres. These small patches will be explored. A small portion of the discussion will be about how to visualize the data.
PostgreSQL is designed to be easily extensible. For this reason, extensions loaded into the database can function just like features that are built in. In this session, we will learn more about PostgreSQL extension framework, how are they built, look at some popular extensions, management of these extensions in your deployments.
In-memory OLTP storage with persistence and transaction supportAlexander Korotkov
Nowadays it becomes evident that single storage engine can't be "one size fits all". PostgreSQL community starts its movement towards pluggable storages. Significant restriction which is imposed in the current approach is compatibility. We consider pluggable storages to be compatible with (at least some) existing index access methods. That means we've long way to go, because we have to extend our index AMs before we can add corresponding features in the pluggable storages themselves.
In this talk we would like look this problem from another angle, and see what can we achieve if we try to make storage completely from scratch (using FDW interface for prototyping). Thus, we would show you a prototype of in-memory OLTP storage with transaction support and snapshot isolation. Internally it's implemented as index-organized table (B-tree) with undo log and optional persistence. That means it's quite different from what we have in PostgreSQL now.
The proved by benchmarks advantages of this in-memory storage are: better multicore scalability (thanks to no buffer manager), reduced bloat (thanks to undo log) and optimized IO (thank to logical WAL logging).
PostgreSQL is designed to be easily extensible. For this reason, extensions loaded into the database can function just like features that are built in. In this session, we will learn more about PostgreSQL extension framework, how are they built, look at some popular extensions, management of these extensions in your deployments.
In-memory OLTP storage with persistence and transaction supportAlexander Korotkov
Nowadays it becomes evident that single storage engine can't be "one size fits all". PostgreSQL community starts its movement towards pluggable storages. Significant restriction which is imposed in the current approach is compatibility. We consider pluggable storages to be compatible with (at least some) existing index access methods. That means we've long way to go, because we have to extend our index AMs before we can add corresponding features in the pluggable storages themselves.
In this talk we would like look this problem from another angle, and see what can we achieve if we try to make storage completely from scratch (using FDW interface for prototyping). Thus, we would show you a prototype of in-memory OLTP storage with transaction support and snapshot isolation. Internally it's implemented as index-organized table (B-tree) with undo log and optional persistence. That means it's quite different from what we have in PostgreSQL now.
The proved by benchmarks advantages of this in-memory storage are: better multicore scalability (thanks to no buffer manager), reduced bloat (thanks to undo log) and optimized IO (thank to logical WAL logging).
"Performance Analysis Methodologies", USENIX/LISA12, San Diego, 2012
Performance analysis methodologies provide guidance, save time, and can find issues that are otherwise overlooked. Example issues include hardware bus saturation, lock contention, recoverable device errors, kernel scheduling issues, and unnecessary workloads. The talk will focus on the USE Method: a simple strategy for all staff for performing a complete check of system performance health, identifying common bottlenecks and errors. Other analysis methods discussed include workload characterization, drill-down analysis, and latency analysis, with example applications from enterprise and cloud computing. Don’t just reach for tools—use a method!
2016년 4월 27일, NDC2016 발표자료입니다.
전통적인 콘텐츠 생산은 기획자, 개발자가 하나하나 컨트롤해야 하는 노동집약적 산업입니다. 저 또한 처음에 입사했을 때 출시를 위해 며칠 만에 이백여 개가 넘는 맵을 찍어야 했던 아픔이 있습니다. 그 뒤에도 플레이어들을 위한 콘텐츠 생산은 계속되었지만, 플레이어들의 콘텐츠 소비 속도는 생산 속도를 뛰어넘은 지 오래되었습니다. 대안은 로그라이크 같은 장르에서 널리 쓰이는 Procedural Contents Generation이라고 생각합니다. 외국 인디 씬에서는 이미 많이 사용되고 있는 이 방법에 최근 점점 더 많이 사용되고 있는 신경망을 활용해서 색다른 콘텐츠 생산 기법을 소개하려 합니다.
Gerenciamento de Backups PostgreSQL com pgbarmanJuliano Atanazio
Tutorial desde a instalação a aplicações práticas do pgbarman.
Apresentação feita originalmente como palestra no evento nacional de PostgreSQL (PgBr 2013.
PgBouncer: Pool, Segurança e Disaster Recovery | Felipe PereiraPGDay Campinas
Como utilizar o PgBouncer para reforçar a segurança do seu banco de dados e ganhar a capacidade de fazer um Disaster Recovery transparente sem ter que mexer na sua aplicação e no seu banco de dados. Ah, ele também é um pool de conexões.
PostgreSQL is a very popular and feature-rich DBMS. At the same time, PostgreSQL has a set of annoying wicked problems, which haven't been resolved in decades. Miraculously, with just a small patch to PostgreSQL core extending this API, it appears possible to solve wicked PostgreSQL problems in a new engine made within an extension.
This presentation covers all aspects of PostgreSQL administration, including installation, security, file structure, configuration, reporting, backup, daily maintenance, monitoring activity, disk space computations, and disaster recovery. It shows how to control host connectivity, configure the server, find the query being run by each session, and find the disk space used by each database.
This ppt was used by Devrim at pgDay Asia 2017. He talked about some important facts about WAL - Transaction Logs or xlogs in PostgreSQL. Some of these can really come handy on a bad day
인프라 모니터링을 위한 시스템을 구축하고 운영하는 데 있어, 다이내믹한 인프라 변화는 어려움으로 다가오고 있습니다.
본 세션에서는 인프라를 운영하는 팀 혹은 운영자 관점에서 바라본 미래 지향적 인프라 모니터링 시스템의 방향성과 이를 구현하기 위해 필요한 구성들을 공유하고자 합니다.
목차
1. NHN 모니터링의 현재
2. 모니터링의 변화
3. 모니터링 방법론
4. 모니터링 절차
5. NHN 모니터링의 미래
대상
- 인프라를 운영하는 시스템 엔지니어
- 인프라 모니터링 시스템에 관심이 있는 분
Spencer Christensen
There are many aspects to managing an RDBMS. Some of these are handled by an experienced DBA, but there are a good many things that any sys admin should be able to take care of if they know what to look for.
This presentation will cover basics of managing Postgres, including creating database clusters, overview of configuration, and logging. We will also look at tools to help monitor Postgres and keep an eye on what is going on. Some of the tools we will review are:
* pgtop
* pg_top
* pgfouine
* check_postgres.pl.
Check_postgres.pl is a great tool that can plug into your Nagios or Cacti monitoring systems, giving you even better visibility into your databases.
2018 아이펀팩토리 데브데이 서버위더스
(iFunFactory DevDay Server,Withus)
제목 : 코드 수정 없이 리눅스 게임 서버 성능 평가
발표자 : 김진욱 CTO
일정 : 2018년 03월 28일
개요 : 서버에 코드 추가 없이도 고수준에서 서버 성능 분석이 가능하다!
실행 중인 서버를 외부에서 관찰하여 최대한 성능을 분석하고 디버깅 하는 법을 공개 합니다.
아래링크를 통해 아이펀팩토리의 더 많은 정보를 얻으실 수 있습니다.
*아이펀팩토리 홈페이지 : https://ifunfactory.com/
*아이펀팩토리 기술 블로그 : https://blog.ifunfactory.com/
"Performance Analysis Methodologies", USENIX/LISA12, San Diego, 2012
Performance analysis methodologies provide guidance, save time, and can find issues that are otherwise overlooked. Example issues include hardware bus saturation, lock contention, recoverable device errors, kernel scheduling issues, and unnecessary workloads. The talk will focus on the USE Method: a simple strategy for all staff for performing a complete check of system performance health, identifying common bottlenecks and errors. Other analysis methods discussed include workload characterization, drill-down analysis, and latency analysis, with example applications from enterprise and cloud computing. Don’t just reach for tools—use a method!
2016년 4월 27일, NDC2016 발표자료입니다.
전통적인 콘텐츠 생산은 기획자, 개발자가 하나하나 컨트롤해야 하는 노동집약적 산업입니다. 저 또한 처음에 입사했을 때 출시를 위해 며칠 만에 이백여 개가 넘는 맵을 찍어야 했던 아픔이 있습니다. 그 뒤에도 플레이어들을 위한 콘텐츠 생산은 계속되었지만, 플레이어들의 콘텐츠 소비 속도는 생산 속도를 뛰어넘은 지 오래되었습니다. 대안은 로그라이크 같은 장르에서 널리 쓰이는 Procedural Contents Generation이라고 생각합니다. 외국 인디 씬에서는 이미 많이 사용되고 있는 이 방법에 최근 점점 더 많이 사용되고 있는 신경망을 활용해서 색다른 콘텐츠 생산 기법을 소개하려 합니다.
Gerenciamento de Backups PostgreSQL com pgbarmanJuliano Atanazio
Tutorial desde a instalação a aplicações práticas do pgbarman.
Apresentação feita originalmente como palestra no evento nacional de PostgreSQL (PgBr 2013.
PgBouncer: Pool, Segurança e Disaster Recovery | Felipe PereiraPGDay Campinas
Como utilizar o PgBouncer para reforçar a segurança do seu banco de dados e ganhar a capacidade de fazer um Disaster Recovery transparente sem ter que mexer na sua aplicação e no seu banco de dados. Ah, ele também é um pool de conexões.
PostgreSQL is a very popular and feature-rich DBMS. At the same time, PostgreSQL has a set of annoying wicked problems, which haven't been resolved in decades. Miraculously, with just a small patch to PostgreSQL core extending this API, it appears possible to solve wicked PostgreSQL problems in a new engine made within an extension.
This presentation covers all aspects of PostgreSQL administration, including installation, security, file structure, configuration, reporting, backup, daily maintenance, monitoring activity, disk space computations, and disaster recovery. It shows how to control host connectivity, configure the server, find the query being run by each session, and find the disk space used by each database.
This ppt was used by Devrim at pgDay Asia 2017. He talked about some important facts about WAL - Transaction Logs or xlogs in PostgreSQL. Some of these can really come handy on a bad day
인프라 모니터링을 위한 시스템을 구축하고 운영하는 데 있어, 다이내믹한 인프라 변화는 어려움으로 다가오고 있습니다.
본 세션에서는 인프라를 운영하는 팀 혹은 운영자 관점에서 바라본 미래 지향적 인프라 모니터링 시스템의 방향성과 이를 구현하기 위해 필요한 구성들을 공유하고자 합니다.
목차
1. NHN 모니터링의 현재
2. 모니터링의 변화
3. 모니터링 방법론
4. 모니터링 절차
5. NHN 모니터링의 미래
대상
- 인프라를 운영하는 시스템 엔지니어
- 인프라 모니터링 시스템에 관심이 있는 분
Spencer Christensen
There are many aspects to managing an RDBMS. Some of these are handled by an experienced DBA, but there are a good many things that any sys admin should be able to take care of if they know what to look for.
This presentation will cover basics of managing Postgres, including creating database clusters, overview of configuration, and logging. We will also look at tools to help monitor Postgres and keep an eye on what is going on. Some of the tools we will review are:
* pgtop
* pg_top
* pgfouine
* check_postgres.pl.
Check_postgres.pl is a great tool that can plug into your Nagios or Cacti monitoring systems, giving you even better visibility into your databases.
2018 아이펀팩토리 데브데이 서버위더스
(iFunFactory DevDay Server,Withus)
제목 : 코드 수정 없이 리눅스 게임 서버 성능 평가
발표자 : 김진욱 CTO
일정 : 2018년 03월 28일
개요 : 서버에 코드 추가 없이도 고수준에서 서버 성능 분석이 가능하다!
실행 중인 서버를 외부에서 관찰하여 최대한 성능을 분석하고 디버깅 하는 법을 공개 합니다.
아래링크를 통해 아이펀팩토리의 더 많은 정보를 얻으실 수 있습니다.
*아이펀팩토리 홈페이지 : https://ifunfactory.com/
*아이펀팩토리 기술 블로그 : https://blog.ifunfactory.com/
pg_top allows you to monitor PostgreSQL processes to view the currently running SQL statement of a process, the query plan of a currently running SELECT statement, locks held by a process, user table statistics, and user index statistics.
This presentation was provided by Chris Conley, Policy Attorney, ACLU-Northern California, for the NISO Two Part Webinar, Understanding Privacy: What Data Is Being Collected And By Whom?, held on March 16, 2016.
This presentation was provided by Alison Macrina, Founder and Director, Library Freedom Project, for the NISO Two Part Webinar, Understanding Privacy: What Data Is Being Collected And By Whom, held on March 16, 2016
This presentation was provided by Eric Hellman, President, GlueJar for the NISO Two Part Webinar, Understanding Privacy: What Data Is Being Collected and By Whom, held on March 16, 2016.
Beautiful Monitoring With Grafana and InfluxDBleesjensen
Query your data streams with the time series database InfluxDB and then visualize the results with stunning Grafana dashboards. Quick and easy to set up. Fully scalable to millions of metrics per second.
Best Practices for Becoming an Exceptional Postgres DBA EDB
Drawing from our teams who support hundreds of Postgres instances and production database systems for customers worldwide, this presentation provides real-real best practices from the nation's top DBAs. Learn top-notch monitoring and maintenance practices, get resource planning advice that can help prevent, resolve, or eliminate common issues, learning top database tuning tricks for increasing system performance and ultimately, gain greater insight into how to improve your effectiveness as a DBA.
Josh Berkus
You've heard that PostgreSQL is the highest-performance transactional open source database, but you're not seeing it on YOUR server. In fact, your PostgreSQL application is kind of poky. What should you do? While doing advanced performance engineering for really high-end systems takes years to learn, you can learn the basics to solve performance issues for 80% of PostgreSQL installations in less than an hour. In this session, you will learn: -- The parts of database application performance -- The performance setup procedure -- Basic troubleshooting tools -- The 13 postgresql.conf settings you need to know -- Where to look for more information.
Postgres has the unique ability to act as a powerful data aggregator in many data centers. This presentation shows how Postgres's extensibility, access to foreign data sources, and ability handle NoSQL-like and data warehousing workloads gives it unmatched capabilities to function in this role.
Operating PostgreSQL at Scale with KubernetesJonathan Katz
The maturation of containerization platforms has changed how people think about creating development environments and has eliminated many inefficiencies for deploying applications. These concept and technologies have made its way into the PostgreSQL ecosystem as well, and tools such as Docker and Kubernetes have enabled teams to run their own “database-as-a-service” on the infrastructure of their choosing.
All this sounds great, but if you are new to the world of containers, it can be very overwhelming to find a place to start. In this talk, which centers around demos, we will see how you can get PostgreSQL up and running in a containerized environment with some advanced sidecars in only a few steps! We will also see how it extends to a larger production environment with Kubernetes, and what the future holds for PostgreSQL in a containerized world.
We will cover the following:
* Why containers are important and what they mean for PostgreSQL
* Create a development environment with PostgreSQL, pgadmin4, monitoring, and more
* How to use Kubernetes to create your own "database-as-a-service"-like PostgreSQL environment
* Trends in the container world and how it will affect PostgreSQL
At the conclusion of the talk, you will understand the fundamentals of how to use container technologies with PostgreSQL and be on your way to running a containerized PostgreSQL environment at scale!
This talk is divided into multiple parts, starting with a basic introduction into the popular opensource database management software postgresql.
Building upon that operational aspects of a postgresql cluster instance will be discussed with a focus on what can, should and must be monitored and what kind of tools and nagios related plugins are available.
The final part of the talk gives an insight into the complex and distributed nature of the postgresql.org infrastructure and how nagios plays a crucial role as the central monitoring and alerting platform for the sysadmin team.
Beyond Postgres: Interesting Projects, Tools and forksSameer Kumar
This ppt was used at Aug Meetup of Postgres User Group Singapore. We talked about
• Cool tools and extensions like PostGIS
• Great projects like pgpool and pgbouncer
• Interesting forks of PostgreSQL like EnterpriseDB, GreenPlum etc
Interesting take-away from the session was -
• Ease Oracle Migration
• Load Balancing in PostgreSQL
• Spatial data in PostgreSQL
• Connection pooling and resource management
• Your next Data Warehouse project
Meetup page- http://www.meetup.com/PUGS-Postgres-Users-Group-Singapore/
Jethro data meetup index base sql on hadoop - oct-2014Eli Singer
JethroData Index based SQL on Hadoop engine.
Architecture comparison of MPP / Full-Scan sql engines such as Impala and Hive to index-based access such as Jethro.
SQL and NoSQL NYC meetup Oct 20 2014
Boaz Raufman
Attack monitoring using ElasticSearch Logstash and KibanaPrajal Kulkarni
With growing trend of Big data, companies are tend to rely on high cost SIEM solutions. However, with introduction of open source and lightweight cluster management solution like ElasticSearch this has been the highlight of the year. Similarly, the log aggregation has been simplified by logstash and kibana providing a visual look to the complex data structure. This presentation will exactly cater to this need of having a appropriate log analysis+Detecting Intrusion+Visualizing data in a powerful interface.
OSDC 2016 - Unifying Logs and Metrics Data with Elastic Beats by Monica SarbuNETWAYS
The Beats are a friendly army of lightweight agents that installed on your servers capture operational data and ship it to Elasticsearch for analysis. They are open source, written in Golang, and maintained by Elastic, the company behind Elasticsearch, Logstash, and Kibana.
This talk will present the first three Beats: Topbeat for system level metrics, Filebeat for log files and Packetbeat for wire data. It will also demonstrate how to combine them with Logstash and Kibana in one advanced monitoring solution, unifying log management, metrics monitoring and system stats. Finally, you will learn how to create a new Beat from scratch using Golang and the libbeat framework to capture any type of information and ship it to Elasticsearch.
Covered:
1. Databases and Schemas
2. Tablespaces
3. Data Type
4. Exploring Databases
5. Locating the database server's message log
6. Locating the database's system identifier
7. Listing databases on this database server
8. How much disk space does a table use?
9. Which are my biggest tables?
10. How many rows are there in a table?
11. Quickly estimating the number of rows in a table
12. Understanding object dependencies
Large Scale Crawling with Apache Nutch and FriendsJulien Nioche
This session will give an overview of Apache Nutch. I will describe its main components and how it fits with other Apache projects such as Hadoop, SOLR, Tika or HBase. The second part of the presentation will be focused on the latest developments in Nutch, the differences between the 1.x and 2.x branch and what we can expect to see in Nutch in the future. This session will cover many practical aspects and should be a good starting point to crawling on a large scale with Apache Nutch and SOLR.
Large Scale Crawling with Apache Nutch and Friendslucenerevolution
Presented by Julien Nioche, Director, DigitalPebble
This session will give an overview of Apache Nutch. I will describe its main components and how it fits with other Apache projects such as Hadoop, SOLR, Tika or HBase. The second part of the presentation will be focused on the latest developments in Nutch, the differences between the 1.x and 2.x branch and what we can expect to see in Nutch in the future. This session will cover many practical aspects and should be a good starting point to crawling on a large scale with Apache Nutch and SOLR.
PGTop for Android: Things I learned making this appMark Wong
This is about a weekend he spent slapping together an Android app that talks directly to Postgres using the Postgres JDBC interface. He'll focus more on the specifics of the development environment, how to use JDBC to connect to Postgres, and gotchas
encountered along the way and not so much on general Android application programming.
pg_proctab: Accessing System Stats in PostgreSQLMark Wong
pg_proctab is a collection of PostgreSQL stored functions that provide access to the operating system process table using SQL. We'll show you which functions are available and where they collect the data, and give examples of their use to collect processor and I/O statistics on SQL queries.
pg_proctab: Accessing System Stats in PostgreSQLMark Wong
pg_proctab is a collection of PostgreSQL stored functions that provide access to the operating system process table using SQL. We'll show you which functions are available and where they collect the data, and give examples of their use to collect processor and I/O statistics on SQL queries. These stored functions currently only work on Linux-based systems.
PostgreSQL Portland Performance Practice Project - Database Test 2 TuningMark Wong
Sixth presentation in a speaker series sponsored by the Portland State University Computer Science Department. The series covers PostgreSQL performance with an OLTP (on-line transaction processing) workload called Database Test 2 (DBT-2). This presentation goes through some experimentation of setting different PostgreSQL global user configuration (GUC) parameters.
pg_proctab: Accessing System Stats in PostgreSQLMark Wong
pc_proctab is a collection of PostgreSQL stored functions that allow you to access the operating system process table using SQL. See examples on how to use these stored functions to collect processor and I/O statistics on SQL statements run against the database.
Filesystem Performance from a Database PerspectiveMark Wong
How do you choose the right filesystem for your database management system?
Administrators have a variety of filesystems to choose from, as well as volume management and hardware or software RAID. This talk will examine how different the performance of filesystems really are, and how do you go about systematically determining which configuration will be the best for your application and hardware.
This talk will present data generated by a group of volunteers running performance tests for database tuning. We were curious if the file systems would really behave like we expected them to, especially when used in conjunction with RAID or volume management.
There is also more to file systems than how fast we can read to or write to them. Reliability is critical for production environments, and proving that is a key part of evaluating performance.
The talk will review and confirm or deny assumptions that many system administrators and developers make about filesystems and databases.
Data shared will include baseline throughput determined with exhaustive fio tests.
PostgreSQL Portland Performance Practice Project - Database Test 2 Filesystem...Mark Wong
Fifth presentation in a speaker series sponsored by the Portland State University Computer Science Department. The series covers PostgreSQL performance with an OLTP (on-line transaction processing) workload called Database Test 2 (DBT-2). This presentation goes through results of different hardware RAID configurations to show why it is important to test your own hardware: it might be performing in way you don't expect.
PostgreSQL Portland Performance Practice Project - Database Test 2 HowtoMark Wong
Fourth presentation in a speaker series sponsored by the Portland State University Computer Science Department. The series covers PostgreSQL performance with an OLTP (on-line transaction processing) workload called Database Test 2 (DBT-2). This presentation is a set of examples to go along with the live presentation given on March 12, 2009.
PostgreSQL Portland Performance Practice Project - Database Test 2 Workload D...Mark Wong
Third presentation in a speaker series sponsored by the Portland State University Computer Science Department. The series covers PostgreSQL performance with an OLTP (on-line transaction processing) workload called Database Test 2 (DBT-2). This presentation go into detail about what the workload does.
A introduction to using system tools to identify what the system is doing. This will go over using top and iostat to determine what the system is physically doing, and then using tools like ps and querying the PostgreSQL system catalog tables to determine what queries are running, what locks have been acquired, and where the tables and indexes are physically located on the system in order to correlate the physical activity to what the database is doing.
Why You Should Replace Windows 11 with Nitrux Linux 3.5.0 for enhanced perfor...SOFTTECHHUB
The choice of an operating system plays a pivotal role in shaping our computing experience. For decades, Microsoft's Windows has dominated the market, offering a familiar and widely adopted platform for personal and professional use. However, as technological advancements continue to push the boundaries of innovation, alternative operating systems have emerged, challenging the status quo and offering users a fresh perspective on computing.
One such alternative that has garnered significant attention and acclaim is Nitrux Linux 3.5.0, a sleek, powerful, and user-friendly Linux distribution that promises to redefine the way we interact with our devices. With its focus on performance, security, and customization, Nitrux Linux presents a compelling case for those seeking to break free from the constraints of proprietary software and embrace the freedom and flexibility of open-source computing.
GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using Deplo...James Anderson
Effective Application Security in Software Delivery lifecycle using Deployment Firewall and DBOM
The modern software delivery process (or the CI/CD process) includes many tools, distributed teams, open-source code, and cloud platforms. Constant focus on speed to release software to market, along with the traditional slow and manual security checks has caused gaps in continuous security as an important piece in the software supply chain. Today organizations feel more susceptible to external and internal cyber threats due to the vast attack surface in their applications supply chain and the lack of end-to-end governance and risk management.
The software team must secure its software delivery process to avoid vulnerability and security breaches. This needs to be achieved with existing tool chains and without extensive rework of the delivery processes. This talk will present strategies and techniques for providing visibility into the true risk of the existing vulnerabilities, preventing the introduction of security issues in the software, resolving vulnerabilities in production environments quickly, and capturing the deployment bill of materials (DBOM).
Speakers:
Bob Boule
Robert Boule is a technology enthusiast with PASSION for technology and making things work along with a knack for helping others understand how things work. He comes with around 20 years of solution engineering experience in application security, software continuous delivery, and SaaS platforms. He is known for his dynamic presentations in CI/CD and application security integrated in software delivery lifecycle.
Gopinath Rebala
Gopinath Rebala is the CTO of OpsMx, where he has overall responsibility for the machine learning and data processing architectures for Secure Software Delivery. Gopi also has a strong connection with our customers, leading design and architecture for strategic implementations. Gopi is a frequent speaker and well-known leader in continuous delivery and integrating security into software delivery.
The Art of the Pitch: WordPress Relationships and SalesLaura Byrne
Clients don’t know what they don’t know. What web solutions are right for them? How does WordPress come into the picture? How do you make sure you understand scope and timeline? What do you do if sometime changes?
All these questions and more will be explored as we talk about matching clients’ needs with what your agency offers without pulling teeth or pulling your hair out. Practical tips, and strategies for successful relationship building that leads to closing the deal.
Smart TV Buyer Insights Survey 2024 by 91mobiles.pdf91mobiles
91mobiles recently conducted a Smart TV Buyer Insights Survey in which we asked over 3,000 respondents about the TV they own, aspects they look at on a new TV, and their TV buying preferences.
GraphSummit Singapore | The Art of the Possible with Graph - Q2 2024Neo4j
Neha Bajwa, Vice President of Product Marketing, Neo4j
Join us as we explore breakthrough innovations enabled by interconnected data and AI. Discover firsthand how organizations use relationships in data to uncover contextual insights and solve our most pressing challenges – from optimizing supply chains, detecting fraud, and improving customer experiences to accelerating drug discoveries.
Transcript: Selling digital books in 2024: Insights from industry leaders - T...BookNet Canada
The publishing industry has been selling digital audiobooks and ebooks for over a decade and has found its groove. What’s changed? What has stayed the same? Where do we go from here? Join a group of leading sales peers from across the industry for a conversation about the lessons learned since the popularization of digital books, best practices, digital book supply chain management, and more.
Link to video recording: https://bnctechforum.ca/sessions/selling-digital-books-in-2024-insights-from-industry-leaders/
Presented by BookNet Canada on May 28, 2024, with support from the Department of Canadian Heritage.
Observability Concepts EVERY Developer Should Know -- DeveloperWeek Europe.pdfPaige Cruz
Monitoring and observability aren’t traditionally found in software curriculums and many of us cobble this knowledge together from whatever vendor or ecosystem we were first introduced to and whatever is a part of your current company’s observability stack.
While the dev and ops silo continues to crumble….many organizations still relegate monitoring & observability as the purview of ops, infra and SRE teams. This is a mistake - achieving a highly observable system requires collaboration up and down the stack.
I, a former op, would like to extend an invitation to all application developers to join the observability party will share these foundational concepts to build on:
UiPath Test Automation using UiPath Test Suite series, part 5DianaGray10
Welcome to UiPath Test Automation using UiPath Test Suite series part 5. In this session, we will cover CI/CD with devops.
Topics covered:
CI/CD with in UiPath
End-to-end overview of CI/CD pipeline with Azure devops
Speaker:
Lyndsey Byblow, Test Suite Sales Engineer @ UiPath, Inc.
GridMate - End to end testing is a critical piece to ensure quality and avoid...ThomasParaiso2
End to end testing is a critical piece to ensure quality and avoid regressions. In this session, we share our journey building an E2E testing pipeline for GridMate components (LWC and Aura) using Cypress, JSForce, FakerJS…
A tale of scale & speed: How the US Navy is enabling software delivery from l...sonjaschweigert1
Rapid and secure feature delivery is a goal across every application team and every branch of the DoD. The Navy’s DevSecOps platform, Party Barge, has achieved:
- Reduction in onboarding time from 5 weeks to 1 day
- Improved developer experience and productivity through actionable findings and reduction of false positives
- Maintenance of superior security standards and inherent policy enforcement with Authorization to Operate (ATO)
Development teams can ship efficiently and ensure applications are cyber ready for Navy Authorizing Officials (AOs). In this webinar, Sigma Defense and Anchore will give attendees a look behind the scenes and demo secure pipeline automation and security artifacts that speed up application ATO and time to production.
We will cover:
- How to remove silos in DevSecOps
- How to build efficient development pipeline roles and component templates
- How to deliver security artifacts that matter for ATO’s (SBOMs, vulnerability reports, and policy evidence)
- How to streamline operations with automated policy checks on container images
State of ICS and IoT Cyber Threat Landscape Report 2024 previewPrayukth K V
The IoT and OT threat landscape report has been prepared by the Threat Research Team at Sectrio using data from Sectrio, cyber threat intelligence farming facilities spread across over 85 cities around the world. In addition, Sectrio also runs AI-based advanced threat and payload engagement facilities that serve as sinks to attract and engage sophisticated threat actors, and newer malware including new variants and latent threats that are at an earlier stage of development.
The latest edition of the OT/ICS and IoT security Threat Landscape Report 2024 also covers:
State of global ICS asset and network exposure
Sectoral targets and attacks as well as the cost of ransom
Global APT activity, AI usage, actor and tactic profiles, and implications
Rise in volumes of AI-powered cyberattacks
Major cyber events in 2024
Malware and malicious payload trends
Cyberattack types and targets
Vulnerability exploit attempts on CVEs
Attacks on counties – USA
Expansion of bot farms – how, where, and why
In-depth analysis of the cyber threat landscape across North America, South America, Europe, APAC, and the Middle East
Why are attacks on smart factories rising?
Cyber risk predictions
Axis of attacks – Europe
Systemic attacks in the Middle East
Download the full report from here:
https://sectrio.com/resources/ot-threat-landscape-reports/sectrio-releases-ot-ics-and-iot-security-threat-landscape-report-2024/
Elevating Tactical DDD Patterns Through Object CalisthenicsDorra BARTAGUIZ
After immersing yourself in the blue book and its red counterpart, attending DDD-focused conferences, and applying tactical patterns, you're left with a crucial question: How do I ensure my design is effective? Tactical patterns within Domain-Driven Design (DDD) serve as guiding principles for creating clear and manageable domain models. However, achieving success with these patterns requires additional guidance. Interestingly, we've observed that a set of constraints initially designed for training purposes remarkably aligns with effective pattern implementation, offering a more ‘mechanical’ approach. Let's explore together how Object Calisthenics can elevate the design of your tactical DDD patterns, offering concrete help for those venturing into DDD for the first time!
Unlocking Productivity: Leveraging the Potential of Copilot in Microsoft 365, a presentation by Christoforos Vlachos, Senior Solutions Manager – Modern Workplace, Uni Systems
1. collectd & PostgreSQL
Mark Wong
markwkm@postgresql.org
mark.wong@myemma.com
PDXPUG
November 17, 2011
2. My Story
• How did I get to collectd?
• What is collectd
• Hacking collectd
• Using collectd with Postgres
• Visualizing the data
markwkm (PDXPUG) collectd & PostgreSQL November 17, 2011 2 / 43
3. Brief background
• Working at a little company called Emma http://myemma.com
• Collect performance data from production systems
markwkm (PDXPUG) collectd & PostgreSQL November 17, 2011 3 / 43
4. What did we have?
• A database with over 1 million database objects
• >500,000 tables
• >1,000,000 indexes
• Tables alone generate 11,000,000 data point per sample
markwkm (PDXPUG) collectd & PostgreSQL November 17, 2011 4 / 43
5. What did we try?
Only free things:
• Cacti http://www.cacti.net/
• Ganglia http://ganglia.info/
• Munin http://munin-monitoring.org/
• Reconnoiter https://labs.omniti.com/labs/reconnoiter
• Zenoss http://community.zenoss.org/
markwkm (PDXPUG) collectd & PostgreSQL November 17, 2011 5 / 43
6. What doesn’t work
Dependency on RRDtool; can’t handle more than hundreds of thousands of
metrics (Application Buffer-Cache Management for Performance: Running the
World’s Largest MRTG by David Plonka, Archit Gupta and Dale Carder, LISA
2007):
• Cacti
• Ganglia
• Munin
• Reconnoiter
• Zenoss
markwkm (PDXPUG) collectd & PostgreSQL November 17, 2011 6 / 43
7. Reconnoiter almost worked for us
Pro’s:
• Write your own SQL queries to collect data from Postgres
• Used Postgres instead of RRDtool for storing data
• JavaScript based on-the-fly charting
• Support for integrating many other health and stats collection solutions
Con’s:
• Data collection still couldn’t keep up; maybe needed more tuning
• Faster hardware? (using VM’s)
• More hardware? (scale out MQ processes)
markwkm (PDXPUG) collectd & PostgreSQL November 17, 2011 7 / 43
8. Couldn’t bring myself to try anything else
• Hands were tied, no resources available to help move forward.
• Can we build something light weight?
• Played with collectd (http://collectd.org/) while evaluating
Reconnoiter
markwkm (PDXPUG) collectd & PostgreSQL November 17, 2011 8 / 43
9. What is collectd?
collectd is a daemon which collects system performance
statistics periodically and provides mechanisms to store the
values in a variety of ways, for example in RRD files.
http://collectd.org/
markwkm (PDXPUG) collectd & PostgreSQL November 17, 2011 9 / 43
10. Does this look familiar?
Note: RRDtool is an option, not a requirement
markwkm (PDXPUG) collectd & PostgreSQL November 17, 2011 10 / 43
11. What is special about collectd?
From their web site:
• it’s written in C for performance and portability
• includes optimizations and features to handle hundreds
of thousands of data sets
• PostgreSQL plugin enables querying the database
• Can collect most operating systems statistics (I say “most” because I
don’t know if anything is missing)
• Over 90 total plugins
http://collectd.org/wiki/index.php/Table_of_Plugins
markwkm (PDXPUG) collectd & PostgreSQL November 17, 2011 11 / 43
12. collectd data description
• time - when the data was collected
• interval - frequency of data collection
• host - server hostname
• plugin - collectd plugin used
• plugin instance - additional plugin information
• type - type of data collected for set of values
• type instance - unique identifier of the metric
• dsnames - names for the values collected
• dstypes - type of data for values collected (e.g. counter, gauge, etc.)
• values - array of values collected
markwkm (PDXPUG) collectd & PostgreSQL November 17, 2011 12 / 43
14. . . . per database.
...
<Database bar>
Interval 60
Service "service_name"
Query backend # predefined
Query magic_tickets
</Database>
</Plugin>
Full details at
http://collectd.org/wiki/index.php/Plugin:PostgreSQL
markwkm (PDXPUG) collectd & PostgreSQL November 17, 2011 14 / 43
15. Hurdles
More meta data:
• Need a way to save schema, table, and index names; can’t differentiate
stats between tables and indexes
• Basic support of meta data in collectd but mostly unused
• How to store data in something other than RRDtool
markwkm (PDXPUG) collectd & PostgreSQL November 17, 2011 15 / 43
16. Wanted: additional meta data
Hack the PostgreSQL plugin to create meta data for:
• database - database name (maybe not needed, same as
plugin instance)
• schemaname - schema name
• tablename - table name
• indexname - index name
• metric - e.g. blks hit, blks read, seq scan, etc.
markwkm (PDXPUG) collectd & PostgreSQL November 17, 2011 16 / 43
17. Another database query for collecting a table statistic
<Query table_stats>
SELECT schemaname, relname, seq_scan
FROM pg_stat_all_tables;
<Query>
markwkm (PDXPUG) collectd & PostgreSQL November 17, 2011 17 / 43
18. Identify the data
<Result>
Type counter
InstancePrefix "seq_scan"
InstancesFrom "schemaname" "relname"
ValuesFrom "seq_scan"
</Result>
markwkm (PDXPUG) collectd & PostgreSQL November 17, 2011 18 / 43
19. Meta data specific parameters
<Database postgres>
Host "localhost"
Query table_stats
SchemanameColumn 0
TablenameColumn 1
</Database>
Note: The database name is set by what is specified in the <Database>tag, if
it is not retrieved by the query.
markwkm (PDXPUG) collectd & PostgreSQL November 17, 2011 19 / 43
20. Example data
• time: 2011-10-20 18:04:17-05
• interval: 300
• host: pong.int
• plugin: postgresql
• plugin instance: sandbox
• type: counter
• type instance: seq scan-pg catalog-pg class
• dsnames: {value}
• dstypes: {counter}
• values: {249873}
markwkm (PDXPUG) collectd & PostgreSQL November 17, 2011 20 / 43
21. Example meta data
• database: sandbox
• schemaname: pg catalog
• tablename: pg class
• indexname:
• metric: seq scan
markwkm (PDXPUG) collectd & PostgreSQL November 17, 2011 21 / 43
22. Now what?
Hand’s were tied (I think I mentioned that earlier); open sourced work to date:
• collectd forked with patches
https://github.com/mwongatemma/collectd
• YAMS https://github.com/myemma/yams
markwkm (PDXPUG) collectd & PostgreSQL November 17, 2011 22 / 43
23. Yet Another Monitoring System
markwkm (PDXPUG) collectd & PostgreSQL November 17, 2011 23 / 43
24. Switching hats and boosting code
Using extracurricular time working on equipment donated to Postgres from
SUN, IBM, and HP to continue proofing collectd changes.
markwkm (PDXPUG) collectd & PostgreSQL November 17, 2011 24 / 43
25. How am I going to move the data?
Options from available write plugins; guess which I used:
• Carbon - Graphite’s storage API to Whisper
http://collectd.org/wiki/index.php/Plugin:Carbon
• CSV http://collectd.org/wiki/index.php/Plugin:CSV
• Network - Send/Receive to other collectd daemons
http://collectd.org/wiki/index.php/Plugin:Network
• RRDCacheD http://collectd.org/wiki/index.php/Plugin:RRDCacheD
• RRDtool http://collectd.org/wiki/index.php/Plugin:RRDtool
• SysLog http://collectd.org/wiki/index.php/Plugin:SysLog
• UnixSock http://collectd.org/wiki/index.php/Plugin:UnixSock
• Write HTTP - PUTVAL (plain text), JSON
http://collectd.org/wiki/index.php/Plugin:Write_HTTP
markwkm (PDXPUG) collectd & PostgreSQL November 17, 2011 25 / 43
26. Process of elimination
If RRDtool (wriiten in C) can’t handle massive volumes of data, a Python
RRD like database probably can’t either:
• Carbon
• CSV
• Network
• RRDCacheD
• RRDtool
• SysLog
• UnixSock
• Write HTTP - PUTVAL (plain text), JSON
markwkm (PDXPUG) collectd & PostgreSQL November 17, 2011 26 / 43
27. Process of elimination
Writing to other collectd daemons or just locally doesn’t seem useful at the
moment:
• CSV
• Network
• SysLog
• UnixSock
• Write HTTP - PUTVAL (plain text), JSON
markwkm (PDXPUG) collectd & PostgreSQL November 17, 2011 27 / 43
29. Random: What Write HTTP PUTVAL data looks like
Note: Each PUTVAL is a single line but is broken up into two lines to fit onto
the slide.
PUTVAL leeloo.lan.home.verplant.org/disk-sda/disk_octets
interval=10 1251533299:197141504:175136768
PUTVAL leeloo.lan.home.verplant.org/disk-sda/disk_ops
interval=10 1251533299:10765:12858
PUTVAL leeloo.lan.home.verplant.org/disk-sda/disk_time
interval=10 1251533299:5:140
PUTVAL leeloo.lan.home.verplant.org/disk-sda/disk_merged
interval=10 1251533299:4658:29899
markwkm (PDXPUG) collectd & PostgreSQL November 17, 2011 29 / 43
30. Random: What the Write HTTP JSON data looks like
Note: Write HTTP packs as much data as it can into a 4KB buffer.
[ {
"values": [197141504, 175136768],
"dstypes": ["counter", "counter"],
"dsnames": ["read", "write"],
"time": 1251533299,
"interval": 10,
"host": "leeloo.lan.home.verplant.org",
"plugin": "disk",
"plugin_instance": "sda",
"type": "disk_octets",
"type_instance": ""
}, ... ]
markwkm (PDXPUG) collectd & PostgreSQL November 17, 2011 30 / 43
31. I didn’t know anything about CouchDB at the time
• Query interface not really suited for retrieving data to visualize
• Insert performance not suited for millions of metrics of data over short
intervals (can insert same data into Postgres several orders of
magnitude faster)
markwkm (PDXPUG) collectd & PostgreSQL November 17, 2011 31 / 43
32. Now where am I going to put the data?
Hoping that using the Write HTTP is still a good choice:
• Write an ETL
• Table partitioning logic; creation of partition tables
• Transform JSON data into INSERT statements
• Use Postgres
markwkm (PDXPUG) collectd & PostgreSQL November 17, 2011 32 / 43
33. Database design
Table "collectd.value_list"
Column | Type | Modifiers
-----------------+--------------------------+-----------
time | timestamp with time zone | not null
interval | integer | not null
host | character varying(64) | not null
plugin | character varying(64) | not null
plugin_instance | character varying(64) |
type | character varying(64) | not null
type_instance | character varying(64) |
dsnames | character varying(512)[] | not null
dstypes | character varying(8)[] | not null
values | numeric[] | not null
markwkm (PDXPUG) collectd & PostgreSQL November 17, 2011 33 / 43
34. Take advantage of partitioning
At least table inheritance in Postgres’ case; partition data by plugin
markwkm (PDXPUG) collectd & PostgreSQL November 17, 2011 34 / 43
35. Child table
Table "collectd.vl_postgresql"
Column | Type | Modifiers
-----------------+--------------------------+-----------
...
database | character varying(64) | not null
schemaname | character varying(64) |
tablename | character varying(64) |
indexname | character varying(64) |
metric | character varying(64) | not null
Check constraints:
"vl_postgresql_plugin_check" CHECK (plugin::text =
’postgresql’::text)
Inherits: value_list
markwkm (PDXPUG) collectd & PostgreSQL November 17, 2011 35 / 43
36. How much partitioning?
Lots of straightforward options:
• Date
• Database
• Schema
• Table
• Index
• Metric
markwkm (PDXPUG) collectd & PostgreSQL November 17, 2011 36 / 43
37. Back to the ETL
Parameters set for fastest path to working prototype:
• Keeping using HTTP POST (Write HTTP plugin) for HTTP protocol
and JSON
• Use Python for built in HTTP Server and JSON parsing (Emma is
primarily a Python shop)
• Use SQLAlchemy/psycopg2
markwkm (PDXPUG) collectd & PostgreSQL November 17, 2011 37 / 43
38. Back again to the ETL
Python didn’t perform; combination of JSON parsing, data transformation,
and INSERT performance still several orders of magnitude below acceptable
levels:
• redis to queue data to transform
• lighttpd for the HTTP interface
• fastcgi C program to push things to redis
• multi-threaded C program using libpq for Postgres API
• pop data out of redis
• table partitioning creation logic
• transform JSON data into INSERT statements
markwkm (PDXPUG) collectd & PostgreSQL November 17, 2011 38 / 43
39. Success?
• Table statistics for 1 million tables collect in approximately 12 minutes.
• Is that acceptable?
• Can we go faster?
markwkm (PDXPUG) collectd & PostgreSQL November 17, 2011 39 / 43
40. If you don’t have millions of data
Easier ways to visualize the data:
• RRDtool
• RRDtool compatible front-ends
http://collectd.org/wiki/index.php/List_of_front-ends
• Graphite with the Carbon and Whisper combo
http://graphite.wikidot.com/
• Reconnoiter
markwkm (PDXPUG) collectd & PostgreSQL November 17, 2011 40 / 43
43. License
This work is licensed under a Creative Commons Attribution 3.0 Unported
License. To view a copy of this license, (a) visit
http://creativecommons.org/licenses/by/3.0/us/; or, (b) send a
letter to Creative Commons, 171 2nd Street, Suite 300, San Francisco,
California, 94105, USA.
markwkm (PDXPUG) collectd & PostgreSQL November 17, 2011 43 / 43