PostgreSQL and ZFS were made for each other. This talk dives downstack into the internals and way that PostgreSQL consumes disk resources and tricks that are available if you run PostgreSQL on ZFS (ZFS on Linux, ZFS on FreeBSD, or ZFS on Illumos). Topics covered will include:
* Performance and sizing considerations
* Workload estimation heuristics
* Standard administrative practices that leverage ZFS
* Recovery using ZFS
* Performing database migrations using ZFS
Presentation slides for running MySQL (InnoDB) on ZFS. Since most databases have analogues to optimisation targets mentioned, it is more broadly applicable.
Talk for PerconaLive 2016 by Brendan Gregg. Video: https://www.youtube.com/watch?v=CbmEDXq7es0 . "Systems performance provides a different perspective for analysis and tuning, and can help you find performance wins for your databases, applications, and the kernel. However, most of us are not performance or kernel engineers, and have limited time to study this topic. This talk summarizes six important areas of Linux systems performance in 50 minutes: observability tools, methodologies, benchmarking, profiling, tracing, and tuning. Included are recipes for Linux performance analysis and tuning (using vmstat, mpstat, iostat, etc), overviews of complex areas including profiling (perf_events), static tracing (tracepoints), and dynamic tracing (kprobes, uprobes), and much advice about what is and isn't important to learn. This talk is aimed at everyone: DBAs, developers, operations, etc, and in any environment running Linux, bare-metal or the cloud."
PostgreSQL and ZFS were made for each other. This talk dives downstack into the internals and way that PostgreSQL consumes disk resources and tricks that are available if you run PostgreSQL on ZFS (ZFS on Linux, ZFS on FreeBSD, or ZFS on Illumos). Topics covered will include:
* Performance and sizing considerations
* Workload estimation heuristics
* Standard administrative practices that leverage ZFS
* Recovery using ZFS
* Performing database migrations using ZFS
Presentation slides for running MySQL (InnoDB) on ZFS. Since most databases have analogues to optimisation targets mentioned, it is more broadly applicable.
Talk for PerconaLive 2016 by Brendan Gregg. Video: https://www.youtube.com/watch?v=CbmEDXq7es0 . "Systems performance provides a different perspective for analysis and tuning, and can help you find performance wins for your databases, applications, and the kernel. However, most of us are not performance or kernel engineers, and have limited time to study this topic. This talk summarizes six important areas of Linux systems performance in 50 minutes: observability tools, methodologies, benchmarking, profiling, tracing, and tuning. Included are recipes for Linux performance analysis and tuning (using vmstat, mpstat, iostat, etc), overviews of complex areas including profiling (perf_events), static tracing (tracepoints), and dynamic tracing (kprobes, uprobes), and much advice about what is and isn't important to learn. This talk is aimed at everyone: DBAs, developers, operations, etc, and in any environment running Linux, bare-metal or the cloud."
zfsday talk (a video is on the last slide). The performance of the file system, or disks, is often the target of blame, especially in multi-tenant cloud environments. At Joyent we deploy a public cloud on ZFS-based systems, and frequently investigate performance with a wide variety of applications in growing environments. This talk is about ZFS performance observability, showing the tools and approaches we use to quickly show what ZFS is doing. This includes observing ZFS I/O throttling, an enhancement added to illumos-ZFS to isolate performance between neighbouring tenants, and the use of DTrace and heat maps to examine latency distributions and locate outliers.
A brief talk on systems performance for the July 2013 meetup "A Midsummer Night's System", video: http://www.youtube.com/watch?v=P3SGzykDE4Q. This summarizes how systems performance has changed from the 1990's to today. This was the reason for writing a new book on systems performance, to provide a reference that is up to date, covering new tools, technologies, and methodologies.
Kernel Recipes 2017: Performance Analysis with BPFBrendan Gregg
Talk by Brendan Gregg at Kernel Recipes 2017 (Paris): "The in-kernel Berkeley Packet Filter (BPF) has been enhanced in recent kernels to do much more than just filtering packets. It can now run user-defined programs on events, such as on tracepoints, kprobes, uprobes, and perf_events, allowing advanced performance analysis tools to be created. These can be used in production as the BPF virtual machine is sandboxed and will reject unsafe code, and are already in use at Netflix.
Beginning with the bpf() syscall in 3.18, enhancements have been added in many kernel versions since, with major features for BPF analysis landing in Linux 4.1, 4.4, 4.7, and 4.9. Specific capabilities these provide include custom in-kernel summaries of metrics, custom latency measurements, and frequency counting kernel and user stack traces on events. One interesting case involves saving stack traces on wake up events, and associating them with the blocked stack trace: so that we can see the blocking stack trace and the waker together, merged in kernel by a BPF program (that particular example is in the kernel as samples/bpf/offwaketime).
This talk will discuss the new BPF capabilities for performance analysis and debugging, and demonstrate the new open source tools that have been developed to use it, many of which are in the Linux Foundation iovisor bcc (BPF Compiler Collection) project. These include tools to analyze the CPU scheduler, TCP performance, file system performance, block I/O, and more."
Make Your Containers Faster: Linux Container Performance ToolsKernel TLV
If you look under the hood, Linux containers are just processes with some isolation features and resource quotas sprinkled on top. In this talk, we will apply modern Linux performance tools to container analysis: get high-level resource utilization on running containers with docker stats, htop, and nsenter; dig into high-CPU issues with perf; detect slow filesystem latency with BPF-based tools; and generate flame graphs of interesting event call stacks.
Sasha Goldshtein is the CTO of Sela Group, a Microsoft MVP and Regional Director, Pluralsight and O'Reilly author, and international consultant and trainer. Sasha is the author of two books and multiple online courses, and a prolific blogger. He is also an active open source contributor to projects focused on system diagnostics, performance monitoring, and tracing -- across multiple operating systems and runtimes. Sasha authored and delivered training courses on Linux performance optimization, event tracing, production debugging, mobile application development, and modern C++. Between his consulting engagements, Sasha speaks at international conferences world-wide.
You can find more details on the meetup page - https://www.meetup.com/Tel-Aviv-Yafo-Linux-Kernel-Meetup/events/245319189/
Kernel Recipes 2017: Using Linux perf at NetflixBrendan Gregg
Talk for Kernel Recipes 2017 by Brendan Gregg. "Linux perf is a crucial performance analysis tool at Netflix, and is used by a self-service GUI for generating CPU flame graphs and other reports. This sounds like an easy task, however, getting perf to work properly in VM guests running Java, Node.js, containers, and other software, has been at times a challenge. This talk summarizes Linux perf, how we use it at Netflix, the various gotchas we have encountered, and a summary of advanced features."
High Availability can be a curiously nebulous term, and most people probably don't care about it until they can't access their online banking service, or their plane crashes.
This presentation examines some of the considerations necessary when building highly available computer systems, then focuses on the HA infrastructure software currently available from the Corosync/OpenAIS, Linux-HA and Pacemaker projects.
Originally presented at Linux Users Victoria in April 2010 (http://luv.asn.au/2010/04/06)
EuroBSDcon 2017 System Performance Analysis MethodologiesBrendan Gregg
keynote by Brendan Gregg. "Traditional performance monitoring makes do with vendor-supplied metrics, often involving interpretation and inference, and with numerous blind spots. Much in the field of systems performance is still living in the past: documentation, procedures, and analysis GUIs built upon the same old metrics. Modern BSD has advanced tracers and PMC tools, providing virtually endless metrics to aid performance analysis. It's time we really used them, but the problem becomes which metrics to use, and how to navigate them quickly to locate the root cause of problems.
There's a new way to approach performance analysis that can guide you through the metrics. Instead of starting with traditional metrics and figuring out their use, you start with the questions you want answered then look for metrics to answer them. Methodologies can provide these questions, as well as a starting point for analysis and guidance for locating the root cause. They also pose questions that the existing metrics may not yet answer, which may be critical in solving the toughest problems. System methodologies include the USE method, workload characterization, drill-down analysis, off-CPU analysis, chain graphs, and more.
This talk will discuss various system performance issues, and the methodologies, tools, and processes used to solve them. Many methodologies will be discussed, from the production proven to the cutting edge, along with recommendations for their implementation on BSD systems. In general, you will learn to think differently about analyzing your systems, and make better use of the modern tools that BSD provides."
PostgreSQL is designed to be easily extensible. For this reason, extensions loaded into the database can function just like features that are built in. In this session, we will learn more about PostgreSQL extension framework, how are they built, look at some popular extensions, management of these extensions in your deployments.
Linux Performance Analysis: New Tools and Old SecretsBrendan Gregg
Talk for USENIX/LISA2014 by Brendan Gregg, Netflix. At Netflix performance is crucial, and we use many high to low level tools to analyze our stack in different ways. In this talk, I will introduce new system observability tools we are using at Netflix, which I've ported from my DTraceToolkit, and are intended for our Linux 3.2 cloud instances. These show that Linux can do more than you may think, by using creative hacks and workarounds with existing kernel features (ftrace, perf_events). While these are solving issues on current versions of Linux, I'll also briefly summarize the future in this space: eBPF, ktap, SystemTap, sysdig, etc.
Norma tecnica de proteccion seguridad y salud de los trabajadores y trabajadoras. cuando y donde deba ser usada, asi como su relacion con los conceptos de seguridad y salud puestos en marcha por los organos competentes dentro de las organizaciones productivas
zfsday talk (a video is on the last slide). The performance of the file system, or disks, is often the target of blame, especially in multi-tenant cloud environments. At Joyent we deploy a public cloud on ZFS-based systems, and frequently investigate performance with a wide variety of applications in growing environments. This talk is about ZFS performance observability, showing the tools and approaches we use to quickly show what ZFS is doing. This includes observing ZFS I/O throttling, an enhancement added to illumos-ZFS to isolate performance between neighbouring tenants, and the use of DTrace and heat maps to examine latency distributions and locate outliers.
A brief talk on systems performance for the July 2013 meetup "A Midsummer Night's System", video: http://www.youtube.com/watch?v=P3SGzykDE4Q. This summarizes how systems performance has changed from the 1990's to today. This was the reason for writing a new book on systems performance, to provide a reference that is up to date, covering new tools, technologies, and methodologies.
Kernel Recipes 2017: Performance Analysis with BPFBrendan Gregg
Talk by Brendan Gregg at Kernel Recipes 2017 (Paris): "The in-kernel Berkeley Packet Filter (BPF) has been enhanced in recent kernels to do much more than just filtering packets. It can now run user-defined programs on events, such as on tracepoints, kprobes, uprobes, and perf_events, allowing advanced performance analysis tools to be created. These can be used in production as the BPF virtual machine is sandboxed and will reject unsafe code, and are already in use at Netflix.
Beginning with the bpf() syscall in 3.18, enhancements have been added in many kernel versions since, with major features for BPF analysis landing in Linux 4.1, 4.4, 4.7, and 4.9. Specific capabilities these provide include custom in-kernel summaries of metrics, custom latency measurements, and frequency counting kernel and user stack traces on events. One interesting case involves saving stack traces on wake up events, and associating them with the blocked stack trace: so that we can see the blocking stack trace and the waker together, merged in kernel by a BPF program (that particular example is in the kernel as samples/bpf/offwaketime).
This talk will discuss the new BPF capabilities for performance analysis and debugging, and demonstrate the new open source tools that have been developed to use it, many of which are in the Linux Foundation iovisor bcc (BPF Compiler Collection) project. These include tools to analyze the CPU scheduler, TCP performance, file system performance, block I/O, and more."
Make Your Containers Faster: Linux Container Performance ToolsKernel TLV
If you look under the hood, Linux containers are just processes with some isolation features and resource quotas sprinkled on top. In this talk, we will apply modern Linux performance tools to container analysis: get high-level resource utilization on running containers with docker stats, htop, and nsenter; dig into high-CPU issues with perf; detect slow filesystem latency with BPF-based tools; and generate flame graphs of interesting event call stacks.
Sasha Goldshtein is the CTO of Sela Group, a Microsoft MVP and Regional Director, Pluralsight and O'Reilly author, and international consultant and trainer. Sasha is the author of two books and multiple online courses, and a prolific blogger. He is also an active open source contributor to projects focused on system diagnostics, performance monitoring, and tracing -- across multiple operating systems and runtimes. Sasha authored and delivered training courses on Linux performance optimization, event tracing, production debugging, mobile application development, and modern C++. Between his consulting engagements, Sasha speaks at international conferences world-wide.
You can find more details on the meetup page - https://www.meetup.com/Tel-Aviv-Yafo-Linux-Kernel-Meetup/events/245319189/
Kernel Recipes 2017: Using Linux perf at NetflixBrendan Gregg
Talk for Kernel Recipes 2017 by Brendan Gregg. "Linux perf is a crucial performance analysis tool at Netflix, and is used by a self-service GUI for generating CPU flame graphs and other reports. This sounds like an easy task, however, getting perf to work properly in VM guests running Java, Node.js, containers, and other software, has been at times a challenge. This talk summarizes Linux perf, how we use it at Netflix, the various gotchas we have encountered, and a summary of advanced features."
High Availability can be a curiously nebulous term, and most people probably don't care about it until they can't access their online banking service, or their plane crashes.
This presentation examines some of the considerations necessary when building highly available computer systems, then focuses on the HA infrastructure software currently available from the Corosync/OpenAIS, Linux-HA and Pacemaker projects.
Originally presented at Linux Users Victoria in April 2010 (http://luv.asn.au/2010/04/06)
EuroBSDcon 2017 System Performance Analysis MethodologiesBrendan Gregg
keynote by Brendan Gregg. "Traditional performance monitoring makes do with vendor-supplied metrics, often involving interpretation and inference, and with numerous blind spots. Much in the field of systems performance is still living in the past: documentation, procedures, and analysis GUIs built upon the same old metrics. Modern BSD has advanced tracers and PMC tools, providing virtually endless metrics to aid performance analysis. It's time we really used them, but the problem becomes which metrics to use, and how to navigate them quickly to locate the root cause of problems.
There's a new way to approach performance analysis that can guide you through the metrics. Instead of starting with traditional metrics and figuring out their use, you start with the questions you want answered then look for metrics to answer them. Methodologies can provide these questions, as well as a starting point for analysis and guidance for locating the root cause. They also pose questions that the existing metrics may not yet answer, which may be critical in solving the toughest problems. System methodologies include the USE method, workload characterization, drill-down analysis, off-CPU analysis, chain graphs, and more.
This talk will discuss various system performance issues, and the methodologies, tools, and processes used to solve them. Many methodologies will be discussed, from the production proven to the cutting edge, along with recommendations for their implementation on BSD systems. In general, you will learn to think differently about analyzing your systems, and make better use of the modern tools that BSD provides."
PostgreSQL is designed to be easily extensible. For this reason, extensions loaded into the database can function just like features that are built in. In this session, we will learn more about PostgreSQL extension framework, how are they built, look at some popular extensions, management of these extensions in your deployments.
Linux Performance Analysis: New Tools and Old SecretsBrendan Gregg
Talk for USENIX/LISA2014 by Brendan Gregg, Netflix. At Netflix performance is crucial, and we use many high to low level tools to analyze our stack in different ways. In this talk, I will introduce new system observability tools we are using at Netflix, which I've ported from my DTraceToolkit, and are intended for our Linux 3.2 cloud instances. These show that Linux can do more than you may think, by using creative hacks and workarounds with existing kernel features (ftrace, perf_events). While these are solving issues on current versions of Linux, I'll also briefly summarize the future in this space: eBPF, ktap, SystemTap, sysdig, etc.
Norma tecnica de proteccion seguridad y salud de los trabajadores y trabajadoras. cuando y donde deba ser usada, asi como su relacion con los conceptos de seguridad y salud puestos en marcha por los organos competentes dentro de las organizaciones productivas
Developing Your Business Through Internal ControlsSkoda Minotti
Internal controls are mandated for public companies, but companies of all sizes can benefit from strong internal controls and process documentation. This presentation will describe what internal controls mean to small businesses and private companies. We will discuss the approach for documenting processes, identifying where the weaknesses are within a company and how to mitigate those risks by developing simple controls.
Practica innovadora rosario cañar y nancy garciaEdwin Riascos
CON EL DESARROLLO DE ESTE TEMA LOS ESTUDIANTES DEL GRADO TERCERO DEL CENTRO EDUCATIVO QUEBRADAHONDA, DE LA INSTITUCION EDUCATIVA NUESTRA SEÑORA DEL CARMEN, MUNICIPIO DE LA FLORIDA NARIÑO; FORTALECERAN EL AMOR POR LOS ANIMALES Y SU CUIDADO.
Presented by Jon Bloom, Founder and CEO, McGrath/Power, Marketing Chair for the Americas Region on the Worldcom Public Relations Group Board of Director, former journalist with the San Jose Mercury News
Presentación utilizada durante la primera sesión del Seminario Permanente "Comunicación, Cultura y Sociedad" de la Facultad de Ciencias de la Comunicación de la Universidad Autónoma de Coahuila, en torno a la sociología de la tecnología, desde una perspectiva constructivista que contempla la Teoría del Actor-Red y la Construcción Social de la Tecnología
CON EL DESARROLLO DE ESTE TEMA LOS ESTUDIANTES DEL GRADO TERCERO DEL CENTRO EDUCATIVO QUEBRADAHONDA, DE LA INSTITUCION EDUCATIVA NUESTRA SEÑORA DEL CARMEN, MUNICIPIO DE LA FLORIDA NARIÑO; FORTALECERAN EL AMOR POR EL MEDIO AMBIENTE.
Salesforce Marketing Cloud Training | Salesforce Training For Beginners - Mar...Edureka!
This Edureka Salesforce Marketing Cloud training video for beginners will help you learn Salesforce marketing cloud benefits, what it is, its various features, use case along with marketing cloud demo. This training video is ideal for beginners to learn Salesforce marketing cloud. You can also read the blog here: https://goo.gl/CS6aV4
Introducing NoSQL and MongoDB to complement Relational Databases (AMIS SIG 14...Lucas Jellema
This presentation gives an brief overview of the history of relational databases, ACID and SQL and presents some of the key strentgths and potential weaknesses. It introduces the rise of NoSQL - why it arose, what is entails, when to use it. The presentation focuses on MongoDB as prime example of NoSQL document store and it shows how to interact with MongoDB from JavaScript (NodeJS) and Java.
Tracxn Research - Insurance Tech Landscape, February 2017Tracxn
Round count hit an all-time sector peak in seed (81), early (72), and late stage (17), with late stage deal activity registering the most growth (46%).
Slides presented at Percona Live Europe Open Source Database Conference 2019, Amsterdam, 2019-10-01.
Imagine a world where all Wikipedia articles disappear due to a human error or software bug. Sounds unreal? According to some estimations, it would take an excess of hundreds of million person-hours to be written again. To prevent that scenario from ever happening, our SRE team at Wikimedia recently refactored the relational database recovery system.
In this session, we will discuss how we backup 550TB of MariaDB data without impacting the 15 billion page views per month we get. We will cover what were our initial plans to replace the old infrastructure, how we achieved recovering 2TB databases in less than 30 minutes while maintaining per-table granularity, as well as the different types of backups we implemented. Lastly, we will talk about lessons learned, what went well, how our original plans changed and future work.
Nagios Conference 2014 - Andy Brist - Nagios XI Failover and HA SolutionsNagios
Andy Brist's presentation on High Availability and Failover Solutions for Nagios XI. The presentation was given during the Nagios World Conference North America held Oct 13th - Oct 16th, 2014 in Saint Paul, MN. For more information on the conference (including photos and videos), visit: http://go.nagios.com/conference
Lock, Stock and Backup: Data GuaranteedJervin Real
Percona Live 2017 - the decisions you need to make, the tools we recommend, the process you need to consider for a successful backup implementation for your MySQL services.
A Backup Today Saves Tomorrow is a presentation from Percona Live 2013 that provides insight into planning and the tools used today to capture MySQL backups.
Percona Live 2022 - PBM - The Backup Open Source Tool for MongoDBJean Da Silva
Backup and restore are two of the most important things for databases. We don't often use the backup, but during a disaster situation, it is crucial to work.
In this session, we will discuss Percona Backup for MongoDB (PBM short).
We will walk through the process of taking backups and executing restores. We will also introduce the newest backup method that PBM offers, the physical backup in addition to the logical backup. After the introduction of the backup methods, we will evaluate the backup and restore times, and how to store the backup on remote backup storage.
Training Slides: 203 - Backup & RecoveryContinuent
Watch this 36min training to learn about planning for backups, what some of the methods and tools are, how to restore backups and more.
TOPICS COVERED
- How to develop a backup plan
- Methods and tools for taking a backup
- Verifying the backup contains the last binary position, and the importance of this
- Restore backups into the cluster
- Provision a replica from an existing datasource
Building tungsten-clusters-with-postgre sql-hot-standby-and-streaming-replica...Command Prompt., Inc
Alex Alexander & Linas Virbalas
Hot standby and streaming replication will move the needle forward for high availability and scaling for a wide number of applications. Tungsten already supports clustering using warm standby. In this talk we will describe how to build clusters using the new PostgreSQL features and give our report from the trenches.
This talk will cover how hot standby and streaming replication work from a user perspective, then dive into a description of how to use them, taking Tungsten as an example. We'll cover the following issues:
* Configuration of warm standby and streaming replication
* Provisioning new standby instances
* Strategies for balancing reads across primary and standby database
* Managing failover
* Troubleshooting and gotchas
Please join us for an enlightening discussion a set of PostgreSQL features that are interesting to a wide range of PostgreSQL users.
This talk is a followup to Deploying systemd at scale that was presented at systemd.conf 2016, and covers the aftermath of the migration of our fleet to CentOS 7. Now that systemd is available everywhere, we found more and more services that started adopting it for their deployment, leveraging its features and occasionally exposing interesting behaviors. At the same time, we've been able to hone our process for integrating and rolling out new versions of systemd on the fleet, and started building tooling to manage and monitor it at scale.
What if …
- Traditional, labour-intensive backup and archive practices for your MySQL, MariaDB, MongoDB and PostgreSQL databases were a thing of the past?
- You could have one backup management solution for all your business data?
- You could ensure integrity of all your backups?
- You could leverage the competitive pricing and almost limitless capacity of cloud-based backup while meeting cost, manageability, and compliance requirements from the business.
Welcome to our webinar on Backup Management with ClusterControl.
ClusterControl’s centralized backup management for open source databases provides you with hot backups of large datasets, point in time recovery in a couple of clicks, at-rest and in-transit data encryption, data integrity via automatic restore verification, cloud backups (AWS, Google and Azure) for Disaster Recovery, retention policies to ensure compliance, and automated alerts and reporting.
Whether you are looking at rebuilding your existing backup infrastructure, or updating it, this webinar is for you!
AGENDA
- Backup and recovery management of local or remote databases
- Logical or physical backups
- Full or Incremental backups
- Position or time-based Point in Time Recovery (for MySQL and PostgreSQL)
- Upload to the cloud (Amazon S3, Google Cloud Storage, Azure Storage)
- Encryption of backup data
- Compression of backup data
- One centralized backup system for your open source databases (Demo)
- Schedule, manage and operate backups
- Define backup policies, retention, history
- Validation - Automatic restore verification
- Backup reporting
SPEAKER
Bartlomiej Oles, Senior Support Engineer at Severalnines, is a MySQL and Oracle DBA, with over 15 years experience in managing highly available production systems at IBM, Nordea Bank, Acxiom, Lufthansa, and other Fortune 500 companies. In the past five years, his focus has been on building and applying automation tools to manage multi-datacenter database environments.
A look at some of the ways available to deploy Postgres in a Kubernetes cloud environment, either in small scale using simple configurations, or in larger scale using tools such as Helm charts and the Crunchy PostgreSQL Operator. A short introduction to Kubernetes will be given to explain the concepts involved, followed by examples from each deployment method and observations on the key differences.
OSBConf 2016: The Database Backup is done - what next? - by Jörg BrüheNETWAYS
“Nobody wants to do a backup – but everybody demands a restore”: This is a pointed description as seen by many system or database administrators.
Only the fact that restore cannot be had without backup ensures that they are given the resources for backup.
But creating a backup does not settle the matter. In their very own interest, admins must ensure that restore will really work, that the backup has no
hidden damage or is otherwise unsuitable. So a test system is needed to perform and verify a restore.
The result should not be simply overwritten: It is real live data, available on a system which is neither loaded nor changed by production!
These are perfect conditions to do load-intensive evaluations, create excerpts for billing or anonymize the data so that they may be used as snapshots for
developments and tests.
All this can be automated, and data privacy rules can be fit in.
I will present the relevant scripts and settings of a MySQL setup with about 30 production instances (each a master-master replication), from which you
may take the concepts and code snippets that fit your environment.
Techniques to optimize the pagerank algorithm usually fall in two categories. One is to try reducing the work per iteration, and the other is to try reducing the number of iterations. These goals are often at odds with one another. Skipping computation on vertices which have already converged has the potential to save iteration time. Skipping in-identical vertices, with the same in-links, helps reduce duplicate computations and thus could help reduce iteration time. Road networks often have chains which can be short-circuited before pagerank computation to improve performance. Final ranks of chain nodes can be easily calculated. This could reduce both the iteration time, and the number of iterations. If a graph has no dangling nodes, pagerank of each strongly connected component can be computed in topological order. This could help reduce the iteration time, no. of iterations, and also enable multi-iteration concurrency in pagerank computation. The combination of all of the above methods is the STICD algorithm. [sticd] For dynamic graphs, unchanged components whose ranks are unaffected can be skipped altogether.
Levelwise PageRank with Loop-Based Dead End Handling Strategy : SHORT REPORT ...Subhajit Sahu
Abstract — Levelwise PageRank is an alternative method of PageRank computation which decomposes the input graph into a directed acyclic block-graph of strongly connected components, and processes them in topological order, one level at a time. This enables calculation for ranks in a distributed fashion without per-iteration communication, unlike the standard method where all vertices are processed in each iteration. It however comes with a precondition of the absence of dead ends in the input graph. Here, the native non-distributed performance of Levelwise PageRank was compared against Monolithic PageRank on a CPU as well as a GPU. To ensure a fair comparison, Monolithic PageRank was also performed on a graph where vertices were split by components. Results indicate that Levelwise PageRank is about as fast as Monolithic PageRank on the CPU, but quite a bit slower on the GPU. Slowdown on the GPU is likely caused by a large submission of small workloads, and expected to be non-issue when the computation is performed on massive graphs.
As Europe's leading economic powerhouse and the fourth-largest hashtag#economy globally, Germany stands at the forefront of innovation and industrial might. Renowned for its precision engineering and high-tech sectors, Germany's economic structure is heavily supported by a robust service industry, accounting for approximately 68% of its GDP. This economic clout and strategic geopolitical stance position Germany as a focal point in the global cyber threat landscape.
In the face of escalating global tensions, particularly those emanating from geopolitical disputes with nations like hashtag#Russia and hashtag#China, hashtag#Germany has witnessed a significant uptick in targeted cyber operations. Our analysis indicates a marked increase in hashtag#cyberattack sophistication aimed at critical infrastructure and key industrial sectors. These attacks range from ransomware campaigns to hashtag#AdvancedPersistentThreats (hashtag#APTs), threatening national security and business integrity.
🔑 Key findings include:
🔍 Increased frequency and complexity of cyber threats.
🔍 Escalation of state-sponsored and criminally motivated cyber operations.
🔍 Active dark web exchanges of malicious tools and tactics.
Our comprehensive report delves into these challenges, using a blend of open-source and proprietary data collection techniques. By monitoring activity on critical networks and analyzing attack patterns, our team provides a detailed overview of the threats facing German entities.
This report aims to equip stakeholders across public and private sectors with the knowledge to enhance their defensive strategies, reduce exposure to cyber risks, and reinforce Germany's resilience against cyber threats.
Show drafts
volume_up
Empowering the Data Analytics Ecosystem: A Laser Focus on Value
The data analytics ecosystem thrives when every component functions at its peak, unlocking the true potential of data. Here's a laser focus on key areas for an empowered ecosystem:
1. Democratize Access, Not Data:
Granular Access Controls: Provide users with self-service tools tailored to their specific needs, preventing data overload and misuse.
Data Catalogs: Implement robust data catalogs for easy discovery and understanding of available data sources.
2. Foster Collaboration with Clear Roles:
Data Mesh Architecture: Break down data silos by creating a distributed data ownership model with clear ownership and responsibilities.
Collaborative Workspaces: Utilize interactive platforms where data scientists, analysts, and domain experts can work seamlessly together.
3. Leverage Advanced Analytics Strategically:
AI-powered Automation: Automate repetitive tasks like data cleaning and feature engineering, freeing up data talent for higher-level analysis.
Right-Tool Selection: Strategically choose the most effective advanced analytics techniques (e.g., AI, ML) based on specific business problems.
4. Prioritize Data Quality with Automation:
Automated Data Validation: Implement automated data quality checks to identify and rectify errors at the source, minimizing downstream issues.
Data Lineage Tracking: Track the flow of data throughout the ecosystem, ensuring transparency and facilitating root cause analysis for errors.
5. Cultivate a Data-Driven Mindset:
Metrics-Driven Performance Management: Align KPIs and performance metrics with data-driven insights to ensure actionable decision making.
Data Storytelling Workshops: Equip stakeholders with the skills to translate complex data findings into compelling narratives that drive action.
Benefits of a Precise Ecosystem:
Sharpened Focus: Precise access and clear roles ensure everyone works with the most relevant data, maximizing efficiency.
Actionable Insights: Strategic analytics and automated quality checks lead to more reliable and actionable data insights.
Continuous Improvement: Data-driven performance management fosters a culture of learning and continuous improvement.
Sustainable Growth: Empowered by data, organizations can make informed decisions to drive sustainable growth and innovation.
By focusing on these precise actions, organizations can create an empowered data analytics ecosystem that delivers real value by driving data-driven decisions and maximizing the return on their data investment.
Adjusting primitives for graph : SHORT REPORT / NOTESSubhajit Sahu
Graph algorithms, like PageRank Compressed Sparse Row (CSR) is an adjacency-list based graph representation that is
Multiply with different modes (map)
1. Performance of sequential execution based vs OpenMP based vector multiply.
2. Comparing various launch configs for CUDA based vector multiply.
Sum with different storage types (reduce)
1. Performance of vector element sum using float vs bfloat16 as the storage type.
Sum with different modes (reduce)
1. Performance of sequential execution based vs OpenMP based vector element sum.
2. Performance of memcpy vs in-place based CUDA based vector element sum.
3. Comparing various launch configs for CUDA based vector element sum (memcpy).
4. Comparing various launch configs for CUDA based vector element sum (in-place).
Sum with in-place strategies of CUDA mode (reduce)
1. Comparing various launch configs for CUDA based vector element sum (in-place).
8. pg_dumpall
Pros:
• Globals
Cons:
● Requires superuser privileges
● No granularity while restoring
● Plaintext only
● Cannot take advantage of faster/parallel restore with pg_restore
8
10. Why do you need both?
The more the merrier:
Database wide restores - Filesystem restore is faster!
Someone dropped a table? - Dump backups are faster!
10
11. pg_basebackup
Advantage:
• In core postgres
• No explicit backup mode required
• Multiple instances can run concurrently
• Backups can be made on master as well as standby
Disadvantage:
• Slower when compared to snapshot file-system technologies
• Backups are always of the entire cluster
11
12. pg_basebackup Requirements
• A superuser or a user with REPLICATION permissions
can be used
• archive_command parameter should be set to true
• max_wal_senders should be at least 1
12
13. Delayed Replicas as Backups
• In situations where accidental damage was done that was realized in a
short period of time.
• May still result in some data loss post recovery, but it depends on the
importance of table, other objects dependent on it, etc.
13
14. Version Controlled DDL
• Committing daily changes to schema design, functions, triggers etc. into
source control
• Can go back to any state
• pg_extractor (https://github.com/omniti-labs/pg_extractor) facilitates
this by creating files for each database object, followed by integration
with git, svn, etc.
14
16. Continuous Restores
Validation is important
Estimates / Expectations
Procedure / External factors
Does it even work?
Development Database
Routine refresh + restore testing
Reporting Databases:
Overnight restores for reporting databases refreshed daily. Great
candidate for daily validation.
16
RTO
17. Sample Validation Log
2015-03-01 10:00:03 : Testing backup for 2015-02-28 from
/data/backup/hot_backup/+/var/log/test/bin/test_backups.sql
2015-03-01 10:00:03 : Starting decompress of
/data/backup/hot_backup/test.local-data-2015-02-28.tar.gz
2015-03-01 10:00:03 : Starting decompress of
/data/backup/hot_backup/test.local-xlog-2015-02-28.tar.gz
2015-03-01 10:00:03 : Waiting for both decompressing processes to finish
2015-03-01 14:36:06 : Decompressing worked, generated directory test with
size: 963G
2015-03-01 14:36:06 : Starting PostgreSQL
server starting
2015-03-01 14:36:07.372 EST @ 3282 LOG: loaded library
"pg_scoreboard.so"
2015-03-01 15:52:36 : PostgreSQL started
2015-03-01 15:52:36 : Validating Database
starting Vacuum
2015-03-02 08:17:56 : Test result: OK
2015-03-02 08:18:11 : All OK.
17
18. Where did Gitlab go wrong?
18
No PITR, no alerts for missing backups, no documentation for backup
location, no restore testing
“…backups…taken once per 24 hours…not yet been able to figure
out where they are stored…don’t appear to be working...”
No retention policy
“Fog gem may have cleaned out older backups”
No monitoring
“Our backups to S3 apparently don’t work either: the bucket is
empty”
No RPO
“…Unless we can pull these from the past 24 hours they will be
lost”
19. Sobering Thought of the Day
19
A chain is only as strong as the weakest link
21. Retention Period
Remove all older than x days
VS
Remove all but latest x backups
Off-server (short term)
VS
Off-site retention period (long term)
21
22. Security
Transfer
One way passwordless SSH access
HTTPS for cloud uploads (e.g. s3cmd)
Storage
Encryption
Access control
PCI Compliance
Private keys
Logical backups preferred
Multi-Tenancy Environment
Backup Server Client
Client Backup server
22
23. Monitoring and Alerts
Alert at runtime
• To detect errors at runtime within script, or change in system/user
environments
• Immediate
• Cron emails
Alert on storage/retention:
• Error in retention logic/implementation
• Secondary/Delayed
• Cron’d script, external monitoring
Alert on backup validation:
• On routine refreshes/restores
23
24. S3 Backups
Ideal for long term backups:
• Cheap
• Ample space
• Transfer in parts, can pick up where you left off in case of errors
• Speed depends on bandwidth
Secure:
• Buckets , Users, Policies
Communication:
• AWS cli / s3cmd
• Access keys
• PGP encryption
24
28. Why Automate?
Reduced Chance of Human Error
pg_dump backups
filesystem level backups
retention scripts
backup monitoring and alerts
off-site and off-server transfer as well as storage
access channels
security keys
cronjobs…
Less work, faster
Move scripts, swap crontabs, ensure all accesses exist
Uniformity, Reliability
28
33. Can Filesystems enhance DR?
33
ZFS
• Available on Solaris, Illumos. ZFS on Linux is new but improving!
• Built in compression
• Protections against data corruption
• Snapshots
• Copy-on-Write
34. How can ZFS help you with DR?
34
Scenario 1: You’re using pg_upgrade for a
major upgrade with hard link option (-k). It fails
in between. Since you used hardlinks, you
cannot turn on the older cluster. What do you
do?
ZFS Rollback – instantaneously rollback a
dataset to a previous state from a snapshot
sudo zfs rollback <snapshot_name>
35. How can ZFS help you with DR?
35
Scenario 2: You’ve accidentally deleted a large
table in a very large database where taking full
backups are infeasible. Is there a faster
alternative to a filesystem restore? ZFS rollback
will overwrite pgdata so you cannot use that.
ZFS Clone – Copy-on-Write. Instantaneously
clone a dataset from snapshot without any
additional space consumptions as long as you
don’t write to it.
sudo zfs clone <snapshot_name> <mountpoint>
37. OmniPITR
37
PITR backups and archives
Backups off of master or standby
Built in backup integrity checks
Quick setup, Minimal requirements
Comprehensive logging
Built in retention
Ideal for infrastructures of all sizes
38. Barman
38
PITR backups and restores
Automatic retention, continuous recovery
Requires its own server
More suitable for larger infrastructures to manage multiple clusters and
multiple locations
Simplicity
Minimal knowledge of PITR inner workings required
Wrappers for recovery process
Supports backups from both primary and standby
39. wal-e
39
PITR backups and restores
Automatic retention, continuous recovery
Minimal setup
Cloud integration:
AWS, Azure, Google, Swift
44. Redundancy is Good
“That Sunday, Thomas deleted remotely stored backups and turned off the automated
backup system. He made some changes to VPN authentication that basically locked
everybody out, and turned off the automatic restart. He deleted internal IT wiki pages,
removed users from a mailing list, deactivated the company's pager notification system,
and a number of other things that basically created a huge mess that the company spent
the whole of Monday sorting out (it turned out there were local copies of the deleted
backups).”
https://www.theregister.co.uk/2017/02/23/michael_thomas_appeals_conviction/
44
45. S3 Backups Monitoring
Sample:
# Get timestamp of latest file in S3 bucket
latest_backup=$(s3cmd ls s3://omniti_client/ | awk {'print $1'} | sort -n | tail -1)
today=$(date --date=today "+%Y-%m-%d")
yesterday=$(date --date=yesterday "+%Y-%m-%d")
# If latest backup is older than yesterday, email dba
if [ $yesterday -ne $latest_backup -a $today -ne $latest_backup ]
then
echo "Offsite Backups Missing for client" | mailx -s "Offsite Backup Missing
for Client " dba@omniti.com
45
46. Backup Documentation
• Detailed backup information - types, locations, retention periods
• Procedures to setup new machines
• Analysis and estimation time for recovery
• Ways to recover
46
