This document discusses MySQL backup solutions and focuses on Percona's XtraBackup tool. It provides an overview of challenges with MySQL backups, available solutions, and key features of XtraBackup. The document demonstrates how to perform full and incremental backups with XtraBackup and provides examples of options like streaming, filtering tablespaces, and parallel copying. It also notes differences between the xtrabackup binary and innobackupex script and highlights new features in XtraBackup version 2.0.
Mydumper - Vinoth kanna @ MySQL meetup MumbaiKarthik .P.R
Mydumper is a parallel mysqldump tool maintained and developed by Percona. Vinoth Kanna has taken this presentation in the MySQL meet up uploading this on behalf of him.
Highly efficient backups with percona xtrabackupNilnandan Joshi
Percona XtraBackup is an open source, free MySQL hot backup software that performs non-blocking backups for InnoDB and XtraDB databases. In this talk we'll describe below things.
- How it works with MySQL/Percona Server and what are the features provided
- Difference between Xtrabackup and Innobackupex
- How to take full/increment/partial backup and restore
- How to use features like streaming, compression, remote and compact backups
- How to troubleshoot the issue with xtrabackup
Mydumper - Vinoth kanna @ MySQL meetup MumbaiKarthik .P.R
Mydumper is a parallel mysqldump tool maintained and developed by Percona. Vinoth Kanna has taken this presentation in the MySQL meet up uploading this on behalf of him.
Highly efficient backups with percona xtrabackupNilnandan Joshi
Percona XtraBackup is an open source, free MySQL hot backup software that performs non-blocking backups for InnoDB and XtraDB databases. In this talk we'll describe below things.
- How it works with MySQL/Percona Server and what are the features provided
- Difference between Xtrabackup and Innobackupex
- How to take full/increment/partial backup and restore
- How to use features like streaming, compression, remote and compact backups
- How to troubleshoot the issue with xtrabackup
Operation Unthinkable – Software Defined Storage @ Booking.com (Peter Buschman)data://disrupted®
The story of the plan that was just crazy enough to work! Learn how Booking.com failed its way to success on a multi-year journey away from single-purpose storage-appliances, predatory-licensing, and over-complicated networking to create a unique storage solution for their hyper-scale private-cloud environment.
Red Hat Enterprise Linux: Open, hyperconverged infrastructureRed_Hat_Storage
The next generation of IT will be built around flexible infrastructures and operational efficiencies, lowering costs and increasing overall business value in the organization.
A hyperconverged infrastructure that's built on Red Hat supported technologies--including Linux, Gluster storage, and oVirt virtualization manager--will run on commodity x86 servers using the performance of local storage, to deliver a cost-effective, modular, highly scalable, and secure hyperconverged solution.
TokuDB is an ACID/transactional storage engine that makes MySQL even better by increasing performance, adding high compression, and allowing for true schema agility. All of these features are made possible by Tokutek's Fractal Tree indexes.
"Data classification" is an umbrella term covering things: locality-aware data placement, SSD/disk or normal/deduplicated/erasure-coded data tiering, HSM, etc. They share most of the same infrastructure, and so are proposed (for now) as a single feature.
Server as the core of the information systems in today's business operations playing a pivotal role, data storage on server is becoming the lifeline to maintain the normal operation of the millions of businesses and organizations. Correct choice of independent control of the server or cluster systems and storage devices can improve system performance and operational capabilities. The user request rate and response time of business server data storage system are dependent on the selection of the storage devices. These devices can greatly affect system performance. To effectively protect data and give linear performance for iSCSI initiator, it is required to modify the iSCSI target driver to provide Gluster file system backend storage as distributed block device. This paper aims at discussing ways of modifying iSCSI Target (ISTGT) driver to support Gluster file system.
Operation Unthinkable – Software Defined Storage @ Booking.com (Peter Buschman)data://disrupted®
The story of the plan that was just crazy enough to work! Learn how Booking.com failed its way to success on a multi-year journey away from single-purpose storage-appliances, predatory-licensing, and over-complicated networking to create a unique storage solution for their hyper-scale private-cloud environment.
Red Hat Enterprise Linux: Open, hyperconverged infrastructureRed_Hat_Storage
The next generation of IT will be built around flexible infrastructures and operational efficiencies, lowering costs and increasing overall business value in the organization.
A hyperconverged infrastructure that's built on Red Hat supported technologies--including Linux, Gluster storage, and oVirt virtualization manager--will run on commodity x86 servers using the performance of local storage, to deliver a cost-effective, modular, highly scalable, and secure hyperconverged solution.
TokuDB is an ACID/transactional storage engine that makes MySQL even better by increasing performance, adding high compression, and allowing for true schema agility. All of these features are made possible by Tokutek's Fractal Tree indexes.
"Data classification" is an umbrella term covering things: locality-aware data placement, SSD/disk or normal/deduplicated/erasure-coded data tiering, HSM, etc. They share most of the same infrastructure, and so are proposed (for now) as a single feature.
Server as the core of the information systems in today's business operations playing a pivotal role, data storage on server is becoming the lifeline to maintain the normal operation of the millions of businesses and organizations. Correct choice of independent control of the server or cluster systems and storage devices can improve system performance and operational capabilities. The user request rate and response time of business server data storage system are dependent on the selection of the storage devices. These devices can greatly affect system performance. To effectively protect data and give linear performance for iSCSI initiator, it is required to modify the iSCSI target driver to provide Gluster file system backend storage as distributed block device. This paper aims at discussing ways of modifying iSCSI Target (ISTGT) driver to support Gluster file system.
PLAM 2015 - Evolving Backups Strategy, Devploying pyxbackupJervin Real
Not all backup requirements are the same, there are simple backups and then there are complex backup and recovery strategies. After working with hundreds of users laying out the ground work, we've decided that although not all installations are the same, all of them run through the same procedure so we have decided to automate most of this things with pyxbackup. This talk will highlight the evolution of the project from the most basic backup implementation to extending capabilities for different scenarios like compression, remote storage, encryption and being a portable, comprehensible backup utility for the novice and experts alike.
The pyxbackup project can be found at:
https://github.com/dotmanila/pyxbackup
Fast Incremental Backups with Percona Server and Percona XtraBackup / PLMCE 2014Laurynas Biveinis
Percona Live 2014 presentation
https://www.percona.com/live/mysql-conference-2014/sessions/fast-incremental-backups-percona-server-and-percona-xtrabackup
in LATVIAN language: Viens no galvenajiem datubāzes administratora uzdevumiem ir veikt datubāzes backup un prast no tā atjaunot datubāzi. Mysql bezmaksas versija nepiedāvā datubāzes administratoram ļoti daudz izvēles. Sava prezentācija es pastāstīšu par šādiem rīkiem:
-- MySQLdump
-- Percona XtraBackup
-- Mysql enterprise backup (MEB)
--Un citiem rīkiem kas palīdz man veikt db backup
"Disaster is inevitable" and "To move forward you must first backup" should be known to all software developers. This presentation will discuss all the options for your valuable data assets in MySQL, and highlight how to maintain site reliability of your data
Online MySQL Backups with Percona XtraBackupKenny Gryp
Percona XtraBackup is a free, open source, complete online backup solution for all versions of Percona Server, MySQL® and MariaDB®.
Percona XtraBackup provides:
* Fast and reliable backups
* Uninterrupted transaction processing during backups
* Savings on disk space and network bandwidth with better compression
* Automatic backup verification
* Higher uptime due to faster restore time
This talk will discuss the various different features of Percona XtraBackup, including:
* Full & Incremental Backups
* Compression, Streaming & Encryption of Backups
* Backing Up To The Cloud (Swift).
* Percona XtraDB Cluster / Galera Cluster.
* Percona Server Specific features
MySQL backups overview. Characteristics of every backup type, including dumps, Xtrabackup and snapshots. Planning proper backup strategies. Why and how to test backups.
Slides presented at Percona Live Europe Open Source Database Conference 2019, Amsterdam, 2019-10-01.
Imagine a world where all Wikipedia articles disappear due to a human error or software bug. Sounds unreal? According to some estimations, it would take an excess of hundreds of million person-hours to be written again. To prevent that scenario from ever happening, our SRE team at Wikimedia recently refactored the relational database recovery system.
In this session, we will discuss how we backup 550TB of MariaDB data without impacting the 15 billion page views per month we get. We will cover what were our initial plans to replace the old infrastructure, how we achieved recovering 2TB databases in less than 30 minutes while maintaining per-table granularity, as well as the different types of backups we implemented. Lastly, we will talk about lessons learned, what went well, how our original plans changed and future work.
OSDC 2012 | Taking hot backups with XtraBackup by Alexey KopytovNETWAYS
Percona XtraBackup is an open source hot backup tool for MySQL. This talk will cover basic operations such as taking full and incremental backups, restoring from backups and setting up replication slaves, as well as advanced and recently added features such as streaming and compressed backups, parallel operations and partial backups.
Percona XtraBackup - New Features and ImprovementsMarcelo Altmann
Percona XtraBackup is an open-source hot backup utility for MySQL - based servers that doesn't lock your database during the backup. In this talk, we will cover the latest development and new features introduced on Xtrabackup and its auxiliary tools: - Page Tracking - Azure Blob Storage Support - Exponential Backoff - Keyring Components - and more.
Backup and restore methods are concepts that everyone knows the importance of. Over the years, open-source tools emerged like MyDumper, Xtrabackup, and Mariabackup. Also, with MySQL 8 new shell, new utils for dump and restore were introduced as well.
In this presentation, we are going to compare the newest backup/restore methods with the most used ones. We will see how parallelization can influence the speed of backup and restore process and also how the compression algorithms can influence the performance.
In this talk, we will compare mysqldump, mydumper/myloader, mysqlpump, MySQL Shell utils, and Xtrabackup.
Become a MySQL DBA - slides: Deciding on a relevant backup solutionSeveralnines
Backup and restore is one of the most important aspects of database administration. If a database crashed and there was no way to recover it, any resulting data loss might lead to devastating results to a business. As the DBA operating a MySQL or Galera cluster in production, you need to ensure your backups are scheduled, executed and regularly tested.
There are multiple ways to take backups, but which method fits your specific needs? How do I implement point in time recovery?
Watch this session on backup strategies for MySQL and Galera clusters led by Krzysztof Książek, Senior Support Engineer at Severalnines.
AGENDA
Logical and Physical Backup methods
Tools
mysqldump
mydumper
xtrabackup
snapshots
How backups are done in ClusterControl
Best practices
Example Setups
On premises / private datacenter
Amazon Web Services
SPEAKER
Krzysztof Książek, Senior Support Engineer at Severalnines, is a MySQL DBA with experience in managing complex database environments for companies like Zendesk, Chegg, Pinterest and Flipboard.
This webinar ‘Become a MySQL DBA - deciding on a relevant backup solution’ will show you the pros and cons of different backup options and help you pick one that goes well with your environment.
Lock, Stock and Backup: Data GuaranteedJervin Real
Percona Live 2017 - the decisions you need to make, the tools we recommend, the process you need to consider for a successful backup implementation for your MySQL services.
A Backup Today Saves Tomorrow is a presentation from Percona Live 2013 that provides insight into planning and the tools used today to capture MySQL backups.
What if …
- Traditional, labour-intensive backup and archive practices for your MySQL, MariaDB, MongoDB and PostgreSQL databases were a thing of the past?
- You could have one backup management solution for all your business data?
- You could ensure integrity of all your backups?
- You could leverage the competitive pricing and almost limitless capacity of cloud-based backup while meeting cost, manageability, and compliance requirements from the business.
Welcome to our webinar on Backup Management with ClusterControl.
ClusterControl’s centralized backup management for open source databases provides you with hot backups of large datasets, point in time recovery in a couple of clicks, at-rest and in-transit data encryption, data integrity via automatic restore verification, cloud backups (AWS, Google and Azure) for Disaster Recovery, retention policies to ensure compliance, and automated alerts and reporting.
Whether you are looking at rebuilding your existing backup infrastructure, or updating it, this webinar is for you!
AGENDA
- Backup and recovery management of local or remote databases
- Logical or physical backups
- Full or Incremental backups
- Position or time-based Point in Time Recovery (for MySQL and PostgreSQL)
- Upload to the cloud (Amazon S3, Google Cloud Storage, Azure Storage)
- Encryption of backup data
- Compression of backup data
- One centralized backup system for your open source databases (Demo)
- Schedule, manage and operate backups
- Define backup policies, retention, history
- Validation - Automatic restore verification
- Backup reporting
SPEAKER
Bartlomiej Oles, Senior Support Engineer at Severalnines, is a MySQL and Oracle DBA, with over 15 years experience in managing highly available production systems at IBM, Nordea Bank, Acxiom, Lufthansa, and other Fortune 500 companies. In the past five years, his focus has been on building and applying automation tools to manage multi-datacenter database environments.
Data deduplication is a hot topic in storage and saves significant disk space for many environments, with some trade offs. We’ll discuss what deduplication is and where the Open Source solutions are versus commercial offerings. Presentation will lean towards the practical – where attendees can use it in their real world projects (what works, what doesn’t, should you use in production, etcetera).
The Proper Care and Feeding of MySQL DatabasesDave Stokes
Many Linux System Administrators are 'also' accidental database administrators. This is a guide for them to keep their MySQL database instances happy, health, and glowing
DevOps and Testing slides at DASA ConnectKari Kakkonen
My and Rik Marselis slides at 30.5.2024 DASA Connect conference. We discuss about what is testing, then what is agile testing and finally what is Testing in DevOps. Finally we had lovely workshop with the participants trying to find out different ways to think about quality and testing in different parts of the DevOps infinity loop.
Elevating Tactical DDD Patterns Through Object CalisthenicsDorra BARTAGUIZ
After immersing yourself in the blue book and its red counterpart, attending DDD-focused conferences, and applying tactical patterns, you're left with a crucial question: How do I ensure my design is effective? Tactical patterns within Domain-Driven Design (DDD) serve as guiding principles for creating clear and manageable domain models. However, achieving success with these patterns requires additional guidance. Interestingly, we've observed that a set of constraints initially designed for training purposes remarkably aligns with effective pattern implementation, offering a more ‘mechanical’ approach. Let's explore together how Object Calisthenics can elevate the design of your tactical DDD patterns, offering concrete help for those venturing into DDD for the first time!
LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...DanBrown980551
Do you want to learn how to model and simulate an electrical network from scratch in under an hour?
Then welcome to this PowSyBl workshop, hosted by Rte, the French Transmission System Operator (TSO)!
During the webinar, you will discover the PowSyBl ecosystem as well as handle and study an electrical network through an interactive Python notebook.
PowSyBl is an open source project hosted by LF Energy, which offers a comprehensive set of features for electrical grid modelling and simulation. Among other advanced features, PowSyBl provides:
- A fully editable and extendable library for grid component modelling;
- Visualization tools to display your network;
- Grid simulation tools, such as power flows, security analyses (with or without remedial actions) and sensitivity analyses;
The framework is mostly written in Java, with a Python binding so that Python developers can access PowSyBl functionalities as well.
What you will learn during the webinar:
- For beginners: discover PowSyBl's functionalities through a quick general presentation and the notebook, without needing any expert coding skills;
- For advanced developers: master the skills to efficiently apply PowSyBl functionalities to your real-world scenarios.
Generating a custom Ruby SDK for your web service or Rails API using Smithyg2nightmarescribd
Have you ever wanted a Ruby client API to communicate with your web service? Smithy is a protocol-agnostic language for defining services and SDKs. Smithy Ruby is an implementation of Smithy that generates a Ruby SDK using a Smithy model. In this talk, we will explore Smithy and Smithy Ruby to learn how to generate custom feature-rich SDKs that can communicate with any web service, such as a Rails JSON API.
Essentials of Automations: Optimizing FME Workflows with ParametersSafe Software
Are you looking to streamline your workflows and boost your projects’ efficiency? Do you find yourself searching for ways to add flexibility and control over your FME workflows? If so, you’re in the right place.
Join us for an insightful dive into the world of FME parameters, a critical element in optimizing workflow efficiency. This webinar marks the beginning of our three-part “Essentials of Automation” series. This first webinar is designed to equip you with the knowledge and skills to utilize parameters effectively: enhancing the flexibility, maintainability, and user control of your FME projects.
Here’s what you’ll gain:
- Essentials of FME Parameters: Understand the pivotal role of parameters, including Reader/Writer, Transformer, User, and FME Flow categories. Discover how they are the key to unlocking automation and optimization within your workflows.
- Practical Applications in FME Form: Delve into key user parameter types including choice, connections, and file URLs. Allow users to control how a workflow runs, making your workflows more reusable. Learn to import values and deliver the best user experience for your workflows while enhancing accuracy.
- Optimization Strategies in FME Flow: Explore the creation and strategic deployment of parameters in FME Flow, including the use of deployment and geometry parameters, to maximize workflow efficiency.
- Pro Tips for Success: Gain insights on parameterizing connections and leveraging new features like Conditional Visibility for clarity and simplicity.
We’ll wrap up with a glimpse into future webinars, followed by a Q&A session to address your specific questions surrounding this topic.
Don’t miss this opportunity to elevate your FME expertise and drive your projects to new heights of efficiency.
Securing your Kubernetes cluster_ a step-by-step guide to success !KatiaHIMEUR1
Today, after several years of existence, an extremely active community and an ultra-dynamic ecosystem, Kubernetes has established itself as the de facto standard in container orchestration. Thanks to a wide range of managed services, it has never been so easy to set up a ready-to-use Kubernetes cluster.
However, this ease of use means that the subject of security in Kubernetes is often left for later, or even neglected. This exposes companies to significant risks.
In this talk, I'll show you step-by-step how to secure your Kubernetes cluster for greater peace of mind and reliability.
Slack (or Teams) Automation for Bonterra Impact Management (fka Social Soluti...Jeffrey Haguewood
Sidekick Solutions uses Bonterra Impact Management (fka Social Solutions Apricot) and automation solutions to integrate data for business workflows.
We believe integration and automation are essential to user experience and the promise of efficient work through technology. Automation is the critical ingredient to realizing that full vision. We develop integration products and services for Bonterra Case Management software to support the deployment of automations for a variety of use cases.
This video focuses on the notifications, alerts, and approval requests using Slack for Bonterra Impact Management. The solutions covered in this webinar can also be deployed for Microsoft Teams.
Interested in deploying notification automations for Bonterra Impact Management? Contact us at sales@sidekicksolutionsllc.com to discuss next steps.
GraphRAG is All You need? LLM & Knowledge GraphGuy Korland
Guy Korland, CEO and Co-founder of FalkorDB, will review two articles on the integration of language models with knowledge graphs.
1. Unifying Large Language Models and Knowledge Graphs: A Roadmap.
https://arxiv.org/abs/2306.08302
2. Microsoft Research's GraphRAG paper and a review paper on various uses of knowledge graphs:
https://www.microsoft.com/en-us/research/blog/graphrag-unlocking-llm-discovery-on-narrative-private-data/
2. Percona
MySQL/LAMP Consulting
● MySQL Support
● (co-)Developers of
○ Percona Server (XtraDB)
○ Percona XtraBackup
○ Percona Toolkit (Maatkit, Aspersa)
○ Percona XtraDB Cluster
http://www.percona.com
http://www.mysqlperformanceblog.com
3. Topics
● Why MySQL Backup is hard to get right
● What's available on the market?
● Percona's Solution: XtraBackup
● Demo
4. MySQL Backup Challenges
● MySQL is multi-engine
○ Different storage engines call for different strategies
■ Are we using filecopy or dumps?
■ MyISAM tables handle filecopy quite well
■ InnoDB files cannot simply be copied elsewhere
● MySQL can be quite busy
○ Do we need to shut it down?
○ If not, how much load is generated?
○ How quickly can we restore?
● MySQL is more than the sum of its datafiles
○ How about binary logging, replication?
○ Does the tool impact MySQL in other ways?
5. Available backup solutions (InnoDB)
Backup Restore
InnoDB Impact Warmth
Time Time
Cold Backup very high cold very fast fast
mysqldump medium hot medium slow
snapshotting high/medium hot/warm fast fast
MySQL EB low hot fast fast
(Enterprise Backup)
XtraBackup low hot fast fast
6. Available backup solutions (mixed)
InnoDB/ Backup Restore
Impact Warmth
MyISAM Time Time
Cold Backup very high cold very fast fast
mysqldump high warm medium slow
snapshotting high/medium hot/warm fast fast
MySQL EB low/medium warm fast fast
(Enterprise Backup)
XtraBackup low/medium warm fast fast
7. XtraBackup Features
● Shared with MySQL Enterprise Backup
○ Non-blocking
○ Support for MyISAM
○ Compression
○ Partial Backups
○ Throttling
○ Incremental Backups
● Unique to XtraBackup
○ Exporting/Importing individual tables
○ Streaming (including incremental streaming since 2.0!)
○ Parallel copying (with use of innodb_file_per_table)
http://www.percona.com/doc/percona-xtrabackup/intro.html
9. XtraBackup/innobackupex Operation
Performing and restoring a mixed full backup:
Backing up (1):
1. Copy datafiles while recording transaction log changes
2. FLUSH TABLES WITH READ LOCK;
3. Get binlog position
4. Copy all .MYD, .MYI, .TRG, .TRN, ... files
5. Stop recording transaction log changes
6. UNLOCK TABLES;
Preparing/Restoring:
7. Prepare backup by applying recorded tlog changes (2)
8. Restoring files to original location (3)
10. Getting down to business...
innobackupex vs. xtrabackup?
xtrabackup - Compiled C application, focuses on InnoDB
only
innobackupex - Perl script, automates surrounding tasks
3 separate "manual" steps are required:
(1) - Backing up
(2) - Preparing InnoDB files for recovery
(3) - Restoring the files
All of these are easily scriptable!
11. Some recipes
Simple full backup and restore:
(1) # innobackupex /path/to/your/backups
(2) # innobackupex --apply-log /path/to/your/backups/2012-06-
21_13-45-00/
(3) # innobackupex --copy-back /path/to/your/backups/2012-06-
21_13-45-00/
# chown -R mysql:mysql /var/lib/mysql
12. Options
● Specified as --variable-name=value
● innobackupex/xtrabackup will read options from your config
○ [client] or [mysql] for credentials
○ [mysqld] for location of datadir, InnoDB startup options
○ Lastly, [xtrabackup] to override any of the previous and
specify xtrabackup-specific options*
* percona.com/doc/percona-xtrabackup/xtrabackup_bin/xbk_option_reference.html
* percona.com/doc/percona-xtrabackup/innobackupex/innobackupex_option_reference.html
13. Some recipes
Incremental backup:
● Repeat (1) to create a full backup at location $fullbackup
● # innobackupex --incremental $incbackup --incremental-
basedir=$fullbackup
● # innobackupex --apply-log --redo-only $fullbackup
■ Preparing the base backup FIRST
● # innobackupex --apply-log --redo-only $fullbackup --incremental-
basedir=$incbackup
■ Applying each incremental backup to the base
● # innobackupex --apply-log $fullbackup
■ Finish preparing the now completed base backup
14. Some recipes
Streaming:
# innobackupex --stream=tar ./ | ssh user@desthost "cat - > /data/backups/backup.
tar"
Note: Extract with tar -i!
Filtering tablespaces: --tables, --databases, --tables-file
Throttling:
● Limit to 100 read/writes IOs: --throttle=100
● In streaming backup, use pv to limit to 10MB/s
○ # innobackupex --stream=tar ./ | pv -q -L10m | ssh user@desthost "cat - >
/data/backups/backup.tar"
Parallel copying of tablespaces: --parallel 2 #threads
15. A note about versions
● xtrabackup 1.6
● xtrabackup 2.0 (GA since 4 April 2012)
○ Backs up galera cluster information
○ Supports parallel compression
○ Streaming incremental backups
○ Backs up LRU dumpfile
Note: When using the xtrabackup binary directly, make sure to
use the correct binary for your MySQL version!