This document provides an introduction and quickstart guide for MyAWR, a tool for collecting and analyzing performance data for MySQL databases. It describes what MyAWR is, its data model involving multiple tables, and how to install it along with its dependencies. Instructions are provided on initializing the MyAWR host table, adding jobs to crontab to periodically collect data and analyze slow logs, and using the MyAWR reporting tool to generate HTML reports on the collected performance metrics.
In this session we will discuss selected areas of InnoDB and XtraDB 5.7 internals that are mostly related to buffer pool management, flushing, and the doublewrite buffer, from a performance and scalability point of view.
PGDay.Amsterdam 2018 - Stefan Fercot - Save your data with pgBackRestPGDay.Amsterdam
Ever heard of Point-in-time recovery? pgBackRest is an awsome tool to handle backups, restores and even helps you build streaming replication ! This talk will introduce the tool, its basic features and how to use it.
In this session we will cover wide area replica sets and using tags for backup. Attendees should be well versed in basic replication and familiar with concepts in the morning's basic replication talk. No beginner topics will be covered in this session
MySQL Replication: Demo Réplica en EspañolKeith Hollman
MySQL Replication technical example in Spanish.
Ejemplo técnico de réplica de MySQL en Español.
Es una guía muy rápida para quitar el miedo a empezar a jugar con réplica.
In this session we will discuss selected areas of InnoDB and XtraDB 5.7 internals that are mostly related to buffer pool management, flushing, and the doublewrite buffer, from a performance and scalability point of view.
PGDay.Amsterdam 2018 - Stefan Fercot - Save your data with pgBackRestPGDay.Amsterdam
Ever heard of Point-in-time recovery? pgBackRest is an awsome tool to handle backups, restores and even helps you build streaming replication ! This talk will introduce the tool, its basic features and how to use it.
In this session we will cover wide area replica sets and using tags for backup. Attendees should be well versed in basic replication and familiar with concepts in the morning's basic replication talk. No beginner topics will be covered in this session
MySQL Replication: Demo Réplica en EspañolKeith Hollman
MySQL Replication technical example in Spanish.
Ejemplo técnico de réplica de MySQL en Español.
Es una guía muy rápida para quitar el miedo a empezar a jugar con réplica.
Learn how upcoming changes in the persistent memory market will affect deployments of in-memory computing and traditional applications. Using software innovations from SanDisk and the broad portfolio of flash storage hardware options, customers and developers can optimize applications for “flash extended memory”, the intersection of in-memory computing and persistent memory technologies.
Rapid Application Design in Financial ServicesAerospike
Applying internet NoSQL design patterns to fraud detection and risk scoring, including when to use SQL and when to use NoSQL. The state of NAND Flash and NVMe is also discussed, as well as storage class memory futures with Intel's 3D Xpoint technology.
This talk was presented in LA at the following meetup:
http://www.meetup.com/scalela/events/233396111/
Esta oferta podría ser la solución para el punto de partida de Flash que junto con Spectrum Scale te da una solución de Software Defined Storage escalable, para cumplir con los requisitos del almacenamiento no estructurado y big data.
Director of Data Center Solutions Marketing, Brian Allison, breaks down the impact of Big Data Flash in this presentation during Flash Memory Summit 2016
Flash memory summit 2015 gary lyng session 301-aGARY LYNG
The World is Ready for Big Data Flash - these workloads, enterprise's at scale and hyperscalers have already discovered the real world benefits of implementing Flash in the data center for high performance and high capacity workloads and @Scale the incredible bottom line savings while raising the SLA's of apps. and the business they support.
ProxySQL and the Tricks Up Its Sleeve - Percona Live 2022.pdfJesmar Cannao'
ProxySQL is well-affirmed into thousands of production environments for the features we all know: multiplexing, query routing, and rewriting to name a few.
Let's go through those use cases which maybe are the least common: from keeping malicious eyes away from your production data to rebuilding your non-production environment, from stopping having hanging transactions to monitor your instance, from query firewalling to changing ProxySQL configuration without a single line of SQL!
When the sky is the limit, ProxySQL gives you some boost!
Elevating Tactical DDD Patterns Through Object CalisthenicsDorra BARTAGUIZ
After immersing yourself in the blue book and its red counterpart, attending DDD-focused conferences, and applying tactical patterns, you're left with a crucial question: How do I ensure my design is effective? Tactical patterns within Domain-Driven Design (DDD) serve as guiding principles for creating clear and manageable domain models. However, achieving success with these patterns requires additional guidance. Interestingly, we've observed that a set of constraints initially designed for training purposes remarkably aligns with effective pattern implementation, offering a more ‘mechanical’ approach. Let's explore together how Object Calisthenics can elevate the design of your tactical DDD patterns, offering concrete help for those venturing into DDD for the first time!
LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...DanBrown980551
Do you want to learn how to model and simulate an electrical network from scratch in under an hour?
Then welcome to this PowSyBl workshop, hosted by Rte, the French Transmission System Operator (TSO)!
During the webinar, you will discover the PowSyBl ecosystem as well as handle and study an electrical network through an interactive Python notebook.
PowSyBl is an open source project hosted by LF Energy, which offers a comprehensive set of features for electrical grid modelling and simulation. Among other advanced features, PowSyBl provides:
- A fully editable and extendable library for grid component modelling;
- Visualization tools to display your network;
- Grid simulation tools, such as power flows, security analyses (with or without remedial actions) and sensitivity analyses;
The framework is mostly written in Java, with a Python binding so that Python developers can access PowSyBl functionalities as well.
What you will learn during the webinar:
- For beginners: discover PowSyBl's functionalities through a quick general presentation and the notebook, without needing any expert coding skills;
- For advanced developers: master the skills to efficiently apply PowSyBl functionalities to your real-world scenarios.
Securing your Kubernetes cluster_ a step-by-step guide to success !KatiaHIMEUR1
Today, after several years of existence, an extremely active community and an ultra-dynamic ecosystem, Kubernetes has established itself as the de facto standard in container orchestration. Thanks to a wide range of managed services, it has never been so easy to set up a ready-to-use Kubernetes cluster.
However, this ease of use means that the subject of security in Kubernetes is often left for later, or even neglected. This exposes companies to significant risks.
In this talk, I'll show you step-by-step how to secure your Kubernetes cluster for greater peace of mind and reliability.
Generative AI Deep Dive: Advancing from Proof of Concept to ProductionAggregage
Join Maher Hanafi, VP of Engineering at Betterworks, in this new session where he'll share a practical framework to transform Gen AI prototypes into impactful products! He'll delve into the complexities of data collection and management, model selection and optimization, and ensuring security, scalability, and responsible use.
SAP Sapphire 2024 - ASUG301 building better apps with SAP Fiori.pdfPeter Spielvogel
Building better applications for business users with SAP Fiori.
• What is SAP Fiori and why it matters to you
• How a better user experience drives measurable business benefits
• How to get started with SAP Fiori today
• How SAP Fiori elements accelerates application development
• How SAP Build Code includes SAP Fiori tools and other generative artificial intelligence capabilities
• How SAP Fiori paves the way for using AI in SAP apps
Alt. GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using ...James Anderson
Effective Application Security in Software Delivery lifecycle using Deployment Firewall and DBOM
The modern software delivery process (or the CI/CD process) includes many tools, distributed teams, open-source code, and cloud platforms. Constant focus on speed to release software to market, along with the traditional slow and manual security checks has caused gaps in continuous security as an important piece in the software supply chain. Today organizations feel more susceptible to external and internal cyber threats due to the vast attack surface in their applications supply chain and the lack of end-to-end governance and risk management.
The software team must secure its software delivery process to avoid vulnerability and security breaches. This needs to be achieved with existing tool chains and without extensive rework of the delivery processes. This talk will present strategies and techniques for providing visibility into the true risk of the existing vulnerabilities, preventing the introduction of security issues in the software, resolving vulnerabilities in production environments quickly, and capturing the deployment bill of materials (DBOM).
Speakers:
Bob Boule
Robert Boule is a technology enthusiast with PASSION for technology and making things work along with a knack for helping others understand how things work. He comes with around 20 years of solution engineering experience in application security, software continuous delivery, and SaaS platforms. He is known for his dynamic presentations in CI/CD and application security integrated in software delivery lifecycle.
Gopinath Rebala
Gopinath Rebala is the CTO of OpsMx, where he has overall responsibility for the machine learning and data processing architectures for Secure Software Delivery. Gopi also has a strong connection with our customers, leading design and architecture for strategic implementations. Gopi is a frequent speaker and well-known leader in continuous delivery and integrating security into software delivery.
PHP Frameworks: I want to break free (IPC Berlin 2024)Ralf Eggert
In this presentation, we examine the challenges and limitations of relying too heavily on PHP frameworks in web development. We discuss the history of PHP and its frameworks to understand how this dependence has evolved. The focus will be on providing concrete tips and strategies to reduce reliance on these frameworks, based on real-world examples and practical considerations. The goal is to equip developers with the skills and knowledge to create more flexible and future-proof web applications. We'll explore the importance of maintaining autonomy in a rapidly changing tech landscape and how to make informed decisions in PHP development.
This talk is aimed at encouraging a more independent approach to using PHP frameworks, moving towards a more flexible and future-proof approach to PHP development.
GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using Deplo...James Anderson
Effective Application Security in Software Delivery lifecycle using Deployment Firewall and DBOM
The modern software delivery process (or the CI/CD process) includes many tools, distributed teams, open-source code, and cloud platforms. Constant focus on speed to release software to market, along with the traditional slow and manual security checks has caused gaps in continuous security as an important piece in the software supply chain. Today organizations feel more susceptible to external and internal cyber threats due to the vast attack surface in their applications supply chain and the lack of end-to-end governance and risk management.
The software team must secure its software delivery process to avoid vulnerability and security breaches. This needs to be achieved with existing tool chains and without extensive rework of the delivery processes. This talk will present strategies and techniques for providing visibility into the true risk of the existing vulnerabilities, preventing the introduction of security issues in the software, resolving vulnerabilities in production environments quickly, and capturing the deployment bill of materials (DBOM).
Speakers:
Bob Boule
Robert Boule is a technology enthusiast with PASSION for technology and making things work along with a knack for helping others understand how things work. He comes with around 20 years of solution engineering experience in application security, software continuous delivery, and SaaS platforms. He is known for his dynamic presentations in CI/CD and application security integrated in software delivery lifecycle.
Gopinath Rebala
Gopinath Rebala is the CTO of OpsMx, where he has overall responsibility for the machine learning and data processing architectures for Secure Software Delivery. Gopi also has a strong connection with our customers, leading design and architecture for strategic implementations. Gopi is a frequent speaker and well-known leader in continuous delivery and integrating security into software delivery.
Observability Concepts EVERY Developer Should Know -- DeveloperWeek Europe.pdfPaige Cruz
Monitoring and observability aren’t traditionally found in software curriculums and many of us cobble this knowledge together from whatever vendor or ecosystem we were first introduced to and whatever is a part of your current company’s observability stack.
While the dev and ops silo continues to crumble….many organizations still relegate monitoring & observability as the purview of ops, infra and SRE teams. This is a mistake - achieving a highly observable system requires collaboration up and down the stack.
I, a former op, would like to extend an invitation to all application developers to join the observability party will share these foundational concepts to build on:
The Art of the Pitch: WordPress Relationships and SalesLaura Byrne
Clients don’t know what they don’t know. What web solutions are right for them? How does WordPress come into the picture? How do you make sure you understand scope and timeline? What do you do if sometime changes?
All these questions and more will be explored as we talk about matching clients’ needs with what your agency offers without pulling teeth or pulling your hair out. Practical tips, and strategies for successful relationship building that leads to closing the deal.
Encryption in Microsoft 365 - ExpertsLive Netherlands 2024Albert Hoitingh
In this session I delve into the encryption technology used in Microsoft 365 and Microsoft Purview. Including the concepts of Customer Key and Double Key Encryption.
DevOps and Testing slides at DASA ConnectKari Kakkonen
My and Rik Marselis slides at 30.5.2024 DASA Connect conference. We discuss about what is testing, then what is agile testing and finally what is Testing in DevOps. Finally we had lovely workshop with the participants trying to find out different ways to think about quality and testing in different parts of the DevOps infinity loop.
GraphRAG is All You need? LLM & Knowledge GraphGuy Korland
Guy Korland, CEO and Co-founder of FalkorDB, will review two articles on the integration of language models with knowledge graphs.
1. Unifying Large Language Models and Knowledge Graphs: A Roadmap.
https://arxiv.org/abs/2306.08302
2. Microsoft Research's GraphRAG paper and a review paper on various uses of knowledge graphs:
https://www.microsoft.com/en-us/research/blog/graphrag-unlocking-llm-discovery-on-narrative-private-data/
In his public lecture, Christian Timmerer provides insights into the fascinating history of video streaming, starting from its humble beginnings before YouTube to the groundbreaking technologies that now dominate platforms like Netflix and ORF ON. Timmerer also presents provocative contributions of his own that have significantly influenced the industry. He concludes by looking at future challenges and invites the audience to join in a discussion.
zkStudyClub - Reef: Fast Succinct Non-Interactive Zero-Knowledge Regex ProofsAlex Pruden
This paper presents Reef, a system for generating publicly verifiable succinct non-interactive zero-knowledge proofs that a committed document matches or does not match a regular expression. We describe applications such as proving the strength of passwords, the provenance of email despite redactions, the validity of oblivious DNS queries, and the existence of mutations in DNA. Reef supports the Perl Compatible Regular Expression syntax, including wildcards, alternation, ranges, capture groups, Kleene star, negations, and lookarounds. Reef introduces a new type of automata, Skipping Alternating Finite Automata (SAFA), that skips irrelevant parts of a document when producing proofs without undermining soundness, and instantiates SAFA with a lookup argument. Our experimental evaluation confirms that Reef can generate proofs for documents with 32M characters; the proofs are small and cheap to verify (under a second).
Paper: https://eprint.iacr.org/2023/1886
UiPath Test Automation using UiPath Test Suite series, part 4DianaGray10
Welcome to UiPath Test Automation using UiPath Test Suite series part 4. In this session, we will cover Test Manager overview along with SAP heatmap.
The UiPath Test Manager overview with SAP heatmap webinar offers a concise yet comprehensive exploration of the role of a Test Manager within SAP environments, coupled with the utilization of heatmaps for effective testing strategies.
Participants will gain insights into the responsibilities, challenges, and best practices associated with test management in SAP projects. Additionally, the webinar delves into the significance of heatmaps as a visual aid for identifying testing priorities, areas of risk, and resource allocation within SAP landscapes. Through this session, attendees can expect to enhance their understanding of test management principles while learning practical approaches to optimize testing processes in SAP environments using heatmap visualization techniques
What will you get from this session?
1. Insights into SAP testing best practices
2. Heatmap utilization for testing
3. Optimization of testing processes
4. Demo
Topics covered:
Execution from the test manager
Orchestrator execution result
Defect reporting
SAP heatmap example with demo
Speaker:
Deepak Rai, Automation Practice Lead, Boundaryless Group and UiPath MVP
2. myawr introduction
2
目录
目录
MyAWR(awr of MySQL) ...............................................................................................................................................................................................................................1
1.What is myawr...................................................................................................................................................................................................................................3
2.Myawr Data Model ...........................................................................................................................................................................................................................4
3. Quickstart.........................................................................................................................................................................................................................................7
3.1 install db.........................................................................................................................................................................................................................................7
3.2 initialize myawr_host.....................................................................................................................................................................................................................8
3.3 add two jobs in crontab .................................................................................................................................................................................................................8
4. Dependencies.................................................................................................................................................................................................................................10
5. Mysql WorkLoad Report ................................................................................................................................................................................................................10
6. Contact me.....................................................................................................................................................................................................................................46
3. myawr introduction
3
1.What is myawr
Myawr is a tool for collecting and analyzing performance data for MySQL database (including os info ,mysql status info and Slow Query Log
all of details). The idea comes from Oracle awr. Myawr periodic collect data and save to the database as snapshots. Myawr was designed as CS
architecture.Myawr depends on performance schema of MySQL database. Myawr consists of two parts:
myawr.pl--------a perl script for collecting mysql performance data
myawrrpt.pl-----a perl script for analyzing mysql performance data
Myawr relies on the Percona Toolkit to do the slow query log collection. Specifically you can run pt-query-digest. To parse your slow logs and
insert them into your server database for reporting and analyzing.
Thanks to orzdba.pl (zhuxu@taobao.com).
5. myawr introduction
5
some key tables:
myawr_host-- mysql instance config table
myawr_snapshot -- snapshot table,exec myawr.pl a time as a shapshot
myawr_query_review_history -- The table in which to store historical values for review trend analysis about slow log.
myawr data model:
myawr_snapshot.host_id reference myawr_host.id;
myawr_query_review_history.hostid_max reference myawr_host.id;
myawr_innodb_info.(host_id,snap_id) reference myawr_snapshot.(host_id,snap_id);
7. myawr introduction
7
3. Quickstart
If you are interesting to use this tool, here's what you need:
1. A MySQL database to store snapshot data and slow log analysis data .
2. pt-query-digest by percona
3. A MySQL server with perl-DBD-mysql
4. slow query logs named like slow_20130521.log,you can switch slow logs every day.
3.1 install db
Connect to the MySQL database where store the performance data and issue the following command in myawr.sql:
CREATE DATABASE `myawr` DEFAULT CHARACTER SET utf8;
grant all on myawr.* to 'user'@'%' identified by "111111";
then create tables in db myawr.
8. myawr introduction
8
3.2 initialize myawr_host
Insert a config record about your mysql instacne,just like:
INSERT INTO `myawr_host`(id,host_name,ip_addr,port,db_role,version) VALUES (6, 'db2.11', '192.168.2.11', 3306, 'master', '5.5.27');
.
3.3 add two jobs in crontab
* * * * * perl /data/mysql/sh/myawr.pl -u user -p 111111 -lh 192.168.2.11 -P 3306 -tu user -tp 111111 -TP 3306 -th
192.168.1.92 -n eth0 -d sdb1 -I 6 >> /data/mysql/sh/myawr_pl.log 2>&1
#
15 14 * * * /data/mysql/sh/pt-query-digest --user=user --password=111111 --review
h=192.168.1.92,D=myawr,t=myawr_query_review --review-history h=192.168.1.92,D=myawr,t=myawr_query_review_history
--no-report --limit=100% --filter="$event->{add_column} = length($event->{arg}) and $event->{hostid}=6"
/data/mysql/sh/slow_`date -d "-1 day" +"%Y%m%d"`.log >> /data/mysql/sh/pt-query_run.log 2>&1
myawr.pl Parameters:
-h,--help Print Help Info.
-i,--interval Time(second) Interval(default 1).
-d,--disk Disk Info(can't be null,default sda1).
-n,--net Net Info(default eth0).
-P,--port Port number to use for local mysql connection(default 3306).
-u,--user user name for local mysql(default user).
9. myawr introduction
9
-p,--pswd user password for local mysql(can't be null).
-lh,--lhost localhost(ip) for mysql where info is got(can't be null).
-TP,--tport Port number to use formysql where info is saved (default 3306)
-tu,--tuser user name for mysql where info is saved(default user).
-tp,--pswd user password for mysql where info is saved(can't be null).
-th,--thost host(ip) for mysql where info is saved(can't be null).
-I,--tid db instance register id(can't be null,Reference myawr_host.id)
pt-query-digest Parameters:
--user user name for mysql where info is saved
--password user password for mysql where info is saved
--review Store a sample of each class of query in this DSN
h host(ip) for mysql where info is saved
D database
t table name
--review-history The table in which to store historical values for review trend analysis.
h host(ip) for mysql where info is saved
D database
t table name
$event->{hostid}=6 db instance register id(Reference myawr_host.id)
The pt-query-digest only support mechanism for switching a slow log file every day just now, named like
slow_20130521.log(slow_date -d "-1 day" +"%Y%m%d".log)
10. myawr introduction
10
4. Dependencies
perl-DBD-mysql
you can install it two way:
yum install perl-DBD-MySQL
or install manually like :
mkdir /tmp/mysqldbd-install
cp /usr/lib64/mysql/*.a /tmp/mysqldbd-install
perl Makefile.PL --libs="-L/tmp/mysqldbd-install -lmysqlclient"
make
make test
make install
5. Mysql WorkLoad Report
We can use myawrrpt.pl to generate mysql workload report.You can execute the script on MySQL database machine where store the
performance data,but perl-DBD-MySQL is required.We also can execute the script in any linux machine with perl-DBD-MySQL installed.
You can execute it for help Info:
perl myawrrpt.pl -h
Info :
Created By noodba (www.noodba.com).
11. myawr introduction
11
References: Oracle awr
Usage :
Command line options :
-h,--help Print Help Info.
-P,--port Port number to use for local mysql connection(default 3306).
-u,--user user name for local mysql(default user).
-p,--pswd user password for local mysql(can't be null).
-lh,--lhost localhost(ip) for mysql where info is got(can't be null).
-I,--tid db instance register id(can't be null,Ref myawr_host.id)
Sample :
shell> perl myawrrpt.pl -p 111111 -lh 192.168.1.111 -I 11
============================================================================
Let use to generate mysql (db2.11,instance id = 6) workload report:
perl myawrrpt.pl -u user -p 111111 -P 3306 -lh 192.168.1.92 -I 6
===================================================
| Welcome to use the myawrrpt tool !
| Date: 2013-05-22
|
| Hostname is: db2.11
| Ip addr is: 192.168.2.11
12. myawr introduction
12
| Port is: 3306
| Db role is: master
|Server version is: 5.5.27
| Uptime is: 0y 2m 2d 7h 55mi 33s
|
| Min snap_id is: 1
| Min snap_time is: 2013-05-21 14:12:02
| Max snap_id is: 1147
| Max snap_time is: 2013-05-22 09:29:02
| snap interval is: 60s
===================================================
Listing the last 2 days Snapshots
---------------------------------
snap_id: 19 snap_time : 2013-05-21 14:30:02
snap_id: 38 snap_time : 2013-05-21 14:49:02
snap_id: 57 snap_time : 2013-05-21 15:08:02
snap_id: 76 snap_time : 2013-05-21 15:27:02
snap_id: 95 snap_time : 2013-05-21 15:46:02
.....................................................
snap_id: 1102 snap_time : 2013-05-22 08:44:02
snap_id: 1121 snap_time : 2013-05-22 09:03:02
snap_id: 1140 snap_time : 2013-05-22 09:22:02
snap_id: 1147 snap_time : 2013-05-22 09:29:02
13. myawr introduction
13
Pls select Start and End Snapshot Id
------------------------------------
Enter value for start_snap:1
Start Snapshot Id Is:1
Enter value for end_snap:589
End Snapshot Id Is:589
Set the Report Name
-------------------
Enter value for report_name:myawr.html
Using the report name :myawr.html
Generating the mysql report for this analysis ...
Generate the mysql report Successfully.
[mysql@test2 myawr]$ ls -al
total 976
drwxrwxr-x 2 mysql mysql 4096 May 22 09:30 .
drwx------ 19 mysql mysql 4096 May 13 10:42 ..
-rw-rw-r-- 1 mysql mysql 73074 May 22 09:30 myawr.html
-rw-rw-r-- 1 mysql mysql 53621 May 11 16:23 myawrrpt.pl
14. myawr introduction
14
Ok,let me show some pictures which come from my test db report:
Mysql WorkLoad Report
Host Name Ip addr Port Db role Version Uptime
db2.11 192.168.2.11 3306 master 5.5.27 0y 2m 2d 7h 55mi 33s
Snap Id Snap Time Threads_connected Threads_running
Begin Snap: 1 2013-05-21 14:12:02 481.00 3.00
End Snap: 589 2013-05-22 00:00:02 450.00 2.00
Elapsed: 35280 (seconds)
Start snap value and end snap value
Begin End</TH< tr>
query_cache_size : 536870912.00 536870912.00
45. myawr introduction
45
17518085470290754732 update test set plag = 2, uptime = now(), opid = 1 where ordid = 116459410
8783231904896245142 SELECT count(id) as count FROM zzz_log where code='200001' and ccdoe='100001'
3360175983660887134 SELECT count(id) as count FROM zzzz where code='200001'
4588707098242522987 select VARIABLE_VALUE from information_schema.GLOBAL_VARIABLES where VARIABLE_NAME='PERFORMANCE_SCHEMA'
.......more...... ........................................more................................
--------The End -----------
For detail report ,pls click myawr.html.
46. myawr introduction
46
6. Contact me
Any questions,pls contact me freely.
EMAIL: qiuwsh@gmail.com
Q Q : 570182914
Phone: (+86)13817963180