The document discusses various techniques for tuning MySQL performance. It begins with an introduction and agenda, then covers top performance issues such as bad SQL queries, long running transactions, and incorrect configurations. The rest of the document provides tips for monitoring different aspects of the system and tuning various configuration options, software, and application design factors to optimize MySQL performance.
MySQL Performance Tuning: Top 10 Tips intended for PHP, Ruby and Java developers on performance tuning and optimization of MySQL. We will cover the deadly mistakes to be avoided. We will take real life examples of optimizing application many times. Here is the summary of what we intend to cover:
• Selection of Storage Engine
• Schema Optimization
• Server Tuning
• Hardware Selection and Tuning
• Effective uses of Index, when to use and when not to use.
• Partitions
• Speeding up using Stored Procedures
• Implementing prepared statements?
• Deadly Sins to be avoided
• Performance Tuning and Benchmarking Tools
Microsoft released SQL Azure more than two years ago - that's enough time for testing (I hope!). So, are you ready to move your data to the Cloud? If you’re considering a business (i.e. a production environment) in the Cloud, you need to think about methods for backing up your data, a backup plan for your data and, eventually, restoring with Red Gate Cloud Services. In this session, you’ll see the differences, functionality, restrictions, and opportunities in SQL Azure and On-Premise SQL Server 2008/2008 R2/2012. We’ll consider topics such as how to be prepared for backup and restore, and which parts of a cloud environment are most important: keys, triggers, indexes, prices, security, service level agreements, etc.
Microsoft released SQL Azure more than two years ago - that's enough time for testing (I hope!). So, are you ready to move your data to the Cloud? If you’re considering a business (i.e. a production environment) in the Cloud, you need to think about methods for backing up your data, a backup plan for your data and, eventually, restoring with Red Gate Cloud Services. In this session, you’ll see the differences, functionality, restrictions, and opportunities in SQL Azure and On-Premise SQL Server 2008/2008 R2/2012. We’ll consider topics such as how to be prepared for backup and restore, and which parts of a cloud environment are most important: keys, triggers, indexes, prices, security, service level agreements, etc.
MySQL Performance Tuning: Top 10 Tips intended for PHP, Ruby and Java developers on performance tuning and optimization of MySQL. We will cover the deadly mistakes to be avoided. We will take real life examples of optimizing application many times. Here is the summary of what we intend to cover:
• Selection of Storage Engine
• Schema Optimization
• Server Tuning
• Hardware Selection and Tuning
• Effective uses of Index, when to use and when not to use.
• Partitions
• Speeding up using Stored Procedures
• Implementing prepared statements?
• Deadly Sins to be avoided
• Performance Tuning and Benchmarking Tools
Microsoft released SQL Azure more than two years ago - that's enough time for testing (I hope!). So, are you ready to move your data to the Cloud? If you’re considering a business (i.e. a production environment) in the Cloud, you need to think about methods for backing up your data, a backup plan for your data and, eventually, restoring with Red Gate Cloud Services. In this session, you’ll see the differences, functionality, restrictions, and opportunities in SQL Azure and On-Premise SQL Server 2008/2008 R2/2012. We’ll consider topics such as how to be prepared for backup and restore, and which parts of a cloud environment are most important: keys, triggers, indexes, prices, security, service level agreements, etc.
Microsoft released SQL Azure more than two years ago - that's enough time for testing (I hope!). So, are you ready to move your data to the Cloud? If you’re considering a business (i.e. a production environment) in the Cloud, you need to think about methods for backing up your data, a backup plan for your data and, eventually, restoring with Red Gate Cloud Services. In this session, you’ll see the differences, functionality, restrictions, and opportunities in SQL Azure and On-Premise SQL Server 2008/2008 R2/2012. We’ll consider topics such as how to be prepared for backup and restore, and which parts of a cloud environment are most important: keys, triggers, indexes, prices, security, service level agreements, etc.
DB12c: All You Need to Know About the Resource ManagerMaris Elsins
This presentation is different from the previous uploads as SLOB was used for the testing.
Oracle Database 12c Multitenant provides the highest level of Oracle Database resource efficiency, driven by an improved resource manager. The 12c resource manager effectively allocates resources both within a single database and between multiple pluggable databases in a container. This presentation will review new features of the 12c resource manager, provide guidelines for migration of your current resource management plan to 12c, and will also look into how much overhead the resource manager introduces.
Severalnines Self-Training: MySQL® Cluster - Part VIISeveralnines
Part VII of our free self-training slides on MySQL Cluster.
In this installment, we cover ’Management and Administration'
* Backup and Restore
* Geographical Redundancy
* Online and Offline Operations
* Ndbinfo tables
* Reporting
* Single User Mode
* Scaling MySQL Cluster
Severalnines Self-Training: MySQL® Cluster - Part IISeveralnines
Part II of our free self-training slides on MySQL Cluster.
In this part we cover 'Detailed Concepts':
* Data Distribution & Partitioning
* Two Phase Commit Protocol
* Transaction Resources
SQL Server Best Practices - Install SQL Server like a boss (RELOADED)Andre Essing
Best practices are recommendations for a rock solid system and high performance. These best practices are based on recommendations from vendors and countless experiences that were made in the community. All these experiences, tips and recommendations combined makes the manual how you should setup and configure a system. This also applies to SQL Server. Some of these best practices were already spotted on some PASS chapter meeting, SQL Saturdays and conferences.
Unfortunately, on the most SQL Servers you can find best practices only in rare cases. Most times performance issues or instability could be solved by implementing just these best practices. Starting at the BIOS settings, going through the Windows settings and for sure the SQL Server itself, I want to show you how to configure your SQL Server to make it a rock solid high performance data monster.
Mining the AWR: Alternative Methods for Identification of the Top SQLs (inclu...Maris Elsins
A typical tuning session on a resource-constrained system starts with a search for "low-hanging fruit." In a CPU-bound database system, it would be the SQL that uses CPU the most, in an I/O-bound system, the SQL doing the most physical reads, and so on. Tuning the TOP statements often allows us to free large portions of the utilized resources and remove bottlenecks. Often, we can use AWR reports to quickly identify the SQL_IDs of the top statements in the database. But what if the AWR report reveals no "low-hanging fruit," and the resource usage is evenly distributed among multiple statements? Where do we start? Is there a better way to identify the starting point for the tuning of a resource-bound system?
This presentation will explain when the AWR reports are misleading and how we can take a look at the data stored in AWR from a different angle to determine the top consumers. Discussion will include a practical demonstration using scripts for AWR mining that attendees can apply to their own challenging database performance tuning problems.
Scripts and the demo log: https://github.com/MarisElsins/TOOLS/tree/master/SQL/C15LV_AWR
The Oracle Database Appliance is a new way to take advantage of the world\'s most popular database—Oracle Database 11g—in a single, easy-to-deploy and manage system. It\'s a complete package of software, server, storage, and networking that\'s engineered for simplicity; saving time and money by simplifying deployment, maintenance, and support of database workloads.
StorMagic have announced a major new release of SvSAN bringing unprecedented flexbility, efficiency and performance to our customers and partners.
Join John Glendenning (SVP Sales and Marketing) and Luke Pruen (Technical Services Director) to learn more about how SvSAN 6 can modernize your IT infrastructure and take advantage of next generation software defined storage built on a proven platform.
StorMagic SvSAN 6 is available to purchase now and is available through StorMagic’s Preferred Partner Network and via the company’s global OEM agreement with Cisco. Order your free trial now by visiting http://stormagic.com/trial/
How to configure SQL Server for SSDs and VMsSolarWinds
Recent trends in virtualized converged infrastructures, coupled with rapidly decreasing costs of solid state storage (SSD), pose new challenges for DBAs and developers alike. Learn answers to to pressing questions like:
• If my array uses compression, are there benefits to using database compression?
• How should I arrange my data files on an SSD array?
• If I have limited SSD space, how should I use it?
• On a hybrid array (spinning disk and SSD) how should I structure policies?
• What changes from physical servers to VMs?
Optimizing Your Postgres ROI Through Best PracticesEDB
With a goal of helping you maximize your investment in Postgres, this presentation addresses commonly asked questions such as:
- Can my existing database architecture and deployment strategy keep up with demand?
- What if I didn’t have to contend with the process and expense of hiring another DBA?
- How can I improve my team’s ability to resolve complex database management issues?
You will gain some short-term and long-term options that can help you avoid problems and make the most of your deployment.
This present will interest both business and technical decision-makers or influencers responsible for the overall strategy and execution of a PostgreSQL and/or an EDB Postgres database.
This presentation will provide the key ingredients for success, but will not dive into the specifics of deployment, architecture, or how to measure your ROI. If your goal is to learn finer details, please email sales@enterprisedb.com and you can easily schedule a call with one of EDB’s technical experts.
Going thru the era of IoT that involves lots more and much bigger data, we need a faster database. MySQL 5.7 gives you 3x speed of its predecessor and able to reach 1.6m qps on our select benchmark.
OSDC 2016 - Tuning Linux for your Database by Colin CharlesNETWAYS
Many operations folk know that performance varies depending on using one of the many Linux filesystems like EXT4 or XFS. They also know of the schedulers available, they see the OOM killer coming and more. However, appropriate configuration is necessary when you're running your databases at scale.
Learn best practices for Linux performance tuning for MariaDB/MySQL (where MyISAM uses the operating system cache, and InnoDB maintains its own aggressive buffer pool), as well as PostgreSQL and MongoDB (more dependent on the operating system). Topics that will be covered include: filesystems, swap and memory management, I/O scheduler settings, using and understanding the tools available (like iostat/vmstat/etc), practical kernel configuration, profiling your database, and using RAID and LVM.
There is a focus on bare metal as well as configuring your cloud instances in.
Learn from practical examples from the trenches.
DB12c: All You Need to Know About the Resource ManagerMaris Elsins
This presentation is different from the previous uploads as SLOB was used for the testing.
Oracle Database 12c Multitenant provides the highest level of Oracle Database resource efficiency, driven by an improved resource manager. The 12c resource manager effectively allocates resources both within a single database and between multiple pluggable databases in a container. This presentation will review new features of the 12c resource manager, provide guidelines for migration of your current resource management plan to 12c, and will also look into how much overhead the resource manager introduces.
Severalnines Self-Training: MySQL® Cluster - Part VIISeveralnines
Part VII of our free self-training slides on MySQL Cluster.
In this installment, we cover ’Management and Administration'
* Backup and Restore
* Geographical Redundancy
* Online and Offline Operations
* Ndbinfo tables
* Reporting
* Single User Mode
* Scaling MySQL Cluster
Severalnines Self-Training: MySQL® Cluster - Part IISeveralnines
Part II of our free self-training slides on MySQL Cluster.
In this part we cover 'Detailed Concepts':
* Data Distribution & Partitioning
* Two Phase Commit Protocol
* Transaction Resources
SQL Server Best Practices - Install SQL Server like a boss (RELOADED)Andre Essing
Best practices are recommendations for a rock solid system and high performance. These best practices are based on recommendations from vendors and countless experiences that were made in the community. All these experiences, tips and recommendations combined makes the manual how you should setup and configure a system. This also applies to SQL Server. Some of these best practices were already spotted on some PASS chapter meeting, SQL Saturdays and conferences.
Unfortunately, on the most SQL Servers you can find best practices only in rare cases. Most times performance issues or instability could be solved by implementing just these best practices. Starting at the BIOS settings, going through the Windows settings and for sure the SQL Server itself, I want to show you how to configure your SQL Server to make it a rock solid high performance data monster.
Mining the AWR: Alternative Methods for Identification of the Top SQLs (inclu...Maris Elsins
A typical tuning session on a resource-constrained system starts with a search for "low-hanging fruit." In a CPU-bound database system, it would be the SQL that uses CPU the most, in an I/O-bound system, the SQL doing the most physical reads, and so on. Tuning the TOP statements often allows us to free large portions of the utilized resources and remove bottlenecks. Often, we can use AWR reports to quickly identify the SQL_IDs of the top statements in the database. But what if the AWR report reveals no "low-hanging fruit," and the resource usage is evenly distributed among multiple statements? Where do we start? Is there a better way to identify the starting point for the tuning of a resource-bound system?
This presentation will explain when the AWR reports are misleading and how we can take a look at the data stored in AWR from a different angle to determine the top consumers. Discussion will include a practical demonstration using scripts for AWR mining that attendees can apply to their own challenging database performance tuning problems.
Scripts and the demo log: https://github.com/MarisElsins/TOOLS/tree/master/SQL/C15LV_AWR
The Oracle Database Appliance is a new way to take advantage of the world\'s most popular database—Oracle Database 11g—in a single, easy-to-deploy and manage system. It\'s a complete package of software, server, storage, and networking that\'s engineered for simplicity; saving time and money by simplifying deployment, maintenance, and support of database workloads.
StorMagic have announced a major new release of SvSAN bringing unprecedented flexbility, efficiency and performance to our customers and partners.
Join John Glendenning (SVP Sales and Marketing) and Luke Pruen (Technical Services Director) to learn more about how SvSAN 6 can modernize your IT infrastructure and take advantage of next generation software defined storage built on a proven platform.
StorMagic SvSAN 6 is available to purchase now and is available through StorMagic’s Preferred Partner Network and via the company’s global OEM agreement with Cisco. Order your free trial now by visiting http://stormagic.com/trial/
How to configure SQL Server for SSDs and VMsSolarWinds
Recent trends in virtualized converged infrastructures, coupled with rapidly decreasing costs of solid state storage (SSD), pose new challenges for DBAs and developers alike. Learn answers to to pressing questions like:
• If my array uses compression, are there benefits to using database compression?
• How should I arrange my data files on an SSD array?
• If I have limited SSD space, how should I use it?
• On a hybrid array (spinning disk and SSD) how should I structure policies?
• What changes from physical servers to VMs?
Optimizing Your Postgres ROI Through Best PracticesEDB
With a goal of helping you maximize your investment in Postgres, this presentation addresses commonly asked questions such as:
- Can my existing database architecture and deployment strategy keep up with demand?
- What if I didn’t have to contend with the process and expense of hiring another DBA?
- How can I improve my team’s ability to resolve complex database management issues?
You will gain some short-term and long-term options that can help you avoid problems and make the most of your deployment.
This present will interest both business and technical decision-makers or influencers responsible for the overall strategy and execution of a PostgreSQL and/or an EDB Postgres database.
This presentation will provide the key ingredients for success, but will not dive into the specifics of deployment, architecture, or how to measure your ROI. If your goal is to learn finer details, please email sales@enterprisedb.com and you can easily schedule a call with one of EDB’s technical experts.
Going thru the era of IoT that involves lots more and much bigger data, we need a faster database. MySQL 5.7 gives you 3x speed of its predecessor and able to reach 1.6m qps on our select benchmark.
OSDC 2016 - Tuning Linux for your Database by Colin CharlesNETWAYS
Many operations folk know that performance varies depending on using one of the many Linux filesystems like EXT4 or XFS. They also know of the schedulers available, they see the OOM killer coming and more. However, appropriate configuration is necessary when you're running your databases at scale.
Learn best practices for Linux performance tuning for MariaDB/MySQL (where MyISAM uses the operating system cache, and InnoDB maintains its own aggressive buffer pool), as well as PostgreSQL and MongoDB (more dependent on the operating system). Topics that will be covered include: filesystems, swap and memory management, I/O scheduler settings, using and understanding the tools available (like iostat/vmstat/etc), practical kernel configuration, profiling your database, and using RAID and LVM.
There is a focus on bare metal as well as configuring your cloud instances in.
Learn from practical examples from the trenches.
Tuning Linux for your database FLOSSUK 2016Colin Charles
Some best practices about tuning Linux for your database workloads. The focus is not just on MySQL or MariaDB Server but also on understanding the OS from hardware/cloud, I/O, filesystems, memory, CPU, network, and resources.
The environment in which your EECMS lives is as important as what can be seen by your clients in their browser. A solid foundation is key to the overall performance, scalability and security of your site. Building on over a decade of server optimization experience, extensive benchmarking and some custom ExpressionEngine extensions this session will show you how to make sure your ExpressionEngine install is ready for prime time.
Windows Server 2012 R2 Software-Defined StorageAidan Finn
In this presentation I taught attendees how to build a Scale-Out File Server (SOFS) using Windows Server 2012 R2, JBODs, Storage Spaces, Failover Clustering, and SMB 3.0 Networking, suitable for storing application data such as Hyper-V and SQL Server.
AI, ML and Graph Algorithms: Real Life Use Cases with Neo4jIvan Zoratti
I gave this presentation at DataOps 19 in Barcelona.
You will find information about Neo4j and how to use it with Graph Algorithms for Machine Learning and Artificial Intelligence.
This is the presentation at the OSIsoft EMEA User Conference in London, 16 October 2017.
Please note that "Open Edge Module" and "FogLAMP" are synonyms.
Time Series From Collection To AnalysisIvan Zoratti
This is my talk at Percona Live 2016 in Santa Clara. It is a quick walkthrough time series workloads and solutions with traditional relational databases and dedicated time series DBs
This is the presentation at Percona Live 2015 on MySQL, MariaDB and Percona Orchestration on bare metal, virtualised environments and clouds (AWS and OpenStack).
These are the slides that I presented at Percona Live London, 4th Dec 2012.
There is lots of content related to the deployment and use of MySQL in the cloud, specifically in Amazon EC2.
These are the slides of my presentation at the NYC MySQL Meetup on Sep 21 2012. There are tips and tricks about MySQL in the cloud and the SkySQL cloud data suite
In the rapidly evolving landscape of technologies, XML continues to play a vital role in structuring, storing, and transporting data across diverse systems. The recent advancements in artificial intelligence (AI) present new methodologies for enhancing XML development workflows, introducing efficiency, automation, and intelligent capabilities. This presentation will outline the scope and perspective of utilizing AI in XML development. The potential benefits and the possible pitfalls will be highlighted, providing a balanced view of the subject.
We will explore the capabilities of AI in understanding XML markup languages and autonomously creating structured XML content. Additionally, we will examine the capacity of AI to enrich plain text with appropriate XML markup. Practical examples and methodological guidelines will be provided to elucidate how AI can be effectively prompted to interpret and generate accurate XML markup.
Further emphasis will be placed on the role of AI in developing XSLT, or schemas such as XSD and Schematron. We will address the techniques and strategies adopted to create prompts for generating code, explaining code, or refactoring the code, and the results achieved.
The discussion will extend to how AI can be used to transform XML content. In particular, the focus will be on the use of AI XPath extension functions in XSLT, Schematron, Schematron Quick Fixes, or for XML content refactoring.
The presentation aims to deliver a comprehensive overview of AI usage in XML development, providing attendees with the necessary knowledge to make informed decisions. Whether you’re at the early stages of adopting AI or considering integrating it in advanced XML development, this presentation will cover all levels of expertise.
By highlighting the potential advantages and challenges of integrating AI with XML development tools and languages, the presentation seeks to inspire thoughtful conversation around the future of XML development. We’ll not only delve into the technical aspects of AI-powered XML development but also discuss practical implications and possible future directions.
GraphSummit Singapore | The Future of Agility: Supercharging Digital Transfor...Neo4j
Leonard Jayamohan, Partner & Generative AI Lead, Deloitte
This keynote will reveal how Deloitte leverages Neo4j’s graph power for groundbreaking digital twin solutions, achieving a staggering 100x performance boost. Discover the essential role knowledge graphs play in successful generative AI implementations. Plus, get an exclusive look at an innovative Neo4j + Generative AI solution Deloitte is developing in-house.
GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using Deplo...James Anderson
Effective Application Security in Software Delivery lifecycle using Deployment Firewall and DBOM
The modern software delivery process (or the CI/CD process) includes many tools, distributed teams, open-source code, and cloud platforms. Constant focus on speed to release software to market, along with the traditional slow and manual security checks has caused gaps in continuous security as an important piece in the software supply chain. Today organizations feel more susceptible to external and internal cyber threats due to the vast attack surface in their applications supply chain and the lack of end-to-end governance and risk management.
The software team must secure its software delivery process to avoid vulnerability and security breaches. This needs to be achieved with existing tool chains and without extensive rework of the delivery processes. This talk will present strategies and techniques for providing visibility into the true risk of the existing vulnerabilities, preventing the introduction of security issues in the software, resolving vulnerabilities in production environments quickly, and capturing the deployment bill of materials (DBOM).
Speakers:
Bob Boule
Robert Boule is a technology enthusiast with PASSION for technology and making things work along with a knack for helping others understand how things work. He comes with around 20 years of solution engineering experience in application security, software continuous delivery, and SaaS platforms. He is known for his dynamic presentations in CI/CD and application security integrated in software delivery lifecycle.
Gopinath Rebala
Gopinath Rebala is the CTO of OpsMx, where he has overall responsibility for the machine learning and data processing architectures for Secure Software Delivery. Gopi also has a strong connection with our customers, leading design and architecture for strategic implementations. Gopi is a frequent speaker and well-known leader in continuous delivery and integrating security into software delivery.
Building RAG with self-deployed Milvus vector database and Snowpark Container...Zilliz
This talk will give hands-on advice on building RAG applications with an open-source Milvus database deployed as a docker container. We will also introduce the integration of Milvus with Snowpark Container Services.
Securing your Kubernetes cluster_ a step-by-step guide to success !KatiaHIMEUR1
Today, after several years of existence, an extremely active community and an ultra-dynamic ecosystem, Kubernetes has established itself as the de facto standard in container orchestration. Thanks to a wide range of managed services, it has never been so easy to set up a ready-to-use Kubernetes cluster.
However, this ease of use means that the subject of security in Kubernetes is often left for later, or even neglected. This exposes companies to significant risks.
In this talk, I'll show you step-by-step how to secure your Kubernetes cluster for greater peace of mind and reliability.
GridMate - End to end testing is a critical piece to ensure quality and avoid...ThomasParaiso2
End to end testing is a critical piece to ensure quality and avoid regressions. In this session, we share our journey building an E2E testing pipeline for GridMate components (LWC and Aura) using Cypress, JSForce, FakerJS…
Removing Uninteresting Bytes in Software FuzzingAftab Hussain
Imagine a world where software fuzzing, the process of mutating bytes in test seeds to uncover hidden and erroneous program behaviors, becomes faster and more effective. A lot depends on the initial seeds, which can significantly dictate the trajectory of a fuzzing campaign, particularly in terms of how long it takes to uncover interesting behaviour in your code. We introduce DIAR, a technique designed to speedup fuzzing campaigns by pinpointing and eliminating those uninteresting bytes in the seeds. Picture this: instead of wasting valuable resources on meaningless mutations in large, bloated seeds, DIAR removes the unnecessary bytes, streamlining the entire process.
In this work, we equipped AFL, a popular fuzzer, with DIAR and examined two critical Linux libraries -- Libxml's xmllint, a tool for parsing xml documents, and Binutil's readelf, an essential debugging and security analysis command-line tool used to display detailed information about ELF (Executable and Linkable Format). Our preliminary results show that AFL+DIAR does not only discover new paths more quickly but also achieves higher coverage overall. This work thus showcases how starting with lean and optimized seeds can lead to faster, more comprehensive fuzzing campaigns -- and DIAR helps you find such seeds.
- These are slides of the talk given at IEEE International Conference on Software Testing Verification and Validation Workshop, ICSTW 2022.
Epistemic Interaction - tuning interfaces to provide information for AI supportAlan Dix
Paper presented at SYNERGY workshop at AVI 2024, Genoa, Italy. 3rd June 2024
https://alandix.com/academic/papers/synergy2024-epistemic/
As machine learning integrates deeper into human-computer interactions, the concept of epistemic interaction emerges, aiming to refine these interactions to enhance system adaptability. This approach encourages minor, intentional adjustments in user behaviour to enrich the data available for system learning. This paper introduces epistemic interaction within the context of human-system communication, illustrating how deliberate interaction design can improve system understanding and adaptation. Through concrete examples, we demonstrate the potential of epistemic interaction to significantly advance human-computer interaction by leveraging intuitive human communication strategies to inform system design and functionality, offering a novel pathway for enriching user-system engagements.
UiPath Test Automation using UiPath Test Suite series, part 6DianaGray10
Welcome to UiPath Test Automation using UiPath Test Suite series part 6. In this session, we will cover Test Automation with generative AI and Open AI.
UiPath Test Automation with generative AI and Open AI webinar offers an in-depth exploration of leveraging cutting-edge technologies for test automation within the UiPath platform. Attendees will delve into the integration of generative AI, a test automation solution, with Open AI advanced natural language processing capabilities.
Throughout the session, participants will discover how this synergy empowers testers to automate repetitive tasks, enhance testing accuracy, and expedite the software testing life cycle. Topics covered include the seamless integration process, practical use cases, and the benefits of harnessing AI-driven automation for UiPath testing initiatives. By attending this webinar, testers, and automation professionals can gain valuable insights into harnessing the power of AI to optimize their test automation workflows within the UiPath ecosystem, ultimately driving efficiency and quality in software development processes.
What will you get from this session?
1. Insights into integrating generative AI.
2. Understanding how this integration enhances test automation within the UiPath platform
3. Practical demonstrations
4. Exploration of real-world use cases illustrating the benefits of AI-driven test automation for UiPath
Topics covered:
What is generative AI
Test Automation with generative AI and Open AI.
UiPath integration with generative AI
Speaker:
Deepak Rai, Automation Practice Lead, Boundaryless Group and UiPath MVP
Full-RAG: A modern architecture for hyper-personalizationZilliz
Mike Del Balso, CEO & Co-Founder at Tecton, presents "Full RAG," a novel approach to AI recommendation systems, aiming to push beyond the limitations of traditional models through a deep integration of contextual insights and real-time data, leveraging the Retrieval-Augmented Generation architecture. This talk will outline Full RAG's potential to significantly enhance personalization, address engineering challenges such as data management and model training, and introduce data enrichment with reranking as a key solution. Attendees will gain crucial insights into the importance of hyperpersonalization in AI, the capabilities of Full RAG for advanced personalization, and strategies for managing complex data integrations for deploying cutting-edge AI solutions.
Generative AI Deep Dive: Advancing from Proof of Concept to ProductionAggregage
Join Maher Hanafi, VP of Engineering at Betterworks, in this new session where he'll share a practical framework to transform Gen AI prototypes into impactful products! He'll delve into the complexities of data collection and management, model selection and optimization, and ensuring security, scalability, and responsible use.
2. WHO I AM
• aadant@jumptrading.com
• 3.5 years @ Oracle MySQL as support engineer
– Filed hundreds of bugs and feature requests
• InnoDB
• Optimizer
• High availability
• Performance
• Joined Jump Trading in 2015
3. AGENDA
• Introduction
• Mostly performance tuning tips
• this presentation is an update from the 50 tips
presentation done while at Oracle for MySQL
5.6
• Questions
4. TOP PERFORMANCE ISSUES
• Bad SQL queries : 90 % of the time
• Long running idle transaction
• high history_list_length
• MDL
• Replication lag
• no primary key and row based replication
• Wrong / bad configuration
• smaller buffer pool, query cache, small redo logs
• OS, hardware
5. PERFORMANCE CHECK LIST
• #1 Monitor the queries …
• tune the apps
• #2 Monitor replication
• tune replication
• #3 Monitor the status variables
• tune the config
• #4 Monitor the OS
• tune the OS
• #5 in doubt, benchmark
6. THE APP (DEFINITION)
• Everything that is not shipped with the database
• schema
• queries
• user connections
• the app data in data files
• …
8. CHANGE PROCESS
• One change at a time, test on dev = prod, then deploy
Development Production
deploy
Change Test
Monitor
OK
Monitor
9. TUNING PROCESS
1. Config
• OS
• Config variables
• App design
• Benchmark
2. Monitor
• App / status variables / replication / mysqld / os
3. Tune using the change process
• App
• Config variables
• Table spaces
• OS
10. CONFIG / OS 1/2
• RAM
• CPU
• Storage
• OS
• OS limits
• Battery backed disk cache
• Memory allocator
• CPU affinity
11. CONFIG / OS 2/2
• I/O scheduler
• File system
• Mount options
• Disk configuration
• NUMA
12. RAM
• The active data should fit in the buffer pool
• MySQL connections and caches take memory
• ECC RAM recommended
• Enable huge pages for large buffer pools : large-pages
• Extra RAM for
• FS cache
• Monitoring
• RAM disk (tmpfs)
• RAM is split over NUMA nodes
13. CPU
• Fast CPU for single threaded performance
• Recent servers have 32 to 80 cores.
• Enable hyper-threading
• MySQL 5.7 scales to 64 cores
• Minimize the number of sockets !
15. STORAGE
• Good for IO bound loads
• HDD for sequential reads and writes
• Bus-attached SSD for random reads and writes
• NVMe : Non-Volatile Memory Express
• Big sata or other disk for log files
• Several disks ! If striping, mind the stripe size, align
• Life time for low end SSD is a concern
16. OPERATING SYSTEM
• Linux !
• pick a modern distribution
• also works well on Windows, FreeBSD, MacOS, Solaris
17. OS LIMITS (LINUX)
• Max open files per process
• ulimit –n
• limits the number of file handles (connections, open
tables, …)
• Max threads per user
• ulimit –u
• limits the number of threads (connections, event
scheduler, shutdown)
18. BATTERY BACKED DISK CACHE
• Usually faster fsyncs
• InnoDB redo logs
• binary logs
• data files
• Crash safety
• Durability (ACID)
• Applies to SSD
19. MEMORY ALLOCATOR
• jemalloc is a good malloc replacement
[mysqld_safe]
malloc-lib=/usr/lib64/libjemalloc.so.1
Default with MariaDB
• tcmalloc shipped on Linux with MySQL
[mysqld_safe]
malloc-lib=tcmalloc
20. CPU USAGE TUNING
• taskset command on Linux for core assignment
• taskset -c 1-4 `pidof mysqld`
• taskset -c 1,2,3,4 `pidof mysqld`
• niceness
[mysqld_safe]
nice=-20
also in /etc/security/limits.conf
• CPU governor
• set to performance
21. FILE SYSTEM
• ext4 best choice for speed and ease of use
• fsyncs a bit slower than ext3
• more reliable
• xfs is excellent (for experts only)
• With innodb_flush_method = O_DIRECT
• less stable recently
• ext3 is also a good choice
22. MOUNT OPTIONS
• ext4 (rw,noatime,nodiratime,nobarrier,data=ordered)*
• xfs (rw, noatime,nodiratime,nobarrier,logbufs=8,logbsize=32k)
• SSD specific
• innodb_use_trim
• innodb_page_size = 4K
• Innodb_flush_neighbors = 0
* use no barrier if you have a battery backed disk cache
23. I/O SCHEDULER
• deadline is generally the best I/O scheduler
• echo deadline > /sys/block/{DEVICE-NAME}/queue/scheduler
• the best value is HW and WL specific
• noop on high end controller (SSD, good RAID card …)
• deadline otherwise
24. DISK CONFIGURATION
• everything on one disk is killing performance
• several disks (RAID)
• or distribute the data files on different disks
• data files (ibd files)
• main InnoDB table space ibdata
• redo logs
• undo logs (if separate)
• binary logs
25. NUMA
• NUMA architecture is the norm nowadays for mysql server
• Problem : RAM is allocated to CPU sockets
• The Buffer pool should be distributed in each RAM
• Percona, MariaDB and MySQL >= 5.6 support NUMA
• The usual trick is using mysql_safe
• flush the FS cache --flush-caches
• use NUMA interleave --numa-interleave
26. NUMA
• From 10.1, 5.7, -- innodb-numa-interleave
• Allocates the buffer pool at startup
• “Replaces” innodb_buffer_pool_populate
• Default off
• “For the innodb_numa_interleave option to be available,
MySQL must be compiled on a NUMA-enabled Linux system
MariaDB : only custom builds MDEV-12924
27. CONFIG / VARIABLES 1/2
• Buffer pool size
• Query cache off
• Use a thread pool
• Cache the tables
• Cache threads
• Per thread memory usage
• Default storage engine
• Buffer pool contention
29. INNODB BUFFER POOL SIZE
• innodb_buffer_pool_size
• Not too large for the data
• Do not swap !
• Beware of memory crash if swapping is disabled
• Active data <= innodb_buffer_pool_size <= 0.8 * RAM
30. DISABLE THE QUERY CACHE
• Single threaded cache, removed in 8.0
• Only if threads_running <= 4
• Becomes fragmented
• Cache should be in the App !
• Off by default from 5.6
• query_cache_type = 0
• query_cache_size =0
• Problem if qcache_free_blocks > 50k
31. ENABLE THE THREAD POOL
• https://www.percona.com/blog/2014/01/23/percona-server-
improve-scalability-percona-thread-pool/
• Stabilize TPS for high
concurrency
• Useful if threads_running
> hardware threads
• Decrease context switches
• Several connections for one
execution thread
• Acts as a Speed Limiter
• MySQL commercial,
Percona, MariaDB
32. TABLE CACHING
• table_open_cache
• not too small, not too big
• opened_tables / sec
• table_definition_cache
• do not forget to increase
• opened_table_definitions / sec
• table_cache_instances = 8 or 16 (MySQL and Percona only)
• innodb_open_files
• mdl_hash_instances = 256 (in 5.7, no more an issue)
33. THREAD CACHING
• Thread creation is expensive, so caching is important
• thread_cache_size
• decreases threads_created rate
• capped by max user processes (see OS limits)
• 5.7.2 refactors this code
• Default value is calculated in 5.7
• http://dev.mysql.com/doc/refman/5.7/en/server-system-
variables.html#sysvar_thread_cache_size
34. INNODB STORAGE ENGINE
• Should be the default storage engine
• Do not use MyISAM unless you know what you are doing
• The most advanced storage engine is InnoDB
• Scalable
• Temporary tables use InnoDB in 5.7
• the memory engine is still used when using less then
tmp_table_size
35. BUFFER POOL CONTENTION
• innodb_buffer_pool_instances >= 8
• Reduce rows_examined / sec (see Bug #68079)
• 8 is the default value in 5.6 !
• innodb_spin_wait_delay = 96 on high concurrency
• Use read only transactions when possible
36. LARGE REDO LOGS
• Redo logs defer the expensive changes to the data files
• Recovery time is no more an issue
• innodb_log_file_size = 2047M before 5.6
• innodb_log_file_size >= 2047M from 5.6
• Bigger is better for write QPS stability
• You want to avoid “furious flushing”
• innodb_log_files_in_group = 2 is usually fine
37. IO CAPACITY
• IO capacity should mirror device IO capacity in IOPS
• innodb_io_capacity should be larger for SSD
• Impacts flushing
• In 5.6, innodb_lru_scan_depth is per buffer pool instance
• so innodb_lru_scan_depth = innodb_io_capacity /
innodb_buffer_pool_instances
• Default innodb_io_capacity_max = min(2000, 2 * innodb_io_capacity)
• From 5.7, innodb_page_cleaners, default = 4
38. PER THREAD MEMORY USAGE
• Each thread allocates memory
• estimates = max_used_connections * (
read_buffer_size +
read_rnd_buffer_size +
join_buffer_size +
sort_buffer_size +
binlog_cache_size +
thread_stack +
2 * net_buffer_length … + n * tmp_table_size where n >= 0
)
For a more accurate measure, check the performance_schema memory metrics
or MariaDB select * from information_schema.processlist.
39. THREAD CONCURRENCY
• No thread pool :
• innodb_thread_concurrency = 16 - 32 in 5.5
• innodb_thread_concurrency = 36 in 5.6
• align to HW threads if less than 32 cores
• Thread pool :
• innodb_thread_concurrency = 0 (unlimited) is fine
• innodb_max_concurrency_tickets : higher for OLAP, lower for
OLTP
40. INNODB FLUSHING
• Redo logs :
• innodb_flush_log_at_trx_commit = 1 // best durability
• innodb_flush_log_at_trx_commit = 2 // better performance
• innodb_flush_log_at_trx_commit = 0 // best performance
• Data files only :
• innodb_flush_method = O_DIRECT // Linux, skips the FS cache
• Increase innodb_adaptive_flushing_lwm (fast disk)
41. INNODB_FILE_PER_TABLE
• Default value : 1 table = 1 table space = 1 ibd file
• Not so good for small tables.
• Good for large tables.
• Default value from 5.6, before all tables in system table space.
• From 5.7, a user defined table space can now host several tables.
• http://dev.mysql.com/doc/refman/5.7/en/create-tablespace.html
42. TRANSACTION ISOLATION
• Default = REPEATABLE-READS
• Oracle database, Sybase ASE default is READ-COMMITTED
• Less locking, no gap locks, less overhead
• transaction-isolation = REPEATABLE-READ
• If you enable READ-COMMITTED, make sure binlog_format=ROW.
43. REPLICATION DURABILITY
• Replication is crash safe from 5.6
• Replication state was stored in files
• Now stored in InnoDB tables in the mysql schema
• However the binary logs are stored on disk :
• sync_binlog = 1
• no reason not to use from 5.6
44. MULTI-THREADED REPLICATION
• MariaDB slave_parallel_threads
• MySQL / Percona : slave_parallel_workers
• >1 when lag is a concern
• For a recent comparison, see https://www.percona.com/live/data-
performance-conference-2016/sites/default/files/slides/
2016-04-21_plmce_mysql_parallel_replication-inventory_use-
case_and_limitations.pdf
45. CONNECTOR TUNING
• Connectors can also be tuned.
• JDBC property for maximum performance :
• userConfigs=maxPerformance
• Use if the server configuration is stable
• Removes frequent
• SHOW COLLATION
• SHOW GLOBAL VARIABLES
• Fast validation query : /* ping */
46. APPLICATION DESIGN 1/2
• Schema design
• Indexes at the right place
• Remove redundant indexes
• Reduce rows examined
• Reduce sent rows
• Minimize locking
• Minimize temporary tables (on disk)
• Minimize sorting on disk
47. APPLICATION DESIGN 2/2
• Avoid long running transactions
• Close prepare statements
• Close idle connection
• Do not use the information_schema in your app
• Views may not not good for performance
– temporary tables (on disk)
• Replace truncate with drop table / create table
• Tune the replication thread
• Cache data in the app
• Scale out, shard
48. SCHEMA DESIGN
• create a PK for each table !
• integer primary keys
• avoid varchar, composite for PK
• latin1 vs. utf8 vs. utf8mb4
• the smallest varchar for a column
• smallest data type for a column
• keep the number of partitions low (< 100, optimal
performance <10) in 5.6. No more an issue in 5.7.
• use compression for blob / text data types
49. INDEXES
• Fast path to data
• B-TREE, R-TREE (spatial), full text
• for sorting / grouping
• without temporary table
• covering indexes
• contain all the selected data
• save access to full record
• reduce random reads
50. REDUNDANT INDEXES
• Not good for write performance
• duplicated data
• resources to update
• confuse the optimizer
• Use SYS schema views
• schema_unused_indexes
51. REDUCE ROWS_EXAMINED
• Rows read from the storage engines
• Rows_examined
• slow query log
• P_S statement digests
• SYS schema
• Handler%
• show session status where variable_name like ‘Handler%’ or
variable_name like ‘%tmp%’;
• optimize if rows_examined > 10 * rows_sent
• usually due to missing indexes
52. REDUCE ROWS_SENT
• Found in the slow query log, the SYS schema
• Number of rows that are returned by queries to the clients
• rows_sent <= rows_examined
• Network / CPU expensive
• Client compression can help.
• Usually bad design.
• Use caching or LIMIT for UI
• No human can parse 10k rows
53. REDUCE LOCKING
• Locking has a performance impact because locks kept are in memory
• Can be seen in show engine innodb status
• UPDATE, SELECT FOR UPDATE, DELETE, INSERT SELECT
• Use a PK ref, UK ref to lock
• Avoid large index range and table scans
• Reduce rows_examined for locking SQL
• Commit when possible
54. TEMPORARY TABLES (ON DISK)
• Large temporary tables on disk
• handler_write (handler_tmp_write, handler_tmp_update in
MariaDB)
• created_tmp_disk_tables
• monitor tmpdir usage
• Frequent temporary tables on disk
• High created_tmp_disk_tables / uptime
• show global status like '%tmp%';
• Available in the SYS schema, select * from sys.statement_analysis
55. MIND THE SORT (ON DISK)
• Key status variable is :
• sort_merge_passes : it is a session variable
• if it occurs often, you can try to up sort_buffer_size
• find the query and fix it with an the index if possible
• http://dev.mysql.com/doc/refman/5.7/en/order-by-
optimization.html
56. LONG RUNNING (IDLE) TRANSACTIONS
• Usually the oldest transactions in show engine innodb status :
• High history_list_length (a status variable in MariaDB)
• Prevent the purge
• Decrease performance
• Can also prevent schema changes (due to MDL locks)
• Can block backups (FTWRL see MDEV-12620)
59. MISCELLANEOUS
• Do not use the information_schema in your App
• Replace truncate with drop table / create table due to
• Bug #68184 Truncate table causes innodb stalls, fixed in MySQL
8.0
• Views may not be good for performance
• temporary tables (on disk)
• joining other views
• Scale out, shard
60. CACHE DATA IN THE APP
• Good for CPU / IO
• Cache the immutable or the expiring !
• referential data
• memcached / redis
• Query cache can be disabled
• Identify frequent statements
• perl mysqldumpslow.pl –s c slow60s.log
• pt-statement-digest
• Possibly use proxies such as ProxySQL, MaxScale
61. MONITOR THE REPLICATION THREADS
• Slow query log with
• log-slow-slave-statements is now dynamic (Bug #59860) from
5.6.11
• MySQL and Percona only
• Performance_schema >= 5.6.14
• binlog_format = ROW
• show global status like ‘Handler%’
• SYS (on table scans), detect tables not having PK
• In case of issue with a particular table, check the MDL locks.
• MariaDB and 5.7 have a way to see them
62. MONITOR / MAINTAIN
• Monitor the database and OS
• Mine the slow query log
• Use the performance_schema and SYS schema
• Backup the database
• Upgrade regularly
63. MONITOR THE DATABASE
• essential for any professional DBA
• part of the tuning and change processes
• alerts
• graphs
• availability and SLAs
• the effect of tuning
• query analysis
64. MINE THE SLOW QUERY LOG
• Dynamic collection
• The right interval
• Top queries
• pt-schema-digest
• Sort by query time desc
• perl mysqldumpslow.pl –s t slow.log
• Sort by rows_examined desc
• Top queries at the 60s range
65. USE THE SYS SCHEMA
• the SYS schema:
• good entry point
• ready to use views
• IO / latency / waits / statement digests
• ideal for dev and staging
• https://github.com/mysql/mysql-sys
• sys 5.6 works with MariaDB 10.1
• overhead is acceptable in most cases (5% for P_S)
66. BACKUP THE DATABASE
• Backup is always required !
• Method depends on your business : logical vs. physical
• Verify the backup
• Decrease the overhead on prod
• LVM has an overhead
• mysqldump eats MySQL resources
• mysqlbackup / xtrabackup copy the data files and verify it
(in parallel)
67. OPTIMIZE THE DATABASE
• Fragmentation has an impact on performance
• internal fragmentation (inside table spaces)
• external fragmentation (on the file system)
• OPTIMIZE TABLE fixes it (see also Bug #57583)
• can be done online
• MariaDB has the iterative defragmentation patch from
Facebook / Kakao.
• https://mariadb.com/kb/en/defragmenting-innodb-
tablespaces/
68. MONITORING FRAGMENTATION
• There is no general formula
• except for fixed length records
• create table t_defrag like t;
insert into t_defrag select * from t limit 20000;
• Fragmentation if Avg_row_length(t) >
Avg_row_length(t_defrag)
• Avg_row_length from show table status
69. MONITORING FRAGMENTATION
• There is no general formula
• except for fixed length records
• create table t_defrag like t;
insert into t_defrag select * from t limit 20000;
• Fragmentation if Avg_row_length(t) >
Avg_row_length(t_defrag)
• Avg_row_length from show table status
70. UPGRADE POLICY
• Security vulnerability fixes
• Bug fixes
• Performance improvements
• Ready for the next GA
• Never upgrade without testing (see change process)
• can be automated
• the DBA is also a QA engineer !
71. POINTERS
• The MySQL manual
• bugs.mysql.com
• Percona’s blog
• https://planet.mysql.com/
• 50 tips to improve MySQL performance (2014)
• Dimitrik’s latest slides and blog
• Bill Karwin’s Tuning MySQL Its about performance (2015)
• Bill Karwin’s SQL anti-patterns
• https://www.percona.com/live/17/resources/slides are freely available !!!
• FOSDEM videos https://video.fosdem.org/2017/H.1309/
• MariaDB
• My blog : http://aadant.com/blog