MySQL Administrator
Basic course
- MySQL 개요
- MySQL 설치 / 설정
- MySQL 아키텍처 - MySQL 스토리지 엔진
- MySQL 관리
- MySQL 백업 / 복구
- MySQL 모니터링
Advanced course
- MySQL Optimization
- MariaDB / Percona
- MySQL HA (High Availability)
- MySQL troubleshooting
네오클로바
http://neoclova.co.kr/
MySQL Administrator
Basic course
- MySQL 개요
- MySQL 설치 / 설정
- MySQL 아키텍처 - MySQL 스토리지 엔진
- MySQL 관리
- MySQL 백업 / 복구
- MySQL 모니터링
Advanced course
- MySQL Optimization
- MariaDB / Percona
- MySQL HA (High Availability)
- MySQL troubleshooting
네오클로바
http://neoclova.co.kr/
Robert Haas
Why does my query need a plan? Sequential scan vs. index scan. Join strategies. Join reordering. Joins you can't reorder. Join removal. Aggregates and DISTINCT. Using EXPLAIN. Row count and cost estimation. Things the query planner doesn't understand. Other ways the planner can fail. Parameters you can tune. Things that are nearly always slow. Redesigning your schema. Upcoming features and future work.
MySQL users commonly ask: Here's my table, what indexes do I need? Why aren't my indexes helping me? Don't indexes cause overhead? This talk gives you some practical answers, with a step by step method for finding the queries you need to optimize, and choosing the best indexes for them.
This tutorial covers all parallel replication implementation in MariaDB 10.0 and 10.1 and MySQL 5.6, 5.7 and 8.0 (including how it works in Group Replication).
MySQL and MariaDB have different types of parallel replication. In this tutorial, we present the different implementations that allow us to understand their limitations and tuning parameters. We cover how to make parallel replication faster and what to avoid for maximizing its benefits. We also present tests from Booking.com workloads.
Some of the subjects that are covered are group commit and optimistic parallel replication in MariaDB, the parallelism interval of MySQL and its Write Set optimization, and the ?slowing down the master to speed up the slave? optimization.
After this tutorial, you will know everything you need to implement and tune parallel replication in your environment. But more importantly, we will show how you can test parallel replication benefit in a non-disruptive way before deployment.
This presentation focuses on optimization of queries in MySQL from developer’s perspective. Developers should care about the performance of the application, which includes optimizing SQL queries. It shows the execution plan in MySQL and explain its different formats - tabular, TREE and JSON/visual explain plans. Optimizer features like optimizer hints and histograms as well as newer features like HASH joins, TREE explain plan and EXPLAIN ANALYZE from latest releases are covered. Some real examples of slow queries are included and their optimization explained.
PostgreSQL (or Postgres) began its life in 1986 as POSTGRES, a research project of the University of California at Berkeley.
PostgreSQL isn't just relational, it's object-relational.it's object-relational. This gives it some advantages over other open source SQL databases like MySQL, MariaDB and Firebird.
Advanced MySQL Query Tuning - talk at Percona Live and MySQL Meetup tour.
Tuning Queries and Schema/Indexes can significantly increase performance of your application and decrease response times.
This year I will cover new MySQL 5.6 and 5.7 algorithms that has been designed to improve query performance and simply tuning.
Topics:
1. Group by and order by optimizations
2. MySQL temporary tables and filesort
3. Using covered indexes to optimize your queries
4. Loose and tight index scan in MySQL
5. Using summary tables to optimize your reporting queries
6. New MySQL 5.6 and 5.7 Optimizer features and improvements
You deployed automation, enabled automatic database master failover and tested it many times: great, you can now sleep at night without being paged by a failing server. However, when you wake up in the morning, things might not have gone the way you expect. This talk will be about such a surprise.
Once upon a time, a failure brought down a MySQL master database. Automation kicked in and fixed things. However, a fancy failure, combined with human errors, an edge-case recovery, and a lack of oversight in tooling and scripting lead to a split-brain and data corruption. This talk will go into details about the convoluted—but still real-world—sequence of events that lead to this disaster. I cover what could have avoided the split-brain and what could have made data reconciliation easier.
MySQL Parallel Replication: All the 5.7 and 8.0 Details (LOGICAL_CLOCK)Jean-François Gagné
To get better replication speed and less lag, MySQL implements parallel replication in the same schema, also known as LOGICAL_CLOCK. But fully benefiting from this feature is not as simple as just enabling it.
In this talk, I explain in detail how this feature works. I also cover how to optimize parallel replication and the improvements made in MySQL 8.0 and back-ported in 5.7 (Write Sets), greatly improving the potential for parallel execution on replicas (but needing RBR).
Come to this talk to get all the details about MySQL 5.7 and 8.0 Parallel Replication.
Replication Troubleshooting in Classic VS GTIDMydbops
This presentation talk will assist you in troubleshooting MySQL replication for the most common issues we might face with a simple comparison of how can we get them solved in the different replication methods (Classic VS GTID).
MySQL Performance Tuning: Top 10 Tips intended for PHP, Ruby and Java developers on performance tuning and optimization of MySQL. We will cover the deadly mistakes to be avoided. We will take real life examples of optimizing application many times. Here is the summary of what we intend to cover:
• Selection of Storage Engine
• Schema Optimization
• Server Tuning
• Hardware Selection and Tuning
• Effective uses of Index, when to use and when not to use.
• Partitions
• Speeding up using Stored Procedures
• Implementing prepared statements?
• Deadly Sins to be avoided
• Performance Tuning and Benchmarking Tools
Josh Berkus
You've heard that PostgreSQL is the highest-performance transactional open source database, but you're not seeing it on YOUR server. In fact, your PostgreSQL application is kind of poky. What should you do? While doing advanced performance engineering for really high-end systems takes years to learn, you can learn the basics to solve performance issues for 80% of PostgreSQL installations in less than an hour. In this session, you will learn: -- The parts of database application performance -- The performance setup procedure -- Basic troubleshooting tools -- The 13 postgresql.conf settings you need to know -- Where to look for more information.
Best Practices for Becoming an Exceptional Postgres DBA EDB
Drawing from our teams who support hundreds of Postgres instances and production database systems for customers worldwide, this presentation provides real-real best practices from the nation's top DBAs. Learn top-notch monitoring and maintenance practices, get resource planning advice that can help prevent, resolve, or eliminate common issues, learning top database tuning tricks for increasing system performance and ultimately, gain greater insight into how to improve your effectiveness as a DBA.
Robert Haas
Why does my query need a plan? Sequential scan vs. index scan. Join strategies. Join reordering. Joins you can't reorder. Join removal. Aggregates and DISTINCT. Using EXPLAIN. Row count and cost estimation. Things the query planner doesn't understand. Other ways the planner can fail. Parameters you can tune. Things that are nearly always slow. Redesigning your schema. Upcoming features and future work.
MySQL users commonly ask: Here's my table, what indexes do I need? Why aren't my indexes helping me? Don't indexes cause overhead? This talk gives you some practical answers, with a step by step method for finding the queries you need to optimize, and choosing the best indexes for them.
This tutorial covers all parallel replication implementation in MariaDB 10.0 and 10.1 and MySQL 5.6, 5.7 and 8.0 (including how it works in Group Replication).
MySQL and MariaDB have different types of parallel replication. In this tutorial, we present the different implementations that allow us to understand their limitations and tuning parameters. We cover how to make parallel replication faster and what to avoid for maximizing its benefits. We also present tests from Booking.com workloads.
Some of the subjects that are covered are group commit and optimistic parallel replication in MariaDB, the parallelism interval of MySQL and its Write Set optimization, and the ?slowing down the master to speed up the slave? optimization.
After this tutorial, you will know everything you need to implement and tune parallel replication in your environment. But more importantly, we will show how you can test parallel replication benefit in a non-disruptive way before deployment.
This presentation focuses on optimization of queries in MySQL from developer’s perspective. Developers should care about the performance of the application, which includes optimizing SQL queries. It shows the execution plan in MySQL and explain its different formats - tabular, TREE and JSON/visual explain plans. Optimizer features like optimizer hints and histograms as well as newer features like HASH joins, TREE explain plan and EXPLAIN ANALYZE from latest releases are covered. Some real examples of slow queries are included and their optimization explained.
PostgreSQL (or Postgres) began its life in 1986 as POSTGRES, a research project of the University of California at Berkeley.
PostgreSQL isn't just relational, it's object-relational.it's object-relational. This gives it some advantages over other open source SQL databases like MySQL, MariaDB and Firebird.
Advanced MySQL Query Tuning - talk at Percona Live and MySQL Meetup tour.
Tuning Queries and Schema/Indexes can significantly increase performance of your application and decrease response times.
This year I will cover new MySQL 5.6 and 5.7 algorithms that has been designed to improve query performance and simply tuning.
Topics:
1. Group by and order by optimizations
2. MySQL temporary tables and filesort
3. Using covered indexes to optimize your queries
4. Loose and tight index scan in MySQL
5. Using summary tables to optimize your reporting queries
6. New MySQL 5.6 and 5.7 Optimizer features and improvements
You deployed automation, enabled automatic database master failover and tested it many times: great, you can now sleep at night without being paged by a failing server. However, when you wake up in the morning, things might not have gone the way you expect. This talk will be about such a surprise.
Once upon a time, a failure brought down a MySQL master database. Automation kicked in and fixed things. However, a fancy failure, combined with human errors, an edge-case recovery, and a lack of oversight in tooling and scripting lead to a split-brain and data corruption. This talk will go into details about the convoluted—but still real-world—sequence of events that lead to this disaster. I cover what could have avoided the split-brain and what could have made data reconciliation easier.
MySQL Parallel Replication: All the 5.7 and 8.0 Details (LOGICAL_CLOCK)Jean-François Gagné
To get better replication speed and less lag, MySQL implements parallel replication in the same schema, also known as LOGICAL_CLOCK. But fully benefiting from this feature is not as simple as just enabling it.
In this talk, I explain in detail how this feature works. I also cover how to optimize parallel replication and the improvements made in MySQL 8.0 and back-ported in 5.7 (Write Sets), greatly improving the potential for parallel execution on replicas (but needing RBR).
Come to this talk to get all the details about MySQL 5.7 and 8.0 Parallel Replication.
Replication Troubleshooting in Classic VS GTIDMydbops
This presentation talk will assist you in troubleshooting MySQL replication for the most common issues we might face with a simple comparison of how can we get them solved in the different replication methods (Classic VS GTID).
MySQL Performance Tuning: Top 10 Tips intended for PHP, Ruby and Java developers on performance tuning and optimization of MySQL. We will cover the deadly mistakes to be avoided. We will take real life examples of optimizing application many times. Here is the summary of what we intend to cover:
• Selection of Storage Engine
• Schema Optimization
• Server Tuning
• Hardware Selection and Tuning
• Effective uses of Index, when to use and when not to use.
• Partitions
• Speeding up using Stored Procedures
• Implementing prepared statements?
• Deadly Sins to be avoided
• Performance Tuning and Benchmarking Tools
Josh Berkus
You've heard that PostgreSQL is the highest-performance transactional open source database, but you're not seeing it on YOUR server. In fact, your PostgreSQL application is kind of poky. What should you do? While doing advanced performance engineering for really high-end systems takes years to learn, you can learn the basics to solve performance issues for 80% of PostgreSQL installations in less than an hour. In this session, you will learn: -- The parts of database application performance -- The performance setup procedure -- Basic troubleshooting tools -- The 13 postgresql.conf settings you need to know -- Where to look for more information.
Best Practices for Becoming an Exceptional Postgres DBA EDB
Drawing from our teams who support hundreds of Postgres instances and production database systems for customers worldwide, this presentation provides real-real best practices from the nation's top DBAs. Learn top-notch monitoring and maintenance practices, get resource planning advice that can help prevent, resolve, or eliminate common issues, learning top database tuning tricks for increasing system performance and ultimately, gain greater insight into how to improve your effectiveness as a DBA.
This presentation explores a broad cross-section of enterprise Postgres deployments to identify key usage patterns and reveals important aspects of performance, scalability, and availability including:
* Challenges organizations encounter most frequently during the stages of database development, deployment and maintenance
* Tuning parameters used most frequently to improve performance of production databases
* Frequently problematic database maintenance processes and configuration parameters
* Most commonly-used database back-up and recovery strategies
Spencer Christensen
There are many aspects to managing an RDBMS. Some of these are handled by an experienced DBA, but there are a good many things that any sys admin should be able to take care of if they know what to look for.
This presentation will cover basics of managing Postgres, including creating database clusters, overview of configuration, and logging. We will also look at tools to help monitor Postgres and keep an eye on what is going on. Some of the tools we will review are:
* pgtop
* pg_top
* pgfouine
* check_postgres.pl.
Check_postgres.pl is a great tool that can plug into your Nagios or Cacti monitoring systems, giving you even better visibility into your databases.
The paperback version is available on lulu.com there http://goo.gl/fraa8o
This is the first volume of the postgresql database administration book. The book covers the steps for installing, configuring and administering a PostgreSQL 9.3 on Linux debian. The book covers the logical and physical aspect of PostgreSQL. Two chapters are dedicated to the backup/restore topic.
Magnus Hagander
PostgreSQL supports several options for securing communications when deployed outside the typical webserver/database combination. This talk will go into some details about the features that make this possible, with some extra focus on the changes in 8.4. The main areas discussed are:
* Securing the channel between client and server using SSL, including an overview of the threats and how to secure against them
* Securing the login process, using LDAP, Kerberos or SSL certificates, including the use of smartcards to log into the database
The talk will not focus on security and access control inside the database once the user is connected and authenticated.
Best Practices for a Complete Postgres Enterprise Architecture SetupEDB
This presentation provides the details of a best-practice reference architecture for deploying Postgres into your enterprise for large scale OLTP solutions. It reviews how to put all the key pieces together to build a robust, reliable and cost-effective Postgres infrastructure, providing recommendations for configuration and deployment guidance.
This presentation reviews:
* Standard requirements for robust and reliable OTLP architecture
* How to use open source based Postgres Plus building blocks to meet those requirements
* High availability system design with streaming replication
* Backup with logical and physical backup recommendations and setup for point-in-time recovery
* Replication – single master and multi-master considerations
* Database infrastructure monitoring with alerts
* Managing and tuning your Postgres database configuration
To listen to the recording visit www.enterprisedb.com - Resources - Webcasts - On Demand webcasts
Email sales@enterprisedb.com with your questions about Postgres.
These are the slides used by Dilip Kumar of EnterpriseDB for his presentation at pgDay Asia 2016, Singpaore. He talked about scalability and performance improvements in PostgreSQL v9.6, which is expected to be released in Dec/2016 - Jan/2017.
This presentation covers all aspects of PostgreSQL administration, including installation, security, file structure, configuration, reporting, backup, daily maintenance, monitoring activity, disk space computations, and disaster recovery. It shows how to control host connectivity, configure the server, find the query being run by each session, and find the disk space used by each database.
Review this presentation to discover the whens and hows of scaling at it provides and overview of multi-session, single-session, and multi-host scaling options and the workloads where these options are appropriate.
Database scaling refers to increasing database throughput. Vertical scaling utilizes additional resources such as I/O, memory, CPU, while horizontal scaling uses additional computers. However, the high concurrency and write requirements of database servers make scaling a challenge. Sometimes scaling is only possible with multiple sessions, while other options require data model adjustments or server configuration changes.
To learn more please visit www.EnterpriseDB.com or contact sales@enterprisedb.com.
Triggers are those little bits of code running in your database that gets executed when something happens that you care about. Whether you are a developer who puts all of your business logic inside of PL/pgSQL functions or someone who uses an ORM and wants to stay away from database code, you will likely end up using triggers at some point. The fact that the most recommend way of implementing table partitioning in PostgreSQL uses triggers accounts for the importance of understanding triggers.
In this talk, we will step through examples of writing various types of triggers using practical uses cases like partitioning and auditing.
The structure of a trigger
BEFORE vs AFTER triggers
Statement Level vs Row Level triggers
Conditional triggers
Event triggers
Debugging triggers
Performance overhead of triggers
All of the examples will be done using PL/pgSQL so in addition to getting an overview of triggers, you will also get a good understanding of how to code in PL/pgSQL.
How to convert from MySQL to PostgreSQL: discuss history of each, current status, when you might wish to convert, what might motivate you to convert, & how to do so. With references.
сравнение производительности СУБД MySQL и PostgreSQL для "типичной задачи стартапа".
Презентация сопровождала тестовую online-сессию и потому не содержит результатов тестирования.
For years, the common industry perception has been that MySQL is faster and easier to use than PostgreSQL. PostgreSQL is perceived as more powerful, more focused on data integrity, and stricter at complying with SQL specifications, but correspondingly slower and more complicated to use.
Like many perceptions formed in the past, these things aren\'t as true with the current generation of releases as they used to be.
Screenshots to accompany the ETL Assistant article posted on LinkedIn for Eric Whitley. ETL Assistant automates the heavy lifting for importing data from ODBC/OLEDB-compliant data sources into Microsoft SQL Server
Part2 Best Practices for Managing Optimizer StatisticsMaria Colgan
Part 2 of the SQL Tuning workshop focuses on Optimizer Statistics and the best practices for managing them, including when and how to gather statistics. It also covers what additional information you may need to give the Optimizer and provides guidance on when not to gather statistics. Finally we look at all of the techniques you can use to speed up statistics gathering including taking advantage of Incremental statistics, parallelism and concurrency.
Data Warehouse Physical Design,Physical Data Model, Tablespaces, Integrity Constraints, ETL (Extract-Transform-Load) ,OLAP Server Architectures, MOLAP vs. ROLAP, Distributed Data Warehouse ,
MemSQL 201: Advanced Tips and Tricks WebcastSingleStore
Topics discussed include differences between columnstore and rowstore engines, data ingestion, data sharding and query tuning, lastly memory and workload management.
Watch the replay at https://memsql.wistia.com/medias/4siccvlorm
концепт и архитектура геймплея в Creach: The Depleted WorldSperasoft
Presentation by Evgeniy Muralev (Sperasoft) and Konstantin Muralev (Trace studio) during Unreal Engine 4 MeetUp at Sperasoft office in St.Petersburg
April 8th, 2017
Sperasoft talks about several important aspects of ECMAScript6 - language widely used for client-side scripting on the web, in the form of several well-known implementations such as JavaScript, JScript and ActionScript.
GraphRAG is All You need? LLM & Knowledge GraphGuy Korland
Guy Korland, CEO and Co-founder of FalkorDB, will review two articles on the integration of language models with knowledge graphs.
1. Unifying Large Language Models and Knowledge Graphs: A Roadmap.
https://arxiv.org/abs/2306.08302
2. Microsoft Research's GraphRAG paper and a review paper on various uses of knowledge graphs:
https://www.microsoft.com/en-us/research/blog/graphrag-unlocking-llm-discovery-on-narrative-private-data/
Generating a custom Ruby SDK for your web service or Rails API using Smithyg2nightmarescribd
Have you ever wanted a Ruby client API to communicate with your web service? Smithy is a protocol-agnostic language for defining services and SDKs. Smithy Ruby is an implementation of Smithy that generates a Ruby SDK using a Smithy model. In this talk, we will explore Smithy and Smithy Ruby to learn how to generate custom feature-rich SDKs that can communicate with any web service, such as a Rails JSON API.
GDG Cloud Southlake #33: Boule & Rebala: Effective AppSec in SDLC using Deplo...James Anderson
Effective Application Security in Software Delivery lifecycle using Deployment Firewall and DBOM
The modern software delivery process (or the CI/CD process) includes many tools, distributed teams, open-source code, and cloud platforms. Constant focus on speed to release software to market, along with the traditional slow and manual security checks has caused gaps in continuous security as an important piece in the software supply chain. Today organizations feel more susceptible to external and internal cyber threats due to the vast attack surface in their applications supply chain and the lack of end-to-end governance and risk management.
The software team must secure its software delivery process to avoid vulnerability and security breaches. This needs to be achieved with existing tool chains and without extensive rework of the delivery processes. This talk will present strategies and techniques for providing visibility into the true risk of the existing vulnerabilities, preventing the introduction of security issues in the software, resolving vulnerabilities in production environments quickly, and capturing the deployment bill of materials (DBOM).
Speakers:
Bob Boule
Robert Boule is a technology enthusiast with PASSION for technology and making things work along with a knack for helping others understand how things work. He comes with around 20 years of solution engineering experience in application security, software continuous delivery, and SaaS platforms. He is known for his dynamic presentations in CI/CD and application security integrated in software delivery lifecycle.
Gopinath Rebala
Gopinath Rebala is the CTO of OpsMx, where he has overall responsibility for the machine learning and data processing architectures for Secure Software Delivery. Gopi also has a strong connection with our customers, leading design and architecture for strategic implementations. Gopi is a frequent speaker and well-known leader in continuous delivery and integrating security into software delivery.
LF Energy Webinar: Electrical Grid Modelling and Simulation Through PowSyBl -...DanBrown980551
Do you want to learn how to model and simulate an electrical network from scratch in under an hour?
Then welcome to this PowSyBl workshop, hosted by Rte, the French Transmission System Operator (TSO)!
During the webinar, you will discover the PowSyBl ecosystem as well as handle and study an electrical network through an interactive Python notebook.
PowSyBl is an open source project hosted by LF Energy, which offers a comprehensive set of features for electrical grid modelling and simulation. Among other advanced features, PowSyBl provides:
- A fully editable and extendable library for grid component modelling;
- Visualization tools to display your network;
- Grid simulation tools, such as power flows, security analyses (with or without remedial actions) and sensitivity analyses;
The framework is mostly written in Java, with a Python binding so that Python developers can access PowSyBl functionalities as well.
What you will learn during the webinar:
- For beginners: discover PowSyBl's functionalities through a quick general presentation and the notebook, without needing any expert coding skills;
- For advanced developers: master the skills to efficiently apply PowSyBl functionalities to your real-world scenarios.
Transcript: Selling digital books in 2024: Insights from industry leaders - T...BookNet Canada
The publishing industry has been selling digital audiobooks and ebooks for over a decade and has found its groove. What’s changed? What has stayed the same? Where do we go from here? Join a group of leading sales peers from across the industry for a conversation about the lessons learned since the popularization of digital books, best practices, digital book supply chain management, and more.
Link to video recording: https://bnctechforum.ca/sessions/selling-digital-books-in-2024-insights-from-industry-leaders/
Presented by BookNet Canada on May 28, 2024, with support from the Department of Canadian Heritage.
The Art of the Pitch: WordPress Relationships and SalesLaura Byrne
Clients don’t know what they don’t know. What web solutions are right for them? How does WordPress come into the picture? How do you make sure you understand scope and timeline? What do you do if sometime changes?
All these questions and more will be explored as we talk about matching clients’ needs with what your agency offers without pulling teeth or pulling your hair out. Practical tips, and strategies for successful relationship building that leads to closing the deal.
DevOps and Testing slides at DASA ConnectKari Kakkonen
My and Rik Marselis slides at 30.5.2024 DASA Connect conference. We discuss about what is testing, then what is agile testing and finally what is Testing in DevOps. Finally we had lovely workshop with the participants trying to find out different ways to think about quality and testing in different parts of the DevOps infinity loop.
Connector Corner: Automate dynamic content and events by pushing a buttonDianaGray10
Here is something new! In our next Connector Corner webinar, we will demonstrate how you can use a single workflow to:
Create a campaign using Mailchimp with merge tags/fields
Send an interactive Slack channel message (using buttons)
Have the message received by managers and peers along with a test email for review
But there’s more:
In a second workflow supporting the same use case, you’ll see:
Your campaign sent to target colleagues for approval
If the “Approve” button is clicked, a Jira/Zendesk ticket is created for the marketing design team
But—if the “Reject” button is pushed, colleagues will be alerted via Slack message
Join us to learn more about this new, human-in-the-loop capability, brought to you by Integration Service connectors.
And...
Speakers:
Akshay Agnihotri, Product Manager
Charlie Greenberg, Host
Elevating Tactical DDD Patterns Through Object CalisthenicsDorra BARTAGUIZ
After immersing yourself in the blue book and its red counterpart, attending DDD-focused conferences, and applying tactical patterns, you're left with a crucial question: How do I ensure my design is effective? Tactical patterns within Domain-Driven Design (DDD) serve as guiding principles for creating clear and manageable domain models. However, achieving success with these patterns requires additional guidance. Interestingly, we've observed that a set of constraints initially designed for training purposes remarkably aligns with effective pattern implementation, offering a more ‘mechanical’ approach. Let's explore together how Object Calisthenics can elevate the design of your tactical DDD patterns, offering concrete help for those venturing into DDD for the first time!
JMeter webinar - integration with InfluxDB and GrafanaRTTS
Watch this recorded webinar about real-time monitoring of application performance. See how to integrate Apache JMeter, the open-source leader in performance testing, with InfluxDB, the open-source time-series database, and Grafana, the open-source analytics and visualization application.
In this webinar, we will review the benefits of leveraging InfluxDB and Grafana when executing load tests and demonstrate how these tools are used to visualize performance metrics.
Length: 30 minutes
Session Overview
-------------------------------------------
During this webinar, we will cover the following topics while demonstrating the integrations of JMeter, InfluxDB and Grafana:
- What out-of-the-box solutions are available for real-time monitoring JMeter tests?
- What are the benefits of integrating InfluxDB and Grafana into the load testing stack?
- Which features are provided by Grafana?
- Demonstration of InfluxDB and Grafana using a practice web application
To view the webinar recording, go to:
https://www.rttsweb.com/jmeter-integration-webinar
Builder.ai Founder Sachin Dev Duggal's Strategic Approach to Create an Innova...Ramesh Iyer
In today's fast-changing business world, Companies that adapt and embrace new ideas often need help to keep up with the competition. However, fostering a culture of innovation takes much work. It takes vision, leadership and willingness to take risks in the right proportion. Sachin Dev Duggal, co-founder of Builder.ai, has perfected the art of this balance, creating a company culture where creativity and growth are nurtured at each stage.
2. Here’s a classic scenario. You work on a project that stores data in a
relational database. The application gets deployed to production
and early on the performance is great, selecting data from the
database is snappy and insert latency goes unnoticed. Over a time
period of days/weeks/months the database starts to get bigger and
queries slow down.
3. A Database Administrator (DBA) will take a look and see that the database is
tuned. They offer suggestions to add certain indexes, move logging to
separate disk partitions, adjust database engine parameters and verify that
the database is healthy. This will buy you more time and may resolve this
issues to a degree.
At a certain point you realize the data in the database is the bottleneck.
There are various approaches that can help you make your application and
database run faster. Let’s take a look at two of them:
• Table partitioning
• Aggregated data tables
4. Main idea: you take one massive table (master table) and split it into many
smaller tables – these smaller tables are called partitions or child tables.
5. Master Table
Also referred to as a Master Partition Table, this table is the template child
tables are created from. This is a normal table, but it doesn’t contain any data
and requires a trigger.
Child Table
These tables inherit their structure (in other words, their Data Definition
Language or DDL for short) from the master table and belong to a single
master table. The child tables contain all of the data. These tables are also
referred to as Table Partitions.
Partition Function
A partition function is a Stored Procedure that determines which child table
should accept a new record. The master table has a trigger which calls a
partition function.
6. Here’s a summary of what should be done:
1.
Create a master table
2.
Create a partition function
3.
Create a table trigger
Let’s assume that we have a rather large table ( ~ 2 500k rows) containing
reports for different dates.
7. There are two typical methodologies for routing records to child tables:
•
By Date Values
•
By Fixed Values
The trigger function does the following:
Creates child table by dynamically generated “CREATE TABLE” statement if
the child table does not exist.
Partitions (child tables) are determined by the values in the “date” column.
One partition per calendar month is created.
The name of each child table will be in the format of
“master_table_name_yyyy-mm”
8. CREATE OR REPLACE FUNCTION partition_function() RETURNS trigger AS
$BODY$
DECLARE
table_master varchar(255) := ‘SOME_LARGE_TABLE';
table_part varchar(255) := ‘';
…
BEGIN
------------------------------------------generate partition name----------------------------------------------------…
table_part := table_master|| '_y' || DATE_PART( 'year', rec_date )::TEXT
|| '_m' || DATE_PART( 'month', rec_date )::TEXT;
-----------------------------------------check if partition already exists--------------------------------------------…
-----------------------------------------if not yet then create new---------------------------------------------------EXECUTE 'CREATE TABLE public.' || quote_ident(table_part) || ' (
CHECK( “RECORD_DATE" >= DATE ' || quote_literal(start_date) || ' AND “RECORD_DATE" <
DATE ' || quote_literal(end_date) || ')) INHERITS ( public.' || quote_ident(table_master) || ')
----------------------------------------create indexes for current partition----------------------------------------EXECUTE 'CREATE INDEX …
END;
$BODY$
LANGUAGE plpgsql VOLATILE
COST 100;
9. Now that the Partition Function has been created an Insert Trigger needs to be
added to the Master Table which will call the partition function when new records
are inserted.
CREATE TRIGGER insert_trigger
BEFORE INSERT
ON “SOME_LARGE_TABLE"
FOR EACH ROW
EXECUTE PROCEDURE partition_function();
At this point you can start inserting rows against the Master Table and see the
rows being inserted into the correct child table.
10. Constraint exclusion is a query optimization technique that improves
performance for partitioned tables
SET constraint_exclusion = on;
The default (and recommended) setting of constraint_exclusion is actually
neither on nor off, but an intermediate setting called partition, which causes the
technique to be applied only to queries that are likely to be working on
partitioned tables. The on setting causes the planner to examine CHECK
constraints in all queries, even simple ones that are unlikely to benefit.
11. SELECT * FROM “SOME_LARGE_TABLE" WHERE “ID" = '0000e124-e7ff-4859-8d4fa3d7b37b521b' AND “RECORD_DATE" BETWEEN '2013-10-01' AND '2013-10-30';
Without partitioning:
With partitioning:
12. Benefits:
• Query performance can be improved dramatically in certain situations;
• Bulk loads and deletes can be accomplished by adding or removing
partitions;
• Seldom-used data can be migrated to cheaper and slower storage media.
Caveats:
• Partitioning should be organized so that queries reference as few tables as
possible.
• The partition key column(s) of a row should never change, or at least do not
change enough to require it to move to another partition.
• Constraint exclusion only works when the query's WHERE clause contains
constants.
• All constraints on all partitions of the master table are examined during
constraint exclusion, so large numbers of partitions are likely to increase
query planning time considerably.
13. Another approach to boost performance is using pre-aggregated data.
One real feature of relational databases is that complex objects are built from
their atomic components at runtime, but this can cause excessive stress if the
same things are being done, over and over.
Without using pre-aggregated data you may see unnecessary repeating largetable full-table scans, as summaries are computed, over and over.
Data aggregation can be used to pre-join tables, presort solution sets, and presummarize complex data information. Because this work is completed in
advance, it gives end users the illusion of instantaneous response time.
14. You can use a set of ordinary tables with triggers and stored procedures for
these purpose but there is another solution available out of the box –
materialized views (PostgreSQL v. 9.3 natively supports materialized views)
A materialized view is a database object that contains the results of a query
Materialized views in PostgreSQL use the rule system like views do, but
persist the results in a table-like form.
Let’s assume that we have a two tables: ‘machines’ (2 abstract machines) and
‘reports’ containing reports for each machine (~100k rows).
15. Let’s create materialized view:
CREATE MATERIALIZED VIEW mvw_reports AS
SELECT reports.id, machines.name || ' ' || machines.location AS
machine_name, reports.reports_qty
FROM reports
INNER JOIN machines ON machines.id = reports.machine_id;
And a simple view for comparison:
CREATE VIEW vw_reports AS
SELECT reports.id, machines.name || ' ' || machines.location AS
machine_name, reports.reports_qty
FROM reports
INNER JOIN machines ON machines.id = reports.machine_id;
16. Executing the same query to simple view:
EXPLAIN ANALYZE SELECT * FROM vw_reports WHERE machines_name = ‘Machine1
Location1';
And for materialized view:
EXPLAIN ANALYZE SELECT * FROM mvw_reports WHERE machines_name = ‘Machine1
Location1';
17. Another advantage compared with simple views is that we can add indexes to
materialized views like for ordinary tables.
CREATE INDEX idx_report_machines_name ON mvw_reports ( machines_name );
Executing the query once more:
EXPLAIN ANALYZE SELECT * FROM mvw_reports WHERE machines_name =
‘Machine1 Location1';
18. In order to have actual data in materialized view it should be refreshed after
each DML operation (INSERT, UPDATE, DELETE) on the target tables.
REFRESH MATERIALIZED VIEW mvw_reports;
This can be done using triggers:
CREATE TRIGGER machines_refresh AFTER INSERT OR UPDATE OR DELETE ON
machines FOR EACH STATEMENT EXECUTE PROCEDURE mvw_reports_refresh( );
CREATE TRIGGER reports_refresh AFTER INSERT OR UPDATE OR DELETE ON
reports FOR EACH STATEMENT EXECUTE PROCEDURE mvw_reports_refresh ( );
19. Benefits:
Query performance can be improved dramatically in situations when there are
relatively few data modifications compared to the queries being performed,
and the queries are very complicated and heavy-weight.
Caveats:
• Materialized views contain a duplicate of data from base tables;
• Depending on the complexity of the underlying query for each MV, and the
amount of data involved, the computation required for refreshing may be
very expensive, and frequent refreshing of MVs may impose an
unacceptable workload on the database server.
20. Table partitioning and aggregated data tables can help a lot. But there is no
ideal solution that always works. Both approaches have their own pluses and
minuses. It all depends on certain situation and circumstances. Hopefully
presented overview gave few tips on when each technique can be useful.
Any questions?