This slide deck describes the Flexviews materialized view toolkit for MySQL:
http://flexvie.ws
Learn how to use incrementally refreshable materialized views, and how they can improve your performance.
This slide deck describes the Flexviews materialized view toolkit for MySQL:
http://flexvie.ws
Learn how to use incrementally refreshable materialized views, and how they can improve your performance.
Flexviews is a materialized view solution for MySQL. This set of slides introduces Flexviews concepts, and gives some examples in how to use it and what to use it for.
MySQL 8.0 New Features -- September 27th presentation for Open Source SummitDave Stokes
MySQL 8.0 has many new features that you probably need to know about but don't. Like default security, window functions, CTEs, CATS (not what you think), JSON_TABLE(), and UTF8MB4 support.
Flexviews is a materialized view solution for MySQL. This set of slides introduces Flexviews concepts, and gives some examples in how to use it and what to use it for.
MySQL 8.0 New Features -- September 27th presentation for Open Source SummitDave Stokes
MySQL 8.0 has many new features that you probably need to know about but don't. Like default security, window functions, CTEs, CATS (not what you think), JSON_TABLE(), and UTF8MB4 support.
We all have tasks from time to time for bulk-loading external data into MySQL. What's the best way of doing this? That's the task I faced recently when I was asked to help benchmark a multi-terrabyte database. We had to find the most efficient method to reload test data repeatedly without taking days to do it each time. In my presentation, I'll show you several alternative methods for bulk data loading, and describe the practical steps to use them efficiently. I'll cover SQL scripts, the mysqlimport tool, MySQL Workbench import, the CSV storage engine, and the Memcached API. I'll also give MySQL tuning tips for data loading, and how to use multi-threaded clients.
Many questions on database newsgroups and forums can be answered with uses of outer joins. Outer joins are part of the standard SQL language and supported by all RDBMS brands. Many programmers are expected to use SQL in their work, but few know how to use outer joins effectively.
Learn to use this powerful feature of SQL, increase your employability, and amaze your friends!
Karwin will explain outer joins, show examples, and demonstrate a Sudoku puzzle solver implemented in a single SQL query.
MySQL High Availability and Disaster Recovery with Continuent, a VMware companyContinuent
Users seeking high availability, disaster recovery and zero downtime maintenance operation for business-critical MySQL applications face confusing choices. Is multi-master or master/slave clustering better? What about synchronous versus asynchronous replication? Using a plain vanilla, stock MySQL or a modified version of it? Which of these choices are right for data-driven businesses that depend on fast, reliable data access?
This no-BS webinar cuts through the FUD to explore the real trade-offs between the different clustering and replication methods, thens show you how Continuent's asynchronous master/slave clusters support these important capabilities for business-critical applications:
- High application write rates Master/slave clustering with Continuent
- Mixed workloads consisting of large and small transactions
- Data across multiple geographically distributed locations
- Failures and more importantly recovery from them
- Zero downtime maintenance and software upgrades
- Use of off-the-shelf MySQL/MariaDB to avoid application changes and allow clusters to improve as MySQL itself does.
We illustrate key points with demonstrations and case studies from deployed systems.
A presentation about how to make MySQL highly available, presented at the San Francisco MySQL Meetup (http://www.sfmysql.org/events/15760472/) on January 26th, 2011.
A video recording of this presentation is available from Ustream: http://ustre.am/fyLk
MySQL High Availability with Group ReplicationNuno Carvalho
MySQL Group Replication is a MySQL plugin that combines distributed systems technologies with database replication. The outcome of fusing such technologies is a resilient, multimaster update-everywhere replication solution for MySQL. At its core is a group communication service providing reliable and dependable message-passing for a group of MySQL servers. This enables automatic and consistent coordination between all the participants in a group. Such automatic orchestration of MySQL servers results in a fault-tolerant replicated state machine providing a highly available MySQL data store.
MySQL Replication Performance Tuning for Fun and Profit!Vitor Oliveira
MySQL Replication, in addition to bringing high-availability, is the foundation to build high-performance MySQL database systems. Using read scale-out and sharding one can design systems that go from the capacity of a single server to supporting the largest internet sites. But to design and operate high-performance, efficient, manageable and reliable deployments requires knowing the intricacies of the underlying technologies.
This session will provide insights on the main factors that affect the performance of Asynchronous Replication and Group Replication, and how to configure them to make the most out of the underlying computing system. It will also show the latest developments in MySQL 5.7 and 8.0, in areas spanning from group communication to the multi-threaded slave applier, and how effective they are in helping meet the performance requirements in terms of throughput, latency and durability to support the most demanding workload types.
MySQL Group Replication - HandsOn TutorialKenny Gryp
During this tutorial, attendees have their hands on virtual machines and migrate standard Master - Slave architecture to the new MySQL native Group Replication.
After explaining briefly what is group replication and how this is important for MySQL HA architecture. We will cover how to verify the workload and the scheme to how GR can be used and configured.
Then we will go trough the migration steps with minimal impact on the live system.
Basic administration tasks are covered such as add/remove a node from the cluster. We also play with performance_schema to monitor our Group Replication cluster and understand how to control it.
The nightmare of locking, blocking and isolation levels!Boris Hristov
am sure you all know that troubleshooting problems related to locking and blocking (hey, sometimes there are deadlocks too) can be a real nightmare! In this session, you will be able to see and understand why and how locking actually works, what problems it causes and how can we use isolation levels and various other techniques to resolve them!
Browser Fuzzing with a Twist (and a Shake) -- ZeroNights 2015Jeremy Brown
The web client is critical software to secure from any perspective. No matter if you're an organization or a casual client, you're typically just as vulnerable as anyone else. OSes are often supplemented with hardening toolsets or built-in mitigations as an extra measure to avoid compromise, but as with all things, they aren't completely solid either. Thus the need for systems that break systems, some of which deploy fuzzing and almost all of them work to find implementation bugs. Browser fuzzing has been explored and improved in many different ways over the past several years. In this presentation, we'll be primarily talking about a mutation engine that provides a somewhat novel technique for finding bugs in a still-ripe attack surface: the browser's rendering engine. This technique has the flexibility to be applied even more broadly than browsers, for example, there's initial support for fuzzing PDF readers. We'll also be discussing the tooling and infrastructure areas of the process, detailing what's needed to build a system that will scale and enable your fuzzing strategies to be successful. Finally, we can conclude the talk with some incubation results and how you can start making use of these fuzzing techniques today to find the bugs you need to exploit browsers or identify and fix the code responsible for each vulnerability.
Application Performance Troubleshooting 1x1 - Part 2 - Noch mehr Schweine und...rschuppe
Application Performance doesn't come easy. How to find the root cause of performance issues in modern and complex applications? All you have is a complaining user to start with?
In this presentation (mainly in German, but understandable for english speakers) I'd reprised the fundamentals of trouble shooting and have some new examples on how to tackle issues.
Follow up presentation to "Performance Trouble Shooting 101 - Schweine, Schlangen und Papierschnitte"
SoftServe's Hadoop Demo Lab - is a project to aggregate log files from 300 Apache HTTPD web-servers and populate them into Hadoop/ElasticSearch cluster for future analysis using Microstrategy and Kibana. It is even more interesting keeping in mind that all deployment is fully automated using Vagrant and Puppet.
Similar to 10x Performance Improvements - A Case Study (20)
Every website wants to become successful. Few websites however undertake the basic and fundamental steps to build a rock solid foundation to ensure a scalable
"Disaster is inevitable" and "To move forward you must first backup" should be known to all software developers. This presentation will discuss all the options for your valuable data assets in MySQL, and highlight how to maintain site reliability of your data
The History and Future of the MySQL ecosystemRonald Bradford
The history and future of the MySQL Ecosystem. This talk sub-titled “Spaghetti and MySQLBalls (with a side of greens)” detailed the beginnings of MySQL, the MySQL acquisition history, described the state of current MySQL versions/variants/forks, storage engines, related vendors, NoSQL and much more.
A video of the presentations is available on YouTube at http://www.youtube.com/watch?v=9mKwkbaB5X8&feature=youtu.be
Lessons Learned Managing Large AWS EnvironmentsRonald Bradford
How to you optimize management of 500+ AWS servers? In this presentation I share my experiences using Amazon Web Servers covering techniques for webscale. Learn how to optimized your cost, handle security, automate and be prepared for handling failure.
Monitoring your technology stack with New RelicRonald Bradford
There is no excuse to not have monitoring of your LAMP stack, NoSQL database like MongoDB/Redis/Cassandra/Memcache, Cloud services and much more when you can use the popular New Relic tool for free. As the MySQL plugin author I can offer the following link will give you access to free monitoring http://j.mp/newrelic-mysql There can never be an excuse to not know how your application is performing, from 1 server to 100+ servers.
While MySQL is a popular and widely used RDBMS, some default features and settings are very foreign in comparison with other commercial RDBMS products. In this discussion, Ronald Bradford will discuss some of the MySQL defaults including a non-transactional state, silent data truncations, date management, and transaction isolation options. These are all critical for data integrity and consistency. He will cover in-depth topics including SQL_MODE that saves the day. He will also cover character sets and collations and the best practices to ensure your UTF8 is stored and retrieved correctly.
Only after a successful preparation covered in IGNITION can you be ready for the implementation and management of a MySQL ecosystem and a successful launch of your product.
We Discuss:
* Escape Options – Before and after backup and recovery situations
* Good to Go – Knowing and confirming your MySQL environment is ready
* Full Throttle – Understanding and Improving MySQL database performance
* A Green Dashboard – Monitoring for Success
* The Human Factor – Nobody is perfect, dealing with the design changes
* Propellant – The murky mess of MySQL versions, patches and variants
* Best Practices – Proven techniques for consistency, automation and reproducibility
Note: This is volume 2 of a two part series
IGNITION is the preparation necessary for a successful launch of a MySQL ecosystem for an Oracle DBA. This volume covers the preparation needed to be ready for ongoing production administration of MySQL.
We discuss:
* Translation – Understanding the MySQL terminology
* Installation – Knowing the options for MySQL distributions
* Protection Detail – Security of MySQL information
* The Dashboard – Understanding what to monitor in MySQL
* Mechanics – Understanding more of MySQL Internals including storage engines
* Redundancy – Maintaining multiple copies via MySQL replication
* Checklists – Double checking and cross referencing your ecosystem
NOTE: This is Volume 1 of a two part series
This presentation discusses the current state of the Drizzle database project including the principles behind this leading open source project, and the healthy and growing community. For more information visit http://drizzle.org
There has been significant movement in recent times towards less structured approaches of storing and retrieving data. No longer the realm of Relational Databases, there is a new crop of structured key/value pair stores and unstructured data offerings. This closing panel debate at Open SQL Camp 2009 discussed the SQL v NoSQL topic.
Example section on MySQL for the Oracle DBA 1 day bootcamp.
In object management we look at the key SQL objects including what differs with Oracle and what is Oracle specific functionality.
We also look at the MySQL data dictionary, the INFORMATION_SCHEMA
Smart TV Buyer Insights Survey 2024 by 91mobiles.pdf91mobiles
91mobiles recently conducted a Smart TV Buyer Insights Survey in which we asked over 3,000 respondents about the TV they own, aspects they look at on a new TV, and their TV buying preferences.
Key Trends Shaping the Future of Infrastructure.pdfCheryl Hung
Keynote at DIGIT West Expo, Glasgow on 29 May 2024.
Cheryl Hung, ochery.com
Sr Director, Infrastructure Ecosystem, Arm.
The key trends across hardware, cloud and open-source; exploring how these areas are likely to mature and develop over the short and long-term, and then considering how organisations can position themselves to adapt and thrive.
"Impact of front-end architecture on development cost", Viktor TurskyiFwdays
I have heard many times that architecture is not important for the front-end. Also, many times I have seen how developers implement features on the front-end just following the standard rules for a framework and think that this is enough to successfully launch the project, and then the project fails. How to prevent this and what approach to choose? I have launched dozens of complex projects and during the talk we will analyze which approaches have worked for me and which have not.
Epistemic Interaction - tuning interfaces to provide information for AI supportAlan Dix
Paper presented at SYNERGY workshop at AVI 2024, Genoa, Italy. 3rd June 2024
https://alandix.com/academic/papers/synergy2024-epistemic/
As machine learning integrates deeper into human-computer interactions, the concept of epistemic interaction emerges, aiming to refine these interactions to enhance system adaptability. This approach encourages minor, intentional adjustments in user behaviour to enrich the data available for system learning. This paper introduces epistemic interaction within the context of human-system communication, illustrating how deliberate interaction design can improve system understanding and adaptation. Through concrete examples, we demonstrate the potential of epistemic interaction to significantly advance human-computer interaction by leveraging intuitive human communication strategies to inform system design and functionality, offering a novel pathway for enriching user-system engagements.
Builder.ai Founder Sachin Dev Duggal's Strategic Approach to Create an Innova...Ramesh Iyer
In today's fast-changing business world, Companies that adapt and embrace new ideas often need help to keep up with the competition. However, fostering a culture of innovation takes much work. It takes vision, leadership and willingness to take risks in the right proportion. Sachin Dev Duggal, co-founder of Builder.ai, has perfected the art of this balance, creating a company culture where creativity and growth are nurtured at each stage.
JMeter webinar - integration with InfluxDB and GrafanaRTTS
Watch this recorded webinar about real-time monitoring of application performance. See how to integrate Apache JMeter, the open-source leader in performance testing, with InfluxDB, the open-source time-series database, and Grafana, the open-source analytics and visualization application.
In this webinar, we will review the benefits of leveraging InfluxDB and Grafana when executing load tests and demonstrate how these tools are used to visualize performance metrics.
Length: 30 minutes
Session Overview
-------------------------------------------
During this webinar, we will cover the following topics while demonstrating the integrations of JMeter, InfluxDB and Grafana:
- What out-of-the-box solutions are available for real-time monitoring JMeter tests?
- What are the benefits of integrating InfluxDB and Grafana into the load testing stack?
- Which features are provided by Grafana?
- Demonstration of InfluxDB and Grafana using a practice web application
To view the webinar recording, go to:
https://www.rttsweb.com/jmeter-integration-webinar
Dev Dives: Train smarter, not harder – active learning and UiPath LLMs for do...UiPathCommunity
💥 Speed, accuracy, and scaling – discover the superpowers of GenAI in action with UiPath Document Understanding and Communications Mining™:
See how to accelerate model training and optimize model performance with active learning
Learn about the latest enhancements to out-of-the-box document processing – with little to no training required
Get an exclusive demo of the new family of UiPath LLMs – GenAI models specialized for processing different types of documents and messages
This is a hands-on session specifically designed for automation developers and AI enthusiasts seeking to enhance their knowledge in leveraging the latest intelligent document processing capabilities offered by UiPath.
Speakers:
👨🏫 Andras Palfi, Senior Product Manager, UiPath
👩🏫 Lenka Dulovicova, Product Program Manager, UiPath
DevOps and Testing slides at DASA ConnectKari Kakkonen
My and Rik Marselis slides at 30.5.2024 DASA Connect conference. We discuss about what is testing, then what is agile testing and finally what is Testing in DevOps. Finally we had lovely workshop with the participants trying to find out different ways to think about quality and testing in different parts of the DevOps infinity loop.
The Art of the Pitch: WordPress Relationships and SalesLaura Byrne
Clients don’t know what they don’t know. What web solutions are right for them? How does WordPress come into the picture? How do you make sure you understand scope and timeline? What do you do if sometime changes?
All these questions and more will be explored as we talk about matching clients’ needs with what your agency offers without pulling teeth or pulling your hair out. Practical tips, and strategies for successful relationship building that leads to closing the deal.
PHP Frameworks: I want to break free (IPC Berlin 2024)Ralf Eggert
In this presentation, we examine the challenges and limitations of relying too heavily on PHP frameworks in web development. We discuss the history of PHP and its frameworks to understand how this dependence has evolved. The focus will be on providing concrete tips and strategies to reduce reliance on these frameworks, based on real-world examples and practical considerations. The goal is to equip developers with the skills and knowledge to create more flexible and future-proof web applications. We'll explore the importance of maintaining autonomy in a rapidly changing tech landscape and how to make informed decisions in PHP development.
This talk is aimed at encouraging a more independent approach to using PHP frameworks, moving towards a more flexible and future-proof approach to PHP development.
GraphRAG is All You need? LLM & Knowledge GraphGuy Korland
Guy Korland, CEO and Co-founder of FalkorDB, will review two articles on the integration of language models with knowledge graphs.
1. Unifying Large Language Models and Knowledge Graphs: A Roadmap.
https://arxiv.org/abs/2306.08302
2. Microsoft Research's GraphRAG paper and a review paper on various uses of knowledge graphs:
https://www.microsoft.com/en-us/research/blog/graphrag-unlocking-llm-discovery-on-narrative-private-data/
Search and Society: Reimagining Information Access for Radical FuturesBhaskar Mitra
The field of Information retrieval (IR) is currently undergoing a transformative shift, at least partly due to the emerging applications of generative AI to information access. In this talk, we will deliberate on the sociotechnical implications of generative AI for information access. We will argue that there is both a critical necessity and an exciting opportunity for the IR community to re-center our research agendas on societal needs while dismantling the artificial separation between the work on fairness, accountability, transparency, and ethics in IR and the rest of IR research. Instead of adopting a reactionary strategy of trying to mitigate potential social harms from emerging technologies, the community should aim to proactively set the research agenda for the kinds of systems we should build inspired by diverse explicitly stated sociotechnical imaginaries. The sociotechnical imaginaries that underpin the design and development of information access technologies needs to be explicitly articulated, and we need to develop theories of change in context of these diverse perspectives. Our guiding future imaginaries must be informed by other academic fields, such as democratic theory and critical theory, and should be co-developed with social science scholars, legal scholars, civil rights and social justice activists, and artists, among others.
State of ICS and IoT Cyber Threat Landscape Report 2024 previewPrayukth K V
The IoT and OT threat landscape report has been prepared by the Threat Research Team at Sectrio using data from Sectrio, cyber threat intelligence farming facilities spread across over 85 cities around the world. In addition, Sectrio also runs AI-based advanced threat and payload engagement facilities that serve as sinks to attract and engage sophisticated threat actors, and newer malware including new variants and latent threats that are at an earlier stage of development.
The latest edition of the OT/ICS and IoT security Threat Landscape Report 2024 also covers:
State of global ICS asset and network exposure
Sectoral targets and attacks as well as the cost of ransom
Global APT activity, AI usage, actor and tactic profiles, and implications
Rise in volumes of AI-powered cyberattacks
Major cyber events in 2024
Malware and malicious payload trends
Cyberattack types and targets
Vulnerability exploit attempts on CVEs
Attacks on counties – USA
Expansion of bot farms – how, where, and why
In-depth analysis of the cyber threat landscape across North America, South America, Europe, APAC, and the Middle East
Why are attacks on smart factories rising?
Cyber risk predictions
Axis of attacks – Europe
Systemic attacks in the Middle East
Download the full report from here:
https://sectrio.com/resources/ot-threat-landscape-reports/sectrio-releases-ot-ics-and-iot-security-threat-landscape-report-2024/
Essentials of Automations: Optimizing FME Workflows with ParametersSafe Software
Are you looking to streamline your workflows and boost your projects’ efficiency? Do you find yourself searching for ways to add flexibility and control over your FME workflows? If so, you’re in the right place.
Join us for an insightful dive into the world of FME parameters, a critical element in optimizing workflow efficiency. This webinar marks the beginning of our three-part “Essentials of Automation” series. This first webinar is designed to equip you with the knowledge and skills to utilize parameters effectively: enhancing the flexibility, maintainability, and user control of your FME projects.
Here’s what you’ll gain:
- Essentials of FME Parameters: Understand the pivotal role of parameters, including Reader/Writer, Transformer, User, and FME Flow categories. Discover how they are the key to unlocking automation and optimization within your workflows.
- Practical Applications in FME Form: Delve into key user parameter types including choice, connections, and file URLs. Allow users to control how a workflow runs, making your workflows more reusable. Learn to import values and deliver the best user experience for your workflows while enhancing accuracy.
- Optimization Strategies in FME Flow: Explore the creation and strategic deployment of parameters in FME Flow, including the use of deployment and geometry parameters, to maximize workflow efficiency.
- Pro Tips for Success: Gain insights on parameterizing connections and leveraging new features like Conditional Visibility for clarity and simplicity.
We’ll wrap up with a glimpse into future webinars, followed by a Q&A session to address your specific questions surrounding this topic.
Don’t miss this opportunity to elevate your FME expertise and drive your projects to new heights of efficiency.
Essentials of Automations: Optimizing FME Workflows with Parameters
10x Performance Improvements - A Case Study
1. 10x Performance Improvements
in 10 steps
A Case Study
Ronald Bradford
http://ronaldbradford.com
FOSDEM - 2010.02
Sunday, February 7, 2010
2. Application
Typical Web 2.0 social media site (Europe based)
• Users - Visitors, Free Members, Paying Members
• Friends
• User Content - Video, Pictures
• Forums, Chat, Email
Sunday, February 7, 2010
3. Server Environment
• 1 Master Database Server (MySQL 5.0.x)
• 3 Slave Database Servers (MySQL 5.0.x)
• 5 Web Servers (Apache/PHP)
• 1 Static Content Server (Nginx)
• 1 Mail Server
Sunday, February 7, 2010
4. Step 1
Monitor, Monitor, Monitor
Sunday, February 7, 2010
6. 1. Monitor, Monitor, Monitor
Action 1
Monitoring Software
• Installation of Cacti http://www.cacti.net/
-
• Installation of MySQL Cacti Templates -
http://code.google.com/p/mysql-cacti-templates/
• (Optional) Installation of MONyog - http://www.webyog.com/
Sunday, February 7, 2010
7. 1. Monitor, Monitor, Monitor
Action 2
Custom Dashboard
• Most important - The state of NOW
• Single Page Alerts - GREEN YELLOW RED
Sunday, February 7, 2010
9. 1. Monitor, Monitor, Monitor
Action 3
Alerting Software
• Installation of Nagios http://www.nagios.org/
-
• MONyog also has some DB specific alerts
Sunday, February 7, 2010
10. 1. Monitor, Monitor, Monitor
Action 4
Application Metrics
• Total page generation time
Sunday, February 7, 2010
11. Step 2
Identify problem SQL
Sunday, February 7, 2010
12. 2. Identify Problem SQL
Identify SQL Statements
• Slow Query Log
• Processlist
• Binary Log
• Status Statistics
Sunday, February 7, 2010
13. 2. Identify Problem SQL
Problems
• Sampling
• Granularity
Solution
• tcpdump + mk-query-digest
Sunday, February 7, 2010
14. 2. Identify Problem SQL
Action 1
• Install maatkit - http://www.maatkit.org
• Install OS tcpdump (if necessary)
• Get sudo access to tcpdump
http://ronaldbradford.com/blog/take-a-look-at-mk-query-digest-2009-10-08/
Sunday, February 7, 2010
15. # Rank Query ID Response time Calls R/Call Item
# ==== ================== ================ ======= ========== ====
# 1 0xB8CE56EEC1A2FBA0 14.0830 26.8% 78 0.180552 SELECT c u
# 2 0x195A4D6CB65C4C53 6.7800 12.9% 257 0.026381 SELECT u
# 3 0xCD107808735A693C 3.7355 7.1% 8 0.466943 SELECT c u
# 4 0xED55DD72AB650884 3.6225 6.9% 77 0.047046 SELECT u
# 5 0xE817EFFFF5F6FFFD 3.3616 6.4% 147 0.022868 SELECT UNION c
# 6 0x15FD03E7DB5F1B75 2.8842 5.5% 2 1.442116 SELECT c u
# 7 0x83027CD415FADB8B 2.8676 5.5% 70 0.040965 SELECT c u
# 8 0x1577013C472FD0C6 1.8703 3.6% 61 0.030660 SELECT c
# 9 0xE565A2ED3959DF4E 1.3962 2.7% 5 0.279241 SELECT c t u
# 10 0xE15AE2542D98CE76 1.3638 2.6% 6 0.227306 SELECT c
# 11 0x8A94BB83CB730494 1.2523 2.4% 148 0.008461 SELECT hv u
# 12 0x959C3B3A967928A6 1.1663 2.2% 5 0.233261 SELECT c t u
# 13 0xBC6E3F701328E95E 1.1122 2.1% 4 0.278044 SELECT c t u
Sunday, February 7, 2010
16. # Query 2: 4.94 QPS, 0.13x concurrency, ID 0x195A4D6CB65C4C53 at byte 4851683
# This item is included in the report because it matches --limit.
# pct total min max avg 95% stddev median
# Count 3 257
# Exec time 10 7s 35us 492ms 26ms 189ms 78ms 332us
# Time range 2009-10-16 11:48:55.896978 to 2009-10-16 11:49:47.760802
# bytes 2 10.75k 41 43 42.85 42.48 0.67 42.48
# Errors 1 none
# Rows affe 0 0 0 0 0 0 0 0
# Warning c 0 0 0 0 0 0 0 0
# Query_time distribution
# 1us
# 10us #
# 100us ################################################################
# 1ms ####
# 10ms ###
# 100ms ########
# 1s
# 10s+
# Tables
# SHOW TABLE STATUS LIKE 'u'G
# SHOW CREATE TABLE `u`G
# EXPLAIN
SELECT ... FROM u ...G
Sunday, February 7, 2010
17. 2. Identify Problem SQL
Action 2
• Wrappers to capture SQL
• Re-run on single/multiple servers
• e.g. Different slave configurations
Sunday, February 7, 2010
18. 2. Identify Problem SQL
Tip
• Enable General Query Log in Development/Testing
• Great for testing Batch Jobs
Sunday, February 7, 2010
19. 2. Identify Problem SQL
Action 3
Application Logic
• Show total master/slave SQL statements executed
• Show all SQL with execution time (admin user only)
Tip
• Have abstracted class/method to execute ALL SQL
Sunday, February 7, 2010
20. Step 3
Analyze problem SQL
Sunday, February 7, 2010
21. 3. Analyze Problem SQL
• Query Execution Plan (QEP)
• EXPLAIN [EXTENDED] SELECT ...
• Table/Index Structure
• SHOW CREATE TABLE <tablename>
• Table Statistics
• SHOW TABLE STATUS <tablename>
Sunday, February 7, 2010
22. 3. Analyze Problem SQL Good
mysql> EXPLAIN SELECT id FROM example_table WHERE id=1G
*************************** 1. row ***************************
id: 1
select_type: SIMPLE
table: example_table
type: const
possible_keys: PRIMARY
key: PRIMARY
key_len: 4
ref: const
rows: 1
Extra: Using index
Sunday, February 7, 2010
23. 3. Analyze Problem SQL Bad
mysql> EXPLAIN SELECT * FROM example_tableG
*************************** 1. row ***************************
id: 1
select_type: SIMPLE
table: example_table
type: ALL
possible_keys: NULL
key: NULL
key_len: NULL
ref: NULL
rows: 59
Extra:
Sunday, February 7, 2010
25. Step 4
The Art of Indexes
Sunday, February 7, 2010
26. 4. The Art of Indexes
• Different Types
• Column
• Concatenated
• Covering
• Partial
http://ronaldbradford.com/blog/understanding-different-mysql-index-implementations-2009-07-22/
Sunday, February 7, 2010
27. 4. The Art of Indexes
Action 1
• EXPLAIN Output
• Possible keys
• Key used
• Key length
• Using Index
Sunday, February 7, 2010
28. 4. The Art of Indexes Tip
• Generally only 1 index used per table
• Make column NOT NULL when possible
• Statistics affects indexes
• Storage engines affect operations
Sunday, February 7, 2010
29. Before (7.88 seconds) After (0.04 seconds)
*************************** 2. row ** *************************** 2. row ***
id: 2 id: 2
select_type: DEPENDENT SUBQUERY select_type: DEPENDENT SUBQUERY
table: h_p table: h_p
type: ALL type: index_subquery
possible_keys: NULL possible_keys: UId
key: NULL key: UId
key_len: NULL key_len: 4
ref: NULL ref: func
rows: 33789 rows: 2
Extra: Using where Extra: Using index
ALTER TABLE h_p ADD INDEX (UId);
Sunday, February 7, 2010
30. mysql> explain SELECT UID, FUID, COUNT(*) AS Count FROM f
GROUP BY UID, FUID ORDER BY Count DESC LIMIT 2000G
*************************** 1. row ***************************
id: 1
select_type: SIMPLE
table: f
type: index
possible_keys: NULL
key: UID
key_len: 8
ref: NULL
rows: 2151326
Extra: Using index; Using temporary; Using filesort
ALTER TABLE f DROP INDEX UID,
ADD INDEX (UID,FUID)
Sunday, February 7, 2010
31. 4. The Art of Indexes
Indexes can hurt performance
Sunday, February 7, 2010
32. Step 5
Offloading Master Load
Sunday, February 7, 2010
33. 5. Offloading Master Load
• Identify statements for READ ONLY slave(s)
• e.g. Long running batch statements
Single point v scalable solution
Sunday, February 7, 2010
35. 6. Improving SQL
• Poor SQL Examples
• ORDER BY RAND()
• SELECT *
• Lookup joins
• ORDER BY
The database is best for storing
and retrieving data not logic
Sunday, February 7, 2010
36. Step 7
Storage Engines
Sunday, February 7, 2010
37. 7. Storage Engines
• MyISAM is default
• Table level locking
• Concurrent SELECT statements
• INSERT/UPDATE/DELETE blocked by long running SELECT
• All SELECT’s blocked by INSERT/UPDATE/DELETE
• Supports FULLTEXT
Sunday, February 7, 2010
38. 7. Storage Engines
• InnoDB supports transactions
• Row level locking with MVCC
• Does not support FULLTEXT
• Different memory management
• Different system variables
Sunday, February 7, 2010
39. 7. Storage Engines
• There are other storage engines
• Memory
• Archive
• Blackhole
• Third party
Sunday, February 7, 2010
40. 7. Storage Engines
Using Multiple Engines
• Different memory management
• Different system variables
• Different monitoring
• Affects backup strategy
Sunday, February 7, 2010
44. 8. Caching
Action 1
• Memcache is your friend http://memcached.org/
-
• Cache query results
• Cache lookup data (eliminate joins)
• Cache aggregated per user information
• Caching Page Content
• Top rated (e.g. for 5 minutes)
Sunday, February 7, 2010
45. 8. Caching
Action 2
• MySQL has a Query Cache
• Determine the real benefit
• Turn on or off dynamically
• SET GLOBAL query_cache_size = 1024*1024*32;
Sunday, February 7, 2010
46. 8. Caching Tip
The best performance
improvement for an SQL
statement is to eliminate it.
Sunday, February 7, 2010
48. 9. Sharding
• Application level horizontal and vertical partitioning
• Vertical Partitioning
• Grouping like structures together (e.g. logging, forums)
• Horizontal Partitioning
• Affecting a smaller set of users (i.e. not 100%)
Sunday, February 7, 2010
49. 9. Sharding
Action 1
• Separate Logging
• Reduced replication load on primary server
Sunday, February 7, 2010
50. Step 10
Database Management
Sunday, February 7, 2010
52. 10. Database Maintenance
Action 1
• Automate slave inclusion/exclusion
• Ability to apply DB changes to slaves
• Master still a problem
Sunday, February 7, 2010
53. 10. Database Maintenance
Action 2
• Install Fail-Over Master Server
• Slave + Master features
• Master extra configuration
• Scripts to switch slaves
• Scripts to enable/disable Master(s)
• Scripts to change application connection
Sunday, February 7, 2010
55. Bonus
Front End Improvements
Sunday, February 7, 2010
56. 11. Front End Improvements
• Know your total website load time http://getfirebug.com/
-
• How much time is actually database related?
• Reduce HTML page size - 15% improvement
• Remove full URL’s, inline css styles
• Reduce/combine css & js files
• Identify blocking elements (e.g. js)
Sunday, February 7, 2010
57. 11. Front End Improvements
• Split static content to different ServerName
• Spread static content over multiple ServerNames (e.g. 3)
• Sprites - Combining lightweight images http://spriteme.org/
-
• Cookie-less domain name for static content
Sunday, February 7, 2010
59. Before
• Users experienced slow or unreliable load times
• Management could observe, but no quantifiable details
• Concern over load for increased growth
• Release of some new features on hold
Sunday, February 7, 2010
60. Now
• Users experienced consistent load times (~60ms)
• Quantifiable and visible real-time results
• Far greater load now supported (Clients + DB)
• Better testability and verification for scaling
• New features can be deployed
Sunday, February 7, 2010