Point to make: MySQL Monitoring & Advisor Service = “Virtual MySQL DBA” There are 3 main features, each with business rewarding features…
Points to make: Enterprise Dashboard – global view of health, security, availability and performance of all MySQL servers. Heat chart – first two columns show up/down status of MySQL server. That one feature alone will save most DBAs at least 20 minutes a day. The Enterprise dashboard helps a DBA find the needle BEFORE they get to the haystack.
Points to make: MySQL Advisors are “areas of concern” categories for all DBAs. MySQL Advisors are comprised of MySQL Advisor Rules that are designed to enforce MySQL best practices. MySQL Advisors provide “Virtual MySQL DBA” expertise!
Points to make: MySQL Advisor rules provide immediate ROI as they go to work as soon as they are scheduled to find and report on problems and performance tuning opportunities. There is no “warm” up period; DBAs beginning receiving expert advice on improving them systems immediately. The simplest MySQL Advisor rule would take even the most skilled MySQL DBA several hours at best to research, code, test, debug, and schedule. Then they have to factor in the time to maintain and monitor. With MySQL Network, the most complex rule can be implemented and set to monitor MANY MySQL servers in less than 1 minute.
Points to make: DBAs can set different SLAs, alerts and notifications for specific groups of servers (Example: Prod vs Dev). Alerts and notifications can be set at application, criticality, responsibility, etc. levels.
Give examples of concrete applications looking at different 9’s Real life application Order management, ecommerce.
According to the Institute of Electrical and Electronics Engineers (IEEE), High availability (HA) refers to the availability of resources in a computer system, in the wake of component failures within the system. This can be achieved leveraging a variety of strategies. They may range from solutions that utilize customized and redundant hardware, to software solutions using commercial off-the-shelf (COTS) hardware components. Customized solutions typically provide a higher degree of availability, but are significantly more expensive to deploy and maintain. This fact has led to the growth of software solutions which leverage COTS hardware. The solutions presented in this paper will focus on achieving MySQL high availability using the latter strategy.
MySQL Features & Implementation
MySQL FeaturesBy Rakesh KumarOSSCube , Delhirakesh@osscube.com
What we will discuss in next 1 hour• General Info.• MySql Enterprise• Other Features of MySQL > MySQL Network Monitoring and Advisory services > Backup and Recovery > Audit Trail > High Availability > Migration Tool Kit
General Info - Overview of MySQL AB• Founded in 1995• Operations in 24 countries• World’s Most Popular Open Source Database > Over 10 million installations > 50,000 downloads each day• Reduces TCO of database management > Reliable, high performance, easy to use• Bundled by over 100 software & hardware companies• Partners and Telecom Service Integrators HP, Dell, Novell, Red Hat, Unisys, Accenture, LogicaCMG, T-Systems and IPL
Key Features• Written in C, C++• Multilayered design with Independent modules• Fully multi-threaded using Kernel Threads• Provides Transactional and Non-transactional storage engines
Key Features Cont…• In memory heap table support• Server available in client server model, or embedded DB.• Security and privilege management• Supports connection over SSL• Large database, large tables (default up to 256 TB)
Key features contd…• Connectors for application connectivity• 64 Index per table, 1-16 columns per index• Built-in tools for query analysis and space analysis• Localization• Many more…
What is MySQL Enterprise?MySQL Enterprise is the leading open source database solution behindtoday’s modern enterprise applications. It includes:• The MySQL Enterprise Server – the most reliable, secure, and up-to-date version of the world’s most popular open source database.• MySQL Network Monitoring and Advisory Services – services to ensure the maximum quality, uptime, and performance of MySQL Enterprise systems.• Production Support – around-the-clock support for the MySQL Enterprise server, along with consultative services and self-help knowledge aids. Designed for production use and web businesses / enteprise customers relying on MySQL for key business applications.
MySQL Enterprise vs. Community ServerOpen SourceBenefits from testing by worldwide CommunityFeature Releases Experimental MatureMonthly Rapid Update Service Pack ProgramQuarterly Service Pack ProgramHot Fix ProgramBug Escalation Privilege24x7 Production SupportPlatform / Application / Training CertifiedExtended End-of-Life PolicyEnterprise Installer and Configuration ManagerOut-of-the-Box Configured for Enterprise useMonitoring and Advisory ServiceAutomated Software Update and Alert ServiceOnline Knowledge BaseExtensive internal/external (Coverity, KLOCwork) testingBenchmarked with 2,000+ connectionsLegal indemnification
What is the MySQL Community Server?The MySQL Community Server is the rapidly evolving, leading edge and non-commercially supported version of the MySQL server.• Leading edge: new and experimental features with every release.• Open source development and innovation. GPL license.• High community participation. Contributions encouraged!• Great for open source developers, technology enthusiasts, and those wanting to experiment and try out MySQL.• Serves as the foundation & proving ground for MySQL Enterprise.• No formal QA, support, or certifications Recommended for non-production applications and those wishing to try out MySQL. Supported by the large MySQL open source community.
Other Features of MySQL• MySql Network Monitoring and Advisory services• Audit Trail• Backup and Recovery• High Availability – Replication and Clusters• Data migration from other RDBMS to MySQL
MySQL Network Monitoring and Advisory Service• Single, consolidated view into entire MySQL environment• Customizable rules-based monitoring and alerts• Identifies problems before they occur• Reduces risk of downtime• Makes it easier to scale-out without requiring more DBAs“Virtual MySQL DBA” Assistant
Top Features & BenefitsEnterprise DashboardPrimary benefits• Provides “at a glance” view into health, security, availability andperformance of all MySQL servers.• Saves DBA time writing, maintaining and monitoringscripts that collect, compile, and report similar metrics.• Assists DBAs in determining “where” and “how” tospend their limited time.Why it’s important• Makes it easy to scale-out on MySQL without addingmore DBAs.
Top Features & Benefits MySQL Network Advisors Security Schema Administration Performance Replication Custom• Protects MySQL • Helps DBA • Helps DBA • Makes •Makes • Allows customer Servers design better better manage suggestions for to create their• Uncovers databases database improving •Identifies own Advisor Security • Uncovers processes database speed collection loopholes • Suggests • Identifies • Provides for Security improvements potential custom best loopholes for smoother performance practice operations bottlenecks enforcement Each contains a set of Advisor Rules designed to enforce MySQL Best Practices for the specific Advisor category. Provide “Virtual MySQL DBA” expertise
Top Features & BenefitsMySQL Network Advisors and RulesPrimary benefits• 100+ Advisor Rules proactively monitor 600+ MySQL and OS specificmetrics for problems before they occur.• Extend DBA skill set to include MySQL expertise.• Find problems/tuning opportunities DBAs cannot findthemselves.Why they are important• Reduce downtime and performance issues by identifyingproblems and corrective actions before costly outagesoccur. Provide “Virtual MySQL DBA” expertise
Top Features & BenefitsCustomizable Monitors & AlertsPrimary Benefits• Saves DBA time and effort by allowing DBAs to group and monitor manyservers as single entity.• User defined Advisor rules allow for Enterprise specific monitoring andalerts.• Notifications (SMTP/SNMP) can be tailored for individual or grouprecipients.Why it’s important• Allows for customized SLAs across largeMySQL deployments.
Value Prop of MySQL Enterprise Gold Silver Silver + Basic + • Monitors and • Monitors all servers Optimizes Memory G • Proactive Usage Notifications • Replication • Expert Advice Monitoring Basic • Optimizes MySQL • Ensures properr e m t s uC ot e u a V • Enterprise Server Configuration Replication • Ensures settings configuration •Ensures Performance l Uptime/Reliability • Ensures • Rapid Problem • Regular Service Packs Recoverability Resolution •Guards against • Self-help Support • Tightens Security • (Hot Fix builds) • Production Support • Extends Support •Optimizes/Tunes • Support! Support! • Custom o Queries • Database Design
MySQL Network Monitoring and Advisory Service How it worksAutomated Data Collection Automated Data Collection “Query Cache Hit Rate”600 + MySQL and OS specific metrics Data Evaluation• Metrics from all SHOW commands -Default or custom expression evaluation “Query Cache Has Sub-optimal Hit Rate”• Metrics from MySQL provided Thresholds scripts -Violation? “Query Cache Hit Ratio < 95%”• OS Metrics from SIGAR library Events and Alerts• MySQL/ OS Metrics from custom -Graphically seen in console scripts “Rule Violation Appears on Dashboard” Notifications -Requires notification? (SMTP/SNMP) “Notification Sent” Expert Advice and Assistance -Advice on alert/problem and how to fix it “Increase the query_cache_variable value” Specific User Action “Virtual MySQL DBA” - Problem resolved by user intervention
MySQL Network Monitoring and Advisory Service Architecture MySQL AB Corporate Data Center Individual Enterprise Software Databases & Updates Advisor Rules Server Updates MySQL Enterprise Web Service Manager Browser Master-Slave Enterprise Dashboard Replication Profiles & Alerts Knowledge Base Production Support MySQL Enterprise Repository Multi-vendor Support, IP Protection Cluster
MySql Backup• Dump or Raw Backup• On-line or Off-line• Table Types and Consistency• Storage Requirements• Replication
Dump or Raw Backup• Dumps – Can be performed remotely – Take a lot of space – CPU intensive and slower – Are plain SQL, so easy to use later – Can do selective restore – Works for any table type• Raw Backups – Faster, there is no translation – Restore “instantly” – Works for ISAM/MyISAM only
On-line or Off-line• On-line – May Impacts running applications (can’t write) – Must deal with data consistency issues – Slower than off-line because of I/O contention• Off-line – Very simple – Speed is not an issue – Can use normal backup software
Table Types & Consistency• All tables need to flushed to disk – FLUSH TABLES• Tables can’t be updated during backup – LOCK TABLES...• Related tables treated as a group, lock all – FLUSH TABLES WITH READ LOCK• Transactional tables have logs to backup
Storage Requirements• How long can you store backups? – Consider staggering older backups – Never know when you’ll want old data• Raw backups take less space – You can keep just the .MYI header – Rebuild indexes later if need be• All forms of backup can be compressed
Replication• Backup your slave instead of your master – Master won’t be interrupted – Slave can probably be shut down – Or at least you can hold read locks for a long time and nobody will care• It may be worth setting up a slave just for doing backups• Remember to backup the master.info file
Tools and Techniques• mysqldump• mysqlhotcopy• InnoDB on-line backup• Off-line backups• Filesystem snapshots• Roll your own
mysqldump• Comes with MySQL• Dumps local or remote tables/databases• Variety of output formats• Handles locking if you need it• Works best for small and medium installations
mysqlhotcopy• Comes with MySQL – Written by Tim Bunce of Perl DBI fame• Raw backups of MyISAM tables• Handles locking• Regular expression support – Match certain databases or tables• Can truncate indexes to save space
mysqlsnapshot• Separate download – http://jeremy.zawodny.com/mysql/• Used for setting up replication snapshots• Or backing up a running server• Always backs up everything – Either one big tar file or one per database• Very quick to restore
InnoDB on-line Backup• Separate download – http://www.innodb.com/• Costs extra $$$• The only good solution for InnoDB backups without replication• Only does InnoDB tables
Off-line Backups• Very easy• Hard to screw up• Use any backup tool you like – Just remember to backup all the MySQL stuff – You wouldn’t want compatibility problems after an upgrade• Restores can be quick
Filesystem Snapshot• Most “expensive” solution• Need hardware/software support – Veritas, Network Appliance, LVM, EMC, etc.• Fastest option available• Uses least disk space• Doesn’t guard against all failures
MySql Auditing• Various logs that help in auditing:• General Query Log• Binary Logs• Error Logs• Slow query log
MySQL HA Solutions Overview• Introduction to MySQL• HA Terms and Concepts• HA Requirements and Considerations• HA Solutions from MySQL• Third-Party HA Solutions for MySQL• Additional HA Resources plus Q & A
What is High Availability?• High Availability (HA) refers to the availability of resources in a computer system despite component failures• Two major categories > Customized/redundant software and hardware > Software-based solutions leveraging COTS components• Continuous Availability > Non-stop service > No disruption of service even during a fail over > Very high standard of availability, often unnecessary
What is Replication?• The duplication of data to more then one location• The use of additional hardware or software for… > Availability > Reliability > Fault tolerance > Performance• Asynchronous vs. Synchronous > Safeguarding data loss• MySQL Replication > Statement vs Row-based Replication• Block-level Replication (DRBD)
What is Replication? • The duplication of data changes to more than one location Asynchronous Replication updatecustomer Synchronous Replication updatecustomer
What is Clustering?• Clusters are often used for: > Operating systems > Databases > Application servers > Load balancers > Web servers > Just about any “service” that needs to be highly available• Redundant nodes provide service when components fail• Eliminate single points of failure > Multiple network connections > Redundant data storage (SAN, RAID)• Shared-Nothing vs. Shared-Disk
MySQL High Availability Solutions• MySQL Replication• MySQL Replication and Linux Heartbeat• MySQL, DRBD and Linux Heartbeat• MySQL Cluster• MySQL Cluster Carrier Grade Edition• MySQL w/ Shared-Storage & Clustering Agents• Other HA Solutions for MySQL
MySQL Replication• Asynchronous Requirements Support > Statement Availability > Row-based (5.1) Automated IP Fail Over No > Mixed (5.1) Automated DB Fail Over No• Read vs. Write Typical Fail Over Time Varies > Response times Resynchronization of Data No > Load balancing Geographic Redundancy Yes > Scale Out Scalability• Administration Built-in Load Balancing Scale-Out > Backups Read Intensive Yes > Upgrades Write Intensive No > Testing Dozens for• # of Nodes Geographic Redundancy Reads
Inside MySQL ReplicationWrites & Reads Web/App Server Writes relay binlog mysqld I/O SQL Thread Thread index & binlogs data Replication binlog data mysqld MySQL Master MySQL Slave
MySQL Replication - Scale Out • Write to one master • Read from many slaves • Perfect for read intensive apps Load BalancerWrites & Reads Reads Reads MySQL Replication
MySQL, Heartbeat & DRBD Cluster IP ManagementActive PassiveDRBD DRBDServer Server Synchronous Block Replication
Managing Failures IP ManagementPassive X Active DRBD DRBDServer Server X X X
Data Resynchronization after Host Failure IP ManagementPassive Active DRBD DRBDServer Server Resynchronization Active Server is still able to process transactions during a resynchronization operation
DRBD Basics• When the failed node comes up > Becomes a secondary node > Must synchronize its data to the primary > Synching happens in the background without service interruption > Only the parts of the device which have changed since the failure will be resynchronized.• Total resynchronization in just several minutes > Regardless of device size > Even after the hard crash of an active node > Resynch rates may be configured per device• Device sizes currently limited to 4 TB
MySQL Cluster• Requirements Support In-Memory Database > Disk Data (5.1) Availability• Shared-nothing storage Automated IP Fail Over No• Synchronous replication Automated DB Fail Over Yes• Fast-automatic data fail over Typical Fail Over Time < 3 Secs• High performance Resynchronization of Data Yes > NDB API Yes• No special hardware Geographic Redundancy (w/ Replication)• Best for primary key lookups Scalability• Transactional Load Balancing Scale-Out• Online backup Read Intensive Yes (Look ups) Write Intensive Yes 5.0: 64 # of Nodes 5.1: 255
MySQL Cluster Carrier Grade Edition• An advanced version of MySQL Cluster for the telecom industry• Ideally suited for the telecom industrys top carriers, operators and equipment manufactures• Used by industry leaders for real-time, scalable data management
MySQL Cluster Carrier Grade Edition Characteristic Benefits • Separate development tree from MySQL Server Stability • Permits prioritized bug fixes & enhancements for customers • Developed & tested with major TEMsQuality & Reliability • Stricter patch and bug acceptance • Backup and restore status/monitoring • Support for larger clusters, up to 255 Nodes • Optimizations to communication protocol enabling API to Data Node over WAN • Circular replication support Scalability • Batched slave replication • Support for user-defined partitioning • Multiple connections to NDB from a single MySQL Server • Optimizations to communication protocol enabling API to Data Node over WAN • More efficient Client/Server data handling and transfers Performance • Other optimizations (Memory, BLOBs, Disk-based data, etc.) • Micro Global Checkpoints • Setting threads to real-time priority • Online Add Column • Support for mixed (x86, SPARC, PowerPC) environments Adaptability • Time-stamp based conflict detection/resolution support in replication • Locking threads to a CPU
MySQL Cluster System RequirementsSystem Component Requirement Hosts Maximum of 255 total nodes (48 Data Nodes) COTS – Advanced TCA Hardware 32 & 64-bit x86, PowerPC & SPARC Memory Varies on size of database, # of hosts, # of replicas Shared-Nothing - Memory & Disk Data Storage SCSI or RAID for I/O performance 100 Mbps or better Network (Gigabit recommended, SCI supported) Operating System Linux (Red Hat, SuSE), Solaris, HP-UX, Mac OSX, others…
Cluster Components • Standard SQL Interface SQL Node • Scale-out for Performance (MySQL) • Enables Replication • High Performance NDB API • C++ API (Application) • Developer’s Guide • Data Storage (Memory/Disk) • Automatic & User-Defined Partitioning Data Node(NDB Storage Engine) • Local & Global Checkpoints • Scale-out for Capacity & Redundancy • ConfigurationManagement • Arbitration Node • Use Two for Redundancy
2 Data Node MySQL Cluster Clients Application/SQL Nodes (NDB API and/or MySQL Server)Management Nodes Data Nodes NDB Storage Engine
4 Data Node MySQL Cluster Clients Application/SQL Nodes (NDB API and/or MySQL Server)Management Nodes Data Nodes NDB Storage Engine
Cluster uses Synchronous ReplicationUpdate:UPDATE author Two-phaseSET country = ‘Cuba’ commit protocolWHERE authid = 2; Node Group authid (PK) fname lname country authid (PK) fname lname country 1 Albert Camus France 1 Albert Camus France 2 Ernest Hemingway Cuba 2 Ernest Hemingway Cuba 3 Johan Goethe Germany 3 Johan Goethe Germany 4 Junichiro Tanizaki Japan 4 Junichiro Tanizaki Japan
Geographic Redundancy• As of 5.1, support for asynchronous replication between Clusters• Enables a higher degree of availability and continuity Asynchronous MySQL Replication NDB API NDB API NDB Storage Engine NDB Storage Engine Cluster 1 Cluster 2
Distribution & Partitioning: 4 Data Nodes Four Data Nodes (Physical)authid (PK) fname lname country * Half of database on each node 1 Albert Camus France 2 Ernest Hemingway USA Two Replicas (Logical) * Two copies of each partition 3 Johan Goethe Germany 4 Junichiro Tanizaki Japan Two Node Groups * Two sets of identical Data Nodes Node Group 1 Node Group 2 authid (PK) fname lname country authid (PK) fname lname country 1 Albert Camus France 2 Ernest Hemingway USA 3 Johan Goethe Germany 4 Junichiro Tanizaki Japan
Distribution & Partitioning: 4 Data NodesQuery:SELECT fname, lnameFROM author The query can only be satisfied by aWHERE authid = 2; Data Node belonging to Node Group 2Result: because the data is not stored within Node Group 1Ernest Hemingway Node Group 1 Node Group 2authid (PK) fname lname country authid (PK) fname lname country 1 Albert Camus France 2 Ernest Hemingway USA 3 Johan Goethe Germany 4 Junichiro Tanizaki Japan
Distribution & Partitioning: 4 Data NodesQuery:SELECT fname, lnameFROM author In a four node cluster, even if weWHERE authid BETWEEN 1 AND 3; loose a Data Node in each NodeResult: Group, we can still satisfy any queryAlbert CamusErnest HemingwayJohan Goethe Node Group 1 Node Group 2authid (PK) fname X lname country X authid (PK) fname lname country 1 Albert Camus France 2 Ernest Hemingway USA 3 Johan Goethe Germany 4 Junichiro Tanizaki Japan
Data Node Failure & Recovery• Server 3 failure is detected• Server 4 handles Server 3 requests 1 2 3 4 Node Group 1 Node Group 2
Data Node Failure & Recovery• Server 3 begins copying data from Server 4• Complete or just changes since failure 1 2 3 4 Node Group 1 Node Group 2
Data Node Failure & Recovery• Once data is fully restored, Server 3 rejoins the cluster 1 2 3 4 Node Group 1 Node Group 2
Network Partitioning – Avoiding “Split Brain” • By default, the Management Node acts as arbitrator • In the event of a network failure, arbitration determines which set of nodes should continue while shutting down the rest NDB API Application Nodes (NDB API and/or MySQL Server)Management Nodes Data Nodes X
MySQL MySQL Replication MySQL, Heartbeat Requirements MySQL Cluster Replication + Heartbeat + DRBD AvailabilityAutomated IP Fail Over No Yes Yes NoAutomated DB Fail Over No No Yes Yes Typical Fail Over Time Varies Varies < 30 secs < 3 secs Auto Resynch of Data No No Yes YesGeographic Redundancy Yes Yes MySQL Replication MySQL Replication Scalability MySQLBuilt-in Load Balancing MySQL Replication MySQL Replication Yes Replication Read Intensive Yes Yes MySQL Replication Yes Write Intensive No No If configured correctly Yes# of Nodes per Cluster Master/Slave(s) Master/Slave(s) Active/Passive 255 # of Slaves Dozens for Reads Dozens for Reads Dozens for Reads Dozens for Reads
MySQL Migration Toolkit• Wizard Interface. Supports...• Oracle• Microsoft SQL Server• Microsoft Access• Sybase• MySQL