Do you know how to do rolling maintenance on your database hosts so you can make changes without stopping applications? How about upgrading schema and applications themselves? Tungsten clusters have a host of features that can help you with everything from basic administration to complex application upgrades. This webinar shows you the different types of administration you need to perform and introduce the procedures to perform them without application downtime. Course Topics • What is rolling maintenance • How to perform simple maintenance on hosts • How to plan, test, and perform a MySQL version upgrade • Standard SQL schema upgrade patterns and how to do them with zero downtime • Combining SQL schema and application code upgrades • Structuring applications to make zero-downtime upgrade work better • Backing out of failed upgrade or maintenance

1. Tungsten University:
Zero Downtime Maintenance
and Schema Upgrade
Robert Hodges, CEO
Giuseppe Maxia, Director of QA
©Continuent 2013

2. Introducing Continuent
• The leading provider of clustering and
replication for open source DBMS
• Our Product: Continuent Tungsten
• Clustering - Commercial-grade HA, performance
scaling and data management for MySQL
• Replication - Flexible, high-performance data
movement
©Continuent 2013 2

3. High Availability and System
Maintenance
©Continuent 2013 3

4. Maintenance and Upgrade Math
Uptime Weekly Maintenance Window
95.2% 8 hours
97.6% 4 hours
98.8% 2 hours
99.4% 1 hour
99.8% 20 minutes
99.99% 1 minute (!!)
True 4 9s availability: 53 minutes downtime per year
©Continuent 2013 4

5. Tungsten Cluster Architecture
in 3 Minutes or Less
©Continuent 2013 5

6. Overview of a Tungsten Cluster
Application Application
Tungsten Connector Tungsten Connector
Monitoring and Control
Monitoring and Control
Db2 Db1 Db3
Manager Manager Manager
Replicator Replicator Replicator
Slave Master Slave
Data Service: sjc
©Continuent 2013 6

7. Introduction to MySQL Maintenance:
Change a my.cnf variable and restart
©Continuent 2013 7

8. Useful cctrl maintenance commands
• set policy automatic | manual | maintenance
• datasource <name> o!ine
• datasource <name> online
• datasource <name> shun
• datasource <name> welcome
• cluster online
©Continuent 2013 8

9. Online Maintenance of a Slave
Step 1
Shun node
and take 1
services
o"ine
Step 2
2 Perform
Step 3 maintenance
Welcome task(s)
slave back to
3
cluster
©Continuent 2013 9

10. Online Maintenance of a Master
Step 2
Step 1 Shun node
Switch and take
master to 2 services
slave 1 o"ine
Step 3
3 Perform
Phase 5 maintenance
Switch slave 5 task(s)
back to
master
4
Step 4
Welcome
slave back
©Continuent 2013 10

11. Rolling Maintenance
Slave 1 Slave 2-N Switch Master
• Shun slave 1, • Repeat for • Switch • Shun old
maintain, additional master master,
welcome back slave(s) maintain,
to cluster welcome back
to cluster
©Continuent 2013 11

12. Extended MySQL Maintenance:
Upgrading to a new MySQL version
©Continuent 2013 12

13. Rolling MySQL Version Upgrade
Slave Upgrade Slave Upgrade Switch Final Upgrade
1. Shun slave 1.Shun slave 1.Switch 1. Shun slave
2. Dump data 2. Upgrade master 2. Upgrade
3. Upgrade MySQL MySQL
MySQL 3. Restore 3. Restore
4. Restore data
5. Welcome slave
6. Backup
©Continuent 2013 13

14. Backups and MySQL Version Upgrade
• Backup before upgrade to protect against
failure
/cluster
>
datasource
DB2
backup
• Backup after 1st upgrade to simplify further
upgrades
/cluster
>
datasource
DB2
backup
• Restore after subsequent node upgrades
/cluster
>
datasource
DB3
restore
©Continuent 2013 14

15. Schema Upgrade:
Performing ALTER TABLE commands
©Continuent 2013 15

16. Tips and Tricks for Schema Upgrade
• Use replication to do simple, non-blocking
changes
• Don’t log schema changes on slaves
mysql>
set
session
sql_log_bin=0;
• Add schema rather than taking away
©Continuent 2013 16

17. Rolling Schema Upgrade
Slave Upgrade Slave Upgrade Switch Final Upgrade
1. Run backup 1.Shun/ 1.Switch 1. Shun/
2. Shun slave upgrade/ master upgrade/
3. Run upgrade welcome welcome
4. Discard and
restore on
failure
5. Welcome slave
©Continuent 2013 17

18. Ways to Make Upgrade and
Maintenance Easy or at
Least Less Painful :)
©Continuent 2013 18

19. Secrets of Success #1: Testing
• Develop a procedure and follow it
• Automate as much as possible
• Run tests before going production
• Use production data and lots of it
©Continuent 2013 19

20. Secrets of Success #2: Minimize Risk
• You can do zero-downtime upgrade without
stopping any apps
• But why take chances?
• Run upgrades at periods of low activity
• Shut down non-essential services
• Upgrade low-value applications "rst
• Upgrade slaves "rst
©Continuent 2013 20

21. Secrets of Success #3: Application
Design
• Avoid “generic” SQL statement that do not
specify columns
• Build schema upgrade into application
installation
Install base schema
Run upgrade script #1
Run upgrade script #2
...
Application ready...
©Continuent 2013 21

22. Secrets of Success #4: Be Prepared
for Trouble
©Continuent 2013 22

23. Wrapping Up
©Continuent 2013 23

24. Tungsten University Sessions
• Zero-Downtime Maintenance and Schema
Operations
(March 21 and 26)
• Geographically Distributed Tungsten Clusters
(April 11 and 16)
• Replicate between MySQL and Oracle
(May 2 and 7)
Send any feedback to: tu@continuent.com
©Continuent 2013 24

25. 560 S. Winchester Blvd., Suite 500 Our Blogs:
San Jose, CA 95128 http://scale-out-blog.blogspot.com
Tel +1 (866) 998-3642 http://datacharmer.org/blog
Fax +1 (408) 668-1009 http://www.continuent.com/news/blogs
e-mail: sales@continuent.com
Continuent Web Page:
http://www.continuent.com
Tungsten Replicator 2.0:
http://code.google.com/p/tungsten-replicator
©Continuent 2012.