Anders Karlssonanders@skysql.comMySQL on Amazon AWS 101Free as inFree BeerSkySQLSolutions Day
Agenda• About Anders Karlsson• Who is this for?• Amazon database options• Amazon AWS EC2 basics• Getting dirty!• Preparing...
About Anders Karlsson• Senior Sales Engineer at SkySQL• Former Database Architect at Recorded Future, SalesEngineer and Co...
Who is this for?• If you are a newcomer to MySQL? Maybe,and maybe not. But I make someassumptions on what you know aboutMy...
Amazon database options• Running in the Amazon cloud you have atleast two options for how to run MySQL– Run it yourself• I...
Amazon database options• This talk is not about Amazon RDS service• This talk is about installing MariaDB in yourown Amazo...
Amazon EC2 – The basics• Amazon AWS (Amazon WebSerices) EC2 (Elastic ComputeCloud) is what we will useInstanceInstance Ins...
Amazon EC2 – Instances• Instance – A virtual machine• Is identified by an instance id• Has 2 network interfaces, 1 interna...
Amazon EC2 – Volumes• EC2 disks are called volumes• Are identified by a Volume id• EC2 disks appear much as normal disks t...
Amazon EC2 - Snapshots• Snapshots are "copies" of EC2 volumes• Are identified by a Snapshot id• New Volumes can be created...
Time to get our hands dirty!• I have created 2 m3.xlarge instances for us toplay with– 15 Gb RAM– 4 "Cores"– No ephermal d...
Our instances in the Amazon console
Hey, you said "no GUI"! Stop doing that!• To begin with, log on to the instance using thecertificate key-pair from when th...
Logging on to the EC2 Instance• In the GUI (OK, so I lied) go to EC2->Instancesand click on the Instance of interest• In t...
Installing EC2 command line tools• Then we want the EC2 tools installed– sudo apt-get install ec2-api-tools– You need to e...
Enabling the EC2 API commands• As we are accessing EC2, we need to identifyourselves• We need a private key and a certific...
Finally, a working commandline!$ export EC2_PRIVATE_KEY=~/.ec2/ec2_pk.pem$ export EC2_CERT=~/.ec2/ec2_cert.pem$ # Now, che...
Now, lets create some disks!• ec2-create-volume to create the volume• ec2-attach-volume to attach it to aninstance• After ...
Disk creation for MySQL in EC2INSTANCE=`curl -s http://169.254.169.254/latest/meta-data/instance-id`AZ=`curl -s http://169...
Time to start MySQL• MySQL binaries are in /usr/local/mariadb1001cd /usr/local/mariadb1001scripts/mysql_install_db --defau...
Now, Backups! Fun!• Backups are best done with EC2 snapshots• I tend to like xfs as we can do a freeze there,so we can hav...
Creating the snapshot – Part 1#!/bin/bash#mysql --skip-column-names -u root <<!EOFflush tables with read lock;! mysql --sk...
Creating the snapshot – Part 2sync; syncsudo xfs_freeze -f /dataVOLID=`cat $HOME/volid.dat`SNAPID=`ec2-create-snapshot $VO...
Provisioning a slave from a backup• Provisioning a slave is done by– Creating a volume from a snapshot of thecorresponding...
Provisioning a new SlaveMasterInstanceMasterInstanceSlaveInstanceSlaveInstanceSnapshotSnapshot VolumeVolume1. Create snaps...
Provisioning a slave – Create VolumeINSTANCE=`curl -s http://169.254.169.254/latest/meta-data/instance-id`AZ=`curl -s http...
Provisioning a slave – Set up Volumesudo pvscansudo lvchange -a y /dev/$VG/$LVsudo mount -t xfs /dev/$VG/$LV $MOUNTPTsudo ...
Provisioning a slave – Set up MySQLcd /usr/local/mariadb1001sudo bin/mysqld_safe --defaults-file=/usr/local/mariadb1001/my...
Questions? Answers!Anders Karlssonanders@skysql.comhttp://karlssonondatabases.blogspot.comThe question is not “What is the...
Upcoming SlideShare
Loading in …5
×

MySQL on AWS 101

2,129 views

Published on

Running MySQL on Amazon AWS is not that difficult, but you want to take advantage of AWS and EC2 fully. This talk is for the MySQL DBA that is about to use Amazon or is just starting to.

Published in: Technology
0 Comments
5 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
2,129
On SlideShare
0
From Embeds
0
Number of Embeds
521
Actions
Shares
0
Downloads
30
Comments
0
Likes
5
Embeds 0
No embeds

No notes for slide

MySQL on AWS 101

  1. 1. Anders Karlssonanders@skysql.comMySQL on Amazon AWS 101Free as inFree BeerSkySQLSolutions Day
  2. 2. Agenda• About Anders Karlsson• Who is this for?• Amazon database options• Amazon AWS EC2 basics• Getting dirty!• Preparing EC2 for MySQL• Installing MySQL on EC2• EC2 features for the MySQL DBA• Backing up MySQL using EC2• Provisioning a new slave using EC2• Questions? Answers?
  3. 3. About Anders Karlsson• Senior Sales Engineer at SkySQL• Former Database Architect at Recorded Future, SalesEngineer and Consultant with Oracle, Informix,TimesTen, MySQL / Sun / Oracle etc.• Has been in the RDBMS business for 20+ years• Has also worked as Tech Support engineer, PortingEngineer and in many other roles• Outside SkySQL I build websites (www.papablues.com),develop Open Source software (MyQuery,mycleaner etc), am a keen photographer, hasan affection for English Real Ales and a greatinterest in computer history25/04/2013 SkySQL Ab 2011 Confidential 3
  4. 4. Who is this for?• If you are a newcomer to MySQL? Maybe,and maybe not. But I make someassumptions on what you know aboutMySQL and MariaDB• If you know EC2 really well and has used itfor a long time? Probably not, you will mostlikely not learn anything new• If you are a MySQL DBA that wants toexplore the Amazon cloud or want to knowmore about it? Stay just where you are,dont touch that dial!
  5. 5. Amazon database options• Running in the Amazon cloud you have atleast two options for how to run MySQL– Run it yourself• Install MySQL or MariaDB on an Amazon instance• Manage the instance yourself• Install all HA options yourself• Cost is the price for the instances, EC2 volumes and allthat– Standard EC2 volumes or provisioned IOPS volumes– Use Amazon RDS• Maintained by Amazon• Simple and easy backup, scaling etc• Much less flexible
  6. 6. Amazon database options• This talk is not about Amazon RDS service• This talk is about installing MariaDB in yourown Amazon AWS instance– We will be using EC2 disks– We will be looking at setting up replication– We will be using Ubuntu (a standard Ubuntuprovided by Amazon)– We will be using mostly Amazon EC2 commandline tools• No GUI stuff!
  7. 7. Amazon EC2 – The basics• Amazon AWS (Amazon WebSerices) EC2 (Elastic ComputeCloud) is what we will useInstanceInstance InstanceInstance InstanceInstanceVolumesVolumes
  8. 8. Amazon EC2 – Instances• Instance – A virtual machine• Is identified by an instance id• Has 2 network interfaces, 1 internal and 1 public• May have a "local" Ephermal disk that is tied tothe instance: If the instance goes, the data goes!• Is of a type which determines amount ofmemory, CPU power and some other things• Is part of a security group which determinesnetwork security• Is located in an Availability Zone
  9. 9. Amazon EC2 – Volumes• EC2 disks are called volumes• Are identified by a Volume id• EC2 disks appear much as normal disks toLinux• EC2 disks are either:– Standard – Using the internal network– Provisioned IOPS – Using a separate network• Are located in an Availability Zone• May be attached to an instance
  10. 10. Amazon EC2 - Snapshots• Snapshots are "copies" of EC2 volumes• Are identified by a Snapshot id• New Volumes can be created from Snapshots• Snapshots are located in an Availability Zone– New volumes has to be in the same AZ as theSnapshot it is created from• Taking a snapshot is fast– Done at the Volume level, not at the Instancelevel. So this is different from an LVM Snapshot
  11. 11. Time to get our hands dirty!• I have created 2 m3.xlarge instances for us toplay with– 15 Gb RAM– 4 "Cores"– No ephermal disk• Lets have a look at theAWS console – We shouldsee 2 instances and 2 EBS disks
  12. 12. Our instances in the Amazon console
  13. 13. Hey, you said "no GUI"! Stop doing that!• To begin with, log on to the instance using thecertificate key-pair from when the instance wascreated– No, you cannot get at it later. You cant.– If you are on Windows and Putty, use puttygen toconvert the certificate to one that putty understandsOK, OK! Lets gocommand linemode then!OK, OK! Lets gocommand linemode then!
  14. 14. Logging on to the EC2 Instance• In the GUI (OK, so I lied) go to EC2->Instancesand click on the Instance of interest• In the lower pane, make sure the"Description" pane is selected, the look for the"Public DNS" field• This is the address you will be connecting to• If you used the default security group, you areOK, else you need to set up port 22 (ssh)access from, for example, 0.0.0.0/0• Then start your ssh client, connect as ubuntu
  15. 15. Installing EC2 command line tools• Then we want the EC2 tools installed– sudo apt-get install ec2-api-tools– You need to enable multiverse• To use these tools, Java and a few other thingsare necessary, but in the Amazon suppliedUbuntu versions, you should have what youneed• Yes, yes I know this is a bit Ubuntu specific butthere isnt any generic method
  16. 16. Enabling the EC2 API commands• As we are accessing EC2, we need to identifyourselves• We need a private key and a certificate thattells EC2 who we are• Send the certificate and private key files to theinstance• Set up, and export, EC2_PRIVATE_KEY andEC2_CERT to point to these files (and put it in.profile for convenience)
  17. 17. Finally, a working commandline!$ export EC2_PRIVATE_KEY=~/.ec2/ec2_pk.pem$ export EC2_CERT=~/.ec2/ec2_cert.pem$ # Now, check who we are!$ curl -s http://169.254.169.254/latest/meta-data/instance-idi-8b77f2e6$ ec2-describe-volumes –F "attachment.instance-id=i-8b77f2e6"VOLUME vol-94b43fcd 8 snap-bcdff2f2us-east-1a in-use 2013-04-19T11:46:17+0000ATTACHMENT vol-94b43fcd i-8b77f2e6/dev/sda1 attached 2013-04-19T11:46:18+0000
  18. 18. Now, lets create some disks!• ec2-create-volume to create the volume• ec2-attach-volume to attach it to aninstance• After this, create a Volume Group, a LogicalVolume, mkfs and mount it, just like youusually doInstanceInstance
  19. 19. Disk creation for MySQL in EC2INSTANCE=`curl -s http://169.254.169.254/latest/meta-data/instance-id`AZ=`curl -s http://169.254.169.254/latest/meta-data/placement/availability-zone`VOLID=`ec2-create-volume --size 200 -z $AZ | awk {print $2}`ec2-create-tags $VOLID --tag role=mysqlmasterec2-attach-volume $VOLID -i $INSTANCE -d =/dev/sdb1while [ ! -b /dev/xvdb1 ]; doecho "Waiting for /dev/xvdb1 to become available"sleep 5donesudo pvcreate /dev/xvdb1sudo vgcreate vg_mysql /dev/xvdb1sudo lvcreate -L $195G -n lv_mysql vg_mysqlsudo mkfs -t xfs /dev/vg_mysql/lv_mysqlsudo mount -t xfs /dev/vb_mysqk/lv_mysql /datasudo chown -R mysql:mysql /data
  20. 20. Time to start MySQL• MySQL binaries are in /usr/local/mariadb1001cd /usr/local/mariadb1001scripts/mysql_install_db --defaults-file=my.cnfbin/mysqld_safe --defaults-file=my.cnf &• Now, insert some test data. Thisscript will create a table t1 andstart inserting into itcd./gendata.sh &
  21. 21. Now, Backups! Fun!• Backups are best done with EC2 snapshots• I tend to like xfs as we can do a freeze there,so we can have a consistent backup "below"LVM• I also use FLUSH TABLES• Lets see it in action!
  22. 22. Creating the snapshot – Part 1#!/bin/bash#mysql --skip-column-names -u root <<!EOFflush tables with read lock;! mysql --skip-column-names -u root -e "show master status" >/data/snappos.dat! $HOME/snapvol.shunlock tables;!EOF
  23. 23. Creating the snapshot – Part 2sync; syncsudo xfs_freeze -f /dataVOLID=`cat $HOME/volid.dat`SNAPID=`ec2-create-snapshot $VOLID -dmysql_master_backup | awk {print $2}`SNAPSTAT=`ec2-describe-snapshots $SNAPID | awk{print $4}`while [ "x$SNAPSTAT" != "xcompleted" ]; doecho "Waiting for snapshot to complete"sleep 1SNAPSTAT=`ec2-describe-snapshots $SNAPID | awk{print $4}`donesudo xfs_freeze -u /data
  24. 24. Provisioning a slave from a backup• Provisioning a slave is done by– Creating a volume from a snapshot of thecorresponding master– Mount that on the slave– Start MySQL– Configure the slave– Start the slave– Wait for the slave to catch up…
  25. 25. Provisioning a new SlaveMasterInstanceMasterInstanceSlaveInstanceSlaveInstanceSnapshotSnapshot VolumeVolume1. Create snapshot from master volume2. Create new volume from snapshot3. Prepare new volume and mount it4. Prepare mysql as a slave and catch up withmaster
  26. 26. Provisioning a slave – Create VolumeINSTANCE=`curl -s http://169.254.169.254/latest/meta-data/instance-id`AZ=`curl -s http://169.254.169.254/latest/meta-data/placement/availability-zone`SNAPID=`ec2-describe-snapshots -Fdescription=mysql_master_backup | awk {print $2}`VOLID=`ec2-create-volume --snapshot $SNAPID --availability-zone $AZ | awk {print $2}`ec2-attach-volume $VOLID -i $INSTANCE -d /dev/sdb1while [ ! -b /dev/xvdb1 ]; doecho "Waiting for /dev/xvdb1 to become available"sleep 5done
  27. 27. Provisioning a slave – Set up Volumesudo pvscansudo lvchange -a y /dev/$VG/$LVsudo mount -t xfs /dev/$VG/$LV $MOUNTPTsudo chown -R mysql /data
  28. 28. Provisioning a slave – Set up MySQLcd /usr/local/mariadb1001sudo bin/mysqld_safe --defaults-file=/usr/local/mariadb1001/my.cnf &while [ ! -S /tmp/mysql.sock ]; dosleep 3doneMASTERFILE=`awk {print $1} < /data/snappos.dat`MASTERPOS=`awk {print $2} < /data/snappos.dat`echo "CHANGE MASTER TO MASTER_LOG_FILE=$MASTERFILE,master_log_pos=$MASTERPOS,master_host=aws101_1,master_port=3306,master_user=repl,master_password=repl;" | mysql -u rootmysql -u root -e "start slave"
  29. 29. Questions? Answers!Anders Karlssonanders@skysql.comhttp://karlssonondatabases.blogspot.comThe question is not “What is theanswer?”, the question is “What is thequestion?”.Henri PoincaréThe question is not “What is theanswer?”, the question is “What is thequestion?”.Henri Poincaré

×