The Sky’s the Limit                    Hints and Best Practices to Integrate and Run MySQL on Standard Servers and in the ...
Who is Ivan                               ?Wednesday, 5 December 12
SkySQL                •#1 provider of MySQL-based                     solutions, for software and                     serv...
Agenda                           • Understanding the Cloud Landscape                           • Anatomy of a MySQL Databa...
Understanding the Cloud LandscapeWednesday, 5 December 12
The Landscape                    Or maybe an aerial view...                                                 PAGE 6Wednesda...
The Landscape                    Or maybe an aerial view...                           Public Clouds                       ...
The Landscape                    Or maybe an aerial view...                           Public Clouds   Hybrid Clouds       ...
The Landscape                    Or maybe an aerial view...                           Public Clouds   Hybrid Clouds   Priv...
The Landscape                    Or maybe an aerial view...                                                      IaaS     ...
The Landscape                    Or maybe an aerial view...                                                     PaaS      ...
The Landscape                    Or maybe an aerial view...                                                    SaaS       ...
The Landscape                    Or maybe an aerial view...                                                               ...
The Landscape                    Or maybe an aerial view...                                                               ...
The Landscape                    Or maybe an aerial view...                                                               ...
The Landscape                    Or maybe an aerial view...                                                 PAGE 7Wednesda...
The Landscape                    Or maybe an aerial view...                                                 PAGE 7Wednesda...
The Landscape                    Or maybe an aerial view...                                                 PAGE 7Wednesda...
The Cloud Promise                           • Agility           • Reduced Costs                           • Compatibility ...
The Cloud Promise                           • Agility                                                   F AC  T!• Reduced ...
Challenges for a MySQL DBA                    in the next 3 years                           • Integration with other datab...
Anatomy of a MySQL Database                                  in the CloudWednesday, 5 December 12
A Mix & Match                           • Deployment                           • Internal / External Communication        ...
Deployment                                 PAGE 12Wednesday, 5 December 12
Deployment                                   Amazon RDS                                    Rackspace                •One-c...
Communication & Networking                                                 PAGE 14Wednesday, 5 December 12
Communication & Networking                                      Amazon RDS                                Rackspace       ...
Highly Availability                                          PAGE 16Wednesday, 5 December 12
Highly Availability                                  Amazon RDS                            Rackspace                •Inter...
Scalability                                  PAGE 18Wednesday, 5 December 12
Scalability                                  Amazon RDS                                    Rackspace                •From ...
Daily Operations                                       PAGE 20Wednesday, 5 December 12
Other Features                                     PAGE 21Wednesday, 5 December 12
Cost Factors                                   PAGE 22Wednesday, 5 December 12
MySQL in AWS - Tips & TricksWednesday, 5 December 12
Some Tips for...                           • New instances                           • Tagging                           •...
Some Tips for...                           • New instances                           • Tagging                           •...
Some Tips for...                           • New instances                           • Tagging                           •...
Hidden Gems                    AWS API                       # Create the actual EC2 instance from the appropriate AMI usi...
Hidden Gems                    AWS API                       # Create the actual EC2 instance from the appropriate AMI usi...
Hidden Gems                    Userdata.sh                                                              Variable definitio...
Hidden Gems                    Userdata.sh     Work with IAMs if you                                    want to interact w...
Creating New Instances              hi1.4xlarge                                              (8 cores, 60GB RAM) provides ...
Using AMIs                                 PAGE 29Wednesday, 5 December 12
Using AMIs                                 PAGE 29Wednesday, 5 December 12
Using AMIs                                 PAGE 29Wednesday, 5 December 12
Using AMIs                                 PAGE 29Wednesday, 5 December 12
Tagging    Volume tagging is also     helpful to match the          instances      Name, Node etc. are       helpful in sc...
Networking             Elastic IP replaces             public IP address                     It reflects the EIP          ...
Security                    Groups                               PAGE 32Wednesday, 5 December 12
Security                    Identity and Access Management                                                     PAGE 33Wedn...
Storage       Use Separate System       and Data storage... or        even file per tables                                ...
Backup & Restore                                                 Even better:                                             ...
Why RDS?                           • One-Click solution, good feature set                           • Peace of mind backup...
Why MySQL/MariaDB on EC2?                           • You can choose among different technologies                         ...
Why MySQL on Rackspace?                           • Fanatical about support!                           • Other systems run...
Why Google Cloud SQL?                           •   You develop and use applications with Google AppEngine                ...
For More Information...                   •       Try the Configurator and How-To Video: http://                          ...
Thank You!                                                              ivan@skysql.com                           www.skys...
Upcoming SlideShare
Loading in …5
×

Sky Is The limit

1,089 views
982 views

Published on

These are the slides that I presented at Percona Live London, 4th Dec 2012.
There is lots of content related to the deployment and use of MySQL in the cloud, specifically in Amazon EC2.

Published in: Technology
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
1,089
On SlideShare
0
From Embeds
0
Number of Embeds
3
Actions
Shares
0
Downloads
3
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

Sky Is The limit

  1. 1. The Sky’s the Limit Hints and Best Practices to Integrate and Run MySQL on Standard Servers and in the Cloud Ivan Zoratti V1212.01Wednesday, 5 December 12
  2. 2. Who is Ivan ?Wednesday, 5 December 12
  3. 3. SkySQL •#1 provider of MySQL-based solutions, for software and services •Backed by Monty Program AB and 3rd parties active in the MySQL ecosystem •Committed to provide the best solutions around MySQL for the Community and for the EnterpriseWednesday, 5 December 12
  4. 4. Agenda • Understanding the Cloud Landscape • Anatomy of a MySQL Database in the Cloud • MySQL in AWS - Tips and Tricks PAGE 4Wednesday, 5 December 12
  5. 5. Understanding the Cloud LandscapeWednesday, 5 December 12
  6. 6. The Landscape Or maybe an aerial view... PAGE 6Wednesday, 5 December 12
  7. 7. The Landscape Or maybe an aerial view... Public Clouds PAGE 6Wednesday, 5 December 12
  8. 8. The Landscape Or maybe an aerial view... Public Clouds Hybrid Clouds PAGE 6Wednesday, 5 December 12
  9. 9. The Landscape Or maybe an aerial view... Public Clouds Hybrid Clouds Private Clouds PAGE 6Wednesday, 5 December 12
  10. 10. The Landscape Or maybe an aerial view... IaaS Infrastructure as a Service Public Clouds Hybrid Clouds Private Clouds PAGE 6Wednesday, 5 December 12
  11. 11. The Landscape Or maybe an aerial view... PaaS Platform as a Service IaaS Infrastructure as a Service Public Clouds Hybrid Clouds Private Clouds PAGE 6Wednesday, 5 December 12
  12. 12. The Landscape Or maybe an aerial view... SaaS Software as a Service PaaS Platform as a Service IaaS Infrastructure as a Service Public Clouds Hybrid Clouds Private Clouds PAGE 6Wednesday, 5 December 12
  13. 13. The Landscape Or maybe an aerial view... SaaS Software as a Service Level of Abstraction PaaS Platform as a Service IaaS Infrastructure as a Service Public Clouds Hybrid Clouds Private Clouds PAGE 6Wednesday, 5 December 12
  14. 14. The Landscape Or maybe an aerial view... Control / Governance SaaS Software as a Service Level of Abstraction PaaS Platform as a Service IaaS Infrastructure as a Service Public Clouds Hybrid Clouds Private Clouds PAGE 6Wednesday, 5 December 12
  15. 15. The Landscape Or maybe an aerial view... Control / Governance SaaS Flexibility of Purpose Software as a Service Level of Abstraction PaaS Platform as a Service IaaS Infrastructure as a Service Public Clouds Hybrid Clouds Private Clouds PAGE 6Wednesday, 5 December 12
  16. 16. The Landscape Or maybe an aerial view... PAGE 7Wednesday, 5 December 12
  17. 17. The Landscape Or maybe an aerial view... PAGE 7Wednesday, 5 December 12
  18. 18. The Landscape Or maybe an aerial view... PAGE 7Wednesday, 5 December 12
  19. 19. The Cloud Promise • Agility • Reduced Costs • Compatibility • Reliability • Elasticity • Security • Low Maintenance • Systems Independence • Multi-tenancy • Virtualization PAGE 8Wednesday, 5 December 12
  20. 20. The Cloud Promise • Agility F AC T!• Reduced Costs • Compatibility lo •ud ] Reliability he C on al (*) • Elasticity T t[ Re • la i tSecurity ! nd ly sN o ie Fr Systems Independence I • Low Maintenance atabas e- • D • Multi-tenancy et... no ty • Virtualization Or PAGE 8Wednesday, 5 December 12
  21. 21. Challenges for a MySQL DBA in the next 3 years • Integration with other databases - primarily NoSQL • Adopt the right technology to make MySQL and the Cloud work together • Integration with provisioning, administration and monitoring tools • Increase/provide better security for the database and for the users’ data • Provide elasticity and reduce TCO PAGE 9Wednesday, 5 December 12
  22. 22. Anatomy of a MySQL Database in the CloudWednesday, 5 December 12
  23. 23. A Mix & Match • Deployment • Internal / External Communication • High Availability • Scalability • Daily Operations • Special (non-InnoDB and non-Replication) features PAGE 11Wednesday, 5 December 12
  24. 24. Deployment PAGE 12Wednesday, 5 December 12
  25. 25. Deployment Amazon RDS Rackspace •One-click •Limited offering (max 4GB RAM) •Master/Slave replicas (up to 5 replicas) •You can get up to 30GB RAM and 1.2TB/server •Choice of instances, IOPs, storage etc. Amazon EC2 Google SQL •Deployment procedure needed •Up to 16GB RAM and 100GB storage •Full control on technology and configuration •Can go on SSDs, High I/O etc. PAGE 13Wednesday, 5 December 12
  26. 26. Communication & Networking PAGE 14Wednesday, 5 December 12
  27. 27. Communication & Networking Amazon RDS Rackspace •AWS vs Rackspace •550MB/min (73Mbps) •No control on Networking 160MB/min (21Mbps) •http://bit.ly/TLrILl Amazon EC2 Google SQL •EC2 Placement Groups - 10Gbps connectivity •No control on Networking PAGE 15Wednesday, 5 December 12
  28. 28. Highly Availability PAGE 16Wednesday, 5 December 12
  29. 29. Highly Availability Amazon RDS Rackspace •Internal NAS replication •SAN Replication •MySQL Replication •Multi-region replication Amazon EC2 Google SQL •BYOR (Bring Your Own Replication) •Internal sync/async replication (not MySQL Replication) •Multi-region replication PAGE 17Wednesday, 5 December 12
  30. 30. Scalability PAGE 18Wednesday, 5 December 12
  31. 31. Scalability Amazon RDS Rackspace •From Micro to XLarge instances (64GB RAM) •No Replication or other solutions •Dedicated Instances •Up to 5 Read Replicas Amazon EC2 Google SQL •From Micro to XLarge instances •No Replication •Hi I/O with SSDs and placement groups •Used for Google applications •Dedicated Instances •Limited to 16MB for external applications •Unlimited Replicas PAGE 19Wednesday, 5 December 12
  32. 32. Daily Operations PAGE 20Wednesday, 5 December 12
  33. 33. Other Features PAGE 21Wednesday, 5 December 12
  34. 34. Cost Factors PAGE 22Wednesday, 5 December 12
  35. 35. MySQL in AWS - Tips & TricksWednesday, 5 December 12
  36. 36. Some Tips for... • New instances • Tagging • Networking • Security • Storage • Backup & Restore PAGE 24Wednesday, 5 December 12
  37. 37. Some Tips for... • New instances • Tagging • Networking • Security • Storage • Backup & Restore PAGE 24Wednesday, 5 December 12
  38. 38. Some Tips for... • New instances • Tagging • Networking • Security • Storage • Backup & Restore PAGE 24Wednesday, 5 December 12
  39. 39. Hidden Gems AWS API # Create the actual EC2 instance from the appropriate AMI using the key pair, security group, IAM role # and user data script we previously created for this deployment resource_id=`ec2-run-instances $ami -k $keypair -g $group -f $udscript -t $size -b /dev/sdf=:$ebssize:false -p $role $extras 2> error | awk $1 == "INSTANCE" { print $2 }` if [ "$resource_id" = "" ]; then         ./logerror $session_id "Create Instance" `cat error`         (echo "Create instance failed" ; cat error) > error.$SystemId         exit 1 fi ./add_inventory "Machine Instance Node $NodeNo" $resource_id if [ "$demo" = "1" ] ; then         ./db.sh "insert into DemoResources values ($SystemId, EC2 Instance, $resource_id);" fi echo $resource_id >> resources # Add tags to the instances so that they can locate each other and synchronise during the boot phase ec2-create-tags $resource_id --tag SystemID=$SystemId --tag NodeNo=$NodeNo         --tag EIP=$EIP --tag IAM=SDS$SystemId          --tag Name="SkySQL Data Suite $SystemId Node $NodeNo"           --tag SystemStatus=pending # Find all the EBS volumes associated with an instance and tag those so that destruction can take place # cleanly cnt=0 devices=`ec2-describe-instances $resource_id | grep BLOCKDEVICE | awk { print $3 }` ... PAGE 25Wednesday, 5 December 12
  40. 40. Hidden Gems AWS API # Create the actual EC2 instance from the appropriate AMI using the key pair, security group, IAM role # and user data script we previously created for this deployment resource_id=`ec2-run-instances $ami -k $keypair -g $group -f $udscript -t $size -b /dev/sdf=:$ebssize:false -p $role $extras 2> error | awk $1 == "INSTANCE" { print $2 }` if [ "$resource_id" = "" ]; then         ./logerror $session_id "Create Instance" `cat error`         (echo "Create instance failed" ; cat error) > error.$SystemId         exit 1 fi ./add_inventory "Machine Instance Node $NodeNo" $resource_id if [ "$demo" = "1" ] ; then         ./db.sh "insert into DemoResources values ($SystemId, EC2 Instance, $resource_id);" fi echo $resource_id >> resources # Add tags to the instances so that they can locate each other and synchronise during the boot phase ec2-create-tags $resource_id --tag SystemID=$SystemId --tag NodeNo=$NodeNo         --tag EIP=$EIP --tag IAM=SDS$SystemId          --tag Name="SkySQL Data Suite $SystemId Node $NodeNo"           --tag SystemStatus=pending # Find all the EBS volumes associated with an instance and tag those so that destruction can take place # cleanly cnt=0 devices=`ec2-describe-instances $resource_id | grep BLOCKDEVICE | awk { print $3 }` ... PAGE 25Wednesday, 5 December 12
  41. 41. Hidden Gems Userdata.sh Variable definition - they will be used later when other scripts are called #!/bin/sh export SystemId=12345 export keypair=SDS12345 export role=SDS12345 export security=SDS12345 export EIP=107.21.101.100 export DiskSize=5 export NodeNo=3 export NumberNodes=3 export InitialPassword=MyPassword export EmailAddress="skysql.test@gmail.com" echo Start boot script execution >> /var/log/SDS.boot.log cat > /etc/my.cnf << MY_CNF_EOF On-the-fly configuration files can be created ... Specific my.cnf ... MY_CNF_EOF if [ -d /usr/local/skysql/init.boot ] ; then for script in /usr/local/skysql/init.boot/* ; do echo Run: $script >> /var/log/SDS.boot.log 2>&1 sh $script >> /var/log/SDS.boot.log 2>&1 done fi Predefined scripts are if [ -d /usr/local/skysql/init.boot.3 ] ; then for script in /usr/local/skysql/init.boot.3/* ; do present in the AMI echo Run: $script >> /var/log/SDS.boot.log 2>&1 sh $script >> /var/log/SDS.boot.log 2>&1 done fi echo Completed boot script execution >> /var/log/SDS.boot.log PAGE 26Wednesday, 5 December 12
  42. 42. Hidden Gems Userdata.sh Work with IAMs if you want to interact with other instances Use custom AMIs with prepared scripts and Paste your boot preloaded software for commands here more complex operations PAGE 27Wednesday, 5 December 12
  43. 43. Creating New Instances hi1.4xlarge (8 cores, 60GB RAM) provides 10GBit networking and placement groups Heavy I/O ops can benefit from the use of EBS optimised instances Best fit for a non-sync replication cluster is m2.4xlarge (8 cores, 68GB RAM) Use VPCs to create a VP DB Cluster or if you have already a VPC for your app PAGE 28Wednesday, 5 December 12
  44. 44. Using AMIs PAGE 29Wednesday, 5 December 12
  45. 45. Using AMIs PAGE 29Wednesday, 5 December 12
  46. 46. Using AMIs PAGE 29Wednesday, 5 December 12
  47. 47. Using AMIs PAGE 29Wednesday, 5 December 12
  48. 48. Tagging Volume tagging is also helpful to match the instances Name, Node etc. are helpful in scripts PAGE 30Wednesday, 5 December 12
  49. 49. Networking Elastic IP replaces public IP address It reflects the EIP Internal IP - it can be used for intra-instance communication PAGE 31Wednesday, 5 December 12
  50. 50. Security Groups PAGE 32Wednesday, 5 December 12
  51. 51. Security Identity and Access Management PAGE 33Wednesday, 5 December 12
  52. 52. Storage Use Separate System and Data storage... or even file per tables PAGE 34Wednesday, 5 December 12
  53. 53. Backup & Restore Even better: •Attach a EBS Volume •Direct the backup to the new volume innobackupex •Stream to S3 --user=user •... --password=password •Keep the last backup for emergencies --stream=tar ./ 2> tmp-file | S3Control.sh stream bucket backup-name #!/bin/sh SkySQLLIB=/usr/local/skysql/skysql_aws/ if [ ! -n "$EC2_HOME" ] ; then export EC2_HOME=/opt/aws/apitools/ec2 fi LIBDIR="${EC2_HOME}/lib" for jar in "${LIBDIR}"/*.jar ; do CP="${CP}:${jar}" done LIBDIR="${EC2_HOME}/lib" for jar in "${LIBDIR}"/*.jar ; do CP="${CP}:${jar}" done CP=.:${SkySQLLIB}SkySQLAWS.jar:${SkySQLLIB}aws-java-sdk-1.3.16.jar:$CP java -classpath $CP S3Control $* PAGE 35Wednesday, 5 December 12
  54. 54. Why RDS? • One-Click solution, good feature set • Peace of mind backup and disaster recovery • Great if you have “few” DB Servers PAGE 36Wednesday, 5 December 12
  55. 55. Why MySQL/MariaDB on EC2? • You can choose among different technologies • MySQL, MariaDB, Percona • MySQL Replication vs Galera, MySQL Cluster, TokuDB, InfiniDB etc. • You want to use High I/O instances and SSDs • You want to control the deployment with tuned configuration params • You want to save between 10% and 25% PAGE 37Wednesday, 5 December 12
  56. 56. Why MySQL on Rackspace? • Fanatical about support! • Other systems running on Rackspace - Rackspace hosting • More control of your instances PAGE 38Wednesday, 5 December 12
  57. 57. Why Google Cloud SQL? • You develop and use applications with Google AppEngine PAGE 39Wednesday, 5 December 12
  58. 58. For More Information... • Try the Configurator and How-To Video: http:// cloud.skysql.com • The Admin Console: http://www.skysql.com/ downloads/community/skysql-console • Forums: https://www.skysql.com/forums/ skysql-cloud-data-suite • Web: https://www.skysql.com/cloud • Bugs: http://bugs.skysql.com • Try it! http://config.skysql.com • Free SkySQL account available for testing • Test on your account PAGE 40Wednesday, 5 December 12
  59. 59. Thank You! ivan@skysql.com www.skysql.com izoratti.blogspot.com www.slideshare.net/izorattiWednesday, 5 December 12

×