MySQL with Enterprise     Storage Presented By Peter     Teitelbaum
Peter’s Background•    Working with MySQL since 2003•    Programming since 1995•    Working with Linux since 1999•    Pres...
Introduction to Storage•    Approach to storage will greatly affect things like availability, scalability,    replication,...
What is LVM•    Logical Volume Management is a way to    virtualize storage•    An abstraction layer over disk storage
LV / VG / PV Relationship Logical   / dev/myvolgroup/ logs     /dev/ myvolgroup/ data     / dev/myvolgroup/ images        ...
Logical / Physical MappingLogical blocks    VA          VA        VA in a volume      LB1         LB2       LB3Physical Bl...
Snapshots vs. Clones•    A snapshot is a static/read only image of the data at a    specific point in time.•    A clone is...
Creating a snapshot or cloneLogical blocks           VA             VB in a volume             LB1            LB1Physical ...
Copy-On-Write (COW)                         (write)Logical blocks           VA                      VB in a volume        ...
LVM doesn’t actually use blocks•    File systems – block size     –         A block is logical container of data with a co...
How it works: Device Mapper•    Device Mapper is a Linux framework used to create create block    devices which are mapped...
Backing up MySQL•    Backup basics:     –   Online vs. offline & physical vs. logical           •               http://dev...
Quiescing MySQLInitiate backupGet dirty page % S % to zero  et  Set session                                       Kill glo...
Recovery•    Reverting to a backup cannot be undone•    Future backups lost
Recovery & Future Backups    Backups
Recovery•    Reverting to a backup cannot be undone•    Future backups lost•    Near instant recovery, no need to copy dat...
File System Architecture                                                             Isolated                             ...
Granular Recovery•    Table or row level recovery•    Use a dedicated data recovery host     –   Create clone of snap (MyS...
Retention•    Process to purge snaps based on age    –   Example: 10d; 7d, 4w, 2m; etc.•    Move archive backups to anothe...
Replication•    Run slaves on clone of parent (master)
Replication – shared diskUnderlying physical disks are shared for master/slave volumes      Master              Slave1    ...
Replication•    Run slaves on clone of parent (master)•    Shared blocks on parent and slaves•    Add slaves quickly to ad...
Resync a slaveRe-syncing a slave or reclaiming disk space is fast and easy      Master1                                   ...
Slave Reclone•    Create a clone of a backup snapshot    –   Will require time for replication to catch up•    Clone from ...
Replication•    Run slaves on clone of parent (master)•    Shared blocks on parent and slaves•    Add slaves quickly to ad...
Other Things to Consider•    Monitor for most recent backup•    Automated recovery testing host•    Monitor for backup qua...
Thanks!Questions/thoughts?
Upcoming SlideShare
Loading in …5
×

My sql with enterprise storage

573 views

Published on

  • Be the first to comment

  • Be the first to like this

My sql with enterprise storage

  1. 1. MySQL with Enterprise Storage Presented By Peter Teitelbaum
  2. 2. Peter’s Background• Working with MySQL since 2003• Programming since 1995• Working with Linux since 1999• Presently Dir. of Database Ops at Clear Channel Media and Entertainment – With Clear Channel since 2008 and lead the DB team – Users generate around 200 million page views per month – MySQL used as primary data store – Many instances of MySQL deployed – Primary applications generate around 15K queries/sec
  3. 3. Introduction to Storage• Approach to storage will greatly affect things like availability, scalability, replication, backup/recovery, and DR.• SAN vs. attached storage• Storage performance/architecture: – Spindles (IOPS) vs. disk capacity – Write caching• Using a hosting facility and leasing SAN storage: – Disks/networking shared with other customers? – API/interface available to take/restore snapshots
  4. 4. What is LVM• Logical Volume Management is a way to virtualize storage• An abstraction layer over disk storage
  5. 5. LV / VG / PV Relationship Logical / dev/myvolgroup/ logs /dev/ myvolgroup/ data / dev/myvolgroup/ images 10GB 600GB 250GBVolumesVolume / dev/myvolgroup 1500GBGroupPhysicalVolumes /dev/sda1 / dev/sdb1 / dev/ sdc1 500GB 500GB 500GB
  6. 6. Logical / Physical MappingLogical blocks VA VA VA in a volume LB1 LB2 LB3Physical Blocks on disk/LUN PB1 PB2 PB3 VA = Volume A
  7. 7. Snapshots vs. Clones• A snapshot is a static/read only image of the data at a specific point in time.• A clone is a dynamic/writeable image of the data at a specific point in time.• A clone can appear to be a snap when mounted read only
  8. 8. Creating a snapshot or cloneLogical blocks VA VB in a volume LB1 LB1Physical Blocks on disk/LUN PB1 VA = volume A, VB = snapshot of VA
  9. 9. Copy-On-Write (COW) (write)Logical blocks VA VB in a volume LB1 LB1 X X XPhysical Blocks on disk/LUN PB1 (copy) PB2 VA = volume A, VB = snapshot of VA
  10. 10. LVM doesn’t actually use blocks• File systems – block size – A block is logical container of data with a configurable fixed size in bytes. A block is written/read from disk in a single disk I/O operation.• RAID – stripe size – The smallest unit of storage allocation that can be written to each disk. It is a fixed size measured in bytes. (i.e. RAID 0,4,5,6)• LVM – extent size – The smallest unit of storage allocated to each physical volume. An extent is a fixed size measured in bytes.
  11. 11. How it works: Device Mapper• Device Mapper is a Linux framework used to create create block devices which are mapped to other block devices. It is the foundation for: • Linux mdadm (RAID) • Linux LVM2 • File system encryption • and moreFor more info: http://mbroz.fedorapeople.org/talks/DeviceMapperBasics/dm.pdf
  12. 12. Backing up MySQL• Backup basics: – Online vs. offline & physical vs. logical • http://dev.mysql.com/doc/refman/5.5/en/backup-types.html – Mysqldump is good for portability, bad for fast recovery • Sequential export, not globally atomic, holds locks, long running processes • Can take a long time to restore• File system copy will be impractical if large dataset• Snapshot is extremely fast• Quiescing the database• Can be more frequent than traditional backups• Transfer backups to another medium
  13. 13. Quiescing MySQLInitiate backupGet dirty page % S % to zero et Set session Kill global lock timeout yes Call lock no Lock monitor Sleep Hanging lock? monitor terminates no Capture binlog Acquire global Dirty pages? Check dirty pages position and Take snapshot Release lock lock Or timeout processlist yes Restore dirty page Sleep Backup complete %
  14. 14. Recovery• Reverting to a backup cannot be undone• Future backups lost
  15. 15. Recovery & Future Backups Backups
  16. 16. Recovery• Reverting to a backup cannot be undone• Future backups lost• Near instant recovery, no need to copy data files, untar or source in a dumpfile• Any slaves will need to be rebuilt• PITR – can be automated or performed manually
  17. 17. File System Architecture Isolated Datadir Binary Logs /var/lib/mysql/ /var/lib/mysql-binlog Unified Datadir & Binary Logs• Isolated /var/lib/mysql/ – Only datadir is restored, binlogs left untouched – Simpler recovery process – Transactions will be duplicated during PITR – Slaves maintain binlog file and position• Unified – Datadir and binlogs are restored – Binlogs must be copied to another location for PITR – More complicated recovery process – Slave replication will fail
  18. 18. Granular Recovery• Table or row level recovery• Use a dedicated data recovery host – Create clone of snap (MySQL needs r/w file system) – Don’t circumvent with clones as backups• Extract via mysqldump then source into production – Use WHERE clause (-w ‘id=123’) – Pipe to sed to convert INSERT to REPLACE, etc.• MyISAM, copy the 3 files, rename, then RENAME TABLE• InnoDB, create dumpfile, source in with a new table name, then RENAME TABLE
  19. 19. Retention• Process to purge snaps based on age – Example: 10d; 7d, 4w, 2m; etc.• Move archive backups to another medium – Will never be restored – Disk usage grows while aging – Greater the difference to parent, the greater IO overhead
  20. 20. Replication• Run slaves on clone of parent (master)
  21. 21. Replication – shared diskUnderlying physical disks are shared for master/slave volumes Master Slave1 Slave2 Slave3 /vol/mysql_master /vol/mysql_slave1 /vol/mysql_slave2 /vol/mysql_slave3 Aggregate
  22. 22. Replication• Run slaves on clone of parent (master)• Shared blocks on parent and slaves• Add slaves quickly to add capacity• Removes need for slave resync tools
  23. 23. Resync a slaveRe-syncing a slave or reclaiming disk space is fast and easy Master1 Slave1 (destroy clone volume) X /vol/mysql_master /vol/mysql_slave1 (old) (create new volume as clone) Slave1 /vol/mysql_slave1 (new)
  24. 24. Slave Reclone• Create a clone of a backup snapshot – Will require time for replication to catch up• Clone from live master – Will need to quiesce master like during backup• Consider urgency and timing
  25. 25. Replication• Run slaves on clone of parent (master)• Shared blocks on parent and slaves• Add slaves quickly to add capacity• Removes need for slave resync tools• Entire reclone process should be automated• No need to back up slaves• Reclone regularly to reclaim space• Reclone after recover from backup• Mostly static or r/o data? Consider no MySQL replication – Replication is inefficient for large volumes of static data – Consider a reclone instead• Easier to rely on SAN for replication to DR facility than replicating multiple MySQL instances
  26. 26. Other Things to Consider• Monitor for most recent backup• Automated recovery testing host• Monitor for backup quality – Recoverable – Dirty• ETL host with automated reclones from backup
  27. 27. Thanks!Questions/thoughts?

×