VMware Backups That Work—Lessons Learned From VADP Performance Benchmark Testing
Upcoming SlideShare
Loading in...5
×
 

VMware Backups That Work—Lessons Learned From VADP Performance Benchmark Testing

on

  • 7,818 views

We’ve pushed the backup performance envelope so that you don’t have to! Industry leaders Cisco, NetApp, VMware and Symantec teamed up to develop a best practice framework and performance benchmark ...

We’ve pushed the backup performance envelope so that you don’t have to! Industry leaders Cisco, NetApp, VMware and Symantec teamed up to develop a best practice framework and performance benchmark based on the vStorage APIs for Data Protection (VADP). The test configuration uses the popular NetApp FlexPod environment. The result that proves that you can easily protect over 4 TB of virtual machine data an hour. Most think that to provide this sort of performance, a mountain of hardware is required. This is not the case. We show you how these performance numbers can be easily obtained with minimal hardware and a small budget. Improving backup performance also creates more reliable backups, shorter backup windows and less impact on the vSphere infrastructure. Whether or not you are using NetBackup, we invite you to attend this hangout where we'll dig deep into VMware VADP and its performance characteristics. We'll share lessons learned from extensive lab benchmarks simulating real production workloads. This will help you design and deploy a backup solution for your VMware vSphere environment that meets your business SLAs.

Bonus exclusively at this hangout: In this hangout, we will provide a sneak preview some of NetBackup for VMware features coming in NetBackup 7.6. We have exciting results to share on how NetBackup is pushing the performance envelope further!

Panel Members:
• Abdul Rasheed, Product Marketing Manager, vExpert
• George Winter, Technical Product Manager, vExpert
• Alex Sakaguchi, Product Marketing Manager

View Hangout: http://bit.ly/1efz3zB

Statistics

Views

Total Views
7,818
Views on SlideShare
7,818
Embed Views
0

Actions

Likes
1
Downloads
287
Comments
0

0 Embeds 0

No embeds

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

VMware Backups That Work—Lessons Learned From VADP Performance Benchmark Testing VMware Backups That Work—Lessons Learned From VADP Performance Benchmark Testing Presentation Transcript

  • Disclaimer This information is about pre-release software. Any unreleased update to the product or other planned modification is subject to ongoing evaluation by Symantec and therefore subject to change. This information is provided without warranty of any kind, express or implied. Customers who purchase Symantec products should make their purchase decision based upon features that are currently available. 1
  • Participate in the discussion! 2 #SYMCHangout http://bit.ly/1efz3zB
  • VMware Backups That Work George Winter Technical Product Manager, Symantec Abdul Rasheed Technical Marketing Manager, Symantec Lessons Learned From VADP Performance Benchmark Testing 3
  • Agenda – NOTE: UPDATE THIS Why Perform These Benchmarks? Benchmark Testing History Benchmark – NBD Transport Benchmark – SAN Transport VADP Process – Quick Review The backup snapshot process HW and Backup Configuration Recommendations Which Method is Fastest? 4
  • Benchmark Testing History 5
  • Three VMware Backup Benchmarks • Requires significant hardware & very slow • VCB has since been deprecated (not going to discuss) VMware Consolidated Backup (VCB) • vStorage API for Data Protection • NetApp FlexPod, Cisco UCS, NetBackup NBD (network) benchmark • vStorage API for Data Protection • Cisco UCS, NetBackup SAN transport benchmark 6
  • Benchmark Environments – Real World Configurations 7 Cisco UCS Blades 1x vCenter, 6x ESXi Nexus NetBackup 5220 NetApp FAS 6080 NetApp V-Series 6080 Block Storage 8Gb Fibre Channel 10Gb Ethernet
  • Why Perform These Benchmarks? Benchmarks Are A Lot Of Work! 8
  • Benchmark Goal – Better Backups For You! Trust but verify – never assume performance Develop independent performance testing techniques Tell You What We Learned Understand hardware and backup API (VADP) limitations Faster backups More reliable Correctly size backup server Shorter backup window Restore performance Restore options 9
  • 10 VADP Backup Process Quick Review
  • VADP Backup Process 11 VMware ESXi Datastore Backup Server ESXi Host
  • VADP Backup Process 12 VMware ESXi Datastore Backup Server VADP Snapshot of Target VM(s) ESXi Host
  • VADP Backup Process 13 VMware ESXi Datastore Backup Server VM Backup Sent to Server ESXi Host
  • VADP Backup Process 14 VMware ESXi Datastore Backup Server Snapshot Released ESXi Host
  • Participate in the discussion! 15 #SYMCHangout http://bit.ly/1efz3zB
  • 16 Backup Snapshots Exposed! Why Do You Care?
  • A Closer Look - What Happens During a Backup? • VSS provider flushes OS buffers in VM • Snapshot of VM is taken (vmdk(s) are frozen) (SCSI reservation of LUN) • Redo log created – all writes redirected to redo log (Redo expanded in 16 MB chunks) • VM is backed up • Redo log data applied to original vmdk(s) • Snapshot released - backup completed During VM backups, VMware creates temp snapshot of VM. Process is as follows: • Every one of these steps involves significant amount of I/O • Reducing number of snapshots per Datastore improves backup perf & reliability • Incrementals are quick and can reduce snapshot impact Why does this matter? A Detailed Look at the VADP Snapshot Process 17 VMware ESXi
  • Reducing Backup I/O Impact – Improve B/U Reliability • Backup during periods of low VM activity • Use incremental backups (CBT) liberally • Limit simultaneous backups per ESX / Datastore (VIP) Shorten time backup snapshot is open • Design backup policies to evenly balance load across ESXi / Datastores • Faster backups = shorter time snapshots are open Configure NetBackup for optimal performance • Snapshots more reliable • Overall backup processing faster Result: 18
  • 19 VADP Snapshot Pop Quiz
  • VIP Configuration Exercise – NBD Transport 20 UCS VMDKVMDK VM2 ESXi ESXi ESXi ESXi ESXi ESXi Network Backup Server • Network = 10GbE • 6 ESXi Hosts • VMs = 100 GB Q: How many streams per ESXi host will provide fastest possible backup? 1, 2, 3, 4, more? 10GbE
  • 1 Stream / ESXi Host (6 streams total) Network = 10 GbE 100 MB/sec per ESXi Each VM = 100 MB/sec Backup 21 VMs / Hour NBD Transport – How Many Streams? 21 UCS VMDKVMDK VM2 ESXi ESXi ESXi ESXi ESXi ESXi 3 Streams / ESXi Host (18 streams total) Network = 10GbE 100 MB/sec per ESXi Each VM = 33 MB/sec Backup 21 VMs / Hour
  • VIP Configuration Example – NBD Transport (Review) 22 UCS VMDKVMDK VM2 ESXi ESXi ESXi ESXi ESXi ESXi Network Backup Server • Network = 10GbE • 6 ESXi Hosts • VMs = 100 GB 10GbE
  • NBD Transport – Impact Of Snapshots Per Datastore? 23 UCS VMDKVMDK VM2 ESXi ESXi ESXi ESXi ESXi ESXi 3 Streams / ESXi Host (18 streams total) Snapshots / DS = 3 Redo Space Req’d: 45 GB Per Datastore 1 Stream / ESXi Host (6 streams total) Snapshot / DS = 1 Redo Space Req’d: 5 GB Per Datastore ???
  • NBD Transport – Impact Of Snapshots Per Datastore? 24 UCS VMDKVMDK VM2 ESXi ESXi ESXi ESXi ESXi ESXi 3 Streams / ESXi Host (18 streams total) Snapshots / DS = 3 Redo Space Req’d: 45 GB Per Datastore 1 Stream / ESXi Host (6 streams total) Snapshot / DS = 1 Redo Space Req’d: 5 GB Per Datastore 45 GB
  • NBD Transport – Impact Of Snapshots Per Datastore? 25 UCS VMDKVMDK VM2 ESXi ESXi ESXi ESXi ESXi ESXi 3 Streams / ESXi Host (18 streams total) Snapshots / DS = 3 Redo Space Req’d: 45 GB Per Datastore 1 Stream / ESXi Host (6 streams total) Snapshot / DS = 1 Redo Space Req’d: 5 GB Per Datastore Why 45 GB for 3 simultaneous backups? • Each VM backs up at slower rate • Snapshot for each VM is open 3 times as long • Time to capture data change in redo is longer • Does not improve overall backup performance!! 45 GB
  • 26 NBD (Network) Transport Lessons From Benchmark Testing
  • NBD (Network) Transport – The Good News Super easy to set up – 5 minutes on a bad day No software required on any VMware component Supports any Datastore type 27
  • NBD (Network) Transport Data Flow – Heat Map 28 LANVMware ESXi Backup Server System Load • ESXi host does all the work  Pulls data from storage  Sends data to backup server
  • NBD (Network) Transport – The Not-So-Good Direct resource impact on ESXi host Backup data path directly from ESXi host to Backup system Backup data transfer through VMkernel port • Our benchmark testing and VMware have verified this • First backup stream throughput approximately 100MB/sec • VMware throttles VADP throughput to reserve VMkernel port bandwidth VMkernel port is bandwidth limited by VMware (e.g. QoS) We’ve developed techniques for working around this…. 29
  • VADP Performance – VMkernel Bandwidth Limits • Single ESXi host – NBD Performance Per Stream 30 # of streams: 1 2 3 4 100 MB/Sec 100 MB/sec 75 MB/Sec 60 MB/Sec 55 MB/Sec 220 MB/sec Creates VMkernel strain on ESXi host Based on Cisco/VMware/NetBackup Benchmark Testing 150 MB/S Total 100 MB/S Total 180 MB/S Total 220 MB/S Total
  • NBD Best Practices For Optimal Backup Performance • Number of simultaneous backups? 31 Aggregate Backup Speed Recommended Settings: • VMware recommends no more than 4 backup streams per ESXi host • We recommend 1 or 2 streams if possible • 10 or more ESXi hosts: Number of streams = 1 • Less than 10 ESXi hosts: Number of streams = 2
  • Participate in the discussion! 32 #SYMCHangout http://bit.ly/1efz3zB
  • SAN Transport 33
  • SAN Transport Data Flow – Heat Map 34 VMware ESXi Backup Server System Load • Skip the ESXi host!
  • SAN Transport – The Good News Supported with Fibre or iSCSI storage (no NFS support) Only “off-host” backup for VMware Near zero impact on ESXi hosts Backup server does all of the work More ESXi resources reserved for your VMs No VMkernel performance limitations Fastest possible backup performance! Restore over SAN also possible regardless of backup transport (check with vendor) 35
  • SAN Transport – The Not-So-Good Storage shared between ESXi host and backup system Need to understand the risks Backup system can overwrite VMware partition 36
  • SAN Transport – Best Practices For Protecting Storage • http://technet.microsoft.com/en-us/library/ff715663.aspx Use Windows SANpolicy: Restrict Administrator or Root access on Windows/Linux backup system Disable Disk Management snap-in Group Policy Disable Windows automount on backup system Configure LUNs as “read only” 37
  • HotAdd Transport 38
  • HotAdd Transport • Can be fast - bypasses the VMkernel port • Supports all storage types • Ideal for remote office environments The Good News • Directly impacts the ESXi host with all backup processing • Proxy VMs need to access the datastore of VMs being protected • Configuration issues – add SCSI controllers in VM • Issues with stuck snapshots, CBT on HotAdd VM, etc. The Not-So-Good 39
  • How About No Transport? 40
  • Array Based Snapshot – Heat Map 41 VMware ESXi Backup Server System Load • Zero load on any vSphere component • No VMkernel bandwidth limit • Ultimate performance 1 2
  • Storage Array Based Snapshot – What To Look For Ability to schedule snapshots just like any other backup Backup application should automatically handle entire data lifecycle •Expensive •Can be required for single file restore No requirement for additional “backup proxies” Automatic VM selection based on storage location •Throttle snapshot creation •Throttle snapshot deletion •Critical for limiting I/O “storms” Control quiesce process – consistent backups 42
  • Storage Array Based Snapshot – What To Look For Ability to schedule snapshots just like any other backup Backup application should automatically handle entire data lifecycle •Expensive •Can be required for single file restore No requirement for additional “backup proxies” Automatic VM selection based on storage location •Throttle snapshot creation •Throttle snapshot deletion •Critical for limiting I/O “storms” Control quiesce process – consistent backups 43 NetBackup 7.6 NetBackup 7.6 NetBackup 7.6 NetBackup 7.6 NetBackup 7.6
  • NetBackup Replication Director For VMware 44 Protect 300 VMs in 300 Seconds!
  • 45 Recommendations
  • Recommendations - Hardware • You are paying VMware for CPU & RAM – use these resources for production Avoid using a VM as backup system • Two socket, 4 cores per socket plenty of processing power Modern CPUs are incredibly fast • 32 GB is minimum • 64 GB if using deduplication • More is always better – future proofs this system RAM • Performance of internal bus important • Consider multiple HBA’s for network (teaming) or SAN (DMP) Pay attention to I/O 46
  • Recommendations - Software •Spread backups across all Datastores •Reduces snapshot impact – I/O and space Limit active backups per Datastore (Resource Limits) •Avoid spending resources protecting VMs that don’t need it •Automatically protect VMs that do need it Use automatic VM backup selection •Extremely fast •Be sure solution controls every aspect: Storage array snapshots add another dimension of protection •300 VMs in 300 minutes Backups, restores, snapshot retention, migration to long term storage 47
  • If your using full backups, your doing it the wrong way! 48
  • Advantages of the NetBackup Accelerator for VMware – Significantly less CPU utilization saves HW and VMware license costs – Backup is significantly faster – backup more VMs with same/less HW – Retain all restore functionality – single file, folder, VM, database, application Backup Processing Impact ESXi Host - CPU Utilization Standard FULL VADP backup (NBD transport) NetBackup Accelerator For VMware 49
  • Advantages of the NetBackup Accelerator Backup (network) – Significantly less network traffic on already busy VMkernel port (NIC) – Less backup hardware required – Shorter backup window Backup Processing Impact ESXi Host - Network Utilization Standard FULL VADP backup (NBD transport) NetBackup Accelerator For VMware 50
  • NetBackup Accelerator for VMware • Protect extremely large VMs in seconds • Faster, more reliable backups 51
  • Deduplication Technologies • Benefits to hard to ignore • Be careful of solutions that are basically compression Quickly replacing standard disk as backup target Trade I/O capacity (expensive) for CPU and RAM (cheap) • Minimums = 64 GB RAM, 2 Sockets, multiple spindles BYOD (Bring Your Own Disk) solutions • NetBackup Appliance – performance tuned • Hardware configuration correctly designed • Easy – start protecting your VMs in 10 minutes Deduplication Appliances 52
  • Thank you! Copyright © 2013 Symantec Corporation. All rights reserved. Symantec and the Symantec Logo are trademarks or registered trademarks of Symantec Corporation or its affiliates in the U.S. and other countries. Other names may be trademarks of their respective owners. This document is provided for informational purposes only and is not intended as advertising. All warranties relating to the information in this document, either express or implied, are disclaimed to the maximum extent allowed by law. The information in this document is subject to change without notice. 53 George Winter @vmxgeorge Abdul Rasheed @AbdulRasheed127