Oracle ZFS Storage ZS3-2 How to Achieve 25TB/hour Backups and 15TB/hour Restores for Your Oracle Database

2. How to Achieve 25TB/hour Backups and 15TB/hour Restores for Your Oracle Database Doug Chamberlain, Principal Product Manager Greg Drobish, Senior Software Engineer Gagan Singh, Intel Corporation

3. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.3 Program Agenda §  Engineered Systems Backup Challenges §  Oracle ZFS Storage §  Exadata Backup Test Results §  Customer Case Study: Intel Corporation

5. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.5 •  Risk –  Data loss – 1 in 7 backups & 1 in 6 restores fail* –  Slow data recovery leads to business outages –  Missed SLAs lead to lost opportunities –  Data-at-rest security and regulatory concerns •  Complexity –  More systems and storage to integrate –  More data and shorter backup windows –  Separate database and storage processes •  Managing Budgets –  Reducing CAPEX & OPEX –  Improving staff efficiency –  Freeing resources for investment Top Backup and Recovery Concerns * Backup and Restore Failure Rate from Enterprise Strategy Group, 2008

7. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.7 Oracle’s New ZS3 Storage Systems •  Over 100M hours in production environments •  Relied on by the majority of the Global F100 •  Foundation of Oracle *IT and Oracle cloud Trusted Storage Architecture •  Fastest database queries (GB/s) •  Fastest database responsiveness (latency) World-Record Performance •  3rd Gen. Hybrid Storage Pool (1/5th $/IOPS) •  2-4x more I/O efficient than everyone •  Advancements in award-winning analytics Best-in-Class Efficiency •  Only with OISP (cut DB tuning time in ½) •  Best-in-class HCC (50x compress, block/file) •  Integrated ADG, EM, OVM, and Eng. Systems Best for Oracle Software

8. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.8 ZFS Storage Appliance for Oracle Engineered Systems Backup & Recovery Application Engineered Storage for Oracle Engineered Systems Backup Less Risk with Extreme Performance 25TB/hour backup, 15TB/hour restore Less Complexity with Fewer Storage Systems and Streamlined Management Lower TCO with Extreme Efficiency

9. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.9 •  Less Business Risk –  Short backup windows with Infiniband and optimized Direct NFS –  Data reduction with encrypted data –  Fast, predictable restore times –  Integrated with StorageTek tape tier for long-term archiving •  Less Complexity –  HCC compression and higher performance mean fewer storage systems –  Secondary processing with no additional storage systems –  Configuration Utility for fast, foolproof setup –  No changes to Oracle best practices •  Lower TCO –  Fewer systems for lower capital, integration, and operating costs –  Unsurpassed backup visibility with DTrace Storage Analytics –  Snap Management Utility for DBA management Oracle ZFS Storage Appliance Advantages Co-Engineered with Oracle Database and Engineered Systems

10. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.10 Oracle ZFS Storage is Fast •  Native InfiniBand Connectivity –  Highest bandwidth for general purpose storage –  Scales horizontally, allowing for upgrades –  Leverages engineered systems’ internally managed InfiniBand network, no network upgrade required •  Backup and restore operations can be automatically parallelized across all database nodes and Sun ZFS Storage Appliance channels and controllers •  Restore throughput 17TB/hr for Exadata 7 TBhr for SuperCluster •  Backup throughput 26TB/hr for Exadata 14TB/hr for SuperCluster

11. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.11 ZFS Storage Is Cost Effective and Complete •  Simple architecture reduces cost •  If not using tape, no media server required –  No backup and recovery software license fees •  Augment Oracle DR utilizing replicated copies of RMAN backups •  Full complement of data services available –  Thin provisioning –  Compression –  Snapshot –  Performance Analytics •  Layers of data integrity –  RMAN protects against logical corruption, ZFS Storage protects against physical corruption –  Self-healing Fault Management Architecture

12. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.12 Hybrid Columnar Compression §  Ingests RMAN image backups of HCC compressed tables §  Snaps and clones RMAN image backup §  Access cloned databases for test and development §  Preserve space savings – no need to uncompress cloned tables before accessing 10x to 15x Storage Space Savings, Unique to Oracle Compression Unit Column1 Column2 Column3 10x to 15x Reduction Column4 Column5

13. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.13 Configuration Utility Simplifies Deployment §  The Oracle Exadata Backup Configuration Utility configures ZFS Storage and Exadata nodes through command-line- based Wizard §  Required manual steps automated through utility; best practice configuration can be enforced §  Available via Oracle Technology Network Exadata Backup

14. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.14 Stay With Current Best Practices §  Implementation –  Tested, validated, and supported –  Integrated InfiniBand simplifies implementation §  Consistent with standard RMAN methodology –  Use a fast recovery area –  Perform an initial RMAN image copy backup –  Perform daily RMAN incremental level 1 backups §  Effectively an “incrementals forever” strategy –  RMAN incremental only back up of changed blocks –  Full backups unnecessarily burden the network

15. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.15 Simplifying Disk-to-Disk-to-Tape §  Disk-to-disk-to-tape (D2D2T) allows backup administrators to have the best of both worlds –  Keeping off-site backups elsewhere –  While retaining fresh backups on disk for ultra-fast restores §  Storage hierarchy that balances cost and performance §  Automated data movement enabled by Oracle Secure Backup or third-party backup and recovery applications –  Backup Backupset: RMAN-aware backup of disk backup on tape –  RMAN Archive to tape: managed and tracked by RMAN

16. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.16 Disk-To-Disk-To-Tape RMAN Oracle Exadata Database Machine ZFS Storage Appliance StorageTek SL3000 Tape Library Media Management Server with Oracle Secure Backup, or third-party backup & recovery software Backup Backupset Tape Backup

18. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.18 Exadata Backup Test Results •  Clustered ZS3-4 •  1 TB DRAM (512GB per controller) •  InfiniBand Connectivity •  ZFS Storage OS8 Software •  No Read Cache SSD Controllers •  8 High Capacity Disk Shelves •  7200 RPM drives •  No Write Log SSD Enclosures §  Peak Sustained Backup: –  23 TB/hr (backupset) –  26 TB/hr (imagecopy) §  Peak Sustained Restore: –  15 TB/hr (backupset) –  17 TB/hr (imagecopy)

19. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.19 Superior RAID-Z Performance Disk Capacity Utilization Mirrored 42.2% Single Parity 69.3% Double Parity 76.7% Disk1 B1 C1 A1 Disk2 B2 Cp A2 Disk3 Bp C2 A3 Disk4 B3 C3 Ap RAID-Z Parity §  High performance RAID-Z gives customers the choice of mirrored or striped parity based on their needs §  Variable stripe width means every write is a full stripe write §  RAID-Z reduces the IOPS overhead

20. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.20 RAID-Z1 vs Mirror: Calculating IOPS Overhead Example Backup – Why is RAID-Z1 Faster? Single Parity 3419 MBPS Mirrored 3337 MBPS •  IOPS per disk correlate to AWR average wait time Mirroring •  IOPS per disk correlate to AWR average wait time Single Parity §  Single Parity –  158.2 IOPS per every 100 MBPS –  Average disk IO size: 700.8 KB §  Mirrored –  250.6 IOPS per every 100 MBPS –  Average disk IO size: 841.1 KB •  Mirroring generates 58% more IOPS •  5x higher DB wait time Conclusion

21. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.21 Optimization For Image Copy or Backupset Image Copy: §  Infinite Incremental, synergies with cloning, superior performance, generally accessible Backupset: §  Tape support, bypasses unused datafile blocks, RMAN default, multi-section backup •  Fewer channels •  More buffers per channel •  1MB buffer sizes Backupset Tuning •  More channels •  Fewer buffers per channel •  4MB buffer sizes Image Copy Tuning General Performance Recommendations Image Copy 6-12 channels per database server, 8 buffers per channel, 4MB buffer size Backupset 2-4 channels per database server, 64 buffers per channel, 1MB buffer size Incremental Merge Limit read-modify-write penalty (narrow stripes or mirroring, consider SSD)

22. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.22 RMAN Tuning Recommendations §  Use filesperset=1 for all backupset and incrementals to improve buffer performance and streamline restores §  Set _file_size_increase_increment=2044M to optimize how RMAN allocates new space for backups §  Use section size 48g to optimize multi-section backups §  Configure Oracle Net Services for optimal performance and availability §  Reset or remove _backup_ksfq_bufsz and _backup_ksfq_bufcnt deprecated parameters RMAN Backup Services Multi-Section Backups

23. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.23 Save Money on SSD, Spend it on HDD •  Transaction groups transfer data from the ARC to HDD TXG •  All writes whether synchronous or asynchronous are first written to the ARC Incoming Writes §  SSD is critical for achieving optimal transactional and mixed-IO performance §  Dedicated RMAN workloads may perform better without SSD •  Synchronous writes are committed to the ZIL before an acknowledgement is returned to the client Synchronous Writes

24. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.24 §  RMAN backupsets require synchronous semantics but are not very latency sensitive §  IO queuing can create a bottleneck and reduce throughput §  Set logbias=throughput will bypass SSD write cache (if present) and write to HDD SSD Is Not Always Faster Example Backup (Backupset) No Write SSD 2590 MBPS 4 Mirrored Write SSD 846 MBPS •  Analytics shows that IO queuing on the 4 SSDs is creating a bottleneck IO Queuing •  Bandwidth intensive workloads benefit from wide IO distribution across the storage pool Sharing the Load

25. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.25 More HDD Improves Performance §  Peak sustained throughput increases as more disk are added to the ZS3 configuration §  For optimal performance always use a single storage pool per controller for RMAN backup/ restore §  High performance drives scale RMAN performance at a steeper rate ZS3 RMAN Performance Scaling Adding Disk Shelves: 7200 RPM High Capacity Drives Single Parity RAID 1 shelf 2 shelves 3 shelves 4 shelves 8 shelves Backup (Image copy) 5.1 TB/hr 9.8 TB/hr 13.8 TB/hr 20.5 TB/hr 26 TB/hr Restore (Image copy) 1.8 TB/hr 4.0 TB/hr 6.3 TB/hr 11.4 TB/hr 17 TB/hr

26. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.26 Infiniband Enables Superior Performance §  Infiniband fabric provides an extremely high performance network for transferring backup data –  Low-latency, high-bandwidth (2.1 GB/s), low CPU utilization –  Seamless integration with Exadata §  Use an active/active IPMP configuration •  Active/active IPMP group provides performance and redundancy Active/Active •  Peak throughput per controller with low CPU overhead CPU

27. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.27 Direct NFS §  dNFS provides integrated performance tuning when backing up an Oracle database to a ZS3 –  Bypasses the operating system –  Data is cached just once in user space with no 2nd copy in kernel space §  Load balance across multiple network interfaces and stripe network buffers over up to 4 addresses •  Oracle Intelligent Storage Protocol (OISP) integrated with dNFS in 12c dynamically tunes record size and logbias variables on the ZFS Storage Appliance to optimize performance OISP Example: Direct NFS Striping Number of Addresses 1 2 4 Backup 4209 MB/s 5261 MB/s 5417 MB/s

29. 29 Oracle Exadata Database Machine Backup and Recovery with the Sun ZFS Backup Appliance Gagan Singh Sr. Database Architect Intel Corporation

30. 30 Agenda • Company Overview • Legacy Data Warehouse Environment & Architecture • Previous Backup Architecture • Impetus for Exadata • Why ZFS ? • Exadata Solution Architecture • New Backup Implementation • Lessons Learnt

31. 31 Intel Corporation Company Overview •  U.S.-based, publicly traded multinational semiconductor chip maker corporation headquartered in California •  Designs and manufactures advanced integrated digital technology platforms •  Microprocessor and chipset •  Develop, sell software and services primarily focused on security and technology integration •  $53.3B in revenue for 2012* * http://www.intc.com/intel-annual-report/2012/

32. 32 Legacy Data Warehouse Environment •  Manufacturing Data Warehouses -Highly automated environment with complex integrated systems -Geographically independent systems, decentralized environment •  Reliability, Availability and Performance •  Robust Backup and Recovery, but lacked performance - Affects scheduling •  No formal VLDB Disaster Recovery - Data not protected from loss of a site

33. 33 Previous Backup Architecture •  All Oracle Databases backed up to Flash Recovery Area •  RMAN Disk-to-Disk-to-Tape backup strategy •  Backup Strategy •  Level 0 to disk; Block Change Tracking for daily Level 1 backups •  Incrementally updated: L1 merged into L0 •  Weekly L0 and Daily L1 also stored on tape •  Challenges •  Image copies needs same space as database size •  Several operational issues managing backups •  Tape bandwidth is limited •  Backup load on Primary Server/Database Oracle Database Server NetBackup Media Server Virtual Tape Library FRA

34. 34 Impetus for Exadata Purchase •  Increase data retention without impacting SLA by leveraging compression; hardware reliability; compute power •  Compelling Value Proposition •  Exadata X3-8 for production and one half-rack for pre-production •  Compression •  Redundancy •  Platinum Support •  Scalability •  Faster backup & recovery times •  I/O Resource Manager

35. 35 Why ZFS ? Speed • Reduced backup windows by more than 2.5X • I/O is external to Storage Area Network, helping reduce backup windows • Improved Recovery Point and Recovery Time Objectives Simplicity • Flexibility to clone databases • Move data between production and pre- production • Manageability through Grid Control Savings • Lower TCO than competitive product • Storage for Offline database • Storage – HCC Compression Support • Single vendor escalation and support • Interactive admin GUI • Platinum Gateway Monitoring Value

36. 36 New Backup Architecture •  ZFS Storage Appliance •  Connected via InfiniBand to Exadata •  RMAN Backupsets from Active Data Guard Host •  Centralized Backup from EM 12c •  Backup Monitoring through EM 12c

37. 37 Lessons Learned 1.  DnFS enabled, channel distribution across Exadata Nodes 2.  Split loads evenly on both controllers 3.  Ability to define logical space within ZFS appliance. 4.  Leverage the ZFS GUI for real time Monitoring and Operations 5.  _backup_disk_bufcnt=64 6.  _backup_file_bufcnt=64

38. 38 Reference Notes Sun Storage 7000 Unified Storage System: FAQ: RMAN backup from Exadata to Sun ZFS Backup Appliance (Doc ID 1354980.1)

39. Thank You

40. Copyright © 2013, Oracle and/or its affiliates. All rights reserved.40 Additional Resources §  Backup and Recovery Performance and Best Practices using Sun ZFS Storage Appliance with Oracle Exadata Database Machine http://www.oracle.com/technetwork/database/features/availability/maa-wp-dbm-zfs-backup-1593252.pdf §  Backup and Recovery Performance and Best Practices for Exadata Cell and the Sun Oracle Database Machine http://www.oracle.com/technetwork/database/features/availability/maa-tech-wp-sundbm- backup-11202-183503.pdf §  Exadata Backup & Recovery My Oracle Support Note: https://support.oracle.com, ID 1354980.1 §  SuperCluster Backup & Recovery My Oracle Support Note: https://support.oracle.com, ID 1517107.1 §  Oracle Exadata Database Machine: http://www.oracle.com/exadata §  Oracle SuperCluster: http://www.oracle.com/supercluster §  Oracle Recovery Manager: http://www.oracle.com/goto/rman §  Sun ZFS Storage Appliance: www.oracle.com/us/products/servers-storage/storage/unified-storage