Your SlideShare is downloading. ×

San in depth

922
views

Published on

Published in: Technology

0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
922
On Slideshare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
35
Comments
0
Likes
0
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide
  • This picture represents the basic framework of RAID-DP that I’ll be using in the rest of the talk. The bracket shows one 4 KB block on each disk. Unlike regular RAID, we divide the blocks on each disk into chunks – four 1 KB chunks in this example. All of the techniques that I’m going to show will apply to every block on the disk, but to keep things simple, I’m just going to focus on this one block.
  • The left 5 disks are handled as regular RAID 4. So here you can see that I’ve put data in the disks using the example from the first page. And sure enough, 3 + 1 + 2 + 3 equals 9. One of the nice things about RAID-DP is that it is a strict super-set of RAID 4, which means that it’s easy to take a RAID 4 group and upgrade it to RAID-DP, or take a RAID-DP group and convert it back to RAID 4, to reclaim the extra disk. TRANSITION: Now let’s look at how the Diagonal Parity works.
  • Here I’ve marked off a diagonal in blue. Notice that the diagonal includes not only the data disks from the RAID 4 array, but also the parity. We store the diagonal parity on the DP disk. Although the diagonal parity goes down the block as a diagonal, the parity calculation itself works just the same. So you can verify in this example that 1 + 2 + 2 + 7 equals 12. Also note that I’ve only filled in numbers for a few of the chunks. Right now, I’m just trying to help you understand the very basic operation of RAID-DP. I’ll fill in more details later. TRANSITION: So now let’s look at what happens if we fail a drive.
  • If we fail just one drive, then we can reconstruct the data just with regular old RAID 4. Take 9 – 3 – 2 – 1 and you get 3, which is what was there. TRANSITION: But suppose a second disk fails… CLICK
  • Now we would be hosed with normal RAID 4, because we are missing two values, but we only have one equation. But notice, we do still have a diagonal row that is missing only one element. So we can use the diagonal to reconstruct the missing block on the second disk. Do the math: 12 – 7 is 5, minus 2 is 3, minus 2 is 1. TRANSITION: Sure enough… CLICK
  • Now we have enough data to do the reconstruction by normal RAID 4. Do the math: 9 minus 3 is 6, minus 2 is 4, minus 1 is 3. TRANSITION: And sure enough… CLICK.
  • At this point, we’ve only reconstructed the missing chunk for the top row, but this simple example should help build your intuition for the next step, when we look at how to reconstruct the missing chunks for all of the rows. So far so good, but things are about to get much more complicated, so let’s review what we are doing. Remember that the bracket identifies 4 KB worth of data on each disk (one WAFL block), and we’ve divided that into four chunks, so that each little red dot represents 1 KB of lost data. The trick now is to show how to extend this same technique to cover all of the missing chunks in the picture. And remember also that this same technique can be applied to each block in the entire disk.
  • You’ll just have to trust me that all of these add up the way they should. But just as an example, let’s look at the pink diagonal: 2 plus 1 is 3, plus 3 is 6, plus 5 is 11. Sure enough. Now is a good time to take a deep breath, look at this whole picture and make sure you understand all the working pieces. TRANSITION: Now let’s kill a couple of drives.
  • The shows a simple configuration for illustrative purposes where there are VMs on two sets of servers in a HA cluster. The VMs have Reservation (lower resource limits) and Limits (higher resource limits) values explicitly. The actual level of usage of the VMs is between these two values. When one of blades fails the failing servers will be restarted on the remaining blades in the HA cluster with the result that the allocated resources to the VMs will be reduced dynamically to a lower value closer to their reservation threshold in order to accommodate the new VMs. This contains a suggested approach for setting resource allocation values in order to configure effective automatic recovery in a HA cluster. The following terms are used to define resource requirements: NS Number of servers in one half of a symmetrical HA cluster defined across both HP sites NPPS Number of processors per server PP Processing power of processor HSH High share resource allocation relative ratio number MSH Medium share resource allocation relative ratio number LSH Low share resource allocation relative ratio number NHVM Number of VMs with a share value set to High for which automatic disaster recovery is to be allowed NMVM Number of VMs with a share value set to Medium for which automatic disaster recovery is to be allowed NLVM Number of VMs with a share value set to Low for which automatic disaster recovery is to be allowed RF Reservation Factor – this is a ceiling for the total of the Reservation values for all virtual machines for which recovery is to be automated. Reservation Factor should be set to less than .5 in order to allow for processing resources for the virtualisation hypervisor. TPMR Total physical machine processing resource capacity RVU Reservation value unit – this is a notional amount of resources that when multiplied by RV Reservation value set for a virtual machine RVH This is the suggested reservation value to be set for a virtual server with a High share resource RVM This is the suggested reservation value to be set for a virtual server with a Medium share resource RVL This is the suggested reservation value to be set for a virtual server with a Low share resource TR This is the total of all the reservation values for virtual machines in one side of a symmetrical
  • The following is one way of determining how the Reservation values should be set. (1) TPMR = NS x NPPS x PP (2) RVU = TPMR x RF / (NHVM x HSH + NMVM x MSH + NLVM x LSH) (3) RVH = RVU x HSH (4) RVM = RVU x MSH (5) RVL = RVU x LSH (6) TR = RVU x (NHVM x HSH + NMVM x MSH + NLVM x LSH) Number of servers in one half of a symmetrical HA cluster defined across both locations 8 Number of processors per server 2 Processing power of processor 3.2 High share resource allocation relative ratio number 2 Medium share resource allocation relative ratio number 1.5 Low share resource allocation relative ratio number 1 Number of VMs with a share value set to High for which automatic disaster recovery is to be allowed 20 Number of VMs with a share value set to Medium for which automatic disaster recovery is to be allowed 20 Number of VMs with a share value set to Low for which automatic disaster recovery is to be allowed 20 Reservation Factor – this is a ceiling for the total of the Reservation values for all virtual machines for which recovery is to be automated. Reservation Factor should be set to less than .5 in order to allow for processing resources for the virtualisation hypervisor. .45 (1) TPMR = 8 x 2 x 5.2 = 51.2 (2) RVU = 51.2 x .45 / (20 x 2 + 20 x 1.5 + 20 x 1) = 0.256 (3) RVH = 0.256 x 2 = 0.512 (4) RVM = 0.256 x 1.52 = 0.384 (5) RVH = 0.256 x 1 = 0.256 (6) TR = 0.256 x (20 x 2 + 20 x 1.5 + 20 x 1) = 23.04
  • VMware ESX Server. A robust, production-proven virtualisation layer run on physical servers that abstracts processor, memory, storage, and networking resources into multiple virtual machines. VirtualCentre Management Server (VirtualCentre Server). The central point for configuring, provisioning, and managing virtualised IT environments. Virtual Infrastructure Client (VI Client). An interface that allows users to connect remotely to the VirtualCentre Server or individual ESX Servers from any Windows PC. VMware Virtual Machine File System (VMFS ). This is a high-performance cluster file system for ESX Server virtual machines. VMware Virtual Symmetric Multi-Processing (SMP). Feature that enables a single virtual machine to use multiple physical processors simultaneously. VMware VMotion. Feature that enables the live migration of running virtual machines from one physical server to another with zero down time, continuous service availability, and complete transaction integrity. VMotion is a technology used by the VMware DRS components VMware HA. Feature that provides easy-to-use, cost-effective high availability for applications running in virtual machines. In the event of server failure, affected virtual machines are automatically restarted on other production servers that have spare capacity. VMware Distributed Resource Scheduler (DRS). Feature that allocates and balances computing capacity dynamically across collections of hardware resources for virtual machines. VMware Consolidated Backup. Provides an easy to use, centralised facility for agent-free backup of virtual machines. It simplifies backup administration and reduces the load on ESX Server installations.
  • This lists sample costs for various VMware configurations. VMware is priced per pair of processors on which the software runs. VirtualCentre is sold separately. Only one VirtualCentre instance is needed for a virtual infrastructure, subject to architectural limits.
  • The elements of this option are: The primary server virtualisation infrastructure consists of two servers There is a separate server to run VirtualCentre to monitor, administer and control the virtual server environment. Data will be stored on a high-capacity, highly-resilient and reliable SAN. Server data will be initially backed-up onto a high-capacity, low-cost disk storage unit. Server data will then be backed-up to a LTO3 tape autoloader unit. This will reduce the manual effort associated with tape handling. The VirtualCentre server will provide centralised management, administration and control of the virtual server infrastructure. In the event of failure of one of the physical servers in the primary site, the HA component of VMware will allow the virtual servers on the failing physical server to be recovered onto the other physical server automatically.
  • Data will be backed-up from the primary SAN to a low-cost, high-capacity disk storage unit. This will enable rapid backup with minimal impact on production systems during the backup process. Data will then be backed-up to an LTO3 tape autoloader. This will reduce the manual effort associated with tape handling during backup.
  • Data will be backed-up from the primary SAN to a low-cost, high-capacity disk storage unit. This will enable rapid backup with minimal impact on production systems during the backup process. Data will then be backed-up to an LTO3 tape autoloader. This will reduce the manual effort associated with tape handling during backup. The backup data on the primary disk backup unit will be copied to a storage unit in the backup site to provide a copy from which data can be restored in the event of failure of the primary site. Backup tapes can be moved from the primary site to the backup site.
  • There are a number of architectural limits that affect large scale implementation: Number of virtual machines (for management server scalability) 1500 Number of physical hosts per DRS cluster 32 Number of physical hosts per HA cluster 16 Number of physical hosts per VirtualCentre server 100 Ultimately this will require two or more entirely separate Virtual Infrastructures each of which will be managed by entirely separate VirtualCentre systems. In this configuration, each Virtual Infrastructure has three blade enclosures of 16 blade servers each in each data centre. This means each Virtual Infrastructure has 96 physical hosts – 48 in each data centre for symmetry. This will impose additional hardware requirements for VirtualCentre systems and VirtualCentre database servers. In reality the number of physical hosts per Virtual Infrastructure may be lower because of the number of virtual machines running on the physical servers. 96 physical hosts should be able to run a minimum of 750 virtual servers which is considerably less than the threshold of 1,500. This minimum of 750 is based on an average of around four virtual machines per blade processor.
  • Multiple Clusters are defined up to the current maximum of 16 physical servers per HA cluster. VMware clusters are defined symmetrically across both sites. So, for a cluster of 16 physical servers, eight are located in each site. The VMware Cluster is designed to maximise recoverability while meeting agreed any SLA terms for resilience and high availability, maximising resource utilisation and long-term flexibility and minimising physical resource requirement. There is no ideal design that optimises all the factors. Some compromise is required. The easiest VMware Cluster design consists of two sets of identical resources across both data centres.
  • Like any IT project, the investment in implementing server virtualisation should be justified to ensure that it delivers real benefits. A cost benefit analysis is important tol enable you to prepare a business case for server virtualisation safe in the knowledge that the information it contains is accurate and detailed. It will equip you with all the facts you need to understand if server virtualisation will deliver you bottom-line business benefits.
  • DSS recommend the nworks SCOM Management Pack for VMware is selected for SCOM integration if required. The nworks MP provides full Alerting and Performance charting on VMware VI3 enterprise system status, as well as operational information. It collects: Performance and Event data for VMware ESX Hosts, either from VirtualCentre or ESX directly Performance and Event data for VMware ESX Guest Virtual Machines, either via VirtualCentre or ESX directly Events and Alerts from VirtualCentre in many categories such as security, status/state-change, object creation/deletion and other management & admin actions taken in VirtualCentre. The Topology of the Virtual Infrastructure within VirtualCentre – Data centres, Folders, Clusters, Hosts and Guests Events and Alerts from nworks own VEM Collector service The detailed data available in the nworks MP is delivered by use of the VMware SDK on VirtualCentre, which gives an accurate picture of the status of VirtualCentre, the managed Hosts, and the Guest Virtual Machines. The SCOM Management Packs runs the nworks Collector. The nworks Collector component is a Windows service which can run on a physical server or a Virtual Machine. The Collector is also referred to as VEM (Virtual Enterprise Monitor). The VEM server can be a virtual server to reduce cost. The nworks Collector architecture does not require the installation of software on the ESX Server. The nworks SCOM Management Pack two versions: VMware Events Only MP for SCOM - handled only VMWare events VMware MP for SCOM - covers both events and performance logging The second version is more expensive but more functional. It can collect up to 300 metrics on the operation of virtual servers.
  • Now that’s all a grand oversimplification, since there are lots of forces at work. Let’s look at what’s really happening. First, let’s talk about Volumes. Volumes are the basic building block—the unit around which all data management is based. Therefore, the tools and processes that we have to manage our data acts on volumes—like snapshots, SnapVault, and backup & restore. When you can act on the smallest unit, you can have very precise control—all these things lead to a push to continue to control data at the volume level. Meanwhile, volumes themselves are getting bigger, and the disks that hold them are getting bigger, faster, and cheaper. (click) so creates an opposing dynamic—towards bigger and bigger physical storage. At one end, we’ve got increasing storage, performance and cost pressures driving the adoption of bigger and bigger disks. (what role does the grid and RAID DP play here?) At the same time, we know that the key to using these big disks efficiently is to have highly customized control over the management of all aspects of the data. Tools like SnapShots, SnapMirror and Snapvault all depend on optimizing configuations at the volume level. (more examples? ILM depends on each volume being managed by the demands of its types of data; automated migration, restore on demand, etc.)
  • In order to accomplish this we are introducing a new entity to capture the physical characteristics of disks – We call it an aggregate. An aggregate is a collection of raid groups and is used to provide a large pool of storage for use by flexible volumes. There can now exist multiple flexible volumes in a single aggregate each of which can be dynamically resized. Reallocation of space is now an instantaneous non-disruptive operation. The thing to note is that from a data management perspective the basic container of data and the basic building block for your storage architecture is still a volume and aside from new features it maintains its properties of the past.
  • Aggregate is representation of physical storage space provided by the combined raid groups – a collection of blocks. As a volume is created it takes some space to set up the meta data, file system view and provide a access point for the user, but no space is carved off of the aggregate space. As data is written in a volume, space from the aggregate is utilized just like we do for qtrees today. The blocks belonging to different flexible volumes are intertwined within an aggregate.
  • Goal of this slide: Demonstrate the difference and consequential value of the N series Unified Architecture approach. Script: One of the best examples of N series innovation is the Unified Architectural Model which provides the foundation for the dramatic differences in value N series is able to provide. First let’s look at the hardware platform model. No matter which of our competitors you look at, they all use the same approach—specialized, incompatible platforms for different functions. They may have a platform for low end, another for mid-range, yet another for high-end, and still another for compliance. Each of these platforms, while robust in its own area, forms an information silo, and an investment dead-end. By contrast, N series systems base solutions on one, extremely broad, scalable and fully compatible platform, totally eliminating the notion of information silos. And to help you get the most out of your investment dollars, every system can be easily upgraded without migrating the data. [Click mouse to build] This model starts to get even more compelling when you look at the software and processes required. The specialized hardware platforms each run their own, incompatible software, each with its own set of processes and “best practices”. In contrast, the N series family all runs the same set of software, with the same processes. So much so that we hear customers say that they only have to test an application with one N series system—they know what works on one, will work on all. [Click mouse to build] Add to this the people side of the equation, and you see that all those incompatible platforms each need their own experts, and getting them to work together requires even more people and expensive integration services. With n series, your people need less training, spend less time on making things work together, and because they’re familiar with the systems, they make fewer mistakes—the leading cause of downtime. I hope this helps you understand how simple concept like “architectural simplicity” can make a big impact to your bottom line.
  • EMC offers strong solutions in each of our markets including primary, secondary and backup. However, we can actually provide the simplification EMC can only talk about. Data ONTAP offers the user consistent management and functionality across all N series platforms. We mean not only in name by actual syntax and operational functionality from the low-end through the high-end. There is no need to re-train staff as another N series solution is added to the environment. EMC wide breadth of solutions have been acquired through a variety of acquisitions and partnerships resulting in not only different operating systems required for their products but often drastically different functionality implementation. Thus, the addition of another platform or the movement of staff to another EMC system required retraining and reeducation as to the capabilities and limitations of that system. Several of EMC’s products have a very narrow functionality limitations. One example is the CLARiiON CX which requires a separate platform to support FC and another platform to support iSCSI. N series allows customers to intermix. Other limitations include Centera’s scalability only by adding an additional frame, lack of a backup solution, lack of tape connectivity, lack of migration. As you can see with the complexity of EMC’s solutions, the only way to provide the integration for the customer is through the involvement of professional services. This not only increases the initial and on-going costs of the solution but locks the customer into the EMC solution.
  • Then use the picture from the Customer Preso… Integrated NAS Protection Key Messages: The co-developed solution integrates all stages of NAS data protection, while increasing performance and simplifying management. While most of the short-term and long-term integration is available today, they have been enhanced and now integrated across each stage. Organization’s can now manage all operations from a single, intuitive interface (NetBackup). Previously, an administrator would have to log into NAS multiple systems and interface with a number of tools to perform each operation. In addition, there was no understanding or logic of what administrators have protected with online snapshots compared to NDMP tape backups. Short-Term (the solution already offered with NetBackup 5.1) – no need to discuss this in a lot of detail here, as it was covered in the Overview section NetBackup (Advanced Client) integrates with NetApp’s Snapshot technology to schedule, manage, and catalog local disk-based snapshots. Snapshots can managed across multiple NSs and locations. Snapshots are space-optimized by providing only a map of the file system at a point-in-time. However, the space required to store snapshots increases in size when data is changes over time. NetBackup (Advanced Client) integrates with NetApp’s SnapRestore to rapidly restore a single file from the local snapshots or rollback a fie system to a point-in-time. Note: Same concept and benefits as NetBackup Advanced Client Instant Recovery feature. Note: This functionality has already been released with NetBackup 5.1. However, it the integration of all components is where customers will find value. Near-Term NetBackup (Advanced Client) integrates with NetApp’s SnapVault technology to provide disk-to-disk backups of NetApp NSs to a consolidated NetApp NearStore system. Backups can be performed at an incremental changed block-level for high-performance backups and reduced storage requirements. Leveraging SnapVault’s ability to send data great distances, organizations will be able to backup remote office NAS systems to a centralize disk repository. Additional benefits of NetBackup managing NetApp SnapVault: - Oracle application interface. - Consolidation of primary and secondary snapshots reduces storage - Ease of use – replaces cumbersome administrative CLI commands which must be run on both the primary and secondary systems. - Provides a “single pane of glass” for NAS NS administration, backups, and restores. - Improved scheduling of snapshot and snapvault transfers with finer time granularity with predictability. - Provides a user restore browse capability enabling efficient user directed restores. (.vs. ~snapshot copies) - Improved snapshot naming conventions combined with NetBackup cataloguing to identify images. ((( Long-Term NetBackup for NDMP Option will migrate (backup) snapshots from the NetApp NearStore to tape for long-term storage. NetBackup 6.0 will bring SSO (drive sharing) for NDMP NAS systems, WORM tape support, and directory level DAR (direct access recovery). )))
  • Another key element of Snapshots are that they are near instantaneous, as they only require copying a simple data structure, not copying the entire data volume. Taking a Snapshot requires virtually no storage. It is only as data changes in the volume that these changes are written. These changes are written to new disk locations thus the Snapshot doesn’t require extraneously copying data. In comparison, mirroring requires significant costs in terms of the bandwidth infrastructure, the potential for downtime, and the computing resource dedicated to doing the copies, as well as the management overhead of these time intensive tasks . Lastly, in comparison to expensive mirroring solutions, Snapshots are bundled in to Data ONTAP and come standard with every system we ship.
  • Aggregate is representation of physical storage space provided by the combined raid groups – a collection of blocks. As a volume is created it takes some space to set up the meta data, file system view and provide a access point for the user, but no space is carved off of the aggregate space. As data is written in a volume, space from the aggregate is utilized just like we do for qtrees today. The blocks belonging to different flexible volumes are intertwined within an aggregate.
  • Aggregate is representation of physical storage space provided by the combined raid groups – a collection of blocks. As a volume is created it takes some space to set up the meta data, file system view and provide a access point for the user, but no space is carved off of the aggregate space. As data is written in a volume, space from the aggregate is utilized just like we do for qtrees today. The blocks belonging to different flexible volumes are intertwined within an aggregate.
  • Aggregate is representation of physical storage space provided by the combined raid groups – a collection of blocks. As a volume is created it takes some space to set up the meta data, file system view and provide a access point for the user, but no space is carved off of the aggregate space. As data is written in a volume, space from the aggregate is utilized just like we do for qtrees today. The blocks belonging to different flexible volumes are intertwined within an aggregate.
  • Why Use NetApp for Exchange?
  • Key Message: NetApp has software specialized for Exchange environments Talking points: SnapManager - currently (Q3CY’03) supports Exchange 5.5 and Exchange 2000. SnapDrive – runs in both ethernet and fiber channel environments Single mailbox recovery software - works with Exchange 5.5 and Exchange 2000. Data Fabric Manager – Provides a central management consol for NetApp systems NetApp Software for Exchange
  • SnapManager for Exchange
  • SnapManager for Exchange Overview
  • SnapMirror (with SME)
  • Single Mailbox Restore
  • PowerControls Software
  • Notes: 1) Unified positioning of NetApp management tools: -Complete set of management tools -Built from the strong base of our existing products 2) Management tools stack composed of 4 software suites targeted to 3 different administrative needs and roles: -storage administrator: storage and data suite -server administrator: server suite -application administrator: application suite 3) Application Suite -provides application solutions on top of NetApp technology by providing an abstraction layer on top of Server, Data, and Storage Suites -application administrator does not need to worry about layers underneath the Application Suite - improves efficiency of application administrator by taking advantage of NetApp technology 4) SnapManager for SQL Server is part of the Application Suite: - allows database administrators to backup, restore, recover and clone the Oracle database with minimum storage knowledge -uses transparently SnapDrive for Windows which is part of the Server Suite
  • Here’s a chart with some of the features & benefits of SnapManager for Microsoft SQL Server. Backup& restore: First and foremost is the ability to make quick backups. As we had talked before, customers will be able to make backups that don’t impact the end user experience. This is a valuable feature. With organizations supporting users who the business application from across the globe, it is extremely hard to find times that the database servers can take a break. With SnapManager for SQL, this restriction can be removed. In the case of a disaster, like an accidental deletion or application misbehaviors, customers can stop their database system and get back to a good copy within minutes. This reduces downtime. The best news for this whole thing is that the benefits of rapid backup and restore can be achieved for any size of database installation. Hot backups to SnapShot : Wizards : One of the questions customers have when they buy a product is, “how long is it going to take to deploy this thing and how much time does it take to learn the product?” The beauty of NetApp’s SnapManager for Microsoft SQL Server, is that it is extremely simple to deploy and extremely simple to learn. The look and feel of the product is very much like Microsoft’s native backup tools that most Windows Administrators are familiar with. This makes the learning process extremely simple. MSCS Support: SnapManager for Microsoft SQL Server supports NetApp Cluster Failover for high availability of storage and integrates with MSCS for high availability of the Server Environment. This makes the entire Database infrastructure highly available. Cluster failover: Depending on customer’s needs, NetApp provides clustered or non-clustered storage appliance. For customers who run mission critical exchange servers, the clustered storage appliance is the best way to maximize on the high availability of the storage.
  • Here’s a chart with some of the features & benefits of SnapManager for Microsoft SQL Server. Volume Mount Point: Support for Volume Mount Points eliminates the limitations with drive letters. This is primarily a limitation for customers who have hundreds of databases. Also, customers might not want to have multiple databases on one/two LUN. Resource Database: Resource database is a read-only database that contains all the system objects in SQL Server 2005. It doesn’t contain any user data or metadata. Each SQL Server has only one instance of this database and is not shared with any other instance. The location of the resource database is dependent on the location of “master database”. This is only supported with SQL Server 2005.
  • Notes: Leverage larger servers to further consolidate
  • Key Message : SnapMirror can protect Exchange data from disasters or catastrophic natural events by replication to a remote site Talking Points: Economical remote replication : SnapMirror replicates Exchange data to a a target filer at a remote site with low impact on network traffic and economical deployment over WAN. SnapMirror only replicates incremental changes thus reducing the bandwidth requirements. Rapid recovery in the event of a disaster : When disaster strikes the primary location, a standby Exchange server at the remote location can connect to the Exchange data on the SnapMirror target volume to provide users with rapid access to their email data.
  • Here’s are the key reasons SnapManager on SQL is valuable to customers: It delivers high availability by making restores simple, reduces backup windows, increases availability of the database infrastructure and does all of this while delivering an easy to manage solution. NetApp’s Storage appliances, software solutions like SnapManager for Microsoft SQL and the services expertise that we bring in, make the transition to using our solution extremely simple, manageable and useful to the end customer. NetApp has a strategic partnership with Microsoft. Both companies collaborate on many fronts and this should give customers and prospects the necessary confidence in using our solutions together. This unbeatable combination of technology, partnership and services should help deliver the best solution for your customer’s environment.
  • Go to Oracle store to buy it, need to license it.
  • ASM provides its own portable Volume Management and File System services. These are Database orientated which aim to give the performance of raw disk with the ease of management of a file system. However, its not a general purpose file system (i.e. does not replace NFS, EXT3 etc.). Oracle’s “Automatic Storage Management” (ASM) is a powerful and portable storage manager designed to manage Oracle Database 10 g™ database files. ASM simplifies storage management so that DBAs worry less about Oracle Database file layout and management. ASM delivers lower total cost of ownership while increasing storage utilization, all without compromising performance or availability. With ASM, a fraction of the time is needed to manage your database files. ASM key features include: Volume Management Database File System with performance of RAW I/O Supports clustering (RAC) and single instance Automatic data distribution Online add/drop/resize disk with automated data relocation Automatic file management Flexible mirror protection
  • Focus on admin productivity across the IT organization Focus on increasing storage flexibility Result is much faster response time and dramatically improved efficiency
  • NetApp provides the other half of this efficient database management solution with SnapManager for Oracle (SMO). NetApp is the first to deliver a tightly integrated disk-based backup with granular recovery at the file level for Oracle customers using ASM technology. SnapManager for Oracle is a host-based management tool that integrates tightly with your Oracle Database to simplify, automate, and optimize database backup, recovery, and cloning Take snapshots with netapp, register with RMAN. SMO understands how ASM diskgroups translate into NetApp volumes. Can recover specific file or use RMAN SMO value is in recovery and cloning.
  • Backup and recovery to ensure availability and uptime is something that is top of mind for most if not all DBA’s. Ensuring high levels of availability means, taking backups often. This results in degraded performance (in hot backup mode) or system being taken offline (in cold backup mode). In addition to performance, backups also take significant time as they are limited by the speed of tape. The time to backup and recover reduces DBA productivity as well. Time to recover from tape is also prohibitive as it is limited by speed of tape. All this results in DBA’s taking backups less frequently. Highlight DBA spends time on maintaining backup scripts
  • A big DBA challenge is balancing BU/recovery, performance and space management. In some studies, work in these areas adds up to 50% of their time. NetApp Snapshot makes it simple. It alleviates the pain points we highlighted in the earlier slides regarding backup and recovery. NetApp allows the DBA to take backups more often as there is no performance or storage overhead. Given that we can store up to 255 snapshot copies, Snapshot can be taken every hour or less if needed Redo/transaction logs-tells you what changed over time
  • SnapManager for Oracle provides capabilities that enable instantaneous and efficient disk-based backups of Oracle ASM-based databases. In addition to fast backups, SnapManager supports rapid restore and recovery of a failed Oracle Database instance within minutes. It leverages Snapshot ™ technology to provide automated, instantaneous, and space-efficient backups of Oracle Databases. It utilizes SnapRestore® technology to provide automated and rapid restore and recovery of the Oracle Databases. It uses FlexClone ™ technology to provide fast, automated creation of database clones within minutes. SnapManager for Oracle combines these with the NetApp intelligent storage infrastructure to simplify and optimize data management operations. SnapManager for Oracle is also protocol agnostic: it provides the same protection across NFS, iSCSI, and FCP.
  • Why do you need to create copies of your database? There are a number of reasons (highlight list) Challenge is to be able to replicate data quickly and cost-effectively. Of
  • There are several ways to copy production data. Offline – stop your application, make sure it’s in a consistent place, then make copies. This isn’t efficient as it impacts production applications unless you have planned downtime. Challenges and pain points Limited storage resources 100% storage capacity overhead per instance, or custom partial extraction scripts Long lead-time requirements Process heavy (I.e., Many “approvals” required.) Storage resource allocation Manual or scripted operations subject to human error Downtime (offline) or degraded production system performance (online) during copy Restoring the baseline requires repeat of this process … your DBAs and application developers could create (and repetitively re-create) a consistent copy of a database application environment……. nearly instantaneously, using negligible incremental storage, as needed, even for individual developers with little or no support of a storage admin? How would that impact the efficiency of your application development team?
  • For example, supposed a volume is created for a production database. A Snapshot of that database is created for instant backup purposes. Recall that, with exception of a very small amount of metadata, the Snapshot does not occupy any more space. New blocks are allocated only as the active volume changes. A FlexClone can be created from that Snapshot, without creating any new blocks and another server can start a database instance against the cloned data (say for development). Additional space is consumed, only as the FlexClone changes. Hence, a rapid replica of a production volume can be created using a fraction of the storage. The benefits are self explanatory
  • So the new methodology, if you take the combined solution, would look something like this where you've got your production copies of the database and you may have a DR copy as well which is something Topio can provide as well. And then you're going to clone potentially off of a DR copy. This is just an example. You can do it right off the production if you like. But basically you would mirror production for initial copy and then use clones off of that copy in order to enable all the functionality we've been talking about. You can also leverage, of course, all the other things that are on the NetApp storage device. Snapshots are one thing that of course you can leverage, besides all the other things like RAID-DP and all the advantages that are part of the WAFL file system. So snapshots are one of those, and bottom line is you can take multiple mirrors and span those out to the multiple use cases or multiple developers.
  • An example. Typical test and dev environment with 3 copies for test and 3 for dev.
  • NetApp consumes disk only for changed blocks. If you assume a 10% change in the data, it results in 67% reduction in storage required. In addition you also have the flexibility to create and delete clones at will! So first of all we talked about reducing the storage capacity which obviously has a direct impact on the overall cost of the solution if they require less storage. That's done by leveraging NetApp first of all for tiered storage which is a lower cost alternative, it has very good price performance. And on that storage eliminating the need to have a full copy, full physical copy of the data. In terms of simplifying operations there is no impact to production applications while you're maintaining the copies. But you can do that without impacting the production environments. The copies can be distributed to multiple locations. As I talked about, you could have them locally or remote, but you could actually have multiple copies at the same time. You could have one local, one remote, maybe two remote. 08:44 We have the capability to do that simultaneously so that if there's people distributed in different areas or you actually have different needs and you want to split off clones at different points in time based on the requirements, you can do that as well. 08:57 And using the capabilities in the NetApp storage those copies are created in a nearly instantaneous fashion. 09:04 So bottom line is, this allows customers to create and manage more copies of their data in less time and in a more efficient manner, and it really enables them to improve their operations. 09:15 They have an always current set of data using the Topio replication technology, and allows them to create them in an on-demand fashion. 09:25
  • Transcript

    • 1. Storage Systems and Business Continuity Overview
    • 2. August 22, 2013 2 Objectives • To information on SAN storage options • To provide details on business continuity and disaster recovery options
    • 3. August 22, 2013 3 Agenda • Types of Storage • Enabling Greater Resource Utilisation Through Storage System Virtualisation • Business Continuity and Disaster Recovery • Systems Center Operations Manager (SCOM) • Managing Disk Based Backup Through Storage Virtualisation Single Instance Storage (Deduplication) • Enabling greater Data Management Through Storage System SnapShots • Enabling Greater Application Resilience Through SnapShot Technologies • Enabling Greater Data Resilience Through Storage System Mirroring • Easing the Pain of Development Through SnapShot Cloning • Rapid Microsoft Exchange Recovery through Storage Systems Technologies • Rapid Microsoft SQL Recovery through Storage Systems Technologies • Rapid Recovery of Oracle DB Through Storage Systems Technologies • Server Virtualisation and Storage • Storage Management and Business Continuity/Disaster Recovery • Storage Management and WAN
    • 4. August 22, 2013 4 Types of Storage • DAS • NAS • SAN
    • 5. August 22, 2013 5 Direct Attached Storage (DAS) • Directly attached to server • Internal or External • Cannot be shared with other servers
    • 6. August 22, 2013 6 Network Attached Storage (NAS) • Storage devices connected to Ethernet network • Can be shared among servers and users • Usually used in places of dedicated file servers • Not for database use (In the Microsoft World)
    • 7. August 22, 2013 7 Storage Attached Network (SAN) • Hosts attached via Fibre Channel Host Bus Adaptors • Connect to storage system via Fibre Channel Switches • Sees pre assigned storage as dedicated free space • Desktops access storage on local server as normal
    • 8. August 22, 2013 8 Storage Attached Network
    • 9. August 22, 2013 9 What Differentiates NAS and SAN?
    • 10. August 22, 2013 10 What Differentiates NAS and SAN? Storage Protocols
    • 11. August 22, 2013 11 What Differentiates NAS and SAN? Storage Protocols • File Level – NAS − Windows File System Share (With no Windows Servers) − ServerNameShareName
    • 12. August 22, 2013 12 What Differentiates NAS and SAN? Storage Protocols • File Level – NAS − Windows File System Share (With no Windows Servers) − ServerNameShareName • Block Level – SAN − Sees provisioned disk as its own drives and formats accordingly. E.g. NTFS, EXT3 − F:Directory Structure
    • 13. August 22, 2013 13 File Level
    • 14. August 22, 2013 14 File Level • CIFS − Common Internet File System − Predominantly Windows Environments
    • 15. August 22, 2013 15 File Level • CIFS − Common Internet File System − Predominantly Windows Environments • NFS − Network File System − Non Windows Environments • Unix, Linux, NetWare, VMware
    • 16. August 22, 2013 16 Block Level
    • 17. August 22, 2013 17 Block Level • Fibre Channel − Uses Fibre Channel Switches • FC-AL • 1Gb, 2Gb, 4Gb
    • 18. August 22, 2013 18 Block Level • Fibre Channel − Uses Fibre Channel Switches • FC-AL • 1Gb, 2Gb, 4Gb • iSCSI − Uses Ethernet Switches • 1GB • 10Gb
    • 19. August 22, 2013 19 Storage Options – Advantages and Disadvantages
    • 20. August 22, 2013 20 DAS - Pros • Inexpensive − Use of large capacity SCSI and SATA drives − No added expense for controllers
    • 21. August 22, 2013 21 DAS - Pros • Inexpensive − Use of large capacity SCSI and SATA drives − No added expense for controllers • Performance − Dedicated disk array with various cache options
    • 22. August 22, 2013 22 DAS - Pros • Inexpensive − Use of large capacity SCSI and SATA drives − No added expense for controllers • Performance − Dedicated disk array with various cache options • Skill Levels − No new skill levels required to mange storage
    • 23. August 22, 2013 23 DAS - Cons • Captive Storage − Storage can only be used by one server
    • 24. August 22, 2013 24 DAS - Cons • Captive Storage − Storage can only be used by one server • Performance − Disk Arrays may be limited to the number of drives that can be used
    • 25. August 22, 2013 25 DAS - Cons • Captive Storage − Storage can only be used by one server • Performance − Disk Arrays may be limited to the number of drives that can be used − Backups can be slow and inconsistent • Expense − Can be expensive in terms of wasted disk space.
    • 26. August 22, 2013 26 NAS - Pros
    • 27. August 22, 2013 27 NAS - Pros • Can replace file servers and introduce enterprise resilience − Windows, Unix
    • 28. August 22, 2013 28 NAS - Pros • Can replace file servers and introduce enterprise resilience − Windows, Unix • Easily expandable − From 36GB to over 0.5PB
    • 29. August 22, 2013 29 NAS - Pros • Can replace file servers and introduce enterprise resilience − Windows, Unix • Easily expandable − From 36GB to over 0.5PB • Cost Effective − Single Appliance replace multiple servers
    • 30. August 22, 2013 30 NAS - Pros • Can replace file servers and introduce enterprise resilience − Windows, Unix • Easily expandable − From 36GB to over 0.5PB • Cost Effective − Single Appliance replace multiple servers • Ease of backup − Can backup all shares from NAS appliance
    • 31. August 22, 2013 31 NAS - Cons
    • 32. August 22, 2013 32 NAS - Cons • Expense − Can be expensive relative to cost of single server
    • 33. August 22, 2013 33 NAS - Cons • Expense − Can be expensive relative to cost of single server • Performance − Depending on protocol
    • 34. August 22, 2013 34 NAS - Cons • Expense − Can be expensive relative to cost of single server • Performance − Depending on protocol • Database Support − No support for MS SQL or MS Exchange
    • 35. August 22, 2013 35 NAS - Cons • Expense − Can be expensive relative to cost of single server • Performance − Depending on protocol • Database Support − No support for MS SQL or MS Exchange • Skill Levels − May require new skill sets
    • 36. August 22, 2013 36 SAN - Pros
    • 37. August 22, 2013 37 SAN - Pros • High Performance − IO/s − Disk Utilisation
    • 38. August 22, 2013 38 SAN - Pros • High Performance − IO/s − Disk Utilisation • Resilience − SnapShots − Mirroring − Replication
    • 39. August 22, 2013 39 SAN - Pros • High Performance − IO/s − Disk Utilisation • Resilience − SnapShots − Mirroring − Replication • Scalability − Scales to PB
    • 40. August 22, 2013 40 SAN - Cons
    • 41. August 22, 2013 41 SAN - Cons • Costs − Initial Capital Cost − Running Costs − Maintenance
    • 42. August 22, 2013 42 SAN - Cons • Costs − Initial Capital Cost − Running Costs − Maintenance • Skill Sets − New skill sets will be required
    • 43. August 22, 2013 43 SAN - Cons • Costs − Initial Capital Cost − Running Costs − Maintenance • Skill Sets − New skill sets will be required • Compatibility − Most vendors require ‘Fork Lift’ upgrades
    • 44. August 22, 2013 44 SAN - Cons • Costs − Initial Capital Cost − Running Costs − Maintenance • Skill Sets − New skill sets will be required • Compatibility − Most vendors require ‘Fork Lift’ upgrades • Business Risk − Lose the SAN and lose data from many servers − Maximum resilience is a must
    • 45. August 22, 2013 45 Which Storage Solution is Right for Me?
    • 46. August 22, 2013 46 NAS or SAN? • Depends on Application requirements • Depends on User Requirements • Depends on Skill Budget
    • 47. August 22, 2013 47 Why Not Both NAS and SAN • Most organisations will benefit from both NAS and SAN • NAS for file serving and low end applications • SAN for greater application performance, OLTP, Exchange, SQL, Oracle • Can be expensive − Use multiprotocol storage systems
    • 48. August 22, 2013 48 Multiprotocol Storage Windows Server UNIX Server GbE switch Windows Server CIFS NFS iSCSI FC fabric FCP
    • 49. August 22, 2013 49 Multiprotocol Storage Systems • No physical boundaries between NAS and SAN • NAS protocols for file serving • SAN protocols for Application Performance • Bring enterprise functionality to NAS environment − NAS data is no less important than SAN data • Greater return on investment
    • 50. August 22, 2013 50 SAN Basics • SAN infrastructure (also called “fabric”) comprises the hardware, cabling and software components that allows data to move into and within the SAN − Server network cards (fibre channel HBAs or Ethernet NICs) and switches • A disk array is a centralised storage pool for servers • Data from multiple servers is stored in dedicated areas called logical unit number (LUNs) • Data can be protected against data loss in the event of multiple disk failures using RAID
    • 51. August 22, 2013 51 What is RAID
    • 52. August 22, 2013 52 What is RAID • Redundant Array of Inexpensive Disks • Allows for single or multiple drive failure • Can increase read and write performance − Depending on environment • Can have an adverse affect on performance − Depending on environment • Dependant on RAID controller
    • 53. August 22, 2013 53 Multiple RAID Levels
    • 54. August 22, 2013 54 Multiple RAID Levels • RAID 0 − No fault tolerance
    • 55. August 22, 2013 55 Multiple RAID Levels • RAID 1 − Hardware Mirror
    • 56. August 22, 2013 56 Multiple RAID Levels • RAID 4 − Single dedicated parity drive
    • 57. August 22, 2013 57 Multiple RAID Levels • RAID 5 − Distributed parity
    • 58. August 22, 2013 58 Multiple RAID Levels RAID 6 (As it should be) − As RAID 4 but with two parity drives with separate parity calculations. Also known as RAID Diagonal Parity, RAID DP
    • 59. August 22, 2013 59 RAID 6 Overview (RAID DP) • Description − Diagonal-Parity RAID – two parity drives per RAID group • Benefits − 2000~4000X data protection compared to RAID 4 or 5 − Protects against 3 modes of double disk failure • Concurrent failure of any 2 disks (very rare) • 2 simultaneous disk uncorrectable errors (also very rare) • A failed disk and an uncorrectable error (most likely) − Comparable operational cost to RAID 4 • Equivalent performance for nearly all workloads • Equally low parity capacity overhead supported − Less system impact during RAID reconstruction
    • 60. August 22, 2013 60 Why is RAID-DP Needed? • ‘Traditional’ single-parity-drive RAID group no longer provides enough protection − Reasonably-sized RAID groups (e.g. 8 drives) are exposed to data loss during reconstruction • Larger disk drives • Disk drive uncorrectable (hard) error rate • RAID 1 is too costly for widespread use − Mirroring doubles the cost of storage − Not affordable for all data
    • 61. August 22, 2013 61 Six Disk “RAID-6” Array { D D D D P DP
    • 62. August 22, 2013 62 Simple RAID 4 Parity 3 1 2 3 9 { D D D D P DP
    • 63. August 22, 2013 63 Add “Diagonal Parity” 3 1 2 1 1 1 3 1 2 2 1 3 3 1 2 2 9 5 8 7 7 12 12 11 { D D D D P DP
    • 64. August 22, 2013 64 Fail One Drive 3 1 2 1 1 1 3 1 2 2 1 3 3 1 2 2 9 5 8 7 7 12 12 11 { D D D D P DP 7
    • 65. August 22, 2013 65 Fail Second Drive 3 1 2 1 1 1 3 1 2 2 1 3 3 1 2 2 9 5 8 7 7 12 12 11 { D D D D P DP 7
    • 66. August 22, 2013 66 Recalculate from Diagonal Parity 3 1 2 1 1 1 3 1 2 2 1 3 3 1 2 2 9 5 8 7 7 12 12 11 { D D D D P DP 7
    • 67. August 22, 2013 67 Recalculate from Row Parity 3 1 2 1 1 1 3 1 2 2 1 3 3 1 2 2 9 5 8 7 7 12 12 11 { D D D D P DP 7
    • 68. August 22, 2013 68 The rest of the block … diagonals everywhere 3 1 2 1 1 1 3 1 2 2 1 3 3 1 2 2 9 5 8 7 7 12 12 11{ D D D D P DP
    • 69. August 22, 2013 69 Business Continuity and Disaster Recovery
    • 70. August 22, 2013 70 Specific Business Continuity and Disaster Recovery Requirements • RTO– Recovery Time Objective − How quickly should critical services be restored • RPO– Recovery Point Objective − From what point before system loss should data be available − How much data loss can be accommodated 1 2RTO Systems Restored SystemLoss 3 Last System Backup/Copy RPO
    • 71. August 22, 2013 71 Options and Issues • Virtualised infrastructure − Virtualise secondary and/or primary server infrastructure • Data replication software − DoubleTake − WANSync • Hardware replication
    • 72. August 22, 2013 72 Possible Core Architecture 1
    • 73. August 22, 2013 73 Possible Core Architecture 2 1. Core server infrastructure virtualised for resilience and fault tolerance 2. Centralised server management and backup 3. SAN for primary data storage 4. Backup to disk for speed 5. Tape backup to LTO3 autoloader for high capacity 6. Two-way data replication
    • 74. August 22, 2013 74 Data Backup and Recovery 1. Servers backed-up to low cost disk - fast backup and reduced backup window 2. Disk backup copied to tape - tape backup to LTO3 autoloader for high capacity and reduced manual intervention 3. Move tapes offsite
    • 75. August 22, 2013 75 Resilience • Virtual infrastructure in VMware HA (High Availability) Cluster • Fault tolerant primary infrastructure • Failing virtual servers automatically restarted • Dynamic reallocation of resources
    • 76. August 22, 2013 76 Disaster Recovery • Failing servers can be recovered on other site • Virtualised infrastructure will allow critical servers to run without the need for physical servers • Virtualisation makes recovery easier – removes any hardware dependencies
    • 77. August 22, 2013 77 Data Replication Options • Option 1 – Direct server replication − Each server replicates to a backup server in the other site • Option 2 – Consolidated virtual server backup and replication of server images for recovery − Copies of virtual servers replicated to other site for recovery • Option 3 – Data replication − Replication of SAN data to other site • Option 4 – Backup data replication − Replication of backup data to other site • Each option has advantages and disadvantages
    • 78. August 22, 2013 78 Option 1 – Direct Server Replication • Install replication software (DoubleTake, Replistor, WANSync) on each server for replication • Continuous replication of changed data • Need active servers to receive replicated data • Active servers can be virtual to reduce resource requirements • Replication software cost of €3,500 per server • Failing servers can be restored • Minimal data loss
    • 79. August 22, 2013 79 Option 2 – Consolidated Virtual Server Backup • Use VCB feature of VMware to capture images of virtual machines • Replicate image copies • Recovery to last image copy • Low bandwidth requirements
    • 80. August 22, 2013 80 Option 3 – SAN Hardware Replication • SAN replication at hardware level • Very high bandwidth requirements - > 1 Gbps each way • Not all SANs support hardware replication • Very fast recovery • Can be an expensive option
    • 81. August 22, 2013 81 Option 4 – Replication of Backup Data • Scripted replication of disk backup data • Recovery to last backup • Low bandwidth requirements • Low cost option
    • 82. August 22, 2013 82 Business Focus on Disaster Recovery • Every year one out of 500 data centres will experience a severe disaster • 43% of companies experiencing disasters never re-open, and 29% close within two years • 93% of business that lost their data centre for 10 days went bankrupt within one year • 81% of CEOs indicated their company plans would not be able to cope with a catastrophic event
    • 83. August 22, 2013 83 DR Recovery Facility Primary Infrastructure Designed for Resilience and Recoverability Processes And Procedures Operational Disaster Recovery And Business Continuity Plan Components of Effective DR
    • 84. August 22, 2013 84 Components of Effective DR • DR Recovery Facility – this will be the second McNamara site • Primary Infrastructure Designed for Recoverability – this will consist of virtualised infrastructure and backup and recovery tools • Processes And Procedures – this is a set of housekeeping tasks that are followed to ensure recovery is possible • Operational Disaster Recovery And Business Continuity Plan – this is a tested plan to achieve recovery at the DR site
    • 85. August 22, 2013 85 Server Virtualisation and Disaster Recovery • Server Virtualisation assists recovery from disaster • Changing disaster recovery requirements − Higher standards are required − More reliability is expected − Faster pace of business generates more critical change − Intense competitive environment requires high service levels
    • 86. August 22, 2013 86 Challenges of Testing Recovery • Hardware bottlenecks − Need a separate target recovery server for each of the primary servers under test − If doing “bare metal” restore, need to locate target recovery hardware matching exactly the primary server configurations • Lengthy process with manual interventions − Configure hardware and partition drives − Install Windows and adjust Registry entries − Install backup agent − Before recovering automatically with the backup server • Personnel not trained − Complex processes and limited equipment availability make it difficult to train personnel
    • 87. August 22, 2013 87 Successful Disaster Recovery • Ensure successful recovery − Diligent use of a reliable backup tool − Regular testing of recovery procedures • Meet the TTR/RTO (Time To Recover/Recovery Time Objective) objectives − Target recovery hardware available − Alternate site available − Processes documented and automated • Put personnel plan in place − Primary and backup DR coordinators designated and trained − Dry runs are conducted regularly
    • 88. August 22, 2013 88 Why Virtual Infrastructure for DR? • Hardware Independence − Flexibility to restore to any hardware • Hardware Consolidation / Pooling / Oversubscription − Test recovery of all systems to one physical server • Speed up recovery − Use pre-configured templates with pre-installed OS & backup agent • Single-step simplified capture and recovery − Different purposes – same procedures – Staging, Deployment, Disaster Recovery − One step system and application recovery − No additional licensing requirements for bare metal restore tools − More trained personnel available
    • 89. August 22, 2013 89 Disaster Recovery at Lower Cost • Hardware / System/ Application independence − No need to worry about the exact hardware configuration − Flexibility to restore to any hardware − Application independent capture and recovery processes • Less hardware required at “hot” failover site • Support for all capture / replication technologies − Tape / Media − Disk-based Back up − Synchronous or Asynchronous Data Replication
    • 90. August 22, 2013 90 Simplified Processes for Recovery • Restore system and application data in one step • Single-step simplified capture and recovery − One step system and application recovery − No Windows registry issues − Easy-to-automate recovery • No need for 3rd party ‘bare metal’ restore tools − Reduce learning and ramp-up − Reduce software licensing expense • Use the same methodology through application lifecycle − Staging /Deployment/ DR • Test once – recover anything − Application independent recovery means simplified testing
    • 91. August 22, 2013 91 Virtual Hardware for Real Recovery • Follow the usual procedure for data backup • For recovery − Find ONE physical server − Install VMware ESX Server − Copy from a template library a virtual machine with the appropriate Windows OS service packs and the Backup Agent pre-installed − Register and start VM, edit IP addresses − Restore from tape into VM using backup server
    • 92. August 22, 2013 92 Compare Recovery Steps Find hardware Configure hardware / partition drives etc. Install Operating System Adjust Registry entries, permissions, accounts Install backup agent Find hardware Install VMware with Templates “Single-step automatic recovery” from backup server “single-step automatic recovery” from backup server PhysicaltoPhysical DoOnceRepeatforeachbox PhysicaltoVirtual Repeatforeachbox
    • 93. August 22, 2013 93 Customer Options for Recovery • 1 - Physical to Physical • 2 - Physical to Virtual • 3 - Virtual to Virtual
    • 94. August 22, 2013 94 Disaster Recovery with SAN Replication • Speed up recovery in solutions based on storage replication − No need to upgrade secondary site server hardware in lock-step with the primary site − Easy to automate and no need for bare metal recovery tools
    • 95. August 22, 2013 95 SAN Replication Issues • Hardware − Synchronous – data is written simultaneously to both SANs. The write operation is not completed until both individual writes are completed. This will require a communications link between both sites operating at least 1 Gbps. − Asynchronous – data is not written real-time to the backup unit. Data is buffered and written in blocks. This will require a communications link between both sites operating at least 2 Mbps. • Software − CommVault QiNetix ContinuousDataReplicator − DoubleTake − RepliStor − WANSync
    • 96. August 22, 2013 96 Virtualisation Resource Allocation and Configuration Analysis • How much resources to leave free to cater for server failure? VM1 VM2 VM3 VM4 Limit Threshold Reservation Threshold Actual Usage VM5 VM6 VM7 VM8 Server1 Server2 HA Cluster
    • 97. August 22, 2013 97 Virtualisation Resource Allocation and Configuration Analysis • Critical (or all virtual servers) will be restarted on other physical server(s) VM1 VM2 VM3 VM4 VM5 VM6 VM7 VM8 Server1 Server2 VM1 VM2 VM3 VM4 HA Cluster
    • 98. August 22, 2013 98 VMware Platforms and Options • VMware Infrastructure 3 Starter NAS or local storage − No HA, DRS, VCB − Restrictions • 4 processors • 8 GB RAM • VMware Infrastructure 3 Standard − HA, DRS, VCB available as separate options • VMware Infrastructure 3 Enterprise − Includes virtual SMP, VMFS, VMotion, HA, DRS, Consolidated Backup • VirtualCentre
    • 99. August 22, 2013 99 VMware Sample Costs Product Rough Cost Annual Software Subscription and Support Year 1 Total Year 2 VMware Infrastructure 3 Starter for 2 processors €781.25 €697.27 €1,478.52 €697.27 VMware Infrastructure 3 Standard for 2 processors €2,929.69 €615.23 €3,544.92 €615.23 VMware Infrastructure 3 Enterprise for 2 processors €4,492.19 €943.36 €5,435.55 €943.36 VMware VirtualCenter Management Server 2 €3,906.25 €625.00 €4,531.25 €625.00 VMWare Enterprise for two 2-processor servers and VirtualCentre €12,890.63 €2,511.72 €15,402.34 €2,511.72 VMWare Enterprise for four 2-processor servers and VirtualCentre €21,875.00 €4,398.44 €26,273.44 €4,398.44 VMWare Enterprise for four 4-processor servers and VirtualCentre €39,843.75 €8,171.88 €48,015.63 €8,171.88
    • 100. August 22, 2013 100 Sample Configurations • Two ESX Servers, VirtualCentre, Backup to Disk, Tape Backup • Two ESX Servers, VirtualCentre, Backup to Disk, Tape Backup, Virtualised DR Facility with Replication • Very Large Scale Implementation
    • 101. August 22, 2013 101 Two ESX Servers, VirtualCentre, Backup to Disk, Tape Backup 1. Two servers running ESX Server – provides resilience in the event of server failure 2. SAN to store data 3. VirtualCentre to administer and manage virtual infrastructure 4. Backup to disk using low cost disk 5. Tape backup unit
    • 102. August 22, 2013 102 Two ESX Servers, VirtualCentre, Backup to Disk, Tape Backup 1. Primary SAN data copied to inexpensive disk – fast backup 2. Disk backup copied to tape/autoloader
    • 103. August 22, 2013 103 Two ESX Servers, VirtualCentre, Backup to Disk, Tape Backup, Virtualised DR Facility with Replication 1. Two servers running ESX Server – provides resilience in the event of server failure 2. SAN to store data 3. VirtualCentre to administer and manage virtual infrastructure 4. Backup to disk using low cost disk 5. Tape backup unit 6. Link for data replication 7. Backup virtual infrastructure for recovery
    • 104. August 22, 2013 104 Two ESX Servers, VirtualCentre, Backup to Disk, Tape Backup, Virtualised DR Facility with Replication 1. Primary SAN data copied to inexpensive disk – fast backup 2. Disk backup copied to tape/autoloader 3. Disk to disk copy to DR location 4. Move tapes to backup location
    • 105. August 22, 2013 105 Two ESX Servers, VirtualCentre, Backup to Disk, Tape Backup, Virtualised DR Facility with Replication
    • 106. August 22, 2013 106 Very Large Scale Implementation
    • 107. August 22, 2013 107 Very Large Scale Implementation
    • 108. August 22, 2013 108 Cost Benefit Analysis • Tangible savings − Server purchases − Operational costs − Administration costs − Power, HVAC − Deferred cost • Intangible savings − Faster server provisioning − Better utilisation − Reduced floorspace − Improved business continuity and disaster recovery
    • 109. August 22, 2013 109 Server Operation Assumptions Server Environmental Details Server Watts/Hour 600 UPS Watt/Hour 25 Server BTU/Hour 2000 Server Operational Hours 8760 kWh Cost €0.10 Total kWh/Server/Year 7227 Total Electricity Cost (Server, UPS, HVAC) €722.70 Maintenance/Server €350.00 Operation Costs Per Server/Year €1,072.70 Server Tasks - Per Server Hours Before Virtualisation Hours After Virtualisation New Server Deployment 16 2 Build / Installs 40 10 Change / Upgrade 12 3 Configuration Changes 2 0.1 Problem Resolution 2 0.1 Rebuilding Test Servers 2 0.1 Installing Software 2 0.1 Rebooting System 2 0.1 Testing 10 0.5 Recovery 8 1
    • 110. August 22, 2013 110 Sample Project Costs and Savings 1 • 16 servers to be virtualised • Avoid 4 new servers a year Virtualisation Project Initial Year 1 Year 2 Year 3 Total Software €21,900.00 €6,100.00 €6,100.00 €6,100.00 €6,100.00 Hardware €16,000.00 Procurement €800.00 Project Costs €25,000.00 Server Operation €3,489.40 €3,489.40 €3,489.40 Maintenance and Support €12,000.00 €12,000.00 €12,000.00 Server Administration €573.73 €573.73 €573.73 Total €63,700.00 €22,163.13 €22,163.13 €22,163.13 €130,189.38 Saving €120,171.68 Existing Servers Initial Year 1 Year 2 Year 3 Total New Server Purchases €32,000.00 €32,000.00 €32,000.00 Procurement €1,600.00 €1,600.00 €1,600.00 Server Operation €22,798.00 €22,798.00 €22,798.00 Server Administration €27,055.69 €27,055.69 €27,055.69 Total €83,453.69 €83,453.69 €83,453.69 €250,361.06 Return on Investment 39 Months
    • 111. August 22, 2013 111 Sample Project Costs and Savings 2 • 32 servers to be virtualised • Avoid 6 new servers a year Virtualisation Project Initial Year 1 Year 2 Year 3 Total Software €29,900.00 €8,300.00 €8,300.00 €8,300.00 €8,300.00 Hardware €32,000.00 Procurement €1,600.00 Project Costs €50,000.00 Server Operation €6,978.80 €6,978.80 €6,978.80 Maintenance and Support €20,000.00 €20,000.00 €20,000.00 Server Administration €1,147.45 €1,147.45 €1,147.45 Total €113,500.00 €36,426.25 €36,426.25 €36,426.25 €222,778.75 Saving €221,107.16 Existing Servers Initial Year 1 Year 2 Year 3 Total New Server Purchases €48,000.00 €48,000.00 €48,000.00 Procurement €2,400.00 €2,400.00 €2,400.00 Server Operation €43,450.60 €43,450.60 €43,450.60 Server Administration €54,111.37 €54,111.37 €54,111.37 Total €147,961.97 €147,961.97 €147,961.97 €443,885.92 Return on Investment 36 Months
    • 112. August 22, 2013 112 Sample Project Costs and Savings 2 • 64 servers to be virtualised • Avoid 8 new servers a year Virtualisation Project Initial Year 1 Year 2 Year 3 Total Software €45,900.00 €12,700.00 €12,700.00 €12,700.00 €12,700.00 Hardware €64,000.00 Procurement €3,200.00 Project Costs €75,000.00 Server Operation €13,957.60 €13,957.60 €13,957.60 Maintenance and Support €25,000.00 €25,000.00 €25,000.00 Server Administration €2,294.90 €2,294.90 €2,294.90 Total €188,100.00 €53,952.50 €53,952.50 €53,952.50 €349,957.51 Saving €424,141.93 Existing Servers Initial Year 1 Year 2 Year 3 Total New Server Purchases €64,000.00 €64,000.00 €64,000.00 Procurement €3,200.00 €3,200.00 €3,200.00 Server Operation €82,610.40 €82,610.40 €82,610.40 Server Administration €108,222.75 €108,222.75 €108,222.75 Total €258,033.15 €258,033.15 €258,033.15 €774,099.44 Return on Investment 30 Months
    • 113. August 22, 2013 113 SAN Options and Vendors
    • 114. August 22, 2013 114 SAN Vendors • Dell/EMC − AXnnn - iSCSI − NSxxx – IP − CXnnn – Fibre Channel − DMX − Centera • IBM − DS series − N Series – multi-protocol • HP − MSA − EVA − XP
    • 115. August 22, 2013 115 System Center Operations Manager (SCOM)
    • 116. August 22, 2013 116 SCOM Configuration
    • 117. August 22, 2013 117 SCOM Components Component Description SCOM Database A Microsoft SQL Server database that stores configuration information and operations data that is produced by the monitoring process. SCOM Management Server A computer that is responsible for monitoring and managing other computers. The SCOM Management Server consists of the Data Access Server, and the SCOM Server and SCOM Agent components. The SCOM Management Server is an essential part of a management group. Data Access Server (DAS) A COM+ application that manages access to the SCOM database. SCOM Server A component that manages the SCOM Agents that monitor computers in a MOM environment. SCOM Agent A component that monitors and collects data from a managed computer. SCOM Reporting Database A SQL Server database that collects and stores the operations data contained in the SCOM Database. User interfaces The Administrator console and Operator console installed by default when you install SCOM. Management Pack A specific extension that provides for the monitoring of a given service/application
    • 118. August 22, 2013 118 SCOM Deployment Options • Agentless Monitoring − SCOM monitors agentless servers. This is aimed at IT environments where agents could not be installed on a few exception nodes. Agentless monitoring is limited to status monitoring only. • Agent Support − Agents are installed on servers. SCOM lets you manage applications running on servers. • Server Discovery Wizard − Allows for server lists to be imported from Active Directory, from a file, or from a typed list. It also allows the list to be filtered using LDAP queries, as well as name– and domain name–based wildcards.
    • 119. August 22, 2013 119 Architecture
    • 120. August 22, 2013 120 SCOM Rule: Unit Of Instruction/Policy • Event Rules − Collection rules − Filtering rules − Missing event rules − Consolidation rules − Duplicate Alert Suppression • Performance Rules − Measuring − Threshold • Alert Rules Rule Provider  NT event log  Perfmon data  WMI  SNMP  Log files  Syslog Criteria Response  Alert  Script  SNMPtrap  Pager  E-Mail  Task  Managed Code  File Transfer •Where source=DCOMand Event ID=1006 Knowledge • Product Knowledge • Links to Vendor • Company Knowledge • Links to Centralised Company knowledge
    • 121. August 22, 2013 121 SCOM Database • The SCOM database is a single authoritative source of all Configuration in a Management Group − Rules, Overrides − Scripts − Computer attributes − Views − SCOM Server and Agent Configurations − Nested Computer Groups − Extensible schema for classes, attributes and associations
    • 122. August 22, 2013 122 UI Consoles • Operator Console − To create and display view instances, Update Alerts • User Customizable Views • Views can be organized in a folder hierarchy • Context Sensitive tasks − Multipane View • Administrator Console − One MMC Snapin per management group − Rules Node – To author, view, modify, Export/Import rules − Config Node – To configure SCOM • Web Console
    • 123. August 22, 2013 123 SCOM Console Views • State View - Provides you with a real-time, consolidated look at the health of the computers within the managed environment by server role, such as Active Directory domain controllers, highlighting the systems that require attention. • Diagram View- Gives you a variety of topological views where the existence of servers and relationships are defined by management packs. The Diagram View allows you to see the status of the servers, access other views, and launch context-sensitive actions, helping you navigate quickly to the root of the problem. • Alerts View - Provides a list of issues requiring action and the current state and severity of each alert. It indicates whether the alerts have been acknowledged, escalated, or resolved, and whether a Service Level Agreement has been breached. • Performance View - Allows you to select and display one or more performance metrics from multiple systems over a period of time. • Events View- Provides a list of events that have occurred on managed servers, a description of each event, and the source of the problem. • Computers and Groups View - Allows you to see the groups to which a computer belongs, the processing rule groups with which it is associated, as well as the attributes of the computer.
    • 124. August 22, 2013 124 SCOM and SQL
    • 125. August 22, 2013 125 The SCOM Administrator Console
    • 126. August 22, 2013 126 SCOM Management Packs • SCOM management packs provide built-in, product-specific operations knowledge for a wide variety of server applications • Management packs contain rules for monitoring an array of server health indicators and creating alerts when problems are detected or reasonable thresholds are exceeded • Monitoring capability is extended by knowledge base content, prescriptive guidance, and actionable tasks that can be associated directly with the relevant alerts included in the management packs • Administrators can then act to prevent or correct situations, such as degraded performance or service interruption, maintaining service availability with greater ease and reliability
    • 127. August 22, 2013 127 SCOM 2005 Management Packs • Standard Management Packs − Exchange 2000 and 2003 Server − Internet Information Services − SCOM 2005 and SCOM 2000 Transition − Security (MBSA) − SQL Server 2000 − Windows Active Directory − Windows Server Cluster − Windows DNS − Windows Server (2000, 2003, NT4) • Tier 2 Management Packs − Windows Update Services − Virtual Server 2005 − Web Services − Application Center 2000 − Terminal Services − DHCP − Remote File Systems − Print Server
    • 128. August 22, 2013 128 Management Packs • Management Pack imported via SCOM Server • Discovery finds computers in need of a given Management Pack • SCOM deploys appropriate Management Packs − No need to touch managed nodes to install Management Packs • Rules: Implement all SCOM monitoring behavior − Watch for indicators of problems − Verify key elements of functionality • Management Packs provide a definition of server health
    • 129. August 22, 2013 129 Management Pack Features • Alerts: Calls attention to critical events that require administrator intervention − Product Knowledge: Provides guidance for administrators to resolve outstanding alerts • Views: Provide targeted drill down details about server health − Performance plots, collections of specific events/alerts, groups of servers , topology, etc. • State Monitoring: At a glance view of the state of my servers and applications by server role − Detail to component level • Tasks: Enable administrators to investigate and repair issues from the SCOM console − Context sensitive diagnostics and remediation • Reports: Historical data analytics − Assess operations performance and capacity planning
    • 130. August 22, 2013 130 Alert Handing and Viewing • When a new alert is identified it will appear in the Alert Pane with a resolution state of “New” • If you highlight that alert its details will appear in the Alert detail Pane • Clicking on the “Properties” tab in the Alert Detail Pane will give you the description (and other details) of the alert • The alert can be classified as: − False Negative − Hardware Issue − Non Hardware Issue
    • 131. August 22, 2013 131 Alert Handling
    • 132. August 22, 2013 132 SCOM VMware Management Pack Integration
    • 133. August 22, 2013 133 SCOM and nWorks Management Pack • nworks Collector is referred to as VEM (Virtual Enterprise Monitor) • The VEM server can be a virtual server to reduce cost
    • 134. August 22, 2013 134 Enabling Greater Resource Utilisation Through Storage System Virtualisation
    • 135. August 22, 2013 135 What is “Storage Virtualisation”? • Abstracted Physical Storage • Storage Pools Created from Physical Blocks of Storage • Virtual Disks created from Storage Pool • Physical Devices and Capacity Distribution Transparent to Servers and Applications
    • 136. August 22, 2013 136 Why Is Storage Virtualisation so Critical?
    • 137. August 22, 2013 137 Opposing Forces on Volume Size Bigger Gives EfficiencySmaller Gives Control  Different classes of data  Different management requirements  Tools work on volumes (Snapshots, etc)  Disks growing  ATA growing faster  More disks for performance  RAID-DP
    • 138. August 22, 2013 138 The Problem: Volumes Tied to Disks What we’ve got today: • Small volumes are impractical • Large volumes are hard to manage What we’d like: • Manage volumes separately from physical disks • Volumes for data; aggregates for disks
    • 139. August 22, 2013 139 14 x 72 GB disks = 1 TB capacity Virtualisation Improve Utilisation Vol 0 Data Parity Database Data Data Data Data Data Data Data Parity Spare Home Directories Data Data Parity Logical Drive 1 = 2 Disks Logical Drive 2 = 8 Disks Logical Drive 3 = 3 Disks 1 Hot spare 140 GB 370 GB 40 GB 550 GB of wasted space
    • 140. August 22, 2013 140 The Solution: Flexible Volumes (FlexVol) • Aggregate contains the physical storage • FlexVol: no longer tied to physical storage • FlexVol: multiple per aggregate • Storage space can be easily reallocated Storage Pool Disks Disks Disks Flexible Volumes
    • 141. August 22, 2013 141 RG1 RG2 RG3 Storage Blocks Storage PoolStorage Pool RG1 RG2 RG3 Storage Pools and Flexible Volumes How Do They Work? • Create RAID groups • Create Storage Pool • Create and populate each flexible volume • No pre allocation of blocks to a specific volume • Storage System allocates space from pool as data is written Flexible Volume 1 Flexible Volume 2 Flexible Volume 3 vol1vol1 vol2vol2 vol3vol3
    • 142. August 22, 2013 142 14 x 72 GB disks = 1 TB capacity Flexible Volumes Improve Utilisation Logical Drive 1 = 144GB Logical Drive 2 = 576GB Logical Drive 3 = 216GB 1 Hot spare SpareData Data Data Data Data Data Data Data Data Data Data Parity Parity Aggregate Database Home DirsVol0 400 GB used 600 GB of Free Space!
    • 143. August 22, 2013 143 Flexible Volume Data Management Benefits • Distinct containers (volumes) for distinct datasets • Flexible Volumes resize to meet space requirements, simple command to adjust size (grow / shrink) • Soft allocation of volumes and LUNs • Free space flows among all Flexible Volumes in a storage pool; space reallocation without any overhead • Flexible Volumes can be: − SnapManaged independently − Backed up independently − Restored without affecting other Flexible Volumes
    • 144. August 22, 2013 144 Compare Benefits FlexibleVolumes Legacy SAN Space Allocation Flexible and dynamic  Volumes can be grown and shrunk Management Spindle Sharing  Preallocated and static  Space is preallocated during configuration  Space can’t be shrunk  Simple  Complex  Automatic sharing of spindles among all volumes, including newly added disks  New spindles are only used when volumes are expanded  Optimal configuration is a daunting task (sliced, striped, etc.)
    • 145. August 22, 2013 145 Compare Benefits Granularity  Volumes can be grown and shrunk in small increments (1MB) without performance or management impact Disruption Rapid Replication  More granularity comes at the expense of performance or management  Growing and shrinking are nondisruptive and instantaneous operations  Shrinking is not possible; growth involves reshuffling of data  Often involves downtime and data copying  FlexClone™ is immediate  No performance implications  Large space savings for similar volumes  Business continuance volumes involve physical replication of the data  No space savings FlexibleVolumes Legacy SAN
    • 146. August 22, 2013 146 LUNs Application-level soft allocation 10 TB 800 GB Flexible Volumes: Enabling Thin Provisioning Flexible Volumes:  Container level:  flexible provisioning  Better utilisation Physical Storage: 1 TB FlexVols: 2TB Container-level soft allocation 1 TB 300 GB 200 GB 200 GB 50 GB 150 GB 100 GB  Application-level:  Higher granularity  Application over- allocation containment  Separates physical allocation from space visible to users  Increases control of space allocation
    • 147. August 22, 2013 147 Managing Complexity through Storage Virtualisation
    • 148. August 22, 2013 148 Unified Management • Storage management and administration is very vendor specific • Most vendors require different skills for different storage systems • Hardware is not cross compatible
    • 149. August 22, 2013 149 The Unified Storage Architecture Advantage Incompatible silos Compatible family Platforms HP, EMC, DELL, IBM Storage Virtualisation Software & Processes Incompatible software; different processes Unified software; Same processes Experts & Integration ServicesLots of experts and integration services Reduced training & service requirements
    • 150. August 22, 2013 150 DMX SeriesCX3-20 CX3-40AX150/S EMC FCEMC FC CX3-80CX3-10 Virtualisation:Virtualisation: Architectural Simplicity Multiple Concurrent Protocols Integrated Mgmnt, DR, BC, ILM, D2D, … NS40G NSXNS80G Centera Celerra Symmetrix / DMX and CX ONLY Virtual Gateways HP, IBM, HDS, SUN CX300iAX150i iSCSI OnlyiSCSI Only The EMC Effect? - ComplexityThe EMC Effect? - Complexity • 8 Dissimilar Operating Systems • 8 Dissimilar Mgmnt GUI’s • Dissimilar DR, BC, … • ILM required CentraStar - 6 1 - FLARE OE 5 - Enginuity 2 - FLARE EMC IPEMC IP NS80NS40NS350 8 - MS Win 3 - Dart 4 - RHEL 2 - FLARE 8 - MS Win Virtual Storage Environment / EMC – Comparison External server w/MS Win and CLARalert required to support CX dial/email home support (compare to AutoSupport). Virtual Gateway Limited iSCSI Support
    • 151. August 22, 2013 151 Managing Disk Based Backup Through Storage Virtualisation Single Instance Storage (Deduplication)
    • 152. August 22, 2013 152 Snapshot and Snapshot Restore Snapshot and Snapshot Restore Backup Integration Backup and Recovery SoftwareBackup and Recovery Software Disk Based TargetDisk Based Target Secondary Storage Secondary Storage Primary Data 9AM 12PM 3PM Snapshot Snapshot Snapshot Primary Storage Primary Storage Instant Recovery Short-Term Local Snapshot Copies Mid- to Long-Term Disk to Disk Block-Level Backups Client Drag-and-Drop Restores Changed Blocks
    • 153. August 22, 2013 153 Advanced Single Instance Storage User1 presentation.ppt 20 x 4K blocks User2 presentation.ppt Identical file 20 x 4K blocks User 3presentation.ppt Edited, 10 x 4K User4 job-cv.doc Different file 8 new 4K blocks = Identical blocks Data Written to Disk: With ASIS: 38 blocks Without ASIS: 75 blocks
    • 154. August 22, 2013 154 Enabling greater Data Management Through Storage System SnapShots
    • 155. August 22, 2013 155 Snapshots Defined • A Snapshot is a reference to a complete point-in-time image of the volume’s file system, “frozen” as read-only. • Taken automatically on a schedule or manually • Readily accessible via “special” subdirectories • Multiple snapshots concurrently for each file system, with no performance degradation. • Snapshots replace a large portion of the “oops!” reasons that backups are normally relied upon for: − Accidental data deletion − Accidental data corruption • Snapshots use minimal disk space (~1% per Snap)
    • 156. August 22, 2013 156 Snapshot Internals - As They Should Be C’ Snapshot File: FILE.DAT • Client modifies data at end of file • Data actually resided in block C on disk System writes modified data block to new location on disk (C’) A B C Active File System File: FILE.DAT Disk blocks
    • 157. August 22, 2013 157 Snapshot Internals Active file system version of FILE.DAT is now composed of disk blocks A, B & C’. Snapshot file system version of FILE.DAT is still composed of blocks A, B & C C’ Snapshot File: FILE.DAT A B C Active File System File: FILE.DAT Disk blocks
    • 158. August 22, 2013 158 User is offered this most recent previous version (and up to 255 older versions) User may drag any of these read-only files back into active service Snapshot-Based Data Recovery
    • 159. August 22, 2013 159 Snapshots are State-of-the-Art Data Protection Snapsho ts sho uld be ne ar instantane o us! To create a point-in-time Snapshot copy requires copying a simple data structure, not copying the entire data volume Additional storage is expended incrementally only for changed blocks only as data changes, not at Snapshot creation time Avoids the significant costs associated with the I/Obandwidth, downtime, CPU cycles dedicated to copying and managing entire volumes
    • 160. August 22, 2013 160 Not all Snapshots Are Equal • What is the disk storage requirement to maintain online data copies? • Will a planned or unplanned or "dirty" system shutdown lose existing data copies? • What is the overall performance impact with snapshots enabled? • How many data copies can be maintained online? • Is the reserve area fixed? Can this "save area" be re-sized on the fly? • Are data copies automatically deleted once the save area is full? • What is the answer to file system recovery? Do they feature a SnapRestore-like capability? • Are snapshots a chargeable item? How much? What is the pricing model? • Is this snapshot method supported across the vendor's entire product line? Questions to ask regarding storage system data copy techniques:
    • 161. August 22, 2013 161 Enabling Greater Application Resilience Through SnapShot Technologies
    • 162. August 22, 2013 162 SnapshotActive File System SnapRestore Recovery 2 N Active File System snap X restore 1 … 2’ N’1’ … Marked as free blocks after Snapshot Restore
    • 163. August 22, 2013 163 Database Recovery 9am 5pm10:00 11:00 12:00 13:00 14:00 15:00 16:00 Snapshots 1 2 3 4 5 6 7 8 9 15:22 Corruption ! Snapshot restore
    • 164. August 22, 2013 164 Enabling Greater Data Resilience Through Storage System Mirroring
    • 165. August 22, 2013 165 Storage Mirroring • Storage Mirroring − Synchronous − Semi Synchronous − Asynchronous
    • 166. August 22, 2013 166 Storage Mirroring Defined • Replicates a filesystem on one storage system to a read-only copy on another storage system (or within the same storage system) • Based on Snapshot technology, only changed blocks are copied once initial mirror is established • Asynchronous or synchronous operation • Runs over IP or FC • Data is accessible read-only at remote site • Replication is volume based
    • 167. August 22, 2013 167 SnapMirror Function …... SAN or NAS Attached hosts Source of source volume(s) Baseline copy …... Source of changed blocks Periodic updates Step 1: Baseline Step 2: Updates Target LAN/WAN Target LAN/WAN SAN or NAS Attached hosts ORImmediate Write Acknowledgement Immediate Write Acknowledgement
    • 168. August 22, 2013 168 Snap A Storage Mirroring Internals Baseline Transfer Source Volume Target Volume
    • 169. August 22, 2013 169 Snap A Storage Mirroring Internals Baseline Transfer Source Volume Target Volume Completed Target file system is now consistent, and a mirror of the Snapshot A file system Source file system continues to change during transfer Common snapshot
    • 170. August 22, 2013 170 Storage Mirroring Internals Incremental Transfer Source Volume Target Volume Snap B Target volume is now consistent, and a mirror of the Snapshot B file system Completed Snap A
    • 171. August 22, 2013 171 Storage Mirroring Internals Source Volume Target Volume Snap C Incremental Transfer Completed Target volume is now consistent, and a mirror of the Snap C file system
    • 172. August 22, 2013 172 Storage Mirroring Applications • Data replication for local read access at remote sites − Slow access to corporate data is eliminated − Offload tape backup CPU cycles to mirror • Isolate testing from production volume − ERP testing, Offline Reporting • Cascading Mirrors − Replicated mirrors on a larger scale • Disaster recovery − Replication to “hot site” for mirror failover and eventual recovery
    • 173. August 22, 2013 173 Data Replication for Warm Backup/Offload • For Corporations with a warm backup site, or need to offload backups from production servers • For generating queries and reports on near-production data MAN/WAN Backup Site Production Sites Tape Library
    • 174. August 22, 2013 174 & WRITEREAD Isolate Testing from Production • Target can temporarily be made read-write for app testing, etc. − Source continues to run online − Resync forward after re-establishing the mirror relationship SnapMirror Production Backup/Test READ & WRITE X Snap C Incremental Transfer SnapMirror Resync (Resync backward works similarly in opposite direction)
    • 175. August 22, 2013 175 Cascading Mirrors • Allows a target volume to be a source to other targets • Each target operates on an independent schedule • Replicate data up to 30 destinations Source NS Source Volume (read + write) SnapMirror Target NS Target Volume (read only) SnapMirror Target NS Target Volume (read only) SnapMirror Target NS Target Volume (read only)
    • 176. August 22, 2013 176 Cascading Replication - Example • Replicate to multiple locations (30) across the continent − Send data only once across the expensive WAN − Reduces resource utilisation on source NS WAN Office 1 Office 2 Office 5 Office 4 Office 3
    • 177. August 22, 2013 177 Disaster Recovery LAN/ WAN • For any corporation that cannot afford the downtime of a full restore from tape. (days) • Data Centric Environments • Reduces “Mean Time To Recovery” when a disaster occurs. Production Site Disaster Recovery Site (redirect) (resync backwards after source restoration) X
    • 178. August 22, 2013 178 Easing the Pain of Development Through SnapShot Cloning
    • 179. August 22, 2013 179 Cloning SnapShots • Write enables SnapShots • Enables multiple, instant data set clones with no storage overhead • Provides dramatic improvement for application test and development environments • Renders alternative methods archaic
    • 180. August 22, 2013 180 Cloned SnapShot Volumes: Ideal for Managing Production Data Sets • Error containment − Bug fixing • Platform upgrades − ERP − CRM • Multiple simulations against a large data set
    • 181. August 22, 2013 181  Start with a volumeVolume 1 Volume 2 (Clone)  Create a clone (a new volume based on the Snapshot copy) Volume Cloning: How It Works Data Written to Disk: Snapshot Copy Snapshot™ Copy of Volume 1  Create a Snapshot copy Result: Independent volume copies, efficiently stored  Modify the cloned vol Cloned Volume Changed Blocks Volume 1 Changed Blocks  Modify the original vol
    • 182. August 22, 2013 182 Volume Splitting  Split volumes when most data is not shared Volume 1 Snapshot™ Copy of Volume 1  Replicate shared blocks in the background Volume 2 Result: Easily create new permanent volume for forking project data
    • 183. August 22, 2013 183 The Pain of Development Prod Volume (200gb) Pre-Prod Volume (200gb) QA Volume (200gb) Dev Volume (200gb) Test Volume (200gb) Sand Box Volume (200gb) 1.4 TB Storage Solution 200 GB Free Create copies of the volume Requires processor time and Physical storage
    • 184. August 22, 2013 184 Clones Remove the Pain Prod Volume (200gb) Pre-Prod Volume QA VolumeDev Volume Test Volume Sand Box Volume 1.4 TB Storage Solution Create Clones of the Volume – no additional space required Start working on Prod Volume and Cloned Volume Only changed blocks get written to disk! 1 Tb Free
    • 185. August 22, 2013 185 Ideally… Primary Production Array Secondary Array Mirror Create Clones from the Read Only mirrored volume Removes development workload from Production Storage!
    • 186. August 22, 2013 186 Rapid Microsoft Exchange Recovery through Storage Systems Technologies
    • 187. August 22, 2013 187 Why use Storage Systems Series for Exchange Data? Just a few off the top…  Snapshot copies “snapshots”  Data and snapshot management, replication  Flexible and easy, dynamic provisioning  Performance  iSCSI, cost effective and gaining on Fibre Channel  Excellent high-end FCP, clustering and MPIO options  Tight Windows OS (incl. MSCS) and Exchange 5.5., 2000, 2003 and 2007 Server integration (SME, VSS on Windows 2003, etc.)
    • 188. August 22, 2013 188 Required Storage Software for Exchange • SnapShot Management − Rapid online backups and restores—integrates with Exchange backup API; runs ESEFILE verification; automates log replay − Intuitive GUI and wizards for configuration, backup, and restore • Server Based Connection Manager − Dynamic disk and volume expansion − Supports both Ethernet and Fibre Channel environments − Supports MSCS and NS Series CFO for high availability • Single mailbox recovery software − Restores single message, mailbox, or folder from a Snapshot™ backup to a live Exchange server or a .pst file
    • 189. August 22, 2013 189 Effective SnapShot Management with Exchange • Manages the entire snapshot backup process • Backup and restore Exchange storage groups • Backups may be scheduled • Each backup is a “full” Exchange backup and is verified using MS provided software, which is integrated into the storage system
    • 190. August 22, 2013 190 SnapShot Management with Exchange Overview • Interacts with Exchange using Exchange backup APIs • interacts with VSS − SnapShot Management is VSS requestor − Exchange is VSS writer − Storage System is VSS hardware provider • Provides point-in-time and up-to-the-minute recovery using snapshots and Exchange database transaction logs
    • 191. August 22, 2013 191 SnapShot Mirroring • SnapShot Mirroring − Automatic mirroring of Exchange data to remote site − Volume based mirroring − Occurs immediately following a Exchange backup and is initiated by Exchange Server − Can replicate over LAN or WAN − Only changed blocks since previous mirror are replicated − Rate of replication can be throttled to minimize impact on network
    • 192. August 22, 2013 192 Single Mailbox Recovery • Allows restores of individual items form Exchange backups in minutes compared to hours or days • Single mailbox recovery is the most requested feature by Exchange customers
    • 193. August 22, 2013 193 Single Mailbox Restore (Exchange) • PowerControls Software − Quickly access Exchange data already stored in the online snapshot backups − Select any data, down to a single message − Restore the data to one of two locations: • An offline mail file (.PST personal storage file) which can be opened in MS Outlook • Connect to a live Exchange server and copy data directly into the users mailbox, making it instantly available
    • 194. August 22, 2013 194 Exchange Single Mailbox Restore (SMBR)
    • 195. August 22, 2013 195 Current Alternatives: Inadequate • Perform daily brick level backups − Pros • Allows quicker recovery of a single mailbox − Cons • Backs up each mailbox separately; one message sent to a 100 people will be copied 100 times • Very time and disk intensive • Impractical to have frequent backups • Brick level backup software is expensive • Have a dedicated recovery server infrastructure − Pros • Reduces the time to recover a single mailbox by eliminating the need to setup a recovery server each time • Eliminates brick level backups − Cons • Still very time and labor intensive (many hours) • Requires additional hardware investments
    • 196. August 22, 2013 196 SMBR and SnapShot Management • SnapShot backs up Exchange in seconds with snapshots • SMBR restores individual mailboxes from snapshots in minutes Primary Data Center Single Mailbox Recovery Software Time to restore: minutes Restore mail box
    • 197. August 22, 2013 197 SMBR: Features • Reads contents of Exchange Information Store without an Exchange server • Extracts mail items at any granularity from an offline copy of the Exchange Information Store (E5.5, E2K, & E2K3) − Folder − Single mailbox − Single message − Single attachment • Restores single mail items to a production Exchange server, alternate server or to an Outlook PST file. • Advanced search and retrieval − Search subject or message body; keyword, user, or date
    • 198. August 22, 2013 198 SMBR: Benefits • Dramatically reduces the time required for single mailbox and single message recovery − From hours or days to just minutes − Simplifies the most dreaded task by Exchange administrators • Eliminates the need for expensive, cumbersome and disk-intensive daily brick level backups • Eliminates the need for recovery server infrastructure • Allows easy search and discovery of email messages and attachments
    • 199. August 22, 2013 199 Rapid Microsoft SQL Recovery through Storage Systems Technologies
    • 200. August 22, 2013 200 SnapShot Management with SQL Server Application consistent data management
    • 201. August 22, 2013 201 SnapShot Management with SQL Server • Provides integrated data management for SQL Server 2000 and SQL Server 2005 databases − Automated, fast, and space-efficient backups using Snapshots − Automated, fast, and granular restore and recovery using SnapShot restore technologies − Integrated with storage system Mirroring for database replication • Provides tight integration with Microsoft technologies such as MSCS, Volume Mount Points.
    • 202. August 22, 2013 202 SnapShot Management with SQL Server – Required Features Features Benefits Rapid hot backup and restore times • Maximizes SQL database availability and helps meet stringent SLAs • Helps organizations recover from accidental user induced errors or application misbehavior • Minimizes SQL downtime and thus reduces cost • Increases the ability of SQL Servers to handle large number of databases and/or higher workloads. Hot backups to Snapshot copies • No performance degradation during backups Configuration, Backup, and Restore wizards with standard Windows GUIs • Ease of use • Virtually no training costs • Cost savings MSCS Support • High availability and enhanced reliability of SQL Server environment Clustered Failover • Further enhances availability of SQL Server Storage Mirroring Integration • Increases SQL Server’s availability – can replicate the database to a secondary storage system for faster recovery in case of a disaster
    • 203. August 22, 2013 203 SnapShot Management with SQL Server – Required Features Features Benefits Online disk addition (storage expansion) • Increases SQL Server’s availability -- additional storage can be added without bringing the SQL Server down Volume Mount Point Support • Support for Volume Mount Points in order to eliminate the limitation with drive letters Native x64 support • Supports 64bit natively on AMD64/EM64T
    • 204. August 22, 2013 204 SnapShot Management for SQL Server (SMSQL) DBA: • Ability to backup DB faster with fewer resources and without any storage knowledge • Reduces Mean Time to Recovery on failure − Quick Restores − More frequent backups  Less logs to replay  Faster Recovery Storage Admin: • Ability to backup and restore DB without any DB knowledge • Space, time & infrastructure efficient backups, restores and clones • Increased productivity and storage utilization
    • 205. August 22, 2013 205 iSCSI or FCP 1 Primary Data Center Benefits: • Simplified, centralized management • Shared storage for improved utilization • Better system availability SQL Server Consolidate SQL Server storage on storage system1 2 2 Add disks and expand volumes on the fly without downtime 3 3 Cluster for higher availability Technical Details – Consolidated SQL Server Storage
    • 206. August 22, 2013 206 Primary Data Center iSCSI or FCP SQL Server • Eliminate backup windows • Automation reduces manual errors • More frequent backups reduce data loss • No performance degradations Benefits: SnapManager automates data management for SQL Server 1 1 Time to backup: seconds Snapshots 2 2 Snapshots for near- instantaneous backups 3 3 Backup multiple databases simultaneously Technical Details – Simplified Backup » More Frequent Backups
    • 207. August 22, 2013 207 Primary Data Center Time to restore: minutes • Fast and accurate restoration of SQL Server • Reduce downtime from outages • Automation saves administrative time Benefits: Near-instant restore from online snapshot Snapshot 1 1 iSCSI or FCP SQL Server Roll transaction logs 2 2 Automated log replay for current image 3 3 Restore single or multiple databases Standby Server 4 4 Rapid failover to standby server Technical Details – Rapid Restores » Less Downtime
    • 208. August 22, 2013 208 Technical Details – Simple & Robust Disaster Recovery Primary Data Center DR Site • Ensures business continuance • Minimizes length of outages • Cost effective – efficient use of existing IP network Benefits: iSCSI or FCP iSCSI or FCP System Mirroring 1 Storage Mirroring replicates SQL Server data to remote location1 Replicate over existing IP networks 2 2 Failover to DR site After Failure Failover DB Server IP network 3 Rebuild primary site from DR site3
    • 209. August 22, 2013 209 Technical Details – Volume Mount Point (VMP) Support • Drive letter limitations in SMSQL − Only 26 available drive letters in a system. − Minimum for 2 LUNs required for database migration. • Limitation for customers who have hundreds of databases. • The customer might not want to have multiple databases on one/two LUN. • Again one database might span multiple LUNs. − LUN restore is performed on whole disk. • To support individual database restore, each database will require its own LUN and drive letter. − Verification will fail on Local server if free drive letter exhausts.
    • 210. August 22, 2013 210 Technical Details – VMP Storing Database Files • All SQL SnapShot related files can reside on a mounted volume, same as that of a Standard Volume: − SQL user databases − SQL system databases − SQL Server transaction log file − SnapInfo directory • Configuration wizard can be used to migrate database files to a mounted volume, same as that of a Standard Volume. − The rules applicable for migrating databases to Standard Volume will apply for Volume Mount Point also.
    • 211. August 22, 2013 211 Technical Details – VMP Rules For Mount Point Root • Database file cannot reside on a LUN which is the root of a mount point: − After LUN restore, all the mount points residing in the LUN will be overwritten. − For example, db1 resides on G:mnt1 • Take backup of the database db1 with SMSQL • Now create a mount point G:mnt1mnt2 • Create a second database db2 in G:mnt1mnt2 • On restoring the backup set for db1, taken before, G:mnt1mnt2 will go off and hence db2 will become inaccessible
    • 212. August 22, 2013 212 Technical Details – VMP Rules • Mounted volumes should not be treated differently from standard volumes. • Configuration rule for multiple databases on one or two LUNs apply for volume mount point also. • Backup, restore and other SQL SnapShot operations will have no difference between mounted volume and standard volume, just longer path for mounted volume.
    • 213. August 22, 2013 213 Technical Details – Backup of Read-Only Databases • Storage System SQL SnapShots now allows backup of Read-Only database • In previous release, read-only databases were not displayed in the list of databases in Configuration Wizard • Now all read-only databases are listed in Configuration wizard, just as normal databases
    • 214. August 22, 2013 214 Technical Details – Resource Database Management • Each instance of SQL Server has one and only one associated mssqlsystemresource.mdf file − Instances do not share this file • The Resource database depends on the location of the master database − If you move the master database, you should also move the Resource database to the same location
    • 215. August 22, 2013 215 Technical Details – Resource Database Management • SMSQL migrates Resource database along with master database − Resource database will not be listed in the Configuration Wizard − Internally SMSQL migrates it while it migrates master database − It will be migrated to the same location as master database • This is supported only for SQL Server 2005
    • 216. August 22, 2013 216 SnapShot Management with SQL Server – Summary • SnapShot Management with SQL Server: − Helps consolidate SQL Server on highly scalable and reliable storage − Efficient, Predictable, Reliable Backup, Restore and Recovery for SQL Server databases − Allows dynamic provisioning of storage for databases − Allows DBAs to efficiently perform database backup, restore, recovery, clone operations with minimum storage knowledge − Facilitates Disaster Recovery and Archiving
    • 217. August 22, 2013 217 Rapid Recovery of Oracle DB Through Storage Systems Technologies
    • 218. August 22, 2013 218 Oracle Enterprise Manager Grid Control Monitor Trends and Threshold Alerts Monitor Key Statistics Monitor Utilization •Ships with Oracle Enterprise Manager •Developed, maintained and licensed separately by Oracle Manage Storage System from Oracle Enterprise Manager 10g Grid Control
    • 219. August 22, 2013 219 Automatic Storage Management Oracle ASM Disks Logical Vol File System 0010 0010 0010 0010 0010 0010 0010 0010 0010 0010Files Tablespace Tables Disk Group Logical Vol File System File Names Tablespace Tables Before ASM ASM Networked Storage (SAN, NAS, DAS)
    • 220. August 22, 2013 220 Compatible Storage Adds Value to Oracle ASM Oracle ASM Compatible Storage Oracle ASM + Compatible Storage Data Resilience Protect against Single Disk Failure Yes Yes Yes Protect against Double Disk failure No Yes Yes Passive Block corruption detection Yes Yes Yes Active Block corruption detection Yes Yes Yes Lost disk write detection No Yes Yes Performance Stripe data across ASM Disks Yes No Yes Balance I/O across ASM Disks Yes No Yes Stripe data across Physical Disks No Yes Yes Balance I/O across Physical Disks No Yes Yes I/O prioritization No Yes Yes Storage Utilization Free space management across physical disks No Yes Yes Thin provisioning of ASM Disks No Yes Yes Space efficient Cloning No Yes Yes Data Protection Storage Snapshot based Backups No Yes Yes Storage Snapshot based Restores No Yes Yes
    • 221. August 22, 2013 221 Integrated Data Management Approach Go from this… Centralized Management + Administrator productivity + Storage flexibility + Efficiency + Response time …to THIS Server-Based Management Application-Based Management Storage Management Inte g ratio n and Auto m atio n Data Se ts and Po licie s X High cost of management X Long process lead times X Rigid structures X Low productivity
    • 222. August 22, 2013 222 SnapDrive SnapShot Management with Oracle Overview Oracle 10g Oracle 9i Storage Systems FCP, iSCSI and NFS* • Provides easy-to-use GUI • Integrates with the host application • Automates complex manual effort − Backup/Restores − Cloning • Tight integration − RMAN − Automated Storage Manager (ASM) SnapShot Management with Oracle
    • 223. August 22, 2013 223 SnapShot Management with Oracle • Database cloning − Ability to clone consistent copies of online databases − GUI support for cloning − Added support for context sensitive cloning • Increased footprint of platforms and protocols − Support for additional flavors of Unix • SuSE 9, RHEL3/4 U3+, Solaris 9/10 − 32-bit and 64-bit − NFS, iSCSI and FCP for various Unix platforms − HP-UX and AIX (NFS) − (Refer to compatibility matrix for specific details) • Product hardening − Increased product stability and usability − Improved performance by utilizing snapshot vs. safecopy − Increase performance when dealing with high number of archive logs
    • 224. August 22, 2013 224 SnapShot Management with Oracle • Database cloning to remote hosts − Ability to clone consistent copies of to remote hosts − Previously clones were assigned to the host (with SMO) that initiated the cloning request • Increased footprint of platforms and protocols − HP-UX and AIX support across NFS, iSCSI and FC
    • 225. August 22, 2013 225 Database Backup and Recovery Challenges • DBA’s time spent on non- value-add backup/restore tasks • Cold backups lead to lower SLAs • Separate backups on each platform • Time-to-recover from tape becomes prohibitive
    • 226. August 22, 2013 226 Backup and Recovery with Snapshot and SnapShot Restore • Significant time savings • Stay online • Reduce system and storage overhead • Consolidated backups • Backup more often Time in Hours Time to Backup Time to Recover To Tape (60GB/Hr Best Case) From Tape Redo Logs 300GB Database 0 1 2 3 4 5 6 7 8 Snapshot™ Redo Logs SnapRestore®
    • 227. August 22, 2013 227 SnapShot Management with Oracle Automates Backup and Recovery Primary Data Center Benefits: • Extremely fast and efficient • No performance degradation • Accurate data restore and recovery • Reduce downtime from outages • Automation reduces errors and saves time • Backups in seconds • Snapshot copies verified • Near instantaneous restores • Dramatically shortened recovery with automated log replays • Automated recovery tasks SnapShot Restore Time to restore: minutes DB Server Storage System Time to backup: seconds Snapshot
    • 228. August 22, 2013 228 Database Cloning and the Application Development Process • Full or partial database copies required for: − App and DB Development − Maintenance (OS, DB upgrade) − Test and QA − Training and Demos − Reporting and DW ETL • Ability to do this quickly, correctly, and efficiently directly impacts Application Development and Deployment PROD SECONDARY (DR) DEV MAINT TEST/QA RPT/ETL
    • 229. August 22, 2013 229 Traditional Approaches to Cloning • Copy − Offline − Online (using a mirror or standby database, snapshots, and log- based consistent recovery) • Redirected restore − From disk- or tape- − based backups • Challenges − Limited storage resources − Long lead-time requirements Test 1 Test 2 Test N Production Mirrored Copy Dev 1 Dev NDev 2
    • 230. August 22, 2013 230 Database Maintenance with Flexible Volume Clones Benefits • Instantaneous copies • Low resource overhead • Easily make copies of a production database without impacting the database − Use clones to test migrations, apply bug fixes, upgrades, and patches Test 1 Test 2 Test N Production Mirrored Copy Dev 1 Dev NDev 2 Production DB Clones
    • 231. August 22, 2013 231 New Database Development Methodology • Mirror PROD for initial copy (DR) − Mirror from and to storage system • Clone database replicas as needed • Create Snapshot copies of replicas for instant SnapShot Restore of working databases PROD Test/Dev/DR Clones Develop ● Test ● Deploy
    • 232. August 22, 2013 232 Traditional Approach: Application Development and Testing Production database 100GB Mirror copy 100GB Development copies 300GB Testing copies 300GB Total: 800GB • 8x actual storage requirement • Time consuming • Resource overheadTest 1 Test 2 Test 3 Production Mirrored Copy Dev 1 Dev 3Dev 2
    • 233. August 22, 2013 233 SAN Approach: Application Development and Testing Production database 100GB Mirror copy 100GB Development copies 30GB Testing copies 30GB Total: 260GB • Over 67% reduction in storage required • Near instantaneous copies • Negligible overhead • Ability to have many more test and dev copies Test 1 Test 2 Test 3 Production Mirrored Copy Dev 1 Dev 3Dev 2 Assumption: up to 10% change in data in the test and dev environments more clones = higher productivity