Data Deduplication: How to Put Data Storage on a Diet

i Data Deduplication Deep Dive 2 Data deduplication: Reducing storage bloat The data explosion threatens to overwhelm storage systems, particularly the backup tier. Here’s how data deduplication can help DATA STORAGE NEEDS continue to grow unabated, It goes like this: Someone in human resources wants to straining backup and disaster recovery systems while send out the new Internet acceptable use policy PDF requiring more online spindles, using more power, and to 100 users on the network. So he or she creates an generating more heat. No one expects a respite from e-mail, addresses it to a mailing list, attaches the PDF, this explosion in data growth. That leaves IT profession- and presses send. The mail server now has 100 copies als to search for technology solutions that can at least of the same attachment in its storage system. Only one lighten the load. copy of the attachment is really necessary, yet with no One solution particularly well-suited to backup and deduplication system in place, all those copies sit in the disaster recovery is data deduplication, which takes mail store taking up space. advantage of the enormous amount of redundancy in Server virtualization is another area rife with duplicate business data. Eliminating duplicate data can reduce the data. The whole idea of virtualization is to “do more with amount of storage space necessary from a 10-1 ratio to less” and maximize hardware utilization by spinning up a 50-1 ratio and beyond, depending on the technology multiple virtual machines in one physical server. This used and the level of redundancy. With a little help from equates to less hardware expense, lower utility costs, data deduplication, admins can reduce costs, lighten and (hopefully) easier management. backup requirements, and accelerate data restoration Each virtualized server is contained in a file. For in the event of an emergency. instance, VMware uses a single VMDK (virtual machine Deduplication takes several different forms, each disk) file as the virtual hard disk for the virtual machine. with its own approach and optimal role in backup and As you would expect, VMDK files tend to be rather disaster recovery scenarios. Ultimately, few doubt that large – at least 2GB in size, and usually much larger. data deduplication technology will extend beyond the One of the great features of virtual machines is that backup tier and apply its benefits across business storage admins can stop the virtual machine, copy the VMDK systems. But first, let’s take a look at why data deduplica- file, and back it up. Simply restart the machine and tion has become so attractive to so many organizations. you’re back online. Now what happens with all of these backup copies? That’s right – a lot of duplicated files TOO MUCH DATA, TOO LITTLE TIME stored on a file server. Admins keep “golden images” of Duplicated data is strewn all over the enterprise. Files are working virtual servers to spawn new virtual machines saved to a file share in the datacenter, with other cop- – not to mention the backup copies. Virtualization is a ies located on an FTP server facing the Internet, and yet fantastic way to get the most out of CPU and memory, another copy (or two) located in users’ personal folders. but without deduplication, virtual hard disks can actually Sometimes copies are made as a backup version prior increase network storage requirements. to exporting to another system or updating to new software. Are users good about deleting these extra copies? STRAINING BACKUP SYSTEMS Not so much. So, how to back up all this data? Old tape backup sys- A classic example of duplicate data is the e-mail blast. tems are too slow and lack the needed capacity. New INFOWORLD.COM DEEP DIVE SERIES M A R C H 2 010

i Data Deduplication Deep Dive 3 high-end tape systems have the performance and capac- This deduplication process reduces the amount of ity but are quite expensive. And no matter how good storage space needed to represent all of the indexed your tape drive is, Murphy’s Law has a tendency to jump files in the system. For example, a file system that has all over tape when it comes to restoration. 100 copies of the same document from HR in each VTLs (virtual tape libraries) provide a modern alterna- employee’s personal folder can be reduced to a single tive to tape, using hard disks in configurations that mimic copy of the original file plus 99 tiny placeholders that standard tape drives. But at what cost? Additional spindles point back to the original file. It’s easy to see how that equal additional cost and additional power consumption. can vastly reduce storage requirements – not to mention VTLs are fast and provide a reliable backup and restore why it makes much more sense to back up the deduped destination, but if there were less data to back up, you’d file system instead of the original file system. have lower hardware and operating costs to begin with. Another benefit of data deduplication is the ability Data glut compounds the difficulty of disaster recov- to keep more backup sets on near line storage. With the ery, making each stage of near line and offline storage amount of backup disk space reduced, more “point-in- more expensive. Keeping a copy of the backup in near time” backups can be kept ready on disk, making file line storage makes restoration of missing or corrupt files restoration faster and easier. This also allows for a longer easy. But depending on the backup set size and the backup history to be maintained. Instead of having three number of backup sets admins want to keep handy, versions of the file to restore, users can have many more, your near line storage can be quite substantial. The next enabling a very granular approach to file backups and tier, offline storage, is comprised of tapes or other media accommodating loads of backup history. copies that get thrown in a vault or sent to some other Disaster recovery is another process that greatly bene- secure location. Again, if the dataset is large and growing, fits from data deduplication. For years, data compression this offline media set must expand to fit. was the only way to reduce the overall size of the off-site Many disaster recovery plans include sending the data set. Add in deduplication and the backup set can be backup set to another geographical location over a reduced even more. Why transfer the same data set each WAN. Unless your company has deep pockets and can night when only a small portion of it changed that day? afford a very fast WAN link, it would be beneficial to Deduplication in disaster recovery makes perfect sense: keep the size of the backup set to a minimum. That goes Not only is the transfer time reduced, but the WAN is double for restoring data. If the set is really large, trying used more efficiently with less overall traffic. to restore from an off-site backup will add downtime and frustration. HOW DATA DEDUPLICATION REALLY WORKS Data deduplication works by analyzing a chunk of DEFINING DATA DEDUPLICATION data – be it a block of data, a series of bits, or the entire AND ITS BENEFITS file. This chunk is run through an algorithm to create a Simply put, deduplication is the process of detecting and hash, and the unique hash in turn is stored in an index. removing duplicate data from a storage medium or file As each new chunk of data gets “hashed,” it is com- system. Detection of duplicate data may be performed pared to the existing hashes in the index. If the hash is at the file, bit, or block level, depending on the type and already in the index, this means that chunk of data is a aggressiveness of the deduplication process. duplicate and doesn’t need to be stored again. If not, the The first time a deduplication system sees a file or hash is added to the index, and so on. It is easy to see that a chunk of file, that data element is identified. There- the number of hashes potentially stored in an index can after, each subsequent identical item is removed from number in the millions or tens of millions. The greater the the system but marked with a small placeholder. The amount of data analyzed, the larger the hash index will be. placeholder points back to the first instance of the data It is possible, if the hashing algorithm isn’t sophisti- chunk so that the dedupped data can be reassembled cated enough, to have a hash collision. In other words, when needed. different chunks of data come out with the same hash INFOWORLD.COM DEEP DIVE SERIES M A R C H 2 010

i Data Deduplication Deep Dive 4 value. The analysis engine must then use other criteria The problem is that if a file is different by just a single to determine which hash value is correct and how to bit, its hash will be different from other versions. This deal with the other chunk of data. If no method is avail- makes file-level deduplication inefficient and impracti- able to sort out the hash collision, both hashes must be cal. For example, one user creates a document in Word discarded and those chunks of data cannot be indexed 2003. Another user opens it and saves a copy (without for deduplication. any changes) to another location, but in Word 2007 Deduplication doesn’t start working right from the instead. Same file, different format – different hash. start. The system needs to spend some time learning Bit- and block-level deduplication are vastly more effi- what data it will see in order to build the initial indexes. cient. In both cases, the analysis engine breaks files into This means there is a “cold” penalty the first time a piece chunks or segments and creates hashes based on those of data passes through the deduplication analysis engine. smaller pieces. The smaller pieces allow for a higher On subsequent passes, when duplicate data passes probability of a match;,the more matches, the greater the through the engine, the duplicates are found and taken reduction of data on the system. Unlike with file-level out of the system. Only the hash reference is stored or deduplication, if a portion of the file changes, bit- and transmitted (depending on the situation), greatly reduc- block-level deduplication only transfers the changed por- ing the number of bits needed to represent a much tion, not the entire file. larger piece of data. The goal of a bit- or block-level system is to have a To see how this works, let’s return to our example very large index of hashes so that it can identify many of the PDF e-mail attachments, which went out to 100 more duplicate items. But, as with many things, there is employees. Each PDF is a whopping 25MB, which a point of diminishing returns. If the placeholder is nearly means the mail server’s storage has just increased by the same size as the removed data, the effort to create roughly 2.5GB (plus the e-mails themselves). a hash isn’t worthwhile. The mail server does a nightly backup to an NAS appliance. The backup, as we saw above, has just SOURCE, TARGET, OR INLINE increased a minimum of 2.5GB from a single e-mail DEDUPLICATION blast. If the network admin wants to keep a week’s worth Just as there are three levels of deduplication, there are of backups for disaster recovery, the weekly backup is also three ways of performing deduplication. Source now 17.5GB larger due to the bulk e-mail. deduplication analyzes files on the original server and Data deduplication to the rescue! By removing the creates hashes for each file, bit, or block. During a backup duplicate attachments and storing only one, deduplica- operation, a backup program with a built-in deduplication keeps the nightly backup to a paltry 25MB – the tion engine creates hashes for the data and compares size of the original attachment. For the weekly backups, them to the backup destination (which is also running instead of storing 17.5GB, the backup system stores only the dedupe backup software). If the hashes are the same, 175MB – a 90-percent reduction in storage require- only the placeholder reference is copied over. If not, ments. Think of how many more backup sets admins then the data is copied and the hash value is updated could keep online or near line with that kind of storage on the backup server. reduction. It would be easy to keep months of history This method is good for keeping network bandwidth instead of only a few days’ worth. under control. Dedupped data is transferred over the LAN/WAN instead of each and every bit. The backup FILE-, BIT-, AND BLOCK-LEVEL destination’s backup set is smaller because it receives DEDUPLICATION optimized data only. Plus, a software package is the only There are three levels of deduplication: file, bit, and block change to the network. No new appliances are required level. In file-level deduplication, each file is analyzed as to dedupe the data. a whole, and a hash value is created. The deduplication Yet source-level deduplication is a good choice only if software looks at the whole file regardless of size. performance isn’t high on the list of requirements. The INFOWORLD.COM DEEP DIVE SERIES M A R C H 2 010

i Data Deduplication Deep Dive 5 overall backup process tends to be slower if backup soft- agnostic) nor does it need additional storage space on the ware does the deduplication, due of the extra overhead destination server. Stored data is already optimized, reduc- needed to create, analyze, and transfer hashes and their ing hardware costs and power consumption. associated chunks of data. Small data sets don’t pay as Because it does the deduplication analysis on the fly, much of a performance penalty, but large sets can really the inline approach also reduces the consumption of net- bring a system to a crawl. work bandwidth by taking redundant bits off the wire. Target deduplication does not impact source server This is especially important when backing up to a server performance in any way. The deduplication is all per- via the WAN. Fewer bits on the wire equals faster transfer. formed on the destination server once the backup is The only real downside to inline data deduplication is finished. Any off-the-shelf backup software package can the cost of the specialized appliance – and the possibility be used with target deduplication since no hashes are that if the appliance is underpowered or the wrong unit created on the source server. Target dedupe is backup is chosen, the device itself can become a bottleneck. For system agnostic – it really doesn’t care how the data gets the most part, that won’t happen, but keep scalability in to the backup server. Duplicate detection is typically very mind when choosing an inline solution. fast in this situation, making it ideal where performance is a primary concern. DEDUPING BEYOND THE BACKUP TIER There are two major drawbacks to target deduplica- If deduplication works so well with backups, why not tion: storage space and network bandwidth. Because the apply it to the primary storage tier? In fact, real-time dedu- deduplication doesn’t take place until after the backup set plication on live storage systems is gaining more attention. arrives, the backup server must have a large amount of Real-time deduplication software watches the local spare disk space to store the backup set until deduplica- file system for duplicate files or chunks of files. Then the tion is complete. One of deduplication’s selling points is usual process ensues: When redundant data is located, the reduction of spindles, but in target deduplication, a few hashes and placeholders point to indexed chunks of more are needed to hold the pre-dedupped backup set. data, reducing online storage requirements. No addi- Target deduplication also requires all of the backup tional hardware is needed, and to the end-user, all the data be sent over the wire to the backup server. Again, dedupped files look and behave just like the original file. because deduplication doesn’t happen until the backup This also works inside virtualized systems, potentially job arrives at the destination, every bit has to be sent over reducing their overall file size. It remains to be seen, the network, in some cases via a slower WAN link. Not however, if there will be a performance penalty from good for disaster recovery unless done in multiple stages. using this type of system. Inline deduplication splits the two previous methods by Data storage needs will never decrease, despite using a specialized appliance to intercept the data in flight strenuous efforts to modify user behavior. Ballooning before it makes it to the backup server. It creates hashes data stores will continue to strain existing storage and on the fly as data is passed through the appliance and backup systems. But with data deduplication removing only sends through the unique chunks. This approach to redundant data from the network, we can at least delay deduplication doesn’t require any special software on the the day when we all have more petabytes than we can source server (like target dedupe, it is backup-application possibly handle. i INFOWORLD.COM DEEP DIVE SERIES M A R C H 2 010

Data Deduplication: How to Put Data Storage on a Diet

by Albino Goncalves on Jun 06, 2010

Accessibility

Categories

Upload Details

Usage Rights

Statistics

Data Deduplication: How to Put Data Storage on a Diet — Document Transcript