1. Symmetrix Local Replication from A-Z:
All the Choices and Which to Choose
EMC Proven Professional™ Knowledge Sharing
October, 2007
Don Fried-Tanzer
EMC Corporation
Don_Fried-Tanzer@emc.com
page 1 of 10

2. Table of Contents
Introduction..................................................................................................................................... 3
TimeFinder Family Overview and Underlying Technologies........................................................ 3
TimeFinder/Mirror.......................................................................................................................... 6
TimeFinder/Clone........................................................................................................................... 7
TimeFinder/Snap............................................................................................................................. 8
Application Factors in Selecting the Best Local Replication Method............................................ 9
When to Choose TimeFinder/Clone ............................................................................................. 10
When to Choose TimeFinder/Mirror ............................................................................................ 10
Conclusion .................................................................................................................................... 10
Disclaimer: The views, processes or methodologies published in this compilation are those of the author.
They do not necessarily reflect EMC Corporation’s views, processes or methodologies.
page 2 of 10

3. Introduction
EMC offers so many ways to replicate data on a Symmetrix®
Array that few are aware of all the
options and even fewer know how to select the best method for a particular situation. In the
case of a Symmetrix Storage Array, we will discuss the Symmetrix Logical Volume SLV that
appears to the host as a hard disk. EMC also offers additional products for remote replication,
file or directory level replication, and automating and managing replication creation and
restoration, all of which are beyond the scope of this paper.
The key EMC local replication methods discussed in this paper include the TimeFinder®
Family:
TimeFinder/Mirror, TimeFinder/Clone, and TimeFinder/Snap. In particular, a detailed
understanding is needed to understand just how TimeFinder/Mirror and TimeFinder/Clone differ
and when it may be most effective to add-on the TimeFinder/Mirror feature above the base
TimeFinder/Clone product.
There are many functions served by SLV replication including data protection, backup, restore,
secondary applications, migration, and upgrade testing. Both the primary and secondary uses
of the data have an impact on the choice of replication. Many factors affect the choice including
the number of copies, ability to make copies of copies, backup functionality, restore
functionality, how soon the copy is available, and the copy’s impact on both the primary and
secondary use performance. This paper includes the latest features available with Enginuity
level 5772 and Solutions Enabler 6.4.
TimeFinder Family Overview and Underlying Technologies
To begin, we must define the different types of replicas available for a logical volume. Replicas
can either be an actual copy or a virtual copy. An actual copy stores the data on separate
storage and can be accessed without a dependency on the original. A virtual copy may allow
independent access, but it is effectively a pointer to the original.
This distinction becomes clear when looking at a single track (the smallest unit used for replica
operations in the Symmetrix). However, it is less clear when looking at a full SLV made up of
many tracks, some of which may be copies and some of which may be virtual "pointers." For
practical purposes, an entire SLV can only be an actual copy once all tracks are copies. If even
a single track is virtual, consider the whole SLV virtual.
Looking at an entire SLV, there must be some mechanism to mark which tracks of the replica
are full copies of the original. In the Symmetrix this is done in two places, either the mirror
position bitmap or an independent SDDF (Symmetrix Differential Data Facility) bitmap. Both of
these structures are stored in Symmetrix Cache memory. When using TimeFinder/Snap, there
is a need for an actual pointer to the data stored in the VDEV (Virtual Device), which is a slim
device stored on disk.
For example, a TimeFinder/Clone replica, known as the target (or TGT), initially has no data on
it and all tracks point indirectly back to the source (or SRC). As tracks are copied to the TGT,
they reside on the TGT and are independent. Once all tracks are copied, the TGT is in the
copied state and can be used completely independent of the SRC.
page 3 of 10

4. Source TargetSource
Before going into more detail about the different features of local replicas, we must consider
how the application wants to use the replica. In some cases, all we need is an independent
point-in-time that is not concerned about whether there is a fully independent copy of all tracks.
Technically, we only need a full copy when the SRC data may no longer be available. However,
when the performance considerations of doing extra IOs to the SRC are considered, having an
actual copy is often desirable. TGT indirect reads of SRC data compete with the primary
application for the same disk and IO path resources.
Next, we will discuss when data gets copied. Copying can be on-demand if it is not necessary to
create a full independent copy. When a new write is made to the SRC, we must write the
original track data to the TGT in order to preserve a consistent point-in-time view. When a new
write is made to the TGT, we must also copy the data from the SRC, because non-overwritten
portions of tracks must be preserved. This is known as Copy on First Write (COFW). COFW
may have a significant performance impact on the primary application, which must wait until the
track copy is complete before completing a write operation.
A full background copy of all tracks can help reduce or fully avoid the primary application’s write
IO performance penalty. The background copy can occur before accessing the TGT begins,
avoiding the write IO performance penalty entirely. Alternatively, background copy can occur at
the same time as TGT IOs are occurring, and it may be possible to change its frequency and
switch it on and off depending on application needs. Please remember that the background
copy reads from the SRC compete with the primary application for the same disk and IO path
resources.
While on the subject of full background copy, we must also consider the feature of differential or
incremental copies for subsequent point-in-times. If the TGT already has much of the
unchanged SRC data, background copy will require far fewer resources and complete more
quickly by avoiding recopying data that is already present. Similar to the earlier discussion
about some mechanism to mark which tracks of the replica are full copies of the original, there
needs to be a mechanism to mark how the SRC and TGT have changed since the point-in-time
in order to synchronize them. In the Symmetrix, this is done using SDDF bitmaps and mirror
position bitmaps.
One of the major differentiators between local replication options is how data is copied. In all
copy cases, an extra write IO is necessary to the TGT to create a replica. This copying can
occur anywhere in the write path for IOs. Selecting the best method depends on where the
impact of the additional IO writes will have the least impact on the entire application
performance mix.
Target
TGT all Virtual
Source Target
TGT Mixed TGT full Copy
page 4 of 10

5. With TimeFinder/Mirror, this write IO occurs using the same mechanism as synchronizing
RAID-1 mirror protection. Once synchronized, duplication of host writes is achieved by
destaging data (writing to disk) from a single Symmetrix cache slot (one track per slot). Data is
written to all the STD mirrors (the standard is the same as the Clone SRC) and the BCV
(Business Continuance Volume similar to the Clone TGT).
With TimeFinder/Clone, it is either the background copy mechanism or COFW that causes the
write (CopyOnAccess writes from read IOs is discussed later). Since the TGT is not the same
device as the SRC, it cannot share the cache slot. As a result, we must use two cache slots.
TimeFinder/Mirror may also need to duplicate cache slots when a point-in-time is selected (a
split operation); this is accomplished through the background split mechanism.
Only COFW causes a write with TimeFinder/Snap. There is no background copy mechanism
since it is designed to support virtual copies. The VDEV has no space to store data, instead it is
used to indirectly point to tracks either on the SRC or on SAVE devices in a pool associated
with each snap session. As a general guideline, if more than 30% of the SRC data has to be
copied to the TGT, then TimeFinder/Snap is probably not the optimal solution. It has extra
overhead since all pointers are indirect and its value is in the lower demand for disk storage for
each replica.
Source VDEV Save Pool
EMC has additional products that can be used to replicate data. Replication can occur at the
host using EMC Open Migrator, or various LVMs (Logical Volume Managers). If there is spare
capacity in this part of the IO path, the higher performance impact of duplicating the entire IO
stream may not be problematic and the additional features of making the host aware of the copy
beneficial. The replication can also occur in the SAN (Storage Area Network) using EMC
Invista. This and other virtualization techniques are beyond the scope of this paper.
Given this background, how do you choose which local replication method to use? There are
two major criteria: features and performance. The requirement of certain features will limit the
replication options. It is more difficult to provide a formula for performance considerations since
performance is always relative. Good performance is satisfactory and not a particular number.
As needs change, what is considered satisfactory changes. The next sections will highlight the
feature and performance differences of the TimeFinder family of local replication products.
page 5 of 10

6. TimeFinder/Mirror
TimeFinder/Mirror was EMC's first local replication method. It grew out of customers' using
SRDF (Symmetrix Remote Data Facility) in an unplanned way to create replicas for secondary
applications. It is unique in the industry and capitalizes on EMC patents using RAID-1 mirroring
technology to achieve the replicas. This has both unique performance advantages and a few
features not currently available with other alternatives. The "dual-life" of the BCV is one of the
unique distinctions of TimeFinder/Mirror. BCV is actually a special attribute of a SLV that allows
it to be either a simple independently accessed volume or to dynamically become the mirror of
another volume (establish operation).
DEV 00
M1
Standard Device 00
DEV 00
M2
BCV Device 01
DEV 01
BCV
Before Establish
BCV Device 01
After Establish
There are a number of significant implications with this implementation. The BCV is not
independently accessible when established with the STD. A host can independently access it
only after synchronization is complete and the BCV is split from the STD. TimeFinder/Mirror
requires sufficient time for synchronization to complete before the point-in-time (split) can be
created. The synchronization method used is RAID-1 mirroring sharing the same cache slots
for all mirrors, which may have performance benefits. This BCV synchronization cannot be
turned off, but it can be significantly slowed using quality of service settings.
Using RAID-1 mirroring techniques means BCVs use the bitmaps in mirror positions to mark
track synchronization. This is a major limitation of how much TimeFinder/Mirror can be applied.
In the Symmetrix Enginuity operating environment, only four mirror positions are supported per
SLV and only two can be dynamic (TimeFinder/Mirror and/or Dynamic SRDF). With local RAID
protection and SRDF protection using these positions there may be only one or no mirror
positions available for BCVs. Mirror position limitations mean the maximum number of
concurrent BCVs established at the same time is only two.
NR to host
DEV 00
M1
Standard Device 00
DEV 00
M2
DEV 01
BCV
DEV 00
M3 (BCV)
page 6 of 10

7. RAID-1 mirroring is a technique that works with back-end Symmetrix hypers. This technique is
not available for RAID-5 and RAID-6 volumes which are made up by multiple back-end hypers
which are presented on the front-end to the host as a single volume. Therefore there is no
support for RAID-5 or RAID-6 BCVs using TimeFinder/Mirror directly. TimeFinder Emulation is
a mechanism that implicitly for RAID-5 or RAID-6 BCVs emulates TimeFinder/Mirror actions
using TimeFinder/Clone. Emulation can also be selected explicitly for other TimeFinder/Mirror
operations instead of rewriting scripts to use TimeFinder/Clone.
Although mirror positions may limit how often TimeFinder/Mirror can be applied, using mirror
position bitmaps is an advantage enabling TimeFinder/Mirror to support up to 16 incremental
BCVs (a feature called multi-BCV). The marking of changes to each STD and BCV is done
using a combination of mirror position bit map and a single SDDF session. Each SLV is
currently limited to 16 SDDF sessions. Since TimeFinder/Mirror uses mirror position bit maps
for all other features, then up to 16 SDDF sessions can be used to support the multi-BCV
feature.
TimeFinder/Mirror has three additional unique features. The first is that when established, the
BCV is a full mirror of the STD, including the ability to satisfy a read IO for the STD from the
BCV if the data is not available on the STD. In fact the optional STD Protect feature of
TimeFinder/Mirror will block splitting away the BCV from the STD, if the STD has no other valid
local mirror (including RAID-1, RAID-5, RAID-6 or RAID-S protection).
The second is a special case for an unprotected R2 with an established BCV, where the BCV
will participate in the Dynamic Mirror Service Policy (DMSP) where read IOs will be optimized as
coming from either the STD R2 or the BCV.
The third is that it is valid for a split BCV to be the SRC of a TimeFinder/Snap or Clone
operation. This is significant, because the differential nature of the original STD BCV
relationship is still present, and the BCV can be incrementally established or restored with the
STD.
TimeFinder/Clone
TimeFinder/Clone is the base product in the TimeFinder family. Over time, TimeFinder/Clone
has been gaining features to match TimeFinder/Mirror and in a number of cases with fewer
limitations. TimeFinder/Clone does not require SLVs with the special BCV attribute and can
create copy sessions between two STDs, two BCVs or a STD and BCV in either direction.
Since the TimeFinder/Clone is always an independent device, and tracks can be virtual copies
of the SRC, the TGT volume can be available for immediate IO access without waiting for any
background copy activity.
Remember that IO to TGT tracks indirectly pointing to SRC tracks generate IO resource
competition with the SRC device. This resource contention can be alleviated by the use of
pre-copy, quality of service settings for background copy, and the ability to turn background
copy on and off.
page 7 of 10

8. Additionally with Enginuity 5772 and Solutions Enabler 6.4, the default no copy mode for
TimeFinder/Clone has been changed to CopyOnWrite, where only writes to the SRC cause
copies to the TGT. When the default option is changed (or in previous releases), the no copy
mode is CopyOnAccess where data is also copied on the first read from the TGT of an indirectly
pointed to SRC track. The potential advantage of CopyOnAccess is subsequent IOs to the TGT
do not have to access the SRC. Depending on the TGT IO locality of reference and the benefit
of reducing IO path resource contention with the SRC, this advantage may not be enough to
offset the performance cost of destaging all first reads to the TGT. In most cases CopyOnWrite
will yield better overall performance, but CopyOnAccess can be optionally selected.
Pre-copy is a key feature in how TimeFinder Emulation mode can use TimeFinder/Clone to fulfill
TimeFinder/Mirror operations. The ways in which emulation support differ from native
TimeFinder/Mirror mode help clarify the differences between TimeFinder/Mirror and
TimeFinder/Clone.
The TimeFinder/Mirror Protected Restore feature can be selected to preserve the BCV point-in-
time from being updated by changes to the STD during a Restore. Since TimeFinder/Clone
uses an independent device, the TGT is always protected, so the option is in effect mandatory.
The Protected BCV Establish feature concurrently establishes both the BCV and its local mirror
with the STD, so when split, both have already been synchronized. Again, the independent
TGT and all (if any) of its mirrors must always be updated, so this feature is also mandatory.
Related to the mandatory nature of Protected BCV Establish, the Reverse Split feature is not
available. The Reverse Split feature is only available for a BCV with a local or remote mirror;
when the BCV is split, instead of its STD updated data being propagated to its mirror(s), it is
restored from the local or remote mirror. Used after a Restore operation, the Reverse Split
achieves the same purpose as the Protected Restore feature.
TimeFinder/Clone uses SDDF sessions to mark the point-in-time and is not limited by mirror
positions, so can support up to 16 concurrent copy sessions. However, since TimeFinder/Clone
also uses SDDF sessions for differential support, a maximum of 8 concurrent differential
sessions are possible (two sessions used per differential clone copy session).
TimeFinder/Snap
As mentioned earlier, TimeFinder/Snap is not designed to support a non-virtual copy and thus
has no built-in background copy mechanism. The value of using TimeFinder/Snap is its space-
saving pointer-based copies. To achieve the benefit of saving space, users must configure the
snap save device pool(s) to be smaller than the amount of data being replicated. As a result, it
is possible for a snap session to fail if it runs out of space. How to configure snap pools
correctly is beyond the scope of this article, but a general rule is that the amount of SRC data
that must be written to the snap pool should be less than 30%.
If more than 30% of the data must actually be copied to the TGT, use either TimeFinder/Clone
or TimeFinder/Mirror. Typical use cases for TimeFinder/Snap are for fast, easily accessible,
short-term copies, often providing extremely small Recovery Point Objectives without the cost of
actual copies.
page 8 of 10

9. TimeFinder/Snap VDEVs are independent devices, with all tracks initially virtual copies of the
SRC (pointers), so the VDEV can be available for immediate IO access. There is no
background copy capability for snap sessions. As a result, there is no tuning available for the
problem of IO resource competition with the SRC device. However, the typical usage of
TimeFinder/Snap copies likely makes this less of an issue.
Application Factors in Selecting the Best Local Replication
Method
It is impossible to list all the different application scenarios and which replication method(s)
would be best for each scenario. However, we can list the key questions that need to be asked
about the application mix before selecting the best local replication method(s).
Does the mix of local replication, SRDF and dynamic SRDF leave any mirror positions available
for TimeFinder/Mirror? Are the SRC volumes RAID-5 or RAID-6 protected? Would there be a
significant advantage with improved availability with the BCV as a true mirror?
How many copies of the data are needed; are they needed concurrently or sequentially? Will
the copy be re-used for another point-in-time, where differential capability would improve
performance? Is there a plan to make copies of copies, and re-use the initial copy for another
point-in-time?
Does the secondary application need immediate access to the point-in-time? Can background
copy synchronization be conveniently scheduled to complete before the point-in-time is
accessed? Are there spare IO resources to allow for contention with the secondary application
indirectly accessing the SRC SLVs? Are there spare IO resources in the IO path at the point
where the chosen replication method will use more resources?
There is no easy formula to evaluate performance considerations. Although replication methods
may differ in performance, if two alternative methods both yield satisfactory performance, the
difference may be unimportant. Even if there is a significant difference, application mix needs at
high priority times may make some performance differences more important than others.
While a chosen replication method may be working fine, gradual growth or some other change
may cause cross a threshold where the old method no longer performs adequately and a
change will be needed. Answering questions like whether TimeFinder/Mirror replication
processing would perform better than TimeFinder/Clone background copy processing probably
would require assistance from EMC.
page 9 of 10

10. When to Choose TimeFinder/Clone
Key features and requirements that require TimeFinder/Clone are:
• Immediate access to copy before any background synchronization completes
• Lack of mirror positions to support TimeFinder/Mirror
• Using between three and eight concurrent copies
(TimeFinder/Mirror is limited by the two dynamic mirror positions limit)
• Using more than one protected BCV establish.
• RAID-5 and RAID-6 BCVs
All of these features are also supported by TimeFinder/Snap, which may be a viable alternative
if the 30% or less criteria applies. Immediate access to the copy is a key distinguishing feature
from TimeFinder/Mirror. The middle three features all relate to the mirror position limitations of
TimeFinder/Mirror. The last feature relates to multiple back-end hyper nature of RAID-5 and
RAID-6 devices which cannot be supported by a RAID-1 mirroring mechanism.
When to Choose TimeFinder/Mirror
Key features and requirements that would require TimeFinder/Mirror are:
• Higher Availability supported by BCV as a true mirror of the STD
• Higher performance provided by the use of DMSP for a BCV of an R2
• High performance BCV mirroring
• Allowable combinations of copies of BCVs
• Using over eight differential copies
The first three features are all a result of TimeFinder/Mirror true mirroring implementation.
Current limitations on copies of copies also offer a significant performance advantage by
avoiding non-differential copies currently necessary if using TimeFinder/Clone. Although
TimeFinder/Clone is limited to 8 differential copies, TimeFinder/Snap also currently supports 16
concurrent copies.
Conclusion
This article provides enough technical background to help you fully interpret the feature
differentiators among the TimeFinder family of local replication solutions. Enginuity
enhancements to TimeFinder/Clone in the past have narrowed the advantages of
TimeFinder/Mirror, and ongoing Enginuity enhancements will continue to add functionality that
may change some of the decision points. Most application scenarios can be satisfied by more
than one local replication solution, though in many cases key features or performance
considerations will direct you to a best local replication solution.
page 10 of 10