2. 2
Agenda
• Introductions
• Key things to know about disaster recovery requirements and
expectations in today’s world
• How a large manufacturer moved to meet disaster recovery
challenges
• Supporting multiple customers through data loss and
recovery
• Q&A
We are live tweeting and answering questions using hash tag: #DellDP. Join us at tweetchat.com/room/DellDP
3. 3
Introducing our speakers
Moderator: George Crump
Storage Switzerland.www.storage-switzerland.com
Rob MacCara,
System Administrator
Maritime Paper Products, Ltd.
Robby Wright
Chief Technical Consultant
Abtech Technologies
We are live tweeting and answering questions using hash tag: #DellDP. Join us at tweetchat.com/room/DellDP
4. 4
Key things to
know about
disaster recovery
requirements
and expectations
in today’s world
We are live tweeting and answering questions using hash tag: #DellDP. Join us at tweetchat.com/room/DellDP
5. 5
About George Crump & Storage Switzerland
• Analyst firm covering storage, cloud and virtualization markets
• Knowledge of these markets is gained through product testing, real-world
implementations and interactions with users and suppliers
• The results of this research are found in the articles, briefing reports, case
studies and lab reports on www.storage-switzerland.com
George Crump
Chief Steward, Storage Switzerland
www.storage-switzerland.com
gcrump@storage-switzerland.com
twitter.com/storageswiss
youtube.com/user/storageswiss
We are live tweeting and answering questions using hash tag: #DellDP. Join us at tweetchat.com/room/DellDP
6. Meeting the Recovery Expectation
We are live tweeting and answering questions using hash tag: #DellDP. Join us at tweetchat.com/room/DellDP
7. • Users expect to up 100% of the time just like
"FaceBook" or they expect outages to be
minimal - Minutes of downtime, not hours
• Meeting this expectation means that data can
no longer be "restored" - The network transfer
is too time consuming
• Data have to be recoverable "in-place" and it
has to be readable the first time - Verification
We are live tweeting and answering questions using hash tag: #DellDP. Join us at tweetchat.com/room/DellDP
Meeting the Recovery Expectation
8. Meeting the Zero Data Loss Expectation
We are live tweeting and answering questions using hash tag: #DellDP. Join us at tweetchat.com/room/DellDP
9. • Users expect you to have every copy and
version of their data protected all the time.
Just like "DropBox"
• Traditional once-a-night backup is no longer
enough. Too much data is created, modified
and potentially deleted in a day
• Backup has to occur at multiple points
throughout the day, potentially hourly, without
impacting performance
We are live tweeting and answering questions using hash tag: #DellDP. Join us at tweetchat.com/room/DellDP
Meeting the Zero Data Loss Expectation
10. Meeting the Keep It Forever Expectation
We are live tweeting and answering questions using hash tag: #DellDP. Join us at tweetchat.com/room/DellDP
11. • Tape makes this difficult because you have
to manage time expectations. Disk makes
this difficult because you have to manage
expense expectations. - Try Tape, Require
scalable deduplicated disk
• Users expect you to keep all their data
forever, near-instantly available...for free
• Reality is that 99% of the data will never be
needed again. The problem is you don't know
where that 1% is going to come from.
We are live tweeting and answering questions using hash tag: #DellDP. Join us at tweetchat.com/room/DellDP
Meeting the Keep It Forever Expectation
12. • Backup may be “all about recovery”
but backups matter
o Granular Backups
o Frequent Backups
o Validated Backups
• Recovery Needs To Change
o In-Place
o Virtual
We are live tweeting and answering questions using hash tag: #DellDP. Join us at tweetchat.com/room/DellDP
Meeting Cloud Expectations
13. 13
How a large
manufacturer
moved to meet
disaster
recovery
challenges
We are live tweeting and answering questions using hash tag: #DellDP. Join us at tweetchat.com/room/DellDP
14. 14
Rob MacCara
• System Administrator, Maritime Paper
Products, Ltd., Nova Scotia
• 30 years of IT experience
• Built his own company building, selling, and
maintaining Digital Video Surveillance
systems for both small and large retail
customers
• 20 years in the Royal Canadian Navy as a
communications technician specializing in
computer systems
15. 15
Maritime Paper Products: manufacturing market leader
• Major corrugated box manufacturer with multi-
continental reach
• Factories across the Canadian Atlantic provinces
• Forward-thinking with both automation and
sustainable manufacturing
• Massive daily output
16. 16
Maritime Paper’s business-critical applications
• Peak summer periods have us running 24
hours a day, with three shifts
• We depend on:
o Domain controller
o Microsoft SQL Server apps
o Microsoft Exchange Server
• A server failure can stop everything
and close plants, resulting in
potentially very large revenue losses
Rob standing in front of one of Maritime Paper’s
high-speed box fabricating machines
17. 17
Challenge of a failing legacy backup scheme
• Time-consuming rotation of tapes, full backups
to removable disks
• Older server population prone to major failures
• Day-long file recoveries
• File backup only
• Virtualization effort
In Maritime’s high-volume production
environment, the pressure is on
to deliver to customers on time so
their customers are also on time.
18. 18
Time for a change
Virtualization was the last straw
• Existing backup software wasn’t up to
the task
• Choice seemed to be either an
expensive upgrade or having to settle
for greatly reduced backup capability
Maritime reached a point where
its existing legacy system would
require a costly upgrade.
19. 19
Looking for solutions
Key requirements
We looked for an advanced backup
and disaster recovery solution that:
• Worked across virtual and physical servers
• Reduced recovery times
• Restored to any type of machine
• Included replication
20. 20
Why we decided on Dell AppAssure
A ”night and day difference”
• Strong ROI and a wealth of advanced backup
and recovery features
• No more tape
• Recoveries in minutes
• Simplified recovery offsite
21. 21
Results
• 95% savings in storage space
• Minutes to recover lost file versus
24 hours with our previous data
protection product
• $28,000 savings in software and
hardware
22. 22
Fast forward to today
A strong DR, migration and testing solution
• Pair of virtual AppAssure Core machines for
local and offsite recoveries
• Push-button failover to virtual
• Simplified creation of DR site
• Fast restores at any level
• Rapid P2V and V2V migrations
• Faster DR testing
23. 23
Recovery proof point
Recent primary domain controller’s RAID backplane failed
• Resulted in a major crash
• Up and running in less than an hour as a virtual
machine, including troubleshooting the server
• Avoided:
o All–nighter
o Panic
o Revenue loss
• Costs and staff time would have been much higher without
Dell AppAssure
26. 26
About Robby Wright & Abtech
Robby Wright
Chief Technical Consultant
Abtech Technologies
www.abtechtechnologies.com
We are live tweeting and answering questions using hash tag: #DellDP. Join us at tweetchat.com/room/DellDP
27. 27
Two Customers – Similar Solutions
• Customer #1 – A power company in Texas that needs to protect against
storm damage at their primary site
– In a hurricane area
– Flat ground and near a river - increases flooding possibilities
– Can’t afford to lose customer information and billing capabilities
– Must be able to pay vendors after disaster damage
• Customer #2 – A global coverage web conference hosting company
– Hosting equipment sites on multiple continents
– Hosts very large conferences – 40,000+ attendees
– Sites back each other up
We are live tweeting and answering questions using hash tag: #DellDP. Join us at tweetchat.com/room/DellDP
28. 28
The Power Company
• Has two sites connected by dark fiber optic cables
– Set up with 10GBit IP connection between sites
• Using NetVault Backup and SmartDisk for normal backup processes
• NetVault SmartDisk’s new replication feature allows painless replication of
deduplicated backup data between sites
• Mix of approximately 60 physical servers plus virtual machines
• Many databases, email, portals
• Needs long backup chains for regulatory compliance – Tape allows this
We are live tweeting and answering questions using hash tag: #DellDP. Join us at tweetchat.com/room/DellDP
29. 29
How do we plan the backups?
• Determine what needs to be backed up
• Determine priorities for both backup and recovery
• Set RPO and RTO for each
• Determine backup window availability
• Do we need special handling for databases or other applications?
• Make sure we have what we need to recover
• Design backup system to meet requirements
We are live tweeting and answering questions using hash tag: #DellDP. Join us at tweetchat.com/room/DellDP
30. 30
All Jobs Start With A Server Survey
Customer Name:
Priority
Server or VM
Name
Operating System
Application on
Server
RTO RPO IP Address
Disk Allocated
(GB)
Disk Used
(GB)
Dependencies
We are live tweeting and answering questions using hash tag: #DellDP. Join us at tweetchat.com/room/DellDP
31. 31
Determine the RPO/RTO For Each Server
RPO Recovery Point Objective How far back can data be lost?
RTO Recovery Time Objective How long after failure until system up and
usable?
Failure Point RTO
RPO
Lost Data Time Down
Time
We are live tweeting and answering questions using hash tag: #DellDP. Join us at tweetchat.com/room/DellDP
32. 32
The Power Company’s Recovery Objectives
• RPO - They can stand up to one day’s loss of data
– Data can be re-input if necessary for time lost
– Backup to both disk and tape provides multiple fallbacks if necessary
• RTO – They need to be back up and running in 8 hours max.
– Bare metal backup and recovery restores O/S and applications
– SmartDisk and/or tape provides recovery of data
We are live tweeting and answering questions using hash tag: #DellDP. Join us at tweetchat.com/room/DellDP
33. 33
What is special handling?
• Exchange, SharePoint and databases either have to be stopped or have some
method for a point-in-time backup to be able to recover it.
• Active file systems require snapshot capabilities for accurate backup
• Virtual machines can be backed up as either a client or as a VM
• Use of plug-ins requires backup as a client, not a VM.
We are live tweeting and answering questions using hash tag: #DellDP. Join us at tweetchat.com/room/DellDP
34. 34
GB/Hr, Real Life
You Must Be Able To Get the Data From The
Disk(s) To the Tape Drive Fast Enough !!!
File size x disk I/O rate = MB/sec.
My tape drive will record 100MBytes/Sec, so why
is my backup slow?
We are live tweeting and answering questions using hash tag: #DellDP. Join us at tweetchat.com/room/DellDP
35. 35
How do we speed up backups?
• Backing up to disk first allows multiple servers to back up at their own speed
• Backups can then be streamed to tape at full speed of the drive.
• Maximizes utility of and saves wear and tear on the tape drives
We are live tweeting and answering questions using hash tag: #DellDP. Join us at tweetchat.com/room/DellDP
36. 36
Disk-To-Disk-To-Tape
SD Server
SD Disk
SD
Agent
NetVault Server Client
NVBK
Server
NV
Client
1. NV server tells client to perform
backup to SD
2. Client sends data to SD server
3. Data is duplicated to tape library
4. If used, SD server de-dupes
data
5. SD server stores data on disk
De-dup
Process
Tape
Library
We are live tweeting and answering questions using hash tag: #DellDP. Join us at tweetchat.com/room/DellDP
37. 37
SmartDisk Replication
• Smart Disk can now
replicate data without
rehydrating it
• This saves bandwidth
SD Server
SD Disk #1
SD
Agent
NetVault Server
NVBK
Server
De-dup
Process
SD Server
SD Disk #2
SD
Agent
De-dup
Process
We are live tweeting and answering questions using hash tag: #DellDP. Join us at tweetchat.com/room/DellDP
38. 38
How Well Does It Work?
• Customer had their main database server fail
• We recovered the OS in ½ hour
• Database data took another hour
• They were up and running in less than 2 hours
• Routine single file recoveries take seconds
• Second site recovery takes the same time
We are live tweeting and answering questions using hash tag: #DellDP. Join us at tweetchat.com/room/DellDP
39. 39
Customer #2: A global coverage web conference hosting
company
• Customer has data centers in the U.S., England and Pacific Rim
• They wanted a backup system that would allow them to recover any
office to another site
• They had very large network pipes between the sites
We are live tweeting and answering questions using hash tag: #DellDP. Join us at tweetchat.com/room/DellDP
40. 40
The Web Conference Company’s RPO/RTO
• RPO - They can stand up to one day’s loss of data
– Clustered servers means one can fail, others in the datacenter will fill in
– Other sites can cover if a data center is lost
– If data is available at another datacenter, it can be spun up quickly
• RTO – They need to be back up in 1 hour max. – Time is money!
– Standardized server image allows quick duplication of a server
– Customer setup data is only important part of recovery
We are live tweeting and answering questions using hash tag: #DellDP. Join us at tweetchat.com/room/DellDP
41. 41
How We Did It....
• NetVault Backup has the capability of easily making a duplicate of a backup
– It is built into the backup setup window
– Just takes a few clicks
– You can specify where you want the duplicate made
– You can specify how you want the duplicate made
– By the client
– By the server
• SmartDisk became the target to allow very fast recovery
We are live tweeting and answering questions using hash tag: #DellDP. Join us at tweetchat.com/room/DellDP
42. 42
Making A Backup Copy – It’s Easy !!!
We are live tweeting and answering questions using hash tag: #DellDP. Join us at tweetchat.com/room/DellDP
44. 44
Recovery....
• Because alternate sites have backup copy, they can recover to other servers
at the alternate site
• If the primary NetVault server is missing, a quick download and install of
NetVault software creates a new NetVault server
• The SmartDisk at the alternate site is imported into NetVault
• NetVault allows you to recover the data to another server by simply selecting
the server as the target
We are live tweeting and answering questions using hash tag: #DellDP. Join us at tweetchat.com/room/DellDP
45. 45
Q&A
We are live tweeting and answering questions using hash tag: #DellDP
Join us at tweetchat.com/room/DellDP
My first introduction to the industry was punch cards and programming in Hex - after that, I knew I’d rather work on the hardware side of things. Then, along came the IBM PC and I was hooked!Along the way I started a business building computers and providing service to businesses, and then shifted focus slightly to the video surveillance market and started building DVRs.I also did twenty some odd years in the Royal Canadian Navy as communications technician - primarily in a support role working with Windows and Unix networks in the naval engineering school – where the big difference between that and a civilian school, is that in the Navy you can tell a student to sit down and pay attention – and they will!A few years after I retired, I ended up working for an amazing company located in Nova Scotia, Canada.
I’ll tell you a little about where I work, so you can get an idea of where we were before starting on the disaster recovery path.We’ve been a major corrugated paper folding and converting manufacturer (or as my son says, “a box factory”) for more than 80 years, and we’ve currently got a workforce of about 125 employees working between four plants located in each of the Canadian Atlantic provinces.From the inception of the company, we’ve looked at the way we manufacture, with an eco friendly and sustainable view. We even have a line of 100% recyclable and FDA approved wax alternative coatings to prevent boxes that used to go to landfills, to now go into the recycling stream.Our market is concentrated in the north eastern part of the US & Canada, Cuba & the Caribbean, and north western Europe.We have a number of machines that can output mind numbing amounts of boxes: up to 25,000 boxes an hour– it’s a staggering amount and they all haveto be tracked right from the making the corrugated paper through till the finished product gets loaded into containers, for shipment by truck or ships to final destination.
Peak summer periods have us running 24 hours a day, with three shifts.We depend on Sequel, Exchange & BES, domain controllers, terminal servers, and file & print servers that all must run smoothly – not to mention other servers looking after our Intranet and the normal mish-mash of servers that make up a small business’s back end.Depending on the season, with the number of boxes we manufacture in just an hour, a server failure could mean a plant shutdown, work stoppages or remote plant closures, and could cost us an amount bigger than I’d care to imagine!
Our existing backup strategy involved rotating tapes and full backups to removable disks – yup, we were saddled with an older backup scheme that was getting worn and showing its age.Indeed, our server population was aging – most of our servers were between 5 & 10 years old, and as a result they were prone to major and minor failures.When there was a system failure, we were only backing up files, so that meant we had to recover the system first, building a replacement system from scratch, update the OS and install software, and then … Crossing our fingers as we restored files to the server from disk and tape.But the daily reality was it would take overnight to recover a lost file from tape – meaning we had to identify the tape the file was on and bring it in the next day from home.--If a Sequel server failed, the entire plant would be shut down while we tried to transition to working with pen and paper…The fallout from this was raising frustrations and costs – and the best we could do was file backups only – we had no provision for any kind of system-level recoveries.But there was hope on the horizon. I’d mentioned that our servers were all getting a little long in the tooth, and I’d just gotten the go ahead to virtualize our server environment.We were also looking at Business Continuity and we wanted a Disaster Recovery plan. Virtualization was going to help tie things together.But this plan was going to leave us with a mix of physical & virtual servers. That meant we needed to be able to back up both kinds of servers.
By the time we started converting old servers to VM’s, we knew our existing backup software wasn’t up to the task.--We’d either have to pay for an expensive upgrade to our legacy software, or be left backing up just a few files to tape.--It all came to a head that we couldn’t ignore any more – we needed something that would look after our complete environment.
I spent about a month looking for a solution that:A - Reduced recovery times of both applications and data from hours to minutes,B - Restored either back to physical or to VMware-based virtual machines,C - Replicated to a planned but not built remote DR site in case of a local site-wide disaster,D - Provided robust domain controller continuity. And finally, it had to back up both virtual and physical servers – being unable to do this (what I thought was a basic task) was a deal-breaker.I quickly discovered that my most basic of requirements - that is, backing up physical AND virtual machines was not going to be so simple to accomplish in a single package. There were the old timers that could only do physical server back ups, and the upstarts that figured everyone only used virtual servers.I sat through several webinars, downloaded various trial packages and checked them out, but hadn’t found anything yet that could do this. I really was about to give up, but through a search engine, up came the name AppAssure, and its website shouted out exactly what I was looking for!
I must say, I raised my eyebrows at all the claims they made, as I couldn’t believe they were the only ones that could “do it all”.So I sat through webinar and was impressed, but I had to see for myself. I downloaded a demo and was convinced.The ROI and the advanced backup and recovery features made it not only a contender, but a no-brainer solution for us.My dream was about to become true - no more going home to get a tape just to recover a file. Well, not so much a dream, but I was happy to be able to recover files quickly!
Dell loves to quantify things in customer case studies so I’ve agreed to this slide. But the most important thing is not numbers so much as peace of mind knowing that the problems we had prior to this are behind us.
We achieved a major upgrade using two virtual AppAssure Core machines – one local and one at our disaster recovery siteThere are now virtual servers being replicated and on standby in case of failure in our 100% virtual DR infrastructure – making it easy to do “push button” failover to virtual. We can do rapid P2V and V2V migrations and DR testing without interrupting production systems. Our system now allows us very fast restores at any level: single file, email or even a complete server. One thing we don’t do though, is to tell users that we can restore individual emails or files lest they use us as their Recycle Bin.And of course – DR testing is nothing to be afraid of anymore!
We’ve had a couple good examples of recovering from major failure.The first was a few years before I started here, and much before we got AppAssure, and it involved a crashed Exchange server. An outside consultant was called in and it was a 24 hour job to get something up and running just to get the email flowing again.The second example was a few months after AppAssure was installed when our primary domain controller crashed – it also handled file shares, DHCP, Print server, etc. The RAID backplane failed and the server was shut down. I’ve always been a hardware guy so my first instinct was to check out the server to see what was wrong. By the time I figured out there was no quick fix, it was back to my desk to bring up the AppAssure console to see what I could do. After a few mouse clicks I had a VMware host selected to place a virtual copy of the physical server on, and a few minutes later we were back in business. Although this was a heart stopping moment, once the new VM was running and it appeared to our users as if nothing had happened (other than the server “was down” for a short while, I marvelled at how easy and quickly what would have been a major disaster, was fixed. This was my first experience with having to recover a production server in a hurry. It went off without a hitch, and I then sat down and specked out a new Dell R515. Once it was delivered and ready to take over from the VM that replaced the old server, I restored the running VM to the new hardware.I can’t imagine the nightmare that would have ensued had I been stuck with minimally useable backup tapes and have to recreate a new server along with all the shares and user permissions, print drivers, DHCP scopes, etc. I’m sure that anyone that’s had a DC with many roles crash, knows it wouldn’t be a fun week trying to recover and restore everything. I think that AppAssure paid for itself in just this one major incident.Since then we’ve recovered plenty of “lost” emails and files for users generally within moments of their panicked phone call, and I can’t imagine going back in time and having to do this any other way!
It’s really amazing how far backups have come over the last few decades. We’ve got entire servers being protected and replicated as often as every 15 minutes and others are backed up hourly or weekly, and all without any intervention. We went from rotating daily tapes that copied a few files and databases, to a system that copied the entire server in a very granular fashion. In the event of a system crash, no longer did we have to source a replacement server, wait for it to show up at the door, install the OS and patches, configure the server, install programs, and then hope that the data from tape was all intact.We’ve incorporated our backup system into a major component of our Business Continuity and Disaster Recovery plan that we hope we never have to use – but knowing it‘s ready should we need it. I don’t want to sound like I’m standing on the stage at an awards night, but I really am thankful that we’ve got AppAssure looking after our environment! Thank you!