My first introduction to the industry was punch cards and programming in Hex - after that, I knew I’d rather work on the hardware side of things. Then, along came the IBM PC and I was hooked!Along the way I started a business building computers and providing service to businesses, and then shifted focus slightly to the video surveillance market and started building DVRs.I also did twenty some odd years in the Royal Canadian Navy as communications technician - primarily in a support role working with Windows and Unix networks in the naval engineering school – where the big difference between that and a civilian school, is that in the Navy you can tell a student to sit down and pay attention – and they will!A few years after I retired, I ended up working for an amazing company located in Nova Scotia, Canada.
I’ll tell you a little about where I work, so you can get an idea of where we were before starting on the disaster recovery path.We’ve been a major corrugated paper folding and converting manufacturer (or as my son says, “a box factory”) for more than 80 years, and we’ve currently got a workforce of about 125 employees working between four plants located in each of the Canadian Atlantic provinces.From the inception of the company, we’ve looked at the way we manufacture, with an eco friendly and sustainable view. We even have a line of 100% recyclable and FDA approved wax alternative coatings to prevent boxes that used to go to landfills, to now go into the recycling stream.Our market is concentrated in the north eastern part of the US & Canada, Cuba & the Caribbean, and north western Europe.We have a number of machines that can output mind numbing amounts of boxes: up to 25,000 boxes an hour– it’s a staggering amount and they all haveto be tracked right from the making the corrugated paper through till the finished product gets loaded into containers, for shipment by truck or ships to final destination.
Peak summer periods have us running 24 hours a day, with three shifts.We depend on Sequel, Exchange & BES, domain controllers, terminal servers, and file & print servers that all must run smoothly – not to mention other servers looking after our Intranet and the normal mish-mash of servers that make up a small business’s back end.Depending on the season, with the number of boxes we manufacture in just an hour, a server failure could mean a plant shutdown, work stoppages or remote plant closures, and could cost us an amount bigger than I’d care to imagine!
Our existing backup strategy involved rotating tapes and full backups to removable disks – yup, we were saddled with an older backup scheme that was getting worn and showing its age.Indeed, our server population was aging – most of our servers were between 5 & 10 years old, and as a result they were prone to major and minor failures.When there was a system failure, we were only backing up files, so that meant we had to recover the system first, building a replacement system from scratch, update the OS and install software, and then … Crossing our fingers as we restored files to the server from disk and tape.But the daily reality was it would take overnight to recover a lost file from tape – meaning we had to identify the tape the file was on and bring it in the next day from home.--If a Sequel server failed, the entire plant would be shut down while we tried to transition to working with pen and paper…The fallout from this was raising frustrations and costs – and the best we could do was file backups only – we had no provision for any kind of system-level recoveries.But there was hope on the horizon. I’d mentioned that our servers were all getting a little long in the tooth, and I’d just gotten the go ahead to virtualize our server environment.We were also looking at Business Continuity and we wanted a Disaster Recovery plan. Virtualization was going to help tie things together.But this plan was going to leave us with a mix of physical & virtual servers. That meant we needed to be able to back up both kinds of servers.
By the time we started converting old servers to VM’s, we knew our existing backup software wasn’t up to the task.--We’d either have to pay for an expensive upgrade to our legacy software, or be left backing up just a few files to tape.--It all came to a head that we couldn’t ignore any more – we needed something that would look after our complete environment.
I spent about a month looking for a solution that:A - Reduced recovery times of both applications and data from hours to minutes,B - Restored either back to physical or to VMware-based virtual machines,C - Replicated to a planned but not built remote DR site in case of a local site-wide disaster,D - Provided robust domain controller continuity. And finally, it had to back up both virtual and physical servers – being unable to do this (what I thought was a basic task) was a deal-breaker.I quickly discovered that my most basic of requirements - that is, backing up physical AND virtual machines was not going to be so simple to accomplish in a single package. There were the old timers that could only do physical server back ups, and the upstarts that figured everyone only used virtual servers.I sat through several webinars, downloaded various trial packages and checked them out, but hadn’t found anything yet that could do this. I really was about to give up, but through a search engine, up came the name AppAssure, and its website shouted out exactly what I was looking for!
I must say, I raised my eyebrows at all the claims they made, as I couldn’t believe they were the only ones that could “do it all”.So I sat through webinar and was impressed, but I had to see for myself. I downloaded a demo and was convinced.The ROI and the advanced backup and recovery features made it not only a contender, but a no-brainer solution for us.My dream was about to become true - no more going home to get a tape just to recover a file. Well, not so much a dream, but I was happy to be able to recover files quickly!
Dell loves to quantify things in customer case studies so I’ve agreed to this slide. But the most important thing is not numbers so much as peace of mind knowing that the problems we had prior to this are behind us.
We achieved a major upgrade using two virtual AppAssure Core machines – one local and one at our disaster recovery siteThere are now virtual servers being replicated and on standby in case of failure in our 100% virtual DR infrastructure – making it easy to do “push button” failover to virtual. We can do rapid P2V and V2V migrations and DR testing without interrupting production systems. Our system now allows us very fast restores at any level: single file, email or even a complete server. One thing we don’t do though, is to tell users that we can restore individual emails or files lest they use us as their Recycle Bin.And of course – DR testing is nothing to be afraid of anymore!
We’ve had a couple good examples of recovering from major failure.The first was a few years before I started here, and much before we got AppAssure, and it involved a crashed Exchange server. An outside consultant was called in and it was a 24 hour job to get something up and running just to get the email flowing again.The second example was a few months after AppAssure was installed when our primary domain controller crashed – it also handled file shares, DHCP, Print server, etc. The RAID backplane failed and the server was shut down. I’ve always been a hardware guy so my first instinct was to check out the server to see what was wrong. By the time I figured out there was no quick fix, it was back to my desk to bring up the AppAssure console to see what I could do. After a few mouse clicks I had a VMware host selected to place a virtual copy of the physical server on, and a few minutes later we were back in business. Although this was a heart stopping moment, once the new VM was running and it appeared to our users as if nothing had happened (other than the server “was down” for a short while, I marvelled at how easy and quickly what would have been a major disaster, was fixed. This was my first experience with having to recover a production server in a hurry. It went off without a hitch, and I then sat down and specked out a new Dell R515. Once it was delivered and ready to take over from the VM that replaced the old server, I restored the running VM to the new hardware.I can’t imagine the nightmare that would have ensued had I been stuck with minimally useable backup tapes and have to recreate a new server along with all the shares and user permissions, print drivers, DHCP scopes, etc. I’m sure that anyone that’s had a DC with many roles crash, knows it wouldn’t be a fun week trying to recover and restore everything. I think that AppAssure paid for itself in just this one major incident.Since then we’ve recovered plenty of “lost” emails and files for users generally within moments of their panicked phone call, and I can’t imagine going back in time and having to do this any other way!
It’s really amazing how far backups have come over the last few decades. We’ve got entire servers being protected and replicated as often as every 15 minutes and others are backed up hourly or weekly, and all without any intervention. We went from rotating daily tapes that copied a few files and databases, to a system that copied the entire server in a very granular fashion. In the event of a system crash, no longer did we have to source a replacement server, wait for it to show up at the door, install the OS and patches, configure the server, install programs, and then hope that the data from tape was all intact.We’ve incorporated our backup system into a major component of our Business Continuity and Disaster Recovery plan that we hope we never have to use – but knowing it‘s ready should we need it. I don’t want to sound like I’m standing on the stage at an awards night, but I really am thankful that we’ve got AppAssure looking after our environment! Thank you!
Disaster recovery. prepare.plan.perform.
Disaster Recovery: Prepare, Plan & Perform
2Agenda• Introductions• Key things to know about disaster recovery requirements andexpectations in today’s world• How a large manufacturer moved to meet disaster recoverychallenges• Supporting multiple customers through data loss andrecovery• Q&AWe are live tweeting and answering questions using hash tag: #DellDP. Join us at tweetchat.com/room/DellDP
3Introducing our speakersModerator: George CrumpStorage Switzerland.www.storage-switzerland.comRob MacCara,System AdministratorMaritime Paper Products, Ltd.Robby WrightChief Technical ConsultantAbtech TechnologiesWe are live tweeting and answering questions using hash tag: #DellDP. Join us at tweetchat.com/room/DellDP
4Key things toknow aboutdisaster recoveryrequirementsand expectationsin today’s worldWe are live tweeting and answering questions using hash tag: #DellDP. Join us at tweetchat.com/room/DellDP
5About George Crump & Storage Switzerland• Analyst firm covering storage, cloud and virtualization markets• Knowledge of these markets is gained through product testing, real-worldimplementations and interactions with users and suppliers• The results of this research are found in the articles, briefing reports, casestudies and lab reports on www.storage-switzerland.comGeorge CrumpChief Steward, Storage Switzerlandwww.firstname.lastname@example.org/storageswissyoutube.com/user/storageswissWe are live tweeting and answering questions using hash tag: #DellDP. Join us at tweetchat.com/room/DellDP
Meeting the Recovery ExpectationWe are live tweeting and answering questions using hash tag: #DellDP. Join us at tweetchat.com/room/DellDP
• Users expect to up 100% of the time just like"FaceBook" or they expect outages to beminimal - Minutes of downtime, not hours• Meeting this expectation means that data canno longer be "restored" - The network transferis too time consuming• Data have to be recoverable "in-place" and ithas to be readable the first time - VerificationWe are live tweeting and answering questions using hash tag: #DellDP. Join us at tweetchat.com/room/DellDPMeeting the Recovery Expectation
Meeting the Zero Data Loss ExpectationWe are live tweeting and answering questions using hash tag: #DellDP. Join us at tweetchat.com/room/DellDP
• Users expect you to have every copy andversion of their data protected all the time.Just like "DropBox"• Traditional once-a-night backup is no longerenough. Too much data is created, modifiedand potentially deleted in a day• Backup has to occur at multiple pointsthroughout the day, potentially hourly, withoutimpacting performanceWe are live tweeting and answering questions using hash tag: #DellDP. Join us at tweetchat.com/room/DellDPMeeting the Zero Data Loss Expectation
Meeting the Keep It Forever ExpectationWe are live tweeting and answering questions using hash tag: #DellDP. Join us at tweetchat.com/room/DellDP
• Tape makes this difficult because you haveto manage time expectations. Disk makesthis difficult because you have to manageexpense expectations. - Try Tape, Requirescalable deduplicated disk• Users expect you to keep all their dataforever, near-instantly available...for free• Reality is that 99% of the data will never beneeded again. The problem is you dont knowwhere that 1% is going to come from.We are live tweeting and answering questions using hash tag: #DellDP. Join us at tweetchat.com/room/DellDPMeeting the Keep It Forever Expectation
• Backup may be “all about recovery”but backups mattero Granular Backupso Frequent Backupso Validated Backups• Recovery Needs To Changeo In-Placeo VirtualWe are live tweeting and answering questions using hash tag: #DellDP. Join us at tweetchat.com/room/DellDPMeeting Cloud Expectations
13How a largemanufacturermoved to meetdisasterrecoverychallengesWe are live tweeting and answering questions using hash tag: #DellDP. Join us at tweetchat.com/room/DellDP
14Rob MacCara• System Administrator, Maritime PaperProducts, Ltd., Nova Scotia• 30 years of IT experience• Built his own company building, selling, andmaintaining Digital Video Surveillancesystems for both small and large retailcustomers• 20 years in the Royal Canadian Navy as acommunications technician specializing incomputer systems
15Maritime Paper Products: manufacturing market leader• Major corrugated box manufacturer with multi-continental reach• Factories across the Canadian Atlantic provinces• Forward-thinking with both automation andsustainable manufacturing• Massive daily output
16Maritime Paper’s business-critical applications• Peak summer periods have us running 24hours a day, with three shifts• We depend on:o Domain controllero Microsoft SQL Server appso Microsoft Exchange Server• A server failure can stop everythingand close plants, resulting inpotentially very large revenue lossesRob standing in front of one of Maritime Paper’shigh-speed box fabricating machines
17Challenge of a failing legacy backup scheme• Time-consuming rotation of tapes, full backupsto removable disks• Older server population prone to major failures• Day-long file recoveries• File backup only• Virtualization effortIn Maritime’s high-volume productionenvironment, the pressure is onto deliver to customers on time sotheir customers are also on time.
18Time for a changeVirtualization was the last straw• Existing backup software wasn’t up tothe task• Choice seemed to be either anexpensive upgrade or having to settlefor greatly reduced backup capabilityMaritime reached a point whereits existing legacy system wouldrequire a costly upgrade.
19Looking for solutionsKey requirementsWe looked for an advanced backupand disaster recovery solution that:• Worked across virtual and physical servers• Reduced recovery times• Restored to any type of machine• Included replication
20Why we decided on Dell AppAssureA ”night and day difference”• Strong ROI and a wealth of advanced backupand recovery features• No more tape• Recoveries in minutes• Simplified recovery offsite
21Results• 95% savings in storage space• Minutes to recover lost file versus24 hours with our previous dataprotection product• $28,000 savings in software andhardware
22Fast forward to todayA strong DR, migration and testing solution• Pair of virtual AppAssure Core machines forlocal and offsite recoveries• Push-button failover to virtual• Simplified creation of DR site• Fast restores at any level• Rapid P2V and V2V migrations• Faster DR testing
23Recovery proof pointRecent primary domain controller’s RAID backplane failed• Resulted in a major crash• Up and running in less than an hour as a virtualmachine, including troubleshooting the server• Avoided:o All–nightero Panico Revenue loss• Costs and staff time would have been much higher withoutDell AppAssure
24What we learnedNow we know:• Our data is safe• It’s being continually backed up
25Supportingmultiplecustomerthrough dataloss andrecoveryWe are live tweeting and answering questions using hash tag: #DellDP. Join us at tweetchat.com/room/DellDP
26About Robby Wright & AbtechRobby WrightChief Technical ConsultantAbtech Technologieswww.abtechtechnologies.comWe are live tweeting and answering questions using hash tag: #DellDP. Join us at tweetchat.com/room/DellDP
27Two Customers – Similar Solutions• Customer #1 – A power company in Texas that needs to protect againststorm damage at their primary site– In a hurricane area– Flat ground and near a river - increases flooding possibilities– Can’t afford to lose customer information and billing capabilities– Must be able to pay vendors after disaster damage• Customer #2 – A global coverage web conference hosting company– Hosting equipment sites on multiple continents– Hosts very large conferences – 40,000+ attendees– Sites back each other upWe are live tweeting and answering questions using hash tag: #DellDP. Join us at tweetchat.com/room/DellDP
28The Power Company• Has two sites connected by dark fiber optic cables– Set up with 10GBit IP connection between sites• Using NetVault Backup and SmartDisk for normal backup processes• NetVault SmartDisk’s new replication feature allows painless replication ofdeduplicated backup data between sites• Mix of approximately 60 physical servers plus virtual machines• Many databases, email, portals• Needs long backup chains for regulatory compliance – Tape allows thisWe are live tweeting and answering questions using hash tag: #DellDP. Join us at tweetchat.com/room/DellDP
29How do we plan the backups?• Determine what needs to be backed up• Determine priorities for both backup and recovery• Set RPO and RTO for each• Determine backup window availability• Do we need special handling for databases or other applications?• Make sure we have what we need to recover• Design backup system to meet requirementsWe are live tweeting and answering questions using hash tag: #DellDP. Join us at tweetchat.com/room/DellDP
30All Jobs Start With A Server SurveyCustomer Name:PriorityServer or VMNameOperating SystemApplication onServerRTO RPO IP AddressDisk Allocated(GB)Disk Used(GB)DependenciesWe are live tweeting and answering questions using hash tag: #DellDP. Join us at tweetchat.com/room/DellDP
31Determine the RPO/RTO For Each ServerRPO Recovery Point Objective How far back can data be lost?RTO Recovery Time Objective How long after failure until system up andusable?Failure Point RTORPOLost Data Time DownTimeWe are live tweeting and answering questions using hash tag: #DellDP. Join us at tweetchat.com/room/DellDP
32The Power Company’s Recovery Objectives• RPO - They can stand up to one day’s loss of data– Data can be re-input if necessary for time lost– Backup to both disk and tape provides multiple fallbacks if necessary• RTO – They need to be back up and running in 8 hours max.– Bare metal backup and recovery restores O/S and applications– SmartDisk and/or tape provides recovery of dataWe are live tweeting and answering questions using hash tag: #DellDP. Join us at tweetchat.com/room/DellDP
33What is special handling?• Exchange, SharePoint and databases either have to be stopped or have somemethod for a point-in-time backup to be able to recover it.• Active file systems require snapshot capabilities for accurate backup• Virtual machines can be backed up as either a client or as a VM• Use of plug-ins requires backup as a client, not a VM.We are live tweeting and answering questions using hash tag: #DellDP. Join us at tweetchat.com/room/DellDP
34GB/Hr, Real LifeYou Must Be Able To Get the Data From TheDisk(s) To the Tape Drive Fast Enough !!!File size x disk I/O rate = MB/sec.My tape drive will record 100MBytes/Sec, so whyis my backup slow?We are live tweeting and answering questions using hash tag: #DellDP. Join us at tweetchat.com/room/DellDP
35How do we speed up backups?• Backing up to disk first allows multiple servers to back up at their own speed• Backups can then be streamed to tape at full speed of the drive.• Maximizes utility of and saves wear and tear on the tape drivesWe are live tweeting and answering questions using hash tag: #DellDP. Join us at tweetchat.com/room/DellDP
36Disk-To-Disk-To-TapeSD ServerSD DiskSDAgentNetVault Server ClientNVBKServerNVClient1. NV server tells client to performbackup to SD2. Client sends data to SD server3. Data is duplicated to tape library4. If used, SD server de-dupesdata5. SD server stores data on diskDe-dupProcessTapeLibraryWe are live tweeting and answering questions using hash tag: #DellDP. Join us at tweetchat.com/room/DellDP
37SmartDisk Replication• Smart Disk can nowreplicate data withoutrehydrating it• This saves bandwidthSD ServerSD Disk #1SDAgentNetVault ServerNVBKServerDe-dupProcessSD ServerSD Disk #2SDAgentDe-dupProcessWe are live tweeting and answering questions using hash tag: #DellDP. Join us at tweetchat.com/room/DellDP
38How Well Does It Work?• Customer had their main database server fail• We recovered the OS in ½ hour• Database data took another hour• They were up and running in less than 2 hours• Routine single file recoveries take seconds• Second site recovery takes the same timeWe are live tweeting and answering questions using hash tag: #DellDP. Join us at tweetchat.com/room/DellDP
39Customer #2: A global coverage web conference hostingcompany• Customer has data centers in the U.S., England and Pacific Rim• They wanted a backup system that would allow them to recover anyoffice to another site• They had very large network pipes between the sitesWe are live tweeting and answering questions using hash tag: #DellDP. Join us at tweetchat.com/room/DellDP
40The Web Conference Company’s RPO/RTO• RPO - They can stand up to one day’s loss of data– Clustered servers means one can fail, others in the datacenter will fill in– Other sites can cover if a data center is lost– If data is available at another datacenter, it can be spun up quickly• RTO – They need to be back up in 1 hour max. – Time is money!– Standardized server image allows quick duplication of a server– Customer setup data is only important part of recoveryWe are live tweeting and answering questions using hash tag: #DellDP. Join us at tweetchat.com/room/DellDP
41How We Did It....• NetVault Backup has the capability of easily making a duplicate of a backup– It is built into the backup setup window– Just takes a few clicks– You can specify where you want the duplicate made– You can specify how you want the duplicate made– By the client– By the server• SmartDisk became the target to allow very fast recoveryWe are live tweeting and answering questions using hash tag: #DellDP. Join us at tweetchat.com/room/DellDP
42Making A Backup Copy – It’s Easy !!!We are live tweeting and answering questions using hash tag: #DellDP. Join us at tweetchat.com/room/DellDP
43Site-To-Site TransfersClientNVClientSDAgentSD DiskClientNVClientSDAgentSD DiskInternetSan JoseLondonHong KongSD DiskNetVault ServerSDAgentNVBKServerWe are live tweeting and answering questions using hash tag: #DellDP. Join us at tweetchat.com/room/DellDP
44Recovery....• Because alternate sites have backup copy, they can recover to other serversat the alternate site• If the primary NetVault server is missing, a quick download and install ofNetVault software creates a new NetVault server• The SmartDisk at the alternate site is imported into NetVault• NetVault allows you to recover the data to another server by simply selectingthe server as the targetWe are live tweeting and answering questions using hash tag: #DellDP. Join us at tweetchat.com/room/DellDP
45Q&AWe are live tweeting and answering questions using hash tag: #DellDPJoin us at tweetchat.com/room/DellDP