Faster Replication of
 Sunrise Clinical
 Manager Databases for
 Running DBCC




Ziaul Mannan
-Senior Technical DBA
Howard...
Yale-New Haven Hospital
           New Haven, Connecticut

 944 Bed Tertiary Teaching Facility
 2600 Medical Staff
 755...
Future Clinical Cancer Center At
    Yale-New Haven Hospital
 112 inpatient beds
 Outpatient treatment rooms
 Expanded ...
Problem
• With continual growth of SCM Master DB, it just
  gets longer to copy the db over to DBCC Server
  for DBCC Chec...
Goals
• Copy database over to the DBCC server and restore in
  shortest possible time
• Run DBCC and scan reports for erro...
YNHH Production Environment
• SCM 4.5 SP5 RU3
  –   Migrating to SCM4.5 with SMM
  –   Total Users Defined – 10,000
  –   ...
YNHH Production Environment
• MS SQL Server
  – SQL Server 2000 EE Build 2195: SP 4
  – Master and Enterprise on their own...
YNHH Production Environment
– Distributed SCM Environment
  • Master Server (MSCS Cluster using EMC
    SRDF/CE) ~ 125 GB ...
YNHH SCM Production
              Environment
MSMQ DCs                                SCM Master DB
                      ...
Time to Complete DBCC
            (Conventional Way)
•   Backup of PROD DB ~ 5hrs
•   Copy over to the DBCC Server via net...
Conventional DBCC Process

 DBCC Server                                           SCM Master DB
                          ...
Solutions/Tools
•   DBCC
•   VDI
•   EMC ERM
•   BCV
•   Time Finder
Database Corruption
•   Compromises the database for logical and physical damage
•   Damages the database in its integrity...
DBCC
• Database Consistency Checker
   – Very important database maintenance tool
   – Highly recommended by Eclipsys Supp...
DBCC
• Eclipsys recommends running DBCC on a restored copy
  of Production DB on a separate Server
• Eclipsys also recomme...
VDI
• Virtual Device Interface (VDI) is the programming interface for the
  Backup Restore API. It provides function calls...
EMC ERM
• EMC Replication Manager creates replicas of mission
  critical information on disk arrays
• ERM creates point in...
General Functionality
• Where is the Original Data
   – Production Host
• Application Set (formerly know as info set)
   –...
System Interfaces
• Server Software
• Agent Software
BCV
• EMC technology
• Stands for Business Continuity Volumes
• Creates mirror images of active production volumes
• Disks...
Time Finder
• EMC Technology
• Creates multiple, independently addressable BVCs
• TimeFinder helps maintain viable and sec...
Time Finder Integration with MS
          SQL Server 2000
• TimeFinder SQL Server Integration Module (TSIM)
• TSIM integra...
Replication Manager Architecture
Replication Manager Architecture
• Console Software
   – This will let you control Replication Manager from Windows or
   ...
Table: Database and File System
            Agents
Agent                                       Information Replicated

Ora...
Agent Purpose and Functions
•   Specify which data to replicate
•   Makes sure that data can be replicated safely
•   Quie...
Determining Replication Goals
• Repurposing
   – Replica can be used carry out other functions; such as
     reporting,ana...
Data Management with Replication
           Manager
• Understanding storage options
   – RM can create replicas of data fo...
ERM in YNHH SCM Production
        Environment

                                                       SCM Master DB
     ...
Step by Step SQL DB Copy (ERM)
Time to Complete DBCC
               (ERM Way)
• Replication of PROD DB and Copy over to the DBCC
  Server via timefinder ...
Benefits
• Faster Database copy
• Copy, restore and DBCC process were fully automated
• Very helpful during migration to S...
Disadvantages
•   Cost
•   Complex Setup
•   Requires extra disks
•   Version compatibility issue with Symmetrix micro cod...
Conclusion
• Minimized time required for the DBBC operation
   – From 12 hours to 2.5 hrs
• Faster detection of corruption...
References
• SunriseXA Monitoring,Maintenance and Troubleshooting
  Guide,Course2052,April2003 by Eclipsys Corporation
• E...
THANK YOU !




 Questions?
Faster Replication of Sunrise Clinical Manager Databases for Running DBCC
Faster Replication of Sunrise Clinical Manager Databases for Running DBCC
Faster Replication of Sunrise Clinical Manager Databases for Running DBCC
Faster Replication of Sunrise Clinical Manager Databases for Running DBCC
Upcoming SlideShare
Loading in …5
×

Faster Replication of Sunrise Clinical Manager Databases for Running DBCC

641 views
577 views

Published on

0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
641
On SlideShare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
0
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

Faster Replication of Sunrise Clinical Manager Databases for Running DBCC

  1. 1. Faster Replication of Sunrise Clinical Manager Databases for Running DBCC Ziaul Mannan -Senior Technical DBA Howard Goldberg -Director, Clinical Systems and Support
  2. 2. Yale-New Haven Hospital New Haven, Connecticut  944 Bed Tertiary Teaching Facility  2600 Medical Staff  7550 Employees  100% CPOE  Average 350,000 orders monthly  Average Daily Census is 724  7 Time “Most Wired” and 3 Time “Most Wireless” Hospital by Hospitals and Health Networks
  3. 3. Future Clinical Cancer Center At Yale-New Haven Hospital  112 inpatient beds  Outpatient treatment rooms  Expanded operating rooms  Infusion suites  Diagnostic imaging services  Therapeutic radiology  Specialized Women's Cancer Center  Yale-New Haven Breast Center/GYN Oncology Center  Tentative Completion Date: 2009
  4. 4. Problem • With continual growth of SCM Master DB, it just gets longer to copy the db over to DBCC Server for DBCC Checks. • In our case ~ 12 Hours. • During this long time there‟s a potential for the Master DB to be online with corruption in it. • Too many steps involved to backup,copy the DB over and run DBCC checks then look for errors.
  5. 5. Goals • Copy database over to the DBCC server and restore in shortest possible time • Run DBCC and scan reports for errors • Notify DBAs via e-mails or alerts on the result of DBCC Challenges • Find an innovative way of duplicating the production DB • Backup and restore production DB without jeopardizing availability, data integrity or performance • Detect corruption in DB before it can spread
  6. 6. YNHH Production Environment • SCM 4.5 SP5 RU3 – Migrating to SCM4.5 with SMM – Total Users Defined – 10,000 – Users logged on at peak hours ~450 – SCM Reports, HL7 Interfaces,CDS,Multum – No CDR – Total disk for data 700GB (all servers) – Total disk for Archive 500 GB
  7. 7. YNHH Production Environment • MS SQL Server – SQL Server 2000 EE Build 2195: SP 4 – Master and Enterprise on their own Servers and both clustered – MSCS and EMC SRDF/CE used as Clustering solution • OS and Hardware – Window 2000 Advance Server SP4 – Local SCSI and EMC Disks on Symmetrix
  8. 8. YNHH Production Environment – Distributed SCM Environment • Master Server (MSCS Cluster using EMC SRDF/CE) ~ 125 GB DB • Enterprise Server (MSCS+EMC SRDF/CE) • HL7 Server (MSCS+EMC SRDF/CE) • Reports Server (MSCS+EMC SRDF/CE) • CDS Server (MSCS+EMC SRDF/CE) • Multum Server (MSCS+EMC SRDF/CE) • Compaq Servers - DL760G2,DL380G3,DL560 • 2-8 CPUs, 3-8 GB RAM
  9. 9. YNHH SCM Production Environment MSMQ DCs SCM Master DB Client Workstations Enterprise Server XAENTER1PA Master Active DB XAENTERP XAMASTER1PA XAENTERCL1 XAMASTERP SCM Client SCM Client XAMASTERCL1 YNHORG2 YNHORG4 HL7 Interface Server HL7 Interface Server Executive Server Manager Server MSMQ XAAPPS2P XAHL71PA XAHL7P XAHL7CL1 SunriseXA Services Notification, CDS and Order Generation Server Multum Server Report Server XACOGNS1PA XAMULTUM1PA XAREPORT1P XACOGNSP XAMULTUMP XAREPORTP XACOGNSCL1 XAMULTUMCL1 XAREPORTCL1
  10. 10. Time to Complete DBCC (Conventional Way) • Backup of PROD DB ~ 5hrs • Copy over to the DBCC Server via network ~5 hrs • Run DBCC on copy of PROD DB ~ 2 hrs • Total time spent before a corruption is detected ~ 12 Hrs
  11. 11. Conventional DBCC Process DBCC Server SCM Master DB Client Workstations Copy over the Master Active network DB XAMASTER1P Enterprise Server A XAENTER1PA Copy of XAENTERP XAMASTERP SCM Master XAENTERCL1 XAMASTERCL SCM Client SCM Client Active DB 1 ERM Client HL7 Interface Server HL7 Interface Server Executive Server Manager Server MSMQ XAAPPS2P XAHL71PA XAHL7P XAHL7CL1 SunriseXA Services Notification, CDS and Order Generation Server Multum Server Report Server XACOGNS1PA XAMULTUM1PA XAREPORT1P XACOGNSP XAMULTUMP XAREPORTP XACOGNSCL1 XAMULTUMCL1 XAREPORTCL1
  12. 12. Solutions/Tools • DBCC • VDI • EMC ERM • BCV • Time Finder
  13. 13. Database Corruption • Compromises the database for logical and physical damage • Damages the database in its integrity and consistency • May impair database‟s functionality • Corruption can be caused by different reasons: – Disk Error – Memory Error – Power loss – Network
  14. 14. DBCC • Database Consistency Checker – Very important database maintenance tool – Highly recommended by Eclipsys Support – It is executed against a database to detect any corruption before it compromises the database • DBCC scans data in database for any corruption • Results are logged in a text file • Different types of DBCC commands used to repair different kind of database corruption • DBCC should not be run on production DB server as it severely effects the performance of the server/DB
  15. 15. DBCC • Eclipsys recommends running DBCC on a restored copy of Production DB on a separate Server • Eclipsys also recommends running DBCC on a regular basis • DBCC statements that Eclipsys wants us to run – DBCC CheckTable - the integrity of the data and index pages for the specified table – DBCC CheckDB - the allocation and structural integrity for all tables in a database – DBCC CheckAlloc - The integrity of disk allocation for a specified database – DBCC CheckCatalog - For consistency in and between system tables in the specified database
  16. 16. VDI • Virtual Device Interface (VDI) is the programming interface for the Backup Restore API. It provides function calls to manipulate the backup and restore functionality in SQL Server. • VDI provides an alternative to using named pipes for backups • Provides interface and allows snapshot backups • VDI is used by several third-party tools that provide capabilities to back up SQL Server databases. • In SQL Server2000 information provided in VDI metadata can be used to restore a database from existing files created by TimeFinder
  17. 17. EMC ERM • EMC Replication Manager creates replicas of mission critical information on disk arrays • ERM creates point in time replicas of databases or file systems • It creates replicas simply, quickly and automatically. • Automates all the important procedures related to data replication
  18. 18. General Functionality • Where is the Original Data – Production Host • Application Set (formerly know as info set) – What is going to be Replicated - „Activities‟ • Filesystem • SQLServer Databases • How and When is the data going to be replicated. – Mount host name – Pre / Post Scripts – Backup Utilities – Mount Programs
  19. 19. System Interfaces • Server Software • Agent Software
  20. 20. BCV • EMC technology • Stands for Business Continuity Volumes • Creates mirror images of active production volumes • Disks are synchronized in the back ground transparent to the users and application • Can be spit and mounted to another host in point in time with out any user outages • Once host processes on the BCV device are complete,it can again be mirrored • Each BCV contains a unique host address and is configured as a single mirror
  21. 21. Time Finder • EMC Technology • Creates multiple, independently addressable BVCs • TimeFinder helps maintain viable and secure system while ensures that each competing application receives data they need
  22. 22. Time Finder Integration with MS SQL Server 2000 • TimeFinder SQL Server Integration Module (TSIM) • TSIM integrates and automates the features of Time Finder,Symmetrix Application Program Interface (API) and Microsoft‟s tool (VDI) supplied with MS SQL Server2000 • TSIM provides several interfaces to back up and restore database through SQL Server VDI • It can be used to reduce back/recovery time
  23. 23. Replication Manager Architecture
  24. 24. Replication Manager Architecture • Console Software – This will let you control Replication Manager from Windows or Unix workstation via TC/IP connection to the server • Server Software – Replication manager Service (IRD) – Replication Manager Database • Agent Software – Replication Manager Agent (IRCCD) – Storage Service Software – Application Agents for each supported interface • Database and File System Agents
  25. 25. Table: Database and File System Agents Agent Information Replicated Oracle Agent Oracle information on raws devices or in datafiles Oracle information in RAC * Information from other applications, that has been organized within Solaris,Linux,HP- UX,IBM AIX or NTFS file systems UDB Agent UDB information on raw devices or in datafiles UDB information in EEE clusters ** Information from other applications, as long as data has been organized within Solaris,Linux, HP-UX or IBM AIX file systems SQL Server for NTFS File System Agent SQL Server 2000/2005 information Information in MSCS clusters* Information from other applications, as long as data has been organized within NTFS file systems Exchange for NTFS File System Agent Exchange 2000/2003 information Information in MSCS clusters* Information from other applications, as long as data has been organized within NTFS file systems *Data stored in clusters can be mounted to a nonclustered environment only **Replication Manager UDB agent supports EEE in only a single-node instance
  26. 26. Agent Purpose and Functions • Specify which data to replicate • Makes sure that data can be replicated safely • Quiesce the database • Returns db to normal operation • Recover db during mounting operation • Shutdown and/or unmount database during restore operation
  27. 27. Determining Replication Goals • Repurposing – Replica can be used carry out other functions; such as reporting,analysis, data mining without impacting production • Backup and Recovery – Replica can be used to backup the data to tape without effecting production performance • Disaster Restart – After a disaster the replica can be used to restart the application. Point in time data recovery is possible using roll forward • Copies of Replica – SAN copy can be used to copy the replica for multi purpose use
  28. 28. Data Management with Replication Manager • Understanding storage options – RM can create replicas of data for Symmetrix DMX, Clarion • Selecting storage types – Flexible to store replicated data • Symmetrix BCVs,Clones,VDEV,Clarion Clone and snapshot • Symmetrix storage option – For Symmetrix DMX - replicas can be created on STDs,BCVs,VDEVs • Clarion storage option – Can use both clone and snapshot functionality
  29. 29. ERM in YNHH SCM Production Environment SCM Master DB Client Workstations Master Active ERM Server DB XAMASTER1P Enterprise Server A XAENTER1PA XAMASTERP XAENTERP XAMASTERCL XAENTERCL1 SCM Client SCM Client 1 ERM Client ERM Copy DBCC Server ERM SQL Server Agent Copy of SCM Master Active DB HL7 Interface Server HL7 Interface Server Executive Server Manager Server MSMQ XAAPPS2P ERM Client XAHL71PA XAHL7P XAHL7CL1 SunriseXA Services Notification, CDS and Order Generation Server Multum Server Report Server XACOGNS1PA XAMULTUM1PA XAREPORT1P XACOGNSP XAMULTUMP XAREPORTP XACOGNSCL1 XAMULTUMCL1 XAREPORTCL1
  30. 30. Step by Step SQL DB Copy (ERM)
  31. 31. Time to Complete DBCC (ERM Way) • Replication of PROD DB and Copy over to the DBCC Server via timefinder ~ 40 min • Database in Quiesce State ~ 20 Sec • Mount the replicas and open DB ~ 40 Sec • Run DBCC on copy of PROD DB ~ 1 hr 40 min • Total time spent before a corruption is detected ~ 2.5 Hrs • Total time saved using ERM ~ 10 hrs
  32. 32. Benefits • Faster Database copy • Copy, restore and DBCC process were fully automated • Very helpful during migration to SMC 4.5XA on new Servers • Standard process makes it easy for support staff to use it • Could be setup as an alternative DR System
  33. 33. Disadvantages • Cost • Complex Setup • Requires extra disks • Version compatibility issue with Symmetrix micro code • Physical replication as oppose to logical replication • Only usable on EMC H/W • Not portable
  34. 34. Conclusion • Minimized time required for the DBBC operation – From 12 hours to 2.5 hrs • Faster detection of corruption (if any) • Easy and frequent replication made possible • Create copy of Production DB on-demand as per request from our analysts to aid in their debugging
  35. 35. References • SunriseXA Monitoring,Maintenance and Troubleshooting Guide,Course2052,April2003 by Eclipsys Corporation • EMC Replication Manager V4.0.0 Product Guide P/N 300-003-099 REV A01 by Eclipsys Corporation, Hopkinton, MA 01748-9103, 2006 • EMC TimeFinder Integration with Microsoft SQL Server2000- Using TimeFinder to Build a Highly Available Microsoft SQL Server 2000 Environment, Eclipsys Corporation, April 2002 Contact Information Ziaul Mannan : Ziaul.Mannan@ynhh.org Howard Goldberg: Howard.Goldberg@ynhh.org
  36. 36. THANK YOU ! Questions?

×