Your SlideShare is downloading. ×
Faster Replication of Sunrise Clinical Manager Databases for Running DBCC
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

Faster Replication of Sunrise Clinical Manager Databases for Running DBCC

458

Published on

0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
458
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
0
Comments
0
Likes
1
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. Faster Replication of Sunrise Clinical Manager Databases for Running DBCC Ziaul Mannan -Senior Technical DBA Howard Goldberg -Director, Clinical Systems and Support
  • 2. Yale-New Haven Hospital New Haven, Connecticut  944 Bed Tertiary Teaching Facility  2600 Medical Staff  7550 Employees  100% CPOE  Average 350,000 orders monthly  Average Daily Census is 724  7 Time “Most Wired” and 3 Time “Most Wireless” Hospital by Hospitals and Health Networks
  • 3. Future Clinical Cancer Center At Yale-New Haven Hospital  112 inpatient beds  Outpatient treatment rooms  Expanded operating rooms  Infusion suites  Diagnostic imaging services  Therapeutic radiology  Specialized Women's Cancer Center  Yale-New Haven Breast Center/GYN Oncology Center  Tentative Completion Date: 2009
  • 4. Problem • With continual growth of SCM Master DB, it just gets longer to copy the db over to DBCC Server for DBCC Checks. • In our case ~ 12 Hours. • During this long time there‟s a potential for the Master DB to be online with corruption in it. • Too many steps involved to backup,copy the DB over and run DBCC checks then look for errors.
  • 5. Goals • Copy database over to the DBCC server and restore in shortest possible time • Run DBCC and scan reports for errors • Notify DBAs via e-mails or alerts on the result of DBCC Challenges • Find an innovative way of duplicating the production DB • Backup and restore production DB without jeopardizing availability, data integrity or performance • Detect corruption in DB before it can spread
  • 6. YNHH Production Environment • SCM 4.5 SP5 RU3 – Migrating to SCM4.5 with SMM – Total Users Defined – 10,000 – Users logged on at peak hours ~450 – SCM Reports, HL7 Interfaces,CDS,Multum – No CDR – Total disk for data 700GB (all servers) – Total disk for Archive 500 GB
  • 7. YNHH Production Environment • MS SQL Server – SQL Server 2000 EE Build 2195: SP 4 – Master and Enterprise on their own Servers and both clustered – MSCS and EMC SRDF/CE used as Clustering solution • OS and Hardware – Window 2000 Advance Server SP4 – Local SCSI and EMC Disks on Symmetrix
  • 8. YNHH Production Environment – Distributed SCM Environment • Master Server (MSCS Cluster using EMC SRDF/CE) ~ 125 GB DB • Enterprise Server (MSCS+EMC SRDF/CE) • HL7 Server (MSCS+EMC SRDF/CE) • Reports Server (MSCS+EMC SRDF/CE) • CDS Server (MSCS+EMC SRDF/CE) • Multum Server (MSCS+EMC SRDF/CE) • Compaq Servers - DL760G2,DL380G3,DL560 • 2-8 CPUs, 3-8 GB RAM
  • 9. YNHH SCM Production Environment MSMQ DCs SCM Master DB Client Workstations Enterprise Server XAENTER1PA Master Active DB XAENTERP XAMASTER1PA XAENTERCL1 XAMASTERP SCM Client SCM Client XAMASTERCL1 YNHORG2 YNHORG4 HL7 Interface Server HL7 Interface Server Executive Server Manager Server MSMQ XAAPPS2P XAHL71PA XAHL7P XAHL7CL1 SunriseXA Services Notification, CDS and Order Generation Server Multum Server Report Server XACOGNS1PA XAMULTUM1PA XAREPORT1P XACOGNSP XAMULTUMP XAREPORTP XACOGNSCL1 XAMULTUMCL1 XAREPORTCL1
  • 10. Time to Complete DBCC (Conventional Way) • Backup of PROD DB ~ 5hrs • Copy over to the DBCC Server via network ~5 hrs • Run DBCC on copy of PROD DB ~ 2 hrs • Total time spent before a corruption is detected ~ 12 Hrs
  • 11. Conventional DBCC Process DBCC Server SCM Master DB Client Workstations Copy over the Master Active network DB XAMASTER1P Enterprise Server A XAENTER1PA Copy of XAENTERP XAMASTERP SCM Master XAENTERCL1 XAMASTERCL SCM Client SCM Client Active DB 1 ERM Client HL7 Interface Server HL7 Interface Server Executive Server Manager Server MSMQ XAAPPS2P XAHL71PA XAHL7P XAHL7CL1 SunriseXA Services Notification, CDS and Order Generation Server Multum Server Report Server XACOGNS1PA XAMULTUM1PA XAREPORT1P XACOGNSP XAMULTUMP XAREPORTP XACOGNSCL1 XAMULTUMCL1 XAREPORTCL1
  • 12. Solutions/Tools • DBCC • VDI • EMC ERM • BCV • Time Finder
  • 13. Database Corruption • Compromises the database for logical and physical damage • Damages the database in its integrity and consistency • May impair database‟s functionality • Corruption can be caused by different reasons: – Disk Error – Memory Error – Power loss – Network
  • 14. DBCC • Database Consistency Checker – Very important database maintenance tool – Highly recommended by Eclipsys Support – It is executed against a database to detect any corruption before it compromises the database • DBCC scans data in database for any corruption • Results are logged in a text file • Different types of DBCC commands used to repair different kind of database corruption • DBCC should not be run on production DB server as it severely effects the performance of the server/DB
  • 15. DBCC • Eclipsys recommends running DBCC on a restored copy of Production DB on a separate Server • Eclipsys also recommends running DBCC on a regular basis • DBCC statements that Eclipsys wants us to run – DBCC CheckTable - the integrity of the data and index pages for the specified table – DBCC CheckDB - the allocation and structural integrity for all tables in a database – DBCC CheckAlloc - The integrity of disk allocation for a specified database – DBCC CheckCatalog - For consistency in and between system tables in the specified database
  • 16. VDI • Virtual Device Interface (VDI) is the programming interface for the Backup Restore API. It provides function calls to manipulate the backup and restore functionality in SQL Server. • VDI provides an alternative to using named pipes for backups • Provides interface and allows snapshot backups • VDI is used by several third-party tools that provide capabilities to back up SQL Server databases. • In SQL Server2000 information provided in VDI metadata can be used to restore a database from existing files created by TimeFinder
  • 17. EMC ERM • EMC Replication Manager creates replicas of mission critical information on disk arrays • ERM creates point in time replicas of databases or file systems • It creates replicas simply, quickly and automatically. • Automates all the important procedures related to data replication
  • 18. General Functionality • Where is the Original Data – Production Host • Application Set (formerly know as info set) – What is going to be Replicated - „Activities‟ • Filesystem • SQLServer Databases • How and When is the data going to be replicated. – Mount host name – Pre / Post Scripts – Backup Utilities – Mount Programs
  • 19. System Interfaces • Server Software • Agent Software
  • 20. BCV • EMC technology • Stands for Business Continuity Volumes • Creates mirror images of active production volumes • Disks are synchronized in the back ground transparent to the users and application • Can be spit and mounted to another host in point in time with out any user outages • Once host processes on the BCV device are complete,it can again be mirrored • Each BCV contains a unique host address and is configured as a single mirror
  • 21. Time Finder • EMC Technology • Creates multiple, independently addressable BVCs • TimeFinder helps maintain viable and secure system while ensures that each competing application receives data they need
  • 22. Time Finder Integration with MS SQL Server 2000 • TimeFinder SQL Server Integration Module (TSIM) • TSIM integrates and automates the features of Time Finder,Symmetrix Application Program Interface (API) and Microsoft‟s tool (VDI) supplied with MS SQL Server2000 • TSIM provides several interfaces to back up and restore database through SQL Server VDI • It can be used to reduce back/recovery time
  • 23. Replication Manager Architecture
  • 24. Replication Manager Architecture • Console Software – This will let you control Replication Manager from Windows or Unix workstation via TC/IP connection to the server • Server Software – Replication manager Service (IRD) – Replication Manager Database • Agent Software – Replication Manager Agent (IRCCD) – Storage Service Software – Application Agents for each supported interface • Database and File System Agents
  • 25. Table: Database and File System Agents Agent Information Replicated Oracle Agent Oracle information on raws devices or in datafiles Oracle information in RAC * Information from other applications, that has been organized within Solaris,Linux,HP- UX,IBM AIX or NTFS file systems UDB Agent UDB information on raw devices or in datafiles UDB information in EEE clusters ** Information from other applications, as long as data has been organized within Solaris,Linux, HP-UX or IBM AIX file systems SQL Server for NTFS File System Agent SQL Server 2000/2005 information Information in MSCS clusters* Information from other applications, as long as data has been organized within NTFS file systems Exchange for NTFS File System Agent Exchange 2000/2003 information Information in MSCS clusters* Information from other applications, as long as data has been organized within NTFS file systems *Data stored in clusters can be mounted to a nonclustered environment only **Replication Manager UDB agent supports EEE in only a single-node instance
  • 26. Agent Purpose and Functions • Specify which data to replicate • Makes sure that data can be replicated safely • Quiesce the database • Returns db to normal operation • Recover db during mounting operation • Shutdown and/or unmount database during restore operation
  • 27. Determining Replication Goals • Repurposing – Replica can be used carry out other functions; such as reporting,analysis, data mining without impacting production • Backup and Recovery – Replica can be used to backup the data to tape without effecting production performance • Disaster Restart – After a disaster the replica can be used to restart the application. Point in time data recovery is possible using roll forward • Copies of Replica – SAN copy can be used to copy the replica for multi purpose use
  • 28. Data Management with Replication Manager • Understanding storage options – RM can create replicas of data for Symmetrix DMX, Clarion • Selecting storage types – Flexible to store replicated data • Symmetrix BCVs,Clones,VDEV,Clarion Clone and snapshot • Symmetrix storage option – For Symmetrix DMX - replicas can be created on STDs,BCVs,VDEVs • Clarion storage option – Can use both clone and snapshot functionality
  • 29. ERM in YNHH SCM Production Environment SCM Master DB Client Workstations Master Active ERM Server DB XAMASTER1P Enterprise Server A XAENTER1PA XAMASTERP XAENTERP XAMASTERCL XAENTERCL1 SCM Client SCM Client 1 ERM Client ERM Copy DBCC Server ERM SQL Server Agent Copy of SCM Master Active DB HL7 Interface Server HL7 Interface Server Executive Server Manager Server MSMQ XAAPPS2P ERM Client XAHL71PA XAHL7P XAHL7CL1 SunriseXA Services Notification, CDS and Order Generation Server Multum Server Report Server XACOGNS1PA XAMULTUM1PA XAREPORT1P XACOGNSP XAMULTUMP XAREPORTP XACOGNSCL1 XAMULTUMCL1 XAREPORTCL1
  • 30. Step by Step SQL DB Copy (ERM)
  • 31. Time to Complete DBCC (ERM Way) • Replication of PROD DB and Copy over to the DBCC Server via timefinder ~ 40 min • Database in Quiesce State ~ 20 Sec • Mount the replicas and open DB ~ 40 Sec • Run DBCC on copy of PROD DB ~ 1 hr 40 min • Total time spent before a corruption is detected ~ 2.5 Hrs • Total time saved using ERM ~ 10 hrs
  • 32. Benefits • Faster Database copy • Copy, restore and DBCC process were fully automated • Very helpful during migration to SMC 4.5XA on new Servers • Standard process makes it easy for support staff to use it • Could be setup as an alternative DR System
  • 33. Disadvantages • Cost • Complex Setup • Requires extra disks • Version compatibility issue with Symmetrix micro code • Physical replication as oppose to logical replication • Only usable on EMC H/W • Not portable
  • 34. Conclusion • Minimized time required for the DBBC operation – From 12 hours to 2.5 hrs • Faster detection of corruption (if any) • Easy and frequent replication made possible • Create copy of Production DB on-demand as per request from our analysts to aid in their debugging
  • 35. References • SunriseXA Monitoring,Maintenance and Troubleshooting Guide,Course2052,April2003 by Eclipsys Corporation • EMC Replication Manager V4.0.0 Product Guide P/N 300-003-099 REV A01 by Eclipsys Corporation, Hopkinton, MA 01748-9103, 2006 • EMC TimeFinder Integration with Microsoft SQL Server2000- Using TimeFinder to Build a Highly Available Microsoft SQL Server 2000 Environment, Eclipsys Corporation, April 2002 Contact Information Ziaul Mannan : Ziaul.Mannan@ynhh.org Howard Goldberg: Howard.Goldberg@ynhh.org
  • 36. THANK YOU ! Questions?

×