Network Infrastructure Validation Conference @UPRA (2003)
Upcoming SlideShare
Loading in...5
×
 

Like this? Share it with your network

Share

Network Infrastructure Validation Conference @UPRA (2003)

on

  • 1,040 views

 

Statistics

Views

Total Views
1,040
Views on SlideShare
1,040
Embed Views
0

Actions

Likes
2
Downloads
41
Comments
0

0 Embeds 0

No embeds

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

Network Infrastructure Validation Conference @UPRA (2003) Presentation Transcript

  • 1. COMPUTER VALIDATIONTESTING FOR IT AND NETWORK EQUIPMENT Presented by: Raúl Soto, BSME Computer Validation Team Leader AstraZeneca Ltd – IPR Pharmaceuticals rsv21@coqui.net rasove_21@yahoo.com 1
  • 2. Presentation Overview1. Introduction • Why Validate Computer Systems? • Computer Validation General Principles2. Areas to be covered • Servers and Network Equipment & Systems Validation IQ and OQ tests • Software Validation GAMP software categories Software Development testing3. Wrapping Up4. Q & A 2
  • 3. Disclaimer This presentation and all opinions therein are solely the responsibility of the author; and not that of the UPR system or AstraZeneca PLC. 3
  • 4. Why validate Computer Systems? 4
  • 5. Why validate Computer Systems?It’s the LAW: • Code of Federal Regulations (CFR)It makes good business sense: • Understanding of your processes / systems • Improved operational efficiency • Reduced risk of failure • Maintenance of quality standards 5
  • 6. Validation is a Regulatory RequirementFDA Requirements for Validation of Computerized SystemsExamples: 21 CFR 211.68 Automatic, mechanical, and electronic equipment 21 CFR 820.70 Production and Process Controls 21 CFR 11.10 Controls for Closed Systems 6
  • 7. Computer Validation General PrinciplesWhat is a “Validation” ?• “Establishing documented evidence which provides a high degree of assurance that a process, equipment, or system will consistently fulfill its intended purpose, meeting its predetermined specifications and quality attributes.” - FDA Guidelines on General Principles of Process Validation (1987) 7
  • 8. Computer Validation General PrinciplesWhich Systems Require Validation?• Systems that automate processes regulated by GxPs• Systems with an impact on product quality, safety, identity, efficacy, or purity.• Systems used to make quality decisions• Systems in scope for 21 CFR Part 11 8
  • 9. Computer Validation General PrinciplesValidation vs. Qualification• We validate systems that automate GxP-regulated processes. Validation is the overall combination of plans, activities, documents, and approvals Validation includes one or more qualifications• Qualifications are test protocols that verify the system or components of the system Infrastructure entities, such as networks or data centers, are qualified for use as part of your overall system validation. 9
  • 10. Computer Validation General PrinciplesQualification Protocol • Qualification tests should be contained in a protocol document. • This document should be approved by the Management of the areas impacted by the validation, and by Quality Assurance, BEFORE any validation testing occurs. • The protocol should contain: overview of the system description of the validation testing strategy detailed description of all tests to be performed pre-established acceptance criteria for all tests 10
  • 11. Computer Validation General PrinciplesInstallation Qualification (IQ) Protocol• Documented verification that a system is installed according to written and pre-approved specifications.Operation Qualification (OQ) Protocol• Documented verification that a system operates according to written and pre-approved specifications throughout all specified operating ranges.Performance Qualification (PQ) Protocol• Documented verification that a system is capable of performing or controlling the activities of the process it is required to perform or control, according to written and pre- approved specifications, while operating in its specified operating environment. 11
  • 12. Computer Validation * Main Areas *Industrial Automation Equipment• Machines or lines controlled by Programmable Logic Controllers (PLCs), ControlLogix, DeviceNet, PCs, or other computerized control systems• Computerized Vision or inspection systemsSoftware• Operating Systems, firmware, or software packages (canned-off-the- shelf, configurable, or custom-made)Networks & Related Equipment• Servers, Routers, switches, cabling, workstations• DNS service, Domain Controllers, DHCP systems• WAN circuits 12
  • 13. IT NETWORK EQUIPMENTVALIDATION 13
  • 14. IT & Network Equipment ValidationWhat is a network (as far as the FDA is concerned) ?• 1. (ISO) An arrangement of nodes and interconnecting branches. 2. A system [transmission channels and supporting hardware and software] that connects several remotely located computers via telecommunications. (Source: FDA - Glossary of Computerized System and Software Development Terminology, Aug 1995)• Includes: Supporting hardware (e.g. servers, workstations, transmission channels) Supporting software (e.g. network operating system) Processes and procedures (e.g. change management) People (e.g. administrators, auditors) 14
  • 15. IT & Network Equipment ValidationWhy must networks be validated ?• Our industry produces two critical outputs: medical products and data.• Medical products are supported and marketed based upon the quality and meaning of the underlying data.• The integrity of this data must be assured and maintained.• The validation process provides the mechanism for assuring and maintaining data and process integrity. 15
  • 16. IT & Network Equipment ValidationWhy must networks be validated ?• Networks are systems that are actively involved in creating, modifying, maintaining, archiving, retrieving, and transmitting data (electronic records and electronic signatures).• Successful network validation offers a “high degree of assurance” that the system will perform its intended functions, according to predetermined specifications.• It’s a regulatory requirement. 16
  • 17. IT & Network Equipment ValidationIT & Network Equipment IQ and OQ• IQ (Installation Qualification) Document that the system has been installed according to predefined specifications & acceptance criteria.• OQ (Operational Qualification) Document that the system performance meets predefined specifications & acceptance criteria. 17
  • 18. IT & Network Equipment ValidationNetwork Validation Example :• GMP Servers• Office Computing Equipment (network printers and PCs)• IT Network Equipment (routers, switches, cabling, WAN circuits)• DNS Service & Domain Controller Servers• DHCP Service 18
  • 19. IT & Network Equipment ValidationIQ for IT Systems 19
  • 20. IT & Network Equipment ValidationIQ Recommended Forms: Cover form • approval signatures, document number, etc. System description and info [one for each device in IQ] • what device is used for • If it’s part of a larger system • what changes are being made to the system (if applicable) Documents, manuals, drawings [one for each device in IQ] • Include version no., location Instruments used • include copies of calibration certifications Software installed [one for each device in IQ] • OS, antivirus, other applications, etc. • Include name, brief description, license #, version #, path. 20
  • 21. IT & Network Equipment ValidationIQ Recommended Forms: Electrical utilities [one for each device in IQ] • Actual vs spec: device voltage (V), phase, current (A) • panel, breaker locations; signature & licence # of electrician cerifying the installation meets National Electric Code (NEC) Signatures log for IQ • Log the name, title, signature, and initials of everyone whose signature or initials appear in any of the IQ forms Spare parts list • if applicable Specific Equipment forms [one for each device in IQ] • server, PC, printer, router, etc. 21
  • 22. IT & Network Equipment ValidationIQ Specific IT Equipment Forms: Server Router Switch Network PC Network Printer Network Cabling WAN Circuit 22
  • 23. IT & Network Equipment Validation IQ form example:Equipment Number ________ IQ Protocol _____________Page _ of _ IQ TEST TITLE(Tests & info collected) Location / Room: ____________________ Manufacturer: ________ Model: __________ Tag No. _________ Tag/ Property # ______ Serial # _________ Server Name: _________ CPU Type/Qty _____ CPU clock Speed ____ IP Address: _____ Domain ________ OS _________ RAM ________ TCP/IP – Ethernet configured (Y/N) ____ Disk Array Configuration _________ HD Qty/Capacity ________ Expansion Boards (Qty, type, model) _______ Input V _____ Input frequency _______ Serial, Parallel, USB Ports (qty each) _______ UPS connected (Y/N) ___ Type ______Complies with Acceptance Criteria: □Y □N □N/A If No or N/A, explain in CommentsComments: 23Performed By: _____________ Verified By: ____________
  • 24. IT & Network Equipment ValidationIQ Attachments (recommended): Network Topology & Racks Diagrams License Evidence • for OS and all applications installed (SQL, antivirus, etc.) • Include copy of document, or printout / screenshot of license number Physical Access Report • list of everyone with physical access to computer rooms and cabinets/closets, • Include name, account #, access level, access status (active/inactive) Electrical Power Equipment Calibration Forms • copies of calibration certifications for multimeters, clamp meters, etc. used in IQ 24
  • 25. IT & Network Equipment ValidationIQ Attachments (recommended): System Information Reports – for Servers or PCs • Start>Programs>Administrative Tools>Computer Management • Click over System Tools>System Information • Click SAVE TEXT REPORT icon, print it and attach it. IP Address Configuration – for Servers or PCs • run IPCONFIG command, print results Electrical Drawings Deviations found during IQ • Log listing all deviations, with description, status • All individual deviation forms, properly completed and approved. 25
  • 26. IT & Network Equipment ValidationServer IQ Form Contents:(including DHCP and Domain Controller servers) Server General Information: • Manufacturer, model, serial no., location, property tag number • Server name, IP Address, domain, installed services • Server Description • Operating System • CPU type and quantity, clock speed • RAM amount • Removable storage drives: type, quantity • Hard drives: quantity, capacity; are they hot-swappable? • Disk Array configuration: None, RAID 0, RAID 1, RAID 4, RAID 5, other • Ports: quantity of USB, serial, parallel ports Expansion Boards • Quantity, type, model 26
  • 27. IT & Network Equipment ValidationServer IQ Form Contents: Communications • Configured for TCP/IP (y/n) • Configured for Ethernet (y/n) Network interface cards • quantity, speed; • other adapters Power • quantity of power supplies • UPS (y/n); if Y then Plant or Stand-alone UPS? Input voltage (V) & frequency (Hz) Room Environment • Operating Temperature (max/min), • Operating RH% (max/min) Room environmental conditions monitoring documentation • (copy of chart recordings) 27
  • 28. IT & Network Equipment ValidationNetwork PC IQ Form Contents:(1 PC per model) Manufacturer, model, serial no., location, property tag number Notebook or Desktop Location (room) CPU, RAM, HD/RAID array, COM ports, Parallel ports, USB ports, other ports OS NICs, TCP/IP or Ethernet IP Address, if static Input voltage & frequency UPS (y/n); if Y then Plant or Stand-alone UPS? Operating Temperature (max/min), Operating RH% (max/min) 28
  • 29. IT & Network Equipment ValidationNetwork Printer IQ Form Contents: Manufacturer, model, serial number Location, property tag number Network printer or stand-alone Amount of RAM IP Address, if networked Input voltage & frequency UPS (y/n); if Y then Plant or Stand-alone UPS? Operating Temperature (max/min), Operating RH% (max/min) 29
  • 30. IT & Network Equipment ValidationRouter IQ Form Contents: Manufacturer, model, serial number IP Address Location, property tag number # of network ports, if they are Ethernet or AUI # of serial ports Input voltage & frequency UPS (y/n); if Y then Plant or Stand-alone UPS? Operating Temperature (max/min), Operating RH% (max/min) 30
  • 31. IT & Network Equipment ValidationSwitch IQ Form Contents: Manufacturer, model, serial number Location, property tag number IP Address Supports 10 BaseT, 100 BaseT, Wireless ? Backbone: UTP, Fiber? Switch configuration settings Input voltage & frequency UPS (y/n); if Y then Plant or Stand-alone UPS? Operating Temperature (max/min), Operating RH% (max/min) 31
  • 32. IT & Network Equipment ValidationNetwork Cabling IQ Form Contents: Fiber or UTP UTP: CAT5 or above? Cabling description Labeling scheme (closet / rack / port) Labeling certification 32
  • 33. IT & Network Equipment ValidationWAN Circuit IQ Form Contents:(1 per circuit) Location Service provider From / to Fiber / microwave / other Data / voice / both Bandwith Routers connected Certified ? – copy of communications circuit certification Circuit exclusive to company (closed system)? 33
  • 34. IT & Network Equipment ValidationOQ/PQ for IT Systems 34
  • 35. IT & Network Equipment ValidationNetwork OQ Testing• Standard OQ Tests • IQ completion and approval verification • Risk assessment – safety department evaluation • Instrument calibration documentation & evidence• Component-specific OQ Tests • Servers • Routers • Switches • etc. 35
  • 36. Server OQ Testing1. Server clock accuracy2. Diagnostic test3. Startup & Shutdown4. Loss of power & UPS test5. Server power supply redundancy test6. Communications redundancy test7. Log files verification8. Virus Protection9. Backup & Restore10. Security 36
  • 37. Server OQ Testing• Server Clock Accuracy Verify time and date displayed are correct, and document this. Using a calibrated chronometer, measure a period of 24 hours. At the end of the 24-hr period, verify that the time displayed in the server corresponds to the time shown in the chronometer, ±2 sec 37
  • 38. Server OQ Testing• Hardware Manufacturer Diagnostic Test Execute the Diagnostic test provided by the server’s manufacturer; print test results and attach them to Raw Data. If Diagnostic test shows any error, document this, and: • Explain if this is acceptable; or • Correct problem, and repeat the diagnostic test. For Compaq Proliant servers: • Turn ON server, and press F10 as the server boots up • In System Setup, select Diagnostic & Utilities, then select Quick Check Diagnostic, and then Start. 38
  • 39. Server OQ Testing• Startup & Shutdown With server turned ON, select Start -> Shutdown, the server should shut down completely. Turn ON server When the Login screen appears, log on using Administrator account Once logged, go to Start -> Programs -> Administrative Tools -> Events Viewer In the Events Viewer select the Application Log and print it. 39
  • 40. Server OQ Testing• Power Loss and UPS Test Ensure UPS is fully charged With the server turned ON, simulate a general power failure by unplugging the UPS from the power outlet. Ensure the UPS provides at least 15 minutes of power to the server, sufficient time to shut down the server properly, or to wait until the site’s emergency power comes online 40
  • 41. Server OQ Testing• Power Supply Redundancy Test With the server ON, disconnect the power cord from one of its power supplies Ensure that the server stays ON, document any messages or warnings displayed. Reconnect the power cord, and repeat the test with the second power cord. 41
  • 42. Server OQ Testing• Communications Redundancy Test With the server ON, use Advanced Server Administrator tools to print the server’s Host Name and the IP Address Disconnect the Ethernet cable from the primary NIC card to simulate a communications loss. Verify that the message “NIC Card Cable Unplugged” appears in the screen, and that the NIC is still working. Use a PC connected to the network to open a DOS window Type NSLOOKUP and the server name. The network should respond with the correct server name and IP address. If a “Request Time Out” message appears, the test has failed. Type EXIT to close DOS window Reconnect primary NIC Ethernet cable, the message “NIC Card Cable Unplugged” should disappear. 42 Repeat steps with the secondary NIC Card Ethernet cable.
  • 43. Server OQ Testing• Log files verification Go to the C:winntsystem32config folder, and open it Verify the existence of the following log files: • SYSEVENT.EVT - System log file • APPEVENT.EVT - Application log file • SECEVENT.EVT - Security log file Go to Start -> Programs -> Administrative Tools -> Event Viewer Right-click the Event Viewer, select Properties for each event log file, and print a screenshot showing Maximum Log Size and Event Log Wrapping information. Ensure that for all event log files: • Maximum Log Size = 5120K • Settings when maximum log size is reached = Overwrite Events as Needed 43
  • 44. Server OQ Testing• Virus Protection Verification Verify the antivirus icon appears in the bottom right corner of the Windows screen. Select this icon and ensure the antivirus software is activated. Perform virus scan of server boot sector and hard drives. Open virus scan report, print it and verify that boot sector and all hard drives are virus free. 44
  • 45. Server OQ Testing• Backup & Restore Use a tool such as Veritas Backup Exec or Symantec Ghost to create a backup or an image of the server and all settings Restore the backup or image (if possible, in a spare server of the same model), and verify the server’s functionality. This description corresponds to the simple case of a single GMP server. For setups with server clusters, servers connected to a mass storage unit (e.g. EMC Symmetrix), or servers with an SQL database, the backup test will be more complex (and beyond the scope of this conference) 45
  • 46. Server OQ Testing• Security Physical Security – Access Control • Try to enter Computer Room using ID Card without access, card reader should not grant access • Repeat with authorized ID Card, card reader should grant access Logical Security – Password Policy • Verify and document that the following controls are in place: Password expiration period set (e.g. 90 days) as per SOP “Password Never Expires” option NOT active Password length limit (e.g. 8 characters) Blank passwords NOT allowed Password Uniqueness enforced Account locked after 3 unsuccessful login attempts Only Administrator can lock / unlock account 46
  • 47. Network Validation• Office Computing Equipment OQ PC Communications test: • Login with an administrator account • PING the Domain Controller server IP Address • Get return reply confirming that communication is OK • Do this for each PC Desktop / notebook model Network Printer Communications test: • Print test page • Ensure the page has the correct printer name, model, date and time • Do this for each network printer model PC Security test: • Attempt to login with different combinations of correct & incorrect user name & password. • Attempt to cause an account lock, get IT to release the account afterwards 47
  • 48. Network Validation• Office Computing Equipment OQ Virus Protection verification: • Refer to SERVER OQ TESTING for Virus Protection test Stand-alone Printer Diagnostic Test • Print a Configuration page • Ensure all diagnostics are ok 48
  • 49. Network Validation• IT Network Equipment OQ Physical Security: • Obtain printout of personnel authorized to enter the room where equipment is located • If room has electronic access control system (e.g. using employee ID cards), test to ensure only authorized personnel can open door. System Security: • Connect physically to router or switch, using a laptop • Open telnet session • Try to enter using incorrect and correct passwords 49
  • 50. Network Validation• IT Network Equipment OQ Diagnostic Tests: • Connect PC to router or switch, open telnet session • Turn off router or switch • Power up • Verify that no start-up errors were generated UPS / Loss of Power Test: • With UPS fully powered, disconnect main power and ensure the UPS provides at least 15 minutes of power Fault Tolerance / Power Supply Redundancy Test: • for routers or switches with dual power supplies • Disconnect power cable from first power supply • Spare power supply should keep equipment running • Re-connect power cable from first power supply • repeat with power cable from second power supply 50
  • 51. Network Validation• IT Network Equipment OQ Communications Circuits redundancy test: • Use this when a WAN connects more than 1 site within the same network, and there are redundant connections (mw and fiber, or a 3rd site) • From one site, PING the destination site’s DHCP server • Get return reply confirming that communication is OK • Disconnect the circuit connecting to the remote site • PING again • Get return reply confirming that communication is OK • Try for all connections Switch Loss of Communication test: • Power LED • RJ45 port status LED should show OK • Unplug the communications cable from the switch • RJ45 LED should show no communication 51 • Plug cable, LED should show OK status
  • 52. Network Validation• IT Network Equipment OQ Network Stress test: • The objective of a stress test is to challenge system performance in a situation where system resources are under unusual or extreme demand in terms of quantity, volume, etc. • Test should challenge, during a high traffic scenario : PC and printer connectivity to network use of the Domain Controller and Domain Name services Use of the Dynamic Host Connectivity Protocol (DHCP) 52
  • 53. Network Validation• IT Network Equipment OQ Network Stress test example: • Use traffic generator software to induce a high volume of network traffic (e.g. 3x measured peak) to the Domain Controller and DHCP Servers • Use various PCs, each connected to a different network node; from each PC execute the following test: Login and open a DOS session PING the IP address for each Domain Controller server NSLOOKUP, the screen should display the Default Server Name and Default Server IP Address, and the “>” prompt Use IPCONFIG/RELEASE to release the dynamic IP address Use IPCONFIG/ALL to display the IP address value, it should be 0.0.0.0 Use IPCONFIG/RENEW to get a new dynamic IP Address. Screen should display the new IP Address, subnet mask, and default gateway. 53 Select a networked printer and print a Test Page
  • 54. Network Validation• DNS Service & Domain Controller Servers OQ All the regular Server OQ tests • For each Domain Controller server, even if they’re all the same model DNS Service Functionality Test DNS Service & Domain Controller Redundancy Test 54
  • 55. Network Validation• DNS Service & Domain Controller Servers OQ DNS Service Functionality Test • Purpose: to verify and document that the DNS service works as specified • Log in from a networked PC • NSLOOKUP, the screen should display the Domain Controller server name and IP address, and the “>” prompt • At the prompt, type the DHCP server name, system should display: SERVER: the Domain Controller server name ADDRESS: the DC server IP Address NAME: the DHCP server name ADDRESS: the DHCP server IP Address 55 • Type EXIT and close DOS window
  • 56. Network Validation• DNS Service & Domain Controller Servers OQ DNS Service & Domain Controller Redundancy Test • Purpose is to verify and document the response, and challenge the redundancy, of the DC and DNS services upon loss of communication. • Disconnect the plant WAN from the corporate network • Disconnect the local Domain Controller server from the network to simulate a communications loss • Log in from a networked PC, and open a DOS window • Type SET, a series of PC settings should be displayed • Record the value of the LOGON SERVER parameter, it contains the name of the Domain Controller server assigned to this PC • NSLOOKUP should display the values for one of the alternate Domain Controller servers in the network. • Execute this test for each Domain Controller server in the network. 56
  • 57. Network Validation• DHCP Servers OQ All the regular Server OQ tests • For each DHCP server, even if they’re all the same model DHCP Servers Emergency Repair • Verify the DHCP Emergency Repair disks exist, document location, backups, person responsible, etc. DHCP Scopes Configuration test • Verify the Address Pools for each site • Verify the Scope Options for each site 57
  • 58. Network Validation• DHCP Servers OQ DHCP Functionality & Redundancy Test • Log in, get IP address • verify it is within the correct Subnet ranges for the site • Verify it is not within the Exclusion ranges for the site • Release IP address • Renew connection (simulate loss of communication) • Display IP address, verify it is from an alternate DHCP server (redundancy) • verify it is within the correct Subnet ranges for the alternate site • Verify it is not within the Exclusion ranges for the alternate site 58
  • 59. Network Validation• Other items which may require Computer Validation : IBM AS400 Computers Laboratory Equipment (e.g. LIMS, HPLC) Remote Access Services SQL Databases w/ GMP data EMC Symmetrix Storage Units 59
  • 60. SOFTWARE VALIDATION 60
  • 61. Software Categories - Summary Category Software Validation Approach Type 1 Operating IQ: Record version (including service pack, if System applicable). The OS will be challenged indirectly by the functional testing of the application. 2 Firmware IQ: For non-configurable firmware, record version. Calibrate instruments as necessary. Verify operation against user requirements. IQ: For configurable firmware, record version and configuration. Calibrate instruments as necessary. OQ: Verify operation against user requirements. Manage custom firmware as Category 5 software.Source: GAMP 4 Guide, Appendix M4
  • 62. Software Categories - Summary Category Software Validation Approach Type 3 Standard IQ: Record version and configuration of Software environment; OQ: verify against user Packages requirements. Consider auditing the supplier for critical and complex applications. 4 Configurable IQ: Record version and configuration of Software environment; OQ: verify against user Packages requirements. Normally audit the supplier for critical and complex applications. Manage any custom programming as Cat-5. 5 Custom Audit supplier and (IQ, OQ) validate complete (Bespoke) system. SoftwareSource: GAMP 4 Guide, Appendix M4
  • 63. Software Development TestingUnit Test• Unit testing focuses on testing configured or customized code at the individual transaction, module, or component level. e.g. Simulated input -> [Module] -> OutputString Test• String testing focuses on testing strings of transactions, modules, or components which are commonly used together. e.g. Simulated input -> [Module 1] -> Output 1 Output 1 -> [Module 2 ] -> Output 2Integrated Testing• Integrated testing focuses on testing integrated scenarios which are intended to simulate entire processes performed by the software. 63
  • 64. Wrapping up …• What have we covered ? Overview of General Validation Concepts Computer Validation tests Servers and IT Network systems Software Validation tests 64
  • 65. We’re Done !