• Share
  • Email
  • Embed
  • Like
  • Save
  • Private Content
Software validation of applications deployed on Windows Azure

Software validation of applications deployed on Windows Azure



Infosys Case Study highlights Challenges & Approaches of Microsoft Azure Cloud Platform over Traditional On-Premise Testing Environment.

Infosys Case Study highlights Challenges & Approaches of Microsoft Azure Cloud Platform over Traditional On-Premise Testing Environment.



Total Views
Views on SlideShare
Embed Views



1 Embed 1

https://si0.twimg.com 1



Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.


11 of 1 previous next

  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
  • very poor font usage.
    Are you sure you want to
    Your message goes here
Post Comment
Edit your comment

    Software validation of applications deployed on Windows Azure Software validation of applications deployed on Windows Azure Document Transcript

    • Software validation of applicationsdeployed on Windows® Azure™Sidharth Subhash Ghag, Divya Sharma, Trupti Sarang May 2011
    • Abstract As a very popular Infosys adage goes, “In God we trust, rest all we test”, it reflects the importance of testing in the software development life cycle. According to Wikipedia1, Software Testing can be stated as the process of validating and verifying whether a software program/application/product – one, it meets the business and technical requirements that guided its design and development; two, works as expected and three, can be implemented with the same characteristics. As software programs get complex, testing becomes vital and error prone. Testing an application on cloud is quite different from testing an application in the traditional on-premise environment. Microsoft® AzureTM is one such cloud platform that runs applications on the cloud. From a testing perspective, Test environment setup on AzureTM is quick and easy which can save considerable time and effort as compared to test infrastructure setup and administration. However by design, the AzureTM platform imposes certain restrictions on services deployed on it. Thus testing an application hosted on AzureTM becomes difficult and complex as compared with their traditional on-premise counterparts. In this paper we shall explain challenges along with approaches to handle different aspects of testing on AzureTM . Also we shall provide an insight into the end-to-end testing lifecycle involved for testing applications to be deployed on Windows® AzureTM .2 | Infosys – White Paper
    • Table of ContentIntroducƟon ....................................................................................................................................... 3Role of tesƟng in development of Azure™ cloud applicaƟons .......................................................... 5Comparison between pracƟces of tesƟng tradiƟonal on-premise vis-a-vis Azure™ applicaƟons .... 5Testing Azure™ applicaƟons ............................................................................................................ 11 Unit TesƟng ................................................................................................................................................................. 11 IntegraƟon TesƟng ...................................................................................................................................................... 12 System TesƟng............................................................................................................................................................. 13 Performance TesƟng ................................................................................................................................................... 15 Security TesƟng ........................................................................................................................................................... 17 Compliance & Regulatory TesƟng ............................................................................................................................... 19 User Acceptance TesƟng ............................................................................................................................................. 20Benefits realized aŌer applicaƟon validaƟon with Azure™ ............................................................ 22Challenges of tesƟng applicaƟons on Azure™ ................................................................................. 23 OrganizaƟon Policies Are Not Implicit For Hosted ApplicaƟons .................................................................................. 23 Development Challenges ............................................................................................................................................. 23 ConnecƟvity ................................................................................................................................................................. 25Summary.......................................................................................................................................... 26Reference ........................................................................................................................................ 28 Infosys – White Paper | 3
    • IntroductionTesting an application is a very important aspect of software development and so is the case with the applications deployedon Microsoft® AzureTM. Similar to the development and testing tools available for on-premise applications, Microsoft®offers tools for application development and testing on AzureTM Platform as well. Microsoft® offers a simulation of the cloudenvironment for developing applications locally through the development fabric. Unit and integration testing can beperformed in a manner similar as it is performed for an on-premise application. But, testing after deploying an application onAzureTM is different from the approach of testing an on-premise application. Applications deployed on AzureTM are managed,governed and controlled by the Windows® AzureTM Fabric controllers. The controllers impose certain infrastructurerestrictions, defined in the form of role definitions, at different levels of the platform and which limit flexibility otherwiseavailable with on-premise systems. The levels of flexibility available to account owners on AzureTM would differ based onthe AzureTM role definitions – Web, Worker or VM (Virtual Machine) Roles. Broadly these allow AzureTM to offer cloudservice model capabilities such as Infrastructure as a Service (IaaS) and Platform as a Service (PaaS).To offer the IaaS service model, Microsoft® announced VM role in PDC 2010 with enhanced control and flexibility on avirtual machine instance. VM role is offered to support development and migration of Windows® server applications whichwould require the creation of a custom Windows® server image, installing required software, applications and managing it aswell. The VM role can be used to realize scenarios where legacy applications built on design practices more suited fo r on-premise deployments that needed to be moved to AzureTM Cloud.Whereas In the PaaS service model offering, AzureTM offers a ready platform for directly deploying and hosting applicationswithout having to worry about installing or managing the software required to run these applications. In this model, theplatform provides an abstraction layer comprising of application runtime services and tools over the underlyinginfrastructure. This layer offers a more managed and standard environment on which applications that would run with theavailable IaaS service models. One can make use of the web and worker roles to leverage the PaaS service models offered byAzureTM. However one must note that with high abstraction levels offered by the web and worker roles, Azure™ has accesscontrols in place which may restrict the free movement of applications from on-premise to cloud. These restrictions wouldalso apply while validating applications deployed under this service model on AzureTM.When one needs to test an Azure™ application, it should be tested on both development and test environments. In thedevelopment environment users can test an application on the development fabric and development storage, using VisualStudio test suite capabilities. Once the application is hosted on the Microsoft® AzureTM Platform, the scope of testingbecomes quite restrictive, which is enforced by the design of AzureTM Platform. At the same time, a ready-to-use platformsaves time and additional cost in setting up various test environments. Setting up environment and scaling it as desired is onlya few clicks away.In this paper, we shall discuss the methodology and approach to test next generation cloud based applications on theWindows® AzureTM platform. The scope of the discussion would be limited to software validation of applications developedon PaaS specific offering of the AzureTM platform i.e. apps leveraging the web/worker role models. We highlight the keychallenges faced with developing PaaS applications along with some approaches to overcome them. “IaaS” style -applications, developed using AzureTM Virtual Machine roles, will not be in the scope of this paper.4 | Infosys – White Paper
    • Role of testing in development of Azure™ cloud applicationsIn a typical Software Development Life Cycle (SDLC), each process has its corresponding verification/validation process toensure that the system works as required. SDLC of AzureTM applications also follows the same verification/validationapproach as in standard SDLC. Testing is an essential part of SDLC to identify any defects and bugs early in the project. If abug is detected later in the SDLC, its impact on the project will be higher and fixing it would require more effort and hencecost more to fix. The diagram below shows the correlation between the SDLC phases and their corresponding testing steps. Figure 1: Correlation between development and validation phasesProper testing of an AzureTM application becomes even more imperative since the owner of application has less control overthe application and data once the application is hosted on AzureTM .Comparison between practices of testing traditional on-premise vis-à-vis Azure™applicationsThough both on-premise and AzureTM application validation process will follow similar phases in the testing life cycle, asdepicted in the diagram below, the execution of these phases will have few significant differences which we shall highlight i nthe subsequent sections. Figure 2: A typical Application Validation life cycle Infosys – White Paper | 5
    • As traditional application testing requires various test environments to set up e.g. development, system test, performance test,quality assurance, user acceptance test, production etc. Whereas, since Microsoft® AzureTM platform offers Platform as aService, it gives liberty to application developers to host applications on a ready to use hosted platform without worryingabout infrastructure management. However, they get limited privileges on infrastructure. These features affect the approachof testing AzureTM applications as compared to that with the traditional application testing methods. The following diagramshows different deployment environments used and roles involved in the typical testing life cycle of a traditional on-premiseapplication compared with that of an AzureTM application. These figures are representative and actual scenarios may differbased on project requirements. FuncƟonal TesƟng Non - FuncƟonal TesƟng Performance Testers Developers Unit Client End User System TesƟng Testers Developers Performance TesƟng Unit TesƟng Security . Testers User ProducƟon IntegraƟon System Acceptance Environment . TesƟng TesƟng TesƟng . Unit TesƟng Security & Compliance TesƟng Development and Test ProducƟon Figure 3: Testing approach for an on-premise applicationThe above figure shows the environment setup required for various stages of an end-to-end testing lifecycle for anytraditional on-premise application. As you can observe one would be required to set up separate test environments withrequired hardware and software installations for the various types of testing. This adds as overhead in terms of efforts andcost involved in managing various test environments.6 | Infosys – White Paper
    • Figure 4: Testing approach for an AzureTM applicationThere are two deployment regions in one hosted AzureTM instance, i.e., AzureTM service viz. staging and production. Any ofthese two deployment regions of an AzureTM service can be used for testing the application based on the availability.Production and staging are exactly identical with respect to hardware and installed software. In the above diagram, we haveconsidered a scenario where we have used two AzureTM services in a project life cycle – one for testing and another forrelease. As depicted in the illustrative scenario above, testing on AzureTM environment saves considerable effort and time byeliminating the overheads in test environment setup and administration.In the traditional testing approach, test environment setup would require setting up several machines with required software,whereas in AzureTM it will only require provisioning an AzureTM account. Creating new AzureTM service will implicitlyallocate hardware resources from a shared pool along with the required software installations required to run your applicatio nin a matter of minutes.A comparative view highlighting the differences of the effort required for testing on traditional on-premise development vis-à-vis AzureTM cloud development is shown below: Traditional Application Azure™ Application Prepare test strategy Prepare test strategyTest plan Plan for test environment Plan for test environment Procure hardware of desired configuration and On-demand provisioning of Azure™ account required software licensesTest Identify test lab (Physical Location) Not Applicable (NA)environmentsetup Infrastructure management team will setup Not Applicable (NA) hardware and install software Perform a quality check of environment setup Not Applicable (NA) Infosys – White Paper | 7
    • Deploy application Deploy application Prepare deployment scripts for application and Not Applicable (NA) database to be deployed –MSI’s Application Prepare installation manuals to guide system Not Applicable (NA) Setup administrators Publish deployment package and database Publish application to target environment scripts to target AzureTM instance Setup master data (if any) Setup master data (if any) Write test cases for all types of testing to be Write test cases for all types of testing to be Test Design performed performed Generate test data Generate test data Unit Testing Unit Testing Debugging/ trouble shooting Debugging/ trouble shooting Unit Test Bug Fixing Bug Fixing Regression Test Regression Test Integrate and deploy the application on Integrate and set up application on dev fabric integration test environment Smoke Testing Smoke Testing Integration Integration Testing Integration Testing Test Debugging/ trouble shooting Debugging/ trouble shooting Bug Fixing Bug Fixing Regression Test Regression Test Use staging deployment region of AzureTM Setup system test environment. Repeat steps service 1 (procured for testing) with mentioned in the Test environment setup Deploy the application on staging environment Deploy the application on system test of Windows® AzureTM subscription for environment development System Test Smoke Testing Smoke Testing System Testing System Testing Identify bugs with code instrumentation and by Debugging/ trouble shooting capturing application data Bug Fixing Bug Fixing Regression Test Regression Test Performance Setup performance test environment. Repeat Use production deployment region of AzureTM8 | Infosys – White Paper
    • Test steps mentioned in the test environment setup service 1 (procured for testing) Deploy the application on performance test Deploy the application on production environment deployment region on AzureTM service 1 Develop Performance Test Scripts using the Develop Performance Test Scripts using the identified Performance Test Tool. identified Performance Test Tool. Perform Dry Runs Perform Dry Runs Setup Performance Test Data Setup Performance Test Data Run Performance Tests Run Performance Tests Capture performance metrics through AzureTM Capture performance metrics diagnostics Analyze output and find out parameters to Analyze output and find out parameters to improve improve Modify application and deploy again Modify application and deploy again Run Performance Tests run post performance Run Performance Tests run post performance defect fix defect fix Setup security test environment. Repeat steps Use staging deployment region of AzureTM mentioned in the test environment setup service 1 (procured for testing) Deploy the application on security test Deploy the application on staging deployment environment region of AzureTM service 1Security Test Security Testing Security Testing Find out any security threats, system Find out any security threats, system vulnerabilities vulnerabilities Modify application and deploy again Modify application and deploy again Security Tests run post defect fix Security Tests run post defect fix Compliance Testing Compliance Testing Find out any non-compliances as per the Find out any non-compliances as per theCompliance organization and Government policies such as organization and government policies such asTest data location, security audit etc. data location, security audit etc. Modify application and deploy again Modify application and deploy again Conduct Audits post defect fix Conduct Audits post defect fix Use staging deployment region of AzureTM Setup user acceptance test environment service 2 (procured for release) Deploy the application on user acceptance test Deploy the application on staging deploymentUser environment region of AzureTM service 2Acceptance Smoke Testing Not Applicable (NA)Testing User Acceptance Testing User Acceptance Testing Find out any defects Find out any defects Modify application and deploy again Modify application and deploy again Infosys – White Paper | 9
    • Conduct tests post defect fix Conduct tests post defect fix Use production deployment region of AzureTM Setup production environment service 2 (procured for release) Setup datacenter – real estate, utility services, communication services, security management, Not Applicable (NA) government and environmental approvals, etc. Align teams- infrastructure, database, Align teams – developers, application application administrators, developers administrators Procure and install hardware and network resources- switches, routers, servers, cables, Not Applicable (NA) racks, storage, H/W load balancers Upgrade to Procure and setup software licenses – operating Production system, application server, database, firewalls, Not Applicable (NA) antivirus, patches, etc. Validate the production environment for security Not Applicable (NA) threats and compliances Harden production environment setup Not Applicable (NA) Install operations tools for infrastructure Subscribe to cloud management vendors for monitoring and management application management and monitoring Prepare deployment scripts for application to be Prepare deployment package for application to deployed –MSI’s be deployed on AzureTM - cspkg Deploy application to production environment Swap deployment from staging to production Table 1: Comparison of efforts in testing on-premise and AzureTM applications Apart from test environment setup, there are dissimilarities in the processes of testing life cycles as well. The table below describes the differences in each phase of the testing life cycle for a traditional on-premise application and an AzureTM application. Traditional Application Azure™ Application Test Plan Prepare the test plan Prepare the test plan Test Design Define test strategy, goals and approach Define test strategy, goals and approach Test Execution Separate test environment setup needs to be Production size instances to be provisioned for done for development / Test / QA / different test environments Production Hardware and Software are to be Hardware and Software details are abstracted and a maintained computation and storage services are provided. Desired software has to be installed Standard MS software components such as IIS, .NET framework etc... are pre-installed with the AzureTM compute services However installing third party components would have to be done by the test engineer by enabling Administrator privileges on the cloud services. Exploring database for test verification is Although in latest releases MS has provided many comparatively simple aiding tools, accessing and exploring AzureTM10 | Infosys – White Paper
    • Storage or SQL Azure™ is complex as compared to their on-premise counterpart. For SQL Azure™ certain features of SQL Server and management studio are not available and for Azure™ storage, developers may need to use third party tools such as Cloudberry, Azure™ storage explorer, Cerebrata’s Cloud storage studio etc. Scaling the environment will require extra Azure™ can be scaled dynamically (both vertically hardware and software setup taking either as well horizontally) by modifying number of the vertical or horizontal scaling route. running instances without any overhead Test Analysis Debugging is easier than cloud applications Debugging cloud applications is a challenge Functionality rich tools are available for Being a nascent technology, the testing tools defect analysis, debugging, trouble shooting available are limited Table 2: Testing life cycle for on-premise and Azure™ applicationsTesting Azure™ applicationsThis section explains how various testing cycles are performed on Azure™ applications. Testing is carried out in bothdevelopment as well as the hosted Azure™ instance; following sections describe the overall process of testing an Azure™application:Unit TestingUnit testing is a verification and validation process for a unit of source code, carried out by the developers to ensure the codeis working as intended. It is performed for an Azure™ application in the same way as it is done for a traditional application.The diagram below shows the steps followed in each phase of unit testing. Figure 5: Steps in Unit TestingAzure™ applications can leverage the test feature of Microsoft® Visual Studio to auto generate unit test cases from theAzure™ source code. Testers can provide input and expected result values to run tests and verify the application logic. Infosys – White Paper | 11
    • Figure 6: Comparative View for unit testing These test cases are executed on development fabric which is a simulation environment of the Windows® Azure™ to simulate the Azure™ fabric locally. Unit testing of Azure™ applications doesn’t require any additional setup. The diagram below shows a comparative view of people, environment and tools required for unit testing an on-premise application vis-à- vis that of an Azure™ application. Integration Testing Integration Testing is performed on integrated modules comprising of individual unit tested modules. Integration testing ensures that the modules communicate right with each other and integrated modules work as specified in high level design. The diagram below shows the steps followed in integration testing. Figure 7: Steps in integration testing12 | Infosys – White Paper
    • The high level design is input of integration testing and integration test plan is prepared based on this. The individual unittested modules are integrated based on the approach chosen to integrate, i.e., top down (the top module is tested first and thenits subordinate modules), bottom up (the low level modules are tested first and then the top level ones calling them) andtested to ensure conformance with high level design.For an Azure™ application, various unit tested modules will be integrated on development fabric and integration testing isperformed on development fabric using the visual studio web tests and/or manual tests. It ensures that application interfacesare working properly and integration did not cause any issues in different modules. This is a part of functional testing ofapplication. Once it is confirmed that application is working fine on development environment then it is moved to Azure™cloud instance. The diagram below shows a comparative view of people, environment and tools required for integrationtesting an on-premise application vis-à-vis that of an Azure™ application. Figure 8: Comparative View for integration testingSystem TestingSystem testing is performed on an application with all its disparate modules integrated to form the system as a whole.Systems tests are carried out to ensure that the system fulfills all the specified functional requirements which the applicationis expected to meet. In the traditional on-premise model, for system testing of an application, a separate system testenvironment is commissioned, pre-requisite hardware and software infrastructure setup is installed. . Whereas for systemtesting of applications deployed on Azure™ Cloud, test environments need not be commissioned separately. A single cloudcomputation service provisioned on Azure™ comprises of a production and a staging environment and the system test isperformed in the staging region of this service. The diagram below illustrates the steps followed in system testing of anAzure™ application. Infosys – White Paper | 13
    • Figure 9: Steps in system testingSystem testing is to extensively test the functionality of application. Once integration testing of the application is performedon development fabric, it is then hosted on Azure™ staging environment and application functionality is tested again. Manual system test cases are created against high level design of the application (Use cases). Web tests created using Microsoft® Visual Studio web test features to test the functionality of the application on development environment during integration test can be reused. The web tests created on development environment can be utilized by changing the application URL in test cases to the hosted application URL. The path of application will be modified using context parameters and tests can be executed for application hosted in staging deployment region of Azure™ account. Since hosted environment provides restricted privilege access to users, system metrics need to be accessed through the Azure™ diagnostics feature. Proper code instrumentation will be required for trouble shooting of hosted application. Developers will have to write tracing and logging code in the application for writing out information through Azure™ logging APIs for application monitoring and analyzing defects (if any). Similar to investigating on-premise, applications developers have access to the event viewer, system logs, IIS logs etc., to analyze application issues. However in Azure™ such facilities are not directly accessible and hence Azure™ diagnostics API (Microsoft.WindowsAzure.Diagnostics) would have to be relied upon. Users can also connect to an instance of their application deployed on Azure™ using remote desktop, monitor and trouble shoot the running instance.The diagram below shows a comparative view of people, environment and tools required for system testing an on-premiseapplication vis-à-vis that of an Azure™ application.14 | Infosys – White Paper
    • Figure 10: Comparative View for system testingPerformance TestingPerformance testing is conducted to confirm how the system would perform under certain workload. Application behavior istested in terms of various performance aspects under different workload conditions. This includes scalability, reliability,availability and response time of the application, ability to handle load, spikes and throughput. The diagram below lists thesteps followed in performance testing of an Azure™ application. Figure 11: Steps in performance testingIn Azure™ applications, performance testing plays a vital role. Since the infrastructure is abstracted and a standard platformis offered, it becomes essential to gauge the performance of application deployed on the Azure™ platform. The number ofcompute instances and VM size will also affect the application performance. Since customers have to pay for number of Infosys – White Paper | 15
    • specified sized compute instances running at a time, the optimum use of Azure™ compute instances becomes important forcost-effective utilization of the Azure™ owned resources. Performance evaluation of application will help derive the factorswhich can aid in optimizing these resources with respect to cost and performance measures. Azure™ supports dynamicscalability which makes it easy to test application performance at varying scales which is made available on-demand.Procuring the compute instances and releasing it based on the load is a manual process. But once requested for resources itwill be available in no time. It can be automated using some third party applications which continuously monitors the load ofdeployed applications and does the resource management.Performance testing is performed on hosted Azure™ instance because application performance should be tested in anenvironment closely similar to the production environment. Based on the performance requirements, test plan is prepared,performance scripts developed and the approach to execute performance test scripts and capture performance metrics isdecided. Performance metrics are captured with performance test scripts execution and analyzed. Since Azure™ platform is amanaged environment, capturing of performance metrics in not easy. Performance parameters are captured using Azure™diagnostics with support of diagnostic monitor. The diagram below shows a comparative view of people, environment andtools required for performance testing an on-premise application vis-à-vis that of an Azure™ application. Figure 12: Comparative View for performance testingThough performance testing is usually performed after system test, the performance goal should be well defined right fromthe application architecture, design and build.16 | Infosys – White Paper
    • Security TestingSecurity testing involves testing the security at two levels – Platform Security Testing – To test the platform where application is hosted, is not easily vulnerable to security attacks. Application Security Testing – To test that application is not easily vulnerable to security attacks.The Azure™ platform offered by Microsoft® is a managed and standardized environment which is protected by the Azure™fabric controllers, network security and personnel level checks and controls making them highly secure. As per Microsoft®6,” Windows® Azure™ operates in the Microsoft® Global Foundation Services (GFS) infrastructure, portions of which areISO27001-certified. ISO27001 is recognized worldwide as one of the premiere international information securitymanagement standards.” Also Azure™ being offered in a shared model, Microsoft® today does not allow performing anyplatform security tests so as to avoid it affecting other tenants hosted on the same host. However the same cannot be saidabout the application hosted on Azure™. Hence as a part of this phase the focus should primarily be on application securitytesting.Application security testing is to ensure that application provides the right data and operations to the right person. Theconfidentiality of data should not be compromised with the functionalities of application. Whatever functionalities are addedto the application, should allow secure access to data and operations. Exposing the applications over internet involves moreconcerns of security since any flaws within the application can make the application vulnerable to security attacks. Thismakes security testing essential for Azure™ applications. Figure 13: Accessibility for on-premise and Azure™ applicationsFollowing are the security aspects to verify in an application security test – Confidentiality: Protect against the disclosure of information to parties other than the intended recipient Integrity: Ensure received information is correct Authentication: Ensuring received information is from intended known user Authorization: Allow access to intended users Availability: Availability of application in case of denial of service attacks and distributed denial of service attacks Non-repudiation: measures should be taken before the concerned parties deny about an action occurred, a message sent by sender, or received by receiver, etc. Infosys – White Paper | 17
    • Figure 14: Steps in security testingIn traditional security testing almost all security measures can be implemented through firewall. Once the application isdeployed, all requests are passed through the firewall and application maintenance team can restrict the application serverfrom users accessing the application. Using this application, the deploying team has control over the users accessing theapplication. Azure™ is a common platform for deploying the application. The user can access the application hosted oncloud ubiquitously.There are firewalls installed on Azure™ platform but it is managed by Microsoft®. The application developer and deployingteam has the least access to infrastructure and they cannot control the firewalls’ behavior. Application security testing mustbe conducted on hosted Azure™ instance to make sure that the application in production will work as intended. Theprocesses of testing an on-premise and Azure™ application are quite similar but since the application and data do not remainwithin an organization’s cont ol, application security testing becomes a major concern. The diagram below shows a rcomparative view of people, environment and tools required for security testing an on-premise application vis-à-vis anAzure™ application. Figure 15: Comparative View for security testing18 | Infosys – White Paper
    • Compliance & Regulatory TestingCompliance & Regulatory testing is conducted to ensure that an application does not violate any organization, industry orgovernment regulations that might lead to legal and possible financial implications. Compliance and regulatory checkbecomes imperative for the applications hosted on cloud since an application hosted on the public cloud is operated andmanaged by a third party. The cloud service providers should provide the required compliance such as information about datalocation, security audits. Hence compliance and regulatory testing is required to ensure that the systems and processes of thecloud service provider have to be compliant with that as is expected by the domain in which the application would operate in.The system must have supportive documentation and/or certification compliance to be handed over to the regulatory agencies(if applicable). Figure 16: Steps in compliance and regulatory testingBelow is the list of common compliance requirements of IT systems – • Sarbanes-Oxley Act (SOX) 3 - The Sarbanes–Oxley Act of 2002 sets new or enhanced standards for all U.S. public company boards, management and public accounting firms. The legislation affects both the financial side of corporations and the IT departments. • Section 404 of SOX requires management and the external auditor to report on the adequacy of the companys internal control over financial reporting (ICFR). • Section 802(a) of the SOX states, whoever knowingly alters, destroys, mutilates, conceals, covers up, falsifies, or makes a false entry in any record, document, or tangible object with the intent to impede, obstruct, or influence the investigation or proper administration of any matter will be panelized.Similar to SOX act in US other countries also have similar act – • Japan – J-SOX • Canada – Bill 198 • Germany - German Corporate Governance Code • Australia - Corporate Law Economic Reform Program Act 2004 (CLERP-9) • France - Financial Security Law of France • Italy - L262/2005 • South Africa – King Report Infosys – White Paper | 19
    • • Health Insurance Portability and Accountability Act (HIPAA) 4 - Title II of HIPAA, known as the Administrative Simplification (AS) provisions, requires the establishment of national standards for electronic health care transactions. This is intended to help people keep their information private and secure. • ISO/IEC 270014- ISO/IEC 270015 is an Information Security Management System (ISMS) standard which requires that management follows the required measures to keep the information secure in case of security threats and vulnerabilities. • Payment Card Industry Data Security Standard (PCI DSS) 6 is a worldwide information security standard defined by the Payment Card Industry Security Standards Council. The standard was created to help organizations that process card payments prevent credit card fraud through increased controls around data and its exposure to compromise.Most of the compliances require that data security and privacy must be maintained and to ensure this they need to conductsecurity audits in a timely manner. The application must follow the required regulatory compliances which client needs tofollow as per their legislation. The developers would need a test to ensure that it follows required regulatory compliances.The diagram below shows a comparative view of people, environment and tools required for regulatory compliance testing anon-premise application vis-à-vis that of an Azure™ application. Figure 17: Comparative View for compliance and regulatory testingUser Acceptance TestingUser Acceptance testing is similar to system testing but this is performed by the application end-user before accepting thesystem. Acceptance testing for Azure™ applications must be performed on the application deployed on the hosted Azure™instance. The user acceptance test environment should be closely similar to the actual production environment.20 | Infosys – White Paper
    • Figure 18: Steps in user acceptance testingUser acceptance testing approach is quite similar for an on-premise application and Azure™ application. The diagram belowshows a comparative view of people, environment and tools required for user acceptance testing an on-premise applicationvis-à-vis that of an Azure™ application. Figure 19: Comparative View for user acceptance testingOnce all the defects are fixed and system is tested, it is deployed on the production deployment region on Azure™ for users. Infosys – White Paper | 21
    • Benefits realized after application validation with Azure™The platform provided by Microsoft® Azure™ offers the capabilities which ease testing applications on Azure™. Followingare the benefits derived – • Testing environment setup made easy – using Microsoft® Azure™ one can easily set up the kind of environment required to test an application. The application can be easily configured to use specified number of compute or storage instances on Azure™ as well as the specified VM size. Apart from this, various test environments would not require any additional efforts such as system hardening, resource access lock down, etc., to ensure that the platform has to be made ready to be hosting applications. This will also eliminate administrative overhead of managing multiple test environments. • Abstraction for infrastructure – As compared to traditional on-premise applications where setting up test environment would mean procuring server class hardware and installing software licenses, Azure™ provides a platform where a user can directly host applications and not worry about setting up the infrastructure. Infrastructure level details are abstracted and are managed by Microsoft®. Since acquiring test and production environments would be easy and ‘time to market’ of the applicationwould be significantly faster. • Standard environment for all test environments on Azure™ – Microsoft® Azure™ offers a standard platform, therefore all test environments setup on Azure™ will be identical. In traditional testing approach this is a common issue that application works fine in testing but fails in production. One major and common reason is that testing and production environments are not same. In Azure™ the staging and production environments are exactly same. Hardware configuration and software installed are consistent throughout and it helps avoiding this kind of issues. • Effective test environment management – The Azure™ platform is well managed and has robust mechanism for ensuring security, recovery and high availability in place out-of-the-box. Providing these features in the traditional set up will require considerable planning, effort and specialized skills. But with Azure™ the QoS criteria mentioned and demanded by any enterprise application is delivered implicitly as a part of the Azure™ services offerings. • Dynamic scalability – While developing or testing an application if user wants to scale the application, this will require setting up new servers along with possible modifications in the application. On Azure™ scaling is just a matter of modifying number of compute and storage instances required by application. • Control over Operating System Updates and Upgrades – Customers can control which operating system updates to apply to their hosted Azure™ instances. With this customers can ensure that any patches or updates that break their application are not rolled out or the required changes are made in application beforehand. • Cost effective – When setting up environment for a traditional testing approach, acquiring hardware, installing software, maintaining infrastructure will add the same cost as setting up actual production environment. Whereas in Azure™ charges are based on utilization, thus saving a lot of time and manpower and resulting in a cost effective solution.22 | Infosys – White Paper
    • Challenges of testing applications on Azure™Despite the fact that Microsoft® Azure™ platform has got potential for hosting applications without worrying aboutinfrastructure level details, it has certain limitations also. With the abstraction comes the lack of control. Microsoft® providesrestricted access privileges to users on the Azure™ platform. Some of these limitations cause difficulties in performingapplication tests on Azure™. These challenges are described as follows –Organization Policies Are Not Implicit For Hosted ApplicationsIn a traditional approach, all incoming and outgoing network traffic will have to pass the organization firewall and certainsecurity measures and policies are implicit with this. Organizations apply many security policies through firewalls like thekind of response data than can be sent outside or the type of incoming requests. When an application is hosted on Azure™, itis no longer in the confinement of the organization, so enforcing the organization policies through firewalls is not possible.Testers will have to perform proper testing to ensure these otherwise implicit parameters are validated and function asexpected.Challenge Workaround/ Supportive Tools/ RecommendationFirewall restriction Though Microsoft® Azure™ provides a highly secure environment, any other security aspect such as imposing site access restrictions, secured transfer of confidential enterprise data, etc., will have to be taken care of by the application and testers will need to test it.Policy enforcement Organization policies can’t be enforced implicitly since applications run out of the boundaries of an organization’s IT system. Application will have to take care of and the testers will need to test it. Table 3: Challenges in enforcing organization policies and their workaroundsDevelopment ChallengesFor an on-premise application, analyzing a bug, debugging and trouble shooting is comparatively easy with support ofnumerous Microsoft® tools, open source tools and third party tools available today. Since Microsoft® Azure™ is a newplatform; this kind of support is currently not available. SQL server provides a proficient way to access database, run profiler,check indexing, database tuning, checking query execution plan etc. SQL Azure™ doesn’t support that level of featurestoday.Azure™ as of today has very limited tools available that can support testing Azure™ applications. Generation of test data,capturing results, capturing performance matrices, analyzing test results, etc., are very easy and convenient in on-premiseapplications using various tools. Unavailability of such tools hampers the overall productivity and amount of efforts bydevelopers and testers. Infosys – White Paper | 23
    • Challenge Workaround/ Supportive Tools/ Recommendation Less support of debugging Windows® Azure™ diagnostics feature can be used for code instrumentation and troubleshooting of Azure™ applications. Developers can add additional functionality in the application to view system event logs, IIS logs and performance monitor logs using Windows® Azure™ Diagnostics feature. Limited tools available for Platform level security is in the control of Microsoft®. Application level security testing security measures such as role and membership based access will have to be taken care of by the application developers. Limited tools available for Windows® Azure™ diagnostics feature along with Diagnostic Monitor performance testing (typeperf utility) of Windows® Azure™ can be leveraged to collect performance parameters of a hosted application. Database access and analysis SQL Azure™ doesn’t support full fledged features of SQL serve e.g. profiler, r. query execution plan, etc. Apart from SQL 2008 R2, Microsoft® provides an online database management tool with project name Houston for DB operations on SQL Azure™. Test data creation Test data can be created manually or programmatically for Azure™ storage and SQL Azure™; inbuilt tools are not available. Microsoft® SQL Azure™ Data Sync tool can be used to replicate database from SQL server to SQL Azure™. Interop APIs The applications using interop APIs are not supported on Azure™ since these require APIs to be registered. This kind of scenario cannot be supported on Azure™. Such applications will have to look at hybrid approaches (mix of on- premise and Azure™ deployment) to build applications on Azure™. Table 4: Development challenges and their workarounds Troubleshooting as a Tester Debugging and trouble shooting an Azure™ application in the development environment works the same way as that of a non-Azure™ application. One can attach debugger to the executing code and debug it in Visual Studio. Or one can use Trace.Write() to get trace related information, If an application hosted on Azure™ platform has to be debugged, one can use below methods – 1. Use IntelliTrace feature of Visual Studio 2010 a. Enable IntelliTrace for roles in Azure™ application b. Publish application on Azure™ subscription c. After deployment, open server explorer and expand add current deployment d. Open IntelliTrace in Windows® Azure™ Compute for deployed application e. View IntelliTrace logs and call stack f. Disable IntelliTrace once debugging is done and before deploying the Azure™ application in production. 2. Use Windows® Azure™ Diagnostics to capture logs24 | Infosys – White Paper
    • a. Write traces in application or modify application configuration (.cscfg) to capture performance counters, Windows® event logs, IIS logs, application crash dumps, infrastructure logs etc. b. Logs will be written into blob and table storage c. Use tools such as Azure™ Storage Explorer, Windows® Azure™ Service Management CmdLets, Windows® Azure™ MMC, etc. to view logs 3. Use Remote Desktop a. Before publishing the application, configure remote desktop connections and enable connections for all roles b. Provide certificate, user name and password for connection c. Go to Azure™ management portal, enable remote access and connect to a role d. Download .rdp file and open it. Provide user name, password and connect to the role with remote access.ConnectivityInternet availability is the basic requirement for developing and testing an application on Azure™. If Internet is down, testerswon’t be able to perform cloud testing. Hence, it is mandatory to have internet connectivity all the time to carry out testing onthe cloud.There are several services of Microsoft® Azure™ such as Windows® Azure™ Platform for hosted application and Azure™storage (Blob, Queue, and Table), AppFabric service bus, Windows® Identity Framework, Access Control Service, SQLAzure™ etc., which an application may be using as building blocks. If an Azure™ application is using any of the abovementioned services then the connectivity to those services and availability of the services needs to be checked beforeinitiating tests. Say in enterprises, owing to security threats access to several external sites are controlled and more oftenblocked. They have a locked down environment where many of the ports are also disabled. If an application uses AppFabricservice bus, then, ensuring that required ports to connect to the service are open is necessary. Apart from this the requestsgoing to and responses coming from application hosted on Azure™ will pass through the organization firewall. Testers willhave to take care of this aspect also. Corresponding ports will have to be opened and firewall policies might have to beamended for sending requests and receiving responses from an organization.Challenge Workaround/ Supportive Tools/ RecommendationInternet connectivity The tester needs to ensure the proper internet connectivity for smooth testing operations.Connectivity to Microsoft® Connectivity to the Azure™ blocks e.g. Azure™ compute, Azure™ storageAzure™ services (blob, queue, table), SQL Azure™, AppFabric service bus, Access control service etc. used by application needs to be ensured. It may require opening some of the ports and few modifications in the firewall policies. Table 5: Challenges in connecting to Azure™ services from an organization and their workaround Infosys – White Paper | 25
    • SummaryAzure™ cloud computing offerings in the PaaS space offers developers a platform to not only host and run applications butalso a ready-to-use test bed which can help projects reduce the overheads with setting up any world class test facility whichhas been discussed in this paper and also further explained later in this section. On the application validation front, Azure™based projects will benefit in terms of reduced time, effort and cost involved in setting up the various test environments.However there are challenges on the other end, more so since the technology is new and the space still lacks tools andutilities for testing, debugging and troubleshooting and which we have highlighted in this paper along with strategies tomitigate them. The figure shown below summarizes the pros and cons of testing an application on Azure™ which developersneed to be aware of. Figure 20: Pros and cons of application testing on Azure™Further to support our views on Azure™ capabilities of accelerating the testing lifecycle, the graph drawn below show arelative comparison of total cost required for testing an on-premise application vis-à-vis to testing an Azure™ application.The graph is illustrative and may differ for different projects.26 | Infosys – White Paper
    • Figure 21: Comparison of total cost in testing an on-premise vis-à-vis Azure™ applicationThe graph above shows the capital and operational expenditure incurred in the test life cycle of an application deployed on-premise as well as on Azure™ . During unit testing and integration testing the costs will be almost identical since both thesestages are performed on-premise with similar infrastructure and software requirements. Moving on to the later stages of anytesting lifecycle, as the need for the test environments become more and more specialized, the capital and operationalexpenses will increase. This is primarily owing to the high infrastructure and software requirements of these environmentsfollowed by specialized skills required to run and maintain it. Whereas in Azure™ , since these factors are abstracted by theplatform, only operational cost would incur which would be more in line to the demands of the test scenarios. Operationalexpenses will occur only for the duration an application is deployed on Azure™. And as we can see, no charges would accruethe minute the application instance is deleted which would be as soon as the tests have been completed. This pay-as-you-usemodel is a very useful option available for applications deployed on Azure™ and needs special attention to be paid by thedevelopers to avoid be charged while no tests are being run. Additionally, since no extra efforts are required for setting uptest environments, the testing life cycle span will reduce and an application can go into production early, thus reducing theoverall time to market.These operational benefits are a big driver for enterprises that are looking at eliminating their non-productive investments inIT. Test beds being one such area, as discussed in this paper, show a huge potential with developing applications on Azure™to help enterprises achieve this goal. Infosys – White Paper | 27
    • Reference Description URL http://en.wikipedia.org/wiki/Systems_Development_Life_Cycle [1] Wikipedia http://en.wikipedia.org/wiki/Security_testing [2] Windows® Azure™ http://www.microsoft.com/Windows®Azure™/whitepapers/papers/default.aspx Security Overview http://en.wikipedia.org/wiki/Sarbanes%E2%80%93Oxley_Act [3] SOX Compliance http://searchcio.techtarget.com/sDefinition/0,,sid182_gci920030,00.html [4] HIPPA http://en.wikipedia.org/wiki/HIPPA [5] ISO/IEC 27001 http://en.wikipedia.org/wiki/ISO/IEC_27001 [6] PCI Data Security http://en.wikipedia.org/wiki/Payment_Card_Industry_Data_Security_Standard Standard MSDN http://msdn.microsoft.com/en-us/library/microsoft.Windows®Azure™.diagnostics.aspx Testing in MSDN http://msdn.microsoft.com/en-us/library/dd286680(VS.100).aspx Unit testing http://msdn.microsoft.com/en-us/library/ms182524.aspx http://msdn.microsoft.com/en-us/library/aa337591.aspx Web test http://msdn.microsoft.com/en-us/library/ms182539.aspx Validation rule http://msdn.microsoft.com/en-us/library/ms404670.aspx http://www.Azure™support.com/2009/12/sql-Azure™-firewall-tutorial/ SQL Azure™ Firewall http://msdn.microsoft.com/en-us/library/ee621783.aspx Azure™ Storage Explorer http://Azure™ storageexplorer.codeplex.com/ OS versioning on Windows® http://blogs.msdn.com/Windows®Azure™ /archive/2010/01/11/operating -system- Azure™ versioning-in-Windows®- Azure™ .aspx Microsoft® SQL Azure™ Data http://www.microsoft.com/Windows®Azure™ /sql Azure™ /datasync/ Sync Azure™ Application http://msdn.microsoft.com/en-us/library/ff966484.aspx Troubleshooting Remote Desktop on Azure™ http://msdn.microsoft.com/en-us/library/gg443832.aspx application28 | Infosys – White Paper
    • AuthorsSidharth Subhash Ghag (Sidharth_Ghag@infosys.com) is a Senior Technology Architect with the Microsoft® TechnologyCenter (MTC) in Infosys. With over twelve years of industry experience, he currently leads solutions in Microsoft®Technologies in the area of Cloud Computing. In the past he has also worked in the areas of SOA and service-enablingmainframe systems. He has been instrumental in helping Infosys clients with the service orientation of their legacymainframe systems. Currently he helps customers adopt Cloud computing within their Enterprise. He has authored papers onCloud computing and service-enabling mainframe systems. Sidharth blogs at http://www.infosysblogs.com/cloudcomputingDivya Sharma (Divya_Sharma01@infosys.com), Technology Analyst with Microsoft® Technology Centre (MTC), Infosysand has more than 4 years of experience of developing and designing solutions in Microsoft® Technologies such asWindows® applications, ASP.net, web services, WCF services etc. Her current work includes exploring Windows® Azure™platform and implementing applications using technology stack of Azure™ and deploying those on Azure™ platform. Divyablogs at http://www.infosysblogs.com/cloudcomputingTrupti Sarang (Trupti_Sarang@infosys.com), Senior Systems Engineer with Microsoft® Technology Center (MTC) inInfosys. She has nearly 3 years of industry experience on Microsoft® .NET. She currently works on implementing anddeploying projects on the Windows® Azure™ platform.AcknowledgementAuthors would like to acknowledge Vijayanathan Naganathan, Senior Technology Architect, Independent ValidationServices (IVS), Infosys Technologies Ltd. and Bhavin Jayantilal Raichura, Senior Technology Architect, ManufacturingIBU, Infosys Technologies Ltd. for their help in reviewing this paper.