Published on

  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide


  1. 1. March 2009 Volume 1 SOFTWARE TESTING New software testing technologies bring new challenges • CHAPTERTEST VIRTUAL 1 LABS MAKE MOST OF CLOUD • CHAPTERRIAS TESTING 2 FOR SECURITY HOLES
  2. 2. EDITOR’S LETTER 1 Testing challenges: The cloud and RIAs p WELCOME TO THE inaugural edition alization technologies as part of their of’s application development process, a Software Testing E-zine. Throughout according to a recent SearchSoftware- EDITOR’S LETTER 2009, our e-zine’s writers—all soft- Quality .com survey. ware testing experts—will deliver tac- Frye explains how virtualization tical how-to and in-depth trend and reduces test labs’ hardware costs, a strategy articles on key issues includ- increases server utilization rates, sim- CHAPTER 1 VIRTUAL ing regression testing, service compo- plifies systems configuration, speeds TEST LABS MAKE MOST nent architecture, mobile application setup and teardown of test configura- OF CLOUD testing, exploratory testing, SOA, tions and more. Users of cloud-based agile development, virtualization, development services can even cloud computing and test manage- bypass many traditional software a ment. configuration tasks. CHAPTER 2 TESTING RIAs This first issue covers two new In “Testing RIAs for security holes,” FOR SECURITY HOLES technology areas—virtualization and information security expert Kevin rich Internet applications (RIAs)— Beaver makes it clear that new Web that are changing software testing 2.0 and Ajax technologies haven’t approaches and presenting new erased Web 1.0 security shortcom- challenges. Our writers, Colleen Frye ings. Sure, he writes, “there are many and Kevin Beaver, each bring over 20 things to exploit with rich Internet years of software industry experience applications,” but he adds that he’s to play in their analysis of these had few security problems with them issues. and explains why. Check out his advice Industry veteran Colleen Frye opens on plugging security holes using auto- the e-zine with a report called “Virtual mated scanning tools and when to test labs make most of cloud.” The use manual analysis on RIAs. I insights and technology updates in this article are timely, as the day when JAN STAFFORD most test and development labs are Executive Editor virtualized is not far off. Already, 44% of software developers are testing in virtual environments and using virtu- 2 SOFTWARE TESTING MARCH 2009
  3. 3. Let Our Global Software Testing Community Help You Find Your Bugs ...So Your Customers Don't Have To. uTest helps QA teams achieve maximum testing coverage and launch higher quality web, mobile and desktop apps. Compliment your in-house QA team by tapping into our community of 14,000+ professional testers from 150 countries around the world. Build a virtual testing team based on location, language, application type, operating system and browser, and only pay for the bugs you approve. Discover what QA teams from companies like Intuit, Thompson Reuters, Internet Brands, Second Rotation and Seesmic already know: uTest helps QA teams meet their launch schedules and release higher quality: » Web Applications » Mobile Applications » Desktop Applications For more information, check out or call 1.800.445.3914 Software Testing Community
  4. 4. CHAPTER 1 1 Virtual test labs make most of cloud QA and testing organizations are embracing the powers of virtualization and cloud-based computing to perform simpler testing at lower costs. BY COLLEEN FRYE a EDITOR’S LETTER p VIRTUALIZED TEST LABS can make she said. “This means that software a it faster and less expensive to set up can languish without being tested due CHAPTER 1 VIRTUAL and tear down test configurations, to poor resource allocation and man- TEST LABS MAKE MOST better utilize resources, and help to agement.” OF CLOUD boost overall software quality. And This bottleneck can be costly, Bal- now cloud computing brings the lou said—an issue that becomes even promise of even less infrastructure more critical in a down economy. “Vir- a to worry about, which can help to fur- tual test lab management can help CHAPTER 2 TESTING RIAs ther alleviate quality assurance (QA) address these kinds of issues, as well FOR SECURITY HOLES bottlenecks due to resource issues. as cutting the cost of infrastructure “QA and testing organizations are by augmenting or replacing physical increasingly embracing virtual test systems.” labs due to the efficiency and cost With the emergence of cloud-based savings that can result,” said Melinda- offerings, the benefits of virtualization Carol Ballou, program director for can be more immediate, Ballou said, application lifecycle management at “because you don’t have to configure IDC. According to Ballou, one of the the software.” barriers to QA is limited access to— and poor management of—physical infrastructure for test labs, as well as STARTUP GAINS CAPACITY, the effort involved in setting up and SAVINGS WITH VIRTUAL TEST managing systems configurations for LAB SOLUTION the labs. For startup Apptio, a provider of “Groups tend to hoard the physical on-demand IT cost transparency resources so that they will have solutions, a virtualized infrastructure access to them when needed, even provided both affordability and the when there is no immediate demand,” capacity to test more broadly. 4 SOFTWARE TESTING MARCH 2009
  5. 5. According to Colin Henry, senior soft- Seattle-based Skytap describes ware engineer at the Bellevue, Wash.- itself as “serving up virtual machines based company, Apptio is developing over the Internet.” According to CEO a data integration add-on for its pri- Scott Roza, “There is no infrastruc- mary product; the add-on will aggre- ture, no upfront expenditures, and it’s gate data from disparate data delivered 100% as a service.” Skytap sources. Virtual Lab “is a multitenant model,” “We realized this portion of the product covered a broad range of systems—ERP, database, etc.—and “Being a startup, we the setup/teardown time for that don’t want to spend the would be costly,” Henry explained. world on getting three a “We knew we needed a virtual infra- EDITOR’S LETTER structure. Being a startup, we don’t racks of servers or a want to spend the world on getting huge box to throw five three racks of servers or a huge box or six VMs on. I knew a to throw five or six [virtual machines] we wanted to go for CHAPTER 1 VIRTUAL on. I knew we wanted to go for virtual- TEST LABS ized space hosted on the cloud.” virtualized space MAKE MOST OF CLOUD Henry said Apptio had performed hosted on the cloud.” manual testing on its primary product —COLIN HENRY and did not have a traditional testing Senior Software Engineer, Apptio a environment, “so we were starting CHAPTER 2 TESTING RIAs from scratch.” FOR SECURITY HOLES The company researched on- he said. “All you need to access it is premise and hosted virtual test labs, a browser. With the browser you can “but any kind of managed service on get access to computing resources; a local machine takes a lot of your you’re not limited by the number of administration time,” he said. Provi- VMs you can run or the number of sioning a box can take three to four users you can have accessing [the hours, according to Henry, and if environment].” you’re adding application infrastruc- While QA and testing organizations ture on top of that, it could take up have been early adopters of virtualiza- to eight. So Apptio decided that tion, Roza said, “very few QA organi- newcomer Skytap Inc. with its Skytap zations are all virtual” because “large Virtual Lab product fit the bill. The database servers and databases his- result? torically don’t work all that well in a “Going from 12 hours to five min- virtualized environment.” He said Sky- utes,” Henry said. “That’s a huge tap offers a hybrid model. “We have increase in productivity, because secure bidirectional VPN technology, you’re not slogged down in installing so you can build part of your lab in the software.” cloud and connect back to your own 5 SOFTWARE TESTING MARCH 2009
  6. 6. data center. The hybrid model can merly VQMS) automates the deploy- take advantage of the cloud as an ment, configuration and teardown of extension of an in-house lab.” complex software environments. Mal- Roza likens the Skytap service to colm said to get started with virtual- a family cell phone plan. Henry said ization, QA organizations need the Apptio has a plan for about 2,000 physical infrastructure to support a hours per month. “We don’t have to virtual lab, a hypervisor platform like keep the infrastructure running full- Microsoft’s or VMware’s, and test time,” Henry said, “so we’re not using automation tools. Surgient integrates unnecessary hours of their time. with HP’s quality management suite When you’re done, you’re done. To and IBM Rational. “The testing tools get rid of a VM you’re not using any- will make the calls to the Surgient a more, you just delete and within a few platform to automate test configura- EDITOR’S LETTER minutes it’s gone from the virtual lab. tions,” he explained. You can manage it relatively easily.” Surgient customers can choose a Roza added that pioneer Software hosted offering, but most install on- a as a Service (SaaS) companies like premise, Malcolm said, and build CHAPTER 1 VIRTUAL broke down barriers what he called an “internal or private TEST LABS MAKE MOST of concern that SaaS might not be cloud.” OF CLOUD secure. For Henry, with their test cases, “we’re not dealing with sensi- tive data, so in that regard it’s not VIRTUAL TEST LABS a much of an issue. For long-lead VMs IMPROVE UTILIZATION CHAPTER 2 TESTING RIAs left on for monthly tests we have it Insuresoft, a software provider for FOR SECURITY HOLES set up over the VPN with Skytap.” the property and casualty insurance industry based in University Park, Ill., decided to install Surgient’s QA/Test AUTOMATION EASES Solution on premise. “We wanted the TEST CONFIGURATION PROCESS flexibility to do what we wanted with But testing organizations don’t need our images and be able to trouble- a cloud-based service to reap benefits shoot ourselves and go through the from virtualization, particularly for process of learning about virtualiza- setting up and tearing down test con- tion,” said Hemanth Guttikonda, figurations. “Automated testing took a Insuresoft’s QA manager. “We had lot of labor out of the process [before some expertise in-house to do some virtualization], but there was still a lot of these things; also we’re not a large labor involved in setting up and tear- enough organization to consider ing down the configurations,” said hosting.” Dave Malcolm, chief technology offi- Guttikonda said Insuresoft turned cer of Surgient Inc., an early pioneer to virtualization to help with testing in the virtualization space. consistency first and foremost, and Surgient’s QA/Test Solution (for- the company ultimately benefited 6 SOFTWARE TESTING MARCH 2009
  7. 7. from better resource utilization as way up.” well. He said the testing lab prior to Consistency and quality are up as Surgient was somewhat ad hoc, and well, he said. “Now people see the they would set up and tear down soft- benefits of how quickly we can get the ware as new versions of their Dia- environment available and ready for mond product, a hosted application, testing. In the past it sometimes took needed to be tested. “We have 15 to 20 companies we service and have physical hardware “People are starting for each company. We had about to understand that 30 to 40 physical servers, which mir- rored our production environment.” [with Surgient] we’re a The process of installing and unin- actually testing in EDITOR’S LETTER stalling software to test for each a true production customer was tedious, he said. environment.” While some testing was done in a the lab, Guttikonda said other testing —HEMANTH GUTTIKONDA CHAPTER 1 QA Manager, Insuresoft VIRTUAL was done on individuals’ physical TEST LABS MAKE MOST machines, which he said wasn’t mim- OF CLOUD icking what the product environment a day or more to get the testing envi- looked like. “The technology we were ronment set up, so people would developing wasn’t being fully tested.” abandon the idea of a central environ- a Insuresoft was already utilizing ment.” Once the organization started CHAPTER 2 TESTING RIAs automated testing tools from HP and using Surgient, “we started noticing FOR SECURITY HOLES saw Surgient, which supports HP, as that we were finding more defects.” a logical choice, he said. These were defects that weren’t Guttikonda said the company’s test showing up when testing on local lab has been reduced from 30 to 10 machines, he said. “People are start- servers. “In essence, we trimmed our ing to understand that [with Surgient] physical servers by over 60%, and we’re actually testing in a true pro- we’re getting more utilization out of duction environment.” the 10 servers than with the 30 in the In addition to getting used to the past,” he said. “But it really means cultural change, Insuresoft had to much more—it’s the ability to tear invest in more powerful servers than down the environment and reuse the what Guttikonda called the “entry- servers.” Previously, he said, each level” ones they had in order to get server was dedicated to one customer the performance benefits of Surgient. and couldn’t be used for any other However, Insuresoft is already seeing purpose. “Now we can use [the serv- the cost savings of better resource er] for one company, and when we’re utilization. “These servers are about done, tear down and deploy it for $5,000, but in the past we would another, so hardware utilization is have had to purchase 15 to 20 desk- 7 SOFTWARE TESTING MARCH 2009
  8. 8. top workstations at $30,000 a piece. cloud computing. There are going to Now we can serve the same number be some changes to processes, and of employees with a lot less hard- you have to embrace that and look at ware.” the upside of the cost savings versus having 10 servers next to my desk. We find a lot of VPs of IT say it’s a threat CLOUD COMPUTING: to [their] way of life, in terms of head- THE NEW HYPERVISOR? count, budget, etc. We try to do as This type of capital expense will be one of the effects of cloud computing, according to Ravi Gururaj, CTO of The cloud will virtualization provider VMLogix Inc. enable QA to extend a in Palo Alto, Calif. “You’ll go from a its internal resources EDITOR’S LETTER CapEx to an OpEx,” he said, as organ- izations are billed for usage. “The to provision from cloud is really the new hypervisor. the cloud the amount a Instead of an ESX [VMware hypervi- of infrastructure CHAPTER 1 VIRTUAL sor] box, you will have a cloud box. required. TEST LABS MAKE MOST The cloud will give you the ability to OF CLOUD provision infrastructure on demand, but the tools themselves will be the much as we can to make it feel like a same, so the tools you’re using on- lab in your data center, with better a premise will be available in the cloud.” collaboration.” CHAPTER 2 TESTING RIAs Another benefit, he said, is that IDC’s Ballou said organizations also FOR SECURITY HOLES “the cloud can scale when you have need to examine how the vendor has a bursting situation.” For instance, structured its cloud strategy. First and in the final stages of QA, scalability foremost, she said, organizations may need to be tested. The cloud need to evaluate the stability and reli- will enable QA to extend its internal ability of the vendor and the service. resources to provision from the cloud For example, she said, is there a plan the amount of infrastructure required. for disaster recovery? Secondly, VMLogix is in the process of cloud- organizations need to look at how the enabling its products, Gururaj said. pricing model is structured and deter- The first stage will enable customers mine whether it is cost-effective: “Do that have VMLogix on premise to I have the resources to run this inter- burst to the cloud when necessary, nally, versus do I have the time? You and that will be followed by a pure have to balance that.” I cloud-based offering, he said. For organizations considering a cloud-based solution, Skytap’s Roza COLLEEN FRYE is a freelance writer and editor in Bridgewater, Mass. said it will require a mindshift. “We’re She has been covering the IT industry in the early adopter phase around for more than 20 years. 8 SOFTWARE TESTING MARCH 2009
  9. 9. CHAPTER 2 1 Testing RIAs for security holes Rich Internet applications bring their own set of vulnera- bilities to software testing, but you can address security problems by looking at apps from the hacker’s point of view. BY KEVIN BEAVER a EDITOR’S LETTER a p WE’RE NOT OUT of the woods yet THE SAME OLD ISSUES CHAPTER 1 VIRTUAL with the old-school Web flaws. One thing has not changed with these TEST LABS MAKE MOST In the days of Web 1.0, Web appli- new client-centric RIA technologies: OF CLOUD cations were chock-full of flat HTML, how the bad guys work. They’re still CGI and the like. Anyone armed with examining your Web applications with modest vulnerability scanners and a malicious eye, seeing what they can a some technical know-how could find exploit in the simplest manner to pro- CHAPTER 2 TESTING RIAs practically every Web security hole vide the highest payoff for ill-gotten FOR SECURITY HOLES left unplugged. gains. That formula will never change, Skip ahead to Web 2.0, Ajax and and that’s why you have to use the client-side processing, a new world of same approach if you’re going to find rich Internet applications (RIAs) that the right Web vulnerabilities. have their own slew of vulnerabilities So whether you’re a developer, a and, of course, new tools and skill sets tester or a security professional, your to evaluate and learn to use. ultimate goal is finding and fixing In my work, I still see more of the Web security flaws before the bad old and less of the new, but things are guys exploit them. You have to step changing pretty quickly. I do see a lot into the attacker’s mindset, looking at of development shops revamping your applications from that point of their sites and apps, implementing view. This requires using an ethical Ajax like it’s going out of style. If hacking approach, which includes you’re serious about Web security, it’s these processes: probably time to take things up a cou- ple of notches. Get on board with the 1 Mapping the application ins and outs of this stuff so you don’t 2 Scanning for common get burned by hackers. vulnerabilities 10 SOFTWARE TESTING MARCH 2009
  10. 10. 3 Exploiting the vulnerabilities scanning tool against your rich Inter- 4 Penetrating the system net applications. A good way to get things kicked off is by mapping the In each step, apply the right tools applications and finding the low- and techniques as both an untrust- hanging fruit. Automated scanners worthy outsider and a trusted user of are especially good at throwing thou- the application. Delving into your sands of similar requests at applica- Web application in this way, you go tions to see how they hold up—very beyond typical black box security tedious work we just can’t do manual- testing and checklist audits and actu- ly. When using vulnerability scanners, ally see what else can be exploited in I’ve found it to be both important and the application. beneficial to use the latest version of a reputable commercial tools. Commer- EDITOR’S LETTER cial tools tend to find more of the FINDING THE FLAWS important flaws—especially client- When getting started with testing, I side flaws such as cross-site scripting a recommend running an automated and cross-site request forgery—and CHAPTER 1 VIRTUAL TEST LABS MAKE MOST OF CLOUD a 1 Wary developers reduce CHAPTER 2 TESTING RIAs FOR SECURITY Web 2.0 security risks HOLES AJAX AND WEB 2.0 technologies have gotten a bad reputation for being riddled with security holes. The general consensus is that client-side code is more accessible to the bad guys and thus more easily exploited. While there are many things to exploit with rich Internet applications, I’ve had few security problems with them, and here’s why. Most RIAs I’ve encountered have been well-coded with only a minimal set of vulnerabilities, especially compared to older Web technologies. These RIAs aren’t just more secure because they’re using new technologies. Instead, applications are being developed securely up front, with good support from management. It’s obvi- ous that Web 2.0 developers have learned a lesson from Web 1.0 security disasters. That’s half the battle. Project managers have learned from past mistakes, too. More and more, I see managers asking consultants and their own teams for extra security checks of their applications. That’s going to result in more secure systems. I expect these double-checks will be a com- mon practice, as more developers use Web 2.0 technologies and systems become more complex. —K.B. 11 SOFTWARE TESTING MARCH 2009
  11. 11. they’re also continually updated with rience overall, it seems the client support readily available in most intervention required in rich Internet cases. applications is sometimes more than There is one problem with automat- automated scanners can handle. In ed scanners. When testing RIAs you fact, my experience has shown that can’t rely on Web vulnerability scan- quite often vulnerability scanners ners to find every weakness. This is won’t know where to go and what to true with previous generation web- do within the application, especially sites and applications, but it’s even when logged in as a trusted user. more important now with client-side Sometimes they just hang up alto- code. That’s been the biggest letdown gether. for me in testing RIAs with traditional Automation tools have gotten bet- a tools. Although I’ve had a good expe- ter over the past year, and I’m confi- EDITOR’S LETTER a CHAPTER 1 VIRTUAL TEST LABS MAKE MOST 1 Set these must-have OF CLOUD controls on RIAs AT A MINIMUM, ensure your RIAs have the following controls in place: a CHAPTER 2 b Reasonable input validation so the application only accepts TESTING RIAs what is expected. FOR SECURITY HOLES b Strong password requirements beyond easy-to-guess words and number combinations. b Intruder lockout that disables accounts after five to 10 failed login attempts. b Generic errors displayed during failed authentications, improper data submissions, etc. b Strong multifactor authentication that can’t be easily manipulated. b Strong session keys and nonpersistent cookies. b Minimal use of hidden fields and related client-side data that’s easy to abuse. b SSL usage throughout the entire site. These controls won’t guarantee Web security, but they create a foundation upon which good security and solid risk management can be built. —K.B. 12 SOFTWARE TESTING MARCH 2009
  12. 12. dent the vendors will make more I With the bigger findings, do improvements soon. But, even with all certain Web browsers behave differ- the scanning tools in the world, they ently than others? While it’s next to still only represent half of the Web impossible to test all versions of all security testing equation. This is browsers, the reality is that users will where manual analysis comes into employ their own browser version play. whether you support it or not. I cannot stress enough that manual USING MANUAL ANALYSIS analysis has to be a part of the Web Some things can only be tested for security equation. The more client- and exploited using manual analysis: side code we have, the more manual a testing needs to be done. EDITOR’S LETTER I Can forms be manipulated so that junk data is appended to a default user selection and sent back to the STAYING SECURE a server? Web 2.0 has introduced what’s CHAPTER 1 I Can client-side form data be arguably the equivalent of open VIRTUAL TEST LABS MAKE MOST manipulated altogether using a Web source software into the World Wide OF CLOUD proxy tool? Web. Rich Internet applications are I Is sensitive information such as often free and open, and the possibili- the username and password being ties are endless. With Web-centric a passed back and forth between the malware generation toolkits such as CHAPTER 2 TESTING RIAs client and server in every HTTP MPack and Webattacker, there’s even FOR SECURITY HOLES request? more reason to be concerned with I How are errors handled? Are these new technologies. detailed messages being generated by Given this new frontier of Web the server or is client code generating interaction, how do you ensure your pop-up messages? Can this be further RIAs are not left vulnerable to both abused? current and forthcoming attacks? You I Is random JavaScript or XML code can take the easy route and invest in a (e.g., via RSS) blindly published and Web application firewall (WAF). Just reflected back to the user with no set it up and forget about it. Unfortu- complaints or errors? nately, that would be like using duct I Are persistent cookies being tape to hold your house together, all stored on the client, which can lead the while knowing it was built on a to further abuse? shaky frame and foundation. It may I Is autocomplete enabled for form work for now, but it’s not the right fields, allowing the Web browser to way to go about things long-term. store potentially sensitive information Don’t get me wrong. WAFs have on the local computer for others to their place, but only after your securi- access? ty policies and development stan- 13 SOFTWARE TESTING MARCH 2009
  13. 13. dards have been solidified and your pundits scream about how scary the codebase is relatively secure. This Web 2.0 world is, but they’d have less means, at the very least, your Web to scream about if these basics were applications need to have reasonable addressed. At the end of the day, rich Internet applications are nice for users and Vendors, analysts great for business. If you focus on using sound development and QA and certain pundits practices—combined with periodic, scream about how consistent and proper security test- scary the Web 2.0 ing—you’ll set your business up for world is, but they’d success, even when the next new a have less to scream wave comes crashing in. I EDITOR’S LETTER about if these basics were addressed. KEVIN BEAVER is an independent a information security consultant, keynote CHAPTER 1 VIRTUAL speaker and expert witness with Atlanta- TEST LABS input filtering that keeps the junk based Principle Logic LLC. He has over MAKE MOST 20 years of experience in the industry OF CLOUD out, solid login mechanisms that and specializes in performing independent informa- aren’t easily broken, and application tion security assessments revolving around compli- logic that can’t be easily abused. In ance and information risk management. Kevin has authored or co-authored seven books on information a addition, you need to patch and hard- security, including the ethical hacking books Hacking CHAPTER 2 TESTING RIAs en Web servers running underneath. For Dummies and Hacking Wireless Networks For Dum- FOR SECURITY RIAs or not, I still see these silly, inex- mies (Wiley). He’s also the creator of the Security On HOLES Wheels information security audio books and blog, cusable oversights in Web-based sys- providing security learning for IT professionals on tems. Vendors, analysts and certain the go. 14 SOFTWARE TESTING MARCH 2009
  14. 14. ABOUT OUR SPONSORS q Surgient Platform Overview q Insuresoft Webinar q Sisters of Mercy Case Study About Surgient: Headquartered in Austin, Texas, Surgient is the market leader in self-service virtualization automation and lab management. The company’s flagship, award-winning product, the Surgient Virtual Automation Platform,™ is a powerful, flexible and mature solution that optimizes IT’s ability to support critical business initiatives, effectively manage diverse virtual resources and eliminate physical server and VM sprawl. Using the Surgient Virtual Automa- tion Platform,™ world-class companies including IBM, Merck, Raymond James, HP, Halliburton, EMC, CA, Iron Mountain, Target, GE, SAP, Microsoft, Siemens, Intuit and others are accelerating their growth and profitability by automating virtual infrastructure in support of their business initiatives. Software Testing Community q Ten Things Every QA Manager Should Know About Building & Leading a World-Class Testing Team q Six Benefits of Using Community-Driven Testing to Compliment Your In-House QA Efforts q Eight Essentials of Crowdsourcing the Design, Development and Testing Of Your Application q Watch A Free Online Demo of uTest About uTest: uTest is the world’s largest marketplace for software testing serv- ices. The company enables in-house QA teams to maximize real-world testing coverage through its community of 14,000+ professional testers from 150 countries around the world. More than 400 QA and development organizations—including those from Intuit, Thompson Reuters, Internet Brands and Gazelle—have signed up with uTest to get their web, mobile and desktop apps tested. And because uTest is on-demand, customers pay only for the bugs that they approve. 15 SOFTWARE TESTING MARCH 2009