Automated softwaretestingmagazine december2011

1,114 views

Published on

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
1,114
On SlideShare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
81
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Automated softwaretestingmagazine december2011

  1. 1. An AUTOMATED TESTING INSTITUTE Publication - www.automatedtestinginstitute.comAutomated .......S T A oftware esting December 2011 $8.95 MAGAZINE Automated M obile esting Cost Effective Test Management Develop a HomeGrown Managment Solution Surviving Obfuscation How To Automate Obfuscated Code Test Management Getting Crowded: Crowdsourcing Inspires New Tool
  2. 2. A Sotera Defense Solutions Company If you’re looking to thrive in a diverse and vibrant professional team, we’re looking for you! Submit your resumé to staffing@spt-inc.com today! Through the SPT PromiseTM, we’ve made a commitment to value our employees, to be understanding and supportive of individual needs and to provide opportunities for career growth. We value and appreciate our employees for the work they do for us as a company and for the clients’ mission. SPT partners with client organizations to design, develop, implement and deploy high-end, custom, technology-based solutions by integrating existing and emerging concepts. Positions are available for professionals looking for exciting and rewarding challenges with significant impact to ongoing missions. Openings include: • Software Engineers • Systems Engineers • Configuration Managers • Systems Administrators • Systems/Software Testers Most positions require an active TS/SCI clearance with full scope or CI polygraph. SPT is EOE, M/F/D/V. .com t-inc w.sp7230 5 • ww Lee D -221 efores 1-854 tD rive, Suite 6 • 30 100 • Columbia, MD 2104 Our promise is to ensure we are committed to continued exellence and success for our clients’ mission. We are devoted to doing what is right for our company, our partners, and our employees to best serve our community and country. © 2011 Software Process Technologies, LLC. (SPT). All rights reserved.
  3. 3. AutomatedS T oftware estingDecember 2011, Volume 3, Issue 5ContentsCover StoryMobile Automated Testing 19For many, mobile devices are fast becoming the computing environment of choice. With so many different devicesand new ones emerging all the time, it’s infeasible to conduct sufficient testing for each, manually. But how willtest automation need to adjust from the PC to mobile? Read this article to find out. By Patrick QuilterFeaturesObfuscation: Avoiding Friendly Fire in the Battle for Security 12How do we handle application dynamics introduced by security measures such as obfuscation? Read this article to get strategies forhandling this challenge that results in modern computing. By Dion Johnson“Integrating MS Office For Cost-Effective Test Management 26Need a cost-effective test management solution? Read this article as Tom Wimsatt identifies key test management features, as well as astep-by-step approach for using common Windows desktop software for creating a tool customized to your needs. By Tom WimsattColumns & DepartmentsEditorial 4 I ‘B’Log To U 38The Mobile Evolution Enjoy a testing comic strip that is funny and informative.How mobile technology effects test management andautomated test execution. Go On A Retweet 40 Read featured microblog posts from the web.Authors and events 6 TestKIT is Hot! 42Learn about AST authors and upcoming events. The KIT is Koming Learn what this year’s conference has to offer.Open Sourcery 8 The Automated Software Testing (AST) Magazine is an Automated TestingTest Management is Getting Crowded Institute (ATI) publication.How Crowdsourcing and the Mobile Revolution Have For more information regarding the magazine visitInspired The Birth of a New QA Tool http://www.astmagazine.automatedtestinginstitute.com December 2011 www.automatedtestinginstitute.com Automated Software Testing Magazine 3
  4. 4. EditorialThe Mobile Evolutionby Dion JohnsonMany computing technologies have beenintroduced over the years and with them,new approaches for test implementation.Linda Hayes wrote a great article calledthe “Evolution of Automated SoftwareTesting” that was published in Volume1, Issue 1 of the Automated SoftwareTesting Magazine; her article chronicledthe evolution in computing fromMainframe dominance all the way upto the Internet taking center stage. Eachevolutionary phase was accompanied bya test automation evolutionary responsethat highlighted the changes to how testautomation was conducted in order tokeep pace with the shift in the computingworld at large.The computing world is at the beginningof a new computing sea change, andI have no doubt that there will need to ...efficient test management is no longerbe a major evolutionary response to thischange in the world of test automation. just a luxury.The sea change will move us further into test management is no longer just a Quilter discusses tools for managingthe world of mobile technology, and the luxury. Many organizations have gone mobile device test cases.move is likely to happen faster than any without any real approach or tool for testof the other computing paradigm shifts management, with the excuse that such Next, Tom Wimsatt reveals an approachto date – evidenced by the faster than an approach or tool would be overkill for for implementing a homegrown testexpected depletion of IPv4 addresses that the way they do testing. With increased management solution in his articlehas forced us quickly into the world of test complexity, however, and teams that entitled, “Integrating MS Office for Cost-IPv6 (see AST Volume 3, Issue 1). may span multiple organizations, and/or Effective Test Management”. In addition continents, test management is becoming to identification of key test managementMobile technology has introduced new increasingly important. Another item features, Wimsatt provides a step-by-stepdevices, new types of software and a whole that we must come to grips with is how approach for using common Windowshost of new challenges for testing and test automation of our test execution will be desktop software for creating a toolautomation. In the past we were largely changed by the mobile evolution. This customized to your needs.concerned about adjusting to a world in issue of the Automated Software Testingwhich setting up lab environments to test In the second feature, Dion Johnson Magazine is the first of many to comemultiple versions and configurations of a (yours truly) shifts gears slightly to offer that focuses on these items and more.handful of browsers was the challenge. strategies for handling another challengeThat’s nothing compared to the 100s, In our cover story entitled, “Mobile that results in modern computing: themaybe 1000s of mobile devices and Automated Testing”, Patrick Quilter challenge posed by security measuresconfigurations that may be running your provides a look into how the explosion such as obfuscation. Obfuscation oftenorganization’s mobile application (app)! of mobile device usage has affected results in dynamic UI properties thatWe need to quickly come to grips with the current day implementation of test execution automation. He discusses how render an application non-automatable.what this means for testing and the toolsthat aid our testing. traditional tools and strategies can be In this article entitled, “Obfuscation – used for mobile app automation with a Avoiding Friendly Fire in the Battle forOne thing this technological shift little assistance from simulators, but also Security”, he provides useful mechanismsmeans is that we need to come to grips addresses test execution automation on for creating stable automated scripts onwith the fact that structured, efficient the actual mobile devices. In addition, an unstable application.4 Automated Software Testing Magazine www.automatedtestinginstitute.com December 2011
  5. 5. Training That’s Process Focused Yet Hands OnSoftware Test Automation Training www . training . automatedtestinginstitute . com Public Courses Software Test Automation Foundations Automated Test Development & Scripting Designing an Automated Test Framework Advanced Automated Test Framework Development Virtual Courses Automated Test Development & Scripting Designing an Automated Test Framework Advanced Automated Test Framework Development Come participate in a set of test automation courses that address both fundamental and advanced concepts from a theoretical and hands on perspective. These courses focus on topics such as test scripting concepts, automated framework creation, ROI calculations and more. In addition, these courses may be used to prepare for the TABOK Certification exam. Public and Virtual Training Available
  6. 6. Authors and EventsWho’s In This Issue? Patrick Quilter Jr. has earned a Bachelor of Science in Computer Science & Mathematics, a Automated S T oftware esting graduate certificate in Homeland Security Technology Management, and is in the process of acquiring a Managing Editor Master degree in Technology Management. Patrick Dion Johnson has been an automation architect for a number of large companies such as Travelers Insurance, Computer Contributing Editor Science Corporation (CSC), and GEICO. While at Donna Vance Travelers Insurance, Patrick received the company’s Edward TorrieOutstanding Achievement Award for an R&D solution that programmaticallygenerated business test cases for automated regression execution. Patrick Director of Marketing and Eventshas also worked as a test consultant providing tools architecture and process Christine Johnsonguidance for both commercial organizations and federal governmentagencies. After 10 years of experience and continued development, he isin the process of patenting a cloud based, automation framework that’s A PUBLICATION OF THE AUTOMATEDbeen configured with many popular automation engines. He is currentlyemployed as a lead automation architect while also assisting organizationsthrough Quilmont LLC, a test process consulting company and cloud based,test automation service provider.Tom Wimsatt is an ISTQB/ASTQB FullyCertified Advanced Tester with Software ProcessTechnologies, Inc. A Sotera Defense Company in CONTACT US AST MagazineColumbia, Maryland. He has been in the field of astmagazine@automatedtestinginstitute.comtesting for more than 20 years beginning in the PartsAnalysis labs with NASA at Goddard in Maryland. ATI Online ReferenceThe last ten years he has been concentrating on testing contact@automatedtestinginstitute.comsoftware specializing in tools and automated testdevelopment. He can be reached at: TomWimsatt@wimsatt.truepath.com Dion Johnson has several years of experience ATI and Partner Events in providing IT services to both government and private industry. With a Bachelor of Science degree January 1 2012 in Electrical Engineering, he has spent much of his TestKIT 2012 Conference Call professional career as a consultant in the areas of quality For Presentations Opens assurance (QA), quality control (QC), software process improvements, requirements analysis and software April 1, 2012 test automation. As a regular conference speaker 4th Annual ATI Automationand presenter, Dion has delivered award winning and highly acclaimed Honors Awards Nominationspresentations at many of the most prestigious industry conferences, including Begin www.atihonors.automatedtestinginstitute.comthe StarEast and StarWest International Conference on Software Testing,Analysis and Review, and the Quality Assurance Institute Conference. He Fall 2012also teaches college level classes relative to testing and test automation, andhas several published articles in various IT publications. TestKIT 2012 Conference6 Automated Software Testing Magazine www.automatedtestinginstitute.com December 2011
  7. 7. K The KIT is Coming gnimoC si TIK ehT gnimoC si TIK ehT Fall 2012 http://www.testkitconference.comDecember 2011 www.automatedtestinginstitute.com Automated Software Testing Magazine 7
  8. 8. Open SourceryTest Management is GettingCrowdedHow Crowdsourcing and the Mobile Revolution Have Inspired The Birth of aNew QA Tool A few years ago, the uTest customers seekingopen source alternatives to have their softwarefor test management tested in diverse,were pretty sparse. The realistic environments.bug tracking portion of Matt Johnston, CMOtest management didn’t of uTest, referred tolack for automated this type of testing astools, but not very many “in the wild testing”.tools filled the needs It was this “in the wildfor features such as test testing” along withcase administration, the large number ofassigning work, status clients involved intracking, reporting, and mobile application (app)other key elements of test development that helpedmanagement. This is far to lead to the decisionfrom being the case now, to create a new testas there are several viable management tool.open source solutions Johnston explainedfor test management. that once it was clear thatAmong the most popular a test management tooltools are TestLink and announcements were not only intriguing was necessary to better administer theXStudio. XStudio isn’t completely open due to the fact that several open source testing process, they polled nearly 1000source, but is free and has some open test case management tools already exist, of their customers to get a feel for whatsource elements. There are many more but also because uTest indicated in these announcements that they were partnering solutions were being used. The feedbackopen source test management tools with Mozilla to produce this new offering. revealed that the test managementbesides these two, however, including That’s right! They were partnering with market was extremely fragmented.RTH (Requirements Testing Hub), the maintainers of Testopia, an already Small companies often didn’t use testSalome-TMF, and Testopia. Testopia is available open source test management management tools at all, while largeran open source test case management tool. AST reached out to uTest to get companies felt their solutions – whetherextension for Mozilla’s open source bug some answers. commercial or open source – had serioustracker, Bugzilla. Given this existing list deficiencies. Ultimately, while most ofof available tools, we were intrigued uTest describes themselves as “the their customers were happy with the bugto hear of a new test management tool world’s largest marketplace for software tracking systems they each employed,known as CaseConductor entering this testing services.” With a communityalready crowded space. A company there was little consensus on the most of 45,000+ professional testers fromknown as uTest announced in April that satisfactory test case management tool. 180 countries around the globe, theythey’d be developing a new test case When asked specifically about tools are founded on a crowdsource testingmanagement system (http://www.utest. such as TestLink and XStudio, Johnston business model that involves a number ofcom/press/utest-announces-partnership- indicated that his customer base gave different testers from different places asmozilla-build-test-case-management- little feedback on these at all. opposed to in-house consultants. Testerssystem), an announcement that came sign-up with uTest and complete a Open source tools such as TestLinkto fruition on October 25, 2011 as technical profile indicating their technical and XStudio were, however, evaluatedthey announced the release of the expertise, along with the platforms, by uTest, along with many commercialbeta version of this system known as environments and/or devices they have tools and, according to Johnston, someCaseConductor (http://blog.utest.com/ access to. These testers are then enlisted key elements necessary for uTest’sutest-and-mozilla-release-beta-version- for testing assignments as necessary approach for doing business wereof-caseconductor/2011/10/). These when their profiles meet the needs of missing. One of the key elements that8 Automated Software Testing Magazine www.automatedtestinginstitute.com December 2011
  9. 9. Open Sourcerywas lacking was the ability to adequately the beta release, CaseConductor was regression tests.handle remote collaboration. For any developed with the following three core 3. Simple interface – The producersgiven testing project, uTest may have values: of CaseConductor set out to keeptesters from all around the world plus 1. Community-oriented – Allow the tool as accessible and simpletesters from a client’s company (for which dynamic distribution of test cases as using Word or Excel since thosethe testing is being conducted) involved across different user roles. With are the tools being used to managein test implementation. Tests or even test CaseConductor, your testing test cases for a big chunk of thesteps may need to be assigned to various team can be a combination of market. Test cases libraries havetesters in different locations, testers may your internal QA members, open the potential to be quite large, butneed to be able to communicate with source testers and uTesters – all the UX has been streamlined toone another, and status may need to be of whom can execute test cases keep assignments and results onappropriately tracked. Another missing simultaneously. Essentially, you one screen.element that the CMO identified was the can create an unlimited testingability to appropriately handle a more Currently, there is no integration team using a single interface. with functional automated test tools, butfluid, agile style of testing. uTest has a lotof customers that develop mobile apps, 2. Environment coverage – In according to Johnston, conversations havewhich Johnston says requires a different today’s software ecosystems, already begun with organizations that aretesting approach than that employed for almost every product needs to interested in developing integrations withlarger, more monolithic applications. support an enormous testing matrix it. AST will keep an eye out for this and consisting of OS versions, device will also be listening to the automation For these reasons and others uTest felt makers, languages, locations community to find out how well you allthat a new tool was necessary, and upon and more. CaseConductor’s receive CaseConductor into the fold ofworking with uTest customer Mozilla, the design around environment test management solutions.makers of the popular Firefox browser, it attributes allows testers to easilywas found that they had many of the same and visually specify complex Wait! There’s More!test management needs that apparently compatibility test metrics andweren’t being filled by Testopia or any select those high priority test cases For more information onother tool. uTest and Mozilla decided for quicker and more targeted CaseConductor, visit theto work together to create a new test compatibility testing. The goal CaseConductor repository atmanagement solution that culminated into for CaseConductor was to enable https://github.com/mozilla/what is now known as CaseConductor. testing managers to improve their caseconductor-platform/ According to the announcement of product coverage when doing fullTraining That’s Process Focused Yet Hands OnSoftware Test Automation Training www . training . automatedtestinginstitute . com Public Courses Software Test Automation Foundations Automated Test Development & Scripting Designing an Automated Test Framework Advanced Automated Test Framework DevelopmentDecember 2011 www.automatedtestinginstitute.com Automated Software Testing Magazine 9
  10. 10. 10 Automated Software Testing Magazine www.automatedtestinginstitute.com December 2011
  11. 11. We have 203 guests online December 2011 www.automatedtestinginstitute.com Automated Software Testing Magazine 11
  12. 12. ObfuscationAvoiding FriendlyFire in the Battlefor SecuritybyDion Johnson12 Automated Software Testing Magazine www.automatedtestinginstitute.com December 2011
  13. 13. “Obfuscation is not the first technique employed to protect code from those that wish to reverse engineer or tamper with it, but is instead another weapon used in a long battle betweenprogrammers and those that seek to use or access the programs in unintended, undesirable ways.”Obfuscated Code - Obfuscated code is source or machine code that has been made difficult to understand for humans.Programmers may deliberately obfuscate code to conceal its purpose (security through obscurity) or its logic to prevent tampering,deter reverse engineering, or as a puzzle or recreational challenge for someone reading the source code. http://en.wikipedia.org/wiki/Obfuscated_code C yber security and the ability to protect code from being reverse engineered have become increasingly important as the threats of a system being reverse engineered or compromised have exploded over the years. It is widely accepted that the risk of these threats cannot be eliminated, but can be greatly reduced by making application code more obscure and costly to decipher in an effort to deter the act of reverse engineering. Increasingly, a method known as obfuscation is being employed to make the code more obscure. In general, obfuscation is the practice of making something more confusing, unclear, and/or difficult to understand. This same definition applies to obfuscated code as well. Obfuscated code is code that has been modified, typically by a program known as an obfuscator, to be more convoluted and difficult for humans to understand and follow its behavior. The obfuscated code is functionally indistinguishable from the original code, but the details surrounding the implementation of those functions have changed. Obfuscation is not the first technique employed to protect code from those that wish to reverse engineer or tamper with it, but is instead another weapon used in a long battle between programmers and those that seek to use or access the programs in unintended, undesirable ways. Unfortunately, it is often the software automator that gets caught in the cross fire from this never ending battle, particularly when the obfuscation affects an application’s user interface (UI). When obfuscation affects an application’s UI, test automation is severally impacted because these tests depend on a certain degree of consistency and/or predictability in how UI elements are referenced. Turning to Figure 1 allows us to look at three commonly exercised elements: a Username textbox, Password textbox and a Login button. Automated tests identify these three elements by property values that have been assigned to them in the application code. For example, each object may have an ID property that defines it. The username textbox may have an ID equal to ‘uname’, the password textbox may have an ID equal to ‘pword’, while the Login button may have an ID equal to ‘Login’. These property names are descriptive and probably fairly constant, thus helping the page remain relatively automatable and maintainable. An obfuscated UI will likely change these IDs in a manner that will negatively affect their ability to be consistently accessed by automated tests. Table 1 and Figure 2 reveal how the ID properties may be affected by obfuscation. Not only do the properties follow no set pattern, they may change from build to build or as often as each time the application is invoked.December 2011 www.automatedtestinginstitute.com Automated Software Testing Magazine 13
  14. 14. • Pre-obfuscation automation • Label-based identification ID property = ‘uname’ • Obfuscation map utilization labels • Image-based automation ID property = ‘pword’ Pre-obfuscation Automation Probably the most obvious solution to ID property = ‘Login’ dealing with obfuscated code is to… well…not deal with obfuscated code. Since obfuscated code is typically the product of a baseline set of code being run through an obfuscator, it stands to reason Figure 1: Login Screen that automation be performed on the code prior to it being obfuscated. Some take umbrage with this solution because it is testing of a code base that is different from that which goes into production. In his blog, Nick Olivo, frequent ATI Object ID First Accessed ID Second Accessed contributor, takes aim at this criticism: User erlwlkjf4 askflaf6 “I would think that obfuscation would Password pewtlj90 aasksj2 impact any tests that were run after the obfuscation process. So is it safe Login alsjfafs3 aljfljlaj5 to say that companies who employ unit tests must be running those tests Table 1: Obfuscated Properties prior to the obfuscator running? If so, should the manager be concerned that the unit tests were run against non-obfuscated code? Obfuscation would also prevent any other type of post-build white box testing; so any tools Ive written that call methods directly from dlls or invoke web service calls would be stopped dead. So whats the best ID property = ‘erlwlkjf4’ solution here? Is it to do the bulk of labels your automated testing against non- obfuscated code and then do manual ID property = ‘pewtlj90’ sanity checks against an obfuscated build? Or is there a better way that Im not seeing?” ID property = ‘alsjfafs3’ (http://autonomicon.blogspot.com/2009/02/ obfuscation-question.html) Label-Based Identification Figure 2: Obfuscated Login Screen Obfuscation is typically a process that is transparent to the user of an application. Therefore, while much of the code isT changed during the obfuscation process, items that we may reasonably expect to remain stable are text-based UI labels, his type of UI instability has brought down many test automation along with their relationship to fields on efforts, because it makes the test automation un-maintainable, a page. For example, Figure 2 has two particularly as the automated test bed grows. text boxes, each preceded by a label that provides information to the user on how There are several proven techniques, however, that may be those textboxes should be used. The labels employed to provide cover for test automation when development are not necessarily directly linked to fields injects obfuscation into the coding battles against hackers and the that they describe, but there is normally a like, including: pattern for assigning labels to fields. Once14 Automated Software Testing Magazine www.automatedtestinginstitute.com December 2011
  15. 15. is moved around the screen and/or its table row 1, table row 1, column 2 properties change. column 1 Cell-based identification is useful when the page objects are contained within a table. Provided that each label/field pair is contained within cells separate from other label/field pairs, the cell configuration table row 2, may be used for uniquely identifying a column 1 table row 2, column 2 field. The first step in making this work is to programmatically locate the label that is associated with the desired field on the Figure 3: Cell-based Object Identification screen, and then get the table cell in which the label exists. Based on the pattern usedthe pattern is identified it can be assessed Thus, gathering all of the page’s objects in the application for storing label/fieldand used for object identification by an into an array named pageElementArray, pairs, the label cell information will beautomated test at run-time. would produce an array or collection with used for uniquely identifying the desired the following items: field. For example, if the label typicallyThree approaches for using labels to exists in the same cell as its associateduniquely identify fields are: pageElementArray[0] = Username label field, then code can be written to get the 1. Location-based identification pageElementArray[1] = erlwlkjf4 textbox child textbox of the same cell element in 2. Cell-based identification pageElementArray[2] = Password label which the label exists. If the field typically exists in the same row, but an adjacent 3. Property-based identification pageElementArray[3] = pewtlj90 textbox column of the label, as illustrated in pageElementArray[4] = alsjfafs3 button Figure 3, code can be written to get theLocation-based identification initially child textbox element of the adjacent cell.seems closely linked to the volatile Upon looping through the array orcoordinate-based automation approach In a roundabout way, the cell-based collection of objects and locating thethat we’ve been taught to steer clear of. approach is dependent on location, desired label, the automation loop canDespite appearances, however, it is not because you’re expecting the field to be in be set to return the very next item in thethe same thing, but is instead a much more a location relative to its associated label. array or collection that is a textbox. So,stable approach to identifying application The third label-based approach, property- if the automated test needs to access thefields. Location-based identification based identification, is the only label- ‘Password’ textbox without using theusing labels is based on the stability of based approach that is not dependent dynamic ID property that is currentlythe label names and the pattern used by on location at all. This approach is most equal to ‘pewtlj90’, it would simplyprogrammers for coupling labels with useful when the label element has an loop through the array of objects andfields, which is a more robust approach attribute that is directly tied to an attribute find the ‘Password’ label. Upon find thethan using the actual position of an object of its corresponding field. ‘Password’ label at array item 2, the loopon a screen. - which has been set up to return the very For example, a label may have a NAMELocation-based identification first requires next array item that is a textbox - will attribute that is identical to the ID attributethat your test automation approach provide return array item 3, which is the ‘pewtlj90’ of its corresponding field, as illustrated ina mechanism for obtaining all fields on a textbox. Provided that the label name text Figure 4. In this situation, automated testpage into an array or collection, based on remains constant, the associated field code can be written to programmaticallythe order in which those objects appear can be found even if the field (and label) obtain the desired label based on its labelon the screen. For web pages, the “all”document object model (DOM) method NAME property = ‘erlwlkjf4’makes this possible. If you are using xpathfor web object identification, “//*” willdo the trick. Once the objects are stored ID property = ‘erlwlkjf4’in an array, a loop may be established totraverse that array or collection in search ID property = ‘pewtlj90’of a label element with the desired labeltext. Then, based on the pattern identifiedfor coupling labels with fields, the desired ID property = ‘alsjfafs3’field can be identified. For example, theapplication screen represented in Figure2 has its fields preceded by labels that NAME property = ‘pewtlj90’represent that field, evidenced by the factthat the ‘Username’ label precedes the Figure 4: Property-based Object Identificationtextbox used for entering a username.December 2011 www.automatedtestinginstitute.com Automated Software Testing Magazine 15
  16. 16. text, and then get the NAME property (TABOK) Guidebook describes Image-associated with that label. The automated based automation as follows: Ccode can then be instructed to find the …often based on Virtual Networkfield with an ID property that is equal tothe label NAME property that has been Computing (VNC), [an image-based automation tool} relies on image ontributecaptured. recognition as opposed to object CUpon identifying the appropriate label- recognition (Context Sensitive) orbased approach, functions and/or methodscan be created to perform the necessary coordinate recognition (Content Sensitive). ontentcomputations in a way that is seamless to VNC-based tools follow the two- Tthe design time test automation process. computer system, with the automatedThe automators would simply need touse a field’s associated label text for tool residing on the client machine and functioning as the VNC client. The server odayidentification of that field as opposed to machine on which the AUT is installedproperties directly related to the desired will run a VNC server that communicatesfield. with and transmits images to the VNC client. These tools, therefore, recognize oBFUSCATION mAPS application objects based on analysis of the transmitted images.At times, when code is passed through Communityan obfuscator, in addition to outputting This mode is more closely related to context sensitive automation than content Comments Boxthe obfuscated code an obfuscation mapmay also be produced. This map provides sensitive, because it is not completelya direct relationship between old property coordinate-based. These images may stillvalues and the new property values of be located if they are moved to a differentapplication fields and other elements. If an screen position or if their internalobfuscation map is not produced, you may properties change. Thus, image-based Announcements &be able to request that the development automation can be useful for handling Blog Poststeam modify the obfuscator to produce applications that utilize obfuscation.such a resource, because it could beinvaluable. With an obfuscation map, a cONCLUSIONutility can be written to synchronize the Obfuscation is a battle between softwareobject information used by the automated producers and their enemies, and if testingscripts with the new object information Automation suffers, ultimately the software producerin the application prior to or during script suffers. Follow the approaches defined Eventsexecution. in this article to avoid being a casualty of obfuscation or any other development IMAGE-Based AUTOMATION approach that injects instability in theThe Test Automation Body of Knowledge application UI field properties. As a registered user you can submit content directly to the site, providing you with content control and the ability to network with like minded individuals. Learn more today at http//www.about.automatedtestinginstitute.com16 Automated Software Testing Magazine www.automatedtestinginstitute.com December 2011
  17. 17. December 2011 www.automatedtestinginstitute.com Automated Software Testing Magazine 17
  18. 18. 18 Automated Software Testing Magazine www.automatedtestinginstitute.com December 2011
  19. 19. A Automated Mobile Automation M obile estingWith Traditional Tools & Beyond by Patrick QuilterHas consumer demand already chosen mobile devices to replace the personal computer?Will traditional webpage design with drop downs and text boxes lose ground to mobiletouch screen data entry? Will everything we know about creating automated tests forapplications change because of mobile devices? Don’t abandon your old automationframeworks just yet. You’ll still need them to expand into the untamed world of automatedmobile testing.Mary Meeker, a renowned Internet analyst with Morgan Stanley, highlighted the significanceof mobile consumer demand in her latest “State of the Internet” report. Meeker predictedthat within the next five years, “more users will connect over mobile devices than desktopPCs.” Since 2010 the trend of smartphone shipments outpacing PC shipments continuesto surge and gives credence to Meeker’s forecast (Ingram, 2010). Anyone can see thata multitude of organizations have already embraced this technology swing by developingapplications explicitly to establish a mobile presence. Recognized companies continue tooffer free or inexpensive apps for both business and entertainment purposes.While having a mobile presence is significant for organizations,it is also becoming imperative to have stable applications and mobile “Simulators onwebsites. PhoCusWright conducted an 800 participant survey to evaluate Windows or Macsuser tolerance for buggy mobile software. The survey revealed that 37% ofsmartphone users indicated that they would be less likely to use that particular allow teams to employapplication again and 28% indicated that they would seek out competitorapplications (Leggatt, 2010). These facts illustrate the importance for sound traditional automationapplication development practices and the ability to automate test cases. Withso many different devices and new ones emerging all the time, it’s infeasible engines like Quickto conduct sufficient testing for each, manually. Automation’s luster is theability to execute large volumes of repeatable scenarios that a person could Test Professional ornot undertake manually. Selenium”December 2011 www.automatedtestinginstitute.com Automated Software Testing Magazine 19
  20. 20. To Simulate or Not to Simulate software under development is going device transactions has spurred the growth to be application based or web based? of application development serviceA typical beginning for application If application based, functionality will providers. A capable service provider willdevelopment teams journeying the path vary depending on device platform such offer an on-demand testing infrastructure,of mobile testing is to create automated as iPhones, Androids, Blackberries, and multiple devices, and automated scriptingscripts that execute on desktop simulators. Windows. Although some solutions capabilities.A simulator has the same look-and- are materializing to allow developers In terms of mobile testing this type offeel of a mobile application displaying the ability to create one codebase that service is invaluable because it accountson a device, however, it executes on a deploys over multiple devices (regardless for all major strategic characteristicsdesktop operating system. Teams utilize of iPhone, Android, etc.) the typical under one solution. But what specifica simulator by constructing business approach is to maintain a separate service provider existing today offers allflow, user designs, and verifying basic codebase per platform. A savvy tester of this?functionality of their applications. will see the vulnerability in maintainingSimulators on Windows or Macs allow multiple codebases. This implies multipleteams to employ traditional automation builds to verify. Meet Keynoteengines like QuickTest Professional or DeviceAnywhereSelenium to create test cases. Device manufacturers and operating A great example of this type of service systems have the biggest impact onAlthough using a simulator adds provider is a company called Keynote how applications function even acrossvalue to an application development DeviceAnywhere headquartered in the platform families. For example, thepractice, verifying functionality is San Mateo, California. Keynote HTC EVO and Samsung Galaxy are bothlimited. Neglecting to conduct tests on a DeviceAnywhere helps organizations of part of the Android family, however, thephysical device causes vulnerabilities to all sizes bring their products, services, same application installed on them couldapplication performance. This realization and IT applications to a mobile platform. function differently. The reason forhas its drawbacks. For one, the list of Their cloud-based solution hosts a phone this is the manufacturer can manipulatepopular devices is growing rapidly, and bank of over 1000 real devices that are how processing, GUI rendering, andtesting must occur for all devices on used for mobile application testing, test form factors affect execution on theirthe list. Secondly, how do application automation, monitoring, and development devices. Web based functionality is moredevelopment teams execute test cases worldwide. This is possible through straightforward because developers andoutside of the PC-based automation tools? their award winning, end-to-end studio testers converge on a single deploymentAre there any strategies to handle these platform that cuts operational expenses of code to a server. Configuration settings A capable service provider will offer an on-demand testing infrastructure, multiple devices, and automated scripting capabilities. determine functionality based on thechallenges? and time associated with testing mobile device a user connects with to the mobile applications (HP/KDA datasheet, 2009). website. Normally, functionality acrossWhat’s the strategy? product families will work the same.The most challenging aspect of an Although not a guarantee, testing a mobile Accessing the Environmentautomation strategy is devising a practice website with an HTC EVO and Samsung Keynote DeviceAnywhere users firstthat optimizes test case execution across Galaxy that is running the same version of gain access by logging into a web-basedmultiple devices. Ideally we’d like to Android should produce the same results. portal that provides a managementcraft a single test case that can execute Testing just one of them would most likely dashboard, training information, andon multiple devices, otherwise we would be sufficient. The device carrier is also links to support resources. From there,have to maintain a separate copy of each significant when considering a strategy for the user downloads the client-sidetest case, per device. Test sets (a collection both mobile applications and web based software onto their individual desktop toof test cases that compose a test plan) software. Network speeds and protocol experiment with the actual devices. Thiswould become maintenance intensive restrictions are common performance gives them a window into the Keynoteover time without this reuse capability. hinderers for mobile software. The key DeviceAnywhere devices hosted in theMobile devices cause a significant amount for any organization will be how to turn cloud. A visual representation of theof variation for development and testing the strategy into reality. physical device displays once the userefforts. First, is the question of whether Luckily consumer demand for mobile acquires a device. Users can license any20 Automated Software Testing Magazine www.automatedtestinginstitute.com December 2011
  21. 21. K The KIT is Coming If you thought ATI’s 2011 event was good, wait until you see 2012. http://www.testkitconference.comDecember 2011 www.automatedtestinginstitute.com Automated Software Testing Magazine 21
  22. 22. Most tools rely on open and published APIs to obtain the discrete properties of a control. Mobile platforms do not provide the traditional APIs, rendering your favorite tools useless in mobileof the hosted Keynote DeviceAnywhere developed for one device it can be cloned script-less actions correspond to thesedevices upon request. Acquiring devices to run on multiple devices. It’s script- screen interaction properties.is an important concept because team less because automation steps are created Walking through an example, a usermembers can share and schedule times by dragging and dropping icons which developing a script for an iPhone will firstfor when they want to test on particular are preset to perform specific actions. acquire the device and is then presenteddevices. Stringing together a series of icons with a window view of the iPhone nextOnce a device has been acquired, the user produces a whole test case. to a blank execution grid. This tells thecan control all aspects of the physical For users that are familiar with traditional scripting component that the particulardevice through mouse clicks and keyboard web-based automation, the biggest implementation is going to interact withstrokes to mimic the actions as if they had adjustment may be getting accustomed an iPhone device. The users will drag-the device in their hand. The user can to how objects are represented on mobile and-drop Keynote DeviceAnywherealso manipulate the features of the devices pages. Tal Barmeir, CEO of Experitest, actions (icons) onto the execution gridsuch as buttons, LCD screen, speaker, in an order that corresponds to test caseringer, battery, power-plug, and data- steps. The actions available to the usercables. The Keynote DeviceAnywhere can search for some text on the screen,control sliders allow the user to change Script-less automation: enter text, click on GUI icons, or validatethese setting and pre-condition thedevice before conducting a test case. “ Script-less means no scripting/ a checkpoint area located on the screen. The actions are separate steps represented programming in the test automationAnother controller known as the Device as separate icons. During execution, the tool’s native language. ThisConsole is also accessible for typing actions perform the key strokes and touch doesn’t mean there are no scriptstext and performing SMS messaging. screen interaction that a human has with involved. Explicitly, this meansThis Console can monitor HTML traffic the iPhone. The iPhone’s screens are while automating test cases, therebetween webpages and the device. The driven by the actions in the execution grid is no more need to program scriptsimportance of controlling these features until the last one has been reached. for each test case”is in the fact that mobile test cases have Once the test case has been created andsome unique verification steps when (Qualitia, 2011). saved, the test case can then be clonedcompared to PC based applications or for other devices. All of the steps thatwebpages. For example, a user may want were created for the iPhone could beto see what happens to their application enabled for an HTC EVO. The benefit ofwhen the phone rings or a text message explains that, “existing industry-leading cloning test cases is that for the most partis received (Keynote DeviceAnywhere, test automation tools are restricted by application flow stays consistent from one2010). Accessing the devices through object recognition limitations. Most device to the next. All action commandthe studio and using the Device Console tools rely on open and published APIs to steps are converted for use on the newallows a user to set the stage for these obtain the discrete properties of a control. device. In some cases, actions thattypes of scenarios. Mobile platforms do not provide the contain specific GUI representations willTo accomplish automation, Keynote traditional APIs, rendering your favorite need updating after the cloning process.DeviceAnywhere provides a module tools useless in mobile” (Barmeir, 2011). Typically these types of updates arespecifically for managing requirements, Unlike web page objects, mobile objects minimal and identifying where to maketest case steps, execution tracking, and are identified differently. Web-based the adjustment is intuitive to the users.result reporting. Each of the items are objects are referenced by property tags This is because the user has the advantageorganized within a project and shared such as an object name or ID. A scripter of using the Keynote DeviceAnywherebetween team members for collaborative can use tools to break down the various technology to stay within a consistenttesting. A user can start creating types of objects on a page and associate framework. Users can also mitigate theautomation steps by accessing Keynote data with those objects. With mobile amount of updates by incorporating anDeviceAnywhere’s device-agnostic, application rendering, property tags are automation development strategy or bestscript-less automation technology (HP/ not referenced in this way. Instead a user practices within the framework that fitsKDA datasheet, 2009). The technology is interacts with the screen by anchoring on their applications. Trial and error willdevice-agnostic because when a script is specific text, icon recognition, and pixel help identify the best strategy. locations. Keynote DeviceAnywhere’s22 Automated Software Testing Magazine www.automatedtestinginstitute.com December 2011
  23. 23. Incorporating Framework an add-in that a user can download for editing complex logic before the compileAutomation this capability. For those that are QTP stage. The Keynote DeviceAnywhere developers, you know that when launching APIs provide a great way for organizationsFor many applications, using the script- QTP you have the option to select which that don’t have expensive commercialless approach is appropriate especially add-ins are loaded for your development tools or the budget to purchase themif functionality changes are minimal and execution sessions. After installation, to still get the most out of the mobilebetween code updates or if the particular the user just needs to check the box in the automated testing experience.application doesn’t have an abundance add-in selection screen for the Keynoteof data entry. An application heavy withlinks and button pushes would be covered DeviceAnywhere add-in. Once this step How We Integrate with is complete, a user can reference any of Keynote DeviceAnywhereadequately. This is also a useful approach the actions they’ve created to drive theif the test case suite is small and easily automation capabilities on any of the Here at Quilmont we have a pre-developedmaintainable. devices. Interacting with a specific device framework that comprises a user-friendlyUsers that have applications with heavy is managed in the QTP object repository interface for managing applicationdata entry in many business scenarios will where every device that is going to be properties and creating test case steps.naturally want to incorporate industry tested is a separate object in the repository. Our out-of-the-box function librariesstandard practices such as creating a data The handshake between the repository integrate with QTP/QC, Selenium,driven or keyword driven framework. This and the devices are controlled by a unique CodeUI, Rational Functional Tester,type of approach enhances script reuse, ID property. From this point a user can and yes, Keynote DeviceAnywhere.quicker maintenance, and offers the ability design a framework and separate the For our Keynote DeviceAnywhereto create test cases without consuming a automation code from their test case data. implementation, we provide plug-ins forKeynote DeviceAnywhere license (the As a best practice, a user should create both QTP and JAVA APIs to deliver alast section provides an example of how a design where QTP does the repetitive framework with keyword and data driventhis can be accomplished). For those well calling of a sequence of Keynote capabilities.versed in automation design, the script- DeviceAnywhere actions while passing In our setup we employ a three layerless approach may sound like a variation parameters to those actions. Keystrokes approach. In the first layer we have severalon record and playback. Also restricted and data entry can be controlled in keyword actions packaged at a project levelaccess to the scripting language may strike spreadsheets or other data harnesses. All for use in the Keynote DeviceAnywhereautomation developers as a hindrance to results and screenshots during execution Studio. Keynote DeviceAnywheretheir effectiveness. In other words there will roll up into the QTP execution report Studio allows projects to be importedare some things a seasoned programmer (Keynote DeviceAnywhere, 2010). and exported. For us, this means wecan accomplish more effectively with code can import and export our keyword actions to any Keynote DeviceAnywhere Keystrokes and data entry can be Studio installation. Our second layer is a driver script which could be the QTP controlled in spreadsheets or other or JAVA API version depending on the customer request. This is the layer that processes the main execution and calls data harnesses. the keywords depending on the test case information it receives from the third A second option for organizations that do layer. The third layer is our frameworkthan relying on script-less actions. Not to not have Functional Testing (QTP) and interface. In this layer our customersworry because Keynote DeviceAnywhere don’t plan to purchase it, is to create a can manage any application propertiesoffers two ways to interact with their data driven or keyword driven framework (for mobile application this would beautomation engine so that a tradition data by utilizing Keynote DeviceAnywhere’s text, icon objects, pixel information)framework can be created. JAVA based APIs to control the actions. and create test case steps. The testFor organizations that use HP/Mercury In the same way a user can create case steps call keyword actions in theFunctional Testing (QTP) and Quality reusable code and incorporate a datasheet Keynote DeviceAnywhere studio whileCenter, there is a tremendous opportunity to drive the commands. Keynote also passing parameters to these actionsto continue utilizing these tools and DeviceAnywhere provides a JAVA editor through the driver script. This allows ourconfigure them to work with the Keynote internally to create the API programs and customers to first create all of their testDeviceAnywhere Studio. This allows conveniently stores all components inside case information outside of the Keynoteorganizations to expand upon the familiar the Keynote DeviceAnywhere Studio. In DeviceAnywhere Studio which cuts downfeatures of QTP and QC as well as enhance addition, the Keynote DeviceAnywhere on license consumption and then log intotheir tester’s skillsets. This option allows Studio can interact with other JAVA the Keynote DeviceAnywhere studioa QTP developer to extend their current editors such as Eclipse to get the full when they are ready to execute their testframework to invoke existing actions in capabilities of a JAVA environment cases.the Keynote DeviceAnywhere Studio. (Keynote DeviceAnywhere, 2010). This An example of how the three layers workKeynote DeviceAnywhere has created is convenient for managing packages and together can be illustrated as follows:December 2011 www.automatedtestinginstitute.com Automated Software Testing Magazine 23
  24. 24. Let’s assume the device we are testing is an HTC EVO and our These three calls will log into the application and drive the screenapplication has a login screen. The login screen has objects for to the next page.a user name, password, and a login button. The goal of our test Statistically we can see that mobile devices are becomingcase is simply to verify that we can log into the application and the preferred way of conducting business transactions forget to the next screen. Starting with layer three (framework consumers. Touch screens provide a more desirable methodinterface) we would represent each of the objects (user name, for collecting customer information than keying data into webpassword, and login button) in the section of the interface thatmanages object information. Since these are mobile objects we forms or scrolling through overpopulated dropdown boxes.would most likely reference the text labels for each object. In Mobile devices have arrived and more companies are embracingthe test case creation section of the interface we would create the the change every year. It’s fair to describe the applicationstep sequence where each step would reference an object name, development techniques for mobile software as still being in itsvalue, and keyword action. early stages. However, applying discipline through traditional application development practices (test automation being one of them) will assist future standardization. From an automation developer’s standpoint, learning the tools and techniques in this early stage will have great implications on a person’s individual Framework Interface: skillset. It’s an amazing opportunity to learn, discover, and help Step 1: User Name | user1 | EnterData standardize the untamed world of automated mobile testing. Step 2: Password | pwd1 | EnterData References: Step 3: Login | button | Click Barneir, Tal (2011), Need for Mobile Test Automation Tool on Various Platforms (Android, iPhone, Blackberry, Symbian & WindowsPhone 7), Mobile ApplicationIn the second layer, the driver script would read these steps Testing. Retrieved: November 17th 2011 From: http://from the framework interface and call the action in the KeynoteDeviceAnywhere Studio that corresponds to the action in the www.mobileappstesting.com/category/mobile-application-step sequence. The driver passes the object name and value to testing/mobile-application-testing-and-automation-tools/#the keyword action. HP/KDA Datasheet (2009), Test Automation for Mobile Devices, Hewlett-Packard. Retrieved: July 16th 2011 From: http://support.openview.hp.com/ Functional Testing (QTP) or JAVA API Driver: GET Step 1 from interface, CALL EnterData in Keynote Keynote DeviceAnywhere (2010), Keynote DeviceAnywhere Studio, PASS User Name | user1 DeviceAnywhere Test Center User Manual for Enterprise GET Step 2 from interface, CALL EnterData in Keynote Deployments, Keynote DeviceAnywhere. Retrieved: June DeviceAnywhere Studio, PASS Password | pwd1 15th 2011 GET Step 3 from interface, CALL Click in Keynote DeviceAnywhere Studio, PASS Login | button Ingram, Mathew (2010), Mary Meeker: Mobile Internet Will Soon Overtake Fixed Internet, Gigaom. Retrieved: October 25th 2011 From: http://gigaom.com/2010/04/12/ mary-meeker-mobile-internet-will-soon-overtake-fixed-In the first layer, the pre-developed keywords receive the internet/parameters from the driver script and use the KeynoteDeviceAnywhere automation engine to translate the action foruse on the HTC EVO. Leggatt, Helen (2010), Travelers display low tolerance for mobile app, website glitches, Gigaom. Retrieved: October 25th 2011 From: http://www.bizreport.com/2010/08/ travelers-display-low-tolerance-for-mobile-app-website- Keywords stored in Keynote DeviceAnywhere Studio: glitches.html# RUN EnterData, FIND User Name, TYPE user1 RUN EnterData, FIND Password, TYPE pwd1 Qualitia (2011), Defining Scriptless, QualitiaSoft. Retrieved: November 17th 2011 From: http://www.qualitiasoft. RUN Click, FIND Login, TOUCH Login com/pdfs/Definition%20of%20Scriptless%20Test%20 Automation.pdf24 Automated Software Testing Magazine www.automatedtestinginstitute.com December 2011
  25. 25. Are You ContributingContent Yet?The Automated Testing Institute relies heavily on the automated testingcommunity in order to deliver up-to-date and relevant content. That’s whywe’ve made it even easier for you to contribute content directly to the ATIOnline Reference! Register and let your voice be heard today! As a registered user you can submit content directly to the site, providing you with content control and the ability to network with like minded individuals. Community Comments Box >> Community Comments Box - This comments box, available on the home page of the site, provides an opportunity for users to post micro comments in real time. >> Announcements & Blog Posts - If you have interesting tool announcements, or you have a concept that you’d like to blog about, submit a post directly to the ATIAnnouncements & Online Reference today. At ATI, you have a community of individuals that would love Blog Posts to hear what you have to say. Your site profile will include a list of your submitted articles. >> Automation Events - Do you know about a cool automated testing meetup, webinar or conference? Let the rest of us know about it by posting it on the ATI site. Automation Add the date, time and venue so people will know where to go and when to be there. Events Learn more today at http//www.about.automatedtestinginstitute.com
  26. 26. t m sat m Wi by To We all have the necessary components on our systems to build a solution of our own. Test Management26 Automated Software Testing Magazine www.automatedtestinginstitute.com December 2011
  27. 27. “A simple system...to coordinate your test procedure development effort.” Integrating MS Office F C or-E ost ffective Test Management A Homegrown, Desktop Solution for Test Management A s in any test effort, we write test procedures and eventually require some way to keep track of all of the documentation. Many of us become spreadsheet jockeys with workbooks of our tests and other supporting information tucked away in the labyrinth of our disk drives. The management becomes increasingly cumbersome leading to duplication and confusion. We forget if we wrote a procedure to cover a case, a new member joins the team and does not know what has been written, or we are unable to locate the needed procedure. Our typical solution is to write a new test and add it to the workbooks for safe keeping. Another complication arises when we want to create a test set for a particular purpose, such as a user acceptance test or regression test set. We may also get requests from management about which tests cover which requirements, or about getting some sort of a test report. Pulling the different parts together to create the needed documents, and managing the whole process becomes time consuming and error prone. In other words, a perfect opportunity for automation! There are a number of nice commercial solutions available that answer the task quite well; but, what if we do not have a large team or a budget that can afford one of the packages? We all have the necessary components on our systems to build a solution of our own. What follows is an explanation of a system I developed with Microsoft (MS) Office, and the same can be adapted for OpenOffice. Org.December 2011 www.automatedtestinginstitute.com Automated Software Testing Magazine 27
  28. 28. RequirementsBefore jumping into the details of thedesign, a few requirements need to beestablished. After having used a numberof tools over the years, I settled on thefollowing basic functions: • Viewing of all tests • Filter and Sort • Test Set Production • Track Status and Author • Allow Attachments • Associate Requirements • Track Team Members • Allow Team Communication Figure 1: Test procedure database directory structureBasic ArchitectureMost of us have the Microsoft Office need to share out the location to your team if you choose to store it on your own worksuite on our machines and it makes sense station. This organization is optional, so, feel free to adjust to your own desire.to develop for something the majority This organization works well, as it helps to keep things organized in a way that seemshas access to. Most of us, though, forget logical. At the root directory is the actual database – in this case it would be underthat Office is more than Word, Excel and the Test Procedures directory. Under that are subdirectories for each of the systemsPowerPoint. At the heart of this system of your particular test effort. If this was an effort to test MS Office, they would beis the MS Access database. The test Word, Excel, PowerPoint, etc. Under each of the system directories are the subsystemprocedures are in MS Excel spreadsheets directories and under that are the test files themselves.to make the formatting somewhateasier, and allow for some additional The naming convention for the files mirrors the above directory structure. It helpsfunctionality without the complication ofWord document formatting. In addition,there is a hook into Outlook to providesome automated communication for theteam.Behind the scenes, MS Office uses VisualBasic for Applications (VBA) for theprogramming language. This makes iteasy to add more functionality or modifyany of the existing functions.The database provides a place to store thedetails about the test procedures, but thefile system contains the actual spreadsheetfiles. Using links in the database, the testscan be displayed and modified. New testsare created from the information enteredinto the database. The test procedurefiles are kept as originals, and modifiedas needed and copies are created andcollected to build test sets for the actualruns.Refer to Figure 1 below. The database isbest stored on a networked drive sharedout to the team. At a minimum, you will Figure 2: Tables and relationships28 Automated Software Testing Magazine www.automatedtestinginstitute.com December 2011

×