Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Taking Testing to Cloud


Published on

Published in: Technology
  • D0WNL0AD FULL ▶ ▶ ▶ ▶ ◀ ◀ ◀ ◀
    Are you sure you want to  Yes  No
    Your message goes here
  • Paid To Write? Earn up to $200/day on with simple writing jobs. ♣♣♣
    Are you sure you want to  Yes  No
    Your message goes here
    Are you sure you want to  Yes  No
    Your message goes here
  • Be the first to like this

Taking Testing to Cloud

  1. 1. Your Monthly Magazine on Software Testing with Smita Mishra Testing Circus Volume 5 - Edition 3 - March 2014
  2. 2. Article submission guidelines – ● Subject of article can be based on any area of Software Testing. If you want to publish your article on theme based subject please read our announcement of monthly theme published in our site. Article can be submitted without any theme based subject. ● There is no minimum and maximum length of article. If you feel the article is lengthy, please divide the article into logically separated parts so that we can print them in a monthly series. ● Give a meaningful title to the article. If you want a sub-title as well , then add that in a different line. ● Add images/pictures if necessary. If you are using any image/picture which is not yours own work, please include the source. Take care of copyrighted materials. Images need to sent separately with the article. ● Send us the article in MS word (doc/docx) format only. Pdf files are not accepted. ● Write a short write up on the author(s). Usually 7/8 liners in 3rd person descriptive language. ● Include photograph of author(s). Preferred in high resolution .jpeg/.png format. Ideal size would be 50mmX 50mm. ● You can send your article any time. Since we publish every month, your article can be included in any month. There is no such thing called as a dead line. ● Send in your article to with a subject line “Article for Testing Circus – Author Name – Title of the article” ● If you think you can write a column in Testing Circus for at least 6 months, please submit 3 articles in advance. We are open to any idea that may improve the user experience of Testing Circus. ● We will publish the articles in our website a week after the pdf magazine is published. Article Submission GuidelinesDo you have something to share with the testing world? We can make your voice heard to testers. March 2014 - 02 -
  3. 3. Volume 5 - Edition 3 - March 2014 Testing Circus March 2014 - 03 - Topic Author Page # Interview with Smita Mishra Jay Philips 5 Examples of Testable Requirements Ulrika Park 10 Taking Testing to the Cloud Vipin Jain & Anubha Jain 14 A Common Pitfall of Test Cases Raj Subramanian 22 Mobile Testing on the Cloud Gagneet Singh 23 How Quicker can mean Slower Peter Morgan 25 7 Types of Testers - What is your identity Arslan Ali 27 Book Review - “Don’t Hire the Best” WoBo 31 A Fake Tester’s Diary, Part - 39 Fake Software Tester 33 Testers to Follow Editorial Team 35 Test Environment for Security Testing Santhosh Tuppad 37 Running Webdriver Script on a VM Mihai Sarlea 39 Test Automation: New Venues, New Challenges 48 TestComplete Support For Automated Mobile Testing 50 Squish Coco Supports Code Coverage Of C# & Tcl Code 52 Testing Events Around the World 53 Testing Circus Team Founder & Editor – Ajoy Kumar Singha Team - Ÿ Srinivas Kadiyala Ÿ Jaijeet Pandey Ÿ Pankaj Sharma Ÿ Bharati Singha Ÿ Chanderkant Saini Ÿ Dwarika Dhish Mishra Editorial Enquiries: Ads and Promotions: Testing Circus India Chaturbhuj Niwas, 1st Floor, Sector 17C, Shukrali, Gurgaon - 122001 India. © Copyright 2010-2014. ALL RIGHTS RESERVED. Any unauthorized reprint or use of articles from this magazine is prohibited. No part of this magazine may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopying, recording, or by any information storage and retrieval system without express written permission from the author / publisher. Edition Number : 42 (since September 2010) *On the Cover Page - Smita Mishra W hat is w here?
  4. 4. Write to us at March 2014 - 04 - Testing Circus is three and half years old. This magazine has been running on volunteer effort, assisted by few passionate testers from time to time. It has been an amazing journey so far. But, like all other good things in life, it is coming to an end. We are announcing discontinuation of Testing Circus after this edition. It is a sad moment for all of us. Running a regular magazine purely based on volunteer effort is hard task, specially when all volunteers including me have full time jobs to look after. We were always supported by many testers by contributing some awesome articles but let me also admit that it is always a challenge to gather good articles when you don’t pay the writers and busy testers do not reply to your emails and direct messages. There was too much effort for our volunteers. Testing Circus site will remain as it is. We will continue to publish our old articles in the site. We will continue to promote good testing stuffs in Facebook/Twitter. But our regular monthly magazine will stop. I would like to thank all you of you who supported us in this wonderful journey. We will cross the roads again. Till then, stay hungry, stay foolish and test passionately. - Ajoy Kumar Singha @TestingCircus // @AjoySingha From the Keyboard of Editor
  5. 5. SMITA MISHRA Smita Mishra is the founder and chief consultant at QAzone Infosystems, which is a pure-play software testing organization. She is a first generation Entrepreneur and is a Test professional who has spent over 12 years practicing testing and leading test efforts of varying sizes, cutting across all key domains and technologies. In the past, she has worked with multiple organizations, likes of - HCL Technologies Ltd, Fidelity Investments, Nucleus Software Exports Ltd, Churchill Insurance (Now RBS). In her current role, she is involved in creating test teams, managing testing for software companies, leading the overall test strategy for them. She is also engaging constantly with different forums to assist growth for women in her field and women in general too. Organization: QAZone Infosystems Pvt Ltd Current Role/Designation: Chief Test Consultant & CEO Location: New Delhi, India Interview with Testers March 2014 - 05 - 1. Tell us about your journey to becoming a software tester. How did it start and how this has been so far? Was it planned or by accident? My journey to become a good tester still continues. However, it began in 2001, with my first job at Nucleus Software Exports Limited, where I was campus placed after my engineering. A batch of about 20 engineers were picked from across the nation and put together as the founding (independent) test team. Honestly, I had no introduction to “Testing” until then other than the few chapters I read on SDLC, Verification and validation in a book of software engineering by “Roger Pressman”. It was planned by my employers not me but wasn’t really an accident. Once in testing, I quickly began to learn it and found it interesting. I will agree that though I really enjoyed my work, the initial years were nothing exciting. But over the years, I realized that I enjoy testing more than anything else. There have been tremendous learning through the 13 yrs spent in the field. The more I learn the more I see I have yet to learn. Hence, I can easily say that the journey so far has been terrific. 2014 seems to be extraordinarily good year for me and I am loving every moment of it, as a test professional. 2. When did you realize your passion was software testing? I would be wrong if I said that it was love at first sight. It was not. But as they say – good music grows on you even if you don’t like it much in the first go. It was similar in my case. I realized I enjoyed testing early in my work life. But it wasn’t until 2006 that I began to love it. I attended the first conference in my life – it was a QAI conference where I was presenting a practice paper on estimations. Met many testers there and I loved the feeling of belonging to a community too. Then in 2007 I went to work in GE Healthcare account under the most dynamic technology leader and sharp as razor – Gazanfar Hakeem and an excellent Test Manager Smita Sethi (who also happens to be a very sharp professional). I got exposed to very technical testing in DW-BI (ETL Testing) and performance testing of these ETLs and implementing lean methodology. I would term that as the turning point in my career on how I began to look at my testing career. I realized I was cut out to do this. 3. Do you regret being associated with software testing today? Given a chance would you move from testing to any other field in IT? * Interviewed by Jay Philips
  6. 6. March 2014 - 06 - No, clearly no. Quietly no. Loudly no. Simply no. I am proud to be a part of an ever evolving field and that has a huge legacy of achievers. I can easily say I never had second thoughts about my work. Also, I would find it very difficult to work on a single technology or domain all my life – and to look conversely at it - which other field can give me so much exposure in understanding “what really matters” in IT field, while keeping me hands on with all latest technology. 4. What does QAZone do? How are its services different from the services offered by other similar organizations? QAZone is a pure play software testing organization that offers Testing solutions and services, Test consulting, Test training and test support services (Test data management, test environment setup). We offer Business as usual testing as well as specialized testing. We have built a client base of over 20 clients in last 3+ years. Our business focus is our key differentiator. Our testing solutions are designed to work for your business. We look at technology as a platform to make business happen. All our testing has a component of domain focus which becomes a very serious need of our clients when we are dealing with regulated environments like Aerospace and Healthcare. Our next differentiator is that against the trend - we believe in hiring locally, even if it means training potential resources at our expense. For example – anytime we need a resource in US, our first approach is to find a resource locally. This has so far proven to be a very fruitful approach. 5. Last year you organized a conference called ThinkTest in India. Was that the first year of the conference? What were some key sessions and feedback? We had planned for a conference ThinkTest 2013, which would have been the first of our initiative in this direction. We were very keen to bring James Bach to this part of India (and we still want this to happen). Unfortunately, due to unforeseen circumstances at my personal end and not having suitable alternatives, we couldn’t take this forward. We will keep the test community updated of future plans related to this. 6. In February 2014 you held the first meetup for Test Practitioner’s Club. What made you want to create this meetup? It always feels easier to work with people whom you have met and can relate to them as a face and as the vibes you share with them. All leading test conferences are still physical conferences not webinars. That’s because learning happens in many forms today and one of them is networking post conferences. With the thought of building a network of co-learners, we created a LinkedIn group – Test Practitioner’s Club and planned for its meet-ups that would help local testers to learn more from each other and get introduced to global platforms too. If we could – we would have such test meet- ups all over the country. We are planning to do one meet every month at the very least. We will keep posting the updates from our meets at my blog, for all to read. 7. I noticed that you are certified in ISTQB and QAI (CSTE). What made you decide to go for both certifications? Yes, that’s true – I have indeed done both the certifications. I went for ISTQB (in 2005) because until then I had no certifications. And under our training and development program, my organization paid for this certification. However, later on I went for QAI (CSTE), as it was not widely done so far due to its cost and students failing in it and this made it SMITA MISHRA
  7. 7. March 2014 - 07 - appear more respected and challenging. I paid for this one myself. 8. Do you recommend that all testers get both certifications? Are there any other certifications you would recommend to other testers? I would let testers decide this for themselves. But I can say it with reasonable confidence that these certifications do not prepare testers to handle serious testing problems. I enjoy being part of miagi-do and discussing actual test issues with real testers there and how to resolve them. I have heard very good things about BBST and most of the leaders I follow are associated with it. I would suggest all testers to atleast go through the content and format and decide for themselves. If learning is the objective and certification is not really the goal, I would suggest testers to go through James Bach’s site for RST classes and RTI (online) classes. 9. According to you, what is lacking in today’s commercialized training industry, especially in testing? With all due respect to the 2 certifications I did, I believe an ideal test certification would be one which would have more practical problems to deal with, as tests for their students and would involve real life situations of testing than focusing on glossary of testing and terminologies. Also, I would invest in domain led testing certifications for my team. So far, there are no such certifications available which would certify testers for regulated environments like those of aerospace and healthcare. I have heard of a new certification that Cem Kaner has come up with. I need to look into that one too, if it has domain based testing certifications. 10. What qualities will you look for in a candidate when you want to recruit someone for software testing job? I like to work with enthusiastic people who enjoy learning new things and working on new areas. This is true not just for software testers but front office executive, accountant, HR and Admin folks - all of them. Many today do not feel the need to look at the past experience and hire only for attitude. However, when I am hiring testers particularly – I like to see the kind of work they have done before and like to hear their stories of projects on exactly what was that they were testing and what were the key defects found. What was their approach and the key challenges faced and how did they bring up the challenges to the notice of relevant stakeholders and how did they do the contingency or mitigation as applicable. Formally speaking, we have 2 set of patterns – one for freshers and other for lateral hiring. For laterals particularly - We have a process that helps us map the required technical skills with the available resumes. So, anytime we have a need we look into our database of testers who have applied with us in past and from the matching resumes, we shortlist based on availability / joining time and expected compensation. Post this, we share certain project links with these potential candidates and expect them to submit us their Bug Reports in a given time frame. Finally, after a candidate is found technically suitable, we setup face to face interactions to evaluate more HR aspects and general communication. For freshers, it’s really about what they have to offer beyond their formal education. No firm expectations, as long as they are able to convince us of them being fast learners and having aspirational attitude, they might. SMITA MISHRA
  8. 8. March 2014 - 08 - 11. What will you suggest to people who want to join IT industry as software testers? I am very happy to be part of the software testing community. And I can only suggest the new folks that learn what is testing and then get into it. I find people to be doing testing for all the wrong reasons like – its easier that Development, it doesn’t need coding, timelines are easier and hence the work pressure, etc. None of this is true. Testing is becoming as challenging as any other part is with the ever evolving IT and When I look back, I think the biggest gap that I see during my initial work tenure was how closed and we were in our approach to testing. working in silos as a closed group and not being exposed to the whole world full of knowledge and awesome teachers and trainers willing to invest time in you. 12. Name few people you would like to thank, people who helped you directly or indirectly in your career as a software testing professional. I would like to thank my family for always standing by me – ALWAYS. But a special mention of my son here for being the world’s most loving and caring son and being very understanding when his mommy needs to work. At work front there are too many names and I would like to thank each good and bad interaction, because it has helped me grow in one way or other. Everyone I know – Thanks for being part of my life. 13. One last question – Do you read Testing Circus Magazine? If yes, what is your feedback to improve this magazine? Yes, I do read Testing Circus Magazine. I first heard about it in 2010 I guess when my manager Akash had been invited to be interviewed in it. But I never followed it then. I began to read Testing Circus since sometime in 2013 onwards. I liked the original format of the magazine. But the new format is truly amazing. I really enjoyed the editions coming out in 2014. And I think they are doing all the right things to get noticed and to put in right content. My only words will be – continue the good work. And maybe you can cover small meetups and group test events happening across the globe as a regular column. This could help other meetups to learn what more can be done and how to be effective. Blog/Site: Twitter ID: @smita_qazone SMITA MISHRA
  9. 9.
  10. 10. The simple thing is this - write your requirement as a test. With defined inputs and outputs. Expected results and expected (and unexpected) data. If you do this, your life will become bright, shiny, and you will live happily ever after. Or at least your soft- ware will become much much, much more reliable. And you’ll probably find out a lot of things about your ideas before you’ve invested in building unnecessary features and details. You can do this with high level requirements, such as business goals and overall objectives, as well as with low level isolated features, and everything in between. I think that the “everything in between” part is where we (software industry people) lacks the most care and insight about the importance of concrete, testable re- quirements. When it comes to high level requirements, we may have business people who do follow up business cases & objectives, i.e. test the results of the investment, at least I have seen it done once or twice. When it comes to very low level requirements, or mi- cro-requirement as my friend @spindelmanne call them, TDD do take care of it to some extent. Such as “When renaming item x the list will keep the same sort order”. It’s hard to separate micro-requirements from real business requirements sometimes. “What is really a valid input string here?” “How should we present the date format” etc but good developers generally can make some good micro-requirements decisions. For the “everything in between” requirements, we have a lot of work to do to make them testable. From what I know it seems as there are mainly 3 ways of communi- cating requirements today. Either you’re “agile” and have a loosely defined prod- uct backlog, filled with short user stories and then not so much more information. Or you have a heavy regulated requirements process, with hundreds of pages of use cases or “shall”-require- ments. Often with abstract statements such as (from real example): “Purchase has generated a receipt” Or the ad-hoc requirements: “Let’s send an email to the developer telling what I need to have”. Example: “We need to update the purchasing order receipt page. Right now it doesn’t show the total. The total need to be there. When can this be done?”. As a requirements analysts / project manager I have seen and practiced a way out of these three abstract, ambiguous, non-informative ways of communicating requirements. Much thanks to developers who serious- ly cared about taking TDD to the next level, and by having the chance to work with testers close by who taught me how to express what I want as test scenarios. I’ll share some examples from a previous project. March 2014 - 10 - - Ulrika Park Requirements People Need Your Help! Examples of Testable Requirements
  11. 11. A testable business requirement I was asked by the business owner to implement a fea- ture: “Cardholders should be able to edit the rights for a whole household to use the money on their bonus card” Since money and banking was involved, it was a bit complicated to implement. My first question to the business owner was: “why?” and how will you know it works?” After quite a lengthy conversation, he said that what he really cared about was that the money on the bonus account was spent. He didn’t want the money to stay on their bonus cards. “How can we verify that this target is achieved?” I asked. “Well.. he said. If the money is spent, then the feature works.” “So.. when in time is realistic that we can check this..?” “Well.. within 6 months we should have a better rate of spending the bonus money than now” he said. “Ok. So what do you mean by ‘better’?” “Hm…” he said. “I’d be content for now if 50% of the total money paid out to customers bonus accounts would be spent”. “Thanks for clarifying! Our feature could help out with achieving that goal. But to achieve this, other things are involved. Customers need to know about how to share bonus money between people in their family. How will they know? Marketing, customer service.. a lot of factors might affect if this feature is used by the customer.” “Yes, of course. But this is what I really care about. And when you have a feature households can use, we should do an effort to inform customers”. Good. Now we had a high level business goal, a testable business requirement. Even though our feature wouldn’t be the sole solution to make the business achieve this goal, knowing the target for sure helped us a lot in developing the feature. What to do when you don’t have access to the business owner? When maybe you just get a bunch of use cases from somewhere to implement? Well, in these cases I try to define my own hypothesis about the main goal and result. “This is how I / we have interpreted the target since we don’t know” and then show for those stakeholders I do have access to. Often I do get some feedback on my hypothetical business goal statement. Even “You’re totally wrong in your assump- tion!!” is good to know before developing anything. And you have a reason to ask for answers. Before testing or developing any feature, we have to know or make a clear defined assumption about the expected result for business. A testable middle level, user requirement So now we knew the business goal of the feature. The feature could be implemented in many ways, with op- tions from everything from printing and scanning paper forms to digital authorization functionality. As a requirements analysts, turning into a tests-before- development tester, I defined some user stories. (We did a lot of other things too to understand what solution might fit, but that’s another story). The main “middle level” user story: As main cardholder I want to authorize other card-holders in my family in order for anyone to use the money on the bonus account. Before communicating this to the development team, I start to think about.. how to test this? What would I test? I brought in a tester for a chat. The tester was busy with other assignments, but he did have a few minutes to help me out. And I asked him “How would you test this story?” “Identify scenarios” he told me. And with some coach- ing I made up some scenarios. (here is just a snapshot to keep the article short) Scenario 1: Give authorization to other cardholder in a household with only 2 cardholders. Scenario 2: Give authorization to other cardholder in a household with several cardholders. Scenario 3: Authorization process is actively canceled by cardholder Scenario 4: Authorization process is canceled by un- planned interruption etc. Then, exemplify these scenarios with Gherkin inspired syntax: March 2014 - 11 -
  12. 12. Scenario 1: Give authorization to other cardholder in a household with only 2 cardholders. Given that: Household has 2 and only 2 cardholders The 2nd cardholder doesn’t currently have the right to use bonus money Main cardholder has actively selected the 2nd cardholder The 2nd cardholder is >= 12 years old Expected result: Information is shown: “You have now given authoriza- tion to <2nd cardholders full name> with SSN: <2nd cardholders SSN>. The 2nd cardholder now has authority to use bonus. Scenario 3: Authorization process is actively canceled by cardholder Given that: Household has 2 and only 2 cardholders The 2nd cardholder doesn’t currently have the right to use bonus money Main cardholder has actively selected the 2nd cardholder and has entered external digital identification application to authorize and shuts down the digital identification application Expected results: The 2nd cardholder doesn’t have authority to use bonus Message to main cardholder: The change has been can- celed. These were just two examples. Doing this I had now: 1. A specification that can be used for testing 2. Got forced to find out the exact business rules regard- ing who actually could be authorized. The age limit, for instance, was discovered by defining the tests. By defin- ing test scenarios, I got a foundation to ask the right questions to domain experts. “Can anyone be autho- rized?” 3. A way to at any time go back and show for who it might concern (developers, stakeholders, customer ser- vice) what exact requirements and rules we built for. These are just small examples from a big domain, so I expect you have a lot of critique “Where are scenario x?? And those examples could for sure cover much more! And aren’t that statement a bit vague? What does ‘ac- tively’ mean really..?” The point here is not to give the full picture. For that I’d need to write a book :-) The point is to show that by defining tests while working with the requirement, the requirement got much more explicit. We saved time for our tester, who could focus on exploratory testing when time came. We saved a lot of time for developers who actually got the information ahead development on what would be tested, and what rules should be applied, so they saved a lot of rework. Also when having conversation with developers about the examples we modified them a bit, and removed unnecessary ones, or added missing. Some of the scenar- ios could even be quite easily automated while develop- ing, which also saved us a lot of time. So, what to do as a tester then? If you’re just not involved in creating the requirement definitions? Maybe the re- quirements are just handed off to you in one way or another, and when you get them they’re not testable at all? One thing you can do as a tester, is to make an effort to be included when other people are working with re- quirements. Offer your help! Only once I have been approached by a tester with this offer! It’s always me who’ve approached testers to help me with making testable requirements. Have you ever as a tester tried to offer your help – and seriously tried? To the guys working with requirements? And there are much more efficient ways than to say “If you need me you know where I am”. A tester could say something like this: “I know I’m going to work on testing for this project a couple of months from now. Is it possible for me to see some of the requirements already now? It will help me get ramped up quicker when I’m in the project. I’m in another project right now, but still I should be able to take one or two hours to look at what you have. And yeah, incomplete use cases or draft user stories will work fine too!” When you do get hold of some kind of requirements early in the process, take a quick look. Try to define some simple test cases or scenarios, and maybe you will get an opportunity to discuss these with the requirement ana- March 2014 - 12 -
  13. 13. lysts (or whatever role who works with the require- ments).. Then why not inviting her for a lunch or a virtual coffee break (or other social excuse) over Skype in case of distributed teams? Requirements people need your help! They just don’t know it yet. Who, if not you, will seriously invite to make their work testable? Ulrika Park is a requirements geek with a passion for testing, methods, learning & the development of prod- ucts & services within organizations and teams. With 15 years of experience in software development, management & business she now works at SmartBear. She believes in the synergy of people, software and quality thinking to change the world. March 2014 - 13 -
  14. 14. Look around in various IT companies, flip pages of some IT magazine, read blogs, see forums and you will be sure to find the today’s buzz word, the Cloud. You cannot find a word that has created so much of ripples than the Cloud. Very dramatically, the thinking process on providing computing resources has been shifted to Cloud computing. With this shift of focus, a lot of confusion has accumulated along the way in people’s mind. Companies are taking this concept very seriously. Many have moved to cloud, and those who are still on the “ground” are planning to move to cloud in next couple of years. People are still confused. They still are not comfortable with the Cloud thought. You ask anyone what a cloud service is and you still won’t get a very confident answer. Even if you get an answer, ask what benefits they offer and what are the disadvantages at stake, and you can see the person will be uncomfortable answering this. Make him skip the discussion by asking how secure they are, and it will be end of discussion. When it comes to testing applications, we are not fully sure whether to adopt the cloud approach of testing or not. What are the benefits we get and what are the hurdles on the way? Let’s take a closer look at the Cloud and how it can benefit out day today testing activities. So what is a Cloud? Cloud computing is a business and economic model. This model has been successfully deployed and executed for various material commodities since its inception, but in the recent years it has been formalized for IT products and services’. Let me try and explain the basic difference between a Cloud service and a Non-Cloud service. Consider that you have to move from one place to another. You can use your car OR use the services of a Taxi. Both of these vehicles have several similarities · Both are automobiles, having very similar structure and machinery. · Both provide basic functionality of transferring people / goods from one place to other. So where lies the difference? The difference is in the business model for the service provided by them. · When the car is owners, the owner has to pay for the fuel, regular maintenance and even possibly a garage. In turn, the car provides the service solely to the owner - you. March 2014 - 14 - - Vipin Jain - Anubha Jain Taking Testing to The “Cloud”
  15. 15. · When the car is a Taxi, the service provided by the taxi cab can be described as ‘Travel as a service’. Who owns the car and who pays the maintenance is not your concern. As a customer, you just have to pay to travel from your place to another desired place. No maintenance fees and no parking fees need to be paid by you. This responsibility lies with the cab driver. The Cloud is synonymous with the phrase ‘On Demand’. You pay only on demand (when you require it.) Cloud gained momentum when IT industry got associated with it. Today, we find that there is a huge range of products and serv- ices available on demand. All are “As a service” e.g. ‘Games as a service’, ‘Java as a service’, ‘Stor- age as a service’ and the list is endless. Fig 1: TaaS or ‘Travel as a service’ – A Metro Cab service How Testing and Cloud work together? Cloud-based testing offers a remarkable combination of low costs, pay-per use model and elimination of initial capital expenditures. The benefits, however, are more than just cost effectiveness. The non-cost factors include on-demand flexibility, a respite from holding various infrastructure assets, enhanced collaboration, higher levels of efficiency and, most importantly, reduced time to-market for key business applications. Economically the vendor and end user gets huge benefits from the Cloud. This comes directly from reduced subscription prices of any product or service. Other benefits are: · Cloud Apps are scalable: The elastic business model, as it is popularly known as, can be customized based on the requirement · Auto-Provisioning: Depending on needs of end users, various Cloud vendors provide and withhold the offerings in a manner which is in an automatic and self-serving format. · Unlimited resources: At least the end user thinks and feels this. The services / products are available as and when they are demanded and in the required quantity The Cloud computing model should be coined as ‘Green Model’ as it maximizes usage of resources and minimizes wastage making it environment friendly. Current state of testing in cloud Many companies are still taking a cautious approach with cloud computing. This is not the case with testing however. They are largely ready for testing in cloud and following reasons will account for that readiness: · As we all know, testing is not a one time, but a periodic exercise and each project will require a new environment to be set up. If a company creates a Test lab, it typically sit unused for longer periods, resulting in a waste of cost, electricity and space. Lots of published reports indicate that more than 50% of the March 2014 - 15 -
  16. 16. technical infrastructure for testing remains underutilized. · Since testing still being considered as a must but non-critical activity for business, taking testing in cloud premises is pretty safe as it doesn’t include important company data and has minimum impact on the organizational business activities. · Today’s applications are increasingly getting dynamic, more complex, distributed across continents and more component-based. Testing them is getting more challenging. For instance, with mobile and Web applications testing, testing needs to be done for multiple operating systems and regular critical updates, various browsers and their versions, variety of hardware and a large pool of concurrent users to understand real time performance. It is pretty difficult to follow the age-old approach of creating so many in-house testing environments. These will become very complex and will need huge capital and resources. Fujitsu in a 2010 research suggested that testing ranked second (57%) as the most likely workload to be put into the cloud after Web sites (61%). The on-demand provisioning by Cloud addresses all the above explained issues with one click. On top of it, the effort and resources saved by using cloud can be redeployed for core business functions. The Cost Factor Economic benefits are the main factor influencing companies to take testing to cloud. Another 2010 survey by IDC hinted the same. As the global economy recesses, companies continue to find ways to regulate costs and improve ROIs. Cloud testing reduces the unit cost of computing. March 2014 - 16 -
  17. 17. Small and medium-size businesses that cannot afford high costs find cloud-based testing as a new lifeline. The companies no longer need to invest in infrastructure and software licenses. They also do not need to worry about various configuration issues and maintenance of test environments, and pay only for what they use. Beyond cost Benefits? Cloud is not all about cost saving. There are lot more benefits companies can extract using it. - A standard infrastructure and pre-configured software is available, reducing efforts in getting servers and licenses. - On-demand provisioning helps companies to think forward instead of spending time to set up test labs. All testing resources required for testing exist within the system and can be called upon instantaneously. - Better analysis and control are offered to test teams to build and execute their tests and identify the bottlenecks. This helps in identifying possible runtime bugs a lot before they are actually found. - It’s a great concept in motion between geographical distributed teams. Once a tester logs in and runs a test, the results are available over the cloud. The developer can then assess it and fix over the cloud itself. This eliminates back-and-forth communication between teams. Limitations - Lack of standards: Absence of universal/ standard solutions to real world problems is a big issue. Each cloud provider can have its own hardware, operating models and prices and may or may not offer any interoperability. This poses a huge challenge for companies when they plan to switch to a new vendors - Security issues: Security in the public cloud is still a huge concern because the data may be stored in a location which is outside a company’s legal and functioning jurisdiction. - Usage: The everyday usage costs increases very rapidly if testing is done without a proper usage of cloud-based test. Though pay-as-you-go clouds are used, it can be expensive if the testing is out of sync with requirements. March 2014 - 17-
  18. 18. - Performance: It is always an issue since public clouds are used by many users in parallel. Situa- tions got created a company had to wait for the required bandwidth to execute their tests. Conclusion This paper tries to explain the benefits and limitations associated with taking testing to Cloud. We have tried to explain that why companies should start small and gain confidence slowly to capture maximum benefits of cloud-based testing. Once they believe that it has led them in speeding time to market, lowering of costs and ensuring standards compliance, they can go big. Using of pay-as-you-go or on-demand services intelligently and efficiently, companies can reduce cost of operation and ownership. Companies should pilot cloud-based testing as early as they feel comfortable before going to mainstream testing. Bibliography - “Determine if a Cloud is Usable,” blog post, Bloomberg Businessweek, Jan. 31, 2011. - “Solving the Challenges of Enterprise Mobile Appli- cation Development With Cloud-Based Testing,” blog post, - CIO, Feb. 17, 2011. - Rajagopal Sattaluri, “Testing Considerations for Ap- plication Migration to Cloud Computing,” - Cloud Computing Journal, Feb. 8, 2011. - “Cloud Computing: The Good, Bad and the Ugly,” blog post, Dynamic Data Inc., Feb. 1, 2011. - “Cloud Testing, A Growing Trend,” blog post, So- nata Software, April 4, 2010. - Nivedan Prakash, “Cloud Testing: Attracting De- mand,” Express Computer, Feb. 1, 2010. March 2014 - 18 - Vipin has dedicated last 10 years of his professional career to the software quality and testing. Currently working with Metacube Software, India, he developed his key skills in developing automation frameworks and automat- ing applications. With a proven record of implementing and refining test processes for various clients across the globe. he is the author of several articles and seven well sold books in India. He is pretty active within the software testing community by speaking at International and national conferences, writing articles and contributing to various blogs and forums. Anubha is working as Associate Professor & Head, IT department, Internation- al College for Girls (ICG), located in Jaipur, India. An academician for last 11 years, she has been involved in teaching and mentoring several students in the field of Computer Science. Knowing that teaching is the best form of giving knowledge back to society, she worked as a lecturer in Subodh college, Jaipur before settling in her current role at ICG. She is currently pursuing her PhD, in field of Information architecture. Anubha is the author of 9 books and a regular contributor in various forums.
  19. 19. To Subscribe Click Here Still relying on reading Testing Circus from tweets & facebook updates? Subscribe just with your email id and get the magazine delivered to your email every month, free! Testing Circus March 2014 - 19 -
  20. 20. I recently gave a talk at a conference. I learnt many things from my interactions with the attendees at my session. One specific thing that stood out for me was how – testers no matter how intelligent, smart, critical thinking they are, still fall into the common pitfall of using test cases for communicating testing progress and test coverage. If anyone reading this article, is still under this mindset, I would ask the following questions- · Do the test cases cover each and every scenario, each and every part of the system under test? · Can you give accurate information of testing effort in terms of test cases? Does it really make sense? · Are we trying to hide from reality and give some sort of number to make us and our project managers feel good when we report the number of test cases covered to communicate testing progress? · What do you actually mean by test coverage? Can any other information about the system influence this? Here is the common pitfall of completely relying on test cases. Firstly, 100% test coverage of the system is impossible. “Exhaustive Testing” is a misnomer. This being said, if anyone says “I have 10000 test cases, I executed all of them and thus all my testing is completed”, it is flawed because it is impossible to cover every scenario of a system by just executing all the test cases · What if there are some hidden scenarios that the tester still hasn’t uncovered? · What if another system influences the system under test in a different way than expected? · What if the system behaves differently under differ- ent situations and our test cases did not cover this? ….and so on. There are so many “What if” questions to ask ourselves. It’s an endless list. So the point is, NO; we cannot possibility write every possible test case to cover each and every scenario of a system. This is where as testers, we think about testing approaches complementary to just executing test cases like exploratory testing, automated testing, combinato- rial testing and other approaches based on the context and scope of what we intend to cover. This may help to uncover other weird scenarios or trigger unexpected outcomes. With this being the case, it may not be valid to say, “I have 100% test coverage because I executed all the test cases”. Secondly, you may then ask, how should I give up- dates on testing progress to my stakeholders? In my experience, I usually do not have a problem in explain- ing testing progress to my stakeholders even if they are old school and believe “test cases are the solution to all the problems”. I would of course have some test cases that I execute, but also do lot of complementary testing to test case execution. Finally, I give the approximate percentage of modules covered/tested in the system as my testing progress. So for example- If there is a system A, I generally split it into different modules M1, M2…….Mn (test areas). I use different testing approaches to test these modules March 2014 - 20 - - Raj Subramanian A Common Pitfall Of Test Cases
  21. 21. and give an update in terms of the percentage of mod- ules I have covered in the system. (This is a general approximation as there is always a possibility that I haven’t thought about some other module in the system which may cause problems. To err is human.) So, say if I had covered M1, M2 out of the 5 modules I have identified i.e M1, M2……M5, then I may have cov- ered about 40% of the system ([2 modules/5 modules] * 100, in case you were wondering how I came up with this percentage ) . This is how I give my updates on testing progress. This by itself helps to give the stake- holders some level of visibility into testing progress and helps to make decisions accordingly. To summarize, I think as testers we know it is ethically wrong to make false claims. That being said, just using test cases to estimate the amount of testing progress and test coverage falls under this ethical problem as we are giving wrong impression to our stakeholders who trust us. Doing this tarnishes tester credibility which is one of the most important factors that influence a tester’s work and helps to establish trust among his peers. Raj Subramanian is a pas- sionate tester, who has previous experience work- ing as a developer and moved to testing to focus on his passion. He gradu- ated from Rochester Insti- tute of Technology with a Masters in Software Engi- neering, worked as a developer for a payroll processing company and currently works as a test engineer for a major insurance company with specific focus on mobile testing. He has been pret- ty active in terms of learning and contributing to the testing society by speaking at conferences, writing articles, blogging and being directly in- volved in various testing related activities. He currently lives in Cleveland, Ohio. Raj blogs at And tweets with @epsilon11 March 2014 - 21 -
  22. 22. moolya sucks we test fast and don’t know to make more money from our customers. we are like this only
  23. 23. You have been asked to test the latest Mobile based application, created painstakingly by your organization over the last few months. All you have with you is your exclusive Smartphone, which you saved for, and bought, over the last few months / years. The choices are now with you, either beg/borrow/steal from col- leagues or do the right thing. Fire up that web browser on your Smartphone, open up ‘Google’ and have a go at searching for ‘Mobile Test Partners on the Cloud’. I would generally as a rule go for the second option and not put ‘my precious’ phone to the test on un-tested applications. What you achieve when you go through the search, is a wide variety of web sites and commercial vendors of- fering this facility to you, at a fraction of the cost that you would spend in setting up your own facilities and test lab. You need to now make an informed decision by researching and figuring out which one of these would serve your purpose the best and also give you value for the money spent. Mobile testing has come a long way. From the initial fragmented scenario of having to check on each kind of screen resolution and phone type and screen size, add to that the variety of mobile browsers being offered by the various vendors, to the current situation of having apps created using HTML5 versus native apps. Adding to the general confusion is the non-app area, where companies wish to check the overall ‘responsiveness’ of their “web pages” across the same wide variety of devices, which includes [and not restricted to] iOS (iPhones and iPads), Android (Tablets and Phones – low end and high end), and recently the Windows 8 (WP8 devices and Windows RT devices and desktop’s with Windows 7 & 8 included) and Chrome OS (mainly low end laptops & devices). This vast array of devices also brings up with them an equally confusing array of browsers along with them (at the last count it was something like 9 browsers and growing). Faced with a similar dilemma, we take refuge in the well-trodden path of checking on the search engines for a Web Accessible Mobile Testing Tool (applications and browser based systems), that would be able to serve our purpose and not cause the management to jump from their warm seats when they finally receive the bill for the services used to test their precious new mobile application or website. To get through all this, I listed down a few of the up and coming offerings which provide a good cross-section of the devices and are reasonable in costs. Although these cloud platforms do provide a service which is extreme- ly useful, keep in mind, you might need to make use of physically handling the device to run certain tests, which cannot be checked with automation (but there are ways and means to handle this, so don’t be disheart- ened). A hybrid cloud installation in these situations can be one of the simpler solutions which come handy. A hybrid cloud provides a small subset of devices at your physical location and the wider variety at a remote location. Of the Could Mobile Test providers, the most important thing to look into is if they provide you with storage for your tests and a way to run the automated tests that you have so painstakingly created for your application from within the cloud infrastructure. This is along with the use of making sure that these tests run March 2014 - 23 - - Gagneet Singh Mobile Application Testing Using the Cloud Infrastructure
  24. 24. over a wide variety of network speeds simulated to provide the 2G, 3G, 4G and Wireless speeds prevalent in the World Wide Networks around the world. Practically a Cloud Based system should be able to cater for most of the requirements outlined above; and the good news is that most of them do exactly that. A few of the prominent ones which I have used recently and the ones which come to my mind are:,,, HP mobile testing lab, to name just a few of the upcoming entrants in this field, where by Gartner estimates, there are 5.6 billion handsets present. has stood out as an ex- cellent upcoming product, being run by engineers who have previously worked with Nokia and other top mo- bile hardware development companies. They have built the framework for testing on mobile devices from the ground up and have got some really cool features which go with it. (Disclaimer: I know one of the Co-Founders as a colleague from my previous companies). I loved the way they have handled the storage of tests and results on the site itself, along with providing features such as mock location maps, allowing the users to experience the different states of the application in separate geographi- cal locations. They have recently launched a feature of having multiple browsers and simplified their launching from within the cloud interface, making things easier for users again. And have launched the latest Android ‘KitKat’ version with the Nexus5. All in all, an excellent package to go for, with reasonable rates. All said, the main purpose of having a cloud based mobile test experience is important for any company wanting to launch its web presence these days. With the advent of HTML5 and other technologies like Founda- tion (more on this later), the web has become a place where people love a responsive site (or application across iOS, Chrome OS and Windows [Phone] 8) that caters for whatever device they are working on, and does not have the staid look and feel when they change from a Desktop -> Laptop -> Handheld Tablet -> Smartphone. They want to get the feel that the web site developer / organization has done their homework and provided them with a site where they do not have to pinch-in and out, just to read content. With the expansion of Smart- phone markets in the developing nations and organiza- tions wanting to tap into the ‘billions’ of people there to advance their products, it has become imperative for these organizations to go through the process of Mobile Testing their web sites and mobile based applications across various low end and high end devices to be useful. Sites like,, etc. are making it easier and faster to send out mobile products into the market by providing the required platform at a fraction of the cost of actually acquiring a complete mobile test lab. That said, you would still need to work out a small subset of your tests at a physical location, but I am sure that this would also become possible over the cloud. With the advances these start-ups are making with tech- nology and improvements on their own feature sets. I certainly would be looking forward to features which provide a friendly interface and let me run across multi- ple devices; and for that the Mobile Test on Cloud pro- viders are definitely a better option than trying to keep up with the influx of the hardware been thrown into the market by all the top end and low end hardware manu- facturers around the world. Gagneet Singh has been working in the Quality Assurance/Test field for the past 8 years (with an additional 4 years in Sys- tem Tools development) and has been involved with companies such as Toshiba, Adobe (Macro- media) , McAfee, Oracle, Yahoo! and recently Mi- crosoft. He likes to blog and to write about the experiences he faced in the various organizations and situa- tions. His work has mostly been with Automation Testing, along with Performance QA. Also, Secu- rity testing over the much hyped “Cloud Comput- ing” (using Hadoop and Azure) has figured in his work area. Currently working out of this place they call the “Down Under“, where he lives in Sydney, New South Wales! March 2014 - 24 -
  25. 25. I want to define what I mean by ‘late’ and ‘early’ and then go on to give some context from a firm I once worked for – let’s give them the fictitious name “Lin- man Manufacturing” for convenience. If I don’t define my terms, we will all get lost. I will use ‘early’ and ‘late’ in an almost intuitive way, although perhaps not as everyone uses them. They are not of the Humpty Dumpty variety (Alice Through the Looking Glass by Lewis Carroll: 'When I use a word,’ Humpty Dumpty said in rather a scornful tone, ‘it means just what I choose it to mean — neither more nor less.’). So, for me ‘early’ is before we were ready, and ‘late’ is after the planned date. These are pragmatic terms – let’s not get argue about them now. My hope is that you will see the usefulness for yourself. To start with, I need to say something about project deadlines. First of all, it is not testers (or test managers) who determine when a work product is implemented (into PROD, obviously). Test resources can speak into, or provide information, that helps others make that kind of decision, but the final decision lies outside “testing”. Secondly, we per- haps all recognize that if the decision were left to tes- ters, work products would not get implemented. We testers are a pessimistic breed, and tend to focus on things that don’t work. It is not always the 390 require- ments that work that get our attention, but the 1 that does not. We concentrate on that. Lastly on deadlines, some deadlines are more impor- tant than others. Take the day I am writing this article. ‘Tomorrow’ I am preaching at a local church, and some- time before standing up, I need to have prepared what I am going to say. That is an immoveable deadline. Similarly, the deadline to ‘post’ this article is midnight tomorrow. If the editor gets it by then, it will be consid- ered for the next edition. Otherwise, it will be a candi- date for the edition that follows. This is another deadline that has significant consequences if it is missed. Some software projects have hard-wired dead- lines that cannot be missed. Perhaps there are legisla- tion changes that need to be complied with. The introduction of the Euro is a good example. However, even when there is a hard (i.e. cannot be moved) dead- line, there is sometimes wriggle-room around the edg- es. Some parts of company accounting in Euro-zone countries needed immediate changes when the Euro was introduced. Other parts did not – activities around the end of the company financial year may not have been needed to be correct on Euro day-1 – only at the end of the company financial year. So even where there is a hard deadline, some parts may not need to be ready on the first day, whilst other parts must be available, and be seen to be working. Right. The pre-amble is over. I want to tell you about my time at “Linman Manufacturing”. In my many months working for them, there were 8 major imple- mentations, where significant new functionality was introduced. Although each implementation was sepa- rate, there were both implicit and explicit points of interaction; data was loaded into and extracted from the same database, with some common database tables used. As time progressed, later functionality relied up- on the data that had been introduced in earlier phases, and was loaded month-on-month. I want to concentrate on the last 4 major deliverables, giving the targeted delivery date into PROD and the actual date. March 2014 - 25 - - Peter Morgan Better to Implement Late Than Early How Quicker can mean Slower
  26. 26. I mentioned that there 8 major implementations. Of course, there were other times that software was promot- ed into the PROD environment. User-requested changes, software platform upgrades, and system tweaking oc- curred throughout this time frame. But if the project team as a whole had been asked before each phase tabled above, for phases F and G the answer would have been ‘yes’, and for phases E and H ‘no’. In the time that was given, the best testing, based upon risk, had been carried out. But there were huge swathes where the answer at best was “we don’t know”, and some parts were clearly not working. The architecture was basically to take mul- tiple-format files on a regular basis (daily / weekly / monthly) and to load this into a standard star schema data warehouse, and then extract the summary informa- tion. For phase H, data had been successfully loaded many times, but the reporting layer only developed in outline. Now it is not until data is output that it is possi- ble to tell whether it was loaded successfully (i.e. it is fit for purpose). And so it proved to be. For phase E, there were immediately amendments re- quired, both to data loading routines and to the report- ing layer. Seven weeks later (and after seven implementations), the solution was stabilizing. Let me tell you, those seven weeks were interesting. Some of the input processes could not be run for part of that time, meaning that reported information was only partially correct, and it was not until nine weeks after the date that users could gain any confidence in the output. January 2009 was the first time that month-end reconciliation (for December 2008) could be attempted. That is quite a long time after the actual implementation date. If anything, the situation was worse with phase H, and interestingly enough from the table above, this phase was both ‘late’ (after the targeted delivery date) and ‘early’ (implemented before it was ready). The first weeks were spent loading a backlog of data, with 10 months of data to load immediately. The reporting layer quickly highlighted ‘problems’ with 7 of the 16 data feeds – problems that could only be rectified by remov- ing any data loaded to date, and either re-engineering the data creation process (before it was made available to load), or changing the data load procedures. There were times that it seemed data was being backed out faster than it was loaded – even though once a data feed was identified as ‘wrong’, nothing else was loaded for that feed until the problem was resolved. A big score-board showed what data had been loaded for which feeds on a month-by-month basis. March 2010 (when data for Feb- ruary 2010 was loaded) was the first month-end with stability. There are business reasons for ensuring that “something is delivered”, and that progress is seen to be made. However, for “Linman Manufacturing” the information loaded into the Data Warehouse was for long-term stra- tegic planning, not for the day-to-day business opera- tion. There were, in one sense, no unmovable project deadlines. If you have a choice, never implement a software solu- tion ‘early’. Doing so makes for a frenetic period of activity just after the implementation date, and some- times aborted product launches. If you take longer, it can mean that the software is available earlier. When that happens, staff can be truly released to go onto the next project without constant drag-back from activities that should have been put to bed! To do otherwise means that quicker is slower. March 2014 - 26 - Peter Morgan is a testing professional who has been involved in the ICT indus- try for more than 30 years, and worked in the free- lance marketplace for much of that time. His time has sometimes moved from testing to ‘develop- ment’, but he would add “always using the mind- set of a tester”. He is passionate about testing and a firm advocate of testing qualifications. An en- thusiastic speaker and author, Peter tries to base his output on hands-on experience, attempting to relate fine sounding ideas back to how it will affect Joe or Jane Tester in their everyday working lives in the war of attrition that we call software testing. He is a regular at EuroSTAR conferences, and is speaking at Belgium Testing Days in 2014 – the forth year in a row. At this time of life, Peter offers experience, and can sometimes say when offered a tricky problem: “Been there, done that, and here are 2 or 3 options that may, just may, work in this situation”. He continues to learn, adding technical skills to his impressive range or hardware / software / business sector portfolio.
  27. 27. I don’t know if anyone has noticed but the world we live in now is different from the world we used to live in just 5 years ago. For me, the change that occurred is far rapid than anticipated, because if we refer this to a 20 or even 10 year past leap then most of the skeptics would say “Why not!, it should have changed!” but the shortness and the speed of this period leaves no room for being skeptic or naïve – We simply need to understand and adapt! Mobility has broken free from just being a device to communicate into an instinct. We are using our hand held devices in so many different dimensions other than it was meant to be that it is changing the complete concept of computing and networking. Hand held devices are now our nodes to the clouds; we are now the new form of terminals; “The Human Terminals”; the value to the word “Touch” has enhanced itself from Touch screen, and then within a very short time it again reboots itself to “Human Senses”! It feels like a history in making where the systems which are categorized as something purely mechanical and electrical combination of components with mix of complex logics and design are re-emerging to sapiens, like an event in space and time; As I see it, the systems are reaching to our senses, our beliefs, daily lives and simply becoming an extension to our self being, like an implanted new body part. Gadgets are now not need to be held in hands; they are wearable, and soon they will represent a mere extension to its users. The human mind is now adapting these effects; see for example our children; how they have adapted the use of touch screen, games and the use of smart phones. How people are getting aware of the communication and sharing bounds and exploits over social networks; Where to “say and share” what is now dependent on “where and how” you are registered as a user; Twitters, Facebook, LinkedIn, Instagram, Four Square, Google, pint it, flicker and so on. We are not structured anymore; we are living multiple lives under multiple scenarios. Somewhat, this is not based entirely on our choices, this is how the environment around is “turning out to be” – and we have to live it! We are now thinking! Or in other more adaptable senses we are now becoming Context Driven! These factors are effecting what we used to know as the solutions and systems; these factors are now playing a major role in provisions of the right solutions to the core requirements and needs of the users. And… the same factors are affecting how we “Test” these systems; In a way, we are now moving to an age where due to the magnitude of the contextual effect on any scenario, generic approaches will simply fall apart and fail! – In so many words, we need specialists! We need empowered human beings and their expertise to address the right issues at the right time; we need to identify the right coverage to find the important bugs, and mind you – We need to identify ALL OF THEM! Under several scenarios and contextual situations and due to certain trendy and cultural bonds we have not yet succeeded in creating our own identities as Testers. We are still based on the “Types” and “Approaches” scenario. Where, Testers when tend to define what they do, start explaining the testing types and the approaches they follow. They tend to list down several technical tools and programming languages as skills, and not as “Tools” to support testing. In this stream of self- March 2014 - 27 - - Arslan Ali 7 Types of Testers What is Your Identity?
  28. 28. discovery they forget the very scarlet thread they cross from being a “Tester” to “being” a Coder. Well don’t worry we have worked that out as well; James Bach has already published twice about the types of testers there are after receiving several queries regarding the latter; in his recent blog post, he mentioned the following as the testers’ types: (I am quoting him here, rather putting in my own words – it is better that way) (SEVEN KINDS OF TESTERS) · Automated? Manual? There is no such thing as manual or automated testing. It’s all just testing. Testing is often supported by tools that attempt to simulate user interaction with the system. This is what people call “test automation” even though it is only automating a crude approximation of one aspect of testing. If you have the ambition to be a one-man test team, it is extremely valuable to learn how to make your own tools. · Exploratory? Scripted? There is no such thing as an exploratory or scripted tester. All good testing is exploratory to some degree and scripted to some degree. · Tester. This is a testing generalist who can contribute to any test team. Sometimes called a QA analyst, QA engineer, or test engineer. I prefer the simplicity of “tester.” · Omega Tester. The omega tester (which I sometimes call a test jumper, after the analogy of a paratrooper) is one who can do anything. An omega tester is equipped to be the only tester in a project team, if necessary. Omega testers can lead testing, or work with a team of other testers. I am an omega tester. I aspire to be a good one. · Performance Tester. The performance tester understands the mathematics and dynamics of the performance of large-scale systems. They use tools that create high loads and measure the performance envelope of systems as they scale up. Performance testers often don’t think of themselves as testers. · Usability Tester. The usability tester is a bit mythical. I have met only two dedicated usability testers in my entire career, but I have seen more of them from a distance. A usability tester specializes in studying how users feel about using and learning a product. · Security Tester. Security testers also often don’t think of themselves as testers. Security is an exciting, specialized form of testing that requires the mastery of a great many facts about a great many technologies. · Testing Toolsmith. A testing Toolsmith is a programmer dedicated to writing and maintaining tools that help testers. This is what a lot of people would call an “automated tester” but you better not use that term around me. · “SDET” Software Development Engineer in Test. This means a full on programmer who does testing using his programming skills. On the other hand there are several characteristics as a tester which can stood up and create an exclusive tester identity; For example, boundary Testing heuristics; now I know that several of us have used this term in our CVs and have somehow read or taken part in forums to talk about this type testing; but how many of us have actually bifurcated this into “Galumphing”, or “Steeplechase” or even “Leap and Creep” - I don’t think any of us! OR While discussing about product coverage we have used the term “San Francisco” depot instead of discussing the missing out requirements, and blaming the developers of not providing the right specs or cursing the Implementation team of not discussing the right requirements; that is what common professional does! Why not stand out and use your own language? I have seen professional testers asking questions and worrying about “Testing without Requirements” or “Finding the right bugs” within a short period of time, but I have not seen any of them discussing about using or creating their own Heuristics? Why not? There is also no denial in discussing Quality and its criteria, but have we ever tried to use and educate ourselves with the use of the “Quality Criteria Heuristic called “FEW-HICCUPPS”? Try and do that! March 2014 - 28 -
  29. 29. Let us understand now. Why can’t we move beyond being a technological freak to a Tester in a meaningful term; Does the mention of the very word “Heuristic” is not a mention of “Tool” in your sense. Technology is a wonderful thing, being technical is in the very genes of a computer professional; but as testers we need to identify ourselves as someone with a niche of being a tester and being technical at the same time; Being technical comes with the package; So why not mention who you really are? “A Tester”! Empower yourself, your language, attitude and reputation as tester. Learn Un-Learn and the re-learn! Try and improve your language as testers, differentiate yourself from the peers of Developers and Implementers. Yet sewed in and work for their support Or I can only say this then; you are in the wrong game baby! Arslan Ali has more than 14 years of Experience related to IT, Industry and Training Institutions with exclusive experience of 5 years in teaching various disciplines and projects in IT Institution. He has worked in various roles in capacity of Software Engineering, Software Tester, Trainer and Quality Assurance Roles. The Major focus of his expertise lies in Coordination, Implementation and Testing of ERPs and Customized Applications. He is also a trainer for Context Driven Testing for various companies and individuals. Arslan is currently working at Sidat Hyder Morshed Associates ( as a “Sr. Consultant – Information Solutions; but beside that he is also an active founding member of TestersTestified ( (@testtified), Outtabox! ( (@OuttaBoxPk) and OISOL – Open Integrated Solutions ( as a training consultant for Software Testing and Context Driven Testing Workshops. You can follow him on twitter @arslan0644 and on LinkedIn at March 2014 - 29 -
  30. 30. Interaction » Insights » Opportunities 11th International Software Testing Conference Today’s tester is a unique being, driven, not by deadlines and mundane tasks, but by a relentless curiosity and a fascination with possibilities. Awaken the tester in you.This is where you want to be; this is NOW. Walk away with exciting ideas and quality insights from other industries. Make your mark by challenging the existing paradigm. Most Popular Conference among Start-Ups to Fortune 1000 Companies ExcitedtoparticipateasaSPEAKER/ DELEGATE/SPONSOR? orcall+91-80-40818888 Over 900+ delegates from India and abroad attended in STeP-IN SUMMIT in 2013 Testing NOW STeP-IN SUMMIT2014 Asia Pacific’s Largest SoftwareTesting Conference Produced by: Hosted by: JUNE 25-27, 2014 AT BANGALORE & JUNE 18 & 19, 2014 AT PUNE & HYDERABAD CONFERENCEPROGRAM View Past Conferences View Past Speakers’Gallery Pre-conferencetutorialworkshops(twoparalleltracks)@Pune,HyderabadandBangalore TargetAudience:TestLeads,TestArchitects,TestManagers,TestAnalysistsandConsultants Conference@Bangalore TargetAudience:CXOs,VicePresidents,Directors,BusinessHeads;Customersandend-usersofITproductsandservices; TestManagementprofessionals–Managers,TestLeads,Software,Test Engineers;Quality/SEPGManagement Professionals–Managers,QALeads,SoftwareQAEngineers; ProductDevelopmentManagers;ChangeAgents 18,19,25 June,2014 26-27 June,2014
  31. 31. BOOK WORM’S CORNER March is the beginning of the appraisal season; that’s why 50% of the organization actually wonder if they should be continuing in the or- ganization or not. That’s why I decided to crawl through “Don’t Hire the Best”. This book is written by Abhijit Bhaduri. In case you are wondering if it is a misguide to hiring the right team, then you are partially incor- rect. This book preaches and lets you come to your own conclusions. The book tries to sell itself by lecturing about good hires and how to make good hires, however, it tries to draw the right line between “hiring right” and “hiring the best”. Unlike many other interviewing books that are filled with choc-a-bloc aptitude and problem solving questions, this book also tries to examine the organization culture and why culture fitment is important for the company. The case studies with the standard disclaimer of having been altered to protect identi- ty is very useful and causes the reader to ponder as to how he could have done things better. Reasons as to why it is important to create a fitment between the interviewee personality and the management ex- pectations, individual traits that need measurement, why making the wrong hire call can result in financial losses are stories talked about in depth in this book. What did it teach me? After reading the book, I ensure that I pass on a set of good and bad candidates to my interviewing team; that varia- tion causes them to be on their toes and interview without bias. If I send only good candidates their way, a form of bias sets in over the time which would result in all of us overlooking the most obvious reason for not hiring the candidate. Why should you buy this book? Because it’s not free; in addition to that, you will learn valuable lessons on how to create a team, why the culture fitment is important for your teams and why you need to ask questions beyond the role competencies while forming teams. When is it important to look into a resume to know what he’s capable of and when should you look beyond the resume? You will ask yourselves these questions if you were to read this book. I hope, after reading the book, you don’t hire the best anymore; you will hire “better than the best”. Love, WoBo March 2014 - 31 -
  32. 32. Request a free demo by sending us an email at Become our fan - March 2014 - 32 -
  33. 33. Part 39 March 2014 - 33 - Random Musings My 39th month with this magazine; that's when I real- ized that I had inadvertently become a career counsel- lor for most of my family members who were in their final year of college. Is there any scope for "Java"? Should I learn ".NET"? Is the industry scope for "SQL" or "Oracle"? With so many questions throw at me from family members, I've stopped visiting any family func- tions or any event where I get to meet my extended family. Quoting client-meetings and a workaholic manager asking for meetings at mid-night, I manage to escape attending most of these functions. And then there is that irritating cousin of mine who keeps brag- ging of his topper credentials in schools and constantly asks me if there's more scope for "development" or "testing"? I have a good mind to tell him that "If people like you join Development teams, then testing career will have a lot of scope", but I dare not. Why do I write this column? --- "When a tester logs a bug and engages the developer in a sadistic discussion asking him to fix the bug, the tester is also torturing himself on some levels; the tester's life is such that he needs such discussions to survive his job" --- This is what a famous manager had to say about corporate culture for testers. Though there has been some amount of hue and cry against this column, there are a few instances that agree that this column unearths what lies beneath. The real thought of documenting this diary is to try and unmask not just the bad guys, but also the horrid practices that have created fake testers all over the place. I also fell prey to fake testing sometime back and believed in the concept of sanity testing, smoke testing, BVT testing and that all of them were different testing types; I also thought that I could become the greatest tester if I cleared all those exams. And now, I stand before you, as the FAKE TESTER. That's what I stand today. And I hope my writings will stop creating more fake testers. So far, in most columns, I have (unsuccessfully) tried to provide an insight as to what's wrong with today's testers. Most of them can be categorized as · the inability to find good and authentic testing teachers · how there are some people sitting in manage- ment who actually decide and state what test- ing is all about and the entire company thinks that's what testing is all about · the low pay of the testing engineer -- how many of them do you think have equal opportunities at growth and how many of them have 40 hour working weeks? · And the inability of the testing team in any company to function as an independent body, but function as a sub-body of another manage- ment function (development or support or mar- keting or sales or .....) there is no independent person or body leading the testing vision for the companies. And at the end of a year, many testers realize the disdain that others have for him; the disdain becomes reality. When this is pointed out to the management, the management don't try to make good the wrong things; they do some actions that makes the tester's life from bad to worse. For a long time, the sole objective of most testing com- panies seems to be a single agenda --- bill the client against the number of test cases executed; to showcase
  34. 34. smartness, they record and replay some of the testing and quote savings in the name of automation. The clients cannot back out due to the legal agreements signed at the beginning. An impact of this is that the test results for periodic testing that's being sent to clients cease to matter; and regardless of the frequency of app usage by customers, testing companies continue this trend. 1 look at the testing economy would tell you that 2.5 Billion Dollars was the money given to off- shoring companies for testing alone in the 3rd quarter of 2013. With new testing companies sprouting all over, this is set double in the next few years. And the testing companies have also started to exploit this. With substandard test cases and haphazard testing, they make hefty profits without providing any value by testing. An indicator of the lack of quality can be seen by the most unimportant projects being handed to the off-shoring testing companies; this can be seen by the very fact that in most of the offshore company cases, not much of business revenue would be lost if the offshore projects are closed down. Secondly, according to the reports of our special correspondent, in the last 5 months, there has been ZERO new product launches in non-Asian markets due to testing done from here. I am unsure if others realize this, but looks like India's testing industry has started to face its worst crisis; my personal thought is that the rest of this year will be gone in people realizing their mistakes and the indus- try auto-correcting itself. What do you think will hap- pen? As I mentioned last month, only time will tell. March 2014 - 34 -
  35. 35. March 2014 - 35 - Anna Royzman Context-Driven Scholar Sharath Byregowda A passionate software test professional, Principal Consultant at SQS, London in their agile group. Mike Lyles International speaker, writer & Sr QA Manager in Performance Testing, Test Automation & Service Virtualization (20+ yrs IT). Joris Meerts Publishing the latest news from the software testing weblogs. Keeper of lists. Testing historian, critic, thinker, member of #DEWT. #Capgemini #Testers2Follow
  36. 36. March 2014 - 36 - #FollowUsAtTwitter Testing Circus Testing Circus is a world’s leading English language magazine for software testers and test enthusiasts. Monthly editions since September 2010. #testing Follow us at
  37. 37. Part 15 - Santhosh Tuppad Santhosh Tuppad specializes in exploratory testing approach and his core interests are security, usability and accessibility amidst other quality criteria. Santhosh loves writing and he has a blog He has also authored several articles and crash courses in the past. He attends conferences and confers with testers he meets. Santhosh is known for his skills in testing and has won the uTest Top Tester of the Year 2010 apart from winning several testing competitions from uTest and Zappers. March 2014 - 37 - Security Testing Tips One of the challenges in security testing is, setting up the test environment. If you are a small scale organiza- tion and there are no lots of processes, then it could be easy for you to go ahead and setup a test environment like you want, but in the bigger organizations when it has lots of processes it could be hard. There could be network related blockers that you may want to clear (Example: You want to download a software which you want to use for security testing purpose, and that source of download is blocked by the network of the organization). If your test environment is a blocker for you, then I would better recommend to not performing security testing and thereby, you can at least save costs. Isolated network of computers It is important to have a separate network dedicated for security testing. This is because, you do not want to affect the other computers on the network if you down- load some software and it is infected with malware / adware or any malicious thing. No website blockers The network shouldn’t be blocking any website that you want to browse for learning about some hacks or downloading any software to aid your hacking activi- ty. Let your network policy doesn’t end up blocking your learning. Administrator rights for computers & other devices In my experience, I have faced lot of blockers when the computer that was given to me did not provide admin- istrative rights. For changing some of the settings in order test for security, I had to e-mail the infrastructure team to change the settings and that consumed time. Before starting, it is important to ask for a computer with all the rights to change / modify any setting. Installations of software before commencement Make sure that you are ready with installations of all required software before you commence security test- ing activity. This is because, it will save you time if something doesn’t get installed or doesn’t work prop- erly while you are testing for security. So, installing the required software like proxy, burp suite, Wireshark, backtrack, kali linux, mantra browser, nmap and lot of many other tools happens without any hassles. No code changes to be done When testers are testing for security, no code changes should be done. It is important that you have separate environment and not the same which developer uses check in his / her code. And also, security testing needs to be done once functional testing is done and all the bugs reported are fixed. DIY: Test Environment for Security Testing
  38. 38. March 2014 - 38 - Visit our facebook page to share. Someone else too could enjoy what you're reading right now.
  39. 39. Since this is my first article for Testing Circus I’d like to thank all the wonderful contributors for sharing info, because there’s no other way to evolve in the IT industry! Why would we need to run tests on a Remote VM? You don’t actually need to do this, but from my own experience I’ve isolated a couple of scenarios where this can prove vital: a. Having a test-script (or suite) running on a JavaScript-rich page that needs to have complete focus all the time. So, no touching the mouse or keyboard while this is running. If this takes 1hr to run, you’re going to get bored and/or sleepy and let’s not forget you have to justify for that particular hour to your management. b. The VM can run on your local machine (making it not that “remote”), it can run on another machine lodged on a cloud-hosting server (e.g. RackSpace) or in your blade-center sitting neatly in your compa- ny’s basement. Unless the VM is running from your computer’s VBox you’re hogging way less resources by running this remotely. c. You might be running across situations where your script runs smooth on your local MacOS / Windows environment, but, for some weird reason you get weird failures on the CI machine running (in most cases) some Linux distribution. How will you debug this? Where to start? VBox setup and install I’m going to run through the steps needed to build your local Ubuntu VM assuming you have a working Host Machine (I’m currently running Mac OS 10.8.5, but this is 99% valid for Windows as well) a. Download and install VBox from Oracle’s website b. Download Ubuntu. I went for 13.04 64bit but you can choose whatever suits you c. Start VirtualBox d. Click the NEW pictogram in the left e. Populate the fields according to your need March 2014 - 39 - - Mihai Sarlea Running Remote Selenium 2 WebDriver Scripts On a VM
  40. 40. March 2014 - 40 -
  41. 41. March 2014 - 41 -
  42. 42. March 2014 - 42-
  43. 43. f. From here on it’s a standard Ubuntu installation the likes of which you can find at Configure Ubuntu I’m just going to run through the basics of Ubuntu configuration so you can get your VM up and running ASAP. a. Install OpenJDK-7 and Maven Since I’m using Java to run my Selenium tests, I’ll need to install the Java Development Kit and Maven (in my case) and I’m willing to share this nice trick useful also when setting up a fresh CI based on Ubuntu: This is the sequence that will get you a clean usable OpenJDK on a Deb machine: apt-get remove openjdk* apt-get remove java* apt-get install openjdk-7-jdk apt-get install openjdk-7-* apt-get install maven b. Test the install java –version mvn -v c. Check the IP address and the “visibility” of your host computer ping your_host_computer_ip March 2014 - 43 -
  44. 44. Here’s how I’ve setup the VM in VBox and Ubuntu: Now you should be able to ping to and from the VBox installed Ubuntu VM. March 2014 - 44 -
  45. 45. Setting up SELENIUM 2 on the VM a. Open up a Terminal window and download the Selenium Server Standalone jar wget server-standalone-2.35.0.jar b. What we’re doing is creating a small Selenium 2 GRID network and that’s why we need to start up a hub here: java -jar selenium-server-standalone-2.35.0.jar -role hub c. Open up a 2nd Termnal window and setup a (single) basic node with default config on the same machine: java -jar selenium-server-standalone-2.35.0.jar -role node -hub http://localhost:4444/grid/register d. Find your VM’s IP address – open up another Terminal window: Ifconfig -a That’s it, all setup! Adding the VM to your Selenium WebDriver Java Framework This is as easy as adding the lines bellow: switch (useThisDriver) { case FIREFOX: aDriver = new FirefoxDriver();//profile); break; case FIREFOX_LOCAL_VM_UBUNTU: DesiredCapabilities capability = DesiredCapabilities.firefox(); aDriver = new RemoteWebDriver(new URL(""), capa- bility); break; DesiredCapabilities capability = DesiredCapabilities.firefox(); driver = new RemoteWebDriver(new URL(""), capability); driver.findElement(…) // whatever you want to do If you’re not running the test as part of a framework, just go ahead and add the lines like this when starting the test: Conclusion I’ve tried to go through every step of setting up a nifty lightweight GRID 2 on a local (or remote) Ubuntu VM. The knowledge is applicable, in much the same way to a Windows VM or any other OS you might need, for the matter. I hope it helps you get up and running with your local *nix VM. Selenium GRID 2 is a very cool feature that can save a lot of time, but we’ll dive more into this in a later arti- cle. Till that time, feel free to check out the documenta- tion Google put up at: More on the framework “Driver” class I used at point 6: ver%E2%80%9D-class-selenium-2-webdriver Mihai Șarlea is an SQA Auto‑ mation Engineer for and sound engineering enthusi- ast. Telecom Engineer by forma- tion, Mihai started flying above radar while working as Simlock Security EU region Specialist for Nokia and, gradually, shifted gears towards QA-ing for Endava Romania on financial projects. For the last year and a half, he's been focusing on automated website testing, continuous integration, light- weight performance scripts and all adjacent tasks at Outside office hours, Mihai occasionally handles Front of House Engineering for various clubs in Cluj-Napoca, Romania and volunteers to help the less fortunate artists record decent, career kick- starter, materials. Feel free to check out his LinkedIn profile: March 2014 - 45 -
  46. 46. Thinking what to do with your career? We can HELP March 2014 - 46 -
  47. 47. Tools Journal Testing Corner Testing Circus in exclusive partnership with Tools Journal ( presents the Tools Journal Testing Corner. 1. Test Automation: New Venues, New Challenges 2. SmartBear TestComplete 10 Brings In Support For Automated Mobile Testing 3. Squish Coco Brings In Support For Code Coverage Of C# & Tcl Code "Thank You" to all subscribers who have joined us and have been giving us some good feedback. Most of all Thank you "Testing Circus" for providing us a good platform to shout our views . March 2014 - 47 - About Us: A start up journal and aspiring social community with an aim to gain and distribute knowledge on software tools and concepts in Testing, Agile, Cloud, Mobile and Enterprise Integration. With over 500 products listed with quality articles, product owner interviews, we are moving swiftly to launch product editorial/user reviews, community module in next 2 months. Connect With Us @toolsjournal Thank You Testing Circus & Tools Journal
  48. 48. While most of the test automation engineers are aware of the many pitfalls of test automation; there are some associated to the automation planning & design considerations. There are some widely used automation frameworks, & hybrid framework tops them all. While it’s a good practice to use hybrid framework with benefits taken from all the conventional frameworks like keyword, data driven etc…; the real trick in coming up with a good design is to cater to all the project needs while keeping the maintenance cost low, and the learning curve short. While there is no single way to implement this solution but again, it’s no rocket science either. One may have to consider various factors that automation framework has to handle – technical & non-technical. On technical side one may have to consider the level of complexity, abstraction-levels, so that the front-end whether it’s a GUI or an excel sheet needs to be easy to use & intuitive at the same time. While one should use excel file for easy readability, overusing it may lead to slow execution – contradicting the very reason automation is used e.g. keyword-driven framework using excel file has its benefits, it makes execution bit slower, each read and write to-from excel file may add up to huge delays by the time automation suite is completed. Extensive use of excel or any other file format may lead to bulky automation leading to slow execution & having its own weight to pull it down. While it is good idea to give maximum control (if not all) to the executor to manipulate suite content , it requires good judgmental call on the skill set of executor , reusability & degree of flexibility that needs to be provided so that its benefits are maximized. Though there have been good amount of focus given lately on application performance, automation performance is another area which is unexplored but said that, doesn’t mean we can overlook the automation execution speeds. · Terminology like Library-architecture etc. is something that is intrinsic to any software project whether it is automation or development, so it becomes really necessary, that we form meaningful library functions/methods, keep them light, reusable and ensure proper error handling. With the advent of various automation tools it becomes really important to keep a track of its advantages & disadvantages; before implementing the same. · With loads of new browsers and devices mushrooming in - Cross-browser & cross-device testing is something which can’t be avoided. But again each browser/device has its own engine/mechanism to handle web elements /objects. Taking an eye off such mechanisms can lead to unexpected results, and automation performance issues. Many times failures in automation needn’t be because of scripting or application alone, browsers also have their own drawbacks which lead to automation script failures. While various studies depict various statistics, one major thing that comes up is IE seems to have many issues when then comes to test automation, such statistics need to be looked at before commencing automation. · Object identification/locator mechanisms also play key role. Those who are well versed with March 2014 - 48 - Test Automation: New Venues, New Challenges
  49. 49. automation testing, don’t have to be told that, each mechanism has its own benefits, like one may be robust mechanism, another may be easy to use e.g. name, id as locators; thereby selecting a mechanism which fits well in all the areas is preferred. · With client side technologies like Silverlight, HTML5, JavaScript etc. dominating the web- pages, it becomes really critical to take them into consideration before committing to the client. Though the pages may appear normal, the tool, browser may have issues to work with. Thereby it is always advisable to do a thorough POC, before dwelling deeper into full-fledged automation. · Like in selenium, it is quite obvious to start automation using Firefox and committing to client on providing complete solution, what further is required is the foresight into client’s future requirements like cross-browser testing, mobile device testing. · It is always recommended to think of the breadth during planning & design phase & during implementation focus on individual automation functionality, creating a self- contained unit, preferably reusable. An incremental approach is highly advised - each automation component or test script needs to be well tested on various browsers & devices before moving to next level. This approach would avoid surprises that come with big-bang approach. (It is advisable to avoid big-bang approach, wherein various automation components are clubbed together to fulfill the requirements. ) · With extensive usage of open source technologies & tools (like selenium), it becomes really important to create a proper test environment, wherein various versions of the each tool, process or platform is well documented. Any change in one may lead to automation failure. · Integration with various other tools &/or application/platform compatibility also is critical, as slightest change in a version may impact automation framework e.g. which version of testlink (test management tool) integrates with which version of selenium or which version of UFT supports which browser & version. Various configurations & versions – inclusive of auto-updates are to be thought about & it is really important that each step is taken carefully to avoid last-minute surprises. · Requirement to automate huge number of test cases, and to get proper ROI, Distributed testing, is fast catching up. Various machines with various configuration, pose a major challenge to distributed automation testing; a single script to be executed for - cross-browser , cross-platform, cross-device each having their own intricate setup (including java version, OS version – service packs installed etc..) – each system depending on the system below it, creating a delicate balance for things to work : eventually all this can add-up to unimaginable permutation & combinations making it mandatory to document the details. Automation is no more limited to a single machine or a single tool, but ranges from various tool integration to its implementation on various machines & browsers, posing new technological challenges to be faced with advent of new versions, compatibility issues, configuration issues making it mandatory to foresee all the major unforeseen pitfalls to be addressed & risks to be averted or mitigated, ensuring successful automation solution delivery. March 2014 - 49 -