uTest STPCon 2011 Presentation
•Download as PPTX, PDF•
1 like•555 views
uTest CMO, Matt Johnston, talks about In-The-Wild testing, a new way of looking at testing some of today's mobile and web applications.
Recommended
Recommended
More Related Content
Similar to uTest STPCon 2011 Presentation
Similar to uTest STPCon 2011 Presentation (20)
More from uTest
More from uTest (8)
Recently uploaded
Recently uploaded (20)
uTest STPCon 2011 Presentation
- 1. In-The-Wild Testing: Your Survival May Depend On It Presented by Matt Johnston | @matjohnston STPCon, Fall 2011 |
- 2. Agenda • Current State of Affairs • SoLoMo Boom • Surviving In-The Wild • Crowdsourcing 101 • Secret Sauce • Key Takeaways | 2
- 3. The Challenge How We Arrived At The Status Quo | 3
- 4. Stuff Gets Built; Then It Breaks 1. Marketing identifies a need 2. Product specifies a solution 3. Engineering builds the product 4. QA tests the application 5. Help desk listens to complaints 6. Sales & execs blame everyone 7. Then we do it all over again and hope for better results… Vicious Cycle | 4
- 5. Testing Innovation Manual vs. Test Automation Managing QA Teams Agile Testing Games, Tours In-House vs. & Quick Attacks Outsource Partnering Testers & Engineers Exploratory vs. Test Cases | 5
- 6. Testing Innovation I n The Wi l | 6
- 7. To Make Matters Worse • Users are less tolerant • Social media & app stores give everyone a megaphone • So their problem… is your problem | 7
- 8. The Challenge Why It Matters More: The SoLoMo Boom | 8
- 9. Enterprise Consumerization • The Consumerization of Enterprise Computing – Technological disruption is pervasive – NO industry is exempt – B2B is B2C – Everyone’s a retailer • Not just for games and personal use – Docs – Data – Publishing • Not just your public site – CRM systems – BI & analytics tools – Productivity suites | 9
- 10. Social: Real-Time Communication • Enterprise social revolution isn’t coming… it’s here – Nothing is disconnected – No industry is immune • The stats on social media adoption in Fortune 100 alone: – 77% leverage Twitter – 69% utilize LinkedIn – 61% enable Facebook – 57% incorporate YouTube • Gartner: By 2016, social integrated w/ nearly all B2B apps | 10
- 11. Social: Impact On Testing • Enterprise adoption of social presents security & privacy hurdles – Internally, a channel for spam, malware and data breaches – Confidential data may be posted and trigger regulatory penalties – Externally, increasingly used for user authentication on your products • And functional testing challenges – Testing code & integrations that aren’t yours – That constantly changes – And you will get the blame | 11
- 12. Local: Can You Hear Me Now? • Businesses use location for more than just marketing • Numerous applications for location-based technologies – Asset tracking – Maps & routing – Location finders – Check-in services – Geo-based personalization • Web and mobile have gone local – 1 of 5 searches has local intent – 1 of 3 mobile searches has local intent | 12
- 13. Local: Impact On Testing • Not just a mobile problem – Localization testing – Geo-based personalization • But it is a mobile problem too – Apps used outside the confines of the QA lab, under in-the-wild conditions • When apps & users are distributed around the country or globe, a portion of testing should be too | 13
- 14. Mobile: Enterprise Impact • ABI Research anticipates worldwide enterprise mobile data revenues to reach $133 billion by 2014 • Enterprise mobility no longer just for email – Business apps: CRM, ERP, HR systems – Productivity apps: docs, spreadsheets, presentations – Collaboration apps: email, IM, publishing – Medical apps: health records, patient education • Tablet-mania, both inside and out – Aflac – Mercedes-Benz – Wells Fargo – SAP | 14
- 15. Mobile: Impact On Testing • Mobile web vs. native apps • Form factors matter again – Feature phones – Smart phones – Tablets • Ever-expanding HW & SW costs • Back to the future – A return to the late 90’s web - Lack of automation or load tools - Lack of security or usability standards - Lack of dev kits or tools - Extremely fluid landscape | 15
- 16. 1. Coverage Web Testing Matrix Is Tough Features OS & Browser | 16
- 17. 1. Coverage Mobile Testing Matrix Is Insane Features Handset Makers OS & Browser & Models | 17
- 18. 1. Coverage Mobile Testing Matrix Is Insane Features Wireless Handset Makers Carriers OS & Browser & Models | 18
- 19. 1. Coverage Mobile Testing Matrix Is Insane Features Location Wireless Handset Makers Carriers OS & Browser & Models | 19
- 20. 2. Managing Resources & Costs • Multiple products, roadmaps & launch schedules • Managing peaks & valleys; unexpected twists & turns | 20
- 21. 3. Time-to-Market • Never-ending race for each new app and version • Can exhaust the best test managers and teams | 21
- 22. SoLoMo: The Bottom Line • Becoming impossible for QA leaders to assure quality | 22
- 23. The Challenge Argh!! What Do We Do? | 23
- 24. From Surviving To Thriving • Keep testing in-the lab – Keep training – Keep automating – Keep innovating • Recognize that it will never again be sufficient • Identify the right way for you to test in-the-wild – Real-world conditions – Mirror your user base - Technologically: OS, browser, anti-virus, device, carrier - Geographically: Continent, country, city, language - Demographically: Age, gender, education, employment, industry, hobby – Maximize relevant testing coverage – Manage signal-to-noise ratio | 24
- 25. Testing In The Wild • Two ways to get there: Beta programs and crowdsourced testing | 25
- 26. Beta “Testing” Programs • Pros: – No direct costs – Lives outside the lab • Cons: – Users ≠ testers - Rarely diagnostic - Poor signal-to-noise ratio – Puts unfinished product in front of customers • Many firms discontinued beta as a core part of QA – Hidden costs – Takes too long – Not actionable results – Didn’t yield higher quality | 26
- 27. Crowdsourced Testing • Pros: – Lives outside the lab – Profiled, professional testers – Mirrors user base – Protects company IP – Much higher signal-to-noise ratio • Cons: – Carries direct costs – Requires careful partner vetting – Requires buy-in from QA leadership • Increasingly popular complement to in-the-lab testing | 27
- 28. The Challenge A Word About Crowdsourcing | 28
- 29. Crowdsourcing Across Industries | 29
- 30. Mob Mentality BUT crowds often look (and act) like unruly mobs | 30
- 31. From Mob to Community And mobs don’t work in every category Example: delivering a skilled service at an enterprise level of predictability and efficiency requires an orderly “community” capable of consistently producing the desired results | 31
- 32. So how do you turn this – | 32
- 33. Into this? | 33
- 34. The Challenge Secret Sauce | 34
- 35. Tenet 1: Know Thy Task | 35
- 36. Tenet 2: Know Thy Partner • Selection criteria are vital: – Referenceable customer successes – By company size – By industry – Ability to adapt to your legacy systems and processes – Ability to satisfy legal requirements – IP protection – NDA – Ability to do the job – Consistent – Predictable – Professional | 36
- 37. Tenet 3: Know Thy Community 1. Community profiling – Technical – Geographic – Demographic 2. Community ratings & micro-ratings – By testing type – By app type – By industry 3. Highly precise matching – Between each project & each tester | 37
- 38. Tenet 4: Know Thyself • Is your culture highly cautious & risk-averse? • Are you in a highly regulated industry? – Defense industry – PCI, PII or PHI • Do you have an appetite for innovation? • Are you centrally organized or decentralized? – Sourcing – IT & IS – Engineering – QA & QE | 38
- 39. Tenet 5: Know Thy Trends (Enterprises That Already Leverage The Crowd) | 39
- 40. The Challenge Enterprise Perspective | 40
- 41. Google Global power Google taps into the crowd to augment its in-house testing resources – Challenge: - Tech execs sought ways to scale testing and achieve in-the-wild coverage - Wanted to ensure testing had real-world relevance to end-user experience – Strategy: - Have invested heavily in outsourcing, in-house resources and test automation - Google was eager to find a better, more scalable approach to app testing - Google explored a wide variety of alternatives and found uTest – Results: - After extensive 2009 pilot project, Google has since expanded its use to 18+ apps – Web, desktop and mobile - Now leverages in-the-wild testing broadly & frequently, as part of core QA playbook - Relationship is pioneering in-the-wild testing as complement to enterprise’s in-the-lab | 41
- 42. Microsoft Security Essentials Security Essentials – a free anti-virus app – Challenge: - v1.0 product in entirely new category, so IP preservation was vital - Needed to test functionality under real-world conditions on vast matrix of hardware, browsers, Windows OSes, and third-party applications - Needed testing coverage in strategic geo-locations (China, India, South America) to mirror expected user-base and fill blind spots – Strategy: - Recognized that it couldn’t solve this solely in the lab - Assemble a targeted team of testers to perform a series of exploratory and test case execution to cover stated testing criteria – Results: - After two months of continuous testing in ten countries on four continents – hundreds of detailed bug reports, executed test cases, and user reviews had been prioritized - The company continues to run regression tests on the software to coincide with major upgrades or new releases | 42
- 43. The Challenge Bottom Line Summary | 43
- 44. Key Takeaways • Apps Universe Has Forever Changed Testing – Exponentially more diverse user environments – Devices – Software configurations – Locations – User demographics – Users less tolerant – Quality issues no longer private • QA Must Play Catch-Up – By improving inside the QA lab – By moving beyond it, where your users live, work & play – Beta programs and/or crowdsourced testing – App quality winners are combining: in-the-lab testing + in-the-wild testing | 44
- 45. The Challenge The End. Well, actually it’s just beginning… (more at InTheWildTesting.com) Matt Johnston | CMO @ uTest mattj@utest.com | @matjohnston | 45
Editor's Notes
- As I mentioned consumerization is a driving force behind this current shift and by looking at the consumer market, we know it is impossible to resist. Whether it is at home or work, it is human nature to want greater social interaction, more control over our surroundings and more actionable information.Speaking of actionable information, Just last week I met with VisaQuate and they have a great business doing just that. They take BI data, which was already transformed and transform it even further into even more actionable information. I love it, at some point in the future, my dashboard at work will be so simple and yet precise that I can predict the next year with a 5 second glance. For now, I only wish I could get better information on one of a million mobile devices in the world.
- As I mentioned consumerization is a driving force behind this current shift and by looking at the consumer market, we know it is impossible to resist. Whether it is at home or work, it is human nature to want greater social interaction, more control over our surroundings and more actionable information.Speaking of actionable information, Just last week I met with VisaQuate and they have a great business doing just that. They take BI data, which was already transformed and transform it even further into even more actionable information. I love it, at some point in the future, my dashboard at work will be so simple and yet precise that I can predict the next year with a 5 second glance. For now, I only wish I could get better information on one of a million mobile devices in the world.
- As I mentioned consumerization is a driving force behind this current shift and by looking at the consumer market, we know it is impossible to resist. Whether it is at home or work, it is human nature to want greater social interaction, more control over our surroundings and more actionable information.Speaking of actionable information, Just last week I met with VisaQuate and they have a great business doing just that. They take BI data, which was already transformed and transform it even further into even more actionable information. I love it, at some point in the future, my dashboard at work will be so simple and yet precise that I can predict the next year with a 5 second glance. For now, I only wish I could get better information on one of a million mobile devices in the world.
- Remember I mentioned that Salesforce bought Manymoon. They also have Chatter and Yammer is there with its on solution, but that is just the tip of the iceberg. Nearly every major enterprise software provider is adding social features in some way. As test engineers, assuming we agree what social features are in general, we have a couple of things we need to consider during test planning.One is collaborative testing and the other is security testing.For collaborative testing I am referring to multiple people testing a feature or set of features in concert. It is like end to end business process testing just in real-time and with a single person orchestrating the test. We have been doing a lot of this at uTest lately for mobile gaming and facebook apps and it has yielded a lot of critical data conflict bugs for those companies.
- For security, I am sure you have seen the headlines. As we open up communication channels both internally and with partners and customers, we open up opportunities for ip to be lost. Some of that IP will be lost through penetration attacks, which we can test for.However, most of it will be lost through loose lipped employees that have a larger forum than just the water cooler. It may not be the QA teams job to train employees on what they can and cannot say or to moderate these channels once they are live. It is our responsibility to raise these concerns during design time and then test preventative and reactive mechanisms used to deal with them.
- Recent developments in technology and our understanding on how to implement it is opening up a whole new world for Geo-Location. One of the more interesting for me is Geo-Fencing. Imagine if you will that STP had a mobile app that each of you installed on your smart phones. By utilizing GPS, triangulation and or basic orienteering, it would know where you are and, more importantly, where you are in relation to a predefined location. With that data, It could have checked you into the conference when you walked through the front doorIt could have routed you to your next session and scanned you in when you sat downIt could offer you the ability to chat about how awesome this keynote is with other people that are in this room or send questions to the host without bothering everyone on twitter. When you leave, it can disable those services automatically.
- Of course, programmers and product managers love this and can think of millions of ways to use it. As testers, boy are we in trouble. Now our test plans just got a lot more complex. In addition to the obvious, like os, hardware and test steps, we also need to consider time, location and connection types. Within just connection types, we have wifi, carriers, speed, rfid, etcIn the last year, uTest actually ran several projects where testers went to specific locations to test products like these all over the world. After all, it is one thing to test a product in a sterile lab, but the real bugs happened when people were interacting with the products in the real world. You would be surprised how much low batteries, changing towers, and poor reception can completely kill an applications usabilityOk, we are closing in on the end now and I think it may be time for some disruption
- That is not surprising and the manufacturers and carriers are counting on it. ABI says we may see $133 billion in data revenue worldwide by 2014To me, given that even the old stogy medical industry is mobilizing quickly, I think that number might be conservative.I further base that statement on the fact that 40% of uTest’s business is now Mobile app related and that number is not restricted to games or consumer appsWe are seeing significant increases in apps for business, productivity and collaboration across all of the major device families.Now, I grant you that many of these apps start as outsourced projects for point solutions, but they are now becoming strategic, coming in house and becoming mission critical.
- That is not surprising and the manufacturers and carriers are counting on it. ABI says we may see $133 billion in data revenue worldwide by 2014To me, given that even the old stogy medical industry is mobilizing quickly, I think that number might be conservative.I further base that statement on the fact that 40% of uTest’s business is now Mobile app related and that number is not restricted to games or consumer appsWe are seeing significant increases in apps for business, productivity and collaboration across all of the major device families.Now, I grant you that many of these apps start as outsourced projects for point solutions, but they are now becoming strategic, coming in house and becoming mission critical.
- That is not surprising and the manufacturers and carriers are counting on it. ABI says we may see $133 billion in data revenue worldwide by 2014To me, given that even the old stogy medical industry is mobilizing quickly, I think that number might be conservative.I further base that statement on the fact that 40% of uTest’s business is now Mobile app related and that number is not restricted to games or consumer appsWe are seeing significant increases in apps for business, productivity and collaboration across all of the major device families.Now, I grant you that many of these apps start as outsourced projects for point solutions, but they are now becoming strategic, coming in house and becoming mission critical.
- That is not surprising and the manufacturers and carriers are counting on it. ABI says we may see $133 billion in data revenue worldwide by 2014To me, given that even the old stogy medical industry is mobilizing quickly, I think that number might be conservative.I further base that statement on the fact that 40% of uTest’s business is now Mobile app related and that number is not restricted to games or consumer appsWe are seeing significant increases in apps for business, productivity and collaboration across all of the major device families.Now, I grant you that many of these apps start as outsourced projects for point solutions, but they are now becoming strategic, coming in house and becoming mission critical.
- That is not surprising and the manufacturers and carriers are counting on it. ABI says we may see $133 billion in data revenue worldwide by 2014To me, given that even the old stogy medical industry is mobilizing quickly, I think that number might be conservative.I further base that statement on the fact that 40% of uTest’s business is now Mobile app related and that number is not restricted to games or consumer appsWe are seeing significant increases in apps for business, productivity and collaboration across all of the major device families.Now, I grant you that many of these apps start as outsourced projects for point solutions, but they are now becoming strategic, coming in house and becoming mission critical.
- That is not surprising and the manufacturers and carriers are counting on it. ABI says we may see $133 billion in data revenue worldwide by 2014To me, given that even the old stogy medical industry is mobilizing quickly, I think that number might be conservative.I further base that statement on the fact that 40% of uTest’s business is now Mobile app related and that number is not restricted to games or consumer appsWe are seeing significant increases in apps for business, productivity and collaboration across all of the major device families.Now, I grant you that many of these apps start as outsourced projects for point solutions, but they are now becoming strategic, coming in house and becoming mission critical.