uTest STPcon spring 2012 presentation

932
-1

Published on

Published in: Technology, Business
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
932
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
27
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

uTest STPcon spring 2012 presentation

  1. 1. The SoLoMo RevolutionAnd The Resulting Tectonic Shift In Testing |Matt Johnston | @matjohnston | CMO @ uTest | March 2012
  2. 2. The Challenge SoLoMo Huh? | 2
  3. 3. SoLoMo Breakdown So Lo Mo Social Local Mobile | In-The-Wild Testing for Functional + Security + Load + Usability + Localization 3
  4. 4. SoLoMo Coming Fast In the past six months… | In-The-Wild Testing for Functional + Security + Load + Usability + Localization 4
  5. 5. QA Leaders Better Sit Down • Media & analysts aren’t the only ones talking about it – Marketing – Product – Engineering – IT – Sales – C-suite • Want a seat at the table? – Get informed – Get involved – Develop an opinion – Advocate strongly | In-The-Wild Testing for Functional + Security + Load + Usability + Localization 5
  6. 6. The Challenge Trends In Social Apps | 6
  7. 7. Not Just For B2C Anymore• The social revolution isn’t coming… it’s here – Nothing is disconnected – No industry is immune• The stats on social media adoption in Fortune 500 firms: – 77% integrate Twitter – 84% incorporate LinkedIn – 93% utilize Facebook – 57% publish on YouTube• Gartner: By 2016, social integrated w/ nearly all apps – Websites & apps – Mobile apps & mobile web – Desktop clients | In-The-Wild Testing for Functional + Security + Load + Usability + Localization 7
  8. 8. Goes Soooo Much Further | In-The-Wild Testing for Functional + Security + Load + Usability + Localization 8
  9. 9. The Challenge Trends In Local Apps | 9
  10. 10. Not Just For Marketing• Businesses use location for more than just marketing• Growing trend toward location-based intelligence in apps – Asset tracking – Maps & directions – Location finders – Check-in services – Geo-based personalization – Location-based ads | In-The-Wild Testing for Functional + Security + Load + Usability + Localization 10
  11. 11. Increasing Granularity Good news for users (and marketers)… good luck to testers. | In-The-Wild Testing for Functional + Security + Load + Usability + Localization 11
  12. 12. The Challenge Trends In Mobile Apps | 12
  13. 13. Mobile Apps Boom Market• ABI: mobile revenues will hit $133 billion in 2014• B2B mobility no longer just for email – Business apps: CRM, ERP, HR systems – Productivity apps: docs, spreadsheets, presentations – Collaboration apps: email, IM, publishing – Medical apps: health records, patient education• B2C mobility growth even steeper – Retail: location-intelligent m-commerce – Media: magazines & newspapers going purely digital – Travel: mobile bookings, check-ins, maps, deals – Education: tablets in every classroom – Healthcare: patient records, physician notes – Social: no explanation required – Gaming: ditto | In-The-Wild Testing for Functional + Security + Load + Usability + Localization 13
  14. 14. Mobile’s Just Getting Started | In-The-Wild Testing for Functional + Security + Load + Usability + Localization 14
  15. 15. Did I Mention… | In-The-Wild Testing for Functional + Security + Load + Usability + Localization 15
  16. 16. Psst… It’s Not Just Mobile • Smartphones & tablets make QA immeasurably tougher • Now imagine a whole world of apps: E-Readers Cars Household Appliances Smart Homes Gaming Consoles Connected TVs | In-The-Wild Testing for Functional + Security + Load + Usability + Localization 16
  17. 17. The Challenge SoLoMo Impact On Testing | 17
  18. 18. Brave New World • Impact on design, dev & IT well-recognized & resourced • Sadly, but not at all surprisingly, not true for testing – Want proof? Specialized designers & devs… but who tests? | In-The-Wild Testing for Functional + Security + Load + Usability + Localization 18
  19. 19. Social: Impact On Testing• Social presents functional test challenges – Testing code & integrations that aren’t yours… – That constantly changes… – And you get the blame• As well as security & privacy pitfalls – Internally, a channel for spam, malware and data breaches – Confidential data may be posted and trigger regulatory penalties – Externally, increasingly used for user authentication on your products | In-The-Wild Testing for Functional + Security + Load + Usability + Localization 19
  20. 20. Social: Impact On Testing• New challenges include how will you: – Regression test multiple social integrations for web & mobile apps? – Write (and execute) test cases with zero visibility into social’s roadmaps? – Automate anything? – Test sharing/liking in a QA environment without publishing via social? – Test your social footprint? - FB page (and apps) - LinkedIn page - Twitter page – Ensure security/privacy if you enable users to use social sign-on? | In-The-Wild Testing for Functional + Security + Load + Usability + Localization 20
  21. 21. Local: Impact On Testing • Not just a mobile problem – Localization testing – Geo-based personalization • But it is a mobile problem too – Apps used outside the confines of the QA lab, under in-the-wild conditions • Impossible to test effectively from only inside the lab | In-The-Wild Testing for Functional + Security + Load + Usability + Localization 21
  22. 22. Local: Impact On Testing • New challenges include, how will you: – Replicate location (answer: you don’t) – Test in NYC, SF, LA, DFW, CHI? – Test abroad - SIM cards - Carriers – Test GPS-based intelligence? – Test IP-based localization? | In-The-Wild Testing for Functional + Security + Load + Usability + Localization 22
  23. 23. Mobile: Impact On Testing• New challenges include how will you: – Native apps: Test across ever-changing OS, devices and carriers? – Mobile web: Test across wide range of OS and mobile browsers? – Either way, how does your testing replicate real-world conditions - Specific and varied locales - Tower-to-tower hopping - Roaming - Low battery - Touch screen actions (pinch, swipe, pan, etc) - Portrait vs. landscape - Display definition | In-The-Wild Testing for Functional + Security + Load + Usability + Localization 23
  24. 24. Mobile: Impact On Testing• Mobile web vs. native apps – 2009: 5-horse race – 2010: 4-horse race – 2011: 2-horse race (depending upon your industry) – 2012: 3 horses (including HTML5-driven mobile web) | In-The-Wild Testing for Functional + Security + Load + Usability + Localization 24
  25. 25. F|R|A|G|M|E|N|T|E|D• Android alone (and that’s only by OS version) | In-The-Wild Testing for Functional + Security + Load + Usability + Localization 25
  26. 26. Mobile Testing Landscape• Form factors matter again – Phones (feature & smart) – Tablets – Hybrids• Prohibitive HW & SW costs• Here we go again – Late-90s web, part deux - Lack of mature tools - Lack of usability standards - Lack of security standards - Extremely fluid landscape | In-The-Wild Testing for Functional + Security + Load + Usability + Localization 26
  27. 27. Multiple Launch Schedules Multiple products, roadmaps & launch schedules | In-The-Wild Testing for Functional + Security + Load + Usability + Localization 27
  28. 28. Test CoverageWeb Testing Matrix Is Tough Features OS & Browser | In-The-Wild Testing for Functional + Security + Load + Usability + Localization 28
  29. 29. Test CoverageMobile Testing Matrix Is Insane Features Handset Makers OS & Browser & Models | In-The-Wild Testing for Functional + Security + Load + Usability + Localization 29
  30. 30. Test CoverageMobile Testing Matrix Is Insane Features Wireless Handset Makers Carriers OS & Browser & Models | 30 In-The-Wild Testing for Functional + Security + Load + Usability + Localization
  31. 31. Test CoverageMobile Testing Matrix Is Insane Features Location Wireless Handset Makers Carriers OS & Browser & Models | 31 In-The-Wild Testing for Functional + Security + Load + Usability + Localization
  32. 32. Increasingly Complex Testing• The laughably simplistic notion of a “test matrix” | In-The-Wild Testing for Functional + Security + Load + Usability + Localization 32
  33. 33. The Challenge Ummmm… So What Do We Do? | 33
  34. 34. Sufficiently Freaked Out?!?• Don’t be… the good news is that testing ain’t dead! – Chance for QA leaders to step up… – To lead the charge… – To defend users and app quality | In-The-Wild Testing for Functional + Security + Load + Usability + Localization 34
  35. 35. …And The Bad News…• You should be scared… need to completely re-think testing – Can’t automate our way out of this – Can’t hire or train our way out – Can’t solve this problem from soley within the lab• More distributed problem than ever before – Need a new testing mindset – Need to bridge the growing gap - Between testing & users - Between artificial and real - Between in-the-lab & in-the-wild testing | In-The-Wild Testing for Functional + Security + Load + Usability + Localization 35
  36. 36. Lab-Based Testing Innovation Manual vs. Automation Managing vs. Leading Teams In-House vs. Outsourced Agile vs. Waterfall Exploratory vs. Testers vs. Test Cases Engineers Offshore vs. Nearshore | In-The-Wild Testing for Functional + Security + Load + Usability + Localization 36
  37. 37. Users Work & Live In The Wild I nThe Wi l In-The-Wild Testing for Functional + Security + Load + Usability + Localization | 37
  38. 38. To Make Matters Worse• Users are less tolerant of quality failures• Social media & app stores give everyone a megaphone• And their problem… is your problem | In-The-Wild Testing for Functional + Security + Load + Usability + Localization 38
  39. 39. The Challenge A Call To Action For Test Leaders (Two, Actually) | 39
  40. 40. 1. Look Beyond The Lab• Yes, keep testing in-the lab – Keep training in-house – Keep outsourcing – Keep automating – Keep innovating• Recognize it will never again be sufficient on its own• Build your company’s plan for testing in-the-wild – Real-world conditions: Live testers, real devices, imperfect connectivity – Mirror your user base - Technologically: OS, browser, anti-virus, device, carrier - Geographically: Continent, country, city, language - Demographically: Age, gender, education, employment, industry, hobby• In-house QA leadership is vitally important – You have the business and product context – You know what your firm cares about – You need to be the orchestrator who leads this symphony - What to test in the wild, how to test in the wild, holistic test planning | In-The-Wild Testing for Functional + Security + Load + Usability + Localization 40
  41. 41. 2. Be The Voice For QA • SoLoMo is coming – with or without QA’s buy-in • Want QA to be heard? Want quality to be built in? – Don’t wait: No one else is coming to the rescue – Get informed: Develop an thoughtful opinion – Get involved: Find out when/where decisions are being made – Be proactive: Build a plan for quality from the ground up – Be protective: You’re a protector of the user… of the brand | In-The-Wild Testing for Functional + Security + Load + Usability + Localization 41
  42. 42. The Challenge SoLoMoWoSto A Few Stories (if there’s time) | 42
  43. 43. The Challenge Questions? Answers. Matt Johnston | CMO @ uTest mattj@utest.com | @matjohnston | 43
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×