Open Source Test Workshop for CIOs, CTOs, Managers


Published on

This Open Source Test Workshop is for senior IT and business executives needing visibility and management tools and methodology into all the demand for IT. Shows how to bring Open Source Testing into your organization.

Published in: Business, Technology, Education
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Open Source Test Workshop for CIOs, CTOs, Managers

  1. 1. Surface and Mitigate Performance Bottlenecks Open Source Test Workshop For CIOs, CTOs, Senior Managers Frank Cohen, +01 (408) 871-0122, June 8, 2011Wednesday, June 8, 2011
  2. 2. Open Source Test Automation Slides on Twitter ‣ ‣ Please retweet ‣ Please follow me ‣ #ost 2Wednesday, June 8, 2011
  3. 3. Open Source Test Automation About PushToTest ‣ Our Mission: We Test, For A More Reliable World ‣ Make It Easier To Move From Manual To Automated Testing ‣ Support Advanced Testing, including Agile, Load, Integration ‣ Open Source Testing (OST) For Everyone 3Wednesday, June 8, 2011
  4. 4. XML, SOA, and Performance Web 2.0 Success Driving Business ‣ Every Organization Wants Modern IT Functions and User Interfaces ‣ Rich User Interfaces (RIA, using Ajax, Flash, Flex) ‣ SOA Composite Applications • Faster Time-To-Market • Increase Revenue • Cost Savings ‣ Service Enabled Applications • Deliver More Integration Faster • Always-On • Less Cost 4Wednesday, June 8, 2011
  5. 5. XML, SOA, and Performance The Reality of Web 2.0 Development ‣ We Build Applications Differently Now ‣ Haze of New Protocols, Software Models, Data Formats ‣ Agile Often Means Shorter Build-Test-Deploy ‘Waterfall-style’ Cycles ‣ Many Still Trying “Test And Trash” Techniques ‣ Uptime Depends on Business Service Management (BSM) Testing to Surface Functional and Performance Issues Every 8 weeks 30% of the Application Changes 5Wednesday, June 8, 2011 30% Tests Re-recorded
  6. 6. Open Source Test Automation Testing Costs ‣ Proprietary Test Tools Offer: Functional Test Load Test Functional Test Platform $19,000 Load Test Platform $20,000 Functional Tests $8,000 Simulated User Licenses $120,000 Protocols: AJAX, SOAP $12,000 Recoding The Test $35,000 Support $5,000 Re-quality The Test $12,000 $40,000 $187,000 ‣ QA Labs for 10,000 Users = $35,000 Per Year ‣ DoesCause Analysis Consultants Root Not Include: Cost of Test Employees, ‣ Does Not Guarantee Success 6Wednesday, June 8, 2011
  7. 7. Open Source Test Automation Proprietary Testing Falling Behind ‣ Adobe Releases Testing APIs for Flex/Flash in 2009 ‣ JQuery, GWT, YUI, Extjs Each Make 5 Releases in 2010 ‣ Appcelerator Titanium Makes 3 Releases in 2010 ‣ Microsoft Draws 8,000 to Mix for StarLight in 2010 ‣ StarEast, Better Software, PyPy, JavaOne, Devoxx, OSBC, LinuxWorld Feature Keynote Talks On Ajax ‣ “HP Refundedcustomer in theFor The Ajax industry. Handler”, a PushToTest Our Payment automotive Protocol 7Wednesday, June 8, 2011
  8. 8. Open Source Test Automation Open Source Testing EmergesWtness Lots Of Options Selenium HtmlUnit soapUI DBUnit Sahi Watir FlexMonkey FIT Windmill Hudson TestMaker jMeter ‣ Signs of Maturity:Logger, Recovery Scenario Manager, Object Iteration Engine, Record/Playback, Data Generation/Importer, 8 RecognizerWednesday, June 8, 2011
  9. 9. XML, SOA, and Performance Depend On A Service Partner ‣ Repurposing: One Test Script For Functional Tests, Load and Performance Tests, and Business Service Monitors ‣ Components: Tests Run Even After Application Changes ‣ Deployment: Run Tests In Your Lab, Ours, or Both! ‣ Services: Add Expertise, Training, Tools To Your Organization ‣ Enthusiasm:Openparticipate and contribute to We Source Test Projects 9Wednesday, June 8, 2011
  10. 10. Open Source Test Automation Manual Testing Process Results 4 1 Recommend Changes Explore Manual Testing Tes 3 t 2 Plan 10 Run The Test Steps Create Test PlansWednesday, June 8, 2011
  11. 11. Open Source Test Automation Side Effects ‣ Exhaustion ‣ Fatigue ‣ Blurry Vision ‣ Night Sweats ‣ Insanity ‣ Hair Loss ‣ Acne ‣ Divorce 11Wednesday, June 8, 2011
  12. 12. Open Source Test Automation OST Architecture Application Under Test PTTMonitor Web, RIA, BPM, SOA CPU, Net, Memory Functional Test Injector Test Authoring Test Bundles ScriptRunners Data Production Libraries SoapUI (Pro) MySQL, HtmlUnit Sahi, Selenium, SoapUI, Java csv files, rdbms queries Sahi (Pro) JUnit Class Framework Load Test Eclipse Java IDE Fitness Test Operation Console Languages: Java, Python Wiki Collaboration Production Designer Grid and Cloud Repository Results Analysis Record/Playback Test Deployment MySQL RDBMS Business Intelligence Tool Monitor Continuous Test Repository Bug Tracker Monitoring Integration Management 12Wednesday, June 8, 2011
  13. 13. Open Source Test Automation An OST Cloud Thousands of virtual users Stores results of every TestNode perform real-life scenarios on Results DB transaction, step, and the Application Under Test monitored resource PTTMonitor Application Under Test Load Injectors 400+ charts identify the root Analysis cause to performance bottlenecks, functional issues Central location from which a TestMaker tester, developer, IT manager, CI Console service runs and monitors tests 13Wednesday, June 8, 2011
  14. 14. Open Source Test Automation An OST Cloud Thousands of virtual users Stores results of every TestNode perform real-life scenarios on Results DB transaction, step, and the Application Under Test monitored resource PTTMonitor Application Under Test Load Injectors 400+ charts identify the root Analysis cause to performance bottlenecks, functional issues Central location from which a TestMaker tester, developer, IT manager, CI Console service runs and monitors tests 14Wednesday, June 8, 2011
  15. 15. Open Source Test Automation Browser Compatibility Tests Application Under Test TestNode Selenium RC SeleniumHtmlUnit Selenese Test TestMaker Console 15Wednesday, June 8, 2011
  16. 16. Open Source Test Automation ScriptRunners ScriptRunner Type Name (langtype) Designer Script IE, Htmlunit designerscript Selenium HtmlUnit Browser selenium Selenium RC IE, Firefox, Chrome seleniumrc Sahi IE, HtmlUnit sahi HTTP Archive Page Optimization HAR Selenium RC IE, Firefox, Chrome seleniumrc SoapUI SOAP and REST soapui TestGen4Web Web 1.0 Tests testgen4web Mozmill Mozilla XUL Tests mozmill Unit Tests jUnit TestCase Java, Jython, JRuby, PHP, Perl, Groovy DotNet .NET Tests dotnet Visual Basic VB tests vb 16 Command Line Desktop and utils commandlineWednesday, June 8, 2011
  17. 17. Open Source Test Automation Component Approach TestScenario Component Application Log-In Load Test 100 Users 500 Users 1000 Users setUp Data runTest Test Use Case Database Production tearDown Development RDBMS DPL Log-In Database CSV DPL Product Search Component QA Regression Order Product Custom DPL Database Confirm Order Component Load Test Log-Out Database Production 17Wednesday, June 8, 2011
  18. 18. Open Source Test Automation Lots of Possibilities TestMaker Test Sent Seattle, Datacenter TestNode Generating Load Dallas, Texas Datacenter You’re Such A Geek! 18Wednesday, June 8, 2011
  19. 19. Open Source Test Automation TestMaker Demonstration ‣ Fun And Easy Test Script Creation ‣ Selenium and TestMaker ‣ Repurposing A Single Test ‣ Functional Test, Load and Performance Test, Production Monitor ‣ Results Analysis ‣ Chart Engine 19Wednesday, June 8, 2011
  20. 20. Open Source Test Automation Root Cause Analysis ‣ PTTMonitor - Agent Based Monitor Gateway ‣ Glassbox - Performance Monitoring for Java Applications ‣ dynaTrace - Analysis In Application and DB Tiers 20Wednesday, June 8, 2011
  21. 21. Open Source Test Automation PTTMonitor ‣ Correlate Test Operation To Resource Usage ‣ Deploy as Web Application (WAR) and Service ‣ Observes CPU, Network, Memory Utilization As Test Runs ‣ Gateway to Glassbox, dynaTrace, Windows PerfMon 21Wednesday, June 8, 2011
  22. 22. Open Source Test Automation Results Analysis Engine 22 And 350 Additional ChartsWednesday, June 8, 2011
  23. 23. Open Source Test Automation Glassbox ‣ Correlate Test Operation To Application Tier Issues ‣ Memory Leaks ‣ Thread Deadlocks ‣ Slow DB Connections ‣ No Instrumentation of Applications ‣ Live Performance Alert View ‣ Run In Production 23Wednesday, June 8, 2011
  24. 24. Open Source Test Automation Glassbox Troubleshooter 24Wednesday, June 8, 2011
  25. 25. Open Source Test Automation dynaTrace PurePaths ‣ PurePath from TestMaker Virtual Users To Application & DB Tier ‣ Answers: What Are The Transactions and Why Are They Slow? dynaTrace monitors applications for functional issues dynaTrace Database Agent and performance bottlenecks. dynaTrace Application Server Agent PurePath Token TestNode TestNode TestNode test test test PTTMonitor Test RunTime 25 TestMaker ConsoleWednesday, June 8, 2011
  26. 26. Open Source Test Automation Pinpoint The Problem ‣ Down to the Line of Source Code 26Wednesday, June 8, 2011
  27. 27. Open Source Test Automation .NET Monitoring ‣ Correlate PerfMon Statistics with Test Operation ‣ PTTMonitor Gateways Data from PerfMon or MS SQL Server .NET Application Tier PerfMon MS SQL Server SOAP HTTP REST TestNode TestNode TestNode PTTMonitor test test test Test RunTime TestMaker Console 27Wednesday, June 8, 2011
  28. 28. Open Source Test Automation Next let’s talk about PushToTest Options ‣ OnDemand ‣ QuickStart ‣ POC Option 28Wednesday, June 8, 2011
  29. 29. Open Source Test Automation Immediate Benefits ‣ Product Solutions ‣ Better, Lower-Cost, More Efficient Test Tool ‣ Business Service Monitoring at the Application Tier (SLA Compliance) ‣ Methodology Solutions ‣ Improved Test Methodology for my Organization and Team ‣ Services ‣ Turnkey Load Testing ‣ Integrate TestMaker Into My Build Environment ‣ Add A Protocol, Chart, Task to TestMaker To Meet My Needs 29Wednesday, June 8, 2011
  30. 30. Open Source Test Automation Dual License ‣ Functional Testing with TestMaker Object Designer ‣ TestMaker Enterprise expands on the success of ‣ TestMaker Community Repurposing Selenium, soapUI Sahi and unit tests into data-driven functional tests, load and performance ‣ Load Testing to Millions of Virtual Users ‣ tests, and business service monitors Cloud and Grid Testing ‣ Load Testing To 50 Virtual Users ‣ Data-driven tests using Data Production Libraries for ‣ Data-driven tests using CSV files. CSV files, RDBMS, Data Generating Objects ‣ PTTMonitor Correlates CPU, Network, Memory ‣ Root cause analysis and mitigation using Glassbox ‣ Utilization As Tests Operate. Mock services for agile service development ‣ 25 Charts and Reports ‣ Flex AMF Protocol Handler for Flex/Flash testing ‣ Community Support and access to all documentation ‣ 1000+ Charts and Reports ‣ Business service monitors, including email notification ‣ Support, QuickStart, Training, Consulting 30Wednesday, June 8, 2011
  31. 31. Open Source Test Automation Pricing ‣ Annual Subscription Model Based On: ‣ Seats for TestMaker Console Installation ‣ Seats for SOAP and REST Test Authoring ‣ Virtual User Levels for Load Testing ‣ Count of Business Service Monitor ‣ Technical support: 1messagesday or faster response to emails, phone calls, instant business 31Wednesday, June 8, 2011
  32. 32. Open Source Test Automation ‣ As-Needed, On-Demand Basis ‣ Test In Your Environment, Or Our Cloud Test Environment, Or Both ‣ Ideal For Consulting Organizations Where Test Needs Are Unknown ‣ AMI Distribution ‣ Pay Only For What You Use ‣ Annual Cap On Total Costs 32Wednesday, June 8, 2011
  33. 33. Open Source Test Automation The QuickStart Option ‣ Your First Test, Built and Operated by PushToTest Experts ‣ Ideal For Organizations With Little or No Load Testing Experience ‣ PushToTest Experts Build Your First Test ‣ PushToTest Experts Operate The Test, Provide Root Cause Analysis and Mitigation ‣ PushToTest Trains Your Team To Build Their Own Tests 33 ‣ $9,750 USDWednesday, June 8, 2011
  34. 34. Open Source Test Automation The POC Option ‣ PushToTest Experts Help Your TestMaker Evaluation ‣ Ensures TestMaker Works In Your Application/System ‣ PushToTest Issues Price Proposal ‣ Applies Engineers To Build and Operate Your Test ‣ PushToTest Trains Your Team To Use TestMaker ‣ Success of POC Confirms Intention To Purchase 34 ‣ Cost: $0Wednesday, June 8, 2011
  35. 35. Open Source Test Automation PushToTest Global Services ‣ Put Our Experts To Work For Your Projects ‣ Methodology, Solutions, Support, Enthusiasm ‣ Offices: USA (Texas and California,) Bangalore, London ‣ The Near Shore Option ‣ 150 Test Experts ‣ Costa Rica: CST, English, Inexpensive, On Site in 3-5 Hours ‣ “PushToTest Uniquely Built Load Tests For Our Customers, Including Pepsi, And Worked With Our Engineers To Surface and Solve Performance Bottlenecks.” Alan Rodriguez, Director of Testing, 35 Tribal DDB DallasWednesday, June 8, 2011
  36. 36. Open Source Test Automation Slides on Twitter ‣ ‣ Please retweet ‣ Please follow me ‣ #ost 36Wednesday, June 8, 2011
  37. 37. Open Source Test Automation Where To Go From Here ‣ Please Complete The Evaluation Form ‣ And We Will Send You The Slides ‣ Watch A Screencast and Tutorial ‣ ‣ Attend An Open Source Test Workshop ‣ ‣ Ask For A Proposal on Licenses, Training, Support, Consulting ‣ Call +01 408 871 0122, 37Wednesday, June 8, 2011