ALM (Application Lifecycle Management)

  • 19,066 views
Uploaded on

ALM describes automated project management. …

ALM describes automated project management.
It includes
* Scrum based task management with issue tracking system.
* Contiguous Build
* Regression Test
* V-Model based Test process
* Defect management process

More in: Technology , Business
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
  • good slide
    Are you sure you want to
    Your message goes here
  • I need ppt for ALM & RM can you please send me. enivetha15@gmail.com
    Are you sure you want to
    Your message goes here
  • Hey. This is author of this PPT (Byunwook). I leaved Oracle. If you want to contact me. Plz 'email' link in Sideshare.
    THanx.
    Are you sure you want to
    Your message goes here
  • How can I download the pt ??
    Are you sure you want to
    Your message goes here
  • how to download this presentation ??
    Are you sure you want to
    Your message goes here
No Downloads

Views

Total Views
19,066
On Slideshare
0
From Embeds
0
Number of Embeds
6

Actions

Shares
Downloads
1,086
Comments
6
Likes
23

Embeds 0

No embeds

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
    No notes for slide

Transcript

  • 1. Practical Application Life cycle Management Framework - DRAFT Version Principle Consultant Byungwook Cho C. (byungwook.cho@oracle.com)
  • 2. AGENDA
    • Realistic of software development
    • Application Lifecycle Management
    • Practical Application Lifecycle Management Framework
    • Delivery Practical ALM Framework
    • Strategy for delivery practical ALM Framework
    • Case study – TBD
  • 3. <Insert Picture Here> Realistic of Software development
  • 4. Software development What we want just well defined process, well structured management In-depth software ”
  • 5. Software development But what ’ s real?
    • Realistic of developer
    • Additional Requirement
    • Changing Requirement
    • Meeting,reporting,education
    • Communication
    • Overtime job
    • Not tracked work
    • Realistic of manager
    • Always 99.999..% complete
    • Always overtime, but what is he doing?
    • When can I release?
    • What are risk items?
    • Wants know status of issue.
    • Task Priority
    • Realistic of software
    • A house of straw
    • Defects are discovered after freezing code
    • No testing for changing
    • Big bag integration
    The three little pigs ‘ A house of straw ’
  • 6. <Insert Picture Here>
    • Many Dr. and genius developed many methodologies
    • RUP,CBD,CMMI
    • Theories
    So …
  • 7.
    • There are only few genius
    • Theory is theory
    • We need more realistic and practical methodology
    But…
  • 8.
    • Practical development methodology
      • Erich Gamma, Joel Spolsky, Kent beck, Andrew Hunt
      • Iterative & Incremental
      • Based on agile
    • Difference between traditional methodology.
      • Traditional methodology assumes that there are no change and no defect in each previous step. (eg. Waterfall model)
      • It assumes there are many potential defects, and focus on finding the defects by frequent testing .
      • It assumes there may be requirement changes, and try to accept the changes.
      • Focus on collaboration and communication
    What is alternative solution? Practical Development
  • 9. <Insert Picture Here> ALM Application Lifecycle Management
  • 10. ALM definition
    • Definition from Wikipedia
    • Scope of ALM
    Application lifecycle management (ALM) is the marriage of business management to software engineering made possible by tools that facilitate and integrate requirements management, architecture, coding, testing, tracking, and release management.[1]
    • Covers whole SDLC
    • Complex and getting theatrical
  • 11. Current ALM market
    • All of vendor uses ALM as a marketing message
      • McCave just has testing solution, but they said “ We are ALM company ”
      • HP also has testing oriented solution, but they said “ We have ALM solution ”
    • There are few pure ALM players but..
      • Serena, Polarion etc already has matured ALM product.
      • But it is complex and hard to apply. (needs genius)
    • We need more practical ALM framework.
      • Depending on teams ’ maturity level.
      • Depending on customer ’ s mega process (methodology)
      • Depending on budget and time
  • 12. Four pillars of ALM
    • Process
      • Cover software life cycle from requirement analysis,implementation to release.
    • Quality Control
      • Provide capabilities to assure software quality
      • For example : testing, short release
    • Tracking
      • Provide visibilities and traceability of project activity.
      • Provide traceability between deliverable. For example traceability between requirement to actual code
    • Collaboration
      • Especially collaborate with Biz dept,.
      • Requirement management
      • Task management etc
    Process Quality Control Tracking Collaboration
  • 13. <Insert Picture Here> Practical ALM Framework
  • 14. Practical ALM
    • Lightweight, more easier and practical
    • Composed with open source tools and cheap popular commercial tools
      • Open source tools  Reduce learning curves
      • Less investment  Free or cheap commercial tools
    • Based on practical and agile methodology
  • 15. Practical ALM use case scenario Developer Perspective
    • Scenario
    • Developer goes to office in morning.
    • Run eclipse IDE.
    • In IDE, there are task list that are assigned to him.
    • He checks tasks with priority and pick up one.
    • Change a status of the task to  In Progress
    • Check out code from source code repository.
    • After implementation, he implement unit test with testing framework.
    • Execute test with standard build script. It automatically generate test result, code coverage rate and run code inspection. Code inspection shows illegal naming convention or potential bugs.
    • He fix the code and re-run the script.
    • He checks code coverage rate in eclipse IDE. After confirming the code coverage rate is meet target rate. He commit the code with the task ID.
    • CI (Contiguous Integration) system get a notify that main source repository has been changed.
    • It check out the code and run build script.
    • Compile and deploy the code to staging machine.
    • Run a test case. If test is broken. It automatically sends a notification to the developer and project manager. And then roll back the code in source repository and staging machine.
    • If the test is passed. it invokes code inspection and coverage analysis. If there is a problem (cannot match up target test code coverage rate or some static bug.) it will also report the problem.
    • Developer notify that all of implementation has been done successfully.
    • He write a his work history in task management system thru eclipse IDE. and change a status to  Resolved
    • Project Manage get a notification the task is resolved. He check up a note of the task and task result etc. And then change a status of the task to Closed
    • Developer come back home.
  • 16. Practical ALM use case scenario Project Leader/Manager perspective
    • Scenario
    • Project manager attend on requirement meeting with customer.
    • He summarize meeting minutes and extract tasks from that with customer's agreement (or confirm)
    • He creates new tasks with priority.
    • He plans the task followed by project milestone.
    • He check up developer's load by using task management system to find who is under lower load.And assign it to appropriate person.
    • Project manager estimate task implementation time with assigned developers.
    • Project manager adjust plan with schedule and estimated time.
    • During the project, project manager tracking the task status and everyday adjust the tasks.
    • Project manager or senior developer checks project health status with unit testing result and code coverage rate.
    • In addition senior developer concentrate testing in high code complexity area. It is founded by code complexity analysis report from CI tool's daily build.
    • Project manager also manage risk by using project status report from task management tool and reports from CI tools. And he check up high priority Opened  task every day to control risk.
  • 17. Practical ALM use case scenario Project Leader/Manager perspective
    • Scenario
    • Customer side PM come to office in the morning.
    • Execute web browser and go to ALM dash board page.
    • Each project has one page dash board. It shows how many tasks are opened and closed for each milestone.How many pending tasks are there. How many critical tasks are opened.
    • He just notify potential risk and project healthy status by using the dash board.
    • Make a meeting with Project Manager or Project Leader to control the risks.
    • If he want to track the critical issue (task). He subscribe the task. All of comment and change are automatically reported. So he doesn't need to make a meeting or make a phone call to check up progress of the critical task.
  • 18. Modules Extract Requirement Task Management Dashboard Source code management Contiguous Build Standard IDE Standard Build Script Testing Process Testing Framework Static Testing Defect Management Wiki based doc mgmt Forum based bi-directional communication Code review
    • Project management
    • Provide development environment
    • Assure quality
    • Leverage collaboration and communication
  • 19. Module 1. Task Management
    • Overview
      • Covers project management like
      • Project schedule management
      • Project resource (human) management
      • Requirement gathering
      • Provide KPI dashboard for management group
    • Scope
    Traditional Development process ALM / Task Management Process (Scrum based customized for Enterprise)
    • Manage major milestone
    • Define mega process
    • Manage daily task
    • Delicate resource and schedule management
  • 20. Module 1. Task Management Extract Requirement
    • Process
      • Extract requirement from meeting with client
      • Gather all of conversation to meeting minutes
      • Summarize meeting minutes and post it into internal wiki
      • Extract action items and get a agreement from customer.
      • And put that action items into task list.
    • Useful tools
      • Google site : gather all meeting minutes into one place with jump start
      • Atalssian confluence wiki : cheap and powerful.
    Customer PL Meeting Minutes Extract Action Item Ex) AI 1. Implement Logging with JMS (JIRA-101) AI 2. Evaluate Logging architecture in Test (JIRA-102 ) Task 1.Log whole of meeting 2. Extract action item in end of meeting. Action item should be agreed between customer and PL WIKI Meeting Task List or Task Management System Register action item to Task Process
  • 21. Module 1. Task Management Extract Requirement
    • Sample
    Action items Task number in task list It provide traceability from requirement to task
  • 22. Module 1. Task Management Task Management
    • Overview
      • Managing schedule and task of project team
      • Agile scrum methodology based.
      • Subset of project management methodology
        • Mega process is up to traditional methodologies or customer ’ s methodologies.
        • Cover delicate daily task management only
  • 23. Module 1. Task Management Task Management
    • Scrum methodology
    • Original scrum methodology is not appropriate Enterprise project management
      • Hard to manage risk.
      • Customer usually has their own methodology.
    • Need to customize for Enterprise project
  • 24. Module 1. Task Management Task Management
    • Task Management Process
      • Prepare product backlog It means requirement list
      • Release Planning – Define major milestone and release
      • Sprint Planning – Planning task for each iteration
      • Track Sprint – Daily tracking progress. Eg. Burn down chart
      • End Sprint – Simply provide demo to customer and get feed back
      • Update Sprint backlog – Update and re priories requirement
      • Retrospective – Review process and mature it
  • 25. Module 1. Task Management Task Management
    • Task Management Process
      • Process
    Duration 4~6 weeks Requirement Use case (Sub) Requirement Task Task Task Analysis Design Implementation Testing Working Item Documentation Meeting Testing Internal Release QA has to test every internal release based on test plan General Available (Release) High Risk Mandatory Functions Non-Functional Requirement Major UI Low Risk Optional Functions Enhancement Scheduling (Before implementation phase) Wik i Task Management System Tracking in WBS (Link to MS Project) Use numbering in title to track of Requirement Task Task Task Task LINK Release Release Sprint Sprint Sprint Requirement Requirement Requirement Requirement Requirement Requirement Meeting minutes Optional Depends on requirement granularity
  • 26. Module 1. Task Management Task Management
    • Task Type
        • Task is real working item to implement requirement
        • Task has to have traceability to requirement
        • Task type definition is very important.
        • Task can have hierarchy.
    [!] Requirement can be separated into more detail sub requirement System Task Sub Task Description Requirement N/A Detailed functional requirement. Requirement cannot contain any child requirement. It is leaf node of requirement tree. Name of requirement has to have number to provide traceability. Ex) 2.1.1 Logging inbound transaction WorkingItem Working items for developer. Eg. Coding,debugging etc. Documentation Write deliverables as document on wiki or document Test Develop and execute test of “ Requirement ” . It includes implementing Unit Test,design, execute etc. Meeting Meeting about specific issue Bug N/A Bug reported by somebody(not a author) after every internal release phase.
  • 27. Module 1. Task Management Task Management
    • Workflow
        • Workflow defines task handling process.
        • It is very important for task management
        • Keep it simple and easy to understand
        • It needs to define task type and hierarchy
  • 28. Module 1. Task Management Task Management
    • Workflow
      • - Requirement management process
    Open Scheduled In Progress Resolved Closed
    • Customer decide to close the child requirement
    • based following decision factor.
    • Code coverage rate
    • Code inspection result
    • Deliverables
    • Unit Test Result
    PL PL PL PL Customer Based on technical use case. Extract Child requirement and register it into JIRA Scheduling child requirement based on iterative development. It is linked to WBS in this phase In Every iteration, when the requirement related sub-task is started, change status to In Progress After finishing all of sub tasks for the child requirement. Change the status to “Resolved”
  • 29. Module 1. Task Management Task Management
    • Workflow
      • - Task management process
    Open Assigned In Progress Resolved Closed PL Need more Information Postponed PL creates detail task for developer. Task is usually finished 1~4 day. (1~2 days are recommended) Discuss with developer and estimate time. Assign the time to task Developer Developer Developer Developer PL PL Developer discuss with PL PL lets developer to postpone the sub task. The postponed task should be finished in same iteration phase. Or closed
  • 30. Module 1. Task Management Task Management
    • Mega Process Summary
  • 31. Module 1. Task Management Task Management
    • Implementation
      • Jump start with Microsoft Excel
    - Product back log - Sprint back log
  • 32. Module 1. Task Management Task Management
    • Implementation
      • Build task management system with Atlassian product
        • JIRA : for task management (2400 USD for professional version)
        • Confluence wiki : for requirement management (2200 USD for 50 user)
        • zAgile portal : Integrate distributed view
        • MS project : Manage schedule by PMO perspective
      • Reference Architecture
    http://www.atlassian.com http://www.zagile.com
  • 33. Module 1. Task Management Task Management
    • Implementation
      • Build task management system with VersionOne product
        • VersionOne provides hosted service. (No install is required, just use it)
        • It is based on agile methodology. (Atlassain JIRA is started from bug tracking system)
        • Sometimes hard to integrated with testing framework, CI tools, SCM etc.
        • http://www.versionone.com
  • 34. Module 1. Task Management Dashboard
    • Overview
      • Provide personalized view for each stakeholder
      • Show KPI for management group (Very important to convince manager)
        • Progress
        • Risk items
        • Velocity of team
        • Workload for resources
        • Escalated issues
      • Has to be integrated with other ALM modules
        • Test Result
        • Build Status
        • Test coverage rate
      • Just use only meaningful KPI
        • “ Patterns in our team velocity and temperature of Seoul are same..?! “
  • 35. Module 1. Task Management Dashboard
    • View
      • For project manager or customer
        • Burn down chart
        • Task or Bug open/close progress
        • Risk item list
        • Escalated task list
      • For project leader or manager in small project
        • Build status
        • Test result and test coverage rate
        • Workload of team members
      • For developer
        • Assigned task
        • Project status (Burn down chart, build status) / To understand whole of project status as a project team member
  • 36. Module 1. Task Management Dashboard
    • Implementation
    Task Management System Contiguous Integration System Testing Framework Dashboard OPEN API FILE PORTAL OR WIKI OR ??
    • Personalization
    • Integrated view
  • 37. Module 1. Task Management Dashboard
    • Example
      • Dashboard with Atlassian JIRA
    Number of task by priority Number of task by status Progress of each iteration TASK CREATE/CLOSE TRACKING Workload of team members
  • 38. Module 1. Task Management Dashboard
    • Example
      • Dashboard with Inflectra/Spira Team
  • 39. Module 2. Build Environment Source Code Management
    • Source Code Management (SCM)
      • Share source code
      • Track change of source
      • Manage branch
  • 40. Module 2. Build Environment Source Code Management
    • Branch management strategy
    Tagging Branch Tagging every build Every release Trunk (main branch)
    • Merge heuristically from Release Branch.
    • Merge systemically from patch branch.
  • 41. Module 2. Build Environment Source Code Management
    • Branch management strategy
      • Release branch
        • This branch is created when software has been released. (Even if it is internal release,manage branch)
        • Bug fix and enhancements of this version are applied to this branch.
        • Bug fix and enhancement of this version are merged to trunk branch by heuristically. (If released versions are different, logic or architecture can be changed, so not to use system merge)
      • Working branch
        • When some co-work is required and it will not be commited during a day, make a working branch.
      • Enhancement or patch branch
        • To make temporary patch or evaluate enhancement, make patch branch before applying fix to trunk branch.
        • It can be applied by using simple system merging.
      • Making a new branch provides additional managing effort.
  • 42. Module 2. Build Environment Standard IDE
    • Standard IDE
      • Enforce all of developer to develop on same environment
        • - Reduce error from environment
      • Save time to set up the environment
    • Process
      • Developer download standard development environment from wiki
      • Just unzip the file to personal computer.
      • Set up personal preference (server ip, directory structure etc )
      • And followed by instruction in wiki , quick start to develop
  • 43. Module 2. Build Environment Standard IDE
    • Standard IDE Packaging
    Zipped achieves IDE JVM Test Framework Std build script JEE Server Property file Install Guide Development Framework
  • 44. Module 2. Build Environment Contiguous Build
    • CI Process
    Build Failed Test failed Tagging current version and Roll back SCM to previous tagging version Event Trigger Send SMS/Email/IMS Check out source Build Packaging Deploy Testing & coverage anaysis Inspect code Roll back code Tagging Reporting Send Notification SCM Commit Event Scheduled time
  • 45. Module 2. Build Environment Contiguous Build
    • System configuration for CI
    Check out code Tagging or Rollback when build fail Invoke Send notification message when build or test failed Use Contiguous Integration Build Script Testing Framework SCM Notifier
  • 46. Module 2. Build Environment Contiguous Build
    • Jobs for CI
    Build Task Description Compile Only compile Packaging Packaging archives Deploy Deploy achieves to server Testing Run Testing Testing and coverage analysis Run Testing with code coverage analysis Code Inspection Run Code Inspection Full build Compile  Packaging  Deploy  Testing with coverage analysis  Code Inspection Job Description Full build Compile  Packaging  Deploy  Testing with coverage analysis  Code Inspection
  • 47. Module 3. Testing Automation Overview
    • Overview
      • One of most important objective in ALM is guarantee quality of software
      • Test Automation Module is designed for cover whole test cycle of V-Model.
      • Unit/Integration tests are covered by development team and System/UAT are covered by QA team. Followed by your role , you can customize this process and framework
    • Theoretical background
      • Mega process is based on ISTQB testing principals.
      • Unit testing and automated regression testing is based on Practical methodology
  • 48. Module 3. Testing Automation
    • Testing Model
      • V-Model : Enhancement of waterfall model
      • Verification & Validation
    < Verification & Validation > < V-Model > Verification Validation To verify the artifacts that has been produced in each development cycle. Valid & estimate the system Review & Inspection with artifacts from each development step Testing with system Static testing Dynamic testing
  • 49. Module 3. Testing Automation
    • Testing Level
    ※ Integration strategies Level Description Testing Type Responsibility Unit Test Verify software component White box test Lead by Dev Team Use Code coverage Integration Test Verify integration between component. Verify software flow ,interface & interaction White box test Lead by Dev Team Use Mock Object (Test Stub ,Driver) System Test Test system over production environment to verify system itself and production environment Include Functional & Non Functional Test (Availibility, Stablity, Extendability, Performance etc) Black box test Lead by QA Team (Specialized for system test) Acceptance Test Verify customer ’ s requirement (End User) Verify customer ’ s Legal issue (Legal) Verify customer ’ s maintanance issue (SM) Black box test Lead by QA Team or Customer Alpha Test Beta Test Top down Bottom up Big-bang Backbone Desc Integrate from top (use test stub) Integrate from bottom (use test driver) Integrate all in time Most important & high risk component first Adv easily find architectural defect Can test implementation logic Short time Ex. BPM SOA svc Small project
  • 50. Module 3. Testing Automation Overview
    • Test Cycle
      • Each test is consists of test cycle
      • Test cycle
        • Pre test - check up testability
        • Main test
        • Conformation test - check up the defect found in previous “ Main test ”
        • Regression test - check up the impact from change
    TEST CYCLE Pre Test Main Test Conformation Test Regression Test
  • 51. Module 3. Testing Automation Testing Process
    • Mega process of Testing
      • Based on ISTQB testing model
    Define test scope objective Risk anal Set up strategy Estimate resource & time Planning PRE TEST TEST Monitoring Report
  • 52. Module 3. Testing Automation Testing Process
    • Step 1. Risk Analysis
      • Risk = Likelihood * impact
      • Risk identification
        • Recommend < 36 identification .
        • Likelihood : Complexity, Implementation technical Level, Size,Developer skill
        • Impact : Biz impact
    Erik van neneendaal, Risk Based Testing, STAREAST,2006 Likelihood Impact factor factor factor factor Risk item Risk item Likelihood Impact STA (Severe Test Area) SSTA (Strong Test Area) FTA (Fundamental Test Area) ITA (Intensive Test Area)
  • 53. Module 3. Testing Automation Testing Process
    • Step 2. Define test strategy
      • Risk based test strategy
      • Other strategies
        • Analytical approaches
        • Model based approaches
        • Methodical approaches
        • Dynamic and heuristic approaches
    Likelihood Impact STA (Severe Test Area) SSTA (Strong Test Area) FTA (Fundamental Test Area) ITA (Intensive Test Area) Likelihood Impact STA (Severe Test Area) SSTA (Strong Test Area) FTA (Fundamental Test Area) ITA (Intensive Test Area) ( Low Level Test) ( High Level Test) - Unit Test - Integration Test - System Test - Acceptance Test
  • 54. Module 3. Testing Automation Testing Process
    • Step 3. Estimate resource and time
  • 55. Module 3. Testing Automation Testing Process
    • Step 4.Test planning & control
      • Define activity of each testing level
      • Define milestone ,resource ,schedule.
      • Make a plan based on analyzed risk factor
      • Define test strategy based on risk
      • Define approach & techniques for testing (testing techniques, coverage, test item, test ware)
      • Must include time & resource for preparing testware
      • Define completion condition
  • 56. Module 3. Testing Automation Testing Process
    • Step 5. Test analysis and design
      • Test analysis
        • Review - test basis, testability
        • Identify test requirement & test data
        • Identify test infra and tools
      • Test design
        • Identify test condition
        • Test case specification
        • Define test procedure
  • 57. Module 3. Testing Automation Testing Process
    • Step 5. Test analysis and design (Technique)
    • Design technique
      • Specification based
        • Equivalence partitioning, Boundary value analysis
        • Pairwise, Decision table testing
        • State transition testing, Usecase testing
      • Structure based
        • Control Flow Test
        • Basic Path Test – Based on Cyclomatic complexity (Edge – Node + 2P) P = number of connected component
        • Elementary comparision test – Based on MC/DC Coverage
        • * Coverage : Statement, Condition, Decision, Condition Decision Coverage, Modified Condition/Descision Coverage(MC/DC) , Multiple Condition Coverage
      • Experience based
        • Exploratory Testing – Adhoc testing, based on test charter, define timezone
        • Classification Tree method
        • CheckList
  • 58. Module 3. Testing Automation Testing Process
    • Step 6. Execute Test
      • Execution
        • Define serverity & priority
        • Develop TC,Test data,Test stub & driver,Test script,Test suite
      • Reporting
        • Logging expected result, real result
        • Current status
        • Time & resource usage
      • Defect tracking
        • Use Bug(Issue) tracking system ( Mantis,Bugzilla,JIRA,Trac)  can be integrated with Task Management module in ALM.
        • Define tracking workflow
    TEST CASE TESTING TOOL TEST DATA MODULE MODULE MODULE MODULE TEST SCRIPT TEST SUITE1 TEST SUITE2 TEST SIMULATOR Test Execution Test Report Defects Issue Tracking System Progress Report Test logs TEST DRIVER TEST STUB TARGET SYSTEM
  • 59. Module 3. Testing Automation Testing Process
    • Step 7. Evaluate exit criteria and reporting
      • Test exit Criteria
        • Coverage rate, Duration, # of unresolved defect
      • Report Test Result
        • Progress report
        • Release advice
        • Final report per testing level
      • Matrix
        • Defects per KLOC, Days test effort per requirement, Unresolved defect impact analysis, Defects / Hour ,Cumulative defects / Day ,Test coverage / Day , Risk vs number of TC
  • 60. Module 3. Testing Automation Testing Process
    • Step 8. Testing closure activities
      • Evaluating
        • Evaluate test target
        • Evaluate test process for maturing next text
      • Final reporting
      • Store test-ware
  • 61. Module 3. Testing Automation Testing Framework
    • Testing framework
      • Tools and environment to support executing testing
    Unit Test Framework Load Testing Tools UI based Automated Testing Monitoring Test case Management Defect Management Tool Contiguous Integration Tool Unit Test Integration Test System Test UAT Regression Test Regression Test Regression Test Report Defects Testcase Management Execute Test
  • 62. Module 3. Testing Automation Testing Framework
    • Unit Test framework
    Test Test Test Test Application Unit Test framework Support J2ee incontainer Testing Validate test itself User Interface Business Logic DAO Interface (RMI,REST,WS*) xUnit xUnit for DBTesting xUnit for Protocol or message Type xUnit for UI In Container Testing Tool Code Coverage Analysis Tool
  • 63. Module 3. Testing Automation Testing Framework
    • Load Testing
      • Support non-functional testing.
      • Generate load.
    • Monitoring
      • Application Performance Monitoring Tools
      • Used for finding bottleneck in performance.
      • Eg. Jennifer(Korea),Wily Introscope. (different from Profiler)
    • UI Based Automated Tool
      • Let tester to free from factory job.
      • Record user activity and replay it.
      • Validate expected action.
      • It is used for user acceptance test. (Especially functional testing)
      • ※ Test case can be reused in Stress testing.
    • Test case management
  • 64. Module 3. Testing Automation Static Testing
    • Static testing techniques
      • Not running code, review code statically.
      • Code Review - Manual Review is done by human
      • Static Analysis - Static review is done by code inspection tools
    Static Testing By human By Automated tool Code Review Static Analysis Automated inspection tools
  • 65.
    • Code Review
      • Introduced by Fagan as Code Inspection.
      • Developed to more vary Code Review.
      • Very effective and low cost action to keep code quality.
      • Kind of Code Review
        • Code Inspection
          • Intend to find a defect. Executed with formal process and team. (For more information. See Fagan’s Code inspection)
          • Team has four roles. (Mediator – project manager, Reader – understand system to be inspected, Designer/Coder – Inspect and generate recommendation, Tester – Test inspected defect and validate) And Six steps in process.
          • To execute code inspection, you need to have formal inspection team in your organization or bring a consulting who has highly trained.
          • Performance of inspection is poor. But effect is very nice. Code Inspection is done after implementation. Recommend to use code inspection in system testing phase.
        • Team Review
          • Less formal than code inspection. But it has process also.
          • Mediator manage Team review session, and code author explains about his code.
          • Find a defects or gather idea for enhancement and write it to code review report. (Recommend to use wiki)
          • The ideas and defects are handled as Task in ALM/Task Management
          • Recommend one time per week. Or after testing, after internal releasing, after sprint.
        • Walkthrough
          • Informal review. It can be used to share idea.
          • Only author has a rule that explains his code. Nobody has a responsibility.
          • All things are up to the author
        • PeerReview
          • Review code on a desk.
          • Usually 2~3 people can attend on review. Review code over a monitor and share idea.
          • It is nice to educate junior developer or who doesn’t have experience about product or skills.
    Module 3. Testing Automation Static Testing
  • 66. Module 3. Testing Automation Static Testing
    • Static Analysis
      • Test code or diagram with “static analysis tool ”
      • Analysis
        • Can find syntax error or well known bugs
        • Validate naming rules
        • Can find dead code or missing parameter
        • Can analysis code complexity & dependency
      • Quality?
        • Efficient but not so smart
        • It need a rule and need a maturing
      • Easy to appeal your manager
      • Tools
        • Open source – PMD, Find Bugs,Jdepend etc.
        • Commercial – Klockworks, Jtest etc.
  • 67. Module 3. Testing Automation Defect management
    • Defect management Process
    Call coordinator Validator Project manager Developer QA Engineer User 1.Create case Filter
    • Summary
    • Description
    • Log and screen shout
    • Reproducible case
    2.Yank & validate 2-1 Request more information 3.Assign 5.Assign 4. Priories and schedule 6.Fix (Workaround or patch) 7. Provide Solution 8. Confirm 9. Commit to main branch with Defect # Defect Reporter Implementation Team Resolved defects list is combined to release note Defect Management System Source code Management system
  • 68. Module 3. Testing Automation Defect management
    • Defect management system
      • Defect management system is integrated with SCM to track change of source code to fix the defect.
      • Web based SCMbrowser shows changes for defect in a web from Defect management system link
      • Task Management system can be used as Defect management system as well.
      • In Big project or if there are so many defects are reported then separate defect management system and integrate it into task management system. It needs to separate organization to implementation and defect handling team
    Defect Management System Source code Management system Web based SCM browser Track changes Link Shows change of code Task Management System Defect is task
  • 69. <Insert Picture Here> Delivery Practical ALM Framework
  • 70. Aapproach to delivery ALM solution
    • There is 3 perspective to delivery ALM.
  • 71. Process Developer Management (PM/PMO/PL) QA Development Lifecycle Testing Model (Based on V-Model) Agile/Scrum Base Practical methodology base V-Model Based Analysis Design Implementation Testing Unit Testing Integration Testing System Test UAT Task Management Process (Traditional Methodology) Task Management Process (ALM/Scrum based) Task Management Process (ALM/Scrum basder) Build Process Test Process SCM Process Test Process Defect Mgmt Process Task Management Process (ALM/Scrum based))
  • 72. Reference Architecture ALM architecture overview
    • Reference architecture describes what we need to build up ALM system with tools and software.
    • Layout of ALM architecture
    Build Status Test Result Task Status Defect Status Check out Check in Management (PM/PMO/PL) Developer QA Assign Task Provide guide & process Manage document. Task Mgmt Contiguous Integration Testing Framework Build Script Source Code Management IDE Build Script Testing Framework Defect Management Testing Framework (System& UAT) Dashboard Wiki
  • 73. Reference Architecture Task Management.
    • Atlassian product is very popular , so easy to get a support and get a resource who can use it.
    • Atlassian has CI tools, SCM connector, Coverage analysis tools,,Wiki etc. So it provides strong integrity.
    • But, origin of JIRA is for Bug Tracking System not a Agile based task management system . It needs a customization in process and tool.
    Atlassian Confluence Wiki Atlassian JIRA zAgile Portal MS Project Requirement Management Task Management Major Milestone management for mgmt org. (eg. PMO) Personalization, Integrated view Scrum based task management process Scrum in JiRA : http://www.slideshare.net/paulrene/adapting-jira-for-scrum-presentation
  • 74. Reference Architecture Task Management – Alternative solution “Spira Team”
    • SpriaTeam has great agile approach . It already contains concept of Release,Iteration,Requirement,Task,Test..
    • Most excellent thing is it provides traceability from Requirement-Task-Test. It enables track whole things in manner of ALM.
    • It also provides some connector to integrate with issue tracking system (JIRA,Bugzilla)
    • But it doesn ’ t support integration with CI tools or coverage analysis tools . It should be implemented with other product.
    • Company looks new and small, so they seems like have few reference yet. But cheap.. 
    Atlassian Confluence Wiki Infrectra Spira Team Requirement Management Requirement Management, Task Management, Schedule mgmt & Test case management Scrum based task management process LINK
  • 75. Reference Architecture Task Management – Another alternative
    • VersionOne ( http://www.versionone.com )
      • - Agile Oriented
      • They provide hosted task management tool and sell it as package.
      • Well support scrum methodology.
      • Free license for one small team for evaluation. Try it … 
    • Rally Enterprise ( http://www.rallydev.com )
      • Agile Oriendted
      • They provide hosted (Saas) Task management tool.
      • It is based on Agile methodology. And they have great agile knowledge.
      • But it is not inconvenient for my practical ALM approach. (It doesn ’ t mean their product is poor)
      • If you want to get a deep consulting about Task management process, I want to recommend this one.
    • Serena (http://www.serena.com)
      • It is traditional pure ALM player .
      • It is very popular and covers big area of ALM.
      • It needs high level team maturity.
      • I just want to recommend this one to Enterprise who really willing to invest ALM and Governance.
    • Polarion (http://www.polarion.com)
      • It is also very powerful ALM vendor.
      • Same as serena, it covers whole of ALM scope and IT governance. It supports CMMI level as well and has very rich capabilities .
      • There is light version- Polarion Track & Wiki (Issue tracking + Wiki + light build mgmt + Eclipse Plug in)
    • Codebeamer
    ※ This evaluation of ALM tools is up to my personal opinion.
  • 76. Reference Architecture Test Automation JUnit EasyMock Cactus Selenium DBUnit XMLUnit Japex Basic testing framework Mockup test J2EE In container test UI Test DB Test XML Test Junit based stress test Cobertura Test Code coverage analysis SourceForge Grinder Stress Test SOAPUI Stress Tes t for Webservice,REST Find Bugs Code inspection Atlassian JIRA Defect tracking Functional Testing Framework (Unit testing/Integration Testing) Non functional testing (System Testing) Static Testing Defect mgmt V model
    • Alternative
    • HP Load Runner (Commercial)
    • Oracle ATS (Commercial)
    • Apache Jmeter (OpenSource)
    • Alternative
    • PMD (Open Source)
    • Alternative
    • Atlassian Clover (Commercial)
    • Alternative
    • Mantis (Free)
    • Bugzilla (OpenSource)
  • 77. Reference Architecture Test Automation
    • Design principal of Test framework is to support automated testing and regression test.
    • Test framework covers functional testing includes unit test and integration test. (Integration test can be viewed as coarse grained unit test)
    • Test framework combines – Functional testing capabilities followed by characteristics of component like DB Testing,UI Testing,XML based messaging and Unit based stress test, coverage analysis and mock up test environment etc.
    • This test framework covers broad range of testing. It means it is not easy. (concentrated on Unit testing and Integration testing. Partially covers System test and UAT. I recommend to use more matured tools for System test and UAT)
    • ROI of code inspection is very high. How to managing rule set is most important.
    • Optionally, I recommend to set up test case management tools from QA team. (ex. Testopia)
  • 78. Reference Architecture Build Environment
    • Apache ant will not be developed anymore, Maven will replace ant. Maven also support unified library management.
    • Hudson is now most popular free CI tools.
      • - It supports a lot of plug ins for JIRA,Junit,Find Bugs etc.
      • It is very easy to learn.
    Functional Testing Framework Test framework Maven Standard Build Script Subversion Source code Management Eclipse Standard IDE Hudson Contiguous Integration Practical methodology ( Joel Spolsky,Kent Beck, Erich Gamma)
    • Alternative
    • Atlassian Bamboo
  • 79. Reference Architecture Collaboration
    • Just provide minimum collaboration environment.
    • Atlassian confluence provides a lot of plug in. Like MS-Office connector etc. It is very useful to sharing your team knowledge.
    • If you already have collaboration infrastructure in your company, do not extend product.
    Atlassian Confluence Wiki Document Management Knowledge Transfer JForum Bi-directional knowledge exchange
  • 80. Reference Architecture Development environment
    • In staging environment, highly recommend to use hardware virtualization (CPU,Memory). It is very useful to validate system capacity. (Scalability test result is be basis of capacity planning)
    Development Environment Staging Production Responsibility Development Team QA Team SM Team GA version (General Available) Internal Release GA Unit Test Integration Test System Test (Software perspective) System Test (Include validating hardware infrastructure) UAT Virtualization is useful
  • 81. Organization Role & Responsibility PMO (Project Mgmt Office) Implementation Team (PL) Application Architect Quality Architect ALM Master Extract Requirement Task Management Dashboard Source code management Contiguous Build Standard IDE Standard Build Script Testing Process Testing Framework for Unit,Integration Testing Static Testing Defect Management Wiki based doc mgmt Forum based bi-directional communication Code review Testing Framework for System,UAT
    • Who has a responsibility to build up process and system?
    Mentoring (Very important)
  • 82. Organization Role & Responsibility Implementation Team (PL) Extract Requirement Task Management Dashboard Source code management Contiguous Build Standard IDE Standard Build Script Testing Framework for Unit,Integration Testing Static Testing Defect Management Wiki based doc mgmt Forum based bi-directional communication Code review
    • Implementation team uses a lot of things in ALM
      • It needs a strong mentoring to use it. Mentoring is not a option but a mandatory..!!
    ALM Master Mentoring (Very important)
  • 83. Organization Role & Responsibility
    • ALM Master
      • Required capabilities : Understand whole ALM architecture & tools and process - May be genius? No. think with customer, act with customer. (Just right understanding of their ALM and help them not to miss a way..)
      • Responsibility
        • Help customer to build up their ALM process and system.
        • Help customer to adapt ALM.
        • Inspect ALM usage in the team. (Inspect task management system note, Review Wiki usage and structure. Check up Build script and CI status. ) and mentor them to right way.
        • Convince customer or project manager that no more free lunch.!! (to do something, let them know it needs a time and resource)
        • Each stakeholder wants to customize ALM for their perspective. In ALM, all of stakeholder has to share their perspective. (Even if they can see different views for their role) – It is liquid flow. So ALM master lead the stakeholder to have same perspective..
      • Help customer to change their culture slowly,smooth without revolt, awareness.
  • 84. <Insert Picture Here> Strategy for delivering Practical ALM Framework
  • 85. Key Strategy
    • Liquid
      • Each process of ALM are integrated as LIQUID
    • Seamless
      • Tools and solution that are used to realize ALM are integrated as seamless.
      • Should be looks as one platform.
    • Process Oriented
      • Tool and Systems are just tool to realize ALM concept.
      • Do not depend on Tool. Process is key.
    • Step by step depends on team level
      • Even if ALM is lightweight. It is hard to apply at once.
      • Apply ALM modules step by step followed by team maturity level.
    • Be Simple
      • Start of ALM is to remove complexity of traditional methodology.
      • Complexity reduce efficiency and requires a lot of learning cost.
    • And need a coach . (ALM Master)
    • Do not enforcing process or tool, but change culture.
      • Most important point of delivering ALM is changing culture.
    • Delivery ALM secretly.
      • Developer should not aware change. It means changes are applied smoothly.
  • 86. Delivery Step
      • Overall road map of applying ALM
    Build Environment Task Management Test Automation Collaboration Level 1 SCM,CI, (AA) Standard BuildScript Level 2 Standard IDE (PL) Excel based Task Management(PL) Code Review (PL) Wiki (PL) Level 3 Branch mgmt strategy (AA) System based Task Mgmt (PM/PMO) Dashboard (PMO) UnitTest (AA or PL) Regression Test (AA or PL) Level 4 Extract Requirement (AA) Static Analysis (AA) Level 5 Testing process (QA) Level 6 Defect mgmt system (QA)