SlideShare a Scribd company logo
Enterprise Metrics on AgileSagi SmolarskiDirector, Software DevelopmentSagi.Smolarski@itg.com[CC] Daniel Goodwin via Wikimedia Commons
DisclaimersThe information contained in this presentation is provided for informational purposes only.  All trademarks, service marks, and trade names not owned by ITG are owned by their respective owners.…in other words, the data presented has been modified and does not represent real data for ITG teams.
AgendaWhy do we need metrics for agile?How do we generate those metrics?Which metrics do we look at?Pros and cons of looking at those metrics.
Investment Technology GroupNYSE: ITGwww.itg.comLeading provider of trading solutionsTrading Front-EndsTrading AlgorithmsResearch (fundamental analytic, pre/post-trade)Trading desksElectronic trading connectivityLiquidity poolDevelopment operation: 300 Developers, 100 QA Analysts,  ~60 teams
Iteration MetricsQuality MetricsProcess BaselineITG’s Agile Transition TimelineITG TritonXPEnterprise rollout planRally+Scrum, +Lean90% of teams have been trained and are using RallyBest Execution Management SystemXP Pilot200020012002200320042005200620072008200920102011Massive transitionInformal, spotty, inconsistent adoption
Our Process Baseline – How We Expect Teams to WorkExcerpt
The journey to Agile is a long, winding road…Are we moving forward?[CC] – Sten via Wikimedia Commons
Why Measure?“If you cannot measure it you cannot improve it.”Lord Kelvin
Why Metrics?Teams (Inspect & Adapt):Are we doing okay?
How can we improve?
Are we doing what the company expect from us?Coaches, PMO (Train and Coach):Are teams using what we taught?
Which teams need our help?Executives (Govern & Steer):What are we getting out of the agile transition?Are teams sticking to our process baseline and to enterprise initiatives?Is productivity/quality improving?
Team ATeams Process HealthData is readily available in RallyTeam BVelocityVelocityvs.Burndown/Story AcceptanceBurndown/Story Acceptancevs.Isn’t this enough?
Why Metrics? - Dumbing down…This is too complex for team members to interpret and monitor…Are the charts okay?
Why Metrics – Scaling upHow do we watch 80 teams? Lines of products? The whole enterprise??
Types of MetricsQualitativeSatisfaction of stakeholdersProduct Management, Client Services, Product Support (deployment, service), Delivery teamTeams adherence to practicesAgility questionnaireQuantitativeQuality metricsProcess health metricsTime-to-MarketWe’ll be focusing on these
What Would We Want to Measure?ProductivityQualityROISatisfactionSome of these are not easy to measure so we have to find proxies
What We Are Actually MeasuringSatisfactionQualityProcessAs a partial proxy for productivityStakeholders SurveysProduction defectsDefects DebtWork-in -ProgressVelocity StabilityFull Completion or workGradual acceptance of workChurnSmall StoriesPractices Radar map
How We Generate MetricsHow do we obtain data?Rally’s Web Services APIRest and SOAPAccess to virtually all data in the toolUsers, Releases, Iterations, Stories, Tasks, Defects, Test cases…Ruby toolkit… or any WS libraryWe use mostly C#Automated generation, monthlyHow do we make sense out of mounds of data?
The Metrics
Quality Metrics
Quality MetricsDefects Found in ProductionMetric #1Production DefectsGoalJanFebMarAprMayJunJulAugSepOctNovDecJanFeb20102011Team Level & Enterprise Level
Metric #2Quality MetricsFix-Bugs-FirstIf you are fixing defects firstAs part of existing development, within the iterationBefore new development for regression defectsThen…The number of defects in your backlog should be smallThe age of open defects should be lowTogether…Defects Debt should be lowDefects Debt = [# open defects] x [avg age of open defects]
Quality MetricsProject-Level Defects DebtMetric #2
Quality MetricsEnterprise Defects DebtMetric #2Defects Debt
Process Health Metrics
Team Process HealthAgility Questionnaire
Scope CreepChurnMetric #3Iteration Cumulative FlowNo Scope CreepPlan EstimatesDay 1Day 2Day 3Day 4Day 5Day 6Day 7Day 8Day 9Day 10Day 11Day 12Day 13Day 14Churn (Scope variation) = 0
Work in Progress & Scope CreepMetric #3Iteration Cumulative FlowPlan EstimatesDay 1Day 2Day 3Day 4Day 5Day 6Day 7Day 8Day 9Day 10Day 11Day 12Day 13Day 14Churn = 30%
Work in Progress & Scope CreepMetric #3Iteration Cumulative FlowPlan EstimatesDay 1Day 2Day 3Day 4Day 5Day 6Day 7Day 8Day 9Day 10Day 11Scope Creep:		Team Cannot Commit	Disruptive (task switching?)	Less efficientDay 12Day 13Day 14Churn = 30%
Work in ProgressWIP AverageMetric #4Iteration Cumulative FlowNo Scope CreepWork In ProgressPlan EstimatesDay 1Day 2Day 3Day 4Day 5Day 6Day 7Day 8Day 9Day 10Day 11Day 12Day 13Day 14PlannedWIP average = 10%In ProgressAccepted
Work in Progress & Scope CreepMetric #4Iteration Cumulative FlowPlan EstimatesDay 1Day 2Day 3Day 4Day 5Day 6Day 7Day 8Day 9Day 10Day 11Day 12Day 13Day 14PlannedWIP average = 70%In ProgressAccepted
Work in Progress & Scope CreepMetric #4Iteration Cumulative Flow	Large WIP = Long Cycle TimeRisk of not completing work within iterationPlan EstimatesDay 1Day 2Day 3Day 4Day 5Day 6Day 7Day 8Day 9Day 10Day 11Day 12Day 13Day 14PlannedWIP average = 70%In ProgressAccepted
Full Acceptance – Final Acceptance %Metric #5Iteration Burn-up100% AcceptedPlan Estimates (Points)50% AcceptanceDay 1Day 2Day 3Day 4Day 5Day 6Day 7Day 8Day 9Day 10Day 11Day 12Day 13Day 14Day 12Day 13Day 14AcceptedAcceptance rate on the last day = 100%
Full AcceptanceMetric #5Iteration Burn-up100% AcceptedPlan Estimates (Points)50% AcceptanceDay 1Day 2Day 3Day 4Day 5Day 6Day 7Day 8Day 9Day 10Day 11Day 12Day 13Day 14Day 13Day 14Day 12Day 13Day 14AcceptedAcceptance rate on the last day = 55%
Full AcceptanceMetric #5Iteration Burn-up100% AcceptedPlan Estimates (Points)50% AcceptanceDay 1Day 2Day 3Day 4Day 5Day 6Day 7Day 8Day 9Day 10Day 11Day 12Day 13Day 14Day 13Day 14	Partial CompletionHard to planGreater cycle timeInefficientDay 12Day 13Day 14AcceptedAcceptance rate on the last day = 55%
Gradual Acceptance – 50% dayMetric #6Iteration Burn-up100% AcceptedPlan Estimates (Points)50% AcceptanceDay 1Day 2Day 3Day 4Day 5Day 6Day 7Day 8Day 9Day 10Day 11Day 12Day 13Day 14>50% points acceptedDay 12Day 13Day 14AcceptedDay of 50% Acceptance = 8days/14days = 57%
Gradual AcceptanceFull AcceptanceMetric #6Iteration Burn-up100% AcceptedPlan Estimates (Points)50% AcceptanceDay 1Day 2Day 3Day 4Day 5Day 6Day 7Day 8Day 9Day 10Day 11Day 12Day 13Day 14Day 12Day 13Day 14AcceptedDay of 50% Acceptance = 14/14 = 100%
Gradual AcceptanceMetric #6Iteration Burn-up100% AcceptedPlan Estimates (Points)50% AcceptanceDay 1Day 2Day 3Day 4Day 5Day 6Day 7Day 8Day 9Day 10Day 11Day 12Day 13Day 14Day 12Day 13Day 14Accepted	Late AcceptanceHigh risk of not completing workLate feedback on workDay of 50% Acceptance = 14/14 = 100%
Velocity StabilityVelocity VariationMetric #7Velocity605040302010Iteration 23Iteration 24Iteration 25Iteration 26Iteration 27Iteration 28Iteration 29Iteration 30Iteration 31Iteration 32Day 12Day 13Day 14Accepted within iterationVelocity is fairly stableAccepted after iterationVariation = 7%Never accepted
Velocity StabilityVelocity VariationMetric #7Velocity605040302010Iteration 23Iteration 24Iteration 25Iteration 26Iteration 27Iteration 28Iteration 29Iteration 30Iteration 31Iteration 32Day 12Day 13Day 14Accepted within iterationVelocity is unstableAccepted after iterationVariation = 87%Never accepted
Velocity StabilityMetric #7Velocity605040302010Iteration 23Iteration 24Iteration 25Iteration 26Iteration 27Iteration 28Iteration 29Iteration 30Iteration 31Iteration 32Day 12Day 13Day 14	Unstable VelocityLow predictabilityHard to planInefficient?Accepted within iterationVelocity is unstableAccepted after iterationVariation = 87%Never accepted
Defects DebtVelocity StabilityChurn% CompletionWIP, Day of 50% Acceptance, Story SizingOur Process Baseline… and metrics we can collect
Team DashboardAll together now…Day 12Day 13Day 14Day 12Day 13Day 14Day 12Day 13Day 14Day 12Day 13Day 14Day 12Day 13Day 14Day 12Day 13Day 14
Teams Process HealthEnterprise Process Health Map(Probably) Healthy ProcessLine of Business #1Line of Business #2Line of Business #3Line of Business #4Line of Business #5ProjYProj kProj XProj BProj NProjאProject AProhj ZProj EProjבProj 1Proj CProj WProj גProj 2Proj דProj 3ProjהProj DProj 4Proj 5Proj MProj LProjו(Possibly) Unhealthy Process

More Related Content

What's hot

Agile Metrics, Value, and Softwre
Agile Metrics, Value, and SoftwreAgile Metrics, Value, and Softwre
Agile Metrics, Value, and Softwre
Don McGreal
 
Agile Workshop: Agile Metrics
Agile Workshop: Agile MetricsAgile Workshop: Agile Metrics
Agile Workshop: Agile Metrics
Siddhi
 
Seven Key Metrics to Improve Agile Performance
Seven Key Metrics to Improve Agile PerformanceSeven Key Metrics to Improve Agile Performance
Seven Key Metrics to Improve Agile Performance
TechWell
 
Top Agile Metrics
Top Agile MetricsTop Agile Metrics
Top Agile Metrics
XBOSoft
 
Agile Metrics
Agile MetricsAgile Metrics
Agile Metrics
Erik Weber
 
Top 10 Agile Metrics
Top 10 Agile MetricsTop 10 Agile Metrics
Top 10 Agile Metrics
XBOSoft
 
Governance of agile Software projects by an automated KPI Cockpit in the Cloud
Governance of agile Software projectsby an automated KPI Cockpit in the CloudGovernance of agile Software projectsby an automated KPI Cockpit in the Cloud
Governance of agile Software projects by an automated KPI Cockpit in the Cloud
pliXos GmbH
 
Big Apple Scrum Day 2015 - Advanced Scrum Metrics Reference Sheet
Big Apple Scrum Day 2015 - Advanced Scrum Metrics Reference SheetBig Apple Scrum Day 2015 - Advanced Scrum Metrics Reference Sheet
Big Apple Scrum Day 2015 - Advanced Scrum Metrics Reference Sheet
Jason Tice
 
Agile Metrics for Senior Managers and Executives
Agile Metrics for Senior Managers and ExecutivesAgile Metrics for Senior Managers and Executives
Agile Metrics for Senior Managers and Executives
VersionOne
 
How smooth is your agile ride
How smooth is your agile rideHow smooth is your agile ride
How smooth is your agile ride
Anu Khendry,ACC,SAFe Aglst, PMI-ACP,CSPO,PMP
 
Agile Metrics
Agile MetricsAgile Metrics
Agile Metrics
Alexey Krivitsky
 
Agile Metrics : A seminal approach for calculating Metrics in Agile Projects
Agile Metrics : A seminal approach for calculating Metrics in Agile ProjectsAgile Metrics : A seminal approach for calculating Metrics in Agile Projects
Agile Metrics : A seminal approach for calculating Metrics in Agile Projects
Prashant Ram
 
Agile by numbers
Agile by numbersAgile by numbers
Agile by numbers
Elena Yatzeck
 
Agile KPIs vs. Traditional KPIs – A mind shift
Agile KPIs vs. Traditional KPIs – A mind shiftAgile KPIs vs. Traditional KPIs – A mind shift
Agile KPIs vs. Traditional KPIs – A mind shift
Javier Espinosa de los Monteros Foret
 
Agile Metrics - how to use metrics to manage agile teams
Agile Metrics - how to use metrics to manage agile teamsAgile Metrics - how to use metrics to manage agile teams
Agile Metrics - how to use metrics to manage agile teams
XBOSoft
 
Agile Metrics
Agile MetricsAgile Metrics
Agile Metrics
Mikalai Alimenkou
 
Simple Lean Agile KPIs
Simple Lean Agile KPIsSimple Lean Agile KPIs
Simple Lean Agile KPIs
Yuval Yeret
 
Agile Metrics
Agile MetricsAgile Metrics
Agile Metrics
nick945
 
Agile dashboard
Agile dashboardAgile dashboard
Agile dashboard
Fadi Stephan
 
Agile metrics - Measure and Improve
Agile metrics - Measure and ImproveAgile metrics - Measure and Improve
Agile metrics - Measure and Improve
WemanityUK
 

What's hot (20)

Agile Metrics, Value, and Softwre
Agile Metrics, Value, and SoftwreAgile Metrics, Value, and Softwre
Agile Metrics, Value, and Softwre
 
Agile Workshop: Agile Metrics
Agile Workshop: Agile MetricsAgile Workshop: Agile Metrics
Agile Workshop: Agile Metrics
 
Seven Key Metrics to Improve Agile Performance
Seven Key Metrics to Improve Agile PerformanceSeven Key Metrics to Improve Agile Performance
Seven Key Metrics to Improve Agile Performance
 
Top Agile Metrics
Top Agile MetricsTop Agile Metrics
Top Agile Metrics
 
Agile Metrics
Agile MetricsAgile Metrics
Agile Metrics
 
Top 10 Agile Metrics
Top 10 Agile MetricsTop 10 Agile Metrics
Top 10 Agile Metrics
 
Governance of agile Software projects by an automated KPI Cockpit in the Cloud
Governance of agile Software projectsby an automated KPI Cockpit in the CloudGovernance of agile Software projectsby an automated KPI Cockpit in the Cloud
Governance of agile Software projects by an automated KPI Cockpit in the Cloud
 
Big Apple Scrum Day 2015 - Advanced Scrum Metrics Reference Sheet
Big Apple Scrum Day 2015 - Advanced Scrum Metrics Reference SheetBig Apple Scrum Day 2015 - Advanced Scrum Metrics Reference Sheet
Big Apple Scrum Day 2015 - Advanced Scrum Metrics Reference Sheet
 
Agile Metrics for Senior Managers and Executives
Agile Metrics for Senior Managers and ExecutivesAgile Metrics for Senior Managers and Executives
Agile Metrics for Senior Managers and Executives
 
How smooth is your agile ride
How smooth is your agile rideHow smooth is your agile ride
How smooth is your agile ride
 
Agile Metrics
Agile MetricsAgile Metrics
Agile Metrics
 
Agile Metrics : A seminal approach for calculating Metrics in Agile Projects
Agile Metrics : A seminal approach for calculating Metrics in Agile ProjectsAgile Metrics : A seminal approach for calculating Metrics in Agile Projects
Agile Metrics : A seminal approach for calculating Metrics in Agile Projects
 
Agile by numbers
Agile by numbersAgile by numbers
Agile by numbers
 
Agile KPIs vs. Traditional KPIs – A mind shift
Agile KPIs vs. Traditional KPIs – A mind shiftAgile KPIs vs. Traditional KPIs – A mind shift
Agile KPIs vs. Traditional KPIs – A mind shift
 
Agile Metrics - how to use metrics to manage agile teams
Agile Metrics - how to use metrics to manage agile teamsAgile Metrics - how to use metrics to manage agile teams
Agile Metrics - how to use metrics to manage agile teams
 
Agile Metrics
Agile MetricsAgile Metrics
Agile Metrics
 
Simple Lean Agile KPIs
Simple Lean Agile KPIsSimple Lean Agile KPIs
Simple Lean Agile KPIs
 
Agile Metrics
Agile MetricsAgile Metrics
Agile Metrics
 
Agile dashboard
Agile dashboardAgile dashboard
Agile dashboard
 
Agile metrics - Measure and Improve
Agile metrics - Measure and ImproveAgile metrics - Measure and Improve
Agile metrics - Measure and Improve
 

Viewers also liked

Making Your Agile Transition and Org Progress Visible
Making Your Agile Transition and Org Progress VisibleMaking Your Agile Transition and Org Progress Visible
Making Your Agile Transition and Org Progress Visible
Jason Little
 
Focus group
Focus groupFocus group
Focus group
10riccinie
 
Agile Transition Framework - presented at Frankfurt PMI Chapter
Agile Transition Framework - presented at Frankfurt PMI ChapterAgile Transition Framework - presented at Frankfurt PMI Chapter
Agile Transition Framework - presented at Frankfurt PMI Chapter
Arno Delhij 웃
 
Unlocking Excellence with Agile Metrics
Unlocking Excellence with Agile MetricsUnlocking Excellence with Agile Metrics
Unlocking Excellence with Agile Metrics
Rally Software
 
Agile Metrics V6
Agile Metrics V6Agile Metrics V6
Agile Metrics V6
Ram Srivastava
 
Focus Group Discussions – a step-by-step guide
Focus Group Discussions – a step-by-step guideFocus Group Discussions – a step-by-step guide
Focus Group Discussions – a step-by-step guide
Annette Gerritsen
 
Focus Group interview - qualitative research
Focus Group interview - qualitative research Focus Group interview - qualitative research
Focus Group interview - qualitative research
Alvis Loo
 
Focus Group Discussion (Fgd)
Focus Group Discussion (Fgd)Focus Group Discussion (Fgd)
Focus Group Discussion (Fgd)
Zamboanga
 
Focus group discussion
Focus group discussionFocus group discussion
Focus group discussion
Abino David
 

Viewers also liked (9)

Making Your Agile Transition and Org Progress Visible
Making Your Agile Transition and Org Progress VisibleMaking Your Agile Transition and Org Progress Visible
Making Your Agile Transition and Org Progress Visible
 
Focus group
Focus groupFocus group
Focus group
 
Agile Transition Framework - presented at Frankfurt PMI Chapter
Agile Transition Framework - presented at Frankfurt PMI ChapterAgile Transition Framework - presented at Frankfurt PMI Chapter
Agile Transition Framework - presented at Frankfurt PMI Chapter
 
Unlocking Excellence with Agile Metrics
Unlocking Excellence with Agile MetricsUnlocking Excellence with Agile Metrics
Unlocking Excellence with Agile Metrics
 
Agile Metrics V6
Agile Metrics V6Agile Metrics V6
Agile Metrics V6
 
Focus Group Discussions – a step-by-step guide
Focus Group Discussions – a step-by-step guideFocus Group Discussions – a step-by-step guide
Focus Group Discussions – a step-by-step guide
 
Focus Group interview - qualitative research
Focus Group interview - qualitative research Focus Group interview - qualitative research
Focus Group interview - qualitative research
 
Focus Group Discussion (Fgd)
Focus Group Discussion (Fgd)Focus Group Discussion (Fgd)
Focus Group Discussion (Fgd)
 
Focus group discussion
Focus group discussionFocus group discussion
Focus group discussion
 

Similar to Sagi Smolarski ITG - Enterprise Metrics on Agile

Steve Lawrence - Agile Metrics
Steve Lawrence - Agile MetricsSteve Lawrence - Agile Metrics
Steve Lawrence - Agile Metrics
AgileNZ Conference
 
The C-Suite Data Advantage: How Workday Executives Reduce Costs and Make Bett...
The C-Suite Data Advantage: How Workday Executives Reduce Costs and Make Bett...The C-Suite Data Advantage: How Workday Executives Reduce Costs and Make Bett...
The C-Suite Data Advantage: How Workday Executives Reduce Costs and Make Bett...
Workday, Inc.
 
Agile metrics at-pmi bangalore
Agile metrics at-pmi bangaloreAgile metrics at-pmi bangalore
Agile metrics at-pmi bangalore
Aguai Solutions Pvt Ltd
 
Problem Solving Frameworks
Problem Solving FrameworksProblem Solving Frameworks
Problem Solving Frameworks
Alan McCloy
 
Deloitte lean agile state of the nation
Deloitte lean   agile state of the nationDeloitte lean   agile state of the nation
Deloitte lean agile state of the nation
Alexis Hui
 
Driving Organizational Change Dreamforce 2014 (Salesforce)
Driving Organizational Change Dreamforce 2014 (Salesforce)Driving Organizational Change Dreamforce 2014 (Salesforce)
Driving Organizational Change Dreamforce 2014 (Salesforce)
Steve Heye
 
Kanban India 2023 | Renjith Achuthanunni and Anoop Kadur Vijayakumar | DevOps...
Kanban India 2023 | Renjith Achuthanunni and Anoop Kadur Vijayakumar | DevOps...Kanban India 2023 | Renjith Achuthanunni and Anoop Kadur Vijayakumar | DevOps...
Kanban India 2023 | Renjith Achuthanunni and Anoop Kadur Vijayakumar | DevOps...
LeanKanbanIndia
 
Six sigma
Six sigmaSix sigma
Six sigma
pcchauhan
 
Puneet Green Belt(13th feb).pptx
Puneet Green Belt(13th feb).pptxPuneet Green Belt(13th feb).pptx
Puneet Green Belt(13th feb).pptx
Puneet Gupta
 
How agile is your team
How agile is your teamHow agile is your team
How agile is your team
Phani Bhushan
 
Logic of Six Sigma DMAIC Y=f(x) to explain the world .ppt
Logic of Six Sigma DMAIC Y=f(x) to explain the world .pptLogic of Six Sigma DMAIC Y=f(x) to explain the world .ppt
Logic of Six Sigma DMAIC Y=f(x) to explain the world .ppt
Stanley Liu Qiang
 
Lokesh Madekar_Resume
Lokesh Madekar_ResumeLokesh Madekar_Resume
Lokesh Madekar_Resume
Lokesh Madekar
 
Agile 2010 conference - a holistic approach to scaling agile at salesforce
Agile 2010 conference - a holistic approach to scaling agile at salesforceAgile 2010 conference - a holistic approach to scaling agile at salesforce
Agile 2010 conference - a holistic approach to scaling agile at salesforce
Steve Greene
 
DOES15 - Ramona Jackson and Aji Rajappan - Continuous Delivery at Cisco IT
DOES15 - Ramona Jackson and Aji Rajappan - Continuous Delivery at Cisco ITDOES15 - Ramona Jackson and Aji Rajappan - Continuous Delivery at Cisco IT
DOES15 - Ramona Jackson and Aji Rajappan - Continuous Delivery at Cisco IT
Gene Kim
 
Lokesh Madekar_Resume
Lokesh Madekar_ResumeLokesh Madekar_Resume
Lokesh Madekar_Resume
Lokesh Madekar
 
Pride Procurement Lean
Pride Procurement LeanPride Procurement Lean
Pride Procurement Lean
Elm Valle
 
Daniel Breston - DevOps metrics that matter
Daniel Breston - DevOps metrics that matterDaniel Breston - DevOps metrics that matter
Daniel Breston - DevOps metrics that matter
itSMF UK
 
KanbanTO - June 2019 - How Agile Are We Executive Dashboard
KanbanTO - June 2019 - How Agile Are We Executive DashboardKanbanTO - June 2019 - How Agile Are We Executive Dashboard
KanbanTO - June 2019 - How Agile Are We Executive Dashboard
Shuman Ip
 
Agile metrics at-pmi bangalore
Agile metrics at-pmi bangaloreAgile metrics at-pmi bangalore
Agile metrics at-pmi bangalore
Bimlesh Gundurao
 
Improving Estimates
Improving EstimatesImproving Estimates
Improving Estimates
Giovanni Scerra ☃
 

Similar to Sagi Smolarski ITG - Enterprise Metrics on Agile (20)

Steve Lawrence - Agile Metrics
Steve Lawrence - Agile MetricsSteve Lawrence - Agile Metrics
Steve Lawrence - Agile Metrics
 
The C-Suite Data Advantage: How Workday Executives Reduce Costs and Make Bett...
The C-Suite Data Advantage: How Workday Executives Reduce Costs and Make Bett...The C-Suite Data Advantage: How Workday Executives Reduce Costs and Make Bett...
The C-Suite Data Advantage: How Workday Executives Reduce Costs and Make Bett...
 
Agile metrics at-pmi bangalore
Agile metrics at-pmi bangaloreAgile metrics at-pmi bangalore
Agile metrics at-pmi bangalore
 
Problem Solving Frameworks
Problem Solving FrameworksProblem Solving Frameworks
Problem Solving Frameworks
 
Deloitte lean agile state of the nation
Deloitte lean   agile state of the nationDeloitte lean   agile state of the nation
Deloitte lean agile state of the nation
 
Driving Organizational Change Dreamforce 2014 (Salesforce)
Driving Organizational Change Dreamforce 2014 (Salesforce)Driving Organizational Change Dreamforce 2014 (Salesforce)
Driving Organizational Change Dreamforce 2014 (Salesforce)
 
Kanban India 2023 | Renjith Achuthanunni and Anoop Kadur Vijayakumar | DevOps...
Kanban India 2023 | Renjith Achuthanunni and Anoop Kadur Vijayakumar | DevOps...Kanban India 2023 | Renjith Achuthanunni and Anoop Kadur Vijayakumar | DevOps...
Kanban India 2023 | Renjith Achuthanunni and Anoop Kadur Vijayakumar | DevOps...
 
Six sigma
Six sigmaSix sigma
Six sigma
 
Puneet Green Belt(13th feb).pptx
Puneet Green Belt(13th feb).pptxPuneet Green Belt(13th feb).pptx
Puneet Green Belt(13th feb).pptx
 
How agile is your team
How agile is your teamHow agile is your team
How agile is your team
 
Logic of Six Sigma DMAIC Y=f(x) to explain the world .ppt
Logic of Six Sigma DMAIC Y=f(x) to explain the world .pptLogic of Six Sigma DMAIC Y=f(x) to explain the world .ppt
Logic of Six Sigma DMAIC Y=f(x) to explain the world .ppt
 
Lokesh Madekar_Resume
Lokesh Madekar_ResumeLokesh Madekar_Resume
Lokesh Madekar_Resume
 
Agile 2010 conference - a holistic approach to scaling agile at salesforce
Agile 2010 conference - a holistic approach to scaling agile at salesforceAgile 2010 conference - a holistic approach to scaling agile at salesforce
Agile 2010 conference - a holistic approach to scaling agile at salesforce
 
DOES15 - Ramona Jackson and Aji Rajappan - Continuous Delivery at Cisco IT
DOES15 - Ramona Jackson and Aji Rajappan - Continuous Delivery at Cisco ITDOES15 - Ramona Jackson and Aji Rajappan - Continuous Delivery at Cisco IT
DOES15 - Ramona Jackson and Aji Rajappan - Continuous Delivery at Cisco IT
 
Lokesh Madekar_Resume
Lokesh Madekar_ResumeLokesh Madekar_Resume
Lokesh Madekar_Resume
 
Pride Procurement Lean
Pride Procurement LeanPride Procurement Lean
Pride Procurement Lean
 
Daniel Breston - DevOps metrics that matter
Daniel Breston - DevOps metrics that matterDaniel Breston - DevOps metrics that matter
Daniel Breston - DevOps metrics that matter
 
KanbanTO - June 2019 - How Agile Are We Executive Dashboard
KanbanTO - June 2019 - How Agile Are We Executive DashboardKanbanTO - June 2019 - How Agile Are We Executive Dashboard
KanbanTO - June 2019 - How Agile Are We Executive Dashboard
 
Agile metrics at-pmi bangalore
Agile metrics at-pmi bangaloreAgile metrics at-pmi bangalore
Agile metrics at-pmi bangalore
 
Improving Estimates
Improving EstimatesImproving Estimates
Improving Estimates
 

More from AgileSparks

What Do Agile Leaders Do by Kurt Bittner
What Do Agile Leaders Do by Kurt Bittner What Do Agile Leaders Do by Kurt Bittner
What Do Agile Leaders Do by Kurt Bittner
AgileSparks
 
Distributed Teams by Kevin Goldsmith
Distributed Teams by Kevin GoldsmithDistributed Teams by Kevin Goldsmith
Distributed Teams by Kevin Goldsmith
AgileSparks
 
A Back-End Approach to Customer Driven by Adi Gostynski
A Back-End Approach to Customer Driven by Adi GostynskiA Back-End Approach to Customer Driven by Adi Gostynski
A Back-End Approach to Customer Driven by Adi Gostynski
AgileSparks
 
Jira Portfolio by Elad Ben-Noam
Jira Portfolio by Elad Ben-NoamJira Portfolio by Elad Ben-Noam
Jira Portfolio by Elad Ben-Noam
AgileSparks
 
Agile Hiring at Scale by Yon Bergman
Agile Hiring at Scale by Yon Bergman Agile Hiring at Scale by Yon Bergman
Agile Hiring at Scale by Yon Bergman
AgileSparks
 
Are We Really Using Our Resources in The Most Effective Way? by Perry Yaqubo...
Are We Really Using Our Resources in The Most Effective Way?  by Perry Yaqubo...Are We Really Using Our Resources in The Most Effective Way?  by Perry Yaqubo...
Are We Really Using Our Resources in The Most Effective Way? by Perry Yaqubo...
AgileSparks
 
Honest Experimentation by Jonathan Bertfield
 Honest Experimentation by Jonathan Bertfield Honest Experimentation by Jonathan Bertfield
Honest Experimentation by Jonathan Bertfield
AgileSparks
 
Pango Journey to an Agile Cloud by Yaniv Kalo
Pango Journey to an Agile Cloud by Yaniv KaloPango Journey to an Agile Cloud by Yaniv Kalo
Pango Journey to an Agile Cloud by Yaniv Kalo
AgileSparks
 
ClickSoftware Agile Tranistion by Meny Duek
ClickSoftware Agile Tranistion by Meny DuekClickSoftware Agile Tranistion by Meny Duek
ClickSoftware Agile Tranistion by Meny Duek
AgileSparks
 
Augury's Journey Towards CD by Assaf Mizrachi
Augury's Journey Towards CD by Assaf Mizrachi Augury's Journey Towards CD by Assaf Mizrachi
Augury's Journey Towards CD by Assaf Mizrachi
AgileSparks
 
Kubernetes is Hard! Lessons Learned Taking Our Apps to Kubernetes by Eldad Assis
Kubernetes is Hard! Lessons Learned Taking Our Apps to Kubernetes by Eldad AssisKubernetes is Hard! Lessons Learned Taking Our Apps to Kubernetes by Eldad Assis
Kubernetes is Hard! Lessons Learned Taking Our Apps to Kubernetes by Eldad Assis
AgileSparks
 
Creating a Culture of Ownership and Trust with Visibility and Transparency by...
Creating a Culture of Ownership and Trust with Visibility and Transparency by...Creating a Culture of Ownership and Trust with Visibility and Transparency by...
Creating a Culture of Ownership and Trust with Visibility and Transparency by...
AgileSparks
 
Real Innovation is with Real Customers by Baat Enosh
Real Innovation is with Real Customers by Baat EnoshReal Innovation is with Real Customers by Baat Enosh
Real Innovation is with Real Customers by Baat Enosh
AgileSparks
 
True Continuous Improvement with Toyota Kata by Jesper Boeg
True Continuous Improvement with Toyota Kata by Jesper BoegTrue Continuous Improvement with Toyota Kata by Jesper Boeg
True Continuous Improvement with Toyota Kata by Jesper Boeg
AgileSparks
 
Homo-Adaptus Agile Worker by Lior Frenkel
Homo-Adaptus Agile Worker by Lior FrenkelHomo-Adaptus Agile Worker by Lior Frenkel
Homo-Adaptus Agile Worker by Lior Frenkel
AgileSparks
 
Intel CHD Case Study by Ronen Ezra
Intel CHD Case Study by Ronen EzraIntel CHD Case Study by Ronen Ezra
Intel CHD Case Study by Ronen Ezra
AgileSparks
 
Leading Innovation by Jonathan Bertfield
Leading Innovation by Jonathan BertfieldLeading Innovation by Jonathan Bertfield
Leading Innovation by Jonathan Bertfield
AgileSparks
 
Organization architecture autonomy and accountability
Organization architecture autonomy and accountability Organization architecture autonomy and accountability
Organization architecture autonomy and accountability
AgileSparks
 
Tribal Unity, Agile Israel 2017
Tribal Unity, Agile Israel 2017Tribal Unity, Agile Israel 2017
Tribal Unity, Agile Israel 2017
AgileSparks
 
The mindful manager, Agile Israel 2017
The mindful manager, Agile Israel 2017The mindful manager, Agile Israel 2017
The mindful manager, Agile Israel 2017
AgileSparks
 

More from AgileSparks (20)

What Do Agile Leaders Do by Kurt Bittner
What Do Agile Leaders Do by Kurt Bittner What Do Agile Leaders Do by Kurt Bittner
What Do Agile Leaders Do by Kurt Bittner
 
Distributed Teams by Kevin Goldsmith
Distributed Teams by Kevin GoldsmithDistributed Teams by Kevin Goldsmith
Distributed Teams by Kevin Goldsmith
 
A Back-End Approach to Customer Driven by Adi Gostynski
A Back-End Approach to Customer Driven by Adi GostynskiA Back-End Approach to Customer Driven by Adi Gostynski
A Back-End Approach to Customer Driven by Adi Gostynski
 
Jira Portfolio by Elad Ben-Noam
Jira Portfolio by Elad Ben-NoamJira Portfolio by Elad Ben-Noam
Jira Portfolio by Elad Ben-Noam
 
Agile Hiring at Scale by Yon Bergman
Agile Hiring at Scale by Yon Bergman Agile Hiring at Scale by Yon Bergman
Agile Hiring at Scale by Yon Bergman
 
Are We Really Using Our Resources in The Most Effective Way? by Perry Yaqubo...
Are We Really Using Our Resources in The Most Effective Way?  by Perry Yaqubo...Are We Really Using Our Resources in The Most Effective Way?  by Perry Yaqubo...
Are We Really Using Our Resources in The Most Effective Way? by Perry Yaqubo...
 
Honest Experimentation by Jonathan Bertfield
 Honest Experimentation by Jonathan Bertfield Honest Experimentation by Jonathan Bertfield
Honest Experimentation by Jonathan Bertfield
 
Pango Journey to an Agile Cloud by Yaniv Kalo
Pango Journey to an Agile Cloud by Yaniv KaloPango Journey to an Agile Cloud by Yaniv Kalo
Pango Journey to an Agile Cloud by Yaniv Kalo
 
ClickSoftware Agile Tranistion by Meny Duek
ClickSoftware Agile Tranistion by Meny DuekClickSoftware Agile Tranistion by Meny Duek
ClickSoftware Agile Tranistion by Meny Duek
 
Augury's Journey Towards CD by Assaf Mizrachi
Augury's Journey Towards CD by Assaf Mizrachi Augury's Journey Towards CD by Assaf Mizrachi
Augury's Journey Towards CD by Assaf Mizrachi
 
Kubernetes is Hard! Lessons Learned Taking Our Apps to Kubernetes by Eldad Assis
Kubernetes is Hard! Lessons Learned Taking Our Apps to Kubernetes by Eldad AssisKubernetes is Hard! Lessons Learned Taking Our Apps to Kubernetes by Eldad Assis
Kubernetes is Hard! Lessons Learned Taking Our Apps to Kubernetes by Eldad Assis
 
Creating a Culture of Ownership and Trust with Visibility and Transparency by...
Creating a Culture of Ownership and Trust with Visibility and Transparency by...Creating a Culture of Ownership and Trust with Visibility and Transparency by...
Creating a Culture of Ownership and Trust with Visibility and Transparency by...
 
Real Innovation is with Real Customers by Baat Enosh
Real Innovation is with Real Customers by Baat EnoshReal Innovation is with Real Customers by Baat Enosh
Real Innovation is with Real Customers by Baat Enosh
 
True Continuous Improvement with Toyota Kata by Jesper Boeg
True Continuous Improvement with Toyota Kata by Jesper BoegTrue Continuous Improvement with Toyota Kata by Jesper Boeg
True Continuous Improvement with Toyota Kata by Jesper Boeg
 
Homo-Adaptus Agile Worker by Lior Frenkel
Homo-Adaptus Agile Worker by Lior FrenkelHomo-Adaptus Agile Worker by Lior Frenkel
Homo-Adaptus Agile Worker by Lior Frenkel
 
Intel CHD Case Study by Ronen Ezra
Intel CHD Case Study by Ronen EzraIntel CHD Case Study by Ronen Ezra
Intel CHD Case Study by Ronen Ezra
 
Leading Innovation by Jonathan Bertfield
Leading Innovation by Jonathan BertfieldLeading Innovation by Jonathan Bertfield
Leading Innovation by Jonathan Bertfield
 
Organization architecture autonomy and accountability
Organization architecture autonomy and accountability Organization architecture autonomy and accountability
Organization architecture autonomy and accountability
 
Tribal Unity, Agile Israel 2017
Tribal Unity, Agile Israel 2017Tribal Unity, Agile Israel 2017
Tribal Unity, Agile Israel 2017
 
The mindful manager, Agile Israel 2017
The mindful manager, Agile Israel 2017The mindful manager, Agile Israel 2017
The mindful manager, Agile Israel 2017
 

Sagi Smolarski ITG - Enterprise Metrics on Agile

  • 1. Enterprise Metrics on AgileSagi SmolarskiDirector, Software DevelopmentSagi.Smolarski@itg.com[CC] Daniel Goodwin via Wikimedia Commons
  • 2. DisclaimersThe information contained in this presentation is provided for informational purposes only.  All trademarks, service marks, and trade names not owned by ITG are owned by their respective owners.…in other words, the data presented has been modified and does not represent real data for ITG teams.
  • 3. AgendaWhy do we need metrics for agile?How do we generate those metrics?Which metrics do we look at?Pros and cons of looking at those metrics.
  • 4. Investment Technology GroupNYSE: ITGwww.itg.comLeading provider of trading solutionsTrading Front-EndsTrading AlgorithmsResearch (fundamental analytic, pre/post-trade)Trading desksElectronic trading connectivityLiquidity poolDevelopment operation: 300 Developers, 100 QA Analysts, ~60 teams
  • 5. Iteration MetricsQuality MetricsProcess BaselineITG’s Agile Transition TimelineITG TritonXPEnterprise rollout planRally+Scrum, +Lean90% of teams have been trained and are using RallyBest Execution Management SystemXP Pilot200020012002200320042005200620072008200920102011Massive transitionInformal, spotty, inconsistent adoption
  • 6. Our Process Baseline – How We Expect Teams to WorkExcerpt
  • 7. The journey to Agile is a long, winding road…Are we moving forward?[CC] – Sten via Wikimedia Commons
  • 8. Why Measure?“If you cannot measure it you cannot improve it.”Lord Kelvin
  • 9. Why Metrics?Teams (Inspect & Adapt):Are we doing okay?
  • 10. How can we improve?
  • 11. Are we doing what the company expect from us?Coaches, PMO (Train and Coach):Are teams using what we taught?
  • 12. Which teams need our help?Executives (Govern & Steer):What are we getting out of the agile transition?Are teams sticking to our process baseline and to enterprise initiatives?Is productivity/quality improving?
  • 13. Team ATeams Process HealthData is readily available in RallyTeam BVelocityVelocityvs.Burndown/Story AcceptanceBurndown/Story Acceptancevs.Isn’t this enough?
  • 14. Why Metrics? - Dumbing down…This is too complex for team members to interpret and monitor…Are the charts okay?
  • 15. Why Metrics – Scaling upHow do we watch 80 teams? Lines of products? The whole enterprise??
  • 16. Types of MetricsQualitativeSatisfaction of stakeholdersProduct Management, Client Services, Product Support (deployment, service), Delivery teamTeams adherence to practicesAgility questionnaireQuantitativeQuality metricsProcess health metricsTime-to-MarketWe’ll be focusing on these
  • 17. What Would We Want to Measure?ProductivityQualityROISatisfactionSome of these are not easy to measure so we have to find proxies
  • 18. What We Are Actually MeasuringSatisfactionQualityProcessAs a partial proxy for productivityStakeholders SurveysProduction defectsDefects DebtWork-in -ProgressVelocity StabilityFull Completion or workGradual acceptance of workChurnSmall StoriesPractices Radar map
  • 19. How We Generate MetricsHow do we obtain data?Rally’s Web Services APIRest and SOAPAccess to virtually all data in the toolUsers, Releases, Iterations, Stories, Tasks, Defects, Test cases…Ruby toolkit… or any WS libraryWe use mostly C#Automated generation, monthlyHow do we make sense out of mounds of data?
  • 22. Quality MetricsDefects Found in ProductionMetric #1Production DefectsGoalJanFebMarAprMayJunJulAugSepOctNovDecJanFeb20102011Team Level & Enterprise Level
  • 23. Metric #2Quality MetricsFix-Bugs-FirstIf you are fixing defects firstAs part of existing development, within the iterationBefore new development for regression defectsThen…The number of defects in your backlog should be smallThe age of open defects should be lowTogether…Defects Debt should be lowDefects Debt = [# open defects] x [avg age of open defects]
  • 25. Quality MetricsEnterprise Defects DebtMetric #2Defects Debt
  • 28. Scope CreepChurnMetric #3Iteration Cumulative FlowNo Scope CreepPlan EstimatesDay 1Day 2Day 3Day 4Day 5Day 6Day 7Day 8Day 9Day 10Day 11Day 12Day 13Day 14Churn (Scope variation) = 0
  • 29. Work in Progress & Scope CreepMetric #3Iteration Cumulative FlowPlan EstimatesDay 1Day 2Day 3Day 4Day 5Day 6Day 7Day 8Day 9Day 10Day 11Day 12Day 13Day 14Churn = 30%
  • 30. Work in Progress & Scope CreepMetric #3Iteration Cumulative FlowPlan EstimatesDay 1Day 2Day 3Day 4Day 5Day 6Day 7Day 8Day 9Day 10Day 11Scope Creep: Team Cannot Commit Disruptive (task switching?) Less efficientDay 12Day 13Day 14Churn = 30%
  • 31. Work in ProgressWIP AverageMetric #4Iteration Cumulative FlowNo Scope CreepWork In ProgressPlan EstimatesDay 1Day 2Day 3Day 4Day 5Day 6Day 7Day 8Day 9Day 10Day 11Day 12Day 13Day 14PlannedWIP average = 10%In ProgressAccepted
  • 32. Work in Progress & Scope CreepMetric #4Iteration Cumulative FlowPlan EstimatesDay 1Day 2Day 3Day 4Day 5Day 6Day 7Day 8Day 9Day 10Day 11Day 12Day 13Day 14PlannedWIP average = 70%In ProgressAccepted
  • 33. Work in Progress & Scope CreepMetric #4Iteration Cumulative Flow Large WIP = Long Cycle TimeRisk of not completing work within iterationPlan EstimatesDay 1Day 2Day 3Day 4Day 5Day 6Day 7Day 8Day 9Day 10Day 11Day 12Day 13Day 14PlannedWIP average = 70%In ProgressAccepted
  • 34. Full Acceptance – Final Acceptance %Metric #5Iteration Burn-up100% AcceptedPlan Estimates (Points)50% AcceptanceDay 1Day 2Day 3Day 4Day 5Day 6Day 7Day 8Day 9Day 10Day 11Day 12Day 13Day 14Day 12Day 13Day 14AcceptedAcceptance rate on the last day = 100%
  • 35. Full AcceptanceMetric #5Iteration Burn-up100% AcceptedPlan Estimates (Points)50% AcceptanceDay 1Day 2Day 3Day 4Day 5Day 6Day 7Day 8Day 9Day 10Day 11Day 12Day 13Day 14Day 13Day 14Day 12Day 13Day 14AcceptedAcceptance rate on the last day = 55%
  • 36. Full AcceptanceMetric #5Iteration Burn-up100% AcceptedPlan Estimates (Points)50% AcceptanceDay 1Day 2Day 3Day 4Day 5Day 6Day 7Day 8Day 9Day 10Day 11Day 12Day 13Day 14Day 13Day 14 Partial CompletionHard to planGreater cycle timeInefficientDay 12Day 13Day 14AcceptedAcceptance rate on the last day = 55%
  • 37. Gradual Acceptance – 50% dayMetric #6Iteration Burn-up100% AcceptedPlan Estimates (Points)50% AcceptanceDay 1Day 2Day 3Day 4Day 5Day 6Day 7Day 8Day 9Day 10Day 11Day 12Day 13Day 14>50% points acceptedDay 12Day 13Day 14AcceptedDay of 50% Acceptance = 8days/14days = 57%
  • 38. Gradual AcceptanceFull AcceptanceMetric #6Iteration Burn-up100% AcceptedPlan Estimates (Points)50% AcceptanceDay 1Day 2Day 3Day 4Day 5Day 6Day 7Day 8Day 9Day 10Day 11Day 12Day 13Day 14Day 12Day 13Day 14AcceptedDay of 50% Acceptance = 14/14 = 100%
  • 39. Gradual AcceptanceMetric #6Iteration Burn-up100% AcceptedPlan Estimates (Points)50% AcceptanceDay 1Day 2Day 3Day 4Day 5Day 6Day 7Day 8Day 9Day 10Day 11Day 12Day 13Day 14Day 12Day 13Day 14Accepted Late AcceptanceHigh risk of not completing workLate feedback on workDay of 50% Acceptance = 14/14 = 100%
  • 40. Velocity StabilityVelocity VariationMetric #7Velocity605040302010Iteration 23Iteration 24Iteration 25Iteration 26Iteration 27Iteration 28Iteration 29Iteration 30Iteration 31Iteration 32Day 12Day 13Day 14Accepted within iterationVelocity is fairly stableAccepted after iterationVariation = 7%Never accepted
  • 41. Velocity StabilityVelocity VariationMetric #7Velocity605040302010Iteration 23Iteration 24Iteration 25Iteration 26Iteration 27Iteration 28Iteration 29Iteration 30Iteration 31Iteration 32Day 12Day 13Day 14Accepted within iterationVelocity is unstableAccepted after iterationVariation = 87%Never accepted
  • 42. Velocity StabilityMetric #7Velocity605040302010Iteration 23Iteration 24Iteration 25Iteration 26Iteration 27Iteration 28Iteration 29Iteration 30Iteration 31Iteration 32Day 12Day 13Day 14 Unstable VelocityLow predictabilityHard to planInefficient?Accepted within iterationVelocity is unstableAccepted after iterationVariation = 87%Never accepted
  • 43. Defects DebtVelocity StabilityChurn% CompletionWIP, Day of 50% Acceptance, Story SizingOur Process Baseline… and metrics we can collect
  • 44. Team DashboardAll together now…Day 12Day 13Day 14Day 12Day 13Day 14Day 12Day 13Day 14Day 12Day 13Day 14Day 12Day 13Day 14Day 12Day 13Day 14
  • 45. Teams Process HealthEnterprise Process Health Map(Probably) Healthy ProcessLine of Business #1Line of Business #2Line of Business #3Line of Business #4Line of Business #5ProjYProj kProj XProj BProj NProjאProject AProhj ZProj EProjבProj 1Proj CProj WProj גProj 2Proj דProj 3ProjהProj DProj 4Proj 5Proj MProj LProjו(Possibly) Unhealthy Process
  • 46. Problems with MetricsMetric-Driven Dysfunctions Teams misreporting defectsNot tracking unpredictable work in RallyWe can limit these by not using these metrics to reward, reprimandFalse positives, False negativesSome metrics are sensitive to how Rally is being useThese metrics don’t cover some important aspects of teams’ processMetrics should be treated as an indication that further examination is neededHey, I just figured out how we can double our quarterly sales. From now on, each quarter will last six months.
  • 47. Looking at Next…Small StoriesTime to MarketTrendsBy Simon A. Eugster (EigenesWerk) [GFDL (www.gnu.org/copyleft/fdl.html) or CC-BY-SA-3.0 (www.creativecommons.org/licenses/by-sa/3.0)], via Wikimedia Commons
  • 48. Q&AQ&AHow are you measuring yourselves?

Editor's Notes

  1. Photo by Daniel Goodwin - http://www.flickr.com/photos/dgoodphoto/
  2. Transition of teams include 3 days training, followed by 2 days at the end of the first iteration.
  3. Sten [CC-BY-SA-3.0-2.5-2.0-1.0 (www.creativecommons.org/licenses/by-sa/3.0) or GFDL (www.gnu.org/copyleft/fdl.html)], via Wikimedia Commons
  4. William Thomson, 1st Baron KelvinDiscovered the first and second law of thermodynamics
  5. How many of you relate to these needs?How many of you are collecting process/quality metrics?
  6. Inspect and Adapt is key and we want to make the most of itTeams have a hard time interpreting charts, especially in situations where they are not very good or very badTeams often don’t take the time to look at the chartsAn easy indication of good/bad can at least tell them that there might be a problem
  7. I’m going to focus on the quantitative side
  8. The First "Computer Bug" Moth found trapped between points at Relay # 70, Panel F, of the Mark II Aiken Relay Calculator while it was being tested at Harvard University, 9 September 1947. The operators affixed the moth to the computer log, with the entry: "First actual case of bug being found". They put out the word that they had "debugged" the machine, thus introducing the term "debugging a computer program". In 1988, the log, with the moth still taped by the entry, was in the Naval Surface Warfare Center Computer Museum at Dahlgren, Virginia, which erroneously dated it 9 September 1945. The Smithsonian Institute's National Museum of American History and other sources have the correct date of 9 September 1947 (Object ID: 1994.0191.01). The Harvard Mark II computer was not complete until the summer of 1947.Photo in the public domain
  9. Is this good or bad?
  10. What is the problem here
  11. What is the problem here
  12. Is this good or bad?