Quality and Business Excellence
Celebration
ASQ Vancouver 25th Anniversary
Statistical Analysis of New Product Development (NPD)
Cycle-time Data Including Applications of Results
Steve Pratt, MEng., PE, CSSBB
Director of Engineering, Alpha Technologies
Alpha Technologies
Company Background
Alpha is a
Full Service Power
Systems Provider to
Multiple Markets
Pratt – Slide 1
Power
Modules
Indoor
Power Systems
Outdoor
Power Systems
Alpha’s Power Solutions Products
Pratt – Slide 2
Phase-Gate New Product
Development (NPD) Process
Deliverables:
• Product Concept
• Business Case
• Preliminary Plan
Deliverables:
•High-Level Design
•Requirements
•Complete Project
Plan
Deliverables:
• Design Outputs
• DFx Reviews
• Preliminary Test
Reports
Deliverables:
• Pilot Doc Pack
• Pilot Build
• Sales Forecasts
Deliverables:
• Compliance
Certifications
• Training
• MarCom docs
Deliverables:
• Sustaining Eng
• Repair/Support
• Cost Reduction
Phase 2
Planning
Phase 1
Concept
Phase 3
Development
Phase 4
Qualification
and Pre-
Production
Phase 5
LA and
Production
Ramp-up
Phase 6
GA and MOL
Pratt – Slide 3
Concurrent Engineering via Cross-
Functional Core Teams
Service
Supply
Chain
Product
Management
Program
Manager
Quality Engineering
Manufacturing
Pratt – Slide 4
Motivation for Study
Continuous Improvement:
• general perception that NPD takes too long
• lack of proper management of resources
• desire to establish performance benchmarks
Pratt – Slide 5
NPD Data Collected and Analyzed
Schedule Data
• Recorded Dates: Start, Gate 1, Gate 2, Gate 3, Gate 4 (LA) & Gate 5 (GA)
• Program Plans: estimated time to LA, estimated time to GA
• Calculated: total time, time per phase, actual vs. plan
Cost Data
• Timecard System: actual total effort (man-hours)
• Program Plans: estimated total effort
• Calculated: effort by function, effort by phase, remaining effort, actual vs. plan
Pratt – Slide 6
Tools Used in the Study
JMP – Statistical Discovery Software
Analyze-Distribution Platform:
• Calculations: Quantiles, Moments
• Plots: Histogram, Quantile Box, Normal Quantile
• Fit Distribution: Normal, Beta, Goodness of Fit Tests
Fit Y by X Platform: Bivariate Analysis
• Fit Line, Fit Polynomial, Summary of Fit
Fit Y by X Platform: One-way Analysis
• Calculations: Means and Standard Deviations
• Plots: Mean Diamonds
• Analysis of Variance (ANOVA)
Fit Model Platform
• Calculations: Standard Least Squares, Summary of Fit, Effect Tests
• Plots: Actual by Predicted, Effect Leverage, Residual by Predicted
• Two-way ANOVA with interactions
Microsoft Excel and PowerPoint
Pratt – Slide 7
External Benchmarking Data – PDMA
Best Practices Research
Provides industry-average
NPD Cycle-Time based on
complexity of programs:
• new-to-the world
• new-to-the firm
• next-gen improvements
• incremental improvements
Pratt – Slide 8
Source: Griffin A., “Product development cycle time for business-to-business products,” Industrial Marketing Management 2002;31: 291-304
Average NPD Cycle-Time
Benchmarking
Pratt – Slide 9
Planned vs. Actual NPD Cycle-Time
Pratt – Slide 10
ANOVA to Identify Significant Factors
for Schedule Data
Pratt – Slide 11
ANOVA to Identify Significant Factors
for Effort Data
Pratt – Slide 12
Categorizing NPD Programs into 9
Buckets
Product Types:
Modules (power modules, shelves, controllers)
Indoor Systems (rack-based systems, racks, distribution)
OSP (outside plant power/battery systems)
Program Complexities:
A– major program requiring advanced development
B – completely new, but no advanced development
C– new with incremental development
Pratt – Slide 13
NPD Summary Data –
Schedule (Gate 2 to LA)
Pratt – Slide 14
NPD Summary Data –
Schedule (Gate 2 to GA)
Pratt – Slide 15
NPD Summary Data –
Effort (Total Development Hours)
Pratt – Slide 16
NPD Schedule & Effort Continuous
Probability Distributions
Pratt – Slide 17
Additional Analyses
• plan vs. actual – schedule and effort
• schedule and effort variance by phase
• time spent per phase
• total effort by function and phase
• default team size and composition
Pratt – Slide 18
Applications
Better Estimating
Pratt – Slide 19
More Accurate Gate 2 Point Estimates
of Schedule and Effort
Previous estimates were:
- subject to negotiation
- overly optimistic and aggressive
- “rose-colored” recollections of past
Pratt – Slide 20
Alpha’s Historical Schedule
Estimation Accuracy
Median MRE = 44%
Pratt – Slide 21
NPD Schedule Point-Estimation via
Historical Mean Values
Median MRE = 19%
Pratt – Slide 22
Basic Schedule Equation Using
Historical Mean Effort Values
Median MRE = 21%
Pratt – Slide 23
Informal Comparisons Equation Using
NPD Mean Values
Median MRE = 20%
Pratt – Slide 24
Mean Magnitude of Relative Error –
Schedule Estimates
Pratt – Slide 25
Alpha’s Historical Effort
Estimation Accuracy
Median MRE = 36%
Pratt – Slide 26
NPD Effort Point-Estimation via
Historical Mean Values
Median MRE = 18%
Pratt – Slide 27
Mean Magnitude of Relative Error –
Effort Estimates
Pratt – Slide 28
Determine the “Cone of Uncertainty”
Pratt – Slide 29
Cone of Uncertainty for NPD Project
Estimates
Pratt – Slide 30
Cone of Uncertainty for NPD Project
Schedule Data
Pratt – Slide 31
Cone of Uncertainty Applied to the 9
NPD Project Buckets
Pratt – Slide 32
Schedule Cone of Uncertainty
Quantified by Program Phase
Pratt – Slide 33
Effort Cone of Uncertainty Quantified
by Program Phase
Pratt – Slide 34
Effort Estimate Ranges by Program
Phase
Pratt – Slide 35
Applications
Top-Down Project Planning
Pratt – Slide 36
Creation of Top-Down Program
Schedules
Pratt – Slide 37
NPD – Time Spent per Phase
Pratt – Slide 38
How Much Time Should Be Spent on
Planning?
Pratt – Slide 39
Applications
Performance Goal Setting
Pratt – Slide 40
SMART Performance Improvement
Goals
• fact-based and statistically valid
• objective and quantifiable
• deterministic at the start of a program
• consistent across all types of programs
• an enabler of continuous improvement
• unambiguous and easy to calculate
An ideal performance improvement measure is:
Pratt – Slide 41
RSD: a Normalized Measure of
Dispersion
Pratt – Slide 42
Program Scoring: RSD as Performance
Unit-of-Measure
Pratt – Slide 43
2013 NPD Cycle-Time Performance
Pratt – Slide 44
Earned-Value Milestones to Score
WIP NPD Programs
Pratt – Slide 45
Applications
Better Project Selection
Pratt – Slide 46
Deterministic Business Case Analysis
“Figures of Merit”
Pratt – Slide 47
Probabilistic Business Case Analysis
Beta Distribution
Pratt – Slide 48
Monte Carlo Simulation of
Cash Flows ($000)
Trial #1 Trial #2 Trial #3 Trial #4 Trial #5000
NPV $556 $461 $681 $447 $621
NRE 562 560597530664
GM$year1
348274 241220 274
GM$year2
604 653 407579 635
GM$year3
451651644447558
Pratt – Slide 49
Probability-Based Figures of Merit
Pratt – Slide 50
Applications
Resource Management
Pratt – Slide 51
Default Team Size and Composition
Effort divided
by Schedule
represents
number of
individuals
Pratt – Slide 52
Roadmap and Budget Planning
Pratt – Slide 53
Program Resource Management
Engineering Pokémon board
• visual resource management tool
• 9 program type/complexity buckets
• default team size/composition
Pratt – Slide 54
NPD Programs are Represented by a
Series of Magnetic Trays
Pratt – Slide 55
Program Trays Incorporate Default
Team Size/Composition
Pratt – Slide 56
NPD Team Members are Represented
by a Series of Cards
Pratt – Slide 57
Engineering Pokémon – Additional
Pieces
Program Status Indicators
Bonus Targets Add-on Resource Trays “Ghost” Cards
Important Date
Indicators
Clear Plastic Overlays
Pratt – Slide 58
The Board in Action
Pratt – Slide 59
Resulting Improvements
Pratt – Slide 60
Improved New Product Development
Better planning and management of resources…
• more stability and focus / less chaos
• increased efficiency
• enhanced organizational understanding
• faster NPD cycle-time
Pratt – Slide 61
In Conclusion:
It is never too late to start recording data!
- even simple data such as key dates and
time spent on activities can facilitate
powerful analyses
- capture data in real-time; don’t rely on
memory or post mortem activities
Pratt – Slide 62
Questions???
Pratt – Slide 63
Thank You!
STEVEN PRATT, MEng., PE
Director of Engineering
www.alpha.ca
Pratt – Slide 64

Statistical Analysis of New Product Development (NPD) Cycle-time Data

  • 1.
    Quality and BusinessExcellence Celebration ASQ Vancouver 25th Anniversary Statistical Analysis of New Product Development (NPD) Cycle-time Data Including Applications of Results Steve Pratt, MEng., PE, CSSBB Director of Engineering, Alpha Technologies
  • 2.
    Alpha Technologies Company Background Alphais a Full Service Power Systems Provider to Multiple Markets Pratt – Slide 1
  • 3.
  • 4.
    Phase-Gate New Product Development(NPD) Process Deliverables: • Product Concept • Business Case • Preliminary Plan Deliverables: •High-Level Design •Requirements •Complete Project Plan Deliverables: • Design Outputs • DFx Reviews • Preliminary Test Reports Deliverables: • Pilot Doc Pack • Pilot Build • Sales Forecasts Deliverables: • Compliance Certifications • Training • MarCom docs Deliverables: • Sustaining Eng • Repair/Support • Cost Reduction Phase 2 Planning Phase 1 Concept Phase 3 Development Phase 4 Qualification and Pre- Production Phase 5 LA and Production Ramp-up Phase 6 GA and MOL Pratt – Slide 3
  • 5.
    Concurrent Engineering viaCross- Functional Core Teams Service Supply Chain Product Management Program Manager Quality Engineering Manufacturing Pratt – Slide 4
  • 6.
    Motivation for Study ContinuousImprovement: • general perception that NPD takes too long • lack of proper management of resources • desire to establish performance benchmarks Pratt – Slide 5
  • 7.
    NPD Data Collectedand Analyzed Schedule Data • Recorded Dates: Start, Gate 1, Gate 2, Gate 3, Gate 4 (LA) & Gate 5 (GA) • Program Plans: estimated time to LA, estimated time to GA • Calculated: total time, time per phase, actual vs. plan Cost Data • Timecard System: actual total effort (man-hours) • Program Plans: estimated total effort • Calculated: effort by function, effort by phase, remaining effort, actual vs. plan Pratt – Slide 6
  • 8.
    Tools Used inthe Study JMP – Statistical Discovery Software Analyze-Distribution Platform: • Calculations: Quantiles, Moments • Plots: Histogram, Quantile Box, Normal Quantile • Fit Distribution: Normal, Beta, Goodness of Fit Tests Fit Y by X Platform: Bivariate Analysis • Fit Line, Fit Polynomial, Summary of Fit Fit Y by X Platform: One-way Analysis • Calculations: Means and Standard Deviations • Plots: Mean Diamonds • Analysis of Variance (ANOVA) Fit Model Platform • Calculations: Standard Least Squares, Summary of Fit, Effect Tests • Plots: Actual by Predicted, Effect Leverage, Residual by Predicted • Two-way ANOVA with interactions Microsoft Excel and PowerPoint Pratt – Slide 7
  • 9.
    External Benchmarking Data– PDMA Best Practices Research Provides industry-average NPD Cycle-Time based on complexity of programs: • new-to-the world • new-to-the firm • next-gen improvements • incremental improvements Pratt – Slide 8
  • 10.
    Source: Griffin A.,“Product development cycle time for business-to-business products,” Industrial Marketing Management 2002;31: 291-304 Average NPD Cycle-Time Benchmarking Pratt – Slide 9
  • 11.
    Planned vs. ActualNPD Cycle-Time Pratt – Slide 10
  • 12.
    ANOVA to IdentifySignificant Factors for Schedule Data Pratt – Slide 11
  • 13.
    ANOVA to IdentifySignificant Factors for Effort Data Pratt – Slide 12
  • 14.
    Categorizing NPD Programsinto 9 Buckets Product Types: Modules (power modules, shelves, controllers) Indoor Systems (rack-based systems, racks, distribution) OSP (outside plant power/battery systems) Program Complexities: A– major program requiring advanced development B – completely new, but no advanced development C– new with incremental development Pratt – Slide 13
  • 15.
    NPD Summary Data– Schedule (Gate 2 to LA) Pratt – Slide 14
  • 16.
    NPD Summary Data– Schedule (Gate 2 to GA) Pratt – Slide 15
  • 17.
    NPD Summary Data– Effort (Total Development Hours) Pratt – Slide 16
  • 18.
    NPD Schedule &Effort Continuous Probability Distributions Pratt – Slide 17
  • 19.
    Additional Analyses • planvs. actual – schedule and effort • schedule and effort variance by phase • time spent per phase • total effort by function and phase • default team size and composition Pratt – Slide 18
  • 20.
  • 21.
    More Accurate Gate2 Point Estimates of Schedule and Effort Previous estimates were: - subject to negotiation - overly optimistic and aggressive - “rose-colored” recollections of past Pratt – Slide 20
  • 22.
    Alpha’s Historical Schedule EstimationAccuracy Median MRE = 44% Pratt – Slide 21
  • 23.
    NPD Schedule Point-Estimationvia Historical Mean Values Median MRE = 19% Pratt – Slide 22
  • 24.
    Basic Schedule EquationUsing Historical Mean Effort Values Median MRE = 21% Pratt – Slide 23
  • 25.
    Informal Comparisons EquationUsing NPD Mean Values Median MRE = 20% Pratt – Slide 24
  • 26.
    Mean Magnitude ofRelative Error – Schedule Estimates Pratt – Slide 25
  • 27.
    Alpha’s Historical Effort EstimationAccuracy Median MRE = 36% Pratt – Slide 26
  • 28.
    NPD Effort Point-Estimationvia Historical Mean Values Median MRE = 18% Pratt – Slide 27
  • 29.
    Mean Magnitude ofRelative Error – Effort Estimates Pratt – Slide 28
  • 30.
    Determine the “Coneof Uncertainty” Pratt – Slide 29
  • 31.
    Cone of Uncertaintyfor NPD Project Estimates Pratt – Slide 30
  • 32.
    Cone of Uncertaintyfor NPD Project Schedule Data Pratt – Slide 31
  • 33.
    Cone of UncertaintyApplied to the 9 NPD Project Buckets Pratt – Slide 32
  • 34.
    Schedule Cone ofUncertainty Quantified by Program Phase Pratt – Slide 33
  • 35.
    Effort Cone ofUncertainty Quantified by Program Phase Pratt – Slide 34
  • 36.
    Effort Estimate Rangesby Program Phase Pratt – Slide 35
  • 37.
  • 38.
    Creation of Top-DownProgram Schedules Pratt – Slide 37
  • 39.
    NPD – TimeSpent per Phase Pratt – Slide 38
  • 40.
    How Much TimeShould Be Spent on Planning? Pratt – Slide 39
  • 41.
  • 42.
    SMART Performance Improvement Goals •fact-based and statistically valid • objective and quantifiable • deterministic at the start of a program • consistent across all types of programs • an enabler of continuous improvement • unambiguous and easy to calculate An ideal performance improvement measure is: Pratt – Slide 41
  • 43.
    RSD: a NormalizedMeasure of Dispersion Pratt – Slide 42
  • 44.
    Program Scoring: RSDas Performance Unit-of-Measure Pratt – Slide 43
  • 45.
    2013 NPD Cycle-TimePerformance Pratt – Slide 44
  • 46.
    Earned-Value Milestones toScore WIP NPD Programs Pratt – Slide 45
  • 47.
  • 48.
    Deterministic Business CaseAnalysis “Figures of Merit” Pratt – Slide 47
  • 49.
    Probabilistic Business CaseAnalysis Beta Distribution Pratt – Slide 48
  • 50.
    Monte Carlo Simulationof Cash Flows ($000) Trial #1 Trial #2 Trial #3 Trial #4 Trial #5000 NPV $556 $461 $681 $447 $621 NRE 562 560597530664 GM$year1 348274 241220 274 GM$year2 604 653 407579 635 GM$year3 451651644447558 Pratt – Slide 49
  • 51.
    Probability-Based Figures ofMerit Pratt – Slide 50
  • 52.
  • 53.
    Default Team Sizeand Composition Effort divided by Schedule represents number of individuals Pratt – Slide 52
  • 54.
    Roadmap and BudgetPlanning Pratt – Slide 53
  • 55.
    Program Resource Management EngineeringPokémon board • visual resource management tool • 9 program type/complexity buckets • default team size/composition Pratt – Slide 54
  • 56.
    NPD Programs areRepresented by a Series of Magnetic Trays Pratt – Slide 55
  • 57.
    Program Trays IncorporateDefault Team Size/Composition Pratt – Slide 56
  • 58.
    NPD Team Membersare Represented by a Series of Cards Pratt – Slide 57
  • 59.
    Engineering Pokémon –Additional Pieces Program Status Indicators Bonus Targets Add-on Resource Trays “Ghost” Cards Important Date Indicators Clear Plastic Overlays Pratt – Slide 58
  • 60.
    The Board inAction Pratt – Slide 59
  • 61.
  • 62.
    Improved New ProductDevelopment Better planning and management of resources… • more stability and focus / less chaos • increased efficiency • enhanced organizational understanding • faster NPD cycle-time Pratt – Slide 61
  • 63.
    In Conclusion: It isnever too late to start recording data! - even simple data such as key dates and time spent on activities can facilitate powerful analyses - capture data in real-time; don’t rely on memory or post mortem activities Pratt – Slide 62
  • 64.
  • 65.
    Thank You! STEVEN PRATT,MEng., PE Director of Engineering www.alpha.ca Pratt – Slide 64