BMNT Partners, profile picture
Uploaded byBMNT Partners
PDF, PPTX1,663 views

Pete Newell & Steve Blank on Diagnostics and Metrics for Innovation

The document discusses the development and implementation of an innovation pipeline designed to enhance innovation within military and defense operations. It emphasizes the need for structured processes, metrics, and diagnostics to efficiently connect innovation activities and rapidly deliver solutions. Key components include technology readiness levels, investment readiness levels, and adoption readiness levels, all aimed at overcoming systemic barriers to innovation.

Embed presentation

Download as PDF, PPTX
1 / 112
2 / 112
3 / 112
4 / 112
5 / 112
6 / 112
7 / 112
8 / 112
9 / 112
10 / 112
11 / 112
12 / 112
13 / 112
14 / 112
15 / 112
16 / 112
17 / 112
18 / 112
19 / 112
20 / 112
21 / 112
22 / 112
23 / 112
24 / 112
25 / 112
26 / 112
27 / 112
28 / 112
29 / 112
30 / 112
31 / 112
32 / 112
33 / 112
34 / 112
35 / 112
36 / 112
37 / 112
38 / 112
39 / 112
40 / 112
41 / 112
42 / 112
43 / 112
44 / 112
45 / 112
46 / 112
47 / 112
48 / 112
49 / 112
50 / 112
51 / 112
52 / 112
53 / 112
54 / 112
55 / 112
56 / 112
57 / 112
58 / 112
59 / 112
60 / 112
61 / 112
62 / 112
63 / 112
64 / 112
65 / 112
66 / 112
67 / 112
68 / 112
69 / 112
70 / 112
71 / 112
72 / 112
73 / 112
74 / 112
75 / 112
76 / 112
77 / 112
78 / 112
79 / 112
80 / 112
81 / 112
82 / 112
83 / 112
84 / 112
85 / 112
86 / 112
87 / 112
88 / 112
89 / 112
90 / 112
91 / 112
92 / 112
93 / 112
94 / 112
95 / 112
96 / 112
97 / 112
98 / 112
99 / 112
100 / 112
101 / 112
102 / 112
103 / 112
104 / 112
105 / 112
106 / 112
107 / 112
108 / 112
109 / 112
110 / 112
111 / 112
112 / 112
The Innovation Pipeline Diagnostics, Metrics and Doctrine Steve Blank, Pete Newell www.steveblank.com www.bmnt.com 1 2019 BMNT & Steve Blank
• US Army Colonel (Retired) • Enlisted National Guardsman • Commanded platoon, company, battalion and brigade Infantry units • Combat tours in Panama (1989) and Iraq (2004 & 2009) • Watch Officer on duty in the National Military Command Center (Pentagon) on 9/11 • Led the U.S. Army Rapid Equipping Force • 360 projects, 115 material deployments, 20 Program of Record transitions • $1.5B invested • Defense News 100 most influential people in the Defense Industry (2012) • Co-Founder and CEO, BMNT (2013) • Forbes top 25 veteran founded start-ups in America (2015) • Inc 1000 list of the fastest growing private companies in America (2017) • Co-Author, Hacking for Defense • Co-Founder and Board Director H4Di.org (2015) • Seed Investor, Board Advisor and Board Director • Visiting Fellow, National Defense University (2014-2016) Pete Newell 2
Steve Blank • Innovation consultant to the Intelligence community • Co-author: Hacking for Defense (H4D) • Co-author: I-Corps @ NSA • Author: NSF I-Corps • Co-creator of the Lean Startup methodology • Participant/founder 8 startups in Silicon Valley • Author The Four Steps to the Epiphany and The Startup Owner’s Manual • Stanford - Adjunct Professor / Columbia – Senior Fellow • USAF 1971-1975 EW/ELINT in TAC and SAC 3
The Innovation Pipeline What got this conversation started 4 2019 BMNT & Steve Blank
2016 Lots of Activities, Lots of Frustration 5 2019 BMNT & Steve Blank
Individual Innovation Innovation Activities Single-Team Innovation Operational Innovation Leadership Process Examples Deliverables Heroic CIA Innovators Alliance One-off successes Frustration w/system 6 2019 BMNT & Steve Blank
Individual Innovation Innovation Activities Single-Team Innovation Operational Innovation Leadership Process Examples Deliverables Heroic Innovators Alliance One-off successes Frustration w/system Innovators – invent things Entrepreneurs – get innovation adopted 7 2019 BMNT & Steve Blank
Individual Innovation Innovation Activities Single-Team Innovation Operational Innovation Leadership Process Examples Deliverables Heroic CIA Innovators Alliance One-off successes Frustration w/system 8 2019 BMNT & Steve Blank
Individual Innovation Innovation Tools & Activities Single-Team Innovation Operational Innovation Process Examples Deliverables Heroic Innovators Alliance One-off successes Frustration w/system Activities/ Training Lean, Design Thinking, Hackathons, Incubators, Outposts, Maker Spaces Innovation Demos, Prototypes People 9 2019 BMNT & Steve Blank
Individual Innovation Innovation Activities Single-Team Innovation Operational Innovation Process Examples Deliverables Heroic Innovators Alliance One-off successes Frustration w/system Activities/ Training Lean, Design Thinking, Hackathons, Incubators, Maker Spaces Innovation Demos, Prototypes People Lean/I-Corps I-Corps@NSA, H4D Problem Curation Evidence-based MVPs Incubator-ready Team We lacked an end-to-end process - from sourcing to deployment 10 2019 BMNT & Steve Blank
Individual Innovation Innovation Activities Single-Team Innovation Mission Driven Innovation Process Examples Deliverables Heroic Innovators Alliance One-off successes Frustration w/system Activities/ Training Lean, Design Thinking, Hackathons, Incubators, Maker Spaces Innovation Demos, Prototypes People Lean/I-Corps I-Corps@NSA, H4D Problem Curation Evidence-based MVPs Pipeline Innovation Pipeline Deliver more, faster Decision Points So we built one 11 2019 BMNT & Steve Blank
2017 We deployed the Innovation Pipeline & a few new tools 12 2019 BMNT & Steve Blank
An Innovation Pipeline That solves the Red Queen Problem Requires a disciplined, evidence-based, data-driven, process for connecting innovation activities into an accountable system that rapidly delivers solutions to hard problems Here is our Example 13
New Tool - Mission Model Canvas 14
New Tool - Portfolio Management 15
2018 We got out of the building and gathered data …and we learned a ton 16
Individual Innovation Innovation Activities Single-Team Innovation Operational Innovation Process Examples Deliverables Heroic Innovators Alliance One-off successes Frustration w/system Activities/ Training Lean, Design Thinking, Hackathons, Incubators, Maker Spaces Innovation Demos, Prototypes People Lean/I-Corps I-Corps@NSA, H4D Problem Curation Evidence-based MVPs Pipeline Innovation Pipeline – CIA H4X Deliver more, faster Decision Points Contracts HRFinance LegalProcurement Security • One of the things we learned was that bottlenecks to innovation existed everywhere • We asked, “Why?” • And how can we fix it? 17 2019 BMNT & Steve Blank
Individual Innovation Innovation Activities Single-Team Innovation Operational Innovation Leadership Process Examples Deliverables Heroic Innovators Alliance One-off successes Frustration w/system Activities/ Training Lean, Design Thinking, Hackathons, Incubators, Maker Spaces Innovation Demos, Prototypes People Lean/I-Corps I-Corps@NSA, H4D Problem Curation Evidence-based MVPs Pipeline Innovation Pipeline – CIA H4X Deliver more, faster Decision Points Guidance/ Investment/Org Decisions • In a perfect world the DNI would extend current guidance to include Innovation Guidance: 1. Replace capabilities that atrophy (tech, tradecraft, etc.) 2. Create new capabilities that get us ahead of adversaries 3. Create new processes to do more faster 4. Create a way to rapidly reconfigure and standup new organizations as new threats and tech emerges 5. Create innovation processes, procedures and orgs to do so Innovation as a parallel system Innovation Doctrine Barrier Reduction 18 2019 BMNT & Steve Blank
Individual Innovation Innovation Activities Single-Team Innovation Operational Innovation Leadership Process Examples Deliverables Heroic Innovators Alliance One-off successes Frustration w/system Activities/ Training Lean, Design Thinking, Hackathons, Incubators, Outposts, Maker Spaces Innovation Demos, Prototypes People Lean/I-Corps I-Corps@NSA, H4D Problem Curation Evidence-based MVPs Incubator-ready Team Pipeline Innovation Pipeline – CIA H4X, ONR Deliver more, faster Decision Points Innovation Network Guidance/ Investment/Org Decisions Innovation as a parallel system Innovation Doctrine Barrier Reduction 19 2019 BMNT & Steve Blank
Individual Innovation Innovation Activities Single-Team Innovation Operational Innovation Leadership Process Examples Deliverables Heroic Innovators Alliance One-off successes Frustration w/system Activities/ Training Lean, Design Thinking, Hackathons, Incubators, Maker Spaces Innovation Demos, Prototypes People Lean/I-Corps I-Corps@NSA, H4D Problem Curation Evidence-based MVPs Pipeline Innovation Pipeline – CIA H4X Deliver more, faster Decision Points Guidance/ Investment/Org Decisions • Innovation Guidance would remove barriers • By creating fast-track processes for innovation Innovation as a parallel system Innovation Doctrine Barrier Reduction Contracts HRFinance LegalProcurement Security X X XX X X 20©2019 BMNT & Steve Blank
Individual Innovation Single-Team Innovation Operational Innovation Leadership Process Examples Deliverables One-off successes Frustration w/system Lean/I-Corps I-Corps@NSA, H4D Problem Curation Evidence-based MVPs Pipeline Innovation Pipeline – CIA H4X Deliver more, faster Decision Points Guidance/ Investment/Org Decisions Innovation Network Problem: • It’s not a perfect world • Leadership is focused on execution • Innovators feels like it is a continual uphill battle Innovation as a Pillar Coffee Cups, Lanyards, posters There is a fix • Diagnostics • Metrics • Doctrine 21 2019 BMNT & Steve Blank
The Innovation Pipeline 2019 22 2019 BMNT & Steve Blank
When operationalized… the innovation pipeline creates a funnel for innovation Ideas Technology Problems People Deployed Solutions 23©2019 BMNT & Steve Blank
The Innovation Pipeline Diagnostics, Metrics and Doctrine 24
What We Learned: The Innovation Pipeline • Leadership needs metrics to defend their decisions and resource requirements • Goals need to be set, investments made and resources applied at every step of the pipeline • Problem curation is more complex and critical than we thought. Prioritization is the output, not a next step • Skilled practitioners and experienced entrepreneurs are needed both inside and outside the agencies • We need to incubate better: We’ve created a “Bermuda triangle” where nascent teams are lost because they are not ready for an accelerator or to transition to a program 252019 BMNT & Steve Blank
What We Learned: The Innovation Pipeline There are 5 critical components to each step in the pipeline: 1. Strategy 2. Pipeline Process 3. Operational Support 4. Activities, Methodologies and Tools 5. Metrics and Data 26 2019 BMNT & Steve Blank
What We Learned: The Innovation Pipeline There are GO/NO-GO stops between each step 27 2019 BMNT & Steve Blank
… and in the next two days you’re going to hear from others who also got out of the building and got real data 1:15-2:30 Setting the tone: Doctrine, Diagnostics and Prime Investments - Part I 2:30-2:45 Break 2:45-4:00 Setting the tone: Doctrine, Diagnostics and Prime Investments - Part II 4:00-4:15 Break 4:15-5:15 Discussion #1: Internal Innovation (Lead by Kevin Keaton, NSA; facilitated by Steve Weinstein) 5:15-6:00 Ranch tour, reset, dinner prep 6:00-8:00 Dinner: Discussion #2: Countering Chinese investment in the US (Lead by Mike Brown, DIU; facilitated by Steve Blank) 9:00-10:00 Discussion #3: Scaling Innovation (Lead by Rich Carlin, ONR; facilitated by William Treseder) 10:00-10:30 Break: Birds of a feather #1 10:30-11:30 Discussion #4: Investing in Innovation (Lead by Chris Moran, LMCO & Steve Bowsher, IQT; facilitated by Heather Richman) 11:30-12:30 Lunch: Discussion #5 (Innovation Education Panel; facilitated by Pete Newell) 12:30-1:30 Discussion #6: Building an Innovation Strategy (Lead by Christy Monaco; facilitated by Doug Weinstein) 1:30-2:30 Discussion #7: Diagnosing the Innovation System (Lead by Doug Cate, DIA; facilitated by Brian Miller) 2:30-2:45 Innovation Pipeline Cohort Results 2:45-3:15 Break: Birds of a feather #2 3:15-4:00 Wrap up - What’s important (Pete / Steve) Today Tomorrow 28
Diagnostics and Metrics Overview 29
Diagnostics & Metrics: The Innovation Pipeline Activities, Methodologies, & Tools Operational Support Pipeline & Infrastructure Process Strategy Metrics & Data 30 2019 BMNT & Steve Blank
Strategy: The Innovation Pipeline Strategy The team responsible for the step needs established goals for the step Leadership needs established goals for each step 31 2019 BMNT & Steve Blank
Pipeline & Infrastructure: The Innovation Pipeline Pipeline & Infrastructure Process Process • What’s the input and where does it come from? • What’s the output (deliverable product of this step)? Team / Resources • What team and what expertise is needed for the team managing the pipeline? • What about for teams going through the pipeline? • What resources are required (funding, contracts vehicles, facilities) 32 2019 BMNT & Steve Blank
Operational Support: The Innovation Pipeline Organizational support • Contracts & contracting officers, Human Resources, Security, access to testing facilities, standing exceptions to policy Operational Support Leadership support • Provide resource & compliance approvals, conduct firefighting, act at key decision points between steps. 33 2019 BMNT & Steve Blank
Activities, Methodologies, Skills & Tools: The Innovation Pipeline Activities and Methodologies • What methodologies are used to perform this step • Skills required and experiences needed Activities, Methodologies, Skills & Tools Skills and Tools • Skills need to enable execution • Tools applied to collect insight 34 2019 BMNT & Steve Blank
Metrics & Data: The Innovation Pipeline Metrics are used to measure throughput and quality of output Metrics & Data Vital signs (data) are used to check performance 35 2019 BMNT & Steve Blank
The Innovation Pipeline: Decision Support Tool 36 2019 BMNT & Steve Blank
TRL+ IRL+ ARL Measure Readiness How to Measure Team Progress Through the Innovation Pipeline 37
Measure Team Progress • Measure Technology Readiness, Investment Readiness and Acquisition/Deployment Readiness • All are data-driven and evidence-based • Each team can track their own progress • Leadership can measure teams progress • Can establish funding and deployment milestones based on TRL/IRL/ARL number 38 2019 BMNT & Steve Blank
Technology + Investment + Adoption Readiness Level TRL Problem validation Complete a first-pass canvas Problem/Solution Validation Low Fidelity MVP Experiments Validate Solution/Mission fit Validate right side of canvas Prototype High-Fidelity MVP Validate left-side of canvas Validated funding/deployment IRL ARL 39 2019 BMNT & Steve Blank
Technology Readiness Level Investment Readiness Level Adoption Readiness Level When to Measure Readiness Levels 40©2019 BMNT & Steve Blank
TRL – Measure Technology Readiness How to Measure Team Progress Through the Innovation Pipeline 41
NASA/DOD Technology Readiness Level (TRL) • Formal Way to assess project maturity • Quantify Relative Risks • Data Driven • Adopted by NASA, DOD, FAA, ESA… 42
NASA/DOD: Technology Readiness Level 43
NASA/DOD Technology Readiness: Levels 1 & 2 Basic Technology Research • Basic principles observed • Technology concept formulated Concept 44
NASA/DOD Technology Readiness Levels 3 & 4 Research to prove Feasibility • Experimental proof of concept • Breadboard validation in lab Research Concept 45
NASA/DOD Technology Readiness Levels 5 & 6 Demo Prototype • Breadboard validation outside the building • System demo in real-world Research Concept Demo 46
NASA/DOD Technology Readiness Levels 7, 8, 9 Deployment • System Development • System deployed in real-world Research Concept Demo 47
TRL Measure Team Progress • TRLs are data-driven and evidence-based • Each team can track their own progress • Leadership can measure teams progress • Can establish funding milestones based on TRL number 48
IRL – Measure Investment Readiness How to Measure Team Progress Through the Innovation Pipeline 49
Investment Readiness Level • A Formal Way to Quantify Relative Risks • Data Driven • Analog to NASA/DOD Technology Readiness Level 50
Investment Readiness Level We can do the same for new mission-driven ventures 51
Investment Readiness Level We can do the same for mission-driven Ventures Emphasis is on data 52
Mission Model Canvas drives the – Investment Readiness Level Mission Model Canvas = hypotheses of how you create and deliver value for the DOD/IC and the warfighter Source: Alexander Osterwalder- Business Model Generation KEY PARTNERS Value Proposition CHANNELS CUSTOMER RELATIONSHIPS CUSTOMER SEGMENTS REVENUE STREAMSCOST STRUCTURE KEY ACTIVITIES KEY RESOURCES Beneficiaries Deployment Buy- in/Support Mission Achievement Value Proposition Activities Resources Partners Costs 53
Mission Model Canvas Source: Alexander Osterwalder- Business Model Generation KEY PARTNERS Value Proposition CHANNELS CUSTOMER RELATIONSHIPS CUSTOMER SEGMENTS REVENUE STREAMSCOST STRUCTURE KEY ACTIVITIES KEY RESOURCES Beneficiaries Deployment Buy- in/Support Mission Achievement Factors Value Proposition Activities Resources Partners Costs How does the team get “Buy- In” from all the beneficiaries? How will we deploy the product to widespread use? What constitutes a successful deployment? Who are our most important customers? Stakeholders? What are their pains/gains? What job do they want us to get done for them? How are we solving each customers pains/gains? What product/service features match their needs? What key activities do we need to be expert in? What key resources do we need to own or acquire? Financial? Human? Who are our key partners? Suppliers? What are we getting from them? Giving them? What is the Mission Budget/Cost? How will we measure Mission Achievement? 54
Investment Readiness Level KEY PARTNERS Value Proposition CHANNELS CUSTOMER RELATIONSHIPS CUSTOMER SEGMENTS REVENUE STREAMSCOST STRUCTURE KEY ACTIVITIES KEY RESOURCES • The TRL answers questions about the technology • The IRL answers questions about the Mission Model Canvas in Curation, and Discovery 55
Investment Readiness Level Problem validation Complete a first-pass canvas Problem/Solution Validation Low Fidelity MVP Experiments Validate Solution/Mission fit Validate right side of canvas Prototype High-Fidelity MVP Validate left-side of canvas Validated funding/deployment KEY PARTNERS Value Proposition CHANNELS CUSTOMER RELATIONSHIPS CUSTOMER SEGMENTS REVENUE STREAMSCOST STRUCTURE KEY ACTIVITIES KEY RESOURCES The IRL answers questions about the Mission Model Canvas in Curation, and Discovery 56
IRL Measure Team Progress • IRLs are data-driven and evidence-based • Each team can track their own progress • Leadership can measure teams progress • Can establish funding milestones based on IRL number 57
Technology Readiness Level + Investment Readiness Level Problem validation Complete a first-pass canvas Problem/Solution Validation Low Fidelity MVP Experiments Validate Solution/Mission fit Validate right side of canvas Prototype High-Fidelity MVP Validate left-side of canvas Validated funding/deployment TRL IRL 58
ARL – Measure Adoption Readiness How to Measure Team Progress Through the Innovation Pipeline 59 2019 BMNT & Steve Blank
ARL - Adoption Readiness Level • The TRL answers questions about the technology • The IRL answers questions about the Mission Model Canvas in Curation, Discovery and Incubation • The ARL answers questions about Incubation & Transition 60 2019 BMNT & Steve Blank
Adoption Readiness Level The ARL answers questions Incubation about Transition Sponsor $s to figure out solution/ mission fit Sponsor & Problem Owner Commitment to find a solution Teams Funded Funded Pilots Field Trials *Final DOTMLPF Deployment Leadership commitment Final Requirement * DOTMLPF is defined in The Joint Capabilities Integration Development System, or JCIDS Process. The JCIDS process provides a solution space that considers solutions (stated in terms of diplomatic or military organization) involving a simulation of doctrine, organization, training, materiel, leadership and education, personnel, and facilities (DOTMLPF) required to accomplish a mission.[2 61 2019 BMNT & Steve Blank
ARL Measure Progress to get a solution adopted • ARLs are data-driven and evidence-based • Each team can track their own progress • Leadership can measure teams progress • Can establish funding milestones based on ARL number 62 2019 BMNT & Steve Blank
TRL+ IRL+ ARL Measure Readiness How to Measure Team Progress Through the Innovation Pipeline 63 2019 BMNT & Steve Blank
Technology + Investment + Adoption Readiness Level TRL Problem validation Complete a first-pass canvas Problem/Solution Validation Low Fidelity MVP Experiments Validate Solution/Mission fit Validate right side of canvas Prototype High-Fidelity MVP Validate left-side of canvas Validated funding/deployment IRL ARL 64 2019 BMNT & Steve Blank
Self Assessment Diagnostics and Metrics 65
Strategy - What’s the goal for the Team - What’s the goal for Leadership Pipeline & Infrastructure Process Process - What’s the input and where does it come from? - What’s the output (deliverable product of this step)? Team / Resources - What team and what expertise is needed? - What resources are required (funding, contracts, exceptions) Operational Support - Type of organizational support required (contracts, HR, Security) - What leadership is required (resource/compliance decisions, firefighting etc) Activities, Methodologies, Tools and Metrics Activities and Methodologies - What methodologies are used to perform this step - Skills required and experiences needed Tools and Skills - Tools applied to collect insight - Skills need to enable execution Data and Assessments - What metrics are used to measure throughput - What vital signs (data) are used to check performance INNOVATION PIPELINE SELF ASSESSMENT KEY 66 2019 BMNT & Steve Blank
M Indicate missing components Highlight program weaknesses Highlight program strengths Comment Score Indicate score for each sub-category 1 - 5 1 – Dysfunctional or missing component 5 – Exceptional Total Total all scores from subcategories INNOVATION PIPELINE SELF ASSESSMENT KEY Teams’ goals are defined to achieve pipeline stage Appropriate input is available to garner success Team & expertise is available to achieve success Organizational support is available for this stage of the pipeline Activities are being executed to generate output Tools defined to collect insight Metrics are defined to determine success on this stage of the pipeline Leadership’s goal is defined to achieve pipeline stage Output is generated to drive next stage of the pipeline Resources are available to achieve success Leadership available & decisions points identified to have progress on this pipeline stage Methodologies defined/used to guide discovery Training is being provided to develop execution skills Data is collected to measure vital signs for this stage of the pipeline Score Score Score Score Score Score Score Score Score Score Score ScoreScore Score 67 2019 BMNT & Steve Blank
Sourcing Diagnostics and Metrics 68
Sourcing – Strategy: The Innovation Pipeline Strategy • Goal for the Innovation Team: Build a repeatable and scalable sourcing process that ensures there is a fresh pool of ideas, tech, people and problems that is exponentially larger than the number of solutions desired to be deployed (using internal and external resources) • Goal for Agency Leadership: Gain the peripheral vision needed to: • provide early warning that disruptive tech may by accelerating • Build systems to enable the org to rapidly seize opportunities to disrupt our enemies • Retaining the agility to respond to the unknown (at scale) 69 2019 BMNT & Steve Blank
Sourcing – Process: The Innovation Pipeline • Input to Sourcing: Active internal / external sourcing efforts • Output to Curation: Large pool of people, tech, ideas & problems (pool continuously refreshed) • Innovation team and expertise needed: Active & aggressive Ecosystem Architects with local knowledge, internal and external sourcing programs, great Bus Dev / Connector skills Pipeline & Infrastructure Process Resources required: • Full time & contract ecosystem sourcing team • Physical space • Contracting, financing, legal and security liaisons • $’s for travel, $ to host programs (ie: Hackathons, Sprints) 70 2019 BMNT & Steve Blank
Sourcing - Operational Support: The Innovation Pipeline • Organizational support required: Contracts, HR, Security, engineering and R&D liaisons with authority for sourcing exceptions Operational Support • Leadership support required: Prioritize current & future problems for the innovation team. Provide the support to allow rapid and fast track internal and external sourcing of tech, problems and people by providing parallel processes for security, contracting, HR, finance, policy and engineering.) 71 2019 BMNT & Steve Blank
Sourcing - Activities, Methodologies, Tools & Metrics: The Innovation Pipeline • Methodologies to perform sourcing: Funnel, Hackathons, Crowd Sourcing, Kickstarters, external partnerships, proxy generation, Outreach Services, Extraction Workshop, H4X • Skills/experiences to perform sourcing: Aggressive connector, team/coalition builder, connection to senior leadership, organizational silo breaker Activities, Methodologies, Skills & Tools • Skills for execution: Aggressive ecosystem development team, H4X, Lean • Tools applied to collect insight: Osterwalder Portfolio Management, innovation pipeline model, input / output ratio / funnel metrics, Problem Sourcing Canvas 72 2019 BMNT & Steve Blank
Sourcing – Metrics & Data: The Innovation Pipeline Metrics & Data Metrics • # of active internal and external sourcing programs • Ratio of new sources netted each event to those actually engaged in other parts of the pipeline over time (passive vs active connections) • Ratio of entities sourced vs entities accepted into next phase (1-1 ration indicates a weak sourcing mechanism, 4-1 may be more preferable) Data • # and diversity of independent sources (tech, people, problems & ideas) across three horizons of potential innovation • # of active engagements from the pool that remain engaged in follow-on steps (ie; # of innovators and entrepreneurs moving into curation) • # of vetted problems accepted into Curation 73 2019 BMNT & Steve Blank
Self Assessment Diagnostics and Metrics 74
Strategy Process Team & Resources Operational Support Activities & Methods Metrics & Data Team’s goals are defined to achieve pipeline stage Leadership’s goal is defined to achieve pipeline stage Appropriate input is available to garner success Output is generated to drive next stage of the pipeline Team & expertise is available to achieve success Resources are available to achieve success Organizational support is available for this stage of the pipeline Leadership available & decisions points identified to have progress on this pipeline stage Activities are being executed to generate output Methodologies defined/used to guide discovery Tools defined to collect insight Training is being provided to develop execution skills Metrics are defined to determine success on this stage of the pipeline Data is collected to measure vital signs for this stage of the pipeline Tools & Skills Score Score Score Score Score Score Score Score Score Score Score Score Score Score *Total Score SOURCE 75 2019 BMNT & Steve Blank
Curation Diagnostics and Metrics 76
Curation – Strategy: The Innovation Pipeline Strategy • Goal for the Innovation Team: Build a repeatable and scalable curation and prioritization process that creates a vetted problem pool that is exponentially larger than the number of solutions desired to be deployed • Goal for Agency Leadership: • Continuous insight into emerging problems and the decay of current assets. • Establish organizational momentum for innovation based on a prioritized portfolio (Horizon 1, 2, 3) of current & future problems. 77 2019 BMNT & Steve Blank
Curation – Process: The Innovation Pipeline • Input from Sourcing: a healthy & diverse pool of ideas, problems, technology and innovators from active internal AND external sourcing programs • Output to Discovery: prioritized portfolio of curated problems. Concept sketch for how the problem presents itself, initial tech & expert ecosystem map, problem sponsor & champion for each problem Pipeline & Infrastructure Process • Innovation team expertise needed: Problem Curation team, technology analysis, ecosystem architects • Resources required: Full time curation team, physical space, contracting, financing, legal and security liaisons. $’s for travel. Organizational OK for curation team to spend time on problems/tech in the pipeline, teams into the pipeline 782019 BMNT & Steve Blank
Curation - Operational Support: The Innovation Pipeline Operational Support • Organizational support required: Contracts, HR, Security, engineering and R&D liaisons with authority for sourcing exceptions • Leadership support required: top-level agreement for fast track sourcing, committed funding, written exceptions for security, contracting, policy, legal for innovation sourcing 79 2019 BMNT & Steve Blank
Curation - Activities, Methodologies, Skills & Tools: The Innovation Pipeline • Activities supporting curation: Active outreach to find problems and sponsors, Problem Curation Workshop, Technology Exploration • Methodologies to perform curation: Problem Curation, Concept (CONOP) development, MVP development, Mission Model Canvas (MMC), Portfolio map, Investment Readiness Level (IRL) / Technology Readiness Level (TRL) / Adoption Readiness Level (ARL) Activities, Methodologies, Skills & Tools • Skills & experiences to perform curation: Problem Curation, Talking to Humans, Interviewing, Lean, Technology Assessment, Design Thinking, MVP development, Iterative Development • Tools applied to collect insight: Problem curation workbook, petal diagrams, Value Proposition Canvas (VPC), Collaboration Map 802019 BMNT & Steve Blank
Curation – Metrics & Data: The Innovation Pipeline Metrics & Data Metrics • Ratio of problems / new concepts curated vs those accepted into Discovery (1-1 reflects a weak pool, 4-1 is closer to historical norms from REF and H4D) • Ratio of problems / new concepts spread across Alex O’s portfolio map vs 3 Horizons • Initial TRL / IRL and ARL established • Quality of teams in Discovery - how many meet minimum standard & remain intact Data • # of beneficiary, innovators, entrepreneurs and tech expert interviews conducted to validate problems & ideas • Diversity of innovators, entrepreneurs and experts engaged in validation 81 2019 BMNT & Steve Blank
Self Assessment Diagnostics and Metrics 82
Strategy Process Team & Resources Operational Support Activities & Methods Metrics & Data Team’s goals are defined to achieve pipeline stage Leadership’s goal is defined to achieve pipeline stage Appropriate input is available to garner success Output is generated to drive next stage of the pipeline Team & expertise is available to achieve success Resources are available to achieve success Organizational support is available for this stage of the pipeline Leadership available & decisions points identified to have progress on this pipeline stage Activities are being executed to generate output Methodologies defined/used to guide discovery Tools defined to collect insight Training is being provided to develop execution skills Metrics are defined to determine success on this stage of the pipeline Data is collected to measure vital signs for this stage of the pipeline Tools & Skills Score Score Score Score Score Score Score Score Score Score Score Score Score Score *Total Score CURATE 83 2019 BMNT & Steve Blank
Discovery Diagnostics and Metrics 84©2019 BMNT & Steve Blank
Discovery – Strategy: The Innovation Pipeline • Goal for the Innovation Team: • Build a repeatable and scalable discovery process that validates problems and solution pathways, while also narrowing the potential solution teams to only the most viable for further investment. • Run a Lean/I-Corps model using synchronous teams, MMC, MVPs Strategy Goal for Agency Leadership: Establish a continuous flow of options to: • Accelerate to replace decaying/lost capabilities • Provide options to disrupt emerging protagonists 85 2019 BMNT & Steve Blank
Discovery – Process: The Innovation Pipeline • Input from Curation: prioritized portfolio of curated problems. For each problem; concept sketch for how the problem presents itself, initial tech & expert ecosystem map, problem sponsor & champion • Output to Incubation: Validated MVPs, test data and a pathway to deliver a solution; teams with the potential to deliver; Early adopters/1st customer ID’d • Innovation Team expertise needed: Lean/I-Corps Instructors, internal entrepreneurs recruited to teams to build agency buy-in & coalition. Pipeline & Infrastructure Process • Expertise for the team in the pipeline: Hypothesis generation, MVP development, MVP testing development, Interviewing techniques, data capture • Resources are required – Full time innovation team, physical space, contracting, financing, legal and security liaisons. $’s for travel. Organizational OK for innovators to spend time on problems/tech in the pipeline, teams into the pipeline. Other orgs bought into 1) fast track process 2) adoption of output 86
Discovery - Operational Support: The Innovation Pipeline • Organizational support required: • Contracts, HR, Security, engineering and R&D liaisons with authority for discovery and MVPs; • Understanding how to “get to Yes” from those groups; • Unfettered access to potential beneficiaries. • Coaches and mentors to support teams in discovery Leadership support required: • Top-level agreement for fast track MVPs; • Funding for teams in the discovery process (travel, prototyping, stipends, pay); w • Written exceptions for security, contracting, policy, legal for innovation sourcing; • Designated/trained organizational POCs in contracting, HR, Resource Management • Firefighting board of organizational leaders capable of reducing barriers for innovation teams Operational Support 87
Discovery - Activities, Methodologies, Tools & Metrics: The Innovation Pipeline • Activities supporting discovery: Hacking for Defense, I-Corps, Exploration Workshops, Discovery workshop, Product Jam / Experimentation Workshop • Methodologies to perform discovery: Lean StartUp, MVP Development, Mission Model Canvas (MMC), Business Model Canvas (BMC), Portfolio Management, Activities, Methodologies, Skills & Tools • Skills & experiences to perform discovery: Innovation teams need innovator + entrepreneurs; Pipeline leadership team needs to build agency buy-in and coalition; Internal & external entrepreneurs recruited to teams; Prototyping ability • Tools applied to collect insight: Mission Model Canvas (MMC), Business Model Canvas (BMC), Innovation Portfolio Map, Investment Decision Support Tool (IRL, TRL, ARL), Funnel Metrics, Workflow maps, User journey map, HMW statements 88 2019 BMNT & Steve Blank
Discovery – Metrics & Data: The Innovation Pipeline Metrics & Data Metrics • Ratio of teams entering discovery vs those moving on to incubation (1-1 ratio is flat, historical norm from venture due diligence, H4D and REF is ~4-1) • TRL, IRL and ARL established as part of decision to move forward into incubation • Ratio of teams in discovery spread across Alex O’s portfolio map vs 3 Horizons • Quality of team transitioning to Incubation (focused on left side of MMC) Data • # of hypothesis and MVP’s generated • # of team pivots • # and diversity of interviews conducted • TRL, IRL and ARL at completion of Discovery • # of external partners and internal parts of organization engaged 89
Self Assessment Diagnostics and Metrics 90
Strategy Process Team & Resources Operational Support Activities & Methods Metrics & Data Team’s goals are defined to achieve pipeline stage Leadership’s goal is defined to achieve pipeline stage Appropriate input is available to garner success Output is generated to drive next stage of the pipeline Team & expertise is available to achieve success Resources are available to achieve success Organizational support is available for this stage of the pipeline Leadership available & decisions points identified to have progress on this pipeline stage Activities are being executed to generate output Methodologies defined/used to guide discovery Tools defined to collect insight Training is being provided to develop execution skills Metrics are defined to determine success on this stage of the pipeline Data is collected to measure vital signs for this stage of the pipeline Tools & Skills Score Score Score Score Score Score Score Score Score Score Score Score Score Score *Total Score DISCOVER 91 2019 BMNT & Steve Blank
Incubation Diagnostics and Metrics 92
Incubation –The Innovation Pipeline • Goal for the Innovation Team: Build a repeatable and scalable incubation process – establish a supply of investible solutions - matching the number desired to be deployed Strategy • Goal for Agency Leadership: • Awareness and visible support of insertion of new solutions into existing program. • Funding for the teams in the incubator, active pipeline participation (mentors, coaching, etc.), continued support 93 2019 BMNT & Steve Blank
Incubation – Process: The Innovation Pipeline • Input from Discovery: • Validated MVPs and test data and a pathway to deliver a solution; teams with the potential to deliver; • Early adopters/1st customer ID’d • Output to Transition – Investible entity with validated IRL, TRL and ARL • Innovation Team expertise needed: Domain expert coaches and mentors, Pipeline leadership to build agency buy-in and coalition. Understand the politics of transition Pipeline & Infrastructure Process • Resources Required: • Recruit developers to teams in pipeline, physical space, contracting, financing, legal and security liaisons. • $’s for travel and funding for teams. Organizational OK for innovators to spend full- time on problems/tech in the pipeline. • Other orgs bought into 1) fast track process 2) adoption of output 94
Incubation - Operational Support: The Innovation Pipeline • Organizational support required: Contracts, HR, Security, engineering and R&D liaisons with authority for discovery and MVPs. Understanding how to “get to Yes” from those groups. Coaches and mentors Operational Support • Leadership support required: • Written agreement for how incubation moves to transition from contracting, security, legal, engineering, etc.; • Funding for teams; Written exceptions for security, contracting, policy, legal for innovation sourcing; • Firefighting board of organizational leaders capable of reducing barriers for innovation teams; • Prioritization for testing and evaluation 95 2019 BMNT & Steve Blank
Incubation - Activities, Methodologies, Skills & Tools: The Innovation Pipeline • Methodologies to perform incubation: MVPs, MMC, active coaching and mentoring, transition plans, scale and deployment plans • Skills/experiences to perform incubation: innovation teams need innovator + entrepreneurs + developers. Pipeline leadership team continue to build agency buy-in and coalition. Recruit developers to teams in pipeline • Skills for execution: Investment due diligence, gov contracts/grants management; prototyping, Agile Development, Scrum, IP assignment, partnership building, Business Development • Tools applied to collect insight: MMC, IRL, TRL, ARL, Funnel metrics Activities, Methodologies, Skills & Tools 96
Incubation - Activities, Methodologies, Skills & Tools: The Innovation Pipeline Metrics & Data Metrics • Ratio of teams in incubation vs those gaining follow-on investment (for H1 teams maybe 2-1, H2 teams 4-1, H3 teams 10-1) • Change in TRL, IRL and ARL of teams throughout Incubation and at departure to transition • Condensation: # of projects in discovery combined into one effort moving forward to incubation Data • # and ratio of MVP’s moving to prototype development • Size / scale of initial and subsequent user pools gained during incubation • $ committed to team during incubation • # and amount of follow-on investment, grants and gov contracts received • # of other organizations engaged 97
Self Assessment Diagnostics and Metrics 98
Strategy Process Team & Resources Operational Support Activities & Methods Metrics & Data Team’s goals are defined to achieve pipeline stage Leadership’s goal is defined to achieve pipeline stage Appropriate input is available to garner success Output is generated to drive next stage of the pipeline Team & expertise is available to achieve success Resources are available to achieve success Organizational support is available for this stage of the pipeline Leadership available & decisions points identified to have progress on this pipeline stage Activities are being executed to generate output Methodologies defined/used to guide discovery Tools defined to collect insight Training is being provided to develop execution skills Metrics are defined to determine success on this stage of the pipeline Data is collected to measure vital signs for this stage of the pipeline Tools & Skills Score Score Score Score Score Score Score Score Score Score Score Score Score Score *Total Score INCUBATE 99 2019 BMNT & Steve Blank
Transition Diagnostics and Metrics 100
Transition – Strategy: The Innovation Pipeline • Goal for the Innovation Team: Establish a pathway for off ramping a team from the innovation team to a transition team or program; manage organizational technology refactoring Strategy • Goal for Agency Leadership: • Support of insertion of new solutions into existing program – specifically by providing Funding and headcount for new solution insertion • Recognize and manage organizational technology refactoring • Establish guidance for transition of solutions that fit in the white space between programs (who is responsible for accepting?) 101 2019 BMNT & Steve Blank
Transition – Process: The Innovation Pipeline • Input from Incubation: Investible entity with validated IRL, TRL and ARL • Output to the agency: A scalable solution that is delivered in time to have an impact • Innovation Team expertise needed: At this step oversight responsibility should move from the the innovation team to a transition team capable of refactoring the solution to deliver at scale Pipeline & Infrastructure Process • Resources Required: • Funds for testing, advanced prototyping and access to facilities; • Funds for training, sustainment for 1 year to support the transition of a prototype into a program of record (POR); • Augment the innovation team with members of the organizational transition team 102 2019 BMNT & Steve Blank
Transition - Operational Support: The Innovation Pipeline • Organizational support required: • Contracts, HR, Security, engineering and R&D liaisons with authority for prototype development, testing & evaluations. • Access to users; Understanding how to “get to Yes” from those groups. Coaches and mentors Operational Support • Leadership support required: • Written agreement for how incubation moves to transition from contracting, security, legal, engineering, etc.; • Funding for teams; Written exceptions for security, contracting, policy, legal for innovation sourcing; • Firefighting board of organizational leaders capable of reducing barriers for scaling teams; • Prioritization for testing, evaluation and forward deployment 103 2019 BMNT & Steve Blank
Transition - Activities, Methodologies, Skills & Tools: The Innovation Pipeline • Methodologies to perform transition: Refactoring, Organizational Change Management, Sustainment packaging, Training Development, Kaizen for continuous improvement (Toyota’s engineering way) • Skills/experiences to perform transition: Agile, Scrum, Deployment planning, Training development, integration, supply chain management • Skills for execution: Transition management, budgeting, experimentation design, supply chain development, POR management, POM planning • Tools applied to collect insight: User surveys, adoptions surveys, localization tools (solution performance at different units, bases etc) Activities, Methodologies, Skills & Tools 104
Transition - Activities, Methodologies, Skills & Tools: The Innovation Pipeline Metrics & Data Metrics • Ratio of teams receiving initial investment leaving incubation to those receiving follow-on investment (different ratio for H1, H2 and H3) • Ratio of teams in transition to those accepted in POR or theater sustainment (different ratio for H1, H2, H3) • Average time spent in transition (shorter for H1, longer for H3) Data • Realized beneficiary value (impact on original problem) • IRL, TRL and ARL at transition to POR • Scaled implementation data (user drop off / pain points / patches or modification required) 105
Self Assessment Diagnostics and Metrics 106
Strategy Process Team & Resources Operational Support Activities & Methods Metrics & Data Team’s goals are defined to achieve pipeline stage Leadership’s goal is defined to achieve pipeline stage Appropriate input is available to garner success Output is generated to drive next stage of the pipeline Team & expertise is available to achieve success Resources are available to achieve success Organizational support is available for this stage of the pipeline Leadership available & decisions points identified to have progress on this pipeline stage Activities are being executed to generate output Methodologies defined/used to guide discovery Tools defined to collect insight Training is being provided to develop execution skills Metrics are defined to determine success on this stage of the pipeline Data is collected to measure vital signs for this stage of the pipeline Tools & Skills Score Score Score Score Score Score Score Score Score Score Score Score Score Score *Total Score TRANSITION 107 2019 BMNT & Steve Blank
Final Thoughts: Why have an innovation pipeline? The result of a pipeline are solutions that: 1. Continuously replenish lost capabilities 2. Create new capabilities that match current and future threats The purpose of innovation pipeline doctrine is to create the organization / process / budget and people to do so 108 2019 BMNT & Steve Blank
The Innovation Pipeline: Decision Support Tool 109 2019 BMNT & Steve Blank
The Innovation Pipeline: Self Assessment View 110©2019 BMNT & Steve Blank
181 GOV/COM entities engaged 1800 People Trained 1391 Problems Sourced 540 Problems Curated 122 Teams in Discovery 9 Solutions Delivered 28 Solutions in Incubation 1391 Problems 38% 23% 23% 32% % moving forward % of original total.6%2%9% US Army REF #’s 2011-2013 840 Problems 360 Projects initiated (42%) 115 Solutions Incubated in theater (31%) 20 Program of Record (17%) H4Di & BMNT 2018 EOY Snapshot 1112019 BMNT & Steve Blank
The Innovation Pipeline Diagnostics, Metrics and Doctrine 112 2019 BMNT & Steve Blank

More Related Content

PDF
Summary - The Four steps to the Epiphany - Steve Blank
byRohit Sharma
 
PPTX
Evidence-based Entrepreneurship by Steve Blank
byLean Startup Co.
 
PDF
Blitzscaling Session 9: Village Stage
byGreylock Partners
 
PPTX
Eric Ries - The Lean Startup - RailsConf 2011
byEric Ries
 
PPTX
Lean Startup Basics - Evidence Based Entrepreneurship
byKelly Schwedland
 
PPTX
Lean Software Startup: Customer Development (lecture)
byJoni Salminen
 
PDF
The Startup Owner's Manual - Founder's Workbook
byZoomstra.com
 
DOCX
The 4 steps to the epiphany - Customer Discovery template - guilhem bertholet
byGuilhem Bertholet
 
Summary - The Four steps to the Epiphany - Steve Blank
byRohit Sharma
 
Evidence-based Entrepreneurship by Steve Blank
byLean Startup Co.
 
Blitzscaling Session 9: Village Stage
byGreylock Partners
 
Eric Ries - The Lean Startup - RailsConf 2011
byEric Ries
 
Lean Startup Basics - Evidence Based Entrepreneurship
byKelly Schwedland
 
Lean Software Startup: Customer Development (lecture)
byJoni Salminen
 
The Startup Owner's Manual - Founder's Workbook
byZoomstra.com
 
The 4 steps to the epiphany - Customer Discovery template - guilhem bertholet
byGuilhem Bertholet
 

What's hot

PDF
BMNT's Innovation Pipeline Self Assessment
byBMNT Partners
 
PPTX
Agile Leadership: A Different Mindset
byAaron Kopel
 
PDF
Agile Transformation in Telco Guide
byACM
 
PDF
What the Heck Is a Product Owner?
byRon Lichty
 
PPTX
Agile Mindset For Executives
byMichael Tarnowski
 
PDF
SAFe® - scaled agile framework in practice
byIntland Software GmbH
 
PDF
Agile Leadership - Beyond the Basics
byMark Levison, CST
 
PDF
10 steps to a successsful enterprise agile transformation global scrum 2018
byAgile Velocity
 
PDF
The Agile Adoption Roadmap (Keynote by Tim Abbott)
byAgile Days Middle East
 
PPTX
Agile Transformation Explained
byLiminalArc
 
PDF
10 Essential SAFe(tm) patterns you should focus on when scaling Agile
byYuval Yeret
 
PDF
Agile transformation Explained: Agile 2017 Session
byLiminalArc
 
PDF
An Introduction to Scaled Agile Framework (SAFe)
byCA Technologies
 
PDF
Agile Transformation v1.27
byLiminalArc
 
PDF
Business Agility Workshop Content
byDavid Siegel
 
PDF
Design Thinking and Agile?
byDr. Tathagat Varma
 
PDF
Empowering Engineering Talent - an update from Spotify
byKevin Goldsmith
 
PPTX
Systematic Inventive Thinking and Process improvements
byKarthik Srinivasan
 
PDF
Agile Placemat v9
byChris Webb
 
PDF
the agile mindset, a learning lab
bynikos batsios
 
BMNT's Innovation Pipeline Self Assessment
byBMNT Partners
 
Agile Leadership: A Different Mindset
byAaron Kopel
 
Agile Transformation in Telco Guide
byACM
 
What the Heck Is a Product Owner?
byRon Lichty
 
Agile Mindset For Executives
byMichael Tarnowski
 
SAFe® - scaled agile framework in practice
byIntland Software GmbH
 
Agile Leadership - Beyond the Basics
byMark Levison, CST
 
10 steps to a successsful enterprise agile transformation global scrum 2018
byAgile Velocity
 
The Agile Adoption Roadmap (Keynote by Tim Abbott)
byAgile Days Middle East
 
Agile Transformation Explained
byLiminalArc
 
10 Essential SAFe(tm) patterns you should focus on when scaling Agile
byYuval Yeret
 
Agile transformation Explained: Agile 2017 Session
byLiminalArc
 
An Introduction to Scaled Agile Framework (SAFe)
byCA Technologies
 
Agile Transformation v1.27
byLiminalArc
 
Business Agility Workshop Content
byDavid Siegel
 
Design Thinking and Agile?
byDr. Tathagat Varma
 
Empowering Engineering Talent - an update from Spotify
byKevin Goldsmith
 
Systematic Inventive Thinking and Process improvements
byKarthik Srinivasan
 
Agile Placemat v9
byChris Webb
 
the agile mindset, a learning lab
bynikos batsios
 

Similar to Pete Newell & Steve Blank on Diagnostics and Metrics for Innovation

PPTX
Innovation at 50x 031616
byStanford University
 
PDF
Berkeley Method of Innovation Leadership
byIkhlaq Sidhu
 
PPTX
Innovation in a rapidly changing world (by Len Middleton)
byRobert Pasick, Ph.D
 
PDF
Managing Innovation: Integrating Technological, Market and Organizational Cha...
byfwxpkkwd024
 
PDF
Managing Innovation: Integrating Technological, Market and Organizational Cha...
byeiderrynne1o
 
PDF
Managing Innovation: Integrating Technological, Market and Organizational Cha...
byvendesinaye
 
PDF
Design Of Innovation Processes Flow From Idea To Market Launch With Higher Sp...
byoiiengmukii
 
PDF
Managing Innovation: Integrating Technological, Market and Organizational Cha...
byabacarmehmi
 
PDF
David cutler slides shared at mit ilp update
bySales Strategy and Innovation Delivery
 
PPTX
Building an Innovation & Strategy Process
byIT Weekend
 
PPT
Book summary of 8 steps to innovation—going from jugaad to excellence
bySuryanaryanan Suri
 
PDF
Most Influential Leaders In the Construction Industry, 2024.pdf
byInsightsSuccess4
 
PPTX
Leading Innovation
byTim Howell
 
PDF
Manufacturing Innovation Model | Has Patel | Lunch & Learn
byUCICove
 
PDF
BMNT Mission Results 2018
byBMNT Partners
 
PPT
The Politics of Innovation
byMalcolm De Leo
 
PPT
Leadership, Ingenuity, and Technology Interaction: Accelerators of Innovation
byPaul Schumann
 
PPT
@Innovation Getting it to getting it one
bytoufiquaanjumgiraman
 
PPT
briefing_280_8_kao_oftweb.ppt
byArindamMondal99
 
PPT
Judith Estrin May 4
bySIIA12
 
Innovation at 50x 031616
byStanford University
 
Berkeley Method of Innovation Leadership
byIkhlaq Sidhu
 
Innovation in a rapidly changing world (by Len Middleton)
byRobert Pasick, Ph.D
 
Managing Innovation: Integrating Technological, Market and Organizational Cha...
byfwxpkkwd024
 
Managing Innovation: Integrating Technological, Market and Organizational Cha...
byeiderrynne1o
 
Managing Innovation: Integrating Technological, Market and Organizational Cha...
byvendesinaye
 
Design Of Innovation Processes Flow From Idea To Market Launch With Higher Sp...
byoiiengmukii
 
Managing Innovation: Integrating Technological, Market and Organizational Cha...
byabacarmehmi
 
David cutler slides shared at mit ilp update
bySales Strategy and Innovation Delivery
 
Building an Innovation & Strategy Process
byIT Weekend
 
Book summary of 8 steps to innovation—going from jugaad to excellence
bySuryanaryanan Suri
 
Most Influential Leaders In the Construction Industry, 2024.pdf
byInsightsSuccess4
 
Leading Innovation
byTim Howell
 
Manufacturing Innovation Model | Has Patel | Lunch & Learn
byUCICove
 
BMNT Mission Results 2018
byBMNT Partners
 
The Politics of Innovation
byMalcolm De Leo
 
Leadership, Ingenuity, and Technology Interaction: Accelerators of Innovation
byPaul Schumann
 
@Innovation Getting it to getting it one
bytoufiquaanjumgiraman
 
briefing_280_8_kao_oftweb.ppt
byArindamMondal99
 
Judith Estrin May 4
bySIIA12
 

Recently uploaded

PDF
Digit Expo 2025 - EICC Edinburgh 27th November
byRay Bugg
 
PPTX
wob-report.pptxwob-report.pptxwob-report.pptx
byssuser0d171c
 
PDF
Day 1 - Cloud Security Strategy and Planning ~ 2nd Sight Lab ~ Cloud Security...
by2nd Sight Lab
 
PPTX
From Backup to Resilience: How MSPs Are Preparing for 2026
byMSP360
 
DOCX
iRobot Post‑Mortem and Alternative Paths - Discussion Document for Boards and...
byDave Litwiller
 
PDF
Day 3 - Data and Application Security - 2nd Sight Lab Cloud Security Class
by2nd Sight Lab
 
PPTX
Conversational Agents – Building Intelligent Assistants [Virtual Hands-on Wor...
byUiPathCommunity
 
PPTX
Coded Agents – with UiPath SDK + LangGraph [Virtual Hands-on Workshop]
byUiPathCommunity
 
PDF
Usage Control for Process Discovery through a Trusted Execution Environment
byValerioGoretti
 
PDF
Energy Storage Landscape Clean Energy Ministerial
bySurajitBanerjee38
 
PDF
Eredità digitale sugli smartphone: cosa resta di noi nei dispositivi mobili
bySpeck&Tech
 
PPTX
THIS IS CYBER SECURITY NOTES USED IN CLASS ON VARIOUS TOPICS USED IN CYBERSEC...
byMelissaGblinwon
 
PDF
Security Technologys: Access Control, Firewall, VPN
byShielaLasala
 
PDF
Is It Possible to Have Wi-Fi Without an Internet Provider
bySidra Jefferi
 
PPTX
AI in Cybersecurity: Digital Defense by Yasir Naveed Riaz
byYasir Naveed Riaz
 
PDF
Making Sense of Raster: From Bit Depth to Better Workflows
bySafe Software
 
PDF
GPUS and How to Program Them by Manya Bansal
byScyllaDB
 
DOCX
Introduction to the World of Computers (Hardware & Software)
byALIABBAS ABASOV
 
PDF
Six Shifts For 2026 (And The Next Six Years)
byDavid Armano
 
PPTX
Unit-4-ARTIFICIAL NEURAL NETWORKS.pptx ANN ppt Artificial neural network
byssuserd4481a
 
Digit Expo 2025 - EICC Edinburgh 27th November
byRay Bugg
 
wob-report.pptxwob-report.pptxwob-report.pptx
byssuser0d171c
 
Day 1 - Cloud Security Strategy and Planning ~ 2nd Sight Lab ~ Cloud Security...
by2nd Sight Lab
 
From Backup to Resilience: How MSPs Are Preparing for 2026
byMSP360
 
iRobot Post‑Mortem and Alternative Paths - Discussion Document for Boards and...
byDave Litwiller
 
Day 3 - Data and Application Security - 2nd Sight Lab Cloud Security Class
by2nd Sight Lab
 
Conversational Agents – Building Intelligent Assistants [Virtual Hands-on Wor...
byUiPathCommunity
 
Coded Agents – with UiPath SDK + LangGraph [Virtual Hands-on Workshop]
byUiPathCommunity
 
Usage Control for Process Discovery through a Trusted Execution Environment
byValerioGoretti
 
Energy Storage Landscape Clean Energy Ministerial
bySurajitBanerjee38
 
Eredità digitale sugli smartphone: cosa resta di noi nei dispositivi mobili
bySpeck&Tech
 
THIS IS CYBER SECURITY NOTES USED IN CLASS ON VARIOUS TOPICS USED IN CYBERSEC...
byMelissaGblinwon
 
Security Technologys: Access Control, Firewall, VPN
byShielaLasala
 
Is It Possible to Have Wi-Fi Without an Internet Provider
bySidra Jefferi
 
AI in Cybersecurity: Digital Defense by Yasir Naveed Riaz
byYasir Naveed Riaz
 
Making Sense of Raster: From Bit Depth to Better Workflows
bySafe Software
 
GPUS and How to Program Them by Manya Bansal
byScyllaDB
 
Introduction to the World of Computers (Hardware & Software)
byALIABBAS ABASOV
 
Six Shifts For 2026 (And The Next Six Years)
byDavid Armano
 
Unit-4-ARTIFICIAL NEURAL NETWORKS.pptx ANN ppt Artificial neural network
byssuserd4481a
 

Pete Newell & Steve Blank on Diagnostics and Metrics for Innovation

  • 1.
    The Innovation Pipeline Diagnostics,Metrics and Doctrine Steve Blank, Pete Newell www.steveblank.com www.bmnt.com 1 2019 BMNT & Steve Blank
  • 2.
    • US ArmyColonel (Retired) • Enlisted National Guardsman • Commanded platoon, company, battalion and brigade Infantry units • Combat tours in Panama (1989) and Iraq (2004 & 2009) • Watch Officer on duty in the National Military Command Center (Pentagon) on 9/11 • Led the U.S. Army Rapid Equipping Force • 360 projects, 115 material deployments, 20 Program of Record transitions • $1.5B invested • Defense News 100 most influential people in the Defense Industry (2012) • Co-Founder and CEO, BMNT (2013) • Forbes top 25 veteran founded start-ups in America (2015) • Inc 1000 list of the fastest growing private companies in America (2017) • Co-Author, Hacking for Defense • Co-Founder and Board Director H4Di.org (2015) • Seed Investor, Board Advisor and Board Director • Visiting Fellow, National Defense University (2014-2016) Pete Newell 2
  • 3.
    Steve Blank • Innovationconsultant to the Intelligence community • Co-author: Hacking for Defense (H4D) • Co-author: I-Corps @ NSA • Author: NSF I-Corps • Co-creator of the Lean Startup methodology • Participant/founder 8 startups in Silicon Valley • Author The Four Steps to the Epiphany and The Startup Owner’s Manual • Stanford - Adjunct Professor / Columbia – Senior Fellow • USAF 1971-1975 EW/ELINT in TAC and SAC 3
  • 4.
    The Innovation Pipeline Whatgot this conversation started 4 2019 BMNT & Steve Blank
  • 5.
    2016 Lots of Activities,Lots of Frustration 5 2019 BMNT & Steve Blank
  • 6.
  • 7.
    Individual Innovation Innovation Activities Single-Team Innovation Operational Innovation Leadership Process Examples Deliverables HeroicInnovators Alliance One-off successes Frustration w/system Innovators – invent things Entrepreneurs – get innovation adopted 7 2019 BMNT & Steve Blank
  • 8.
  • 9.
    Individual Innovation Innovation Tools & Activities Single-Team Innovation Operational Innovation Process ExamplesDeliverables Heroic Innovators Alliance One-off successes Frustration w/system Activities/ Training Lean, Design Thinking, Hackathons, Incubators, Outposts, Maker Spaces Innovation Demos, Prototypes People 9 2019 BMNT & Steve Blank
  • 10.
    Individual Innovation Innovation Activities Single-Team Innovation Operational Innovation Process Examples Deliverables HeroicInnovators Alliance One-off successes Frustration w/system Activities/ Training Lean, Design Thinking, Hackathons, Incubators, Maker Spaces Innovation Demos, Prototypes People Lean/I-Corps I-Corps@NSA, H4D Problem Curation Evidence-based MVPs Incubator-ready Team We lacked an end-to-end process - from sourcing to deployment 10 2019 BMNT & Steve Blank
  • 11.
    Individual Innovation Innovation Activities Single-Team Innovation Mission Driven Innovation Process Examples Deliverables HeroicInnovators Alliance One-off successes Frustration w/system Activities/ Training Lean, Design Thinking, Hackathons, Incubators, Maker Spaces Innovation Demos, Prototypes People Lean/I-Corps I-Corps@NSA, H4D Problem Curation Evidence-based MVPs Pipeline Innovation Pipeline Deliver more, faster Decision Points So we built one 11 2019 BMNT & Steve Blank
  • 12.
    2017 We deployed theInnovation Pipeline & a few new tools 12 2019 BMNT & Steve Blank
  • 13.
    An Innovation Pipeline Thatsolves the Red Queen Problem Requires a disciplined, evidence-based, data-driven, process for connecting innovation activities into an accountable system that rapidly delivers solutions to hard problems Here is our Example 13
  • 14.
    New Tool -Mission Model Canvas 14
  • 15.
    New Tool -Portfolio Management 15
  • 16.
    2018 We got outof the building and gathered data …and we learned a ton 16
  • 17.
    Individual Innovation Innovation Activities Single-Team Innovation Operational Innovation Process Examples Deliverables HeroicInnovators Alliance One-off successes Frustration w/system Activities/ Training Lean, Design Thinking, Hackathons, Incubators, Maker Spaces Innovation Demos, Prototypes People Lean/I-Corps I-Corps@NSA, H4D Problem Curation Evidence-based MVPs Pipeline Innovation Pipeline – CIA H4X Deliver more, faster Decision Points Contracts HRFinance LegalProcurement Security • One of the things we learned was that bottlenecks to innovation existed everywhere • We asked, “Why?” • And how can we fix it? 17 2019 BMNT & Steve Blank
  • 18.
    Individual Innovation Innovation Activities Single-Team Innovation Operational Innovation Leadership Process Examples Deliverables HeroicInnovators Alliance One-off successes Frustration w/system Activities/ Training Lean, Design Thinking, Hackathons, Incubators, Maker Spaces Innovation Demos, Prototypes People Lean/I-Corps I-Corps@NSA, H4D Problem Curation Evidence-based MVPs Pipeline Innovation Pipeline – CIA H4X Deliver more, faster Decision Points Guidance/ Investment/Org Decisions • In a perfect world the DNI would extend current guidance to include Innovation Guidance: 1. Replace capabilities that atrophy (tech, tradecraft, etc.) 2. Create new capabilities that get us ahead of adversaries 3. Create new processes to do more faster 4. Create a way to rapidly reconfigure and standup new organizations as new threats and tech emerges 5. Create innovation processes, procedures and orgs to do so Innovation as a parallel system Innovation Doctrine Barrier Reduction 18 2019 BMNT & Steve Blank
  • 19.
    Individual Innovation Innovation Activities Single-Team Innovation Operational Innovation Leadership Process Examples Deliverables HeroicInnovators Alliance One-off successes Frustration w/system Activities/ Training Lean, Design Thinking, Hackathons, Incubators, Outposts, Maker Spaces Innovation Demos, Prototypes People Lean/I-Corps I-Corps@NSA, H4D Problem Curation Evidence-based MVPs Incubator-ready Team Pipeline Innovation Pipeline – CIA H4X, ONR Deliver more, faster Decision Points Innovation Network Guidance/ Investment/Org Decisions Innovation as a parallel system Innovation Doctrine Barrier Reduction 19 2019 BMNT & Steve Blank
  • 20.
    Individual Innovation Innovation Activities Single-Team Innovation Operational Innovation Leadership Process Examples Deliverables HeroicInnovators Alliance One-off successes Frustration w/system Activities/ Training Lean, Design Thinking, Hackathons, Incubators, Maker Spaces Innovation Demos, Prototypes People Lean/I-Corps I-Corps@NSA, H4D Problem Curation Evidence-based MVPs Pipeline Innovation Pipeline – CIA H4X Deliver more, faster Decision Points Guidance/ Investment/Org Decisions • Innovation Guidance would remove barriers • By creating fast-track processes for innovation Innovation as a parallel system Innovation Doctrine Barrier Reduction Contracts HRFinance LegalProcurement Security X X XX X X 20©2019 BMNT & Steve Blank
  • 21.
    Individual Innovation Single-Team Innovation Operational Innovation Leadership Process Examples Deliverables One-offsuccesses Frustration w/system Lean/I-Corps I-Corps@NSA, H4D Problem Curation Evidence-based MVPs Pipeline Innovation Pipeline – CIA H4X Deliver more, faster Decision Points Guidance/ Investment/Org Decisions Innovation Network Problem: • It’s not a perfect world • Leadership is focused on execution • Innovators feels like it is a continual uphill battle Innovation as a Pillar Coffee Cups, Lanyards, posters There is a fix • Diagnostics • Metrics • Doctrine 21 2019 BMNT & Steve Blank
  • 22.
  • 23.
    When operationalized… the innovationpipeline creates a funnel for innovation Ideas Technology Problems People Deployed Solutions 23©2019 BMNT & Steve Blank
  • 24.
    The Innovation Pipeline Diagnostics,Metrics and Doctrine 24
  • 25.
    What We Learned:The Innovation Pipeline • Leadership needs metrics to defend their decisions and resource requirements • Goals need to be set, investments made and resources applied at every step of the pipeline • Problem curation is more complex and critical than we thought. Prioritization is the output, not a next step • Skilled practitioners and experienced entrepreneurs are needed both inside and outside the agencies • We need to incubate better: We’ve created a “Bermuda triangle” where nascent teams are lost because they are not ready for an accelerator or to transition to a program 252019 BMNT & Steve Blank
  • 26.
    What We Learned:The Innovation Pipeline There are 5 critical components to each step in the pipeline: 1. Strategy 2. Pipeline Process 3. Operational Support 4. Activities, Methodologies and Tools 5. Metrics and Data 26 2019 BMNT & Steve Blank
  • 27.
    What We Learned:The Innovation Pipeline There are GO/NO-GO stops between each step 27 2019 BMNT & Steve Blank
  • 28.
    … and inthe next two days you’re going to hear from others who also got out of the building and got real data 1:15-2:30 Setting the tone: Doctrine, Diagnostics and Prime Investments - Part I 2:30-2:45 Break 2:45-4:00 Setting the tone: Doctrine, Diagnostics and Prime Investments - Part II 4:00-4:15 Break 4:15-5:15 Discussion #1: Internal Innovation (Lead by Kevin Keaton, NSA; facilitated by Steve Weinstein) 5:15-6:00 Ranch tour, reset, dinner prep 6:00-8:00 Dinner: Discussion #2: Countering Chinese investment in the US (Lead by Mike Brown, DIU; facilitated by Steve Blank) 9:00-10:00 Discussion #3: Scaling Innovation (Lead by Rich Carlin, ONR; facilitated by William Treseder) 10:00-10:30 Break: Birds of a feather #1 10:30-11:30 Discussion #4: Investing in Innovation (Lead by Chris Moran, LMCO & Steve Bowsher, IQT; facilitated by Heather Richman) 11:30-12:30 Lunch: Discussion #5 (Innovation Education Panel; facilitated by Pete Newell) 12:30-1:30 Discussion #6: Building an Innovation Strategy (Lead by Christy Monaco; facilitated by Doug Weinstein) 1:30-2:30 Discussion #7: Diagnosing the Innovation System (Lead by Doug Cate, DIA; facilitated by Brian Miller) 2:30-2:45 Innovation Pipeline Cohort Results 2:45-3:15 Break: Birds of a feather #2 3:15-4:00 Wrap up - What’s important (Pete / Steve) Today Tomorrow 28
  • 29.
  • 30.
    Diagnostics & Metrics:The Innovation Pipeline Activities, Methodologies, & Tools Operational Support Pipeline & Infrastructure Process Strategy Metrics & Data 30 2019 BMNT & Steve Blank
  • 31.
    Strategy: The InnovationPipeline Strategy The team responsible for the step needs established goals for the step Leadership needs established goals for each step 31 2019 BMNT & Steve Blank
  • 32.
    Pipeline & Infrastructure:The Innovation Pipeline Pipeline & Infrastructure Process Process • What’s the input and where does it come from? • What’s the output (deliverable product of this step)? Team / Resources • What team and what expertise is needed for the team managing the pipeline? • What about for teams going through the pipeline? • What resources are required (funding, contracts vehicles, facilities) 32 2019 BMNT & Steve Blank
  • 33.
    Operational Support: TheInnovation Pipeline Organizational support • Contracts & contracting officers, Human Resources, Security, access to testing facilities, standing exceptions to policy Operational Support Leadership support • Provide resource & compliance approvals, conduct firefighting, act at key decision points between steps. 33 2019 BMNT & Steve Blank
  • 34.
    Activities, Methodologies, Skills& Tools: The Innovation Pipeline Activities and Methodologies • What methodologies are used to perform this step • Skills required and experiences needed Activities, Methodologies, Skills & Tools Skills and Tools • Skills need to enable execution • Tools applied to collect insight 34 2019 BMNT & Steve Blank
  • 35.
    Metrics & Data:The Innovation Pipeline Metrics are used to measure throughput and quality of output Metrics & Data Vital signs (data) are used to check performance 35 2019 BMNT & Steve Blank
  • 36.
    The Innovation Pipeline:Decision Support Tool 36 2019 BMNT & Steve Blank
  • 37.
    TRL+ IRL+ ARL MeasureReadiness How to Measure Team Progress Through the Innovation Pipeline 37
  • 38.
    Measure Team Progress •Measure Technology Readiness, Investment Readiness and Acquisition/Deployment Readiness • All are data-driven and evidence-based • Each team can track their own progress • Leadership can measure teams progress • Can establish funding and deployment milestones based on TRL/IRL/ARL number 38 2019 BMNT & Steve Blank
  • 39.
    Technology + Investment+ Adoption Readiness Level TRL Problem validation Complete a first-pass canvas Problem/Solution Validation Low Fidelity MVP Experiments Validate Solution/Mission fit Validate right side of canvas Prototype High-Fidelity MVP Validate left-side of canvas Validated funding/deployment IRL ARL 39 2019 BMNT & Steve Blank
  • 40.
    Technology Readiness Level InvestmentReadiness Level Adoption Readiness Level When to Measure Readiness Levels 40©2019 BMNT & Steve Blank
  • 41.
    TRL – Measure TechnologyReadiness How to Measure Team Progress Through the Innovation Pipeline 41
  • 42.
    NASA/DOD Technology Readiness Level(TRL) • Formal Way to assess project maturity • Quantify Relative Risks • Data Driven • Adopted by NASA, DOD, FAA, ESA… 42
  • 43.
  • 44.
    NASA/DOD Technology Readiness: Levels1 & 2 Basic Technology Research • Basic principles observed • Technology concept formulated Concept 44
  • 45.
    NASA/DOD Technology Readiness Levels3 & 4 Research to prove Feasibility • Experimental proof of concept • Breadboard validation in lab Research Concept 45
  • 46.
    NASA/DOD Technology Readiness Levels5 & 6 Demo Prototype • Breadboard validation outside the building • System demo in real-world Research Concept Demo 46
  • 47.
    NASA/DOD Technology Readiness Levels7, 8, 9 Deployment • System Development • System deployed in real-world Research Concept Demo 47
  • 48.
    TRL Measure TeamProgress • TRLs are data-driven and evidence-based • Each team can track their own progress • Leadership can measure teams progress • Can establish funding milestones based on TRL number 48
  • 49.
    IRL – Measure InvestmentReadiness How to Measure Team Progress Through the Innovation Pipeline 49
  • 50.
    Investment Readiness Level •A Formal Way to Quantify Relative Risks • Data Driven • Analog to NASA/DOD Technology Readiness Level 50
  • 51.
    Investment Readiness Level Wecan do the same for new mission-driven ventures 51
  • 52.
    Investment Readiness Level Wecan do the same for mission-driven Ventures Emphasis is on data 52
  • 53.
    Mission Model Canvasdrives the – Investment Readiness Level Mission Model Canvas = hypotheses of how you create and deliver value for the DOD/IC and the warfighter Source: Alexander Osterwalder- Business Model Generation KEY PARTNERS Value Proposition CHANNELS CUSTOMER RELATIONSHIPS CUSTOMER SEGMENTS REVENUE STREAMSCOST STRUCTURE KEY ACTIVITIES KEY RESOURCES Beneficiaries Deployment Buy- in/Support Mission Achievement Value Proposition Activities Resources Partners Costs 53
  • 54.
    Mission Model Canvas Source:Alexander Osterwalder- Business Model Generation KEY PARTNERS Value Proposition CHANNELS CUSTOMER RELATIONSHIPS CUSTOMER SEGMENTS REVENUE STREAMSCOST STRUCTURE KEY ACTIVITIES KEY RESOURCES Beneficiaries Deployment Buy- in/Support Mission Achievement Factors Value Proposition Activities Resources Partners Costs How does the team get “Buy- In” from all the beneficiaries? How will we deploy the product to widespread use? What constitutes a successful deployment? Who are our most important customers? Stakeholders? What are their pains/gains? What job do they want us to get done for them? How are we solving each customers pains/gains? What product/service features match their needs? What key activities do we need to be expert in? What key resources do we need to own or acquire? Financial? Human? Who are our key partners? Suppliers? What are we getting from them? Giving them? What is the Mission Budget/Cost? How will we measure Mission Achievement? 54
  • 55.
    Investment Readiness Level KEY PARTNERS Value Proposition CHANNELS CUSTOMER RELATIONSHIPS CUSTOMER SEGMENTS REVENUESTREAMSCOST STRUCTURE KEY ACTIVITIES KEY RESOURCES • The TRL answers questions about the technology • The IRL answers questions about the Mission Model Canvas in Curation, and Discovery 55
  • 56.
    Investment Readiness Level Problemvalidation Complete a first-pass canvas Problem/Solution Validation Low Fidelity MVP Experiments Validate Solution/Mission fit Validate right side of canvas Prototype High-Fidelity MVP Validate left-side of canvas Validated funding/deployment KEY PARTNERS Value Proposition CHANNELS CUSTOMER RELATIONSHIPS CUSTOMER SEGMENTS REVENUE STREAMSCOST STRUCTURE KEY ACTIVITIES KEY RESOURCES The IRL answers questions about the Mission Model Canvas in Curation, and Discovery 56
  • 57.
    IRL Measure TeamProgress • IRLs are data-driven and evidence-based • Each team can track their own progress • Leadership can measure teams progress • Can establish funding milestones based on IRL number 57
  • 58.
    Technology Readiness Level+ Investment Readiness Level Problem validation Complete a first-pass canvas Problem/Solution Validation Low Fidelity MVP Experiments Validate Solution/Mission fit Validate right side of canvas Prototype High-Fidelity MVP Validate left-side of canvas Validated funding/deployment TRL IRL 58
  • 59.
    ARL – Measure AdoptionReadiness How to Measure Team Progress Through the Innovation Pipeline 59 2019 BMNT & Steve Blank
  • 60.
    ARL - Adoption ReadinessLevel • The TRL answers questions about the technology • The IRL answers questions about the Mission Model Canvas in Curation, Discovery and Incubation • The ARL answers questions about Incubation & Transition 60 2019 BMNT & Steve Blank
  • 61.
    Adoption Readiness Level The ARLanswers questions Incubation about Transition Sponsor $s to figure out solution/ mission fit Sponsor & Problem Owner Commitment to find a solution Teams Funded Funded Pilots Field Trials *Final DOTMLPF Deployment Leadership commitment Final Requirement * DOTMLPF is defined in The Joint Capabilities Integration Development System, or JCIDS Process. The JCIDS process provides a solution space that considers solutions (stated in terms of diplomatic or military organization) involving a simulation of doctrine, organization, training, materiel, leadership and education, personnel, and facilities (DOTMLPF) required to accomplish a mission.[2 61 2019 BMNT & Steve Blank
  • 62.
    ARL Measure Progress toget a solution adopted • ARLs are data-driven and evidence-based • Each team can track their own progress • Leadership can measure teams progress • Can establish funding milestones based on ARL number 62 2019 BMNT & Steve Blank
  • 63.
    TRL+ IRL+ ARL MeasureReadiness How to Measure Team Progress Through the Innovation Pipeline 63 2019 BMNT & Steve Blank
  • 64.
    Technology + Investment+ Adoption Readiness Level TRL Problem validation Complete a first-pass canvas Problem/Solution Validation Low Fidelity MVP Experiments Validate Solution/Mission fit Validate right side of canvas Prototype High-Fidelity MVP Validate left-side of canvas Validated funding/deployment IRL ARL 64 2019 BMNT & Steve Blank
  • 65.
  • 66.
    Strategy - What’s thegoal for the Team - What’s the goal for Leadership Pipeline & Infrastructure Process Process - What’s the input and where does it come from? - What’s the output (deliverable product of this step)? Team / Resources - What team and what expertise is needed? - What resources are required (funding, contracts, exceptions) Operational Support - Type of organizational support required (contracts, HR, Security) - What leadership is required (resource/compliance decisions, firefighting etc) Activities, Methodologies, Tools and Metrics Activities and Methodologies - What methodologies are used to perform this step - Skills required and experiences needed Tools and Skills - Tools applied to collect insight - Skills need to enable execution Data and Assessments - What metrics are used to measure throughput - What vital signs (data) are used to check performance INNOVATION PIPELINE SELF ASSESSMENT KEY 66 2019 BMNT & Steve Blank
  • 67.
    M Indicate missingcomponents Highlight program weaknesses Highlight program strengths Comment Score Indicate score for each sub-category 1 - 5 1 – Dysfunctional or missing component 5 – Exceptional Total Total all scores from subcategories INNOVATION PIPELINE SELF ASSESSMENT KEY Teams’ goals are defined to achieve pipeline stage Appropriate input is available to garner success Team & expertise is available to achieve success Organizational support is available for this stage of the pipeline Activities are being executed to generate output Tools defined to collect insight Metrics are defined to determine success on this stage of the pipeline Leadership’s goal is defined to achieve pipeline stage Output is generated to drive next stage of the pipeline Resources are available to achieve success Leadership available & decisions points identified to have progress on this pipeline stage Methodologies defined/used to guide discovery Training is being provided to develop execution skills Data is collected to measure vital signs for this stage of the pipeline Score Score Score Score Score Score Score Score Score Score Score ScoreScore Score 67 2019 BMNT & Steve Blank
  • 68.
  • 69.
    Sourcing – Strategy:The Innovation Pipeline Strategy • Goal for the Innovation Team: Build a repeatable and scalable sourcing process that ensures there is a fresh pool of ideas, tech, people and problems that is exponentially larger than the number of solutions desired to be deployed (using internal and external resources) • Goal for Agency Leadership: Gain the peripheral vision needed to: • provide early warning that disruptive tech may by accelerating • Build systems to enable the org to rapidly seize opportunities to disrupt our enemies • Retaining the agility to respond to the unknown (at scale) 69 2019 BMNT & Steve Blank
  • 70.
    Sourcing – Process:The Innovation Pipeline • Input to Sourcing: Active internal / external sourcing efforts • Output to Curation: Large pool of people, tech, ideas & problems (pool continuously refreshed) • Innovation team and expertise needed: Active & aggressive Ecosystem Architects with local knowledge, internal and external sourcing programs, great Bus Dev / Connector skills Pipeline & Infrastructure Process Resources required: • Full time & contract ecosystem sourcing team • Physical space • Contracting, financing, legal and security liaisons • $’s for travel, $ to host programs (ie: Hackathons, Sprints) 70 2019 BMNT & Steve Blank
  • 71.
    Sourcing - OperationalSupport: The Innovation Pipeline • Organizational support required: Contracts, HR, Security, engineering and R&D liaisons with authority for sourcing exceptions Operational Support • Leadership support required: Prioritize current & future problems for the innovation team. Provide the support to allow rapid and fast track internal and external sourcing of tech, problems and people by providing parallel processes for security, contracting, HR, finance, policy and engineering.) 71 2019 BMNT & Steve Blank
  • 72.
    Sourcing - Activities,Methodologies, Tools & Metrics: The Innovation Pipeline • Methodologies to perform sourcing: Funnel, Hackathons, Crowd Sourcing, Kickstarters, external partnerships, proxy generation, Outreach Services, Extraction Workshop, H4X • Skills/experiences to perform sourcing: Aggressive connector, team/coalition builder, connection to senior leadership, organizational silo breaker Activities, Methodologies, Skills & Tools • Skills for execution: Aggressive ecosystem development team, H4X, Lean • Tools applied to collect insight: Osterwalder Portfolio Management, innovation pipeline model, input / output ratio / funnel metrics, Problem Sourcing Canvas 72 2019 BMNT & Steve Blank
  • 73.
    Sourcing – Metrics& Data: The Innovation Pipeline Metrics & Data Metrics • # of active internal and external sourcing programs • Ratio of new sources netted each event to those actually engaged in other parts of the pipeline over time (passive vs active connections) • Ratio of entities sourced vs entities accepted into next phase (1-1 ration indicates a weak sourcing mechanism, 4-1 may be more preferable) Data • # and diversity of independent sources (tech, people, problems & ideas) across three horizons of potential innovation • # of active engagements from the pool that remain engaged in follow-on steps (ie; # of innovators and entrepreneurs moving into curation) • # of vetted problems accepted into Curation 73 2019 BMNT & Steve Blank
  • 74.
  • 75.
    Strategy Process Team & Resources Operational Support Activities & Methods Metrics &Data Team’s goals are defined to achieve pipeline stage Leadership’s goal is defined to achieve pipeline stage Appropriate input is available to garner success Output is generated to drive next stage of the pipeline Team & expertise is available to achieve success Resources are available to achieve success Organizational support is available for this stage of the pipeline Leadership available & decisions points identified to have progress on this pipeline stage Activities are being executed to generate output Methodologies defined/used to guide discovery Tools defined to collect insight Training is being provided to develop execution skills Metrics are defined to determine success on this stage of the pipeline Data is collected to measure vital signs for this stage of the pipeline Tools & Skills Score Score Score Score Score Score Score Score Score Score Score Score Score Score *Total Score SOURCE 75 2019 BMNT & Steve Blank
  • 76.
  • 77.
    Curation – Strategy:The Innovation Pipeline Strategy • Goal for the Innovation Team: Build a repeatable and scalable curation and prioritization process that creates a vetted problem pool that is exponentially larger than the number of solutions desired to be deployed • Goal for Agency Leadership: • Continuous insight into emerging problems and the decay of current assets. • Establish organizational momentum for innovation based on a prioritized portfolio (Horizon 1, 2, 3) of current & future problems. 77 2019 BMNT & Steve Blank
  • 78.
    Curation – Process:The Innovation Pipeline • Input from Sourcing: a healthy & diverse pool of ideas, problems, technology and innovators from active internal AND external sourcing programs • Output to Discovery: prioritized portfolio of curated problems. Concept sketch for how the problem presents itself, initial tech & expert ecosystem map, problem sponsor & champion for each problem Pipeline & Infrastructure Process • Innovation team expertise needed: Problem Curation team, technology analysis, ecosystem architects • Resources required: Full time curation team, physical space, contracting, financing, legal and security liaisons. $’s for travel. Organizational OK for curation team to spend time on problems/tech in the pipeline, teams into the pipeline 782019 BMNT & Steve Blank
  • 79.
    Curation - OperationalSupport: The Innovation Pipeline Operational Support • Organizational support required: Contracts, HR, Security, engineering and R&D liaisons with authority for sourcing exceptions • Leadership support required: top-level agreement for fast track sourcing, committed funding, written exceptions for security, contracting, policy, legal for innovation sourcing 79 2019 BMNT & Steve Blank
  • 80.
    Curation - Activities,Methodologies, Skills & Tools: The Innovation Pipeline • Activities supporting curation: Active outreach to find problems and sponsors, Problem Curation Workshop, Technology Exploration • Methodologies to perform curation: Problem Curation, Concept (CONOP) development, MVP development, Mission Model Canvas (MMC), Portfolio map, Investment Readiness Level (IRL) / Technology Readiness Level (TRL) / Adoption Readiness Level (ARL) Activities, Methodologies, Skills & Tools • Skills & experiences to perform curation: Problem Curation, Talking to Humans, Interviewing, Lean, Technology Assessment, Design Thinking, MVP development, Iterative Development • Tools applied to collect insight: Problem curation workbook, petal diagrams, Value Proposition Canvas (VPC), Collaboration Map 802019 BMNT & Steve Blank
  • 81.
    Curation – Metrics& Data: The Innovation Pipeline Metrics & Data Metrics • Ratio of problems / new concepts curated vs those accepted into Discovery (1-1 reflects a weak pool, 4-1 is closer to historical norms from REF and H4D) • Ratio of problems / new concepts spread across Alex O’s portfolio map vs 3 Horizons • Initial TRL / IRL and ARL established • Quality of teams in Discovery - how many meet minimum standard & remain intact Data • # of beneficiary, innovators, entrepreneurs and tech expert interviews conducted to validate problems & ideas • Diversity of innovators, entrepreneurs and experts engaged in validation 81 2019 BMNT & Steve Blank
  • 82.
  • 83.
    Strategy Process Team & Resources Operational Support Activities & Methods Metrics &Data Team’s goals are defined to achieve pipeline stage Leadership’s goal is defined to achieve pipeline stage Appropriate input is available to garner success Output is generated to drive next stage of the pipeline Team & expertise is available to achieve success Resources are available to achieve success Organizational support is available for this stage of the pipeline Leadership available & decisions points identified to have progress on this pipeline stage Activities are being executed to generate output Methodologies defined/used to guide discovery Tools defined to collect insight Training is being provided to develop execution skills Metrics are defined to determine success on this stage of the pipeline Data is collected to measure vital signs for this stage of the pipeline Tools & Skills Score Score Score Score Score Score Score Score Score Score Score Score Score Score *Total Score CURATE 83 2019 BMNT & Steve Blank
  • 84.
  • 85.
    Discovery – Strategy:The Innovation Pipeline • Goal for the Innovation Team: • Build a repeatable and scalable discovery process that validates problems and solution pathways, while also narrowing the potential solution teams to only the most viable for further investment. • Run a Lean/I-Corps model using synchronous teams, MMC, MVPs Strategy Goal for Agency Leadership: Establish a continuous flow of options to: • Accelerate to replace decaying/lost capabilities • Provide options to disrupt emerging protagonists 85 2019 BMNT & Steve Blank
  • 86.
    Discovery – Process:The Innovation Pipeline • Input from Curation: prioritized portfolio of curated problems. For each problem; concept sketch for how the problem presents itself, initial tech & expert ecosystem map, problem sponsor & champion • Output to Incubation: Validated MVPs, test data and a pathway to deliver a solution; teams with the potential to deliver; Early adopters/1st customer ID’d • Innovation Team expertise needed: Lean/I-Corps Instructors, internal entrepreneurs recruited to teams to build agency buy-in & coalition. Pipeline & Infrastructure Process • Expertise for the team in the pipeline: Hypothesis generation, MVP development, MVP testing development, Interviewing techniques, data capture • Resources are required – Full time innovation team, physical space, contracting, financing, legal and security liaisons. $’s for travel. Organizational OK for innovators to spend time on problems/tech in the pipeline, teams into the pipeline. Other orgs bought into 1) fast track process 2) adoption of output 86
  • 87.
    Discovery - OperationalSupport: The Innovation Pipeline • Organizational support required: • Contracts, HR, Security, engineering and R&D liaisons with authority for discovery and MVPs; • Understanding how to “get to Yes” from those groups; • Unfettered access to potential beneficiaries. • Coaches and mentors to support teams in discovery Leadership support required: • Top-level agreement for fast track MVPs; • Funding for teams in the discovery process (travel, prototyping, stipends, pay); w • Written exceptions for security, contracting, policy, legal for innovation sourcing; • Designated/trained organizational POCs in contracting, HR, Resource Management • Firefighting board of organizational leaders capable of reducing barriers for innovation teams Operational Support 87
  • 88.
    Discovery - Activities,Methodologies, Tools & Metrics: The Innovation Pipeline • Activities supporting discovery: Hacking for Defense, I-Corps, Exploration Workshops, Discovery workshop, Product Jam / Experimentation Workshop • Methodologies to perform discovery: Lean StartUp, MVP Development, Mission Model Canvas (MMC), Business Model Canvas (BMC), Portfolio Management, Activities, Methodologies, Skills & Tools • Skills & experiences to perform discovery: Innovation teams need innovator + entrepreneurs; Pipeline leadership team needs to build agency buy-in and coalition; Internal & external entrepreneurs recruited to teams; Prototyping ability • Tools applied to collect insight: Mission Model Canvas (MMC), Business Model Canvas (BMC), Innovation Portfolio Map, Investment Decision Support Tool (IRL, TRL, ARL), Funnel Metrics, Workflow maps, User journey map, HMW statements 88 2019 BMNT & Steve Blank
  • 89.
    Discovery – Metrics& Data: The Innovation Pipeline Metrics & Data Metrics • Ratio of teams entering discovery vs those moving on to incubation (1-1 ratio is flat, historical norm from venture due diligence, H4D and REF is ~4-1) • TRL, IRL and ARL established as part of decision to move forward into incubation • Ratio of teams in discovery spread across Alex O’s portfolio map vs 3 Horizons • Quality of team transitioning to Incubation (focused on left side of MMC) Data • # of hypothesis and MVP’s generated • # of team pivots • # and diversity of interviews conducted • TRL, IRL and ARL at completion of Discovery • # of external partners and internal parts of organization engaged 89
  • 90.
  • 91.
    Strategy Process Team & Resources Operational Support Activities & Methods Metrics &Data Team’s goals are defined to achieve pipeline stage Leadership’s goal is defined to achieve pipeline stage Appropriate input is available to garner success Output is generated to drive next stage of the pipeline Team & expertise is available to achieve success Resources are available to achieve success Organizational support is available for this stage of the pipeline Leadership available & decisions points identified to have progress on this pipeline stage Activities are being executed to generate output Methodologies defined/used to guide discovery Tools defined to collect insight Training is being provided to develop execution skills Metrics are defined to determine success on this stage of the pipeline Data is collected to measure vital signs for this stage of the pipeline Tools & Skills Score Score Score Score Score Score Score Score Score Score Score Score Score Score *Total Score DISCOVER 91 2019 BMNT & Steve Blank
  • 92.
  • 93.
    Incubation –The InnovationPipeline • Goal for the Innovation Team: Build a repeatable and scalable incubation process – establish a supply of investible solutions - matching the number desired to be deployed Strategy • Goal for Agency Leadership: • Awareness and visible support of insertion of new solutions into existing program. • Funding for the teams in the incubator, active pipeline participation (mentors, coaching, etc.), continued support 93 2019 BMNT & Steve Blank
  • 94.
    Incubation – Process:The Innovation Pipeline • Input from Discovery: • Validated MVPs and test data and a pathway to deliver a solution; teams with the potential to deliver; • Early adopters/1st customer ID’d • Output to Transition – Investible entity with validated IRL, TRL and ARL • Innovation Team expertise needed: Domain expert coaches and mentors, Pipeline leadership to build agency buy-in and coalition. Understand the politics of transition Pipeline & Infrastructure Process • Resources Required: • Recruit developers to teams in pipeline, physical space, contracting, financing, legal and security liaisons. • $’s for travel and funding for teams. Organizational OK for innovators to spend full- time on problems/tech in the pipeline. • Other orgs bought into 1) fast track process 2) adoption of output 94
  • 95.
    Incubation - OperationalSupport: The Innovation Pipeline • Organizational support required: Contracts, HR, Security, engineering and R&D liaisons with authority for discovery and MVPs. Understanding how to “get to Yes” from those groups. Coaches and mentors Operational Support • Leadership support required: • Written agreement for how incubation moves to transition from contracting, security, legal, engineering, etc.; • Funding for teams; Written exceptions for security, contracting, policy, legal for innovation sourcing; • Firefighting board of organizational leaders capable of reducing barriers for innovation teams; • Prioritization for testing and evaluation 95 2019 BMNT & Steve Blank
  • 96.
    Incubation - Activities,Methodologies, Skills & Tools: The Innovation Pipeline • Methodologies to perform incubation: MVPs, MMC, active coaching and mentoring, transition plans, scale and deployment plans • Skills/experiences to perform incubation: innovation teams need innovator + entrepreneurs + developers. Pipeline leadership team continue to build agency buy-in and coalition. Recruit developers to teams in pipeline • Skills for execution: Investment due diligence, gov contracts/grants management; prototyping, Agile Development, Scrum, IP assignment, partnership building, Business Development • Tools applied to collect insight: MMC, IRL, TRL, ARL, Funnel metrics Activities, Methodologies, Skills & Tools 96
  • 97.
    Incubation - Activities,Methodologies, Skills & Tools: The Innovation Pipeline Metrics & Data Metrics • Ratio of teams in incubation vs those gaining follow-on investment (for H1 teams maybe 2-1, H2 teams 4-1, H3 teams 10-1) • Change in TRL, IRL and ARL of teams throughout Incubation and at departure to transition • Condensation: # of projects in discovery combined into one effort moving forward to incubation Data • # and ratio of MVP’s moving to prototype development • Size / scale of initial and subsequent user pools gained during incubation • $ committed to team during incubation • # and amount of follow-on investment, grants and gov contracts received • # of other organizations engaged 97
  • 98.
  • 99.
    Strategy Process Team & Resources Operational Support Activities & Methods Metrics &Data Team’s goals are defined to achieve pipeline stage Leadership’s goal is defined to achieve pipeline stage Appropriate input is available to garner success Output is generated to drive next stage of the pipeline Team & expertise is available to achieve success Resources are available to achieve success Organizational support is available for this stage of the pipeline Leadership available & decisions points identified to have progress on this pipeline stage Activities are being executed to generate output Methodologies defined/used to guide discovery Tools defined to collect insight Training is being provided to develop execution skills Metrics are defined to determine success on this stage of the pipeline Data is collected to measure vital signs for this stage of the pipeline Tools & Skills Score Score Score Score Score Score Score Score Score Score Score Score Score Score *Total Score INCUBATE 99 2019 BMNT & Steve Blank
  • 100.
  • 101.
    Transition – Strategy:The Innovation Pipeline • Goal for the Innovation Team: Establish a pathway for off ramping a team from the innovation team to a transition team or program; manage organizational technology refactoring Strategy • Goal for Agency Leadership: • Support of insertion of new solutions into existing program – specifically by providing Funding and headcount for new solution insertion • Recognize and manage organizational technology refactoring • Establish guidance for transition of solutions that fit in the white space between programs (who is responsible for accepting?) 101 2019 BMNT & Steve Blank
  • 102.
    Transition – Process:The Innovation Pipeline • Input from Incubation: Investible entity with validated IRL, TRL and ARL • Output to the agency: A scalable solution that is delivered in time to have an impact • Innovation Team expertise needed: At this step oversight responsibility should move from the the innovation team to a transition team capable of refactoring the solution to deliver at scale Pipeline & Infrastructure Process • Resources Required: • Funds for testing, advanced prototyping and access to facilities; • Funds for training, sustainment for 1 year to support the transition of a prototype into a program of record (POR); • Augment the innovation team with members of the organizational transition team 102 2019 BMNT & Steve Blank
  • 103.
    Transition - OperationalSupport: The Innovation Pipeline • Organizational support required: • Contracts, HR, Security, engineering and R&D liaisons with authority for prototype development, testing & evaluations. • Access to users; Understanding how to “get to Yes” from those groups. Coaches and mentors Operational Support • Leadership support required: • Written agreement for how incubation moves to transition from contracting, security, legal, engineering, etc.; • Funding for teams; Written exceptions for security, contracting, policy, legal for innovation sourcing; • Firefighting board of organizational leaders capable of reducing barriers for scaling teams; • Prioritization for testing, evaluation and forward deployment 103 2019 BMNT & Steve Blank
  • 104.
    Transition - Activities,Methodologies, Skills & Tools: The Innovation Pipeline • Methodologies to perform transition: Refactoring, Organizational Change Management, Sustainment packaging, Training Development, Kaizen for continuous improvement (Toyota’s engineering way) • Skills/experiences to perform transition: Agile, Scrum, Deployment planning, Training development, integration, supply chain management • Skills for execution: Transition management, budgeting, experimentation design, supply chain development, POR management, POM planning • Tools applied to collect insight: User surveys, adoptions surveys, localization tools (solution performance at different units, bases etc) Activities, Methodologies, Skills & Tools 104
  • 105.
    Transition - Activities,Methodologies, Skills & Tools: The Innovation Pipeline Metrics & Data Metrics • Ratio of teams receiving initial investment leaving incubation to those receiving follow-on investment (different ratio for H1, H2 and H3) • Ratio of teams in transition to those accepted in POR or theater sustainment (different ratio for H1, H2, H3) • Average time spent in transition (shorter for H1, longer for H3) Data • Realized beneficiary value (impact on original problem) • IRL, TRL and ARL at transition to POR • Scaled implementation data (user drop off / pain points / patches or modification required) 105
  • 106.
  • 107.
    Strategy Process Team & Resources Operational Support Activities & Methods Metrics &Data Team’s goals are defined to achieve pipeline stage Leadership’s goal is defined to achieve pipeline stage Appropriate input is available to garner success Output is generated to drive next stage of the pipeline Team & expertise is available to achieve success Resources are available to achieve success Organizational support is available for this stage of the pipeline Leadership available & decisions points identified to have progress on this pipeline stage Activities are being executed to generate output Methodologies defined/used to guide discovery Tools defined to collect insight Training is being provided to develop execution skills Metrics are defined to determine success on this stage of the pipeline Data is collected to measure vital signs for this stage of the pipeline Tools & Skills Score Score Score Score Score Score Score Score Score Score Score Score Score Score *Total Score TRANSITION 107 2019 BMNT & Steve Blank
  • 108.
    Final Thoughts: Whyhave an innovation pipeline? The result of a pipeline are solutions that: 1. Continuously replenish lost capabilities 2. Create new capabilities that match current and future threats The purpose of innovation pipeline doctrine is to create the organization / process / budget and people to do so 108 2019 BMNT & Steve Blank
  • 109.
    The Innovation Pipeline:Decision Support Tool 109 2019 BMNT & Steve Blank
  • 110.
    The Innovation Pipeline:Self Assessment View 110©2019 BMNT & Steve Blank
  • 111.
    181 GOV/COM entities engaged 1800 People Trained 1391 Problems Sourced 540 Problems Curated 122 Teams in Discovery 9 Solutions Delivered 28 Solutions in Incubation 1391 Problems 38%23% 23% 32% % moving forward % of original total.6%2%9% US Army REF #’s 2011-2013 840 Problems 360 Projects initiated (42%) 115 Solutions Incubated in theater (31%) 20 Program of Record (17%) H4Di & BMNT 2018 EOY Snapshot 1112019 BMNT & Steve Blank
  • 112.
    The Innovation Pipeline Diagnostics,Metrics and Doctrine 112 2019 BMNT & Steve Blank