Managing Revolutionary Research                    Marc G. Millis                        2009-Feb-24  Management lessons f...
DefinitionEvolutionary Research           Revolutionary Research    Mastery of Known                Pioneering into Unknow...
Revolutionary ResearchPerformance                When Needed?                                                         Poin...
Revolutions, then Evolutions                                                           Reputable        Pioneers          ...
Contrasting Attributes           Pioneers                                  Masters• Highest-Gain benefits sought          ...
Revolutionary Research Challenges• Breaks from the Norm ("Out of the Box")  – Draws attention to what we dont know rather ...
Revolutionary Research             Management Challenges     Visionary                   Credible     Unfamiliar          ...
History Lessons• Organizational Patterns   Foster, Shepherd, Henderson, Utterback,     Miller…• Scientific Revolutions   K...
Organizational PatternsPerformance                                    Point of Diminishing Returns                        ...
Vision Limiting Values• Prior values tailored to Method, not Goals    –   Sail Effectiveness for Steam Ships ?    –   Spec...
Sir Clarkes 3 Laws    Sir Arthur C. Clarke, (1972), Profiles of the Future, Bantam1. When a distinguished but elderly scie...
Reflexive Dismissals“Space travel is utter bilge.”                - Dr. Richard van der Riet Wooley (one year before Sputn...
Reflections on Prior Scientific Revolutions              Dyson                                        Kuhn          1997 I...
Great Researchers & Important Problems1986 lecture, Richard Hamming, distinctions between good and great researchers  •   ...
Applying Clarkes 2nd Law:  The only way of discovering the limits of the  possible is to venture a little way past them in...
Applying Clarkes 2nd Law:   The only way of discovering the limits of the possible is to venture a   little way past them ...
Next Challenge: Fringe Tainting•   30% of projects unsolicited correspondence from    amateurs sharing their ideas•   30% ...
Responding to Amateur / Fringe• Do not reply to correspondence displaying delusions  of grandeur and/or paranoia (Fringe)•...
Compilation of Advice from History• Identify diminishing returns, specifically where  revolutionary research is needed• Lo...
Reactionary versus Strategic     Typical Reactionary Approach                             Strategic Approach•   New hot to...
NASA Breakthrough Propulsion Physics ProjectExceed the fundamental limits ofexisting propulsion by furtheradvancing physic...
Project Operating Principles• Reliable – Define success as gaining reliable knowledge rather than   claiming breakthroughs...
Other "Revolutionary" Efforts  Org        Scope        Avoiding Incumbent          Taking Risks        Funding            ...
Project Activities / Products   • 1996-2002: Small Research Tasks   • 2003-2008: Published Findings      – 10 Approaches  ...
Book: Frontiers of Propulsion Science• AIAA Progress in Aeronautics  and Astronautics Series• Editors:   – Marc Millis (NA...
Project Tactics      Visionary                                       Credible• Define success as gaining reliable knowledg...
Distinguishing Crackpots from Visionaries         Food for thought, or a crackpot filled with half-baked baloney? •   Chec...
Other Crackpot Filters• Carl Sagan Baloney Detection Kit  http://www.carlsagan.com/revamp/carlsagan/baloney.html• John Bae...
Project Tactics, continued     Unfamiliar                   Understandable• Identify the grand challenges  and important p...
Project Tactics, continuedLong-Term Goals                             Near-Term Progress• Break long range goals into imme...
Traceability Map       Research linked to goals and credible foundationsKnowledge            Compare Emerging Knowledge to...
Traceability Map from Book (FPS)         #s = Chapter citations                                                           ...
Project Tactics, continued Divergent Options                       Need to Focus• Prioritization criteria consistent with ...
Research Selection Process (1999)Selection Criteria concurred by key players (1996-97)Reviewers do NOT judge feasibility, ...
Projects Research Evaluation Factors Relevance   1:   Gain – Magnitude of performance improvement, assuming the technology...
Measuring Applied Science Progress                           Technology Readiness Levels                           8      ...
Evaluation Equation• Multiplicative criteria (1 failed criteria fails whole proposal)• Scholastic gradations (A through F)...
Proposal ScoresStandard deviation used to flag disparate scores                                                   38
Traceability Map of Resulting Research                                         39
Project Tactics, continuedSufficient Investment                                     Affordable   • Overall progress made f...
Metrics of Project Value1.   Number of visionary notions converted into research tasks2.   Number of incremental unknowns,...
Revolutionary Research Risks    Reliability and Performance of Information, not Hardware        Risks                     ...
Project Lessons LearnedDID WELL• Addressed a diversified set of relevant research approaches• Produced and published genui...
Summary                                                      Consider the                                                 ...
Upcoming SlideShare
Loading in …5
×

Millis.marc

14,811 views

Published on

Published in: Technology, Education
0 Comments
1 Like
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
14,811
On SlideShare
0
From Embeds
0
Number of Embeds
4
Actions
Shares
0
Downloads
27
Comments
0
Likes
1
Embeds 0
No embeds

No notes for slide

Millis.marc

  1. 1. Managing Revolutionary Research Marc G. Millis 2009-Feb-24 Management lessons from NASAs Breakthrough Propulsion Physics Project are presented. This project investigated such notions as gravity control and faster-than-light travel, assessing 10 approaches, producing 16 journal articles, an award-winning website, and garnering positive media coverage for NASA – all for a total cost of $1.6 M spread over 1996 to 2002. The key tactic was to combine vision with rigor and to apply the lessons from history regarding scientific and technological revolutions. 1
  2. 2. DefinitionEvolutionary Research Revolutionary Research Mastery of Known Pioneering into Unknown 2
  3. 3. Revolutionary ResearchPerformance When Needed? Point of Diminishing Returns Rev Revo Evolutionary olluttio u ion narry ay Resources To exceed the limits of prior technology must seek entirely different technology (Foster, Innovation - The Attacker’s Advantage, 1986) 3
  4. 4. Revolutions, then Evolutions Reputable Pioneers Masters EstablishedBrBre Sophisticated eak ak ffro rommp priio r or r tto o se sea arc rchh ffo or n r ne eww "Breakthrough" Point "Breakthrough" Point Unrefined After which the "Edgy" value is obvious to Not Yet Understood general community 4
  5. 5. Contrasting Attributes Pioneers Masters• Highest-Gain benefits sought • Maintain the knowledge base• Expose ignorance - knowledge • Refine established knowledge gaps • Quantify comparisons, “trades”• Create new knowledge • Procedural progress• Can NOT quantify comparisons • Minimum Risk sought• Intuitive progress Institutions • Must sustain preeminence & reputation • Must assess risk -vs- benefits • Must stay within budgets • Must produce progress 5
  6. 6. Revolutionary Research Challenges• Breaks from the Norm ("Out of the Box") – Draws attention to what we dont know rather than flexing our prowess for the known – Different type work: pioneering rather than mastery – Difficult to assess, quantify benefits• Risky – Most ideas will fail to perform – Evokes hype, sensationalism, fringe – Evokes pedantic disdain – Difficult to sort viable crazy ideas from the fringe – Success will be disruptive 6
  7. 7. Revolutionary Research Management Challenges Visionary Credible Unfamiliar Understandable Long-Term Goals Near-Term Progress Divergent Options Need to FocusSufficient Investment Affordable 7
  8. 8. History Lessons• Organizational Patterns Foster, Shepherd, Henderson, Utterback, Miller…• Scientific Revolutions Kuhn (Paradigms), Dyson (Tool-Driven), Compiled Clarke (3 Laws), Anderson (Horizon), Hamming (Great vs Good Researchers) Advice Emme (Sci-fi/Sci-fact)• Fringe Tainting Langmuir (Pathological Sci), Park (Voodoo Sci), Sagan (Baloney filters), Baez (Crackpot Index), Kruger (Unskilled & Unaware)… 8
  9. 9. Organizational PatternsPerformance Point of Diminishing Returns Mature = Entrenched Revolutionary Revolutionary Resources 9
  10. 10. Vision Limiting Values• Prior values tailored to Method, not Goals – Sail Effectiveness for Steam Ships ? – Specific Impulse for Warp Drive ?• Values tailored to legacy customers, not future – Cold War Prowess -vs- Space Tourism – 1950s Colliers Magazine -vs- • Affordability • Robotics • Environmental monitoring • Asteroid deflection • International collaboration• Revolutionary ideas tend to take root outside the incumbent organizations 10
  11. 11. Sir Clarkes 3 Laws Sir Arthur C. Clarke, (1972), Profiles of the Future, Bantam1. When a distinguished but elderly scientist states that something is possible, he is almost certainly right. When he states that something is impossible, he is very probably wrong.2. The only way of discovering the limits of the possible is to venture a little way past them into the impossible.3. Any sufficiently advanced technology is indistinguishable from magic. 11
  12. 12. Reflexive Dismissals“Space travel is utter bilge.” - Dr. Richard van der Riet Wooley (one year before Sputnik 1)“The secrets of flight will not be mastered within our lifetime, not within a thousand years.” - Wilbur Wright (two years before Kittyhawk)“Heavier than air flying machines are impossible, X-rays are a hoax.” - William Thomson (Lord Kelvin) But don’t forget There were also crazy ideas at the time that were crazy. 12
  13. 13. Reflections on Prior Scientific Revolutions Dyson Kuhn 1997 Imagined Worlds 1962 Structure of Scientific Revolutions "Tool-Driven" "Paradigm Shift"• Telescope → Galileo • Epicycles → Heliocentric (Copernicus)• X-ray diffraction → Watson/Crick • Gravity & Motion → Newtons laws • Michelson-Morley Relativity (Einstein) • Blackbody spectrum • Photoelectric effect Quantum • Wave/Particle dual• COBE & WMAP • Dark Matter• Hubble Space Telescope • Dark Energy• Superconductivity • Quantum Zero Point• Internet networking • Cosmological Const• Nano Engineering • GR/QM incompatibility • Gravity itself 13
  14. 14. Great Researchers & Important Problems1986 lecture, Richard Hamming, distinctions between good and great researchers • Have courage to tackle Important Problems – Grand challenges that will make a real difference, not just "safe" research – Attackable; there is a way to begin solving the problem • Start with independent thoughts and then collaborate • Make steady progress, driven and focused • Learn things beyond own work; "Knowledge is like compound interest" • Redirect what is difficult to something easier (convert liabilities to assets) • Honest with personal flaws & work to overcome • Tolerate ambiguity – Believe in self enough to proceed – Doubt self enough to honestly see flaws 14
  15. 15. Applying Clarkes 2nd Law: The only way of discovering the limits of the possible is to venture a little way past them into the impossible. Via Science Fiction ?Emme, E., ed, (1982) Science Fiction & Space Futures Past & Present, American Astronautical Society History Series • Science Fiction inspires pioneers - definitely ! • Science Fiction is not an accurate predictor • Science Fiction akin to is brainstorming 15
  16. 16. Applying Clarkes 2nd Law: The only way of discovering the limits of the possible is to venture a little way past them into the impossible. Via "Horizon Mission Methodology" Anderson, J. L, , (1996) "Leaps of the Imagination: Interstellar Flight and the Horizon Mission Methodology," JBIS, 49 1. Set Impossible Goals (break from mere extrapolations) 2. Assume it can be done in far future (Sci-fi brainstorming) 3. Look back from future, identifying critical assumptions 4. Identify the critical challenges (contrast knowledge to goals) 5. Identify knowledge gaps ("important questions") 16
  17. 17. Next Challenge: Fringe Tainting• 30% of projects unsolicited correspondence from amateurs sharing their ideas• 30% of the amateur correspondence displayed delusion of grandeur and/or paranoia (Fringe)• Voodoo Science [Park 2000]• "Among the Fringe," S.Weinberger (2006 June 14), <http://www.defensetech.org/archives/002493.html>• "Unskilled and Unaware…" [Kruger 1999] Personality and Social Psychology, V.77, p.121- 17
  18. 18. Responding to Amateur / Fringe• Do not reply to correspondence displaying delusions of grandeur and/or paranoia (Fringe)• When replying to those with nonviable ideas, use statements like, "appears to violate well-established laws of nature," rather than, "violates physics"• Give them a next-step task to help them better understand their concept, rather than trying to teach them the principles…• Make completion of that task a condition for continued correspondence, "to convince us… need to perform… to a higher standard of proof…" 18
  19. 19. Compilation of Advice from History• Identify diminishing returns, specifically where revolutionary research is needed• Look across multiple disciplines and new tools for pioneers and possibilities (seeds of next S-curves)• Employ the "Horizon" method to identify "Important Problems" - converting objections into objectives• Develop more fundamental selection metrics than those used for prior methods (e.g. energy, not specific impulse)• Familiarize the decision-makers with scholarly examples of emerging possibilities and how they apply to goals• Dissect research approaches into short-term next-step research tasks (more affordable, less threatening)• Promise to produce progress, not breakthroughs 19
  20. 20. Reactionary versus Strategic Typical Reactionary Approach Strategic Approach• New hot topic gains attention. • Lead person acts as an impartial broker.• Funds sought for hot topic only. • Various relevant approaches solicited and• Other approaches not comparatively assessed comparatively evaluated. in a rigorous manner. • Selection criteria concurred with customers and researchers. • Scope of each task set to the minimal effort needed to resolve an immediate “go / no-go” question. Typical Results Results Sought• Success is defined in terms of whether the • Success defined as gaining reliable knowledge approach worked. to guide next steps.• Negative results not published. • Results, pro or con, published to set foundation• In the event of a null result, support ebbs. for future decisions.• Window closes on all other approaches for • Opportunity open for sequels to the positive addressing these same challenges. results, and to redirections around null results. 20
  21. 21. NASA Breakthrough Propulsion Physics ProjectExceed the fundamental limits ofexisting propulsion by furtheradvancing physics to discoverthe breakthroughs that couldrevolutionize spaceflight andenable interstellar voyages. Technical Programmatic Target the greatest Conduct visionary challenges of deep- research in a credible spaceflight. manner. 21
  22. 22. Project Operating Principles• Reliable – Define success as gaining reliable knowledge rather than claiming breakthroughs (puts emphasis where needed)• Immediate – Focus on immediate make-or-break issues, unknowns, or curious effects (just enough for “go/no-go”), not the whole thing• Iterated – Gain knowledge via cycles of short-term, incremental tasks• Diversified – Support multiple, divergent research (not just hot topics)• Measured – Track applicability and progress from cycles of research• Impartial – Research selected via competitive peer assessments, where reviewers judge reliability of results, not feasibility of concept• Empirical – Emphasis on experiments over pure theory or studies• Published – Results published, both pro and con 22
  23. 23. Other "Revolutionary" Efforts Org Scope Avoiding Incumbent Taking Risks Funding Limitations ($M/yr)This Emerging - "Impossible" goals - Emphasis on 0.2project physics for - Physics, not Tech reliability, not propulsion - External solicitations claims and in-house work - Publish null resultsNASA Revolutionary Solicitations only "Dont let your 3.4Inst. Adv. space outside NASA preoccupationConcepts systems with reality stifle(NIAC) your imagination"DARPA Defense Rotate staff ≤ 6yr to "Freedom to Fail" 500 avoid entrenchment (space portion) 23
  24. 24. Project Activities / Products • 1996-2002: Small Research Tasks • 2003-2008: Published Findings – 10 Approaches – 16 Journal Articles – Compiled Book 24
  25. 25. Book: Frontiers of Propulsion Science• AIAA Progress in Aeronautics and Astronautics Series• Editors: – Marc Millis (NASA GRC) – Eric Davis (Inst. Adv. Studies, Austin TX)• 18 Authors• 22 Chapters – Gravity control – Faster-than-light – Energy conversion – Project Management• Publication Date: 2009-Feb-2 25
  26. 26. Project Tactics Visionary Credible• Define success as gaining reliable knowledge rather than on achieving a breakthrough• Ask reviewers to judge rigor and credibility, not feasibility (easier to detect a lack of rigor than to assess feasibility)• Edgy physics pursued in aerospace, aerospace goals into physics• Convert objections into objectives (e.g. conservation of momentum - to seeking alternate sources of reaction mass)• Seek credible, published risk-takers• Proposals contingent to prior peer-reviewed publication• Submit progress to peer-reviewed journals – Free peer reviews – More credible publication venue 26
  27. 27. Distinguishing Crackpots from Visionaries Food for thought, or a crackpot filled with half-baked baloney? • Check for self-criticality – Does the author realize the critical make/break issues? – Is a discriminating test suggested? • Check for awareness – Must demonstrate understanding of existing approaches – Must show advantage over existing approaches – Check for legitimate citations of compared approaches • Rigorous on data, playful with interpretations – New idea must be consistent with credible data, but can disagree with existing interpretations of the data (Copernicus analogy) – Check for legitimate citations of data • Check track record – Prior publications – Prior products 27
  28. 28. Other Crackpot Filters• Carl Sagan Baloney Detection Kit http://www.carlsagan.com/revamp/carlsagan/baloney.html• John Baez Crackpot Index http://math.ucr.edu/home/baez/crackpot.html 28
  29. 29. Project Tactics, continued Unfamiliar Understandable• Identify the grand challenges and important problems (Horizon Method)• Publish, publish, publish• Include executive level explanations in technical publications• Be accessible to the Media, teaching, not advocating 29
  30. 30. Project Tactics, continuedLong-Term Goals Near-Term Progress• Break long range goals into immediate questions – Critical Unknowns – Make-Break Issues – Curious Effects• Narrow scope of research – Near-term results (1-3yr task duration) – Only address most critical questions, not the whole system • Less threatening to reflexive challenges • More affordable • More likely to reach completion• Traceability Map to explain linkage of research to goals and credible foundations 30
  31. 31. Traceability Map Research linked to goals and credible foundationsKnowledge Compare Emerging Knowledge to Goals GoalsFoundational Curious Effects, Concepts Grand Physics Unknowns, & Issues & Devices Challenges mpvp = mrvrConservation Laws Quantum Fluctuations Space DrivesQuantum Mechanics 1. Zero Propellant Anomalous Rotation Rates 2. Faster-Than-LightGeneral Relativity (“Dark Matter”) 3. Onboard Energy Warp Drives & Wormholes Anomalous Red-shifts (“Dark Energy”) Cosmology Vacuum Battery ? 31
  32. 32. Traceability Map from Book (FPS) #s = Chapter citations "Important Problems" Rigorous Foundations 22 Goal-Driven VisionsDisciplines & Tools Curiosities Hypotheses & Tests Issues & Unknowns Concepts & Devices Categories Modify gravitationl or Machs Principle inertial scalars 3 Inertia modification Tests of Mach-Lorentz Indigenous reaction matter 3,11 thrusters 3,11,13 3,11 3 Modify quantum vacuum Kinematics 3,4,12,13 MASS: Thrust without propellant or beamed energy Reactive Machian frame Maclay dynamic Casimir Space drive: sails MEMS / NEMS quantum electromagnetic 3,11 effect 12 3,12 Casimir experiments momentum in media 4,12,18 10 Woodward Mach-Lorentz Electrodynamics Reactive quantum vacuum thruster 3,11 inertia as SED vacuum Space drive: fields 3,4,12 Corum DxB & Brito 3 drag 13 Inertia via quantum "EMIM" 10 Quantum vacuum Lifter tests vacuum Biefeld-Brown 8,9 fluctuations 8-9 4,13 Negative matterObservational cosmology 3,4,12,13,15,16,18 Oscillators & gyro propulsion 3,4 Yamishita tests Reactive spacetime antigravity 6 7 3 Yamishita electrogravitics Electromagnetic Cosmic microwave Hathaway tests gravity 7 background Tajmar apparent frame- techniques Superconductivity shield 4,5,7-12 3 3,5 dragging Podkletnov gravity shield 3,5,22 3,5 Superconductors/ Forward g-dipole gravitomagnetics 4 Mechanical techniques 5 Electromagnetic - Levi-Civita effect Anomalous intra-galactic 4 6 gravitational coupling Quantum theory gravity Negative energy Felber gravity-repulsion 3,4,5,7,8,9,11,13 4,15 ("Dark Matter") 3,4 4 Gravitational Spacetime modification / wave propulsion 4,5 gravity control Heim space theory Pinto levitation 4 4 3,4 Computational tools & Existence of negative Anomalous supernovae matter Alzofon antigrav 4 conventions redshifts 3,4 21 Definition of exotic matter Quantum approaches to ("Dark Energy") Heim-Lorentz force & energy conditions 4 gravity control 3,4 Existence of negative 15 4 energy Warp drives & wormholes 4,15 Cosmological constant QED -vs- SED 15 General relativity 13,18 Retrocausal SPEED: faster-than-light 15,18 Conservation laws for Brute fast communication. 16 14 Gravitomagnetic Tachyon hypothesis modifying spacetime 14 15 approximations Forward Casimir battery 18 Spacetime modification for Relativistic No-signal theorems MEG device lifetimes 16 Causality, retrocausal 20 faster-than-light 14 15 paradoxes, & the definition Nachamkin resonant Special relativity Simultaneous linear of time spheres 18 coherence and 14,15,16 Koch voltage fluctuation Quantum nonlocality for Cerenkov radiation entanglement 16 coils 18 faster-than-light 14 Distinguish causal information Nuclear and particle Polchinski nonlinear EPR nonlocality from Ground state suppression 16 superluminal or 18 physics FTL phone 16 retrocausal signaling 16 Cyclic Casimir 18 Quantum entanglement Shoulders EV energy Quantum vacuum energy Leplace-Beltrami ENERGY 16 Degradable quantum tapping 18,20 conversion nonlinear QM in curved 18 Thermodynamics vacuum Potapov water swirl spacetime 16 18 chamber 20 Energetic heavy-water Sonolum energy Sono-luminescence 19 reactions 19 Sono-fusion 19 harvesting 19 Novel nuclear processes 19 32
  33. 33. Project Tactics, continued Divergent Options Need to Focus• Prioritization criteria consistent with goals and an emphasis on the reliability of information• Select a small suite of options, divergent approaches• Progress measured relative to applied science advancements (not technology readiness scale)• Scoring system easily zeroes-out fringe submissions• While individual tasks are near-term, sustained progress gained from iterating cycles of short-term tasks, adapting to the results 33
  34. 34. Research Selection Process (1999)Selection Criteria concurred by key players (1996-97)Reviewers do NOT judge feasibility, instead judge: – Project Relevance – Credibility (reliable results upon which to make future decisions.) – Resources2 - Stage Review Process – Peers numerically grade proposals Minimum of 4 reviews per proposal (for statistics). Multiplicative, mandatory criteria. – Customer team reviews scores to select winnersNASA in-house work subject to same review process 34
  35. 35. Projects Research Evaluation Factors Relevance 1: Gain – Magnitude of performance improvement, assuming the technology ultimately reaches fruition. 2: Empiricism – Tangible effects or just theory? 3: Readiness – The present maturity of the topic/concept under study. 4: Progress – Magnitude of progress to be achieved, as measured by the difference in the readiness now, and the anticipated readiness upon completion of the task. Credibility 5: Foundations – Based on credible references. 6: Contrasts – Compared to current credible competing work. 7: Tests – Leading toward a discriminating test. 8: Results – Probability that the task will result in a reliable foundation for future decisions. Resources 9: Triage – Will it be done anyway or is it unique to this Project? 10: Cost – Funding required (reciprocal scoring factor). 11: Time – Time required to complete task (reciprocal scoring factor). 35
  36. 36. Measuring Applied Science Progress Technology Readiness Levels 8 Flight Qualified 7 Prototype demo in relevant environment 6 Prototype demo 5 Breadboard test in relevant environment 4 Breadboard lab test 3 Proof of concept 2 Application concept formulated 1 Basic principles reportedConjecture Speculation Science Technology Commerce Scientific Method Steps 4 Hypothesis empirically tested 3 Hypothesis proposed Ø 1 2 3 4 Ø 1 2 3 4 Ø 1 2 3 4 2 Data collected 1 Problem formulated General Critical Desired Ø Pre-science: Science Issues Effects • Anomalous effect observed, or • Knowledge gap realized 36
  37. 37. Evaluation Equation• Multiplicative criteria (1 failed criteria fails whole proposal)• Scholastic gradations (A through F) where possible• Team-generated criteriaWhere: – A, B, C represent criteria scores – a, b, c are weighting factors (where 1 is the maximum value, and lower priorities are fractions of 1) – NA, NB, NC are normalizing functions – Cmin is a preset value to prevent the parenthetical term from equaling zero, thereby making criteria C non-mandatory. 37
  38. 38. Proposal ScoresStandard deviation used to flag disparate scores 38
  39. 39. Traceability Map of Resulting Research 39
  40. 40. Project Tactics, continuedSufficient Investment Affordable • Overall progress made from sustaining cycles of short-term tasks and using the findings to affect the next solicitation (can interweave for better continuity) • Support diversified portfolio of approaches (avoid tendency to only support hot topics) Research Research ResearchSolicitation Assess Solicitation Assess Solicitation Findings Findings 40
  41. 41. Metrics of Project Value1. Number of visionary notions converted into research tasks2. Number of incremental unknowns, issues, or curious effects resolved3. Degree of progress per task using Applied Science Progress Scale4. Number of findings published in peer-literature5. Number of citations of published works6. Number of students inspired (can only count those who send comments)7. Number of spin-offs8. Number of educational materials produced 41
  42. 42. Revolutionary Research Risks Reliability and Performance of Information, not Hardware Risks MitigationsCredibility damaged by • Emphasize reliable advances in knowledge, rather thannon-rigorous reporting requiring breakthroughs (the journey, not the destination) • Collaborate with academia and other institutions for peer reviewsLeadership stature • Sustain active scouting for ongoing development insidedamaged by neglecting and external to the organizationrelevant advancements • Pursue visionary research beyond the known - beyond what other organization address • Forge widespread collaborations(For commercial research) • Threshold of attention is when device can be engineeredCompetitive advantage • Disclose only enough for independent verification of keyweakened from premature principles, not device, not best demodisclosure • After independent verification, advertise improved version whose performance is more pronounced than verification demo 42
  43. 43. Project Lessons LearnedDID WELL• Addressed a diversified set of relevant research approaches• Produced and published genuine research progress• Improved the credibility of topic• Cost-effectively accomplished all of the aboveIMPROVEMENT NEEDED• Review opportunities as they emerge• Identify key issues that need the most attention• Link viable sponsors to reliable researchersFOR NEXT TIME• Equal emphasis on in-house research for coverage of unaddressed issues, continuity, and sustaining competence• Explore non-traditional techniques to leverage best of academia, industry, and government – and to inspire and equip future pioneers 43
  44. 44. Summary Consider the possibilities 1. Combine Vision & Credibility Research 2. Identify "Important Problems" rigorously & impartially 3. Use appropriate comparisons (e.g. energy, not specific impulse) 4. Small increments of progress (affordable, less provocative) 5. Emphasis on physical Evaluate rigor and observables impartiality – not 6. Publish reliably, publish often feasibilityMarc G. Millis, 2009 44

×