The document outlines the development of an automated documentation tool (Week 1) and subsequent integration of that tool with a chatbot interface (Week 10). In Week 1, the problem of limited analyst knowledge retention due to lack of documentation is addressed by developing a software tool to automate documentation. In Week 10, the problem of useful tools not being integrated into analyst workflow is addressed by developing a chatbot interface to integrate various software tools and databases. The mission model canvas then provides details on partners, activities, value propositions, resources, and deployment for the Week 10 chatbot integration solution.
2. Will Papper
Product
B.S. SymSys ‘18
Sean Konz
Data Science
B.S. EE & CS ‘20
Holden Gibbons
Strategy/BD
B.S. MS&E ‘19
Anna Zeng
Software
Engineering
B.S. CS ‘18
3. Week 1:
Problem: Analyst
knowledge retention is
limited due to lacking
documentation tools
Solution: Develop a
software tool to automate
analyst workflow
documentation efforts
Week 10:
Problem: Useful tools are
not being used by analysts
because they are not a part
of their workflow
Solution: Develop an AI-
enhanced chatbot interface
to integrate various
software tools and
databases
7. The Librarian
MISSION MODEL CANVAS
Week 1
Mission Budget / Costs
Fixed:
Software development
Integration with government services
Variable:
Onboarding for analysts
System / Database integration
Mission Achievement
Reduce administrative overhead for all analysts regarding
role transitions
Bring new analysts up to speed on shared knowledge
faster
Provide a simple, intuitive experience for easy adoption
Key Partners
Administrators of ispace, rspace,
etc.
Administrators of Intellipedia
Administrators of shared drives
Analysis leadership (i.e. DA4)
across agencies, to encourage
documentation in performance
evaluations
Branch managers in the DA (i.e.
K, our sponsor)
ODNI, to implement our tool
across agencies
Enterprise HR departments, can
help onboard new employees and
aid with exit processes
Key Activities
Research processes to allow for
background documentation
Create & test mocks
Integrate multiple knowledge
sources
Develop a working prototype
Value Proposition
Reduce onboarding time for
new analysts
Improve convenience of
documentation, so analysts are
able to spend less time on
documentation
Consolidate multiple redundant
systems
Speed up analyst transition
time between projects, reducing
administrative burden from HR in
the process
Ensure a clean knowledge
transfer that is less dependent on
contractors
Reduce constant need to
consult former analysts, allowing
for complete focus on new
assignment
Buy-In / Support
Sorely needed by our sponsor;
Analysts can take a year on a 3-5
year project to get up to speed
Sponsor has tried multiple
software solutions without
success
Sponsor’s office has large degree
of autonomy to implement
prospective solutions
Beneficiaries
Primary:
New analysts can get up to
speed on tribal knowledge faster
than before
Transitioning analysts can focus
more on their work by not needing
to allot six months to transition
materials
Current analysts working on
projects enjoy a reduced
documentation load
Secondary:
Leadership can see an overall
picture of key work produced
HR is less burdened by exit
procedures
Non-sponsor Enterprise:
Applicability and potential traction
is being found outside of the
scope of our sponsor
Key Resources
Agency personnel (current and
retired) for documentation
knowledge & in-the-field
experiences
Non-government contacts
experiencing information retention
problems
R&D team contacts
Wide array of industries/job titles
available to contact.
Deployment
Pilot software with one team
within one agency
Expand to similar agencies and
share information based on
clearance level
Integrate with additional
agencies and non-government
organizations
Value Proposition
● Reduce onboarding time
● Improve convenience
● Consolidate
● Speed up analyst transition time
● Ensure a clean knowledge transfer
● Reduce constant need to consult
8. The Librarian
MISSION MODEL CANVAS
Week 1
Mission Budget / Costs
Fixed:
Software development
Integration with government services
Variable:
Onboarding for analysts
System / Database integration
Mission Achievement
Reduce administrative overhead for all analysts regarding
role transitions
Bring new analysts up to speed on shared knowledge
faster
Provide a simple, intuitive experience for easy adoption
Key Partners
Administrators of ispace, rspace,
etc.
Administrators of Intellipedia
Administrators of shared drives
Analysis leadership (i.e. DA4)
across agencies, to encourage
documentation in performance
evaluations
Branch managers in the DA (i.e.
K, our sponsor)
ODNI, to implement our tool
across agencies
Enterprise HR departments, can
help onboard new employees and
aid with exit processes
Key Activities
Research processes to allow for
background documentation
Create & test mocks
Integrate multiple knowledge
sources
Develop a working prototype
Value Proposition
Reduce onboarding time for
new analysts
Improve convenience of
documentation, so analysts are
able to spend less time on
documentation
Consolidate multiple redundant
systems
Speed up analyst transition
time between projects, reducing
administrative burden from HR in
the process
Ensure a clean knowledge
transfer that is less dependent on
contractors
Reduce constant need to
consult former analysts, allowing
for complete focus on new
assignment
Buy-In / Support
Sorely needed by our sponsor;
Analysts can take a year on a 3-5
year project to get up to speed
Sponsor has tried multiple
software solutions without
success
Sponsor’s office has large degree
of autonomy to implement
prospective solutions
Beneficiaries
Primary:
New analysts can get up to
speed on tribal knowledge faster
than before
Transitioning analysts can focus
more on their work by not needing
to allot six months to transition
materials
Current analysts working on
projects enjoy a reduced
documentation load
Secondary:
Leadership can see an overall
picture of key work produced
HR is less burdened by exit
procedures
Non-sponsor Enterprise:
Applicability and potential traction
is being found outside of the
scope of our sponsor
Key Resources
Agency personnel (current and
retired) for documentation
knowledge & in-the-field
experiences
Non-government contacts
experiencing information retention
problems
R&D team contacts
Wide array of industries/job titles
available to contact.
Deployment
Pilot software with one team
within one agency
Expand to similar agencies and
share information based on
clearance level
Integrate with additional
agencies and non-government
organizations
Beneficiaries
Primary:
● New analysts
● Transitioning analysts
● Current analysts
Secondary:
● Leadership
● HR
9. The Librarian
MISSION MODEL CANVAS
Week 1
Mission Budget / Costs
Fixed:
Software development
Integration with government services
Variable:
Onboarding for analysts
System / Database integration
Mission Achievement
Reduce administrative overhead for all analysts regarding
role transitions
Bring new analysts up to speed on shared knowledge
faster
Provide a simple, intuitive experience for easy adoption
Key Partners
Administrators of ispace, rspace,
etc.
Administrators of Intellipedia
Administrators of shared drives
Analysis leadership (i.e. DA4)
across agencies, to encourage
documentation in performance
evaluations
Branch managers in the DA (i.e.
K, our sponsor)
ODNI, to implement our tool
across agencies
Enterprise HR departments, can
help onboard new employees and
aid with exit processes
Key Activities
Research processes to allow for
background documentation
Create & test mocks
Integrate multiple knowledge
sources
Develop a working prototype
Value Proposition
Reduce onboarding time for
new analysts
Improve convenience of
documentation, so analysts are
able to spend less time on
documentation
Consolidate multiple redundant
systems
Speed up analyst transition
time between projects, reducing
administrative burden from HR in
the process
Ensure a clean knowledge
transfer that is less dependent on
contractors
Reduce constant need to
consult former analysts, allowing
for complete focus on new
assignment
Buy-In / Support
Sorely needed by our sponsor;
Analysts can take a year on a 3-5
year project to get up to speed
Sponsor has tried multiple
software solutions without
success
Sponsor’s office has large degree
of autonomy to implement
prospective solutions
Beneficiaries
Primary:
New analysts can get up to
speed on tribal knowledge faster
than before
Transitioning analysts can focus
more on their work by not needing
to allot six months to transition
materials
Current analysts working on
projects enjoy a reduced
documentation load
Secondary:
Leadership can see an overall
picture of key work produced
HR is less burdened by exit
procedures
Non-sponsor Enterprise:
Applicability and potential traction
is being found outside of the
scope of our sponsor
Key Resources
Agency personnel (current and
retired) for documentation
knowledge & in-the-field
experiences
Non-government contacts
experiencing information retention
problems
R&D team contacts
Wide array of industries/job titles
available to contact.
Deployment
Pilot software with one team
within one agency
Expand to similar agencies and
share information based on
clearance level
Integrate with additional
agencies and non-government
organizations
Mission Achievement
● Reduce administrative overhead
● Bring new analysts up to speed faster
● Provide a simple, intuitive user
experience
18. Weeks 1-3 Weeks 4-6 Weeks 7-8 Weeks 9-10
“... it’s not part of your daily
workflow to put your work
somewhere for others to have it.”
- CIA Branch Manager
24. “When can I get it?”
• USAF Operations Director
Weeks 1-3
Weeks 4-6 Weeks 7-8 Weeks 9-10
25. Key Partners
All-Source analysts, particularly
at CIA
Administrators of
Jive/Intellipedia
Administrators of shared drives
Branch managers in the DA to
add collaboration as a metric
Director of the Directorate of
Analysis to issue a top-down rule
regarding collaboration*
Associate Deputy Director and
Deputy Director of the
Directorate of Digital Innovation*
ODNI, to implement our tool
across agencies
In-Q-Tel, potential incubation
and funding opportunities*
Key Activities
Deeply understand analyst
workflow
-Search processes
-Sharing processes
Understand current systems in
use
-Jive
-Intellipedia
-Shared Drives
-Research procurement
practices at sponsor.
Value Proposition
Get up to speed on shared
knowledge quickly
Ease transitions to new projects
by taking documentation with you
Increase ability to accumulate
social clout for skills and
tenure.(being named a librarian or
shares/comments on own page)
See processes used to come to
a conclusion.
Ensure a clean knowledge
transfer that is less dependent on
contractors
Consolidation of databases and
tools within and across agencies
Buy-In / Support
Sorely needed by our sponsor;
Analysts can take a year on a 3-5
year project to get up to speed
Sponsor has tried multiple
software solutions without
success
Sponsor’s office has large degree
of autonomy to implement
prospective solutions
Beneficiaries
Primary:
Incoming analysts can get up to
speed on tribal knowledge faster
than before
Transitioning analysts can focus
more on their work by not needing
to allot six months to transition
materials
Current analysts working on
projects enjoy a reduced
documentation load
Secondary:
Leadership can see an overall
picture of key work produced
IT has fewer databases to
maintain
Non-sponsor Enterprise:
Applicability and potential traction
is being found outside of the
scope of our sponsor
Key Resources
All-source analysts (current and
retired) for workflow and
documentation knowledge
IT Contractors with deep
knowledge of current
databases
Branch managers/leadership to
understand culture
Wide array of industries/job titles
available to contact.
Deployment
Pilot software with one team of
analysts within CIA
Expand to multiple teams to test
collaboration capabilities.
Secure funding from sponsors
budget allocation.
Potential staged development
via In-Q-Tel.*
Roll out collaboration software
across CIA.
The Librarian
MISSION MODEL CANVAS
Mission Budget / Costs
Fixed:
Software development
Integration with government services
Variable:
Onboarding for analysts
System / Database integration
Mission Achievement
Bring new analysts up to speed on shared knowledge faster
Director of the DA: 20% fewer analysts*
Deputy Director and Associate Deputy Director of DDI: Easily access
an interoperable system that brings together multiple data sources*
Security and Legal: Perceive no additional threats to security*
Value Proposition
● Share Knowledge quickly
● Ease transitions
● Accumulate Social Clout
● Understand thought
processes
● Ensure a clean knowledge
transfer
● Consolidate databases &
Tools
26. Key Partners
All-Source analysts, particularly
in IC
Administrators of
Jive/Intellipedia
Administrators of shared drives
Branch managers in the DA to
add collaboration as a metric
Director of the Directorate of
Analysis to issue a top-down rule
regarding collaboration*
Deputy Director & Associate
Deputy Director of the
Directorate of Digital Innovation*
ODNI, to implement our tool
across agencies
In-Q-Tel potential incubation and
funding opportunities
Key Activities
Deeply understand analyst
effectiveness, including Search
& Sharing workflows
Understand current systems in
use, including Jive, Intellipedia,
and Shared Drives
Research procurement
practices at sponsor
Value Proposition
Quickly get up to speed on
shared knowledge
Ease transitions to new projects
by taking documentation with you
Increase ability to accumulate
social clout for skills and tenure
Visibility into what analysts are
working on, as well as processes
used to come to a conclusion
Ensure a clean knowledge
transfer that is less dependent on
contractors
Consolidation of databases and
tools within and across agencies
Buy-In / Support
Sorely needed by our sponsor;
Analysts can take a year on a 3-5
year project to get up to speed
Sponsor has tried multiple
software solutions without
success
Sponsor’s office has large degree
of autonomy to implement
prospective solutions
Beneficiaries
Primary:
Analysts can access shared
databases faster than before
Branch managers can get
visibility into what their analysts
are working on
DA Leadership can have visibility
into metrics on the performance
of the whole organization
Secondary:
Contractors see increased usage
for under-utilized applications
Security/Legal now has a single
audit log, rather than audit logs
across multiple databases
Non-sponsor Enterprise:
Applicability and potential traction
is being found outside of the
scope of our sponsor
Key Resources
All-source analysts (current and
retired) for workflow and
documentation knowledge
IT Contractors with deep
knowledge of current databases
Branch managers and leadership
to understand culture
Wide array of analysts and
related roles in industry available
to contact
Deployment
Pilot software with one team of
analysts within CIA
Expand to multiple teams to test
collaboration capabilities
Secure funding from sponsors
budget allocation
Potential staged development
via In-Q-Tel
Roll out collaboration software
across CIA
The Librarian
MISSION MODEL CANVAS
Week 10
Mission Budget / Costs
Fixed:
Software development
Integration with government services
Variable:
System / Database integration
Mission Achievement
Director of the DA: Requires 20% fewer analysts
All-source Analysts: Get up to speed on new topic areas within one
workday
Deputy Director and Associate Deputy Director of DDI: Easily
access an interoperable system that brings together multiple data
sources
Security and Legal: Perceive no additional threats to security
27.
28.
29. In other words… (3 year)
Option 1. Reduce HR budget,
level productivity.
• Work done before Librarian: 100 units
• Work done after Librarian: 100 units
• HR budget before Librarian: $100
• HR Budget after Librarian: $67 (-33%)
Option 2. Same HR budget,
expanded productivity.
• Work done before Librarian: 100 units
• Work done after Librarian: 150 units (+50%)
• HR budget before Librarian: $100
• HR Budget after Librarian: $100
32. Conversational
Chatbot
Social Nudges
Group Common Folders
Document Tagging
Current Feature
Exploration
Product Feature Diagram
- Personal database
- Group contact database
- Recommended Readings
- Moderated subject Lists
- Database Integrations
- Suggested actions
- Relevant Tool Integration
- Workflow monitoring
33. What’s next?
• WE ARE ALL CONTINUING
• Seeking grants and funding
• Continuing to build-out the Librarian V 1.0
Weeks 1-3
Weeks 4-6 Weeks 7-8 Weeks 9-10
34. Acknowledgements
Sponsor: Thank you for the amazing support!
Mentor: Tom Bedecarre
TA Mentor: Darren Hau
Teaching Team: Steve Blank, Joe Felter, Pete Newell, Steve
Weinstein
TA Team, H4D mentors, everyone we interviewed
37. Key Features
• Social nudges
• Tagging
• Sorting
Weeks 1-3
Weeks 4-6 Weeks 7-8 Weeks 9-10
38. Social Nudges
• Social proof
• Following colleagues
• No need to change incentives
Weeks 1-3
Weeks 4-6 Weeks 7-8 Weeks 9-10
39. Tagging and Sorting
• Make organizational tools front and center
• Combination of push and pull mechanisms
Weeks 1-3
Weeks 4-6 Weeks 7-8 Weeks 9-10
40. What will the Intelligence Community
look like with The Librarian?
• Track down information more quickly than before
• Ensure that time is not wasted at critical moments.
Weeks 1-3
Weeks 4-6 Weeks 7-8 Weeks 9-10
41. What will the world look like with The
Librarian?
• Our ideal is similar to The Librarian in Snow Crash: We
want to allow you to learn from a bot that contains the
sum of your organization’s knowledge.
Weeks 1-3
Weeks 4-6 Weeks 7-8 Weeks 9-10
43. Beneficiary Archetype: Incoming
Analyst
● Always the “new guy”
● Reads heavily for work
● 12 months of training (or more)
● 1 Year of onsite ramp-up time “before
you know what you’re talking about or,
more importantly, [before] anyone
listens.”
44. Beneficiary Archetype: Outgoing
Analyst
25-35 year-old intelligent / high aptitude individual, leaving from a 3-5 year
assignment
● Reads heavily for work
● May be resistant to digital tools
● Current equilibrium: 6 Months of spool-
down, documentation time
47. The Librarian
Knowledge transfer for analysts
Creating a single chatbot interface for the many databases and software platforms
will allow us to improve knowledge transfer for new analysts within the intelligence
community.
Will Papper Sean Konz Anna Zeng Holden
Gibbons
Product / Software Engineering Data Science
Software
Engineering Strategy
48. The Librarian Team
Will Papper Sean Konz Anna Zeng Holden Gibbons
Academic Program
B.S. SymSys 2018 B.S. EE & CS 2020 B.S. CS 2018 B.S. MS&E 2019
LinkedIn
https://www.linkedin.com/in/wpapper/
https://www.linkedin.com/in/sean-
k-19a327b2/?locale=
https://www.linkedin.com/in/ann
a-z-11774096
https://www.linkedin.com/in/j-holden-
gibbons-669296a3/
Subject Matter Expert?
Details available
No
(Computer Vision /
Machine Learning)
No
(Software Engineering)
Yes, Veteran
Role
Product Data Science Hacker Hustler
Experience Solving a
Problem that Seemed
Impossible
- In one month, developed a
proof-of-concept social
network for a P2P mesh
networking platform
- Worked with multiple federal
agencies
- Developed Novel
approach to eye tracking
interfacing in a populated
consumer space
- Parallel processing
experience
- Pivoted the direction of
three Microsoft products
- Coordinated the first EE
makerspace in Stanford
- Founded non-profit urban
farming initiative in the inner
city and a small real estate
development firm.
50. What will the world look like with The
Librarian?
• As an analyst or researcher, you had to pull from dozens of
databases (both internal and external). You would not necessarily
know where to look for information, and could miss vital info.
• With The Librarian, you can query every database simultaneously
and never miss important information.
52. Beneficiary Archetype: Incoming Analyst
22-30 year-old intelligent / high aptitude individual, coming for a 3-5 year assignment
● Values mission recognition over pay incentives
● Strong sense of patriotism/service
● Capable of being highly interchangeable
● Often operate as ‘snowflakes’, covering down
on issues as need be
● Reads heavily for work
● 12 months of training (or more)
● 1 Year of onsite ramp-up time “before you
know what you’re talking about or, more
importantly, [before] anyone listens.”
53. Beneficiary Archetype: Outgoing Analyst
25-35 year-old intelligent / high aptitude individual, leaving from a 3-5 year assignment
● Values mission recognition over pay incentives
● Strong sense of patriotism/service
● Capable of being highly interchangeable
● Often operate as ‘snowflakes’, covering down
on issues as need be
● Reads heavily for work
● May be resistant to digital tools
● Current equilibrium: 6 Months of spool-
down, documentation time
54. Beneficiary Archetype: Agency Leadership
30-45 year-old intelligent / high aptitude individual
● Strong sense of patriotism/service
○ Sacrificed family time & higher income
elsewhere
● Served multiple rotations. some as analysts
● Has a good sense of the agency’s ecosphere
of digital tools
● Perceives organizational culture & limitations
● Mission-focused, but aware of the need for
tools to improve mission effectiveness
55. Customer Workflow: Life of an All-Source Analyst
NEW
ANALYST
(0-1 year)
OLD
ANALYST
(3+ years)
read
write
review
publish
document
analyst
assigned to
a new area
of expertise
analyst given
history book
and loads of
background
reading
revise
scratch work in i-space
or equivalent
executive briefing
Summary reports
Intellipediai-space or eq.
Reports & briefings
raw intel reports
boolean search feed
Project
Lifecycle
56. Customer Workflow: Life of an All-Source Analyst
NEW
ANALYST
(0-1 year)
OLD
ANALYST
(3+ years)
read
write
review
publish
document
analyst
assigned to
a new area
of expertise
analyst given
history book
and loads of
background
reading
revise
scratch work in i-space
or equivalent
executive briefing
Summary reports
Intellipediai-space or eq.
Reports & briefings
raw intel reports
boolean search feed
Project
Lifecycle
AUTOMATE
57. Customer Workflow: Life of an All-Source Analyst
NEW
ANALYST
(0-1 year)
OLD
ANALYST
(3+ years)
read
write
review
publish
&
document
analyst
assigned to
a new area
of expertise
analyst given
history book
and loads of
background
reading
revise
scratch work in i-space
or equivalent
executive briefing
Summary reports
Intellipedia
i-space or eq.
Reports & briefings
raw intel reports
boolean search feed
Project
Lifecycle
60. Directorate office
creates
requirements
Information Technology
Enterprises (in DDI)
searches for solutions
If contractors are
addressing the
requirements, they
are looped in
If an existing solution
cannot be repurposed,
ITE looks in industry
LARGE COMPANY:
Digital Futures brings it
into the internal network
STARTUP:
In-Q-Tel provides
milestone-based funding
62. Information Technology
Enterprises (in DDI)
searches for solutions
If contractors are
addressing the
requirements, they
are looped in
If an existing solution
cannot be repurposed,
ITE looks in industry
LARGE COMPANY:
Digital Futures brings it
into the internal network
STARTUP:
In-Q-Tel provides
milestone-based funding
Directorate leadership
requests solution from
DDI leadership
63. Directorate leadership
requests solution from
DDI leadership
DDI leadership & ITE
create requirements
If contractors are
addressing the
requirements, they
are looped in
If an existing solution
cannot be repurposed,
ITE looks in industry
LARGE COMPANY:
Digital Futures brings it
into the internal network
STARTUP:
DDI leadership
recommends In-Q-Tel
evaluate product
64. Directorate leadership
requests solution from
DDI leadership
DDI leadership & ITE
create requirements
If contractors are
addressing the
requirements, they
are looped in
If an existing solution
cannot be repurposed,
ITE looks in industry
LARGE COMPANY:
Digital Futures brings it
into the internal network
STARTUP:
DDI leadership
recommends In-Q-Tel
evaluate product
Implementation & beta
testing through ITE;
DDI leadership removes
roadblocks
66. Key Partners
All-Source analysts, particularly
in IC
Administrators of
Jive/Intellipedia
Administrators of shared drives
Branch managers in the DA to
add collaboration as a metric
Director of the Directorate of
Analysis to issue a top-down rule
regarding collaboration*
Deputy Director & Associate
Deputy Director of the
Directorate of Digital Innovation*
ODNI, to implement our tool
across agencies
In-Q-Tel potential incubation and
funding opportunities
Key Activities
Deeply understand analyst
effectiveness, including Search
& Sharing workflows
Understand current systems in
use, including Jive, Intellipedia,
and Shared Drives
Research procurement
practices at sponsor
Value Proposition
Quickly get up to speed on
shared knowledge
Ease transitions to new projects
by taking documentation with you
Increase ability to accumulate
social clout for skills and tenure
Visibility into what analysts are
working on, as well as processes
used to come to a conclusion
Ensure a clean knowledge
transfer that is less dependent on
contractors
Consolidation of databases and
tools within and across agencies
Buy-In / Support
Sorely needed by our sponsor;
Analysts can take a year on a 3-5
year project to get up to speed
Sponsor has tried multiple
software solutions without
success
Sponsor’s office has large degree
of autonomy to implement
prospective solutions
Beneficiaries
Primary:
Analysts can access shared
databases faster than before
Branch managers can get
visibility into what their analysts
are working on
DA Leadership can have visibility
into metrics on the performance
of the whole organization
Secondary:
Contractors see increased usage
for under-utilized applications
Security/Legal now has a single
audit log, rather than audit logs
across multiple databases
Non-sponsor Enterprise:
Applicability and potential traction
is being found outside of the
scope of our sponsor
Key Resources
All-source analysts (current and
retired) for workflow and
documentation knowledge
IT Contractors with deep
knowledge of current databases
Branch managers and leadership
to understand culture
Wide array of analysts and
related roles in industry available
to contact
Deployment
Pilot software with one team of
analysts within CIA
Expand to multiple teams to test
collaboration capabilities
Secure funding from sponsors
budget allocation
Potential staged development
via In-Q-Tel
Roll out collaboration software
across CIA
The Librarian
MISSION MODEL CANVAS
Mission Budget / Costs
Fixed:
Software development
Integration with government services
Variable:
System / Database integration
Mission Achievement
Director of the DA: Requires 20% fewer analysts
All-source Analysts: Get up to speed on new topic areas
within one workday
Deputy Director and Associate Deputy Director of DDI:
Easily access an interoperable system that brings together
multiple data sources
Security and Legal: Perceive no additional threats to
security
67. The Librarian: Value Proposition Canvas
Products
& Services
Knowledge
management for
analysts
Bring past documentation and
work with you to new projects
Tool to reference past work and
recommendations from former
analysts (no need to call) Customer
Jobs
Get up to speed
on a new body
of knowledge
Analysts lose access to prior
work when switching roles
Existing documentation tools
are cumbersome and scattered
across systems
Incoming All-Source Analysts
Gains
Gain
Creators
Pain
Relievers
Locate documentation and
past work in one place
Track down knowledge of
analysts who have retired or
changed projects without
needing to call
Put prior work into an IC-wide
system rather than team
shared drives or email accounts
Documentation is handled by
one software solution rather
than multiple tools with
different processes
Pains
68. The Librarian: Value Proposition Canvas
Products
& Services
Knowledge
management for
analysts
Documentation is handled
transparently in the background
Creates a clear list of materials
to be transitioned
Customer
Jobs
Pass along
knowledge to
remaining team
members
No established procedures for
transitioning out
Two weeks is not enough time
to get materials together
Process is chaotic, informal, and
ad-hoc
Outgoing All-Source Analysts
Gains
Gain
Creators
Pain
Relievers
Documentation is handled
in the background over the
course of a project,
reducing transition time
Adds formal processes to
transition materials
Move documentation from an
active burden into a
background process
Documentation is handled by
one software solution rather
than multiple tools with
different processes
Pains
69. The Librarian: Value Proposition Canvas
Products
& Services
Knowledge
management for
analysts
Research processes are stored
in an organized application
rather than shared drives
Branch manager interface:
present important metrics on
analyst activities
Customer
Jobs
Synthesize
documents from
analysts for
decision-making
Finished analyses are visible, but
for controversial claims it is
difficult to see a thought process
Focus on publications creates a
lack of insight into daily work
Branch Managers
Gains
Gain
Creators
Pain
Relievers
See what documents went into a
finished product
Gain insights for what analysts are
doing (beyond publications)
Help individual analysts based on
actions
Report’s interpretation of
source material is indicated
clearly
Documentation is handled by
one software solution rather
than multiple tools with
different processes
Pains
70. The Librarian: Value Proposition Canvas
Products
& Services
Knowledge
management for
analysts
Librarian helps provide metrics
across the entire organization,
when there was previously no
visibility
Customer
Jobs
Ensure that
analytic
resources are
directed most
efficiently
Too many offices to keep track
of in one spot
Need to get information from
briefings, rarely able to dive
deeper
Director of the Directorate of Analysis
Gains
Gain
Creators
Pain
Relievers
Needs to direct analysts to
topic areas focused on by
policymakers
Dashboard allows Director to
monitor high-level behavior of
offices within the organization
Can independently browse
dashboard, no need to rely on
branch managers completely
Pains
71. The Librarian: Value Proposition Canvas
Products
& Services
Knowledge
management for
analysts
Contractors employed by org to
serve as implementation
coordinators = $$
Integration with our system for
analyst use Customer
Jobs
Deploy solutions
that meet
requirementsHave multiple systems that they need to
integrate with, which slows down
progress
Already have systems that they own
solving similar problems, and are
protective of their turf
Contractors might not want to put in the
effort to build APIs/help us integrate
Contractors
Gains
Gain
Creators
Pain
Relievers
Benefit from contracts
due to service as
implementation liaisons
Increased contractor tool
use through system
integrations
Managing a single profile could
make integration with other
software easier
If APIs present, we can
integrate without them
Pains
72. The Librarian: Value Proposition Canvas
Products
& Services
Knowledge
management for
analysts
Security Interface
Analyst action signaling
Customer
Jobs
Keep information
from getting
compromisedHuge database of how analysts
think/their current progress that
is a giant target for foreign
intelligence services
Another system to worry about
secure access
Security / Legal
Gains
Gain
Creators
Pain
Relievers
Immediate insight into the
actions of analysts using
signaling mechanisms
Procedure is already in place to
implement this, since this type
of database; precedent exists
Database APIs already exist
We don’t need data copies, just
links
Pains
73. Cultural Resistance
• Workflow
• If something isn’t in the day-to-day expectations, it won’t be used
• Need-to-know
• Your information is yours, and you shouldn’t share it
• Silos
• Incentives
74. Incentives: How are analysts promoted?
• Analysts are evaluated on papers produced and the decisions
that these papers affect
• Collaboration is not a great help in the promotion process, so it
is not common practice
• Who can change this?
• Branch Managers: “If it’s not shared, I’m not counting it.”
• Directorate of Analysis Leadership: Top-down requirement
77. MVP - Analysts
We are introducing a chatbot interface to connect new all-source
analysts with the information they need from existing databases in
the intelligence community.
Analysts will be able to reach crucial background information, identify
subject matter experts, retrieve existing reports, etc. in no time at all.
78. MVP - Branch Managers
Currently, branch managers do not have much insight into reports
that are in progress. Documents need to actively be moved into
shared drives, which only provides very granular insight.
By reporting aggregate queries from analysts to branch managers,
branch managers can most efficiently help analysts.
81. GET KEEP GROW
GET
Awareness:
- MVP testing
- Present underutilized tools
- Integrate with org chat client
Interest:
- More robust document discovery
- Ease of contacting SMEs
Consideration:
- How integration impacts workflow
- If discovering different document
pools (recommendations) would
really be beneficial
Purchase:
- Procurement process
KEEP
- Quantify usage of databases
with/without tool
- Additional tool integrations based
on user desires
- Leadership/user feedback provided
on end intel product quality
- Ensure budget for system
upgrades (re: compatibility)
- Tool should learn over time
- Better suggestions, relevant
documents based on
suggestion doc usage
GROW
Referrals:
- Introduce the tool into other
agencies (other IC, DoD, DIUx)
- Dual-use applications: additional AI
training opportunities
Cross-Sell:
- Develop machine learning to the
point of making suggestions
Next-Sell:
- Add additional internal tool
integrations
Up-Sell:
- Introduce cross-agency information
exchange / integration
82. Mission Achievement - Product
• Do people come back to this tool after trying it the first two times?
(Focus is on abandonment or lack thereof)
• How many times do people consult the tool per day?
• Does this tool reduce the number of times analysts access other
databases?
• How many database integrations do we have?
• Is the tool getting smarter?
83. Mission Achievement - Stakeholders
Party Benefit
Director of the Directorate of Analysis Requires 20% fewer analysts
All-source Analysts Get up to speed on new topic areas
within one workday, increase breadth of
relevant information presented to
analysts
Deputy Director and Associate Deputy
Director of DDI
Easily access an interoperable system
that brings together multiple data
sources
Security and Legal Perceive no additional threats to safety
84. Work Distribution
Will - Product: Ensure that the MVP we’re building matches
customer needs, handle User Experience
Sean - Data Science: Data Science on Database metrics,
NLP framework
Anna - Hacker: Database integrations, Software Engineering,
User Experience
Holden - Hustler: Find new customers and channels, maintain
contact with current beneficiaries
85. Next Steps
• Start building a chatbot!
• Integrate with an open source database API
• Begin to train the chatbot on open source data
• Test interactive demo with analysts
• Test dashboard with branch managers
86. We want to hear from you!
Email:
thelibrarian@lists.stanford.edu
87.
88.
89.
90.
91.
92.
93.
94.
95.
96.
97.
98.
99.
100.
101.
102.
103.
104.
105. Buy-In and Support
Analyst: Easier to get up to speed on new topics, front-ends are
consolidated to allow for simple querying.
Branch Managers: Stay current on the topics that analysts are
working on. Have more insight into ways to provide help.
Contractors: Improve usage of applications that may not fit into an
analyst’s workflow, increasing chances of renewal.
Security: Easily detect anomalous activity, since the activity will be in
one central audit log.
106. Expectation
Chatbot interface is new for
employees and will require a
shift in mindset.
Result
Asking other analysts questions
over IM is very common.
Analysts can now bother a bot
instead of bothering their
colleagues.
107. Expectation
The Librarian would be useful
for easily constructing lists of
tags
Result
There is a database for this, but
it is not used much. The
Librarian could act as a front-
end.
108. Expectation
There are a few crucial
databases to pull from, and
people consult those heavily.
Result
The Library has hundreds of
databases, but they are
underutilized. We can create an
interface for many databases
that are not pulled from.
109. Expectation
Lists of relevant documents do
not exist. The Librarian could
create these lists seamlessly.
Result
There are lists of documents
that exist in NGT, but analysts
do not realize that this feature
exists. Furthermore, it is not in
the workflow that analysts
follow.
We can add a front-end for
these lists, allowing them to be
queried easily.
111. GET KEEP GROW
GET
Awareness:
- MVP testing
- Present underutilized tools
- Integrate with org chat client
Interest:
- More robust document discovery
- Ease of contacting SMEs
Consideration:
- How integration impacts workflow
- If discovering different document
pools (recommendations) would
really be beneficial
Purchase:
- Procurement process (Appendix)
KEEP
- Quantify usage of databases
with/without tool
- Additional tool integrations based
on user desires
- Leadership/user feedback provided
on end intel product quality
- Ensure budget for system
upgrades (re: compatibility)
- Tool should learn over time
- Better suggestions, relevant
documents based on
suggestion doc usage
GROW
Referrals:
- Introduce the tool into other
agencies (other IC, DoD, DIUx)
- Dual-use applications: additional AI
training opportunities
Cross-Sell:
- Develop machine learning to the
point of making suggestions
Next-Sell:
- Add additional internal tool
integrations
Up-Sell:
- Introduce cross-agency information
exchange / integration
112. Mission Achievement - Product
• Do people come back to this tool after trying it the first two times?
(Focus is on abandonment or lack thereof)
• How many times do people consult the tool per day?
• Does this tool reduce the number of times analysts access other
databases?
• How many database integrations do we have?
• Is the tool getting smarter?
113. Mission Achievement - Stakeholders
Party Effect
Director of the Directorate of Analysis Requires 20% fewer analysts
All-source Analysts Get up to speed on new topic areas
within one workday, increase breadth of
relevant information presented to
analysts
Deputy Director and Associate Deputy
Director of DDI
Easily access an interoperable system
that brings together multiple data
sources
Security and Legal Perceive no additional threats to safety
114. Next Steps
• Interview more analysts to understand different types of workflows
• Drill down on more specific use cases for the chatbot
• Understand which areas have the biggest efficiency improvements
• Understand which database APIs exist already
• Understand the work required to add new database integrations
115. Beneficiary Archetype: Incoming Analyst
Male/Female:22-30 years old
-Intelligent/high aptitude individuals.
-Values mission recognition over pay incentives.
-Strong sense of patriotism/service.
-Though capable of being highly
interchangeable, often operate as ‘snowflakes’,
covering down on issues as need be.
-Reads heavily for work.
-12 month training but can be longer with follow
along training opportunities.
- ~3-5 year assignments.
-1 Year of spool-up time. “Before you know
what you’re talking about or more
importantly, anyone listens”. -K
116. Beneficiary Archetype: Outgoing Analysts
Male/Female:25-35
-Intelligent/high aptitude individuals.
-Values mission recognition over pay incentives.
-Strong sense of patriotism/service.
-Though capable of being highly interchangeable,
often operate as ‘snowflakes’, covering down on
issues as need be.
-Reads heavily for work.
-12 month training but can be longer with follow
along training opportunities.
- ~3-5 year assignments.
-May or may not be hoarder/e-allergic.
-Current equilibrium: 6 Months of spool-down,
documentation time.
117. The Librarian: Value Proposition Canvas
Products
& Services
Knowledge
management for
analysts
- Key documents used by an
analyst are included in a
single system
-Research processes are
stored in an organized
application rather than shared
drives Customer
Jobs
Synthesize
documents from
analysts for
decision-making
-Finished analyses are visible,
but for controversial claims it is
difficult to see a thought process
-Focus on publications creates a
lack of insight into daily work
Branch Managers
Gains
Pains
Gain
Creators
Pain
Relievers
- See what documents went into a
finished product
- Get more insight into what analysts
are doing (beyond publications)
- Report’s interpretation of
source material is indicated
clearly
-Documentation is handled by
one software solution rather
than multiple tools with
different processes
118. Beneficiary Archetype: Team/Mis. Cntr. Managers
Male/Female: 28-40
-Responsible for bringing new analysts up to
speed.
-Intelligent/high aptitude individuals.
-Strong sense of patriotism/service manifesting in
potential sacrificed family time/higher income
elsewhere.
-Most likely served multiple mission centers 3+
years each as an analyst.
-Possesses good sense of what the ecosphere of
digital tools consists of within the agency.
-Good sense of organizational culture/limitations.
-Mission focused on day-to-day but aware of the
need for tools to be revamped to improve mission
effectiveness.
119. The Librarian: Value Proposition Canvas
Products
& Services
Knowledge
management for
analysts
Librarian lists help maintain
information, contacts,
glossary, prominent people
and thus reduce time spent
finding intel and more time
producing it.
Customer
Jobs
Manage cadre of
analysts and their
tools.
-Doesn’t have enough analysts
or at least the current ones
aren’t efficient enough due to
knowledge being lost at lower
levels creating redundant and
less than fully informed
conclusions.. Budget and
manpower she or he is afforded
is outside of their control. .
Director of the Directorate of
Analysis
Gains
Pains
Gain
Creators
Pain
Relievers
Able to either dig deeper in current
areas or potentially spread current
labor footprint into more/nuanced
issues for better understanding of the
world.
Automated systems for
analysts allow for less
variation in human behavior
leading to more uniformity
and more efficiency in
analysis.
120. Beneficiary Archetype: Director of Directorate of Analysis
-Female/Male: 40-50
-Personally responsible for allocation of analytical
manpower across array of needs.
-Ultimately responsible to Director of the agency for quality
of analytical product.
-Most likely served in several mission centers and well
versed on past and current culture. (not much changes)
-Has the authority to mandate workflow procedures such
as software utilization.
-Highly motivated and often has best in mind but has many
boiling pots to tend.
121. The Librarian: Value Proposition Canvas
Products
& Services
Knowledge
management for
analysts
-Single search interface would
reduce proliferation of
databases
Customer
Jobs
Implement new
tools that are
relevant to the
Agency
-Do not have proper API
integrations for existing
databases
Deputy Director and
Associate Deputy Director of
DDI
Gains
Pains
Gain
Creators
Pain
Relievers
-Would be able to consolidate multiple
databases and eliminate failed
solutions
-Focus on interoperability
could reduce cost to add new
solutions on top of our system
122. The Librarian: Value Proposition Canvas
Products
& Services
Knowledge
management for
analysts
-Contractors employed by org
to serve as implementation
coordinators = $$
Customer
Jobs
Deploy solutions
that meet
requirements
-Have multiple systems that
they need to integrate with,
which slows down progress
-Already have systems that they
own solving similar problems,
and are protective of their turf
Contractors
Gains
Pains
Gain
Creators
Pain
Relievers
-Benefit from contracts due to service
as implementation liaisons
-A single profile could make
integration with other
software easier
123. The Librarian: Value Proposition Canvas
Products
& Services
Knowledge
management for
analysts
Customer
Jobs
Keep information
from getting
compromised
-Huge database of how analysts
think/their current progress that
is a giant target for foreign
intelligence services
Security/Legal
Gains
Pains
Gain
Creators
Pain
Relievers
-Procedure is already in place
to implement this, since this
type of database. Precedent
exists.
124. Day in the Life
Workflow of our primary beneficiaries (analysts)
7:30AM- Boolean Search yielding 100s or 1000s of hits.
8:00AM- Triage/speed-read for use in daily brief,
construct notes in Word.
8:30AM- Team daily brief, trending items, taskings, etc.
9-Noon- Follow up with more detail.
Where our product fits in
125. Cultural Resistance
• Workflow
• If something isn’t in the day-to-day expectations, it
won’t be used
• Need-to-know
• Your information is yours, and you shouldn’t share
it
126. How do people get promoted?
• Analysts are measured on the papers you produce and
the decisions that your papers affect
• Collaboration is not part of the promotion process, so
it is not done
• Who can do this?
• Branch Managers - “If it’s not shared, I’m not
counting it.”
• Directorate of Analysis Leadership - Top-down rule
127. “We got an automated water cooler… [but no one] is
responsible for maintaining it. It’s about
accountability.”
- Chris, Pathfinder PM, on Intellipedia
… what if we made documentation part of an
analyst’s performance review?
129. Procurement Life cycle
1.Market/internal research for tools serving a given purpose
a.Contractors generally involved in conducting this process
2. Budget analysis: Is it cheaper to build a solution, or to buy? Which is
more efficient?
3.If buying, Numerous routes: In-Q-Tel, Digital Futures within DDI
130. How Procurement Works in Theory (6-12mo)
1. An office within a directorate, such as the Office of Advanced Analytics
within the Directorate of Analysis, produces a set of requirements.
2. Information Technology Enterprises within the Directorate of Digital
Innovation searches for solutions to meet those requirements. If the
requirements are currently being addressed in some form by a
contractor already, the contractor is brought into the loop as well.
3. If an existing solution cannot be modified, then a commercial solution is
131. How Procurement Works in Theory (6-12mo)
4. If the commercial solution is from a large company, Digital Futures within
the Directorate of Digital Innovation will work to bring the company onto the
internal network.
5. If the commercial solution is from a startup, In-Q-Tel will provide
milestone-based funding to help the startup get the solution onto the
internal network. To beta test the software, a work program is initiated that
acts as a limited rollout of the software.
132. How Procurement Works in Practice (3mo)
1. Someone high up in a relevant office (for example, the Office of
Advanced Analytics within the Directorate of Analysis) expresses
strong support for a solution to someone high up in the Directorate of
Digital Innovation (Associate Deputy Director or Deputy Director)
2. Deputy Director of DDI works with Information Technology Enterprises
to determine requirements. May also recommend that In-Q-Tel
evaluate the solution if the company is a startup.
133. Procurement Life cycle con’t
Procurement through DDI:
1.Team hired by supplier to handle implementation
a.Government team holds necessary clearances, experience etc.
b.Atypical: Government creates a team specifically for implementation of
particular tools
2.Requirements set by IC (in our case, DDI)
3.Implementation team works with both org and supplier to meet
requirements
a.Will ask supplier to alter tool slightly to fit security requirements
b.Project sponsors in leadership make the connections to push
134. Procurement Life cycle con’t
Procurement through In-Q-Tel:
Younger companies in need of seed funding targeted
1.Companies with a desired tool are found through market research
process
2.Companies are provided funding tied to milestones. Milestones
help them progress toward getting their software ready for an IC
environment.
3.When the product is ready for use on classified networks, a work
program is initiated to beta test the software.
135. Scope and implementation takeaways.
-Comparable to mid-sized company in size of workforce that will be
utilizing our product
-Can be most easily implemented via one ‘top down’ directive from
Director of Directorate of Analysis.
-Can also be implemented by individual buy-in of all lower level
managers. This is not a favorable path.
136. Key Takeaways
“[Having leadership support] is probably one of the top five
things you need to be successful in getting a solution through”
- Interviewee 7, Integration Contractor
Having org leadership invested in the solution is a key to successful integration
137. Key Takeaways
“Our Government Chief knew that this was something that was
needed, so she took it upon herself to make it happen”
-Interviewee 7, Integration Contractor
If a solution is needed, leadership will recognize it is needed, and will collaborate
to make it a reality
138. MVP Development: Three Concepts
• Profile based: search a subject -> get back ranked profiles ->
view a timeline
• Document List
• People List
139. Document Based MVP
• Topics linked to Subject
Matter Experts
• Subject matter experts have
suggested readings on their
topic
• Results return documents
authored by subject expert
140. Search Interface MVP
-Relevancy based on
degrees of separation
- “Suggestion” panel
features subject matter
experts and top
documents
- People linked to contact
information
-Documents linked to
databases
141. Profile Based MVP
• Provides information on analysts and what they have viewed
• Acts as a landing page from the document and search solutions
• Could either integrate with or replace existing profile solutions
• Could pull data from HR databases to auto-populate profiles
142. Expectation
If an Intel product is available,
an author’s contact information
can be tracked down from their
group
Result
If the contact information is not
included in an intel product,
consumers of that product
cannot contact the authors
Contact information can be
grabbed using auditing
apparatus
143. Expectation
Analysts will find it helpful to
share information about what
they are working on
automatically.
Result
Teams may not find it preferable
to automatically share
potentially sensitive
information. One team does
manually collect a list of key
topics the team focuses on, but
distributes it via word document
/ email to specific collaborators.
144. Key Takeaways - MVP received support (with
caveats)
MVP would be “super helpful”
With caveats:
“Yes. As long as it was kept up to date and could
highlight the major themes, at the very least it would be
a starting point for a senior analyst to talk to a starting
analyst.”
-Interviewee #3, manages analysts and was an analyst
145. Can we do this?
Notes on how a systems like the ones we’re suggesting would fit into
the current information systems used in the IC
-Can we link directly to documents?
-Can we link directly to people?
-Can we only link documents to citations and users need to go get the
actual doc on a different database? (fatal)
-Would we be able to display search results from docs already in the
IC databases?
146. Next Steps
• Evaluate whether automating clustering and tagging is feasible
• Evaluate whether a top-down rule will be followed
• Bring together multiple MVPs through interview tests
• Gain access to think tank data sets (Used to build a subject
recognition and recommendation system)