Watson Curation Playback 0 - Draft v8 2

ECM Design Studio . confidential
Playback -1  
v2
IBM Watson Curator
May 14, 2015

2
IBM Watson Curation : The Team (internal stakeholders)
the team
Product Management
Stephen Hussey
-Program Director,  
Product Management
Jos Olminkhof
-Product Manager,  
Product Management
Designers
Dana Morris
-Program Director,  
ECM UX Design
Tina Adams
-UX Design Lead and Visual
Designer
Stacia Sambar
-Information Designer
Product Development
Mike Marin
-Distinguished Engineer,
Case Manager Architect, US CA
Oren Paikowsky
-SW Development Manager, Israel
Josemina Magdalen
-ECM Content Discovery &
Classification Product Architect,
Master Inventor, Israel
Manjula Hosurmath
-Sr Engineering Manager, India
Seema Meena
-Developer, India
Vishnu N KV
-Developer, India
Ravi Ray
-Developer, India
Sathis Marudanayagam
-Tester, India
Sunil Satti
-Developer, India

3
IBM Watson Curation : Mission
Mission
Watson Curator enables Customers to deliver
relevant, qualified and governed information
collections for Watson

4
Technical Foundation
• New Services and architecture changes
•Auto Classification – Content Classification
•Document transformation– Datacap
•Deployment acceleration – Docker
•Version to version update/migration
• Standards and certifications
• Globalization
• FISMA / ISO (27000/27001)
• Accessibility
• SaaS operating support for Federal
• Regulated data
• FIPS 140-2
• IPV6
Federal Compliance factors Status and mitigation plan for any factor not
compliant
Will the offering claim FISMA compliance? Will target to be FISMA ready by 06/30/2015
Is the offering accessible as per Section 508 of the Americans with Disabilities Act
(ADA)?
Will target to be 508 compliant by 06/30/2015
Does the offering support compliance to FIPS 140-2 (Cryptographic Components)? Will target to be FIPS 140-2 compliant by 06/30/2015
Does the offering support IPV6? Will target to be IPV6 compliant by 06/30/2015
The request was reviewed and approved by Steve Mills on 11/11/2014
Date deviation approval was granted by Steve Mills – 11/11/2014

Watson Curator
Curator Collection
Curator Collection
Curator Collection
Curator Collection
Curator Collection
Curator Collection
Curator Collection
Curator Collection
Curator Collection
 
Watson 
Engagement 
Advisor (WEA)
Unstructured 
Data
Curator Mission
5

Team Lead
Watson Deployment
Lead, provides 
information criteria
Content Curator
Manages information
collections and assigns
work items
Data Expert
Locates unstructured
content
Domain Experts
Classify, review and
approve information
Curator roles

Compelling reasons for a new Watson Curator release
Claim Who says that? Priority?
1 Source documents are not easily consumed by Watson Engagement Advisor.
They are often in the wrong format and require formatting
‘Project Jewel’ and
other customers
high
2 The current product does not comply with federal industry standards that are
considered important for this type of solution
IBM mandates high
3 The curation tasks are as not self-explanatory and intuitive Customers / Prospects high
4 Collaboration is crucial for efficiency and accuracy, but the user interface does not
provide enough capabilities to collaborate
‘Project Jewel’
high

5 Source documents are not easily consumed by Watson Engagement Advisor. The
text format requires editing, e.g. most are in declarative format, they all need to be
in conversational format.
‘Project Jewel’
high

6 Executing curation tasks requires too much effort and is time consuming Customers / Prospects high

10
Year 2015
Hills
3
Katherine a
Curator can:
easily locate,
transform and
analyze data of any
kind into useable
information
in a single collection
1
Katherine a
Curator can:
optimize curation
tasks and activities
to help the system
improve
automatically
resulting in a
reduction of effort
2
Katherine a
Curator can:
recruit a team, set
goals, assign
engaging and
intuitive curation
tasks and activities
in a way that suits
the way they work
and become more
efficient over time
who
what
WOW

Product visual framework will ingest IBM ‘Galaxy’ which is
currently under visual refresh by Q3/4 2015.

What we will show around foundational look & feel is an
approximation of a snapshot in time from Galaxy and will
evolve as Galaxy solidifies their UI.

We will present:

As-Is

(current product UI)

To-Be

(general idea of the UI as Galaxy develops & Dev resources allow)

Future

(robust UI as Dev resources allow)

12
Hill 1
Katherine a Curator can: optimize curation tasks and
activities to help the system improve automatically resulting
in a reduction of effort.

13
Katherine a Curator can optimize curation tasks and activities to help the system improve automatically resulting in a reduction of effort 
As-is
Entirely manual processing, routing and classiﬁcation
effort as part of Curator and SME tasks
The need
Reduce the human effort to consistently curate
documents over the life of the collection
1

14
the scenario:
Katherine, a Curator, is tasked with creating a new collection called Cancer;
• to be populated with a minimum of 6000 documents
• 7 classification categories containing min 2000 documents
• covering different area's of expertise
• with reasonable accuracy level settings for auto classification
As a result of enabling auto classification, the SME, over time, will see a significant
reduction in the number of documents they will need to classify, and Katherine will have
some detailed metrics with which to base her human resource and document decisions
upon.
Let’s look at Katherine’s experience first!
1

1
As-Is: Curator My Collection view

Katherine logs in
and selects the
Add Collection
Button.
The look and feel will
change when Galaxy
develops their new UI
which we will adopt.
New screens are
showing the Galaxy
direction.
1
To-Be: Curator My Collection view

17
1
FUTURE VIEW: Curator My Collection view
Cancer
2,487
2
597
Collected
675
Working
2,487
Curated
870
Excluded
completeness
based on
collection
details
alerts
collectionteam
(Curator, SMEs, Project Lead, Data Expert)
analytics

18
1
As-Is: Curator: Add New Collection View (no auto classification in current product)

19
To-Be: Add Collection Screen SINGLE VIEW
IBM Watson CuratorKatherine
Curator
Collections Work Add Collection
Collection Details
* Collection name:
Collection ID:
Expiration date:
Attachments:
Description:
Scope:
Purpose:
No file selectedBrowse File 1.pdf File 2.pdf
Clear
Cancer
C-10010234
This collection contains all types of human cancer.
Scope text here
This collection is to be used by Clinicians and Researchers in
Cancer diagnosis and treatment.
11 / 24 / 16 12:00 pm EMT
Expert Groups
Name Description
Stanford Medical
UCLA Medical
Stanford cancer specialists
UCLA cancer specialists
Topics
Classifications
Define classification types that will assist users in properly classifying documents in the collection.For each classification type, you can also add one or more classification values.
Categories Values DescAutoclassification Confidence levelsDesc
Enter new classification tvalue here
Drugs
Radiation
Chemotherapy
Eastern Medicine
Treatment
Cancer Type
Symptoms
Diagnosis
OFFON
OFFON
OFFON
OFFON
Min Docs/category: 200067%
Enter new category here
1
Katherine enters
in all of the
pertinent
collection data.
She enables auto
classification for
all categories
with the
exception of
Symptoms, at the
requester’s
guidelines.
Katherine adds
values to each
Category

1
To-Be: Add Collection Screen ALTERNATE TABBED VIEW 1
Curator
ClassificationCollection Details Topics Expert GroupsCollection Details
* Collection name:
Collection ID:
Expiration date:
Attachments:
Description:
Scope:
Purpose:
No file selectedBrowse
File 1.pdf
File 2.pdf
Clear
Cancer
C-10010234
This collection contains all types of human cancer.
Scope text here
This collection is to be used by Clinicians and Researchers in
Cancer diagnosis and treatment.
11 / 24 / 16 12:00 pm EMT
ON
Minimum Included:
Collection Frequency:
Activity Reports:6,000 Daily AM
Daily AM

1
Curator
ClassificationCollection Details Topics Expert GroupsClassifications
Define classification types that will assist users in properly classifying documents in the collection.For each classification type, you can also add one or more classification values.
Categories Values DescAutoclassification Confidence levelsDesc
Enter new classification tvalue here
Drugs
Radiation
Chemotherapy
Eastern Medicine
Treatment
Cancer Type
Symptoms
Diagnosis
OFFON
OFFON
OFFON
OFFON
Enter new category here

1
Curator
ClassificationCollection Details Topics Expert GroupsExpert Groups
Name Description
UCLA Medical UCLA cancer specialists
Stanford Medical Stanford cancer specialists
Jana Jenkins
Bob Piercy
Beth Owens
408-555-1212
408-333-1212
408-444-1212
Cancer Diagnostics Specialist
Head of Stanford Cancer Department
Stanford Cancer Resident Physician
janaj@stanford.edu
bpiercy@stanford.edu
bethO@stanford.edu
Enter new group member here
Find me experts
Send an invitation

23
Let’s see how autoclassification metrics can help
Katherine make fast and easy decisions as she does
her daily job.
Value auditing
Workload rebalancing
1

24
1
To-Be: Value Auditing
Katherine can see:
• all the categories
at a glance
• previous
accuracy level
adjustments
• system
suggestions to
merge categories
• underperforming
values
• when values
have reached the
collection
minimum
• when values are
under the
minimum (grey)
• make inline auto
classification
adjustments
without having to
leave the screen
• see popup of
detailed
descriptions of
each category
and value
Curator
Collections Work Value Auditing
Cancer
Bone
Colon
Esophageal
Gall Bladder
Ovarian
Liver
Skin
Cancer Type (7)
Treatment
Symptoms
Diagnosis
Drugs
Radiation
Chemotherapy
Easterm Medicine
2310
2004
100
2032
OFFON Min Docs/category: 200090%
OFFON
2025
1500
270
328
2570
140
370

25
1
FUTURE VIEW: Value Auditing alternate deep dive
Curator
Collections Work
Treatment | Drugs
Treatment | Radiation
Treatment | Chemotherapy
Treatment | Eastern Medicine
Treatment | Surgery
Treatment | Alternative
February March April May
Today
1
overlapping
values pairs
1
accuracy level
adjustments
20
sources
Projected Value goal 5/17/15
2,000 documents/value Included
2000Min docs/value
67%
projected min docs date based on
previous ingestion metrics
accuracy level
Cancer
Value auditing projection
1
underperforming
values

26
1
FUTURE VIEW: Value Auditing alternate deep dive
Curator
Collections Work
Treatment | Drugs
Treatment | Radiation
Treatment | Chemotherapy
Treatment | Eastern Medicine
Treatment | Surgery
Treatment | Alternative
February March April May
Today
1
overlapping
values pairs
1
accuracy level
adjustments
20
sources
Projected Value goal 5/17/15
2,000 documents/value Included
2000Min docs/value
67%
projected min docs date based on
previous ingestion metrics
accuracy level
Cancer
Value auditing projection
ChemotherapyActions
DrugsMerge with:
Adjust: Minimum documents/value: 200
CancelSave
1
underperforming
values

27
1
To-Be: Workload viewing/rebalancing
Katherine can
now immediately
see potential
problem areas at
a glance.
• overdue
• unassigned
• assigned
She is able to
quickly reassign
tasks to different
groups or specific
SMEs.
She can quickly
assign tasks or
ingestions to
groups or specific
SMEs.
IBM Watson CuratorJanet Bauer
Curator
15
My Collections
597
Collected
675
Working
2,487
Curated
870
Excluded
Domain Expert would
not see Collected
click this and
go to details
Curation task 1/22/15
300 documents
100 documents
20 documents
500 documents
50 documents
UCLA Medical
UCLA Medical
UCLA Medical
UCLA Medical
Overdue (970) UCLA Medical
3/23/15
3/23/15
3/23/15
3/23/15
1:00 pm GMT
9:00 am GMT
10:00 pm GMT
4:00 pm GMT
Unassigned (525) UCLA Medical
Assigned (473) UCLA Medical
Ingestion 3/12/15
Ingestion 3/17/15
Ingestion 4/1/15
Ingestion 4/3/15
ingestion 4/4/15
200 documents
100 documents
50 documents
75 documents
100 documents
By status
By task
By group
By person
By date due
By status
Cancer
Stanford MedicalDue: Assign to:5 / 24 / 15 12:00 pm EMT
Stanford MedicalDue: Assign to:5 / 20 / 15 2:00 pm EMT
Group/personDue: Assign to:5 / 24 / 15 12:00 pm EMT

28
Now let’s look at the SMEs experience!
1

29
1
FUTURE VIEW: SME Dashboard view

30
1
FUTURE VIEW: SME Dashboard view

1
FUTURE VIEW: SME My Work idea
This could be a
possible view
into My Work as
an SME.
This idea also
shows
collaboration:
• query
colleague for a
review
• send a note to
team member
• redirect a
document to a
team member
• make personal
notes on a
document
SME
My Work
15
Cancer
Tumors
Viral Disseases
Motaba
0
5,000
4,207
Liver Disseases
4,207
2,487
Recent Collections 597
Collected
675
Working
2,487
Curated
870
Excluded
Bone_Metastasis.pdf
32 pages
17 pages
48 pages
11 pages
Added by DSanderson on 1/14/2015 at 3:02 pm
Source: IBM Stored IQ
Modified by DSanderson on 1/15/2015 at 11:57 am
Today at 1:39: to @Susan Longhi:
@Stephen Hussey, I need your
opinion on this document.
12/29/14 at 2:54: from @Susan Longhi:
I took a look at it and think we may need
to call in another Domain Expert due to
the nature of this document. I have seen...
Today at 1:42:
need to follow up on this source,
alerted Stephen & Susam.
Yesterday at 3:37:
unsure of this source
Cancer_treatment_study.pdf
Melanoma_study.pdf
Lymphoma_study.pdf
12/29/14 at 2:54: @Curator:
This file is corrupted, so I think we nee to
transform again.
12/29/14 at 2:54: #reassign@Seema Meena:
I need you to classify this document based
on your expertise
V
by urgency
by priority
by urgency
by activity
by name
my notiﬁcations my notes
12/29/14 at 2:54: @Curator:
transform again.
Domain Expert would
not see Collected
Could also have comments and
notes on a TASK level too!!!!
Search
click this and
go to details

1
To-Be: SME Classification Screen (autoclassification has just been enabled, no suggestions yet)
When the system
first starts to
learn, it cannot
make any
suggestions for
classification to
the SME yet.
Domain Expert
My Work | Cancer | Document_1.html
15
Cancer_1002.pdf: 67% confidence
Classiﬁcation Summary Description Statistics Collection Goals
Search in document
Include Exclude
2
Previous | Next
View similar documents in the collection
Search
Submit new
Bone
Ovarian
Colon
Espphageal
Gall Bladder
Liver
Skin
Search
Biopsy
Blood Test
X-Ray
CAT Scan
PET scan
Spinal Tap
Ultrasound
Search
Submit new
Submit new Submit new
Drugs
Radiation
Surgery
Eastern medicine
Search
Submit new
Exhaustion
Immune disorders
Infection
Mobility
Pain
Rash
Weakness
Search
Stage 0
Stage 1
Stage 2
Stage 3
Stage 4
End-Stage
PrognosisDiagnosis
SymptomsSymptomsTreatment TypeCancer Type
Autoclassification just turned on-> No suggestions yet

1
To-Be: SME Classification Screen (autoclassification has just been observing over time)
As an SME, we
now can see that
Curator is making
suggestions for us
to look at and
consider.
Domain Expert
15
Classiﬁcation Summary Description Statistics Collection GoalsInclude Exclude
2
Previous | Next
Search
Submit new
Bone (30%)
Ovarian (30%)
Colon
Espphageal
Gall Bladder
Liver
Skin
Search
Biopsy (55%)
Blood Test (46%)
X-Ray (22%)
CAT Scan
PET scan
Spinal Tap
Ultrasound
Search
Submit new
Drugs (30%)
Radiation (28%)
Surgery (25%)
Eastern medicine
Search
Submit new
Exhaustion
Immune disorders
Infection
Mobility
Pain
Rash
Weakness
Search
Stage 0 (37%)
Stage 1 (35%)
Stage 2 (33%)
Stage 3 (33%)
Stage 4 (21%)
End-Stage
Prognosis (77%)Diagnosis (33%)
SymptomsSymptomsTreatment Type (72%)Cancer Type (40%)
Autoclassification is learning and making suggestions.

1
To-Be: SME Classification Screen (autoclassification is running smoothly)
We now see that
Curator has
reached it’s stride
and has provided
us with
confidence levels
and solid category
value choices,
allowing us to be
able to quickly
make decisions
and improve the
accuracy of the
classification.
In the future,
could show
ground truth WEA
mapping overlay
for docs that need
to be re-curated.
Domain Expert
15
Classiﬁcation Summary Description Statistics Collection Goals
Search in document
Include Exclude
2
Previous | Next
Search
Submit new
Bone (55%)
Ovarian (30%)
Colon
Espphageal
Gall Bladder
Liver
Skin
Search
Biopsy (100%)
Blood Test (88%)
X-Ray (58%)
CAT Scan
PET scan
Spinal Tap
Ultrasound
Search
Submit new
Drugs (100%)
Radiation (89%)
Surgery (44%)
Eastern medicine
Search
Submit new
Exhaustion
Immune disorders
Infection
Mobility
Pain
Rash
Weakness
Search
Stage 0 (97%)
Stage 1 (88%)
Stage 2 (70%)
Stage 3 (47%)
Stage 4 (32%)
End-Stage
Prognosis (77%)Diagnosis (33%)Diagno
SymptomsSymptomsTreatment Type (72%)Treatment Type (7Cancer Type (40%)Cancer Ty
Autoclassification has learded after a time and is making confident suggestions

1
As-Is: Curator: collection properties dialog

36
Hill 2
Katherine a Curator can: recruit a team, set goals, assign
engaging and intuitive curation tasks and activities in a way that
suits the way they work and become more efficient over time.

37
Katherine a Curator can: recruit a team, set goals, assign engaging and intuitive curation tasks and activities in
a way that suits the way they work and become more efficient over time
As-is
Curator team engagement is manual and repetitive.
SME context, inputs and work management is limited.
The experience requires training
The need
Optimize the Curator’s use of a team
Present the SME task experience in an optimized layout for
information and actions
33
2

38
Overview of the To Be process
34
Curator
Select item

From
inbasket
Classify

and Tag
Include/
Exclude
Repeat
SME
Administer
classification
categories as
needed
Tune
classification to
reduce effort
Assign work to
SME
As Katherine creates a new collection,
she specifies a scope and purpose for the
collection, helping the entire team to stay
focused.
Create new
collection
Initial config
auto
classificatio
n
Curate for
xxx days/
weeks
The SME collaborates with Katherine, and with other SMEs,
reassigns the curation task, or sub-task(s) for individual documents in the task,
and saves time by leveraging the expertise of other SME's.
With a few mouse clicks, Katherine
then easily assigns a task to a SME
requesting to review several documents
to be curated.
2

39
the scenario:
Katherine, a Curator, needs to assign and manage SME groups in her new collection called
Cancer;
• to
As a result, SMEs….
Let’s look at Katherine’s experience first!
1

1
To-Be: Curator Expert Groups assignment & management
Katherine creates
expert groups and
starts to assign
members.
In the future,
Curator could
keep track of all
the various SMEs
and make
suggestions
based on
specifics defined
in Collection
Details.
In the future,
Katherine could
track SME
availability and
send a notice right
from this screen.
Curator
ClassificationCollection Details Topics Expert GroupsExpert Groups
Name Description
UCLA Medical UCLA cancer specialists
Stanford Medical Stanford cancer specialists
Jana Jenkins
Bob Piercy
Beth Owens
408-555-1212
408-333-1212
408-444-1212
Cancer Diagnostics Specialist
Head of Stanford Cancer Department
Stanford Cancer Resident Physician
janaj@stanford.edu
bpiercy@stanford.edu
bethO@stanford.edu
Enter new group member here
Find me experts
Send an invitation

41
Curator can view Document analytics in a collection and adjust based on time, priority, and SME
2
FUTURE VIEW: Curator Collection Analytics view
Katherine is able
to assess
workload and
able to shift
tasks between
either people or
Expert Groups.

42
2
FUTURE VIEW: SME Work screen: Collaboration
SME
My Work
15
Cancer
Tumors
Viral Disseases
Motaba
0
5,000
4,207
Liver Disseases
4,207
2,487
Recent Collections 597
Collected
675
Working
2,487
Curated
870
Excluded
Bone_Metastasis.pdf
32 pages
17 pages
48 pages
11 pages
Today at 1:39: to @Susan Longhi:
@Stephen Hussey, I need your
opinion on this document.
12/29/14 at 2:54: from @Susan Longhi:
I took a look at it and think we may need
to call in another Domain Expert due to
the nature of this document. I have seen...
Today at 1:42:
need to follow up on this source,
alerted Stephen & Susam.
Yesterday at 3:37:
unsure of this source
Cancer_treatment_study.pdf
Melanoma_study.pdf
Lymphoma_study.pdf
12/29/14 at 2:54: @Curator:
transform again.
12/29/14 at 2:54: #reassign@Seema Meena:
I need you to classify this document based
on your expertise
V
by urgency
by priority
by urgency
by activity
by name
12/29/14 at 2:54: @Curator:
transform again.
Domain Expert would
not see Collected
Could also have comments and
notes on a TASK level too!!!!
Search
click this and
go to details
SME can see
message/
collaboration
threads about
specific
documents and
perform actions
such as reassign
to a colleague in
their collection
team.
SME can also
make personal
notes on a
specific
document as
well.

43
Hill 3
Katherine a Curator can: easily locate, transform and
analyze data of any kind into useable information in a single
collection.

44
Katherine a Curator can: easily locate, transform and analyze data of any kind into useable information in a single collection
As-is
Valuable information is inaccessible to Watson. E.g.
PDF and Word files with multi column table
based information are inaccessible without
manual editing. Data with inconsistent layouts
and formatting are extremely challenging to use.
The need
Auto transform information like PDF and Word into
well structured HTML for WEA consumption
3

45
Overview of the
To Be process
41
Curator
Select item

From
Inbasket
Classify

and Tag
Edit
Document or
Version
Include Repeat
SME
Curate for xxx
days/weeks
Transform

To HTML
PDF
DOC
(XLS)
Compare w

previous version Transform text

for WEA consumption
Base formats supported:
Linear format / Normalized
Changes are highlighted
New document
New version
Always / On demand
3
Katherine a Curator can: easily locate, transform and analyze data of any kind into useable information in a single
collection

46
Curator can view Document analytics in a collection and adjust based on time, priority, and SME
FUTURE VIEW: SME version compare
SME is able to
easily see where
WEA mapping
exists and
changes were
made between
versions.
All rolest
15
77% confidence
can we do a bg diff and
only show those areas that
are different?
those areas that are mapped
in gt?
Supported:
word, pdf, htm
Transforn:
orig vs transformed
ability to approve transform,
or rejectl
version 1.0 version 2.0
accepting replaces the curren v1.0
with this version, v1.0 moves to Excluded
rejecting keeps v1.0 in the collection,
v2.0 moves to Excluded
Visual Diff button
Edit button
3

47
Project dates, milestones, and details…

48
Timeline
Q1 Q2 Q3 Q4
Change Management
Q3 Q4
Data transformation and extraction
Auto classification
User efficiency
Q1 Q2

Milestones for V2
Hills Playback
8 April
Playback 0
1 May
Playback 1
27 May
Playback 2
24 June
Playback #...
22 July V2 Availability
Offering
Business
Plan
5th May
Product
ReadinessPMOM
49
~4 Weeks

User research and sponsor users Validating assumptions
Org Type Contact Hill NDA Approached Confirmed Engaged
Geico Customer Jack Jones 1 and 2 N Y N N
IBM Consultant Sridhar Sudarsan 3 N/A N N N
IBM Sales Maig Worel 1,2,3 N/A Y Y N
50

Feedback
• Huzefa – The variable timeline
•Challenges with go to market activities. We will have to figure out the lead time required to trigger GTM and
line it up with the last playback
• Jos – Product dependencies
•Introducing final delivery uncertainties across teams will have to be managed.
• Manjula – Migration
•Add version to version update/migration to the Foundation
52

To-be: Curator Create New Collection View: with scope and purpose
2

Watson Curation Playback 0 - Draft v8 2

Recommended

Recommended

More Related Content

Viewers also liked

Viewers also liked (7)

Similar to Watson Curation Playback 0 - Draft v8 2

Similar to Watson Curation Playback 0 - Draft v8 2 (20)

More from tadams76

More from tadams76 (8)

Watson Curation Playback 0 - Draft v8 2