Lisa Fast
@lisavation
#GOAT2017
Not Too Minimal:
UX Research in a Government
Minimum Viable Product

2
Original Minimum Viable Product View – focus on user need
http://blog.crisp.se/2016/01/25/henrikkniberg/making-sense-of-mvp

3
“The MVP Build-Measure-Learn
feedback loop is at the core of the
Lean Startup model. An experiment
is … also a first product.”
- Eric Ries

4
Minimum Valuable Product
https://www.quora.com/What-is-the-difference-between-a-
prototype-and-a-minimum-viable-product-MVP

5
“A Minimum Viable Product is the
smallest thing you can build that
delivers customer value (and as a
bonus captures some of that value
back).”
– Ash Maurya
https://blog.leanstack.com/minimum-viable-product-mvp-7e280b0b9418

6
https://hackernoon.com/the-mvp-is-dead-long-live-the-rat-233d5d16ab02

7
Risky assumption testing requires a level of
Agile maturity not present in most of the
Government of Canada.
In this project, the idea of starting small
was a struggle, most expected to get the
complete product at launch – despite
ongoing mega-project troubles…

8

9
Receptive moment in government

10
UX Research is all about
Value & Viability
Understanding needs for value
Ensuring they can derive that value

11
The challenge: Email-only Consultations for Gazette Part I
Part I: “a final opportunity to review and comment on a proposed regulation at the last
stages of the regulation-making process, before it is enacted”

12
Starting Point: US eRegulations -18F Comment Pilot Test in June 2016
https://18f.gsa.gov/2016/07/26/new-pilot-aims-to-streamline-notice-and-comment-process/
https://epa-notice.usa.gov/

13
LEAN UX Pilot assumptions:
We believe that online regulatory consultations will lead to effective commenting & analysis –
teams just fell into emails & spreadsheets as people stopped sending letters
This will improve:
the quality and efficiency of commenting by Canadians and analysis by government teams
We will have demonstrated this when we can measure*:
More time to think about comments – less time ’handling’ them:
• A decrease in frequency of copy/paste for receivers & analysts and decrease in ‘lost’ comments
• A decrease in time from closing date to completed analysis/review
• Improved quality and coverage of comments in review and decision process
Clearer and more thorough comments from Canadians:
• An increase in comments clearly associated with a specific section of the RIAS or proposed regulatory text
• An increase in the number of comments overall
* Measurement will be ‘loose’ for this pilot since so few submissions are expected for the selected regulation.

14
Start with prototype
before contract as
proof of concept
- assume that 18F
design is good start

15
UX Research sessions
started on prototype,
progressed to dev site as
features were implemented
Exploratory sessions – use and talk

16
Task: Get started - Orient the commenter
PROTOTYPE COMMENT MVP
Name and submit was at bottom of page
• First participant looked for it at top, wanted an idea of
what to expect, what was possible
Moved name etc. to top & added ‘Skip to first comment box’
to orient users to where they could enter comments.
Add ‘View summary /Submit’ buttons to top of page

17
Task: Enter comments
• Research participants wanted reassurance it was saved
P2: “I'm always worried about saving– that’s what drives
us back to Word.”
• Had been burned by other experiences
• P1: “3 times I went to submit and it cleared my input - 3
times!!
PROTOTYPE COMMENT MVP
• Save button and saved message added
• P3: “Good to see that saved message!”
• P4: “This is very straightforward”

18
Task: Review summary before submit
PROTOTYPE
Participants were clear they need a record of their
submission for their members. Some post it on their
website. Would prefer a table view of their comments.
P1: If there was a table.”
P2: “I want to see a table of ALL possible comments with
blanks where I missed them”
Changed to table view in MVP –for Pilot, must return to main
page to edit/enter, long term should build it into here, also want to
email a copy of this to themselves
P4: “Display empty comments - That makes sense to see what you missed
especially if it was very long. I might put ‘read it’ in there for myself”
COMMENT MVP

19
Aha! Associations Perform Same Share-Receive-Analyze Process
Just like government, associations gather member comments, review & synthesize the
comments into a submission - need support & integration, not add-on copy/paste work
Participant 2: “So I could send them a link to see our actual submission- I like that idea as long as it was easy for them to do”.
Participant 4: “if we could do this, would we even keep a Word copy? Because it’s tedious… this could streamline our process”
Future is participative
democracy: invite
them in to the system,
let them use it with
their members & feed
their submission in
directly
Association members
Association
submission
Individual commenter

20
On-time launch of Comment Pilot in French & English
Site launched on-time on March 4th
simultaneous with Gazette on-line
launch of the proposed amendment

21
Hold up stickies to vote on 4 UX research sessions
– in 6 weeks from contract to launch
Too much
Too minimal
Just right
Minimum viable
Not
enough
Not minimal

22
External Stakeholders really got ‘minimal’
• Result: Only 1 of 5 association stakeholders submitted their comments online – the rest stuck to email
and the HC team entered their emailed comments into the system themselves to learn
• 18F pilot had slightly more, about half of submissions came in via the pilot, was better integrated but
their participants also sought in-progress drafts and mult—session editing
Participant 1: “I get that this is much better for the government, then it’s all laid out in a table for them…
But it creates more work for me. It’s a second layer – I put it in a table and say hey members, send out
a table to them and get them to do it. There’s no save and share option here.”

23
Make notes on stickies during next bit
Not enough
Too minimal
Just right Too much
Not minimal
Your noteYour noteYour note

24
External Commenter Site Internal Analysis Toolkit
Now to the other VALUE - the back end (18F MVP did not do this)

25
UX Research: Kicked off with Discovery workshop to discover
various roles, user needs & pain points in current process
Prepare Gazette Receive (Reg Management)
Worry: lose a comment or
lose track of a comment
Analyze (Programs)
Worry: miss an important
comment or issue
Review (all and TBS)

26
Discovery – your vote?
Too much
Too minimal
Just right
Minimum viable
Not
enough
Not minimal

27
Process from Discovery Maps Onto Key Tasks in Toolkit
demo

28
Data Model

29
Consultation Toolkit MVP co-designed with Health Canada team
UX research: Sketching, 4 co-design workshops & 2 usability test sessions with team members

30
Toolkit: View comments by
submission (commentary) or
by document section

31
Notes capture
analysis, progress and
decisions:
- See public notes by
other analysts
- Assign note a status
- Add public and
private notes

32
Co-design process – your vote?
Too much
Too minimal
Just right
Minimum viable
Not
enough
Not minimal

33
18F MVP did not provide analysis toolkit – it was
performed by an external consultant. Made their job easier

34
Analysis toolkit had more discovery time – got farther
Got closer to meeting needs of receivers, analysts and reviewers – co-created site with them
“I definitely think we are headed in the right direction – Very supportive of proceeding”
“I’m supportive of proceeding with changes to terminology, documentation and reporting”.
All consultations entered online - team performed analysis & review on the Toolkit
Lots of unimplemented bits – preparation of the document for launch was done by hand

35
eRegs Pilot Consultation System - Design Components Map
External Commenter Site
Internal Analysis Toolkit
Written in Swift

36
Open Source Code
3500 lines of Swift written for the MVP and 52000
lines of Swift including the open source used.
750 lines of Javascript written on WET/ JQuery base
250 lines of CSS with WET and Bootstrap reuse
2800 lines of web page template code
“My experience is that Swift was a productive choice. Good quality, stability,
maintainability. Swift and Vapor are in rapid evolution phases and this will
require ongoing code maintenance to stay current.” – Steve Hume

37
No documentation or
automated test case
development - it’s an
MVP
- ran out of time and $

38
“We recognise that it would be a lot for someone to take
on the code base as it stands, in such an early prototype
state. The limited time to deliver an MVP, leads to the
code being less robust than we’d like. However, by
sharing the project progress and research we hope it will
be useful to any council considering a similar technical
tool for their ways into work programme.”
https://blog.wearefuturegov.com/lets-open-source-by-default-fb06f509dd9e

39
Open source code base – your vote?
Too much
Too minimal
Just right
Minimum viable
Not
enough
Not minimal

40
Minimum Viable Product is a team approach – learn together
Learning from 1st MVP
experiment dictates what
to build for next
experiment
Each iteration adds &
tests new assumptions
Product Owner
maintains this vision,
makes final call on
each experiment step,
is present throughout
the process
* We acted as product
owner – Government
needs to be product
owner!

Questions?
MVP regulation consultation for commenters – demo link: https://dev.regconsultation.ca:8080/
• Article on LinkedIn: https://www.linkedin.com/pulse/government-minimum-viable-product-learning-from-small-lisa-fast
• US 18F pilot results: https://github.com/18F/epa-notice/files/548513/FinalDemo-Phase3UserResearchFindings.pdf
• Repository https://github.com/canada-ca/regs-consult-wet
@lisavation
lisa@vation.ca