Hear how Mediacurrent teams with Mass.gov on a user testing strategy leveraging Drupal 8 and Google Optimize.
Watch how to deliver a constituent experience that is discoverable, accessible, and truly “for the people, by the people.” Follow along with our process. Learn tips to user test your way to better website UX.
- Our approach to testing and gathering user feedback
- The A/B testing variants we used to steer site navigation and layout
- Deep dive into testing with Google Optimize
Watch the recording:
Mediacurrent is a full-service digital agency that
implements world class open source software
development, strategy, and design to achieve
deﬁned goals for enterprise organizations
seeking a better return on investment.
Mass.gov Mission Statement
Leverage the best technology and information to make
people’s digital interactions with government
fast, easy, and wicked awesome.
I. Where This All Started
II. Mass.gov UX philosophy
III. User Feedback: What We Learned
IV. Why Google Optimize A/B Testing?
V. Testing process and Results
VI. The Google Optimize Experiment Containers
VII. Lessons Learned
● Help developers and content
specialists gather user feedback
A/B Testing Goals
● Test menu variations
● Measure user interaction with a new contextual pages
menu and page relationship indicator links
Test Method, Measurement & Success Criteria
Build Google Optimize
Experiments, with up to 4
variations to test each
● Testing will be done with
randomly selected visitors
being sent to one of the four
● Test pages will be manually
selected and edited using
the Google Optimize
administration and interface
● The percent of visitors that
clicked a link in the
● The percent of yes/no votes
of each test grouping on at
least one of the test pages.
● The number of pages per
session / session length for
people in each test group.
Plan Method Measure
A/B Testing Experiment 1
● Contextual links or global navigation?
● Would it be equally eﬀective to hide the global
navigation so that users would use the site search
● Which is more engaging: Contextual links pointing to
pages within the same service, or to pages related to
● Would test variants provide a better overall experience?
Moderated User Testing
To determine which of these three types of navigation
oﬀers users the most intuitive and eﬃcient navigation
● Global Navigation (current site navigation)
● Contextual Navigation (proposed navigation)
● No navigation (which would instead push users into
using in-page links and site search)
We need to see a beneﬁt to the user based on opinions — did the
links prove to be useful?
↓ time on site
↓ number of pages per session
↑ increase in satisfaction.
Simple engagement isn’t enough.
Each scenario began with a story of the journey that brought the user to the starting point and an
additional task to complete that would require them to move laterally through the site toward an
● Starting point:
● Story: You’ve been researching how to ﬁle your tax return, and have just ﬁnished reading this page.
● Task: Now you want to ﬁle your taxes.
Each interview included 3 scenarios, consisting of a unique topic and navigation.
Follow Up Questions
➔ Could you say which
scenario was easiest to
navigate, and why?
➔ Can you name any
between the three
➔ Could you say whether
any of the three
scenarios was easy or
diﬃcult to navigate, and
We considered three factors when determining which of these
navigations was the most eﬃcient and intuitive:
● Which took users the shortest amount of time to reach their goal?
○ We tracked the time of each test and produced an average for each
● Did users ﬁnd one scenario signiﬁcantly easier to navigate?
○ We recorded user answers and documented trends
● Did the interviewer ﬁnd an observable diﬀerence in the ease with which users
navigated the diﬀerent scenarios?
○ Interviewers recorder observations and documented trends
A/B Testing Experiment 2
● Would moving the Main Navigation oﬀ screen have a
● Would updating the page relationship indicators help
users navigate within the website?
Global Navigation Change
H: Hiding the global navigation
on all pages except the home
page will have minimal negative
impact to the user experience.
G: Discover if hiding the
navigation in a “hamburger menu”
will have a negative impact on
Relationship Indicator Change
H: Making the relationship indicator
more visible to users visiting a service
family will increase the likelihood that
they use it to navigate within the
G: Discover if moving the relationship
indicator helps users navigate within the
website when they need to ﬁnd
additional content related to the current
A/B tests were conducted
over the course of a
Total visitors to test pages.
No negative impact on
No negative impact on
session duration or
pages per session.
44%One of the variations
saw a big increase in
impact on overall
● We proved the changes have no harmful impact on site
● Observations from qualitative user testing and research have
suggested that these improvements will be helpful to users.
The A/B test allowed us to manage the risk of releasing this
change to a large audience.
● We are validating this by examining whether these changes
have had a measurable impact for speciﬁc groups of users or
in speciﬁc areas of mass.gov.
Were We Successful?
Updates made in the editor are added after the
DOM originally loads. This can cause race
handlers and event triggers on elements after they