All testers have users with unique needs. Are these needs included in your requirements? Lisa denDekker-Redemann says that was not always the case at UPS. Were we testing the mobile systems that our crew members use like we should? Sometimes to get it right, we have to go out into the wild—or in our case, out into the wild blue yonder. To get the whole story (and awesome test coverage), we boarded an aircraft to design scripts and test live 18,000 feet in the air. We found real conditions our users face that didn’t make the user stories and test conditions. Lisa shares why she went into the wild—building relationships with real users, the importance of environment factors, and how it all makes for better testing. What are some items that may be missing from your test coverage? Join Lisa to find out why they got into the cockpit—and more importantly, what they got out of it.

1.
T8
Test
Techniques
10/5/17
11:15
User
Experience
Testing—with
the
Pilots
at
18,000
Feet
Presented
by:
Lisa
denDekker-­‐Redemann
UPS
Brought
to
you
by:
350
Corporate
Way,
Suite
400,
Orange
Park,
FL
32073
888-­‐-­‐-­‐268-­‐-­‐-­‐8770
·∙·∙
904-­‐-­‐-­‐278-­‐-­‐-­‐0524
-­‐
info@techwell.com
-­‐
http://www.starwest.techwell.com/

2.
Lisa
denDekker-­‐Redemann
UPS
With
thirteen
years
of
QA
experience
across
various
logistic
domains,
Lisa
denDekker-­‐Redemann
is
the
UPS
flight
and
crew
QA
manager,
working
on
mobile
solutions,
creating
a
mobile
lab,
and
transitioning
her
area
to
DevOps.
Lisa’s
responsibilities
have
included
transitioning
the
QA
role
to
agile,
re-­‐defining
QA
artifacts,
and
championing
the
QA
discipline
in
general.
Lisa
is
passionate
about
QA,
learning
and
innovation,
and
her
family.
You
never
know
where
she
will
pop
up,
what
wild
idea
she
may
cook
up,
or
what
she
will
be
testing
professionally
or
personally.
Lisa
is
a
wife,
mom
of
three,
photography
enthusiast,
and
a
classic
movie
lover.
Feel
free
to
tweet
her
@LisaRedemann.

3. User Experience Testing
at 18,000 feet
Lisa denDekker-Redemann

4. Who am I?
• Lisa denDekker-Redemann
• QA Manager of UPS Airlines Flight & Crew Domain
• Passionate QA for 14 years
• Drawn to process improvement and innovation
• Twitter: @lisaredemann
• Instagram: lisaredemann
• Also….Mom of three, classic film lover, a Disney person,etc.

5. Why did I fly?

6. The Device: EFK (iPad)
• Our users hail as the best improvement we have made
• Replaces the Flight Bag/Kit
• Primary use is to replace paper navigational charts and manuals
• Secondary use is for training
• We are upgrading our users to the iPad Pro from iPad2-iPad Air

7. The Problems/Issues
• Lack of conditional requirements
• Lack of Non-Functional requirements
• Little understanding of user issues
• Lack of “Real World” process

8. Lack of Conditional Requirements
• Using Wifi and Not
• Weather Issues
• Turbulence Issues

9. Lack of Non-Functional Requirements
• When the device should stay awake?
• When messaging to the device is allowed?
• What should be preferred method of logging in?
• What should be preferred method for toggling apps?
• Is there a standard for the apps to be organized?

10. Little Understanding of User Issues
• Varying range of technical ability in user base
• What now? Attitudes
• User id/pw
• Adoption rates
• Length of time to get comfortable
• Frequency of updates to Aircraft Briefing Guide

11. Lack of “Real World” Process
• The defined process vs. what happens
• The understanding of/documentation of variations

12. The Hypothesis
• By observing the users using the device and applications in a day of
ops scenario, we will have a better understanding of test coverage
needs.

13. Scope
• The scope was focused on activities directly correlated to a flight.
• No other activities on the device were considered for this activity

14. Pre-Flight Work
• Document what we think we know
• Determine what we want to observe
• Setup a current and future test device

15. What we think we know
• The documentation that the business used to certify IOS builds
• The “requirements” we had
• Information given to the users

16. What do we want to observe
• The business processes as a whole, this includes activities that don’t
include the device directly
• How the user interacts with the device
• What the user expects to happen

17. Exploratory List
BP Item Observation
Pre-Flight Do they come to work with everything ready?
Pre-Flight What other activities are performed with and without
the device?
In Flight Does the dock of the device hinder the user?
In Flight Are they switching b/t day and night mode?
In Flight What are they using the device for in flight?
Post Flight What post flight activities if any is the device used for?
All How are they logging in?
All How do the users access the apps?

18. What did we bring with us?
• Current Device “As-Is” Model
• The Future Device “To-Be” Module
• List of items to observe/exploratory list
• Notebook and Pen
• Critical Eye

19. Business Processes
• Pre-Flight activities
• In Flight Activities

20. Pre-Flight Activities
• Checking in
• Updating manuals
• Receiving Briefing Packet
• Looking up MEL (issues with the aircraft)
• Downloading correct Navigation for the flight path

21. Pre-Flight Activities
• Crew Review of briefing packet
• Getting to the aircraft
• Aircraft walk off
• Closing the door
• Airplane Mode

22. In Flight Activities
• Take-off
• Reaching 18,000 feet
• Communicating with
• Adjusting the map for current location
• Observing airspace

25. Post Flight Activities
• Switching out of Airplane mode
• Turnover of any Aircraft Issues to the Ground Crew
• Disembark from the Aircraft

26. Layover
• Review observations thus far
• Edit the list for the return flight
• Document any questions or items the users need us to follow up on
• List any process improvement suggestions

27. Observations
• The users were having to manually adjust the nav charts in flight
• The users dependency on the device
• The users need apple tip/trick knowledge
• The users true workflow
• The mount of the device in the aircraft

28. List Edits
• Will any settings be modified due to time of day?
• Will the pre-flight activities be changed since the flight plan was
already reviewed?

29. User Questions
• Gestures
• Setting wake options on applications
• Difference between the current and new device
• They noticed having to many flight paths stored in the app reduced
performance
• Navigation app spun for a little bit every time it is opened

30. Process Improvements
• Developing an in app toggle for the two primary in flight apps
• Digital briefing packet linked to digital manuals for info lookup
• Digital briefing packet linked to navigation app
• Default setup of critical apps

31. Findings and Action Items
• User Training Items
• Modification or Functional Change Ideas
• Business Process Updates
• User Responses to Questions

32. Summary
• Received information to create better test coverage
• Gave the users some advise and a method to report issues
• Produced some process improvement ideas for our peers

33. Questions