The document discusses optimizing usability and human-robot interaction in medical robotics. It includes presentations from David Shane of Boston Engineering, Daniel Melanz of Energid Technologies, and Dorothy Shamonsky of ICS. Shane discusses Boston Engineering's work evaluating human-robot interfaces for DARPA. Melanz describes Energid's robotics software Actin, which provides simulation, control and coordination of complex robotic systems. Shamonsky emphasizes the importance of user experience design and outlines best practices for working with users to understand requirements and test prototypes. The overall document focuses on challenges in developing intuitive interfaces for emerging medical robotics.

1. David Shane, Boston Engineering
Daniel Melanz, Energid Technologies
Dorothy Shamonsky, Ph.D., ICS
Optimizing Usability and
Human-Robot Interaction (HRI)
in Medical Robotics
October 1, 2020

2. David Shane
PM and Business Development Manager
Boston Engineering’s Advanced Systems Group

3. 3
Improving the Way People Work and Live Through
Innovative Product Design and Novel Engineering
Innovation from Concept to Commercialization

4. 4
Boston Engineering’s Centers of Excellence

5. 5
Case Study – DARPA Robotics Challenge
Worked with DARPA to evaluate the Human Robotic
Interface (HRI) of large humanoid robots:
● Remote operation with select autonomy
● Multiple sequential tasks
● High risk environment
The team evaluated:
● Effectiveness of user interfaces
● Task balance and approach
● Generated predictions of outcomes

6. Categories of Operator Effort
• Situational Awareness –
Visual & Audible
• Emergency – Haptic
• Situational Awareness
& Emergency Events –
Visual
6
Interacting with the
robot through the
interface
Generating environmental
awareness and assessing how the
robot can interact with it
Interacting with the
environment

7. Predictions
How Well Does the
Interface Fuse and
Balance All Tasks?
How Well Can the
Interface Address
the Tasking?
How Well Can the
Interface
Fundamentally
Function?
8 Tasks, 8 Point Scale (tol ± 1)
Overall
Operation
Subtask 1
Subtask 2
Subtask n
1st Order
Manipulation
2nd Order
Manipulation
Obstructed
Traversal
Unobstructed
Traversal
Subtask 1
Subtask 2
Subtask n
Overall
Operation
● “Mechanics” of
Interface
● Task Alignment
Analysis
● Multiple Resource
Theory (multi-modal)
7
Our Assessment
71% Correct!!
Team Self Assessment
45% Correct

8. Getting it “Right”
8

9. Getting it “Right”
9

10. Daniel Melanz
Robotics Engineer
Energid Technologies
Optimizing Usability and
Human-Robot Interaction (HRI)
in Medical Robotics
October 1, 2020

11. Energid Technologies Corporation
• 40-person team focused on development and integration of innovative, robotic software
• Founded in 2001, headquarters in Bedford, MA
• Acquired by Teradyne in 2018
• Moved under Universal Robotics on April 1, 2020
• Core business is licensing software (Actin) for aerospace, medical, energy, and industrial
applications
11
Engineering Resource Mix
~50% of team works remotely

12. Early Projects
• Energid got its start developing software for space robotics with high degrees of freedom
12

13. Actin: A Robotics Operating System
Simulation and control software for any
robotic system:
• Multi-robot coordination
• Dynamic collision avoidance
• Singularity avoidance
• Kinematically redundant mechanisms
• Complex kinematic chains
• Global path planning
• Real-time dynamic simulation
• IO and sensor feedback
• Easy integration of new hardware components
• Integration with CAD
• Desktop applications for Windows, Linux, OS X
• Control on VxWorks, Real-Time Linux, RTOS32, QNX
• Distributed processing over DDS
13

14. Natural Tasking in High DOF Systems
• Constrain only what is required for the task to allow for simpler programming
• Define motion constraints to satisfy the task, allow Actin to optimize the remaining degrees of freedom
• Create a series of motions to accomplish a task
14

15. Multi-Robot Systems Using DDS Communication
• Oil and gas mining
• Collaborative pick and place
15

16. Actin Surgical Applications
• Energid has worked with several companies to develop control
solutions and simulations for complex medical robotic
systems
• Traditionally performed by hand, there has been a lot of
development to perform laparoscopic surgeries with the help
of robots
16

17. Actin’s Product Development Workflow
17

18. Actin Simulation: Design
Actin simulation tools can load a model from a CAD format and automatically create a control system
• Using the Actin SolidWorks plugin, you can convert any mechanism for Actin control and simulation
• Mass properties can be taken from CAD for dynamic simulation
• Design optimization can be performed in Actin
18

19. Actin Simulation: Dynamics
Actin provides an accurate dynamic simulation capability
• Includes full and accurate Newton-Euler rigid body dynamics on all articulated links and impact
dynamics between obstacles
• Dynamics are calculated for nontraditional joint types as well.
• Both the Composite Rigid Body Inertia (CRBI) algorithm and the Articulated Body Inertia (ARBI) algorithm
are implemented
19

20. Actin Simulation: Preoperative Planning
As a preoperative planner, Actin can help surgeons leverage patient and procedure information to configure
their complex surgical robots before the procedure even begins
• Complex tool paths can be exported along with CAD
• Generate collision-free path to a goal location in the work (end-effector) space using RRT
20

21. Actin Control: Constraints and Optimization
The Actin control framework is built on a powerful motion constraint and optimization engine:
• Supports mechanical and virtual RCM
• Mechanical RCM is easy to control, but limits the workspace
• Virtual RCM uses software to impose the RCM constraint. This supports larger workspaces and the
opportunity to use the same robot for more varied procedures
21

22. Actin Control: Coordination
With Actin, surgeons are free to focus on what the robot does and where the hands and tools should be, not
on how they get there:
• Developed from the ground up to control many-axis systems (>7DOF)
• General inverse kinematics will work with any robotic arm without change in code
• Actin’s control algorithms can account for velocity, acceleration, and jerk limits
22

23. Actin Control: Performance and Hardware
The Actin SDK can be used with any type of robot or complex articulated mechanism:
• Actin uses DDS to enable cross-vendor coordination
• Actin runs on some of the top robotics systems in the world: Universal Robot, Motoman, Kuka, Han’s,
etc.
23

24. Actin Control: Dynamic Collision Avoidance
Actin doesn’t just stop its systems to avoid collisions but rather moves its robotic assets out of each others’
way in real time:
• Best-in-class algorithms for collision prevention and avoidance
• Continuous computation of proximity to collision that stops the system if a collision is imminent
• Optimize configuration of the arms to dynamically avoid collisions. The laparoscopic arm, for instance,
might move its elbow joint out of the way of one of the surgical arms while maintaining the camera
position
24

25. Summary
Energid develops software for simulation and control of any robotic system
• Actin provides a flexible programming system that allows user to naturally set tasks and manage
systems with large degrees of freedom
• Actin's extensible real-time control framework integrates motion constraints and optimizations, resulting
in dynamic robot response to changing environments
• Actin's simulation capabilities can be leveraged to test and validate your robotic systems before they are
even built, saving time and reducing risk
• Actin enables collaboration, coordination, and cooperation between multiple robots using DDS-based
communication
25

26. Dorothy Shamonsky, Ph.D.
Chief UX Strategy Officer
ICS & Boston UX

27. About ICS and Boston UX
27
Delivering Smart Devices for a Connected World
ICS
● Founded in 1987
● Provides full-stack medical, industrial and consumer development
● ICS Software Development Process (SDP) is 13485 (QMS) and 62304
(SDLC) compliant
● Largest source of independent Qt expertise in North America
● QNX reseller and service provider
● HQ in Waltham, MA with offices in California, Canada, Europe
Boston UX
● The innovative UX design studio of ICS
● Specialize in intuitive touchscreen and multimodal interfaces for high-impact
embedded and connected medical, industrial and consumer devices

28. A Few of Our Customers
28

29. Decreases training time
Increases ease of use and enhances user satisfaction
Improves task performance and optimal device use
Reduces use error and facilitates the recovery from use errors
Increases safety
Improves patient outcomes
Reduces product liability risks
Preempts device complaints
Facilitates the regulatory approval process
A Good User Experience Gives Your Product a
Competitive Advantage
29

30. 30
“How do I achieve a good user experience?”

31. What Usability Designers Love to Do
● Work with users
● Clarify user requirements
● Unify design
● Create consistency
● Make it comprehensible
● Make it intuitive
● Streamline and simplify
31
Old Interface New Interface

32. FOCUS: Work with Users
● Users are key in emerging tech - discover interactive patterns and best
practices
● Have techniques to work with users effectively - we manage the process
● Force a shift in focus to user-centric - emerging tech is by nature very tech-
centric
32

33. 1. Are you clear on your user requirements
(what your users need and want)?
33

34. 1. Are you clear on your user requirements
(what your users need and want)?
34
2. If you’re not clear, do you know how to find out what
they need and want?

35. 1. Are you clear on your user requirements
(what your users need and want)?
35
3. Do you know how to figure out what they really want and need,
which they aren’t necessarily telling you?
2. If you’re not clear, do you know how to find out what
they need and want?

36. Imagine this scenario...
36

37. Truth: if you take every suggestion from a
user literally you will NOT end up with a
product with good usability
37
USERS /= DESIGNERS
USER DATA /= DESIGNERS

38. 38
Patient outcomes...

39. Users are the key to unlocking good usability
39

40. Do you know your user requirements?
40

41. As a user I
want to:
At a high level, we already know
these answers
But we need to dig in to your
product for specifics
● Be safe!
● The interaction to feel as
intuitive and natural as
possible
● Do as little system
training as possible
● Experience no mistakes
or errors in using your
system
● Get my work done
without thinking about
your system
41

42. Best Practices for User Research - Over Time
Refinement of product - idea to delivery
42
Interview
&
Observe
Brainstor
m &
Discuss
Feedback
on Rough
Prototype
s
Feedback
on
Functiona
l
Prototype
Testing
on Alpha
Rigorous
Testing
on Beta
Rigorous
Testing in
Location
on Beta
Formative Testing Summative Testing
Observing & Discussing
Start early with users

43. Truth: Usability in Medical Devices and
Robotics is Very Challenging
43
EMERGING & COMPLEX

44. Mature Tech
Relatively mature UX practices
● Well established OS
platforms with interaction
conventions
● Best practices are proven
44
Emerging Tech
Relatively immature UX practices
● Tech is still changing, some
established interaction
conventions
● Best practices are emerging
1. Patterns & Best Practices Are Still Emerging
Desktop Web Mobile Embedded IoT Robotics AR/VR/XR

45. Mature Tech
Relatively mature UX practices
● Well established OS
platforms with interaction
conventions
● Best practices are proven
45
Emerging Tech
Relatively immature UX practices
● Some established interaction
conventions
● Best practices are emerging
1. Patterns & Best Practices Are Still Emerging
Desktop Web Mobile Embedded IoT Robotics AR/VR/XR
Discover new knowledge, patterns,
solutions
Rely on templates, patterns,
repeatable solutions

46. 46
● Screen interaction
● Situational awareness (view past and
present)
● Navigating in 3-D space (with a 2-D display)
● Multimodal - haptic, aural, speech, gesture
● Physical buttons, hand controllers, feet
controllers
● Ergonomic
● Affordances and constraints of movement
● Autonomy and AI
● Multi-user and/or multi-robot
● Risks and safety
2. Complexity of Multifaceted UX Requirements

47. 47
● Screen interaction
● Situational awareness (view past and
present)
● Navigating in 3-D space (with a 2-D display)
● Multimodal - haptic, aural, speech, gesture
● Physical buttons, hand controllers, feet
controllers
● Ergonomic
● Affordances and constraints of movement
● Autonomy and AI
● Multi-user and/or multi-robot
● Risks and safety
2. Complexity of Multifaceted UX Requirements

48. 48
Complexity of Multifaceted UX Requirements, cont.
● Screen interaction
● Situational awareness (view past and
present)
● Navigating in 3-D space (with a 2-D display)
● Multimodal - haptic, aural, speech, gesture
● Physical buttons, hand controllers, feet
controllers
● Ergonomic
● Affordances and constraints of movement
● Autonomy and AI
● Multi-user and/or multi-robot
● Risks and safety

49. You can discover
usability
knowledge by
working with
users
Best to work with a usability
professional
● User Research is technology-
agnostic
● User Research is a mature field
with established best practices
● There isn’t just one way to work
with users - many techniques
49

50. Thank You for Attending
Questions?
50