1. A New Augmented Reality
Interface for Game Based Stroke
TeleRehabilitation (AR-GBSTR)
Arzang Kasiri
Mentor: Walt Scacchi Ph.D. Donald Bren School of
Information and Computer Science
3. What is Augmented Reality
Augment reality is a means of human-computer interaction in which users can
perceive and interact with virtual objects that are juxtaposed with physical objects.
5. AR Stroke Rehabilitation - Related Research
Common themes among other AR + Stroke TeleRehabilitation are the use of large
and complex aparati, gloves, or body tracking with the Microsoft Kinect
Assistive Technology Rehabilitation Glove Kinect-Based Rehabilitation
6. AR Related Research - IoT Based Sensing Devices
Inertial Measurement
Unit Glove
Camera with AR-Marker VicoVR (Kinect-like device)
Leap Motion Intel RealSense
9. Assembling the Prototype - Sensor Placement
Leap Motion has two types of placements: Desktop and VR
Desktop Placement
VR Placement
VR Placement on Prototype
13. Formative Assessment
Inspected and critiqued by Doctor Steven C. Cramer and other stroke therapists in
the Cramer Research Lab at the Gross Stem Cell Research Center
Pros:
technology looks promising
technology will benefit from further improvement and refinement
Cons:
Problems with accuracy
Could be confusing for stroke patients
Recommendation is to pursue the dual-screen RealSense system alternative
14. Future Work
Following the assessment, we will pursue the different alternative
In the Fall, with access to new AR-IoT technologies, we will
study a dual screen system with Intel RealSense
Work will continue in the Fall in the Cramer Research Lab with Doctor Scacchi’s
mentorship
Goal: Repeat the initial AR study with the new system we will develop to see how
the new results compare
Editor's Notes
Earlier Work I Did
Multiplayer Simon
Emoji Chat
Setup a Python server to support both of these
Prior work - AR 1
Research into AR
different devices
different AR implementations
AR prototype
different models/features; reasons for features
the process of building it
the result (formative evaluation by Doc. Cramer)
Introduction
What is Augmented Reality
Why we use Augmented Reality in Stroke TeleRehabilitation
Research into AR
IoT devices
Other rehab AR implementations
AR prototype
different models/features; reasons for features
the process of building it
the result (formative evaluation by Doc. Cramer)
What is AR? AR is ...
We are incorporating it into GBSTR. GBSTR is ...
We are incorporating it because ...
TeleRehab: Rehab at home via a networked system
Rehab: repetetive motor control movements
Game Based Stroke TeleRehabilitation:
A fancy way of saying we are having stroke patients play games to regain control of their arms.
We’ve incorporating augmented reality because it has been found to result in increased benefits in gbstr.
What did this study find:
1st-person play can produce better results than 2nd-person play
Limitations:
limited functionality: only trained upper-arm fine motor control
unreliable
costly and not easy to reproduce; not scalable
“””
This is the first AR console developed, before I joined the project. <Explain how it works>
-- talk about result of study; talk about limitations --
So this is where I come in. I’m working to develop the next version of the AR system, one that is versatile and scalable. I will be doing this using IoT sensors, actuators, and controllers.
“””
Talk about...
common themes of ar in stroke
each is good at what they do, but they fall short of what we aim to do
“these three groups can be broken down further into two groups: systems that use devices designed specifically for stroke and those that repurpose consumer products...” -> our project has grown out of the kinect
assistive technology for stroke rehabilitation - specialized technology for stroke rehab: large complicated costly devices, glove devices
first two methods (gloves, large tech) are a holistic solution that supports a range of therapeutic movements, so we are building off of the third method, using non-native tech to provide a holistic solution to stroke rehab
<our goal is to make an upper-body focused, independently usable system that incorporates 1st person play and is scalable>
All of these are great at what they do, but they fall short of a more holistic solution. What we are aiming to accomplish is a holistic solution, on that supports as large a range of exercises as possible, can be used independently, and is scalable.
How does ours differ from the Kinect-based rehab games though? Some of them are upper-body-only games.
> Kinect systems offer potential but they have yet to be refine, also they haven’t been used in TELErehabilitaiton as far as we know
> I’d say ours differs in that it support fine and gross movements. I don’t believe the Kinect-only systems can support fine movements.
Talk about...
what each device is capable of doing
mention that head mounted AR are not possible/safe for stroke patients
each is a commercial product with full software support
(or in the case of camera + ar markers there are commercial options: Eon VR)
to note: these are commercial products with mature(?)/full software development kits. Because of this we can focus on GBSTR design challenges, rather than underlying hardware and software challenges.
AR1: custom prototype (hardware and software)
when talking about VicoVR, mention that the kinect has been officially discontinued
Talk about...
pros and cons of each system
1st person windowed type of play
why we chose the we chose
say we couldn’t build it because of
End: The one we think is best is... . We were constrained in what we had access to. <transition>
Include a model with music glove + imu?
To find the best option, we looked at different alternatives
<Talk about pros and cons of each model>
<explain why we selected the dual camera model>
<explain why we were limited to only a simplified version>
All of these are designed to support fine and gross motor control for the entire arm.
non-standard input devices + gesture tracking camera:
easy to assemble (hence scalable), but lacks full 1st person play and play using physical items
gesture tracking camera + optically sensed user input devices:
complicated to assemble, but supports full 1st person play and physical items
front facing and rear facing gesture tracking camera:
easy to assemble, but lacks physical items
(proprioception? no that’s the thing where you’re aware of what you’re doing outside of your field of view)
We only had access to a leap motion, tablet, and an arm. As a result we build this system
Talk about why we picked VR placement, rather than desktop
Leap motion has two types of sensor placements: desktop and vr. Because there is a 3 inch area in front of the sensor where it is too close to identify hands, and we want the those that use this to be able to work on the desk, we need to not have the sensor on the desk. I had tried setting up a few feet in front of the user facing them, but the leap had trouble tracking the hands. I believe it’s software is specifically designed to recognize hands from below and from behind.
We used a mature industry standard game development environment, Unity 5, that handles integration with AR software libraries. It takes care of the low level development device centric challenges, letting us focus design challenges.
The leap motion SDK supports development in the Unity. They have provided a variety of assets and scripts for using the Leap Motion with Unity.
I took the rig for hands from example scene provided by Leap Motion and used it in building a demo game
Talk about:
how I am performing a standard stroke motor control assessment: box and blocks
“there are limitations that I will discuss next”
hand accuracy was low
on the fringe of the camera view
when extrapolating what the camera could not see
identifying individual fingers when the rest were down (example: index & middle, pinky & ring)
flipping perspective (when a hand entered the camera’s view from above)
hands weren’t immediately picked up
Depth perception was confusing
Grabbing was difficult so we used pinch grab, but that isn’t intuitive (things would go flying off following some finger when the user let go)
