Haptic Playback Simulation

Haptic Playback
l b k
G.K. Ananthasuresh
Mechanical Engineering
M h i lE i i
Indian Institute of Science
Bangalore

Work done with
Shanthanu Chakravarthy
P. Muddukrishna
Avinash Kumar
Santosh D. Bhargav

2

Outline

• About rbccps
• Haptics and CPS
• The context
The context
– Tele‐operation with force feedback
– Micro to macro
Micro to macro
– From here to there
• Ha ti layba k
Haptic playback
– Endoscopy simulation and playback
• The details
• Main points G. K. Ananthasuresh, IISc., Mar., 2013

3

Robert Bosch Centre
for
Cyber Physical Systems
• A philanthropic grant from the Robert Bosch Foundation
(Rs. 11.5 crores per year for 10 years)
• Five verticals
– Agriculture; Buildings; Healthcare
– Transportation; Water
Transportation; Water
• 20 projects  Centre staff  50 faculty  Students
• Liaison with industries
Internships!

rbccps.iisc.ernet.in rbccps@gmail.com
Send e‐mail for summer and other internships with
“Internship” as the subject of the e‐mail.

G. K. Ananthasuresh, IISc., Mar., 2013

4
Cyber Surgery and Remote Patient Care
Investigators
Ashitava Ghosal, ME Dept., IISc, Bangalore
G K Ananthasuresh, ME Dept., IISc Bangalore
V Natarajan, CSA Dept., IISc Bangalore
Chandra Shekar Seelamantula, EE Dept., IISc Bangalore
Ch d Sh k S l l EE D IIS B l
B Gurumoorthy, CPDM, IISc Bangalore
Dibakar Sen, CPDM, IISc Bangalore

Dr. Pradeep Rebala, Asian Institute of Gastroenterology, Hyderabad
Dr. D Nageswara Reddy, Asian Institute of Gastroenterology, Hyderabad

Project Staff & Students
Dr. Parama Pal – RBCCPS, IISc Bangalore
Aditya K – ME Dept., IISc Bangalore
y p , g
Abin Jose – EE Dept., IISc Bangalore
Avinash Kumar – ME Dept., IISc Bangalore
Sarvesh Kolekar – CPDM, IISc Bangalore
S. Sagar – CPDM, IISc Bangalore
Rahul Sharma – CSA Dept., IISc Bangalore
Shanthanu Chakravarthy – ME Dept., IISc Bangalore G. K. Ananthasuresh, IISc., Mar., 2013

5

Cyber Surgery and Remote Patient Care
oscope

Virtual exploration of objects
Virtual exploration of objects WP3

ination
ator

Haptic simulation WP4
ble endo

c simula

nt exami

Haptic control WP5
ke flexib

doscopic

e patien

Interactive visualization
WP6
Snake‐lik

Heterogeneous material
Remote
End

representation WP7
robot

Image processing
I i
S

WP8

WP1 WP2 WP9

S
Snake‐lik
ke flexib
ble endo
oscope

WP1
robot

End
doscopic
c simula
ator

WP2
Remote
e patien
nt exami
ination

WP9
SMA‐activated

Silicone tube with nylon braid
6


S
Snake‐lik
ke flexib
ble endo
oscope

WP1
robot

End
doscopic
c simula
ator

WP2
Remote
e patien
nt exami
ination

WP9
7


8

Patient in the
ambulance
oscope

ination
ator
ble endo

c simula

nt exami
ke flexib

doscopic

e patien

Doctor in the
Snake‐lik

hospital
Remote
End
robot
S

WP1 WP2 WP9

9



Haptic control WP5

WP6

representation WP7

Image processing
I i WP8


10



Haptic control WP5

WP6

representation WP7

Image processing
I i WP8


11

Position Virtual exploration of objects


User Haptic control WP5

Haptic Interactive visualization
Device WP6

Force representation WP7
Input Path
Output Path
O t t P th Image processing
I i WP8


12



Haptic control WP5

WP6

representation WP7

Image processing
I i WP8


13



Haptic control WP5

WP6

representation WP7

Image processing
I i WP8


14



Haptic control WP5

WP6

representation WP7

Image processing
I i WP8


15

The Centre also has teeny weeny
ψφ projects.
 Desalination bottle
 Mobile diagnostic tool
 Breath sensor
Breath sensor
 Touch‐screen anywhere
 Granular flow sensor
Granular flow sensor
 Cognitive jewellery


16

Cognitive
Cognitive
Jewellery
Technology + Aesthetics + Fashion
N
Novel ways to interface with the physical world.
l t i t f ith th h i l ld
One can wear biomedical sensors in style.

With …
Anand P.
Pragathi M.
Dhruv S.
Deepika M. S.


17

Outline
• Haptics and CPS
• The context
– Tele‐operation with force feedback
p
– Micro to macro
– From here to there
From here to there
• Haptic playback
– Endoscopy simulation and playback
Endoscopy simulation and playback
• The details
• Main points

18

Micromanipulation with haptics
Miniature grippers to hold and
I Compliant
I   ・ Input ports Cell grasped x
O O
x
manipulate cells.
actuated by fine‐
micro with  two
motion stages
I mechanism
O  ・ Output ports in
contacts for
contacts for Haptic interface
interface.
manipulation
contact with cell
x
x  ・ Observation ports Mechanical characterization by
for tracking and force
x O computation Compliant solving inverse problems in
Cell probed with micro
a single contact Light source mechanis mechanics.
^y^
x^ z m I I
Tiltable arm
Gross
Fine motion
I I motion
stage
stage

z y Circular motion
x t
stage

Haptic interface
for human operator
Microscope
Controller

CCD camera

PC Image processing
Computation of forces and
displacements
I/O to controller and haptic interface


19

Micromanipulation station


Miniature grippers 20

that also sense forces
that also sense forces
Reddy, A. N., Sahu, D. K., Maheswari, N., and Ananthasuresh, G. K.,
“Miniature Compliant Grippers with Vision‐based Force‐sensing,” IEEE
Transactions on Robotics, Vol. 26, No. 5, pp. 867‐877, 2010.

Miniature gripper

Zebrafish egg in jaws

Gripper small than “1” in the one
rupee coin Squeezing HeLa cells G. K. Ananthasuresh, IISc., Mar., 2013

21

Piercing into a zebrafish egg cell


22

Master‐slave motion with force‐
feedback in tissue‐cutting
f db k i ti tti


23
Setup

Visual
display

Slave
device

Compliant cutting tool
p g
Phantom tissue
Ph i


24

MP 285

CMOS
camera

Scalpel


25

Passive control with a compliant end-effector
Anchored Output displacement Anchored

Rigid-body
translation of
the
plate (Xa)
l t Rigid plate of end-
1 2 effector casing

Y

X Ensuring cutting force (Fc)
Phantom tissue


26
Phantom tissue with a hard inclusion

Gelatin-based hydrogel

PDMS


27
Comparing rigid end-effector with
p
one that is compliant

Saturation of the
load-cell


28

Haptic Playback
Haptic Playback


29

Vision can be captured by camera , which can be viewed
later.
l t
Audio can be captured using sound recorder and
replayed later using a speaker

What about other senses?
Photo courtesy Santosh

Haptic replay

Force Haptic replay is difficult
Environment because of the duality of
force and displacement
Position


30

Endoscopy Playback
Playing back a procedure done in the past…

Snapshot from endoscope
image sensor
image sensor
 Case for haptic replay
 Palpation is one of the important modes of diagnosis and in current
endoscopy procedure, no force information is stored for later replay.
py p p y
 Important information like stiffness from earlier examination can be
replayed to the doctor to help in diagnosis.


31

Haptic playback with an
endoscopy simulator
Step one
Step one Step two
Step two Step three
Step three

Endoscopic session
Endoscopic session
Rough estimation of tissue
carried out using a
properties
modified endoscope.

User
Haptic
Device
Data acquisition from the Creating a reality‐based
instrumented endoscope
instrumented endoscope haptic model
haptic model
Doctors can explore the
virtual model to recall
what was experienced
earlier.


32

Virtual haptic model
Position

Collision Object Object
Sensor geometry DB
C Detection
O/P
H
Data Physical properties
A Collision information
Collision information
User I of Object

Physically based
3 simulation
Haptic Actuato D
(FEM)
Device r
I/P
Data Haptics Rendering
Force engine
g
Input Path
“The Haptics Loop”
Output Path

Object geometry, haptic robot workspace and graphics screen are
synchronized.

33

Force measurement

Modified
Falcon grip

Data
acquisition
acquisition
board

Force sensor
Force sensor


34

Position measurement
Both position and forces have to be measured
simultaneously
y


x
z

 x
 R ( ) y
T (t )   z 
 z
 
 0 0 0 1

Forward kinematics for position estimation Linear position measurement device

Rotations at every time instance is measured using
encoders


35

Rough estimation of tissue
properties for reality‐based
ti f lit b d
modelingg

 Fexp erimental 
Ei 1  Ei  
 FFEM 

Fexp erimental  FFEM  
Synthetic data
S h d


36
Local elastic modulus

Initial guess for E
I iti l f E

U from experiment

Nonlinear Finite Element
Method

Fexp erimental  FFEM  

 Fexp erimental 
Ei 1  Ei  
 FFEM  E(U) for the synthetic data
considered


37

Grid method for collision detection
 Impose  lookup grid inside the
bounding box containing the
bounding box containing the
polygon.

 Categorize  each cell  as being
Ca ego i e eac ce as ei g
fully inside, fully outside, or
indeterminate  with respect to the
p yg
polygon.

 For the indeterminate cells use
line crossing test to see if the point
is inside.
i i id

 Once collision is detected use
pre‐computed cell data to get the
pre computed cell data to get the
nearest node.
Haines, Point in Polygon Strategies, Graphics Gems IV, 1994.

38

Deformation mechanics
 Problem Definition:
   b  0 in 
U  0 on  fixed
   tr  E  I  2 E
U  U h on some portion of 

 Problem in discretized frame work:
Haptic DOF

 K11 K12 K13  U 1   F1 
     
 K 21 K 22 K 23  U 2    F2 
K K 33  U   F 
 31 K 32   3  3

Fixed DOF G. K. Ananthasuresh, IISc., Mar., 2013

39

Lagrange multiplier Method
 Optimization 1 T
Problem Min U KU - U T F Subj. to : U=Uh
j
U 2
 Equivalent Problem

1 T 1
Min U KU - U F  CU - U h
U, 2
T

2
 
where C isconstrained matrix
 isLagrange multiplier vector
g g p
 K CU  F 
CT 0    U 
    h 
Cook et al., Concepts and Applications of Finite Element Analysis. G. K. Ananthasuresh, IISc., Mar., 2013

40

Small area touch paradigm
p g
1
U  K C  F 
  T  U 
 C 0   h 
 Inverse of partitioned matrix
I f d
1K1   K1C CTK1C1  K1CT K1C 
 K C  
CT 0    1 
    K C  C K C 
1 T T 1
 
 Advantages with this method
• Solution is exact
• Lagrange multipliers are reaction forces
Lagrange multipliers are reaction forces
1
• Reaction forces are CT K 1C Uh
 

41

Large deformation model
g

Warping the stiffness along the rotation field

f elastic  R K ( R 1 x  X 0 )
Rotations are computed using geometric algebra
Volume expansion in linear simulation

R

x  X0

X0 R 1 x  X 0 R 1 x

42

Model reduction
 Suppressing interior DOF

 K 11 K 12 K 13  U1  F1 Zero
     
 K 21 K 22 K 23   Uint erior   Fint erior 
K
 31 K 32 K 33 

U
 3
 F
  3

 

Uint erior = - K 22-1K 21U1 - K 22-1K 23 U 3
i
 fixed
 Reduced stiffness matrix

 K 11  K 12K 22-1K 21 K 13  K12K 22-1K 23   U1  F1 
     
 K 31  K 32K 22 K 21 K 33  K 32K 22 K 23 
-1
1 -1
1
U 3  F3 

43
Path replay
Method
one

Force rendered to the user
using force‐displacement
model

Global deformation update computed using real‐time FEM

44

Region of attraction
Method
two

A controller draws you to
the points that were
i ea e eoe O e e
interacted before. Once the
point is reached the user
can experience the forces
that were experienced
before

Global deformation update computed using real‐time FEM

45
Controller
Disturbance from
human hand


Path to nearest 
snapping point PID Robot

Thorough evaluation of the
controller is still to be done


46

2D demonstration

Modified
Falcon grip

Data
acquisition
q
board

Force sensor
Exploration of a 2D stomach model using a
E l f D h d l
falcon device with modified grip

Both force and position information is read in the same simulation loop (Visual C++)


47

Haptic capture


48
Haptic playback


49

Main points
• Haptic playback
• Endoscopic simulator can be used for
haptic playback of endoscopy.
p p y py
• Challenges and issues
– Synchronous measurement of position and
Synchronous measurement of position and
force
– Real‐time large deformation analysis
Real time large deformation analysis
– Control
– The hardware
The hardware


50

Thank you.
Thank you

www.mecheng.iisc.ernet.in/~m2d2

The Multi disciplinary and
Multi‐disciplinary and
M2D2 Multi‐scale
Device and
group Design
D i


Haptic Playback Simulation

Recommended

Recommended

More Related Content

Recently uploaded

Recently uploaded (20)

Featured

Featured (20)

Haptic Playback Simulation