This paper presents a cognitive architecture for a camera robotic assistant aimed at providing the proper camera view of the operating area in an autonomous way. The robotic system is composed of a miniature camera robot and an external robotic arm. The camera robot is introduced into the abdominal cavity and handled by the external robot through magnetic interaction. The cognitive architecture is provided with a long-term memory, which stores surgical knowledge, behaviors of the camera and learning mechanisms, and a short-term memory that recognizes the actual state of the task and triggers the corresponding camera behavior. To provide the proper camera view, each state of the task is characterized by a Focus of Attention (FOA), defined by an object, a position of the object in the image, and a zoom factor. The architecture also includes a learning mechanism to take into account particular preferences of surgeons concerning the viewpoint of the scene. The architecture proposed is validated through a set of in-vitro experiments.

1. TOWARDS A COGNITIVE CAMERA
ROBOTIC ASSISTANT
I. Rivas-Blanco,
B. Estebanez, M. Cuevas-Rodriguez, E. Bauzano, V.F. Muñoz
System Engineering and Automation, University of Málaga, Spain
IEEE RAS/EMBS BIOROB 2014
August 14, 2014

2. INTRODUCTION
 Camera robotic assistant
– Robotic unit that holds the endoscope during a minimally invasive
surgery
– Direct control interfaces
 Intelligent camera robotic assistant
– Collaboration with surgeons in a natural and autonomous way
– Control interface emulating human communication
– Surgical knowledge for decision making
– Learning mechanism to improve the assistant behaviour
Cognitive architecture
Voice commands
Teleoperation

3. INTRODUCTION
 State of the art in autonomous camera robotic assistants
– Surgical tools tracking
– Markov chains to enhanced tools tracking by predicting tools
positon
– Modelling of surgical procedures to provide different viewpoints of
the endoscope for each state
 Our proposal
– To provide the proper camera view for each state of a surgical
procedure
– Each state is characterized by a Focus Of Attention (FOA), defined
by a position of an object in the image and an image zoom factor
– Learning mechanism to take into account particular preferences of
surgeons

4. COGNITIVE ARCHITECTURE
Long-term memory
Procedural memory Episodic memory
Short-term memory
Perceptual system HMI
PATIENT SURGEON
FOA generator Surgical state estimation
Camera motion
Semantic memory

5. SEMANTIC MEMORY
 Scenario knowledge
– Objects of the scenario
– Relation between objects
– State of the objects
 Surgical knowledge
– Surgical task model
S0
T01
S1 S2
T23
S3
T12
Manuevers
Manuever completed

6. COGNITIVE ARCHITECTURE
Long-term memory
Procedural memory Episodic memory
Short-term memory
Perceptual system HMI
PATIENT SURGEON
FOA generator Surgical state estimation
Camera motion
Semantic memory

7. PROCEDURAL MEMORY
 Knowledge of how to perform behaviours of the robot
– Function that outputs the camera motion to place an object O in a
particular position (r, d) in the image with a particular image zoom
– r and d are comprised between 0 and 1
Camera_motion = FOA (O, r, d, zoom)

8. COGNITIVE ARCHITECTURE
Long-term memory
Procedural memory Episodic memory
Short-term memory
Perceptual system HMI
PATIENT SURGEON
FOA generator Surgical state estimation
Camera motion
Semantic memory

9. EPISODIC MEMORY
 Learning mechanism
– User’s profile to take into account particular preferences of
surgeons
User
corrections
Camera viewDefault FOA parameters
User profile
User FOA parameters
HMI (voice commands)

10. COGNITIVE ARCHITECTURE
Long-term memory
Procedural memory Episodic memory
Short-term memory
Perceptual system HMI
PATIENT SURGEON
FOA generator Surgical state estimation
Camera motion
Semantic memory

11. SURGICAL STATE ESTIMATION
 Manuevers recognition using Hidden Markov Models (HMMs)
Surgical task
model
Recognition
system (HMMs)
Perceptual
system
Actual surgical
state
State transition
Maneuver completed
Tools data
Manuevers
library
Off-line

12. FOA GENERATOR
 FOA parameters for the actual surgical state
FOA generator
Actual surgical
state
FOA function
call
FOA parameters
Camera motion

13. IMPLEMENTATION
 Camera robotic assistant

14. EXPERIMENTS
 Experiment design
– Surgical task: suture

15. EXPERIMENTS
 In-vitro experiments
– 5 users perform a suture task with the robotic assistant
– Each user performs 6 trials:
• 3 trials using voice commands to control the camera
• 3 trials using the cognitive architecture to control the camera
 Objectives of the experiment
– To compare the number of voice commands and the performance
time with and without the cognitive architecture
– To validate the learning mechanism of the system

16. EXPERIMENTS

17. EXPERIMENTS: RESULTS
 First 3 trials (just voice commands)
 Last 3 trials (cognitive architecture)

18. CONCLUSIONS & FUTURE WORK
 Intelligent camera robotic assistant by implementing a cognitive
architecture to provide the optimal camera view during each state of a
surgical procedure
 Cognitive architecture based on two types of memory: a long-term
memory that stores the robot’s knowledge and a short-term memory for
decision making
 Learning mechanism to take into account particular preferences of
surgeons
 Results show that the cognitive architecture reduces the surgeon’s
workload during the procedure
 Future work will be focused at increasing the long-term memory to
provide more generalization to the architecture and an improvement of
the learning mechanism

19. THANKS FOR YOUR ATTENTION
QUESTIONS?