This document discusses methods for image restoration and denoising to enable object detection using a vision-based tactile sensor called FingerVision. It proposes two methods: FVIC which uses OpenCV functions like masking, restoration and denoising; and FVICNN which is based on a convolutional neural network. An experiment uses these methods to process noisy FingerVision images and test object detection accuracy using YOLOv3, finding that FVIC has slightly better results but is slower than FVICNN. The document concludes it is possible to obtain multimodal sensing through appropriate image processing for FingerVision.

1. Image Restoration and Denoising for
Simultaneous Tactile Sensing and Object
Detection Using FingerVision
Yamasaki, Kakeru
1

2. Yamasaki, Kakeru
Image Restoration and Denoising for Simultaneous Tactile Sensing and Object Detection Using Finger Vision
Vision-based tactile sensor
• Reproduces the sense of
touch by using a camera to
read the membrane that
changes when it comes into
contact with an object.
• FingerVision is a kind of
vision-based tactile sensors.
Structure
2

3. Yamasaki, Kakeru
Image Restoration and Denoising for Simultaneous Tactile Sensing and Object Detection Using Finger Vision
Feature
• Shape recognition
• Tactile data can be acquired as images.
• Object recognition
3
Vision-based tactile sensor
Yuan, Wenzhen, Siyuan Dong, and Edward H. Adelson. "Gelsight: High-resolution robot tactile sensors for estimating geometry and force." Sensors 17.12 (2017): 2762.

4. Yamasaki, Kakeru
Image Restoration and Denoising for Simultaneous Tactile Sensing and Object Detection Using Finger Vision
Sensor type
4
Vision-based tactile sensor
GelSight TacTip FingerVision
https://news.mit.edu/2014/fingertip-sensor-gives-robot-dexterity-0919 http://www.brl.ac.uk/researchthemes/medicalrobotics/tactip.aspx

5. Yamasaki, Kakeru
Image Restoration and Denoising for Simultaneous Tactile Sensing and Object Detection Using Finger Vision
GelSight
Vision-based tactile sensor
5
Yuan, Wenzhen, Siyuan Dong, and Edward H. Adelson. "Gelsight: High-resolution robot tactile sensors for estimating geometry and force." Sensors 17.12 (2017): 2762.

6. Yamasaki, Kakeru
Image Restoration and Denoising for Simultaneous Tactile Sensing and Object Detection Using Finger Vision
TacTip
Vision-based tactile sensor
6
https://softroboticstoolkit.com/tactip
Winstone, Benjamin, et al. "TACTIP—Tactile fingertip device, challenges in reduction of size to ready for
robot hand integration." 2012 IEEE International Conference on Robotics and Biomimetics (ROBIO). IEEE, 2012.

7. Yamasaki, Kakeru
Image Restoration and Denoising for Simultaneous Tactile Sensing and Object Detection Using Finger Vision
FingerVision
Vision-based tactile sensor
7

8. Yamasaki, Kakeru
Image Restoration and Denoising for Simultaneous Tactile Sensing and Object Detection Using Finger Vision
FingerVision vs. others
Vision-based tactile sensor
FingerVision can recognize objects directly.
8

9. Yamasaki, Kakeru
Image Restoration and Denoising for Simultaneous Tactile Sensing and Object Detection Using Finger Vision
FingerVision vs. others
Vision-based tactile sensor
FingerVision can recognize objects directly.
• If it's not pressurized, it can still detect slip.
• It can detect object.
9
http://akihikoy.net/notes/?project%2FFingerVision

10. 10

11. Yamasaki, Kakeru
Image Restoration and Denoising for Simultaneous Tactile Sensing and Object Detection Using Finger Vision
FingerVision vs. others
Vision-based tactile sensor
FingerVision can recognize objects directly.
• If it's not pressurized, it can still detect slip.
•It can detect object.
11
http://akihikoy.net/notes/?project%2FFingerVision

12. Yamasaki, Kakeru
Image Restoration and Denoising for Simultaneous Tactile Sensing and Object Detection Using Finger Vision
Object detection
FingerVision
• There is not object detection research that displays bounding boxes
and class probabilities in FV.
12
class probability
bounding box

13. Yamasaki, Kakeru
Image Restoration and Denoising for Simultaneous Tactile Sensing and Object Detection Using Finger Vision
Object detection issues
FingerVision
The silicon membrane can be the noise.
• Circle grid
• Re
fl
ection of light
13

14. Yamasaki, Kakeru
Image Restoration and Denoising for Simultaneous Tactile Sensing and Object Detection Using Finger Vision
Object detection methods
FingerVision
1. Learning models of object detection in images including circle grids
2. Use existing trained models after denoising
14

15. Yamasaki, Kakeru
Image Restoration and Denoising for Simultaneous Tactile Sensing and Object Detection Using Finger Vision
Object detection methods
FingerVision
1. Learning models of object detection in images including circle grids
2. Use existing trained models after denoising
We need data for the number of objects we want to recognize.
Existing models cannot be used available.
15

16. Yamasaki, Kakeru
Image Restoration and Denoising for Simultaneous Tactile Sensing and Object Detection Using Finger Vision
Object detection methods
FingerVision
1. Learning models of object detection in images including circle grids
2. Use existing trained models after denoising
We only need a data set for denoising and image restoration.
Existing models can be used.
16

17. Yamasaki, Kakeru
Image Restoration and Denoising for Simultaneous Tactile Sensing and Object Detection Using Finger Vision
Purpose
17
obtained from FingerVision
Denoising
Image restration
Object detection
FingerVison Image Restoration and denoising

18. Yamasaki, Kakeru
Image Restoration and Denoising for Simultaneous Tactile Sensing and Object Detection Using Finger Vision
Related works
Raindrop Image Restoration
18
Image restration
Shaodi You, Robby T Tan, Rei Kawakami, Yasuhiro Mukaigawa, and Katsushi Ikeuchi. Adherent raindrop mod- eling, detectionand removal in video.
IEEE transactions on pattern analysis and machine intelligence, Vol. 38, No. 9, pp. 1721–1733, 2015. (Yamashita Atushi+,2007)
Using multiple cameras with parallax
background difference method
(Shaodi You+,2015)
Modeling raindrops based on the laws of physics
(Takahashi Saki+,2017)
Image style transfer using CNN

19. Yamasaki, Kakeru
Image Restoration and Denoising for Simultaneous Tactile Sensing and Object Detection Using Finger Vision
• Based on OpenCV
19
Denoising and Image Restoration Methods
FVIC
(FingerVision Image Correction)
FVICNN
(FingerVision Inpainting Convolution Neural Network)
• Based on Convolution Neural
Network

20. Yamasaki, Kakeru
Image Restoration and Denoising for Simultaneous Tactile Sensing and Object Detection Using Finger Vision
Based on OpenCV
① Mask
② Restoration
③ Denoise
FVIC
20
①Mask
②Restoration
③Denoise

21. Yamasaki, Kakeru
Image Restoration and Denoising for Simultaneous Tactile Sensing and Object Detection Using Finger Vision
Based on OpenCV
① Mask
② Restoration
③ Denoise
Manual work
The cost is low because the
position of the dot does not
change once it is set.
FVIC
21
①Mask
②Restoration
③Denoise

22. Yamasaki, Kakeru
Image Restoration and Denoising for Simultaneous Tactile Sensing and Object Detection Using Finger Vision
Based on OpenCV
① Mask
② Restoration
③ Denoise
Fast Marching Method
FVIC
22
①Mask
②Restoration
③Denoise

23. Yamasaki, Kakeru
Image Restoration and Denoising for Simultaneous Tactile Sensing and Object Detection Using Finger Vision
OpenCV cv.inpaint(img,mask,3,cv.INPAINT_TELEA)
Normalized weighted average of
all the known pixels of the
neighborhood replace Missing
pixel.
Estimates the color of the
missing pixel to be repaired
using known pixels of the
neighborhood and gradients.
Fast Marching Method
23
https://docs.opencv.org/master/df/d3d/tutorial_py_inpainting.html

24. Yamasaki, Kakeru
Image Restoration and Denoising for Simultaneous Tactile Sensing and Object Detection Using Finger Vision
Based on OpenCV
① Mask
② Restoration
③ Denoise
Non-local means
fi
lter
FVIC
24
①Mask
②Restoration
③Denoise

25. Yamasaki, Kakeru
Image Restoration and Denoising for Simultaneous Tactile Sensing and Object Detection Using Finger Vision
OpenCV cv.fastNlMeansDenoisingColored()
Depending on the similarity, the
weight of the points that are
similar to the template is
increased and the weight of the
points that are not similar to the
template is decreased to correct
the pixels of interest.
Non-local means filter
25
https://www.slideshare.net/masayukitanaka1975/ssii2014

26. Yamasaki, Kakeru
Image Restoration and Denoising for Simultaneous Tactile Sensing and Object Detection Using Finger Vision
FVICNN
Based on Convolution Neural Network
• Since noise data can be obtained, supervised learning is used.
• Although there is a GAN in the generation model, we did not use it in
this report because it is di
ffi
cult to set parameters and the training
may not converge. (Future issues.)
26

27. Yamasaki, Kakeru
Image Restoration and Denoising for Simultaneous Tactile Sensing and Object Detection Using Finger Vision
• It is based on DnCNN, which
is widely used for denoising,
and SRCNN, which is used for
high resolution.
• 400epochs
• Batch size = 8
• Learning rate = 5e-3
FVICNN
27
i
Conv+ReLU
Kernel=(3, 3),filter=32 strides=(1, 1)
Conv+BatchNormalization+ReLU
Kernel=(3, 3),filter=32 strides=(1, 1)
×6
Conv+BatchNormalization+ReLU
Kernel=(9, 9),filter=32 strides=(1, 1)
Conv+BatchNormalization+ReLU
Kernel=(1, 1),filter=16 strides=(1, 1)
Conv+BatchNormalization+tanh
Kernel=(5, 5),filter=3 strides=(1, 1)
input
output
Based on Convolution Neural Network
480×640×3
480×640×3

28. Yamasaki, Kakeru
Image Restoration and Denoising for Simultaneous Tactile Sensing and Object Detection Using Finger Vision
FVICNN
• Prepare data without
membranes and data with
membranes.
• 10 randomly selected images
were used as test data.
• 29 images are saved for
validation data.(Hold-out)
Data set
28
×300
×300

29. 29

30. Yamasaki, Kakeru
Image Restoration and Denoising for Simultaneous Tactile Sensing and Object Detection Using Finger Vision
FVICNN
• Loss function
• mean squared error
• Optimizer
• Adam
• lr=0.005,β1=0.900,
β2=0.999
Learn result
30

31. Yamasaki, Kakeru
Image Restoration and Denoising for Simultaneous Tactile Sensing and Object Detection Using Finger Vision
Object detection
• In order to evaluate our
experiment, we use an object
detection model, YOLOv3.
• Class = 80 categories
• We selected banana class
from that category for
experiment.
YOLOv3
31

32. Yamasaki, Kakeru
Image Restoration and Denoising for Simultaneous Tactile Sensing and Object Detection Using Finger Vision
Accuracy Veri
fi
cation for FVIC and FVICNN
1. Process the noise data with
FVIC and FVICNN,
respectively.
2. Input no-noise data, data
obtained by FVIC and data
obtained by FVICNN to
YOLOv3.
Experiment
32
FVIC
FVICNN

33. Yamasaki, Kakeru
Image Restoration and Denoising for Simultaneous Tactile Sensing and Object Detection Using Finger Vision
FVIC and FVICNN output
33

34. Yamasaki, Kakeru
Image Restoration and Denoising for Simultaneous Tactile Sensing and Object Detection Using Finger Vision
Accuracy Veri
fi
cation for FVIC and FVICNN
• By images restoration and
denoising, only one of the
fi
ve
images could be used to detect
an object. FVIC processing has
slightly better detection
accuracy than FVICNN.
• processing time
FVIC 1.6[s],FVICNN 7.5[s]
Result
34

35. Yamasaki, Kakeru
Image Restoration and Denoising for Simultaneous Tactile Sensing and Object Detection Using Finger Vision
Discussions
• My research was shown that it is possible to detect objects with existing
models by using appropriate image processing, even if the circle grid is
embedded in the image.
• The object detection results depend greatly on the accuracy of image
restoration and denoising, and we con
fi
rmed that the accuracy is greatly
reduced compared to the case without noise.
• Since the number of datasets was only 600, a more e
ffi
cient method of
creating the dataset allows for the creation of a larger dataset, which leads
to improved accuracy.
35

36. Yamasaki, Kakeru
Image Restoration and Denoising for Simultaneous Tactile Sensing and Object Detection Using Finger Vision
Discussions
• Another possibility is to use a large data set such as CIFAR-10 to learn
image restoration and then adapt the model to FingerVision.
• Although the same camera was used for the two FingerVision systems
in this study, we believe that a clearer image can be obtained by using
a high-resolution camera to create the no noise data.
36

37. Yamasaki, Kakeru
Image Restoration and Denoising for Simultaneous Tactile Sensing and Object Detection Using Finger Vision
summary
• There has not been enough research on the task of object detection
using FingerVision.
• The most important point in this research is that there is a possibility
to obtain multimodal information (visual and haptic) with a single
inexpensive mechanism by using image restoration and denoising for
FingerVision.
37

38. Yamasaki, Kakeru
Image Restoration and Denoising for Simultaneous Tactile Sensing and Object Detection Using Finger Vision
Future Issues
• Implementation of the GAN
• Increase the dataset
• use a large dataset such as CIFAR-10 to learn image restoratin and
apply it to FingerVision
• High-resolution cameras to create no noising data
38