This document discusses computer vision techniques for microscopes. It describes the MiCRoN European Union project, which aims to develop computer vision tools to manipulate and track microscopic objects. It provides examples of hardware and software used in the project, including a camera, microscope, and the Mimas computer vision toolkit. Geometric hashing is presented as a technique for recognizing objects in 2D and 3D under different degrees of freedom.

1. Computer Vision for Microscopes -1- Modelling Research Centre, 16th Dec 2005
Computer Vision for Microscopes
Microsystem & Machine Vision Laboratory
16th Dec 2005
MMVL http://vision.eng.shu.ac.uk/mmvl/ (vision/www)
MRC http://www.shu.ac.uk/research/meri/modelling_rc.html
MERI http://www.shu.ac.uk/research/meri/
SHU http://www.shu.ac.uk/
Microsystem & Machine Vision
Sheffield Hallam University
Laboratory

2. Computer Vision for Microscopes -2- Modelling Research Centre, 16th Dec 2005
MiCRoN
European Union IST project
Uppsala, Lausanne, St. Ingbert, Athens, Pisa, Barcelona, Karlsruhe
project goals
• Manipulate µm-sized
objects
• Closed-loop control of
robots
• 3D object recognition
and tracking
image acquisition
• micro-camera
http://wwwipr.ira.uka.de/~micron/ • microscope
http://www.cordis.lu/ist/
Microsystem & Machine Vision
Sheffield Hallam University
Laboratory

3. Computer Vision for Microscopes -3- Modelling Research Centre, 16th Dec 2005
MiCRoN
Microsystem & Machine Vision
Sheffield Hallam University
Laboratory

4. Computer Vision for Microscopes -4- Modelling Research Centre, 16th Dec 2005
MiCRoN hardware (i)
camera
Microsystem & Machine Vision
Sheffield Hallam University
Laboratory

5. Computer Vision for Microscopes -5- Modelling Research Centre, 16th Dec 2005
test hardware (i)
test environment
Microsystem & Machine Vision
Sheffield Hallam University
Laboratory

6. Computer Vision for Microscopes -6- Modelling Research Centre, 16th Dec 2005
MiCRoN hardware (ii)
first pictures
video
Microsystem & Machine Vision
Sheffield Hallam University
Laboratory

7. Computer Vision for Microscopes -7- Modelling Research Centre, 16th Dec 2005
test hardware (ii)
microscope environment
camera sensor actors
Microsystem & Machine Vision
Sheffield Hallam University
Laboratory

8. Computer Vision for Microscopes -8- Modelling Research Centre, 16th Dec 2005
software
Mimas computer vision toolkit
open source computer vision library
deliver solution ⇐ restrict vision domain
⇐ develop reusable toolset
• Sensor data: V4L, gstreamer-interface
• Segmentation: LSI-filters, morphology, disparity-estimator, DFT,
image-processing
• Feature-extraction: edges, corners, ...
• Feature-matching: optic flow, SVD-correspondence, correlation, champfer
matching, PGH, fast POL
• pose-estimation: particle filter, hough transform
http://vision.eng.shu.ac.uk/mediawiki/index.php/Mimas
Microsystem & Machine Vision
Sheffield Hallam University
Laboratory

9. Computer Vision for Microscopes -9- Modelling Research Centre, 16th Dec 2005
Canny-like edge detector
video edges
Microsystem & Machine Vision
Sheffield Hallam University
Laboratory

10. Computer Vision for Microscopes - 10 - Modelling Research Centre, 16th Dec 2005
geometric hashing
1988, Lamdan & Wolfson
Microsystem & Machine Vision
Sheffield Hallam University
Laboratory

11. Computer Vision for Microscopes - 11 - Modelling Research Centre, 16th Dec 2005
geometric hashing
preprocessing recognition
Microsystem & Machine Vision
Sheffield Hallam University
Laboratory

12. Computer Vision for Microscopes - 12 - Modelling Research Centre, 16th Dec 2005
geometric hashing
2-D / 2 DOF
Microsystem & Machine Vision
Sheffield Hallam University
Laboratory

13. Computer Vision for Microscopes - 13 - Modelling Research Centre, 16th Dec 2005
geometric hashing
2-D / 2 DOF
 
t1  
 
t2  = s − m 1
  VM (  , m)
1
0
s
0 5 5
1
3
m
m
Microsystem & Machine Vision
Sheffield Hallam University
Laboratory

14. Computer Vision for Microscopes - 14 (i) - Modelling Research Centre, 16th Dec 2005
geometric hashing
2-D / 3 DOF
scene features
Microsystem & Machine Vision
Sheffield Hallam University
Laboratory

15. Computer Vision for Microscopes - 14 (ii) - Modelling Research Centre, 16th Dec 2005
geometric hashing
2-D / 3 DOF
scene features hash table
pose (3 DOF)
misses hits
80
120
Microsystem & Machine Vision
Sheffield Hallam University
Laboratory

16. Computer Vision for Microscopes - 14 (iii) - Modelling Research Centre, 16th Dec 2005
geometric hashing
2-D / 3 DOF
scene features hash table
pose (3 DOF)
misses hits
80 10
120 190
Microsystem & Machine Vision
Sheffield Hallam University
Laboratory

17. Computer Vision for Microscopes - 14 (iv) - Modelling Research Centre, 16th Dec 2005
geometric hashing
2-D / 3 DOF
scene features hash table
pose (3 DOF)
misses hits
80 10 50
120 190 150
Microsystem & Machine Vision
Sheffield Hallam University
Laboratory

18. Computer Vision for Microscopes - 14 (v) - Modelling Research Centre, 16th Dec 2005
geometric hashing
2-D / 3 DOF
scene features hash table
pose (3 DOF)
misses hits
80 10 50 200∗
120 190 150 0∗
Microsystem & Machine Vision
Sheffield Hallam University
Laboratory

19. Computer Vision for Microscopes - 15 - Modelling Research Centre, 16th Dec 2005
geometric hashing
2-D / 3 DOF
test on artificial sequence test on pipette
Microsystem & Machine Vision
Sheffield Hallam University
Laboratory

20. Computer Vision for Microscopes - 16 - Modelling Research Centre, 16th Dec 2005
geometric hashing
focus stacks as model database
Microsystem & Machine Vision
Sheffield Hallam University
Laboratory

21. Computer Vision for Microscopes - 17 - Modelling Research Centre, 16th Dec 2005
geometric hashing
3-D/4 DOF
test on povray sequence (time lapse)
Microsystem & Machine Vision
Sheffield Hallam University
Laboratory

22. Computer Vision for Microscopes - 18 - Modelling Research Centre, 16th Dec 2005
bounded hough transform
2001/2004, Greenspan, Shang & Jasiobedzki
Microsystem & Machine Vision
Sheffield Hallam University
Laboratory

23. Computer Vision for Microscopes - 19 - Modelling Research Centre, 16th Dec 2005
bounded hough transform
2001/2004, Greenspan, Shang & Jasiobedzki
Microsystem & Machine Vision
Sheffield Hallam University
Laboratory

24. Computer Vision for Microscopes - 20 - Modelling Research Centre, 16th Dec 2005
bounded hough transform
inverse transformation
Microsystem & Machine Vision
Sheffield Hallam University
Laboratory

25. Computer Vision for Microscopes - 21 - Modelling Research Centre, 16th Dec 2005
bounded hough transform
houghspace for relative shifts
Microsystem & Machine Vision
Sheffield Hallam University
Laboratory

26. Computer Vision for Microscopes - 22 - Modelling Research Centre, 16th Dec 2005
bounded hough transform
focus stacks for 4 DOF
Microsystem & Machine Vision
Sheffield Hallam University
Laboratory

27. Computer Vision for Microscopes - 23 - Modelling Research Centre, 16th Dec 2005
software architecture
application layers
Microsystem & Machine Vision
Sheffield Hallam University
Laboratory

28. Computer Vision for Microscopes - 24 - Modelling Research Centre, 16th Dec 2005
software architecture
UML static structure
Microsystem & Machine Vision
Sheffield Hallam University
Laboratory

29. Computer Vision for Microscopes - 25 - Modelling Research Centre, 16th Dec 2005
software architecture
UML collaboration diagram
Microsystem & Machine Vision
Sheffield Hallam University
Laboratory

30. Computer Vision for Microscopes - 26 - Modelling Research Centre, 16th Dec 2005
command-line tool
Microsystem & Machine Vision
Sheffield Hallam University
Laboratory

31. Computer Vision for Microscopes - 27 - Modelling Research Centre, 16th Dec 2005
tests
syringe-chip
video
Microsystem & Machine Vision
Sheffield Hallam University
Laboratory

32. Computer Vision for Microscopes - 28 - Modelling Research Centre, 16th Dec 2005
tests
gripper
video
Microsystem & Machine Vision
Sheffield Hallam University
Laboratory

33. Computer Vision for Microscopes - 29 - Modelling Research Centre, 16th Dec 2005
tests
gripper2
video
Microsystem & Machine Vision
Sheffield Hallam University
Laboratory

34. Computer Vision for Microscopes - 30 - Modelling Research Centre, 16th Dec 2005
tests
full automation
video video
Microsystem & Machine Vision
Sheffield Hallam University
Laboratory

35. Computer Vision for Microscopes - 31 - Modelling Research Centre, 16th Dec 2005
MiCRoN setup (i)
Microsystem & Machine Vision
Sheffield Hallam University
Laboratory

36. Computer Vision for Microscopes - 32 - Modelling Research Centre, 16th Dec 2005
MiCRoN setup (ii)
Microsystem & Machine Vision
Sheffield Hallam University
Laboratory

37. Computer Vision for Microscopes - 33 - Modelling Research Centre, 16th Dec 2005
MiCRoN setup (iii)
Microsystem & Machine Vision
Sheffield Hallam University
Laboratory

38. Computer Vision for Microscopes - 34 - Modelling Research Centre, 16th Dec 2005
MiCRoN setup (iv)
Microsystem & Machine Vision
Sheffield Hallam University
Laboratory

39. Computer Vision for Microscopes - 35 - Modelling Research Centre, 16th Dec 2005
MiCRoN
assembly mockup
video
Microsystem & Machine Vision
Sheffield Hallam University
Laboratory

40. Computer Vision for Microscopes - 36 - Modelling Research Centre, 16th Dec 2005
MiCRoN
assembly mockup
video
Microsystem & Machine Vision
Sheffield Hallam University
Laboratory

41. Computer Vision for Microscopes - 37 - Modelling Research Centre, 16th Dec 2005
demonstration
macro-scale demonstration
Microsystem & Machine Vision
Sheffield Hallam University
Laboratory

42. Computer Vision for Microscopes - 38 - Modelling Research Centre, 16th Dec 2005
acknowledgements
• Jon Travis: technical suggestions, administrative support
• Fabio Caparrelli: camera drive/electronics/driver software, electronics setup,
PI driver software, management
• Balasundram Amavasai: proposal, computer vision suggestions, mimas
long-term strategy
• Arul Selvan: students supervision, telecentric optics, shift estimation
• Manuel Boissenin: realtime tracking of multiple micro-objects, parallelisation
• Jan Wedekind: software architecture and integration, realtime recognition of
multiple micro-objects
Microsystem & Machine Vision
Sheffield Hallam University
Laboratory

43. Computer Vision for Microscopes - 39 - Modelling Research Centre, 16th Dec 2005
Microsystem & Machine Vision
Sheffield Hallam University
Laboratory

44. Computer Vision for Microscopes - 40 - Modelling Research Centre, 16th Dec 2005
Microsystem & Machine Vision
Sheffield Hallam University
Laboratory

45. Computer Vision for Microscopes - 41 - Modelling Research Centre, 16th Dec 2005
Microsystem & Machine Vision
Sheffield Hallam University
Laboratory

46. Computer Vision for Microscopes - 42 - Modelling Research Centre, 16th Dec 2005
Microsystem & Machine Vision
Sheffield Hallam University
Laboratory

47. Computer Vision for Microscopes - 43 - Modelling Research Centre, 16th Dec 2005
Happy Christmas!
http://vision.eng.shu.ac.uk/mediawiki/
Upcoming: MiCRoN public final report
Microsystem & Machine Vision
Sheffield Hallam University
Laboratory