This document summarizes a master's thesis on developing a library for organizing image recognition systems. It outlines the thesis, which proposes a modular plug-in based system called AMORS that standardizes interfaces between acquisition, processing, and display modules. It then discusses applications of AMORS to automatic recognition of micro-objects, human brain cells (using segmentation, features and self-organizing maps), and counting bacteria. Finally, it describes the library developed containing classes to handle images, objects and configurations to support portability.

1. Library for organization of image
recognition systems
Master’s thesis
by
Oleksandr Shyrokov
http://www.ece.unh.edu/svpal/
Thesis adviser: Dr. Messner
May 2002

2. Outline
n Introduction
n Approach
n Automatic micro object recognition system
n Automatic human brain cell recognition
n Bacteria counting
n Library
n Conclusion

3. Where are we?
n Introduction
n Approach
n Automatic micro object recognition system
n Automatic human brain cell recognition
n Bacteria counting
n Library
n Conclusion

4. Introduction
n What is a Recognition System?
n Who needs it?
n What is done?
n What can be done?

5. Introduction (What)
n Acquire
– Get the information from the sensors (camera)
n Processing
– Enhance image; locate region of interests; get
additional information (features)
n Make a decision
– Label the situation
n React
– Perform a corresponding action

6. Introduction (Who)
n Industry
– High speed
– Low error rate
n Science
– Repetitive tasks (measurements,…)
n Hazardous environment
– Radiation, distance

7. Introduction (What is done)
n Dedicated companies
– Special hardware and software (high cost)
» Datacube
» Zeiss
n Software packages
– Matlab, mathcad, etc…
n Absence of the common convention
– Hard to reuse the work done by the others

8. Introduction (What can be done)
n Modularity
– Acquisition
– Object separation
– Feature extraction
– Classification
– Displaying
n Standard
– Predefined interface

9. Where are we?
n Introduction
n Approach
n Automatic micro object recognition system
n Automatic human brain cell recognition
n Bacteria counting
n Library
n Conclusion

10. Approach
n Human brain cell recognition research by
Mr. Pawlak (HBCRR)
– Matlab scripts
n Generalization
– Bacteria counting
n Plug-in concept
– Extendable system
n Interface definition
– Exported functions and parameters

11. Approach (HBCRR)
n Human brain cell recognition research Acquired data (tiff images)
– Acquisition
» Acquire data
Segmentation
– Object separation
» Locate cells
Polar transform
– Feature extraction
» Make polar transform
» More features Features calculation
– Classification
» Classify each cell Fuzzy classification
– Displaying
» Show the result Result

12. Approach (Generalization)
n Bacteria counting Acquired data (tiff images)
– Acquisition
» Acquire data
– Object separation Find bacterias
» Locate bacteria
– Feature extraction Find area of each
» Find area of each bacteria bacteria
– Classification
» Count bacteria Count bacterias
– Displaying
» Show the result
Result

13. Approach (Plug-ins)
n What should be implemented as a plug-in?
– Acquisition plug-in
– Object Separation plug-in
– Feature extraction plug-in
– Database plug-in
– Displaying plug-in

14. Approach (Interfacing)
n Acquisition
– List of variables
– List of images
n Segmentation
– List of bounding rectangles
n Features
– Add list of variables to the bounding rectangles
n Classification
– Uses list of variables for each object
n Displaying
– Access to all object data

15. Where are we?
n Introduction
n Approach
n Automatic micro object recognition
system (AMORS)
n Automatic human brain cell recognition
n Bacteria counting
n Library
n Conclusion

16. AMORS
n Main program
n System organization
n Modes of operation
– Normal mode (run-time mode)
– Research mode (active setup)
n Graphical User Interface (GUI)

17. AMORS (Program)
n Main Program
– Why AMORS
– Plug-ins
– GUI
– Configuration file
Main program
Setup Information, functions Execution
Setup, exectuon Execution
Configuration file Plug-ins Modes

18. AMORS (System)
n Three main blocks
– Acquisition Acquisition
» Acquisition plug-in
– Processing
» Object Separation plug-in Processing
» Feature extraction plug-in
» Classification plug-in
– Displaying Displaying
» Displaying plug-in

19. AMORS (Acquisition)
n Acquisition
Acqusition module
– Camera
Hardware
– Microscope
– XYZ translation stage Camera
Camera
manager
Microscope
XYZ stage
manager
XYZ stage
Acquisition
manager

20. AMORS (Processing)
n Processing
– Separation Processing module
» Find objects Object Separation Object
plug-in separation
– Feature extraction
» Multiple features FE Plug-in 1
… Feature
– Classification …. extraction(FE)
» Database FE Plug-in N
Database Classification

21. AMORS (Mode 1)
n Normal mode (Run-time)
– Minimum interaction with the user
Data Frames Processing Result Result
acquisition block displaying
Feedback (user input, automatic adjustments)

22. AMORS (Mode 2)
n Research mode (active setup)
– A user can find parameters of the plug-ins
Data Frames
User feedback
acquisition
Result New
displaying configuration
Processing User feedback
Results
block

23. AMORS (GUI 1)
n Main window
– Configure
– Setup
– Launch
– Observe progress

24. AMORS (GUI 1)
n Main window
– Configure
– Setup
– Launch
– Observe progress

25. AMORS (GUI 2)
n Configuration dialog box
– Unique identifier for plug-ins
– Plug-ins type is detected
– Ordering of feature plug-ins

26. Where are we?
n Introduction
n Approach
n Automatic micro object recognition system
n Automatic human brain cell recognition
(HBCR)
n Bacteria counting
n Library
n Conclusion

27. HBCR (Problem)
n Problem
– Huntington’s Disease (HD)
– Brain tissue
– Tracing cells
– Identification by experts
» Oligodendrocyte (Oligo)
» Astrocyte (Astro)
» Neuron (Neuro)
» Microgila
» Endothelial

28. HBCR (Cells)
Examples of cells:

29. HBCR (Original)
n Mr. Pawlak’s research (Matlab scripts)
– Image segmentation
» Region Growing method
– Polar transform
– Feature extraction
» Area, mean value, etc…
– Fuzzy logic
» Membership functions

30. HBCR (AMORS)
n Application for AMORS
– Acquisition
» DummyAcq plug-in for TIFF (Tagged Image File Format) files
– Image segmentation
» RegGrow plug-in
– Feature extraction
» Polar transform plug-in
» pawlak_features plug-in
– Classification
» IniDB plug-in for fuzzy logic
– Displaying
» SimpleDispaly plug-in

31. HBCR (Displaying 1)
n SimpleDisplay
plug-in

32. HBCR (Displaying 2)
n Histograms
– Image histogram
– Histogram for all numerical
parameters

33. HBCR (Displaying 3)
n Histogram region selection

34. HBCR (Improvements)
n Improvements
– 3D Acquisition
» Analyze stack of images taken with different focal
depths
n Voting
– Self-Organizing Map (SOM)
» Solves problem of defining the membership
functions

35. HBCR (SOM 1)
n Mapping from the input data space Rn onto
a regular 2D array of nodes
– Node i; parametric vector mi∈ Rn associated with node
i; Node location is in form (x,y);
– This map has the size 5 by 4
0,0 1,0 2,0 3,0 4,0
0,1 1,1 2,1 3,1 4,1
0,2 1,2 2,2 3,2 4,2
0,3 1,3 2,3 3,3 4,3

36. HBCR (SOM 2)
n Let me explain SOM using the following
example
– Need to separate cucumbers, tomatoes and watermelons
– We are given the area occupied by the vegetable and
the maximum distance, as shown below
Maximum distance
Maximum distance Maximum distance
Cucumber Tomato
Watermelon

37. HBCR (SOM 3)
n Training data
– One need a predefined data to train the map
Samples
Cucumber Watermelon
Maximum distance
Tomato
Area

38. HBCR (SOM 4)
n Learning: Initializing
– mi(0) is assigned randomly
0,0 1,0 3,0 4,0
2,0
3,1
Maximum distance
0,1 1,1 4,1
2,1
4,2
3,2
0,2 1,2
2,2 4,3
2,3
0,3 1,3 3,3
Area

39. HBCR (SOM 5)
n Learning: Adjusting the map
– Shifting the closest node with its neighborhood
closer to the input vector
mi (t + 1) = mi (t ) + hci (t ) ⋅ [ x(t ) − mi (t )]
t – discrete-time coordinate; i – node index; mi – node vector;
x – input vector; hci – so-called neighborhood kernel

40. HBCR (SOM 6)
n Learning: Neighborhood kernel
– Position dependence
» hci =h(||rc - ri ||;t), where rc∈R2 and ri∈R2 – radius vectors of
nodes c and i
» hci goes to 0 with increasing ||rc - ri||
» defines the “stiffness” of the “elastic surface”
– Time dependence
» get smaller with time
» learning rate
– Two steps
» Ordering
» Fine tuning

41. HBCR (SOM 7)
n Learning: Training result
0,0 1,0 3,0 4,0
2,0
0,0 1,0 3,0 4,0
2,0
3,1
Maximum distance
Maximum distance
0,1 2,1
1,1 3,1
0,1 1,1 4,1
3,2
4,2 4,1
2,1
4,2
3,2
4,3
0,2 1,2
2,2 4,3 1,2
0,2
3,3
2,3 2,2
0,3 1,3 3,3 2,3
0,3 1,3
Area Area

42. HBCR (SOM 8)
n Learning: Representation
– high dimensions do not allow the direct
presentation of mapping process
– nodes are forming 2D grid
0,0 1,0 2,0 3,0 4,0
0,1 1,1 2,1 3,1 4,1
0,2 1,2 2,2 3,2 4,2
0,3 1,3 2,3 3,3 4,3

43. HBCR (SOM 9)
n Learning: Map calibration
– Assigning labels of training samples to the
nodes ( C – cucumber, T – tomato, W –
watermelon)
C
0,0 C
1,0 C
2,0 W
3,0 W
4,0
C
0,1 C
1,1 W
2,1 W
3,1 W
4,1
T
0,2 T
1,2 T
2,2 W
3,2 W
4,2
T
0,3 T
1,3 T
2,3 3,3 4,3

44. HBCR (SOM 10)
n Learning: Quantization error
– finite number of nodes
– approximation of sample distribution
– quantization error – average difference between
the input vector and the best-match vector

45. HBCR (SOM 11)
n Using SOM for cell recognition
– SOM_PAK software package
» randinit – initializes the map for training
» vsom – trains the map
» vcal – labels the map vectors
» visual – finds best-matching nodes
» qerror – quantization error
– SOMDB plug-in
» Parameter setup for the training

46. HBCR (SOM 12)
n SOMDB plug-in
setup

47. HBCR (SOM 13)
n Parameters used for training
NameStep Ordering step Fine tuning
Number of iterations 1,000 10,000
Learning rate 0.05 0.01
Radius of training area 10 3
n Example with the grid size 12x8 nodes
N - Neuron cells, A - Astrocyte cells, O - Oligodendrocyte cells, * - unknown cells or clutter

48. HBCR (Future work)
n Work to be done
– Testing of the different separation methods, by
using different plug-ins
– Make analysis to find the best learning
parameters for SOM
– Features, which give maximum separation of
the cells

49. Where are we?
n Introduction
n Approach
n Automatic micro object recognition system
n Automatic human brain cell recognition
n Bacteria counting (BC)
n Library
n Conclusion

50. BC (Problem)
n Pure culture vs. industrial specimen
Industrial specimen Pure culture (inverted)

51. BC (Solution)
n Acquisition
– MaxPci
– Zeiss
– FromFileAcq plug-in
n Methods
– Threshold
– Morphology operations
– Clusters division

52. BC (Result)
n Accuracy (95 %) the same as for human
n Speed more then 5 times faster than for human
Input (inverted) Counted (bacteria)

53. BC (Future work)
n Work to be done
– Automatic acquisition
– Automatic thresholding
– Counting in the industrial specimens

54. Where are we?
n Introduction
n Approach
n Automatic micro object recognition system
n Automatic human brain cell recognition
n Bacteria counting (BC)
n Library
n Conclusion

55. Library
n Statically linked library
– interfaces.lib
n Portability
– Standard Template Library
n Classes
– CSJIniFile (configuration files)
– CSJBuffer (2D data)
– CSJImage (Color images)
– CSJImageList (List of images)
– CSJObjectInfo (Object description)
– CSJObjectInfoList (List of objects)
– CSJSpecimen (Description of the specimen)

56. Library (CD)
n Open sources
n Templates
n Examples
n Documentation
n Applications

57. Library (Future work)
n Work to be done
– Test for different operating systems
– Non -linear connection of the plug-ins
– Feedbacks
– Standard
– Code maintenance

58. Where are we?
n Introduction
n Approach
n Automatic micro object recognition system
n Automatic human brain cell recognition
n Bacteria counting (BC)
n Library
n Conclusion

59. Conclusion
n Ready-to-use procedure of creating the
recognition system
n Free
n Examples
n Templates
n Documentation

60. References
nA. J. Pawlak, "Automatic human brain cell
recognition", Master's Thesis, September
1998
n R.O. Duda, P.E. Hart, D.G. Stork, “Pattern
Calssification”, 2nd edition, Wiley-
Interscience Publications, 2000
n Full list online: http://www.ece.unh.edu/svpal/

61. The end
Questions, comments?
Maybe a little demo?
Thank you!