Image signal processing: developing adequate solutions for high-demanding applications. A case study presented by Verhaert on Innovation Day 06

1. IMAGE SIGNAL PROCESSING
VERHAERTINNOVATIONDAY – OCTOBER 20th, 2006
www.mastersininnovation.com
IMAGE SIGNAL PROCESSING
Frederik Wouters
frederik.wouters@verhaert.com
www.verhaert.com
Commercially confidence – This presentation contains ideas and information which are proprietary of VERHAERT, Masters in Innovation®*, it is given in confidence. You are authorized to
open and view the electronic copy of this document and to print a single copy. Otherwise, the material may not in whole or in part be copied, stored electronically or communicated to third
parties without prior agreement of VERHAERT, Masters in Innovation®*.
* VERHAERT, Masters in Innovation is a registered trade name of Verhaert Consultancies N.V.
www.mastersininnovation.com
20.10.2006 Slide 1

2. IMAGE SIGNAL PROCESSING
Agenda
www.mastersininnovation.com
Application System Definition Management
& Challenges
Market
Spec’s & Technical
Costs
constraints solutions
20.10.2006 Slide 2

3. IMAGE SIGNAL PROCESSING
Product Development
www.mastersininnovation.com
Integrated Control
Sensors Systems
Mechanisms & Cabinets &
Robotics Housings
20.10.2006 Slide 3

4. IMAGE SIGNAL PROCESSING
Application & Market
www.mastersininnovation.com
- 1m/s
- Working temperature range 15 – 40°C
Movement tire ± 1 cm
-
Req. precision < 20 µm
-
20.10.2006 Slide 4

5. IMAGE SIGNAL PROCESSING
Specification Definition
www.mastersininnovation.com
1.0508 ⋅ 10 −7 ⋅ x 3 − 7.8461 ⋅ 10 −5 ⋅ x 2 + 1.0339 ⋅ 10 −2 ⋅ x + 166.6760
20.10.2006 Slide 5

6. IMAGE SIGNAL PROCESSING
Specification Definition
www.mastersininnovation.com
Range Precision Repeatability (2 sigma)
Toe +/- 10° +/- 0,5’ +/-0,1’
Camber +/- 10° +/- 1’ +/-0,1’
Caster 0°-10° +/- 3’ +/- 1’
SAI 0°-20° +/- 0,3° +/- 0,2°
20.10.2006 Slide 6

7. IMAGE SIGNAL PROCESSING
Specification Definition
- Pricing Pricing
www.mastersininnovation.com
8.000
6.000
Processing
Euro
Camera
4.000
Total
2.000
Target pricing
0
Giga Ethernet Firewire
- Options
- 2 x DSP with IEEE 1394 -> custom design
- 1 x Floating point DSP + 1 FPGA with IEEE 1394 -> COTS
- 1 x Fixed point DSP + 1 FPGA with IEEE 1394 -> custom design
- ...
20.10.2006 Slide 7

8. IMAGE SIGNAL PROCESSING
Thermal Challenges
- Thermal expansion of distance CCD – light source i.e.
- Normal distance of 250 mm
www.mastersininnovation.com
- Working temperature range 15 – 40°C
- I.e. 3 micron error introduced
20.10.2006 Slide 8

9. IMAGE SIGNAL PROCESSING
Optical Challenges
- Optical errors
www.mastersininnovation.com
- Speckle
- Optical elements
- Aberration
- Non linearity
20.10.2006 Slide 9

10. IMAGE SIGNAL PROCESSING
Processing Challenges
- Hard Realtime vs. Soft Realtime
www.mastersininnovation.com
Carnegie Mellon University RTOS Real-time Performance
18-849b Dependable Embedded Systems by John A. Carbone, VP,
Kanaka Juvva Marketing
http://www.ece.cmu.edu/~koopman/des_s99/real_time/ Express Logic, Inc
20.10.2006 Slide 10

11. IMAGE SIGNAL PROCESSING
System Definition
- Hard Realtime vs. Soft Realtime
www.mastersininnovation.com
- guaranteed worst-case response times
- 1m/s 30 fps (1image 10 Mb per 33,3 ms)
- 1280*1024 , 8 bit, 30 fps 300 Mbps 1.0508 ⋅ 10 −7 ⋅ x 3 − 7.8461 ⋅ 10 −5 ⋅ x 2 + 1.0339 ⋅ 10 −2 ⋅ x + 166.6760
- 300 Mbps 5 x fixed point
32 bit integer
20.10.2006 Slide 11

12. IMAGE SIGNAL PROCESSING
Algorithms
www.mastersininnovation.com
20.10.2006 Slide 12

13. IMAGE SIGNAL PROCESSING
Algorithms
- Noise filtering - Smoothing
www.mastersininnovation.com
- Binning - Denoising
50
50
- Compensate for saturation
100
100
150
150
- Erode - Decrease noise
200
200
(Shrinking object)
250
250
- Dilate - Increase signal
300
300
(Object growing)
350
400
400
- Multiply - AND operation
450
450
50 100 150 200 250 300 350 400 450
50 100 150 200 250 300 350 400 450
- Classification - Which points belong together ?
- Fitting - Polynominal fit
20.10.2006 Slide 13

14. IMAGE SIGNAL PROCESSING
Algorithms
- Dilate & Erode
www.mastersininnovation.com
20.10.2006 Slide 14

15. IMAGE SIGNAL PROCESSING
Algorithms
- Noise filtering - fixed, non recursive, line based
www.mastersininnovation.com
- Binning - fixed point or integer, no recursive, line based
- Erode - fixed point or integer, no recursive, line based
- Dilate - fixed point or integer, no recursive, line based
- Multiply - binary operation, pixel based
- Classification - recursive search algorithm, pixel based
- Fitting - floating point
20.10.2006 Slide 15

16. IMAGE SIGNAL PROCESSING
Processing
Matlab - CPU Power
www.mastersininnovation.com
Noise Filtering
Bin by thresholding
Bin by local maxima Comparison of Processing speed
Erode & Dilate
1000.00
Multiplication
Classify
Polynomial fit
100.00
Seconds
Log
10.00
1.00
MATLAB - PC Single DSP Dual DSP DSP &
PC FPGA
20.10.2006 Slide 16

17. IMAGE SIGNAL PROCESSING
Processing Requirements
- MMACS – Million Multiply and Accumulate operations per second
(example)
www.mastersininnovation.com
- Noise filtering convolution with 16 x 1 Byte window filter
- 1280 * 1024 @ 30 fps
- I.e. 1280 * 1024 * 30 * 16 = 630 MMAC’s per second.
- Floating Point Multipliers in hardware
- PC & ARM: 1 multiplier
- DSP: upto 6 multipliers
- FPGA: built to requirement
- Test of noise filtering on PC P4 2.8GHZ, 1Gbyte RAM
- Processing time = [13.22 – 18.78] ms for a 492x460 image
- I.e. for a 1280*1024 image will this be about 100ms.
20.10.2006 Slide 17

18. IMAGE SIGNAL PROCESSING
PowerPC
- 4000 - 8000 MFLOPS
- 1 Floating point multiplier
www.mastersininnovation.com
- 1 Fixed point multiplier
- RAM access upto 128 bit @ 200 Mhz
20.10.2006 Slide 18

19. IMAGE SIGNAL PROCESSING
Floating Point DSP
- 1800 – 3600 MFLOPS
www.mastersininnovation.com
- Upto 6 Floating point multipliers @ 32 bit
- 2 Fixed point multipliers
- RAM access 32 bit @ 100 Mhz
- 2 MB (TI) upto 3 MB (Analog) memory onboard
20.10.2006 Slide 19

20. IMAGE SIGNAL PROCESSING
Fixed Point DSP
- 3200 – 8000 MIPS
- 4000 MMACs @ 32 bit
www.mastersininnovation.com
- 2 MB (TI) upto 3 MB memory onboard (Analog)
- RAM access 64 bit @ 133 Mhz (TI)
20.10.2006 Slide 20

21. IMAGE SIGNAL PROCESSING
Memory requirements
- Bandwidth to External Memory - Theoretical limits:
www.mastersininnovation.com
- ARM interface to external RAM is 32 bit @ 400 Mhz
- Floating Point DSP interface to external RAM is 32 bit standard @ 100MHz:
- 3200Mbps: 320 images/sec can be read or written to the external memory. At 30fps:
(320/30)/ 2 (read/write) = 5 image handlings per cycle.
- Fixed Point DSP interface to external RAM is 64 bit standard @ 133MHz:
- 8512 Mbps: 851 images/sec can be read or written to the external memory. At 30fps
(851/30) / 2 (read/write) = 14 image handlings per cycle.
- FPGA interface external RAM is DDR2 32 bit @ 400 Mhz
- 12800Mbps: 1280 images/sec can be read or written to the external memory. At 30fps
(12800/30)/ 2 (read/write) = 21 image handlings per cycle.
20.10.2006 Slide 21

22. IMAGE SIGNAL PROCESSING
Processing selection
- DSP for floating operations
www.mastersininnovation.com
- Algorithm steps for FPGA parallel implementation :
- Denoising with filter, binning steps and erode & dilate image processing
- AND functions of 2 or more images
- (optionally) classification 1000.0
100.0
MATLAB
10.0
PC
DSP
1.0
Gain of 10 ms due to fixed DSP & FPGA
l
it
g g e y
n ta
a
sif ial f
lat atio
r in ldin xim To
Di as
lte o c
a
Fi Cl m
li
sh no
l m d e & ltip
op’s in FPGA ise th re oca oly
u
o
No b y Er M P
l
by
in n
Bi
B
20.10.2006 Slide 22

23. IMAGE SIGNAL PROCESSING
Management
- Trade off Performance – Cost – Development Risk
- Algorithm vs. Processing hardware
www.mastersininnovation.com
- Camera’s: 1 Mpixels, 2 Mpixels
- More precision in algorithms & calibration
- More subpixel resolution 1/25 1/50
less smoothing, better peak detectors, better fittings
more floating point operations, more MMACs required, ....
20.10.2006 Slide 23

24. IMAGE SIGNAL PROCESSING
Management
- Trade off Performance – Cost – Development Risk
- Frame rate vs. CPU power
www.mastersininnovation.com
- Camera’s: 15 fps 30 fps 60 fps
- Doubling required processing power, ....
- More statistical algorithms that improve accuracy
20.10.2006 Slide 24

25. IMAGE SIGNAL PROCESSING
Management of Development
www.mastersininnovation.com
- Multi core DSP vs. DSP and FPGA
- COTS image processing board vs. Custom image processing board
- TI, meanwhile, has a roadmap in place to quickly move the 'C6x floating platform
to 3,000 MMAC, said Rick Rienhart, 'C6000 product line manager at TI in
Houston.
- TI expects to produce devices achieving speeds of 3 trillion instructions per
second by 2010.
- In the future processing power will as good as for for free (More’s Law)
20.10.2006 Slide 25

26. IMAGE SIGNAL PROCESSING
Conclusions
www.mastersininnovation.com
- Choice of DSP + FPGA on COTS board
- Waiting for DSP implementations with GMAC processing
- Algorithms is mathematics
- Manage development risks by early stage breadboarding.
- System cost impacts earning capabilities in our prior application and its transfer
opportunties to other market applications, thus interact with business
development from the start of the program.
20.10.2006 Slide 26

27. IMAGE SIGNAL PROCESSING
www.mastersininnovation.com
Verhaert New Products & Services nv
Hogenakkerhoekstraat 21
9150 Kruibeke
Belgium
Tel +32 (0)3 250 19 00
Fax +32 (0)3 254 10 08
www.verhaert.com
info@verhaert.com
www.mastersininnovation.com
20.10.2006 Slide 27