Polarization Image Processing with Scilab

1. 1www.esi-group.com
Copyright © Fraunhofer IISCopyright © Fraunhofer IIS
www.esi-group.com
Schöberl et al, Polarization Image Processing with Scilab, Scilab Conference, Nov 2018
Polarization
Image Processing
with Scilab
K. Kasnakli, S. Freitag, M. Schöberl
2018-11-20

2. 2www.esi-group.com
Copyright © Fraunhofer IIS
Agenda
• Polarization of light
• Measuring Polarization
• Algorithm development
• From development to product
• ARGO project
• Demo 2
Schöberl et al, Polarization Image Processing with Scilab, Scilab Conference, Nov 2018

3. 3www.esi-group.com
Copyright © Fraunhofer IIS
Properties of Light
Dimensions of Electromagnetic Radiation
Light is an electromagnetic wave, with
• Amplitude perceived as brightness
• Wavelength perceived as color
• But the orientation of the
wave oscillation is not visible
Schöberl et al, Polarization Image Processing with Scilab, Scilab Conference, Nov 2018

4. 4www.esi-group.com
Copyright © Fraunhofer IIS
Properties of Polarization
Dimensions of Electromagnetic Radiation
• All orientations equally present
 degree of polarization 0%
• Only one strong orientation
 degree of polarization 100%
• All orientations are present
but at different strength
 partially polarized, e.g. 50%
• Dominant angle of polarization
 another degree of freedom
Schöberl et al, Polarization Image Processing with Scilab, Scilab Conference, Nov 2018

5. 5www.esi-group.com
Copyright © Fraunhofer IIS
Where does it come from?
Polarization of Light
Most light sources (except lasers) are unpolarized
• Reflection gives polarized light,
think of water surfaces, glass, beamsplitters
surface inspection, ellipsometry
• Reflection at non-metallic surfaces
like human skin, plants, plastics,
object classification
Schöberl et al, Polarization Image Processing with Scilab, Scilab Conference, Nov 2018
Pajs, Brewsters angle, public domain

6. 6www.esi-group.com
Copyright © Fraunhofer IIS
Where does it come from?
Polarization of Light
• Nanostructures can create polarization
in transmission and reflection, useful
for polarization filters, carbon fiber inspection
Schöberl et al, Polarization Image Processing with Scilab, Scilab Conference, Nov 2018

7. 7www.esi-group.com
Copyright © Fraunhofer IIS
Application Carbon Fiber Reinforced Plastics (CFRP) Inspection

8. 8www.esi-group.com
Copyright © Fraunhofer IIS
Application Carbon Fiber Reinforced Plastics (CFRP) Inspection

9. 9www.esi-group.com
Copyright © Fraunhofer IIS
Where does it come from?
Polarization of Light
• Rayleigh scattering of light
at small particles gives polarization
(animals, particle analysis)
Schöberl et al, Polarization Image Processing with Scilab, Scilab Conference, Nov 2018
Polarization by Scattering, OpenStax College, CC BY 4.0

10. 10www.esi-group.com
Copyright © Fraunhofer IIS
Where does it come from?
Polarization of Light
• Birefringence in optical anisotrop
material causes polarization,
for example in calcit, quartz
• Stress birefringence is present
in glass and plastics
Schöberl et al, Polarization Image Processing with Scilab, Scilab Conference, Nov 2018
Mikael Häggström, positively birefringent material, public domain

11. 11www.esi-group.com
Copyright © Fraunhofer IIS
Application Stress Inspection in Glass

12. 12www.esi-group.com
Copyright © Fraunhofer IIS
Basic Principle
Measuring Polarization
• Polarization has 3/4 degrees of freedom
• Intensity of light
• Degree of polarization
• Angle of maximum polarization
• Circularity
• We can use ≥3 measurement with polarization filters for analysis
(and some math to obtain Stokes parameters)
Schöberl et al, Polarization Image Processing with Scilab, Scilab Conference, Nov 2018

13. 13www.esi-group.com
Copyright © Fraunhofer IIS
Camera Technology
Measuring Polarization
• It is possible to measure sequentially
(if you have the time to do so)
• Electronic shutters – sequential
and subject to temperature drift
• With side-by-side cameras
• With a multi-line scanning
camera for moving objects
• Beam splitter – expensive components
and precise optical alignment
Schöberl et al, Polarization Image Processing with Scilab, Scilab Conference, Nov 2018
Fluxdata FD-1665P polarization camera with beam splitters

14. 14www.esi-group.com
Copyright © Fraunhofer IIS
Camera Technology
Measuring Polarization
• Or with division-of-focal-plane
similar to almost all RGB color cameras
• Glue the filter to the sensor
• With nano-structured filters
embedded in the CMOS sensor
manufacturing process
Schöberl et al, Polarization Image Processing with Scilab, Scilab Conference, Nov 2018

15. 15www.esi-group.com
Copyright © Fraunhofer IIS
Camera Technology
Measuring Polarization
• 5MPixel Sony Polarization Sensor
becoming available
• Cameras from most camera
manufacturers announced:
AVT, Baumer, Flir, IDS, jAi,
Lucid Vision Labs, Navitar,
Ricoh, Sony, SVS Vistec,
Teledyne Dalsa
Schöberl et al, Polarization Image Processing with Scilab, Scilab Conference, Nov 2018
Polarization cameras from SVS Vistec, Teledyne Dalsa, AVT, jAi, Baumer, Sony, Lucid Vision labs, IDS, FLIR

16. 16www.esi-group.com
Copyright © Fraunhofer IIS
Challenges (we are adressing)
Polarization
• How to process the raw sensor data?
• Different from RGB debayering
• How do you calibrate measurements?
• Achieving reliable data for measurements
• How to solve the applications
• What is a good setup?
• The user does not need polarization values
• More processing is required to calssify
industrial products for quality (good/bad)
Schöberl et al, Polarization Image Processing with Scilab, Scilab Conference, Nov 2018
?

17. 17www.esi-group.com
Copyright © Fraunhofer IIS
With internal prototype camera
Algorithm development
• Image processing for evaluating
different modes/calculations
• Extending SciLab
• Interfacing an industrial USB
camera for live image capture
• Interfacing a step motor controller
for defined filter rotation
Schöberl et al, Polarization Image Processing with Scilab, Scilab Conference, Nov 2018
Regular
industrial
camera
Motorized
polarization
filter
Object
under
test
Fixed
polarization
filter
Light source
PC
with Scilab
Motor
control
Raw
image
data

18. 18www.esi-group.com
Copyright © Fraunhofer IIS
Processing Stack for Prototype System
Algorithm development
Schöberl et al, Polarization Image Processing with Scilab, Scilab Conference, Nov 2018
4 raw images
Stokes
parameters
S0 S1 S2
4 raw images
4 raw images
4 raw images
Stokes
parameters
S0 S1 S2
Stokes
parameters
S0 S1 S2
Retardation,
ATN
Retardation,
ATN,
MoS, OoS
for general
polarization imaging
for glass
inspection
data from
prototype camera

19. 19www.esi-group.com
Copyright © Fraunhofer IIS
For integration into industrial inspection systems
Product Development
• Use upcoming industrial cameras
• 5 MPixel, single-shot imaging
• High frame rate for inline inspection
• Provide the software as extension/plugin/.dll
for machine vision software
• Calibrated measurements of retardation/ATN
• Measurement of magnitude and orientation of stress
Schöberl et al, Polarization Image Processing with Scilab, Scilab Conference, Nov 2018

20. 20www.esi-group.com
Copyright © Fraunhofer IIS
From Scilab Code to a .dll
Product Development
• Re-programm everything again
• ARGO EU project http://www.argo-project.eu/
„WCET-Aware Parallelization of Model-Based Applications
for Heterogeneous Parallel Systems”
• WCET
 Worst-case execution time
 hard realtime applications
• Transfer into parallel C code
• Utilize parallel architectures
• Still ongoing project
Schöberl et al, Polarization Image Processing with Scilab, Scilab Conference, Nov 2018

21. 21www.esi-group.com
Copyright © Fraunhofer IIS
The tool flow we try to use
Product Development
Schöberl et al, Polarization Image Processing with Scilab, Scilab Conference, Nov 2018
.c .h
.c .h
.dll Industrial
application
algorithms
interface
code

22. 22www.esi-group.com
Copyright © Fraunhofer IIS
Software for Industrial Polarization Imaging
Product Development
• Software as a Halcon Extension
for use with all industrial cameras
• Calculate polarization values
(DOLP/AOMP, Stokes values S0 S1 S2)
• Calculate glass stress
(retardation, ATN, magnitude of stress, orientation of stress)
• Demo 2: Inspection of tempered tableware glass
Schöberl et al, Polarization Image Processing with Scilab, Scilab Conference, Nov 2018

23. 23www.esi-group.com
Copyright © Fraunhofer IIS
Processing Stack for Commercial Solution
Product Development
Schöberl et al, Polarization Image Processing with Scilab, Scilab Conference, Nov 2018
Stokes
parameters
S0 S1 S2raw image
Stokes
parameters
S0 S1 S2
Stokes
parameters
S0 S1 S2
Retardation,
ATN
Retardation,
ATN,
MoS, OoS
for general
polarization imaging
for glass
inspection
data from
industrial camera

24. 24www.esi-group.com
Copyright © Fraunhofer IIS
Polarization Imaging
Summary
• Polarization is a novel modality that is
coming to more and more applications
• Application-specific processing required
• ARGO Tool flow starting with Scilab
to a .dll for software product delivery
Schöberl et al, Polarization Image Processing with Scilab, Scilab Conference, Nov 2018
Dr.
Michael Schöberl
Group Manager
Imaging Solutions
michael.schoeberl@iis.fraunhofer.de
www.iis.fraunhofer.de/imaging-solutions