Hyperspectral Imaging

1. Process
Analysis &
Technology
Prof. Dr. Rudolf Kessler
Prof. Dipl. Phys. W. Kessler
STZ Technology Transfer Center
Process Control and Data Analysis
72762 Reutlingen, Germany
Prof. emer. Dr. R. W. Kessler
Process Analysis and Technology PA&T
Reutlingen University, Germany
Hyperspectral Imaging: General Introduction

2. © Prof. Dr. Rudolf Kessler, STZ Prozesskontrolle und Datenanalyse, Process Analysis & Technology, Reutlingen University
Further Reading
R. W. Kessler, Perspectives in process analysis.
J. Chemometrics, 2013, 27: 369–378. doi: 10.1002/cem.2549
B. Boldrini, W. Kessler, K. Rebner and R. W. Kessler
Hyperspectral imaging: a review of best practice, performance and pitfalls for inline and
online applications, Journal of Near Infrared Spectroscopy 2012, 20, 438–508

3. © Prof. Dr. Rudolf Kessler, STZ Prozesskontrolle und Datenanalyse, Process Analysis & Technology, Reutlingen University
Agenda
• Concepts and trends in manufacturing
• Taxonomy of spectral imaging techniques
• Sensitivity, selectivity and robustness
• Selected examples
• Focus on wood
• Summary

4. © Prof. Dr. Rudolf Kessler, STZ Prozesskontrolle und Datenanalyse, Process Analysis & Technology, Reutlingen University
Agenda
• Concepts and trends in manufacturing
• Taxonomy of spectral imaging techniques
• Sensitivity, selectivity and robustness
• Selected examples
• Focus on wood
• Summary

5. © Prof. Dr. Rudolf Kessler, STZ Prozesskontrolle und Datenanalyse, Process Analysis & Technology, Reutlingen University
• Manufuture of the EU
• PAT/QbD of the FDA
• BDI 2006: 32 Thesen
• Factbook 06 (VCI) „Chemie 2030 - Globalisierung gestalten“
• EU’s 20-20-20 goals (20% increase in energy efficiency, 20%
reduction of CO2 emissions, and 20% renewables by 2020)
• Namur road map: Prozess-Sensoren 2015+
• World Manufacturing Forum 2012
• Industrie 4.0
Concepts
Manufacturing and Processing Industry:
• Provides 70% of the wealth of the German society although only around 30 %
of the population work in the manufacturing industry!!!,
• 90% of IT research is financed by the manufacturing industry
Trends:
Aging of the population: medical systems
Urbanisation and megacities
Personalization of products and goods
„Internet of Things“

6. © Prof. Dr. Rudolf Kessler, STZ Prozesskontrolle und Datenanalyse, Process Analysis & Technology, Reutlingen University
First
Principles
&
Mechanistic Models
Causality
Correlative and Descriptive Models
Knowledge Based Production and Causality
Adaptive
Processing
Knowledge
based
Production
Fixed
Process
Consistent
Output
Variable
Output
Variable
Material
Input
Models
- First principles
- DoE
- Soft modelling
- Molecular markers
(specroscopy)

7. © Prof. Dr. Rudolf Kessler, STZ Prozesskontrolle und Datenanalyse, Process Analysis & Technology, Reutlingen University
but…….. Culture …….
"Heaven is where the police are British,
the chefs French, the mechanics German
and the lovers Italian
and it is all organised by Swiss”
“Hell is where the chefs are British,
the mechanics French, the lovers Swiss,
the police German
and it is all organised by the Italians."

8. © Prof. Dr. Rudolf Kessler, STZ Prozesskontrolle und Datenanalyse, Process Analysis & Technology, Reutlingen University
spatial scan (x,y)
x

y
Transmission


Toolbox Multimodal Optical Spectroscopy
Reflection
Different
experimental
setup
UV/VIS
I
N
NIR
N
Fluorescence
N
MIR/Raman
N
Raman
UV NIR IR
Fluorescence
wide spectral range
Hyperspectral imaging

9. © Prof. Dr. Rudolf Kessler, STZ Prozesskontrolle und Datenanalyse, Process Analysis & Technology, Reutlingen University
Key-Issue: Spectroscopy in Scattering Systems
 Use Scatter as Information!!!
 Separate Scatter from Absorption
 Integrate this Information
into your Modelling:
e.g. MCR, SBC, Multiblock
The Basic Idea:
Absorption AND Scatter
= Chemistry and Morphology
Theory: you need more than 1 measurement e.g.
Kubelka Munk, RTE

10. © Prof. Dr. Rudolf Kessler, STZ Prozesskontrolle und Datenanalyse, Process Analysis & Technology, Reutlingen University
Some information
+
redundant
information
Some information
+
non-specific
information
Some
information +
no information
Find small portion of
useful information
 Univariate data analysis
 Explorative multivariate
data analysis (PCA, etc.)
 Multivariate regression and
classification (MLR, PLS,
RBF, Kohonen.......)
 Optical principal component
analysis
 MCR, Multiblock, …
Toolbox Chemometrics: Increase Selectivity!!

11. © Prof. Dr. Rudolf Kessler, STZ Prozesskontrolle und Datenanalyse, Process Analysis & Technology, Reutlingen University
Measurements Data Knowledge Causality
The Japanese eat very little fat and suffer fewer heart attacks
than the British or the Americans.
The French eat a lot of fat and also suffer fewer heart attacks
than the British or the Americans.
The Japanese drink very little red wine and suffer fewer heart attacks
than the British or the Americans.
The Italians drink a lot of red wine and also suffer fewer heart attacks
than the British or the Americans.
Conclusion:
Eat and drink whatever you like. It's speaking English that kills you.
What is wrong?
To draw conclusions from random or spurious correlations

12. © Prof. Dr. Rudolf Kessler, STZ Prozesskontrolle und Datenanalyse, Process Analysis & Technology, Reutlingen University
Agenda
• Concepts and trends in manufacturing
• Taxonomy of spectral imaging techniques
• Sensitivity, selectivity and robustness
• Selected examples
• Focus on wood
• Summary

13. © Prof. Dr. Rudolf Kessler, STZ Prozesskontrolle und Datenanalyse, Process Analysis & Technology, Reutlingen University
Chemical/Spectral Imaging
Absorption
+
Scattering
CHEMICAL
CHARACTERI-
ZATION
IMAGING
PHYSICAL/
MORPHOLOGICAL
CHARACTERI-
ZATIONCHEMICAL
IMAGING =

14. © Prof. Dr. Rudolf Kessler, STZ Prozesskontrolle und Datenanalyse, Process Analysis & Technology, Reutlingen University
Chemical Imaging Techniques
Whiskbroom Imaging
Sequential (x, y, )
Staring Imaging
Simultaneous (x, y)
Sequential ()
Pushbroom Imaging
Simultaneous (x, )
Sequential (y)
Wavelengths:
UV/VIS
2D- Fluorescence with
FLIMS
NIR
IR
Raman
Specular and Diffuse Reflectance, Transmittance, Polarisation
Hyperspectral imaging
y

Whiskbroom
x
Staring

15. © Prof. Dr. Rudolf Kessler, STZ Prozesskontrolle und Datenanalyse, Process Analysis & Technology, Reutlingen University
Advantage Whiskbroom: high flexibility
• High Optical Throughput
• High Sensitivity (high S/N), Dark Field, Bright Field
• Fast Scanning System with High Lateral Resolution
• Multimodal Spectroscopy, Transmittance, Reflectance
• Same Sample- Same Location- Different Wavelength- no optical changes
• No Photon Diffusion (Illumination = Detection)
• Easy to calibrate
• all Microscopy Techniques Optional
• High Collection Efficiency
But: time consuming

16. © Prof. Dr. Rudolf Kessler, STZ Prozesskontrolle und Datenanalyse, Process Analysis & Technology, Reutlingen University
Example Whiskbroom Imaging

17. © Prof. Dr. Rudolf Kessler, STZ Prozesskontrolle und Datenanalyse, Process Analysis & Technology, Reutlingen University
FLIM 2D Fluorescence
NIR
spectrometer
laser single-mode
fiber
objectiv
piezo
scanner
edge filter
white light
CCD
z-stage
CCD
multi-mode
fiber
holographic
beam splitter
Raman
spectrometer
SNOM
unit
CCDVis
spectrometer
CCD
Glioblastoma
reflection
transmission
WITEC – PA&T System
Nearfield unit with Solid immersion lens
(SIL)
brightfield darkfield
Glioblastoma
nearfield image
Glioblastoma

18. © Prof. Dr. Rudolf Kessler, STZ Prozesskontrolle und Datenanalyse, Process Analysis & Technology, Reutlingen University
Staring Imaging
advantage
- high lateral resolution
- easy to implement
- high information density
- but: motion stop needed
- but: calibration and focus difficult,
homogeneous illumination may be difficult
0.05
0.10
0.15
0.20
0.25
0.30
0.35
1000 1100 1200 1300 1400 1500 1600 1700 1800 1900 2000
Wellenlänge [nm]
Extinktion

19. © Prof. Dr. Rudolf Kessler, STZ Prozesskontrolle und Datenanalyse, Process Analysis & Technology, Reutlingen University
Staring Imaging:

450 nm
550 nm
650 nm
0.200
0.300
0.400
0.500
0.600
0.700
0.800
0.900
1.000
1.100
1.200
420 440 460 480 500 520 540 560 580 600 620 640 660 680 700 720
Wellenlänge [nm]
Absorption
Contrast enhancement by
absorption

20. © Prof. Dr. Rudolf Kessler, STZ Prozesskontrolle und Datenanalyse, Process Analysis & Technology, Reutlingen University
Pushbroom Imaging
advantage
- flexible inline and real time applications
- no motion stop needed
- good compromise between spatial and wavelength resolution
- but: different optical and spatial resolution in x- und y- direction
LateraleAchse
Spektrale
Achse
Extinktion

21. © Prof. Dr. Rudolf Kessler, STZ Prozesskontrolle und Datenanalyse, Process Analysis & Technology, Reutlingen University
Inline: Pushbroom Imaging Technology

22. DEMO video available on:
http://vimeo.com/77218620
SNAP SHOT Imaging

23. © Prof. Dr. Rudolf Kessler, STZ Prozesskontrolle und Datenanalyse, Process Analysis & Technology, Reutlingen University

x-axis
(1st spatial dimension)
(2ndspatialdimension)
(timecoordinate,
ifsampleismoving)
y-axis
Distribution map for one wavelength
Spectrum for one pixel
Data Cube in Spectral Imaging

24. © Prof. Dr. Rudolf Kessler, STZ Prozesskontrolle und Datenanalyse, Process Analysis & Technology, Reutlingen University
Agenda
• Concepts and trends in manufacturing
• Taxonomy of spectral imaging techniques
• Sensitivity, selectivity and robustness
• Selected examples
• Focus on wood
• Summary

25. © Prof. Dr. Rudolf Kessler, STZ Prozesskontrolle und Datenanalyse, Process Analysis & Technology, Reutlingen University
Example Darkfield Glioblastoma Vis Backscattering Light RGB: TP53

26. © Prof. Dr. Rudolf Kessler, STZ Prozesskontrolle und Datenanalyse, Process Analysis & Technology, Reutlingen University
Example Label Free Karyotyping by Backscattering

27. © Prof. Dr. Rudolf Kessler, STZ Prozesskontrolle und Datenanalyse, Process Analysis & Technology, Reutlingen University
Enhanced Selectivity: Derivative Spectroscopy: UV-Vis, NIR…..
concentration A + 2B
concentration A + B
concentration A
original
2nd derivative
1st derivative
be aware:
E = h c/λ
200nm = 50 000 cm-1
250nm = 40 000 cm-1
Δ = 50nm, Δ = 10 000 cm-1
Δ Raman, MIR app.
4000 cm-1!!!!

28. © Prof. Dr. Rudolf Kessler, STZ Prozesskontrolle und Datenanalyse, Process Analysis & Technology, Reutlingen University
Robustness: Specular and Diffuse Reflectance of a Cavity on a Surface
parallell
crossed
Specular
Reflection
I0
Δ n
Model System:
cellulose/
dyed cellulose
high lateral
resolution!!
low lateral
resolution due to
photon diffusion!!
high
concentration ?

29. © Prof. Dr. Rudolf Kessler, STZ Prozesskontrolle und Datenanalyse, Process Analysis & Technology, Reutlingen University
Robustness: Inline Illumination – Detection Set Up
light
source
sample moving direction
Specrograph
CCD
Array
45R45 45R0 dR0
light
source
sample moving direction
CCD
Array
Specrograph
diffuse
light
source
sample moving direction
CCD
Array
Specrograph

30. © Prof. Dr. Rudolf Kessler, STZ Prozesskontrolle und Datenanalyse, Process Analysis & Technology, Reutlingen University
diffusors
light source
pushbroom imager
Inline Illumination

31. © Prof. Dr. Rudolf Kessler, STZ Prozesskontrolle und Datenanalyse, Process Analysis & Technology, Reutlingen University
Absorption and Scattering: Penetration of Photons
measured calculated
R
T
Real Life ASA tablet
Penetration Depth!!!!

32. © Prof. Dr. Rudolf Kessler, STZ Prozesskontrolle und Datenanalyse, Process Analysis & Technology, Reutlingen University
Scale of Scrutiny
0
0.05
0.1
0.15
0.2
0.25
0.3
400 600 800 1000 1200 1400 1600 1800 2000
281 Mpa
156 Mpa
Penetrationdepth/cm
wavelength/nm
Optical penetration depth of Theophyllin tablets with different
API concentrations, calculated from S and K
Si-based CCD
3rd
overtone
in NIR
InGas-based detectors
mixed???
many small
measurement spots
are better than one
large spot in
spectroscopy!!!

33. © Prof. Dr. Rudolf Kessler, STZ Prozesskontrolle und Datenanalyse, Process Analysis & Technology, Reutlingen University
Agenda
• Concepts and trends in manufacturing
• Taxonomy of spectral imaging techniques
• Sensitivity, selectivity and robustness
• Selected examples
• Focus on wood
• Summary

34. © Prof. Dr. Rudolf Kessler, STZ Prozesskontrolle und Datenanalyse, Process Analysis & Technology, Reutlingen University 34
Chemical Imaging Plastic Sorting
Conveyor belt and/or chute
for material transportation
Valve Block
Illumination
AWDL
Metal Sensor
HELIOS
HELIOS NIR/SWIR
Smart Camera System
• Sensor Data Processing
– Smile-, Keystone-, Black Drift Corr.,
– Defectpixel Elim., Calibration, etc...
• Spectra Preprocessing
– Derivative, Norm., etc...
• Feature Extraction
– Mean Intens., Baseline Slope, etc..
• Feature Combination
– FG / BG Segmentation, etc...
• Classification
– Spectra, Colour
• Object Processing
– Colour, Shape, Size, Structure, Material
• Decision
– e.g. Control of Air Valves
Hard Real Time Processing of
> 80 000 high-res. Spectra / Second
> 300 high-res. Spectral Images / Second

35. © Prof. Dr. Rudolf Kessler, STZ Prozesskontrolle und Datenanalyse, Process Analysis & Technology, Reutlingen University
e.g. PP/PES in Non Wovens: enhanced Selectivity using Chemometrics
~210°
C
 [nm]
1000 1200 1400 1600 1800 2000 2200 2400 2600
E
0.0
0.2
0.4
0.6
0.8
PES PP
PC 1
-2 -1 0 1 2
PC2
-1.2
-0.6
0.0
0.6
1.2
PP/PES
100/0
75/25
70/30
50/50
0/100
20/80
measured
0 20 40 60 80 100 120
predicted
0
20
40
60
80
100
120
PLS
NIR spectra

36. © Prof. Dr. Rudolf Kessler, STZ Prozesskontrolle und Datenanalyse, Process Analysis & Technology, Reutlingen University
Imaging Thin Films on Aluminium
250 nm 550 nm
705 nm 805 nm645 nm
brightfield illumination with polarisation, magnification for visualisation 500
415 nm
0
20
40
60
80
100
300 400 500 600 700 800
Wellenlänge [nm]
R[%]
Messung
Simulation
- ein.Winkel = 30°
- dt = 445 nm
- n2 = 1,35 k2 = 0,01
(b)

37. © Prof. Dr. Rudolf Kessler, STZ Prozesskontrolle und Datenanalyse, Process Analysis & Technology, Reutlingen University
Counterfeit Detection through blister packaging (N.Lewis, Malvern)

38. © Prof. Dr. Rudolf Kessler, STZ Prozesskontrolle und Datenanalyse, Process Analysis & Technology, Reutlingen University
Key Issue: Sample Presentation: Food and Feed (e.g. Bühler)
see also: 60 000 rice kernels/sec!!

39. © Prof. Dr. Rudolf Kessler, STZ Prozesskontrolle und Datenanalyse, Process Analysis & Technology, Reutlingen University
Multiplexing: e.g. Reactive Extrusion resp. Hot Melt Extrusion
Courtesy of Rottendorf Pharma GmbH
Hot Melt Extrusion
reactive Extrusion
Pushbroom Imaging System
Entrance slit with
fiber optic mounting Camera
prism / grating / prism
optics
x
λ
Reaction
Tomography!!!

40. © Prof. Dr. Rudolf Kessler, STZ Prozesskontrolle und Datenanalyse, Process Analysis & Technology, Reutlingen University
Multipoint Spectroscopy
Extend to many measurement positions
by using several fibre bundles
Very flexible, adjustable to any need

41. © Prof. Dr. Rudolf Kessler, STZ Prozesskontrolle und Datenanalyse, Process Analysis & Technology, Reutlingen University
 = 80 cm-1
 = 5 cm-1
I0
Illumination
2mm thickness
Tablet
 = 180 cm-1
 = 2 cm-1
I0
Illumination
2mm thickness + Coating
Coated
Tablet
Photon Diffusion Spectroscopy
API
Absorbance
decreases
with coating
thickness
But:
Increases
with
distance
!

42. © Prof. Dr. Rudolf Kessler, STZ Prozesskontrolle und Datenanalyse, Process Analysis & Technology, Reutlingen University
Camera
Pushbromm
Imager
Objective
Microreactor
Absorbance
x, s
Microreactor
Reaction Tomography in a Microreactor

43. © Prof. Dr. Rudolf Kessler, STZ Prozesskontrolle und Datenanalyse, Process Analysis & Technology, Reutlingen University
Pushbroom
Catalyst
Illumination
Optical Spectroscopy and Reactor Tomography

44. © Prof. Dr. Rudolf Kessler, STZ Prozesskontrolle und Datenanalyse, Process Analysis & Technology, Reutlingen University
Agenda
• Concepts and trends in manufacturing
• Taxonomy of spectral imaging techniques
• Sensitivity, selectivity and robustness
• Selected examples
• Focus on wood
• Summary

45. © Prof. Dr. Rudolf Kessler, STZ Prozesskontrolle und Datenanalyse, Process Analysis & Technology, Reutlingen University
Wood
chips
Example: dry- and wet processing of wood
DefibratorBlow line
Dry
Process
Drying Non woven Moulding press
Wet
ProcessVat
Fibre
mat
Hot
press

46. © Prof. Dr. Rudolf Kessler, STZ Prozesskontrolle und Datenanalyse, Process Analysis & Technology, Reutlingen University
Concept: Multi Information Manufacturing

47. © Prof. Dr. Rudolf Kessler, STZ Prozesskontrolle und Datenanalyse, Process Analysis & Technology, Reutlingen University
In-Line Control during Manufacturing: Control of Flutter
Diffuse Reflectance Probe and Spectral Imaging

48. © Prof. Dr. Rudolf Kessler, STZ Prozesskontrolle und Datenanalyse, Process Analysis & Technology, Reutlingen University
Morphology and Chemistry
Spruce Beech
Cross-section 20 µm
magnification 100 x
reminder: sensitivity absorption and scatter

49. © Prof. Dr. Rudolf Kessler, STZ Prozesskontrolle und Datenanalyse, Process Analysis & Technology, Reutlingen University
Spectra 370-600nm, Selection
original
2nd derivative
3rd derivative
1st derivative
process variables: DoE at the production plant
• severity steam treatment (temperature-time)
• severity mechanical refining (type, distance, rotation)
• Wood mixture (spruce, spruce with bark, spruce/beech)
Absorbance

50. © Prof. Dr. Rudolf Kessler, STZ Prozesskontrolle und Datenanalyse, Process Analysis & Technology, Reutlingen University
380-700 nm
reflectance
PCA-Analysis
of Spectra
Clustering in wood mixtures
Classification of fineness
PCA of Vis-Spectra of Fibreboards: DoE at the Plant
Classification of Severity factor (SFC)

51. © Prof. Dr. Rudolf Kessler, STZ Prozesskontrolle und Datenanalyse, Process Analysis & Technology, Reutlingen University
Distribution of a Resin on Wood Chips OSB

52. © Prof. Dr. Rudolf Kessler, STZ Prozesskontrolle und Datenanalyse, Process Analysis & Technology, Reutlingen University
Agro-Food-Manufacturing-Process
Industry
grain quality inspection
identification of contaminants in soil, animal
products and food
food safety and authenticity
Environment
plastics sorting and recycling
detection of dangerous waste
monitoring pollution in
enviromental air and water.
Health
detection of tooth decay
neonatal and fetal brain diagnostic (non invasive and painless )
surgery monitoring
Summary Spectral Imaging: The Benefit for Society
Earth observation
disaster monitoring
water resource management
climate change
observation
NASA/Goddard

53. © Prof. Dr. Rudolf Kessler, STZ Prozesskontrolle und Datenanalyse, Process Analysis & Technology, Reutlingen University
Many thanks for their active support to our PhD students at Reutlingen
Research Institute, Reutlingen University
Karsten Rebner
Tobias Merz
Barbara Boldrini
Edwin Ostertag
Lieselotte Barac
Sören Hummel
Anita Lorenz
Sabrina Luckow-Markgraf
…. and many others
Steinbeis-University Berlin: Prof. W. Kessler
ILM Ulm: Prof. Dr. Hibst, Prof. Dr. Kienle
University of Tübingen: Prof. Dr. D. Oelkrug
Finacial Support by BMBF, Landesstiftung BW, EU, ... Industry......
Acknowledgement
Thank You for Your Attention