This document discusses image-based phenotyping and how digital images capture the world in pixels. It explains that phenotypes are complex, encompassing interactions between genotype, environment, scale, time, and measurement approach. Advanced imaging technologies and computational analysis are allowing researchers to quantify dynamic and multivariate phenotypes, and identify hidden traits through large-scale measurements. This enables better mapping of genotypes to phenotypes across various scales.

1. Image-based Phenotyping:
Capturing and quantifying the world in pixels

2. Digital images are comprised of pixels
image tutorial
Bit is binary 1 or 0
Image is 10 pix across x 11 tall pix = 110 pixels
If it were 10” across x 11” tall then = 1 DPI

3. One byte in an 8-bit image has 8 slots or 28 = 256 combinations of 1/0
Bit #7 1/0
Bit #6 1/0
Bit #5 1/0
Bit #4 1/0
Bit #3 1/0
Bit #2 1/0
Bit #1 1/0
Bit #0 1/0
8-bit 16-bit 64-bit
These can be expressed as shades of gray or colors
One byte in a 32-bit image 232 = 4.29 x 109 combinations of 1/0
One byte in a 16-bit image has 216 = 65,536 combinations of 1/0
information content aka increasingly realistic representation of reality

4. The living world is dynamic
So how do we define phenotype?

5. Phenomics has a modest goal:
to understand the entirety of phenotype at multiple
scales across the breadth of genetic variation

6. Mapping genotype to phenotype:
But what is phenotype?
Does phenotype simply correlate to genotype?

7. Chitwood and Topp COPB 2015

8. Mapping genotype to phenotype:
Actually Phenotype space is enormous
=
Genotype
(all gene interactions)

9. Mapping genotype to phenotype:
Phenotype is at least Genotype x Environment
B73 maize grown in 3 environments

10. Mapping genotype to phenotype:
Phenotype space is enormous
=
Genotype
(all gene interactions)
x
Environment

11. Varieties of the phenotypic
experience: small scale
Atomic force
microscopy
High content cell screening
2D gel
electrophoresis
arstechnica.com

12. Varieties of the phenotypic
experience: Field Scale
Field phenotyping
with hyperspectral
imaging

13. Mapping genotype to phenotype:
Phenotype space is enormous
=
Genotype
(all gene interactions)
x
Environment
x
Scale
(from submolecular to ecological scales)

14. Varieties of the phenotypic experience

15. Mapping genotype to phenotype:
Phenotype space is enormous
=
Genotype (all gene interactions)
x
Environment
x
Scale (from submolecular to ecological scales and those
interactions)
x
Time (dynamics)

21. allele A
allele B
a simple example of a multivariate trait
Cheng-Ruei Lee

22. allele A
allele B
no difference in allele effects on one univariate axis
Width
Cheng-Ruei Lee

23. allele A
allele B
no difference in allele effects on one axis or the other
Depth
Cheng-Ruei Lee

24. Width
Depth
allele A
allele B
e.g.
Multivariate
Trait
= 2*Depth - 3*Width
a new multivariate axis separates allele effects
Cheng-Ruei Lee

27. Mapping genotype to phenotype:
Phenotype space is enormous
=
Genotype (all gene interactions)
x
Environment
x
Scale (from submolecular to ecological scales and those
interactions)
x
Time (dynamics)
x
The way in which we study it (phenotyping approach)

28. Video informatics uses imaging and computational
methods to quantify complex and dynamic phenotypes
Sportsvision

29. Video informatics uses imaging and computational
methods to quantify complex and dynamic phenotypes

30. Latent (hidden) phenotypes can be identified through
multivariate measurements and analysis

31. Latent (hidden) phenotypes can be identified through
multivariate measurements and analysis

32. Latent (hidden) phenotypes can be identified through
multivariate measurements and analysis
multivariate QTL
identified

33. We can leverage advances in robotics, computer
vision/AI, and engineering systems for science!

34. FIJI: Fiji Is Just ImageJ

38. IMAGING TECHNOLOGY

39. RAW DATA ACQUISITION – computer cannot distinguish object of
interest from background

40. IMAGE PROCESSING – computer can now distinguish object of interest
from background

41. IMAGE ANALYSIS – Feature Extraction: object of interest measured in
different ways

42. Plant shape metrics
Pseudocolored by NIR signal
Image background
Original image
After thresholding and erosion steps
After binary thresholding
Sobel filtered (x-axis)
Sobel filtered (y-axis)
Sobel filtered (x and y-axis)
Laplacian filtered
Laplacian sharpened
Sobel filtered inverted
After background subtraction
Laplacian sharpened + Sobel filtered
Final plant isolated
Object identification
Image maskBitwise OR join of both methods Original image
Supplemental Figure S11. A visual illustration of the pipeline used to threshold plant tissue from background within grayscale NIRFahlgren, Feldman, Gehan et al. Plant Phys. In press
Image analysis pipeline