The document summarizes the lifecycle of a pixel from capturing light with a sensor to final storage as a JPEG file. It discusses how light is converted to electrical signals via photochemistry, thermal physics, or photophysics. A CMOS sensor array captures this data as pixels organized into rows and columns. A Bayer filter arranges RGB colors on the sensor. The raw pixel data is converted to RGB and compressed to JPEG format. JPEG uses discrete cosine transform (DCT) and quantization to remove high frequency data and compress the file, resulting in some loss of quality but much smaller file size.

1. Lifecycle of a pixel
From a photon to Jpeg

2. Three basic principles of light
sensing
● photochemistry: light renders silver halide grains in film
“emulsion” “developable”
● thermal physics: heating effect of incident light heats
sensor that measures temperature
● photophysics: interaction of light with matter frees
electrons
○ light absorbed by metal surfaces causes current to be ejected from
them
○ light absorbed by semiconductors causes their conductivity to
increase

3. Capturing a digital
image

4. Photoelectric Effect
● Einstein's Nobel prize
● Light hits the conductor or semiconductor, electrons are
ejected and voltage is generated
● Principle behind solar cells and photosensitive diodes

5. Semiconductor Technologies
● CCD (“charge coupled device”), mostly obsolete
● CMOS (“complementary metal oxide semiconductor”)
○ originally: naked memory chips
○ currently: “camera on a chip” designs
● CID (“charge injection device”)
○ Mostly used in speciality cameras, by NASA and similar

6. CMOS imager array
● Two dimensional array of photosensitive diodes is
organized into rows and columns
● Each row charges are moved to a BUS at a time
● Color separation by filters and prisms
● Faster than CCD, cheaper, uses less power, and works
better in low light, but generates more noise

7. Bayer filter
● Color filter array (CFA) for arranging RGB color filters on
a square grid of photosensors
● 50% green, 25% blue, 25% red
● GRGB arrangement

10. Raw to RGB

11. Conversion to RGB
● Need to convert to RGB or similar color space
● Need to interpolate missing pixel values
● Prone to errors
● Moire effect

14. RGB to JPEG

15. What is JPEG?
● JPEG is NOT an image format, it is compression standard
● JPEG stands for Joint Photographic Experts Group
● JFIF is standard file format that holds JPEG compressed
images, JPEG File Interchange Format
● Exif, Exchangeable Image Format is actual format
modern cameras use

16. JPEG compression assumptions
● Human eye sees grayscale better than color
● Human eye are more sensitive to green than red or blue
● Human eye does not see high frequency changes well

17. Transformation to YCbCr space
● Channels luma, blue and red
● Unlike in RGB, each channel is stored separately
● Luma holds luminosity and green data
● YCbCr color space separates luminance and
chrominance
● Chroma channels are downsampled
● Usually by factor of 2, therefore 4x less space
● Each channel is array of bytes, same as RGB pixel
components, therefore it holds same amount of data

18. Separating chroma and luma

19. Extremely downsampled
chroma

20. Discrete Cosine Transform (DCT)
● Locally, image is a combination of high and low frequency
cosine functions
● For example, shape of a hand is high frequency, shape of
knitting on the sweater is high frequency component
● Image is divided into 8x8 pixel blocks, each block is
encoded as sum of 64 preset cosine functions
● This sorts the data according to its frequency

21. DCT 8x8 table

22. How images are composed

23. Arranged in zig-zag

24. DCT calculation
and compression

25. Source 8x8 block

26. Represent pixels with values

27. Subtract 128 from each value

28. Calculate DCT for the matrix

29. Divide by quantization table
value and round to nearest
integer

30. Resulting block

31. DCT quantization
● We will find that large values are in low frequency area
● On most images, high frequency cosines don’t contribute
much, and our eye can’t see them well
● Most values will be zero, only those that contribute a lot
to the image will stay
● High frequency values in quantization table are big, so
they will nearly sure be zeroed out

32. Compression
● Huffman encoding is very good in removing multiple
values
● We store quantization table used and DCT coefficients in
JFIF format so process can be reversed when
decompression
● Chroma tables have much higher penalty than luma
● Decompressed image has relatively small differences
from original while reducing size significantly

33. Disadvantages of JPEG
● It is lossy, so not ideal for scientific images
● No coherence between blocks -> JPEG artifacts
● Not good for text, signs, etc, images with sharp edges and
high contrast due to JPEG artifacts
● Text violates our assumption that high frequency does
not contribute a lot to the image
● When DCTing image of the text, DCT coefficients are not
concentrated in low frequency area, a lot gets lost

34. Steganography
● Neat trick of encoding secret messages in images
● Simplest approach: encode in least significant bits
○ Can be detected as least significant bits will be random noise
● More complex approach: encode in DCT coefficients
○ Not detectable by visual inspection
○ Can be detected by statistics, distribution will be off
○ JSteg, f5, etc, many are open source
● Modern algorithms use DCT, but take into account
statistics, some success detecting with machine learning
● Used for protecting media by watermarking

35. The End.