ECCO: An Electron Counting Implementation for Image Compression and Optimization of Cryo-EM Data Management

ECCO.
An Electron Counting Implementation for
Image Compression and Optimization of
Cryo-EM Data Management.
December 18th, 2019
Room 2.1.5, Politecnico di Milano
Candidate:
Chiara Coletti, matr. 897815
Supervisor:
Prof. Marco Domenico Santambrogio
Co-supervisors:
Prof. Sara Vinco, Politecnico di Torino
Dr. Eleonora D'Arnese

INTRODUCTION
ECCO. An Electron Counting Implementation
for Image Compression and Optimization of

ECCO - Master Thesis Dissertation Candidate: Chiara Coletti, 8978153
Cryogenic Electron Microscopy
17 Å
Recorded on film
[Fernández, 2007]
Revolution in cryo-EM biological structure determination resolution:
example of RNA Polymerase III.
2007 2010 2015 2015
Atomic model
Direct e- detectors
[Hoffmann, 2015]
10 Å
Recorded on film
[Fernández, 2010]
3.9 Å
Direct e- detectors
[Hoffmann, 2015]

Image Processing Framework
4
Cryogenic electron
microscope
Supercomputing
facility
…
Raw data stream
400 Gb/s network connection
…
4D camera
87000 fps 
576 x 576 pixels
4D camera
75000 fps limit 
576 x 576 pixels
Noisy data with
very low SNR+

Edge Computing Approach
5
ECCO
Raw images
Processed
images
460 Gb/s 8 Gb/s
Cryogenic electron
microscope
Supercomputing
facility
Data compression
before network
transfer
Noise rejection
before further
processing+

Electron Counting
10.4
0.2
0.2
0.1
0.07
0.03
Electron
hits detector
Energy
scattering
Charge
integration
Electron
counting
Improved DQE across all spatial frequencies and 55:1 data compression
DQE = detective quantum efficiency [Booth, 2013]
1 2 3 4

PROBLEM
DEFINITION

Problem Statement
Counting
efficiency
Super
resolution
Collision
loss reduction
Real-time
implementation
Maximizing true
counts, while
minimizing false
ones.
Improving
precision in
events
localization.
Avoiding the
fusion of
overlapping
electrons.
Considering the
experimental
setting time
constraints.

MATERIALS
AND METHODS

Experimental Dataset
2k x 2k pixels images from a 4D Scanning
Transmission Electron Microscope (4D
STEM), acquired under vacuum probe
conditions with K2 direct detection
camera at 400fps.
No ground truth available, electrons
energy deposition is a stochastic
phenomena.
Example of a single noisy
diffraction pattern.

Example of a single counted
diffraction pattern.
TP
FP
11
Evaluation Methods
The state-of-the-art implementation is
included in the py4DSTEM tool [https://
github.com/bsavitzky/py4DSTEM].
From the counted diffraction patterns:
- TP : electrons counted inside the
diffraction disk;
- FP : electrons counted outside the
diffraction disk.
precision =
TP
TP + FP

PROPOSED
SOLUTIONS

Solution: Denoising
Histogram of diffraction pattern intensities
Density
Pixels intensity values
Gaussian noise
threshold = μ+4σ
The threshold to remove Gaussian noise is chosen as μ+4σ, assuming a
clear distinction between signal and noise distributions.
Gaussian noise
distribution parameters:
μ = 16.60
σ = 2.80

Solution: Denoising
Histogram of denoised diffraction pattern intensities
Density
Pixels intensity values
Gaussian noise
threshold = μ+4σ
The introduction of a denoising step enhances the signal with respect to
the noise, that is shrunk on a more narrow interval of intensities.
Gaussian noise
distribution parameters:
μ = 16.60
σ = 2.80

Denoising Algorithms
Approaches for
image denoising
Unsupervised
algorithms
Supervised deep
learning
Self-supervised deep
learning [Batson, 2019]
Block Matching
3D
Non-Local
Means
Denoising
CNN
Baby
UNet

Approaches for
image denoising
Unsupervised
algorithms
Supervised deep
learning
Block Matching
3D
Non-Local
Means
Denoising
CNN
Baby
UNet

Solution: Super-Resolution
2
Thresholded
diffraction pattern
Neighbourhoods
identification
Event localisation in
maximal point
1 3
Py4DSTEM implementation localizes the electron event in the maximal point of
a neighbourhood of non-zero pixels, keeping the original spatial resolution.

2 3
20
Solution: Super-Resolution
Thresholded
diffraction pattern
Connected pixels
identification
Event localisation at
center of mass
1
ECCO implementation localizes the electron event in the center of mass of a set
of connected non-zero pixels, eventually doubling (or more) the original spatial
resolution.

Solution: Collision Loss Reduction
2e-
1e-
N°ofevents
Events intensities
3e- …
Histogram of events intensities after counting
The counts' intensities follow the theoretical Landau distribution, defining
several peaks that can be used to identify overlapping electrons and reduce
collision losses.

RESULTS AND
CONCLUSIONS

Results: Denoising
The aim is the optimization of the precision, balancing TP and FP,
while keeping a reasonably low execution time for real-time implementation.
NLM BM3D DnCNN Baby UNet
py4D
STEM
- - Superv. Self-sup. Superv. Self-sup. -
Precision [%] 96.75 97.19 93.45 91.86 49.44 19.74 89.64
TP 744 968 599 553 88 30 727
FP 25 28 42 49 90 122 84
Run time [s] 3.98 192.15 9.71 8.59 55.68 49.53 0.40

Results: GPU Acceleration
284x 94x
GPU NLM
parallel algorithm
acceleration factor
with respect to CPU
GPU DnCNN
testing phase
acceleration factor
with respect to CPU
20x
GPU DnCNN
training phase
acceleration factor
with respect to CPU
GPU execution time:
0.01s per frame
GPU execution time:
0.11s per frame
GPU execution time:
20.83s per frame

Contributions
The introduction
of NLM for noise
reduction
increases the
precision of
7.11%.
The innovative
events
localization
allows sub-pixel
accuracy in
counting.
The novel
classification
approach
increases images
information
content.
The 284x speed-
up with GPU
implementation
shows potential
for real-time
execution.

Future Works
FPGA
Extension to different
datasets to test the
solution's robustness.
Implementation of new
denoising algorithms for
alternative solutions.
Design of hardware
implementations on
different platforms.

Thank you for the attention.
Candidate:
ECCO: An Electron Counting Implementation
Questions?

Supporting Materials
Candidate:
ECCO: An Electron Counting Implementation

Py4DSTEM Implementation
Dark reference
subtraction
Gaussian noise
thresholding
X-rays
thresholding
NN maximal
point selection
-Median filter
along strikes
direction
-Filtered pattern
subtraction from
original
-Fitting image  
to a gaussian
distribution:
N(μg,σg2)
-Thresholding
image at μg+kσg
-Computing μ and
σ2 of the image
-Thresholding at
μ+hσ (where h>>k)
-Consider 8x8
neighbourhoods in
binarized image
-Put e- event at
maximal point

Our Counting Implementation
Dark
reference
subtraction
Gaussian
noise
thresholding
Connected
pixel CoM
-Median filter
along strikes
direction
-Filtered pattern
subtraction
from original
-Fitting image  
to a gaussian
distribution:
N(μg,σg2)
-Thresholding
image at μg+kσg
-Finding
connected
components
-Assigning
integral signal
to their CoM
Image
denoising
-Enhancement
with different
approaches:
- NLM
- BM3D
- DnCNN
Events
classification
-Recognizing
events’ overlap
-Possibility to
assign
probabilistic
labels

Non Local Means
Variance law in probability theory demonstrates that if pixels are averaged, the noise
standard deviation of the average is divided by ⟹ we replace the intensity of a pixel with
an average of the intensities of “similar” pixels [Buades, 2011].
n
n

Block Matching 3D

Denoising CNN
Noisy pattern Noise estimate
Convolutional
Convolutional
Convolutional
ReLU
ReLU
BatchNorm
128 x 128 128 x 128128 x 128 x 64 128 x 128 x 64 128 x 128 x 64
• Convolution + ReLU to gradually separate image structure from noisy
observation;
• Batch normalisation to speed up learning and improve performances;
• Residual learning to learn residuals of the image, easier than learning
the denoised image representation.
[Zhang, 2017]

2 x 642 x 32 642 x 32
1282
34
Unet
1282 x 16
Input image
642 x 16 642 x 32
642 x 32 642 x 32
3 x 1282 x 16 1282 x 16 1282
Output image
Legend:
Convolutional
Pooling
Upsampling
Concatenation

Results: Denoising
Original DP BM3D outputNLM output
DnCNN output UNet outputNLM thresholded DP

Results: Classification
Binary Discrete Probabilistic py4DSTEM
Precision [%] 96.75 93.92 96.16 89.64
TP 744 1375 3975 727
FP 25 89 159 84
1
1
1 1
2
3 0.8
1.7
3.2
Binary
classification
Discrete
classification
Probabilistic
classification

Results: GPU Acceleration
NLM DnCNN py4DSTEM
- Testing Training -
CPU run time [s] 3.98 10.44 417.52 0.40
GPU run time [s] 0.01 0.11 20.83 -
284x
94xGPU NLM
acceleration factor
GPU DnCNN
acceleration factor

Bibliography
Fernández-Tornero, C., Böttcher, B., Riva, M., Carles, C., Steuerwald, U., Ruigrok, R. W., ... &
Schoehn, G. (2007). Insights into transcription initiation and termination from the electron
microscopy structure of yeast RNA polymerase III. Molecular cell, 25(6), 813-823.
Fernández‐Tornero, C., Böttcher, B., Rashid, U. J., Steuerwald, U., Flörchinger, B., Devos, D. P., ... &
Müller, C. W. (2010). Conformational flexibility of RNA polymerase III during transcriptional
elongation. The EMBO journal, 29(22), 3762-3772.
Hoffmann, N. A., Jakobi, A. J., Moreno-Morcillo, M., Glatt, S., Kosinski, J., Hagen, W. J., ... & Müller, C.
W. (2015). Molecular structures of unbound and transcribing RNA polymerase
III. Nature, 528(7581), 231.
Booth, C., Mooney, P. (2013), Applications of electron-counting direct-detection cameras in high-
resolution cryo-electron microscopy. Microscopy and Analysis, 27(6):13-21 (AM).
Zhang, K., Zuo, W., Chen, Y., Meng, D., & Zhang, L. (2017). Beyond a gaussian denoiser: Residual
learning of deep cnn for image denoising. IEEE Transactions on Image Processing, 26(7),
3142-3155.
Batson, J., & Royer, L. (2019). Noise2self: Blind denoising by self-supervision. arXiv preprint arXiv:
1901.11365.

ECCO: An Electron Counting Implementation for Image Compression and Optimization of Cryo-EM Data Management

More Related Content

What's hot

Similar to ECCO: An Electron Counting Implementation for Image Compression and Optimization of Cryo-EM Data Management

More from NECST Lab @ Politecnico di Milano

Recently uploaded

ECCO: An Electron Counting Implementation for Image Compression and Optimization of Cryo-EM Data Management