This document evaluates the built-in error correction code (ECC) of FPGA on-chip memories for mitigating faults caused by aggressive undervolting. Experimental results on multiple FPGA platforms show that undervolting below the minimum safe voltage introduces mostly single-bit faults in BRAMs. The built-in ECC is over 90% effective in correcting faults and prevents significant accuracy loss in a neural network implementation when voltages drop below safe levels. While ECC adds minimal area and power overhead, it effectively allows further energy savings from aggressive undervolting of FPGA memories.

1. www.bsc.es
Evaluating Built-in ECC of FPGA on-Chip
Memories for the Mitigation of
Undervolting Faults
Behzad Salami, Osman S. Unsal, and Adrian C. Kestelman
Barcelona Supercomptuing Center (BSC)
Barcelona, Spain.
27th Euromicro International Conference on Parallel, Distributed, and Network-based Processing
(PDP), Pavia, Italy. 14th Feb, 2019.

2. 2
Aggressive Undervolting
 Aggressive undervolting- Underscaling the supply voltage below the nominal
and safe level:
 Power/Energy Efficiency: Reduces dynamic and static power quadratically and
linearly, respectively.
 Reliability: Increases the circuit delay and in turn, causes timing faults.
 Dual/Multi-Vdd, DVS, and DVFS: Similar but different mechanisms to
aggressive undervolting:
 Similarity: Underscaling the supply voltage.
 Difference: Undervolting is until a certain safe level, usually constrained by
vendors.
Reliability
Power/Energy
Efficiency

3. 3
State-of-the-art
1. Aggressive undervolting has shown significant efficiency
to reduce the energy consumption.
 Devices:
 CPUs: Itanium II (ISCA2014), X86 (IOLTS2017), ARM (HPCA2017)
 GPUs: NVidia (Micro2015)
 DRAMs: Multiple Brands (Sigmetrics2017)
 FPGAs: [Micro18, FPL18, This work]
 Focus of the previous works:
 Voltage guardband
 Minimum safe voltage, i.e., Vmin prediction
 Fault characterization and mitigation
 Chip-to-chip, core-to-core, and workload-to-workload variation
 ….
2. More straightforward and more parameters
but less precise
 ASIC DNN: Minerva (Micro2016), Thundervolt (DAC2018)
 CPU: Bravo (HPCA2017 )
 Network On-Chip (HPCA2014)
Real hardware:
Simulation-based studies:
for on-chip memories via built-in ECC

4. 4
Undervolting on FPGAs: Motivation
Contribution of FPGAs in large data centers is growing, expected to
be in 30% of datacenter servers by 2020 (Top500 news).
 In comparison to ASICs,
energy efficiency of FPGAs is
a serious concern, i.e., 10X-
100X less-efficient.
 Nominal voltage reduction
of FPGAs is naturally applied
for different generations.
Undervolting
[Intel/Altera]
[Xilinx]

5. 5
Outline
1. Background on FPGAs Undervolting
 Experimental Methodology
 Voltage guardband
 Overall power and reliability trade-off
2. Undervolting Faults in FPGA on-chip memories
 Efficiency of the built-in ECC
 Fault characterization
3. Low-voltage FPGA-based Neural Network (NN)
 Power consumption and NN accuracy characterization
 Efficiency of the built-in ECC to prevent accuracy loss

6. 6
Voltage Scaling Capability in Xilinx
VC707: performance-efficient design
KC705: power-efficient design (A & B)
Evaluated Xilinx platforms
VC707
Voltage distribution on Xilinx platforms
Voltage regulator
 Power Management Bus (PMBus).
 Hardwired to the host.
ZC702: ARM integrated with FPGA
VCCINT
VCCBRAM

7. 7
Floorplan of VC707
Experimental Methodology
 FPGA BRAMs:
 Hierarchy of set of bit-cells
distributed over the chip.
 Size of each BRAM: 16-kbits
 Experimental Methodology:
 HW: Transfer content of BRAMs to the host.
 SW: Analyze data, and adjust voltage of BRAMs. (2060 BRAMs)

8. 8
Overall Voltage Behavior
Vnom
Vmin
Vcrash
SAFE (Guardband)
CRITICAL
CRASH
0
0.2
0.4
0.6
0.8
1
VC707 ZC702 KC705-A KC705-B
VCCBRAM(V)
Platform
SAFE (Guardband)
CRITICAL
CRASH
 FPGA stops operating below
Vcrash, min operating voltageCRASH
 No observable fault
 Voltage Guardband below Vnom
SAFE
 Faults manifest
 Below Vmin, min safe voltageCRITICAL
 Voltage guardband: to ensure
the worst-case environmental
and process technologies.
 Experimental conditions: At
ambient temperature and
maximum operating frequency.

9. 9
0
200
400
600
800
0
1
2
3
1 0.95 0.9 0.85 0.8 0.75 0.7 0.65 0.6 0.55
FaultRate(per1Mbit)
BRAMPower(Watts)
VCCBRAM (V)
BRAM Power
Fault Rate
Vmin=0.61V
Vcrash=0.54V
0
400
800
0
0.1
0.2
0.3
0.4
0.61 0.6 0.59 0.58 0.57 0.56 0.55 0.54
per1Mbit
Watts
Vnom=1V
Overall Trade-offs on BRAMs- Power & Reliability
VC707

10. 10
0
150
300
0
0.05
0.1
0.15
0.61 0.6 0.59 0.58 0.57 0.56 0.55 0.54
per1Mbit
Watts
Overall Trade-offs on BRAMs- Multiple Platforms
0
200
400
600
800
0
1
2
3
1 0.95 0.9 0.85 0.8 0.75 0.7 0.65 0.6 0.55
FaultRate
(per1Mbit)
BRAMPower
(Watts)
VCCBRAM (V)
Vnom=1V
Vmin=0.61V
0
400
800
0
0.2
0.4
0.61 0.6 0.59 0.58 0.57 0.56 0.55 0.54
per1Mbit
Watts
VC707
0
50
100
150
200
0
10
20
30
1 0.95 0.9 0.85 0.8 0.75 0.7 0.65 0.6 0.55
FaultRate
(per1Mbit)
BRAMPower
(mWatts)
VCCBRAM (V)
Vnom=1V
Vcrash=0.53V
0
100
200
0
2
4
0.59 0.58 0.57 0.56 0.55 0.54 0.53
per1Mbit
mWatts
ZC702
0
100
200
300
0
1
2
3
1 0.95 0.9 0.85 0.8 0.75 0.7 0.65 0.6 0.55
FaultRate
(per1Mbit)
BRAMPower
(Watts)
VCCBRAM (V)
Vnom=1V
Vcrash=0.54V
Vmin=0.61V
KC705-A
0
20
40
60
80
0
1
2
3
1 0.95 0.9 0.85 0.8 0.75 0.7 0.65 0.6 0.55
FaultRate
(per1Mbit)
BRAMPower
(Watts)
VCCBRAM (V)
Vnom=1V
Vmin=0.57V
Vcrash=0.54V
0
40
80
0
0.05
0.1
0.15
0.57 0.56 0.55 0.54
per1Mbit
Watts
KC705-B
Vmin=0.59V
Vcrash=0.54V

11. 11
Outline
1. Background on FPGAs Undervolting
 Experimental Methodology
 Voltage guardband
 Overall power and reliability trade-off
2. Undervolting Faults in FPGA on-chip memories
 Efficiency of the built-in ECC
 Fault characterization
3. Low-voltage FPGA-based Neural Network (NN)
 Power consumption and NN accuracy characterization
 Efficiency of the built-in ECC to prevent accuracy loss

12. 12
Built-in ECC
 Built-in ECC of FPGA BRAMs:
 Hamming-code.
 Two (2) additional bits per row are reserved as parities.
 SECDED (Single-Error Correction and Double-Error Detection).
 Experimental Methodology:
 Activate built-in ECC under low-voltage read operations.
 Experimental Observations:
 >90% fault correction
 >7% fault detection (not correction)
0
200
400
600
800
0.61 0.6 0.59 0.58 0.57 0.56 0.55 0.54
Faultrate(per1Mbit)
VCCBRAM (V)
Without ECC With ECC
Parity Bits
single-bit
double-bit
multiple-bit

13. 13
Fault Characterization
Location of undervolting faults:
Fault Inclusion Property (FIP)
 FIP: A corrupted bit at a specific voltage stays faulty in lower voltages as well.
 FIP can be used in mitigation techniques.
0.1
1
10
100
1000
10000
0.61 0.6 0.59 0.58 0.57 0.56 0.55 0.54
FaultRate(per1Mbit)
logscale
VCCBRAM (V)
Illustration of FIP FIP shown as fault rate for VC707

14. 14
Fault Characterization
 Fully non-uniform fault distribution.
 Majority of BRAMs do not experience many faults.
Fault variability among FPGA BRAMs:
Fully non-uniform fault distribution
VC707 (2060 BRAMs)
VCCBRAM@ Vcrash= 0.54V
Temperature@ Ambient
0.0%
0.3%
0.6%
0.9%
1.2%
1.5%
BRAMFaultRate(%)
%BRAMs Average Fault Rate (%)
1.8% 0.86%
High-vulnerable
9.4% 0.24%
Mid-vulnerable
52.3% 0.03%
Low-vulnerable
36.3% 0.0%
Zero-vulnerable
K-means clustering

15. 15
Outline
1. Background on FPGAs Undervolting
 Experimental Methodology
 Voltage guardband
 Overall power and reliability trade-off
2. Undervolting Faults in FPGA on-chip memories
 Efficiency of the built-in ECC
 Fault characterization
3. Low-voltage FPGA-based Neural Network (NN)
 Power consumption and NN accuracy characterization
 Efficiency of the built-in ECC to prevent accuracy loss

16. 16
Experimental Methodology
Neural Network (NN)
Type Fully-connected classifier
Total number of weights ~1.5 millions
Activation function Logsig (logarithmic sigmoid)
Major benchmark
Name-type MNIST- handwritten digit images
Number of images Training: 60000, Classification: 10000
Number of pixels per image 28*28=256
Number of output classes 10
Additional benchmarks
Names Forest and Reuters
Data representation model
Type 16-bits fixed-point
Precision Minimum sign and digit per layer
An example implementation on VC707
Frequency 100 Mhz
BRAM usage (total: 2060) 70.8%

17. 17
NN Implementation on FPGA
 Input data: off-chip DDR memory.
 Weights: on-chip FPGA BRAM.
 Computation: Streaming data
onto DSPs and LUTs.
 We undervolt VCCBRAM:
 Weights of the NN are
potentially affected.
FPGA Implementation

18. 18
Low-Voltage FPGA-based NN
 Significant power reduction until
the minimum safe voltage, i.e., Vmin
(By eliminating the voltage
guardband).
 Additional 40% power reduction
below the voltage guardband.
 The NN classification error exponentially
increases from 2.56% (inherent
classification error) to 6.74% through
undervolting BRAMs beyond Vmin.
 Fault mitigation techniques to prevent
the accuracy loss:
 Built-in ECC
Power saving
NN accuracy loss
2.39
0.25 0.15
6.47
6.47 6.47
0
2
4
6
8
10
Vnom= 1 V Vmin= 0.61V Vcrash= 0.54V
On-chipPower(Watts)
BRAM Rest
0
100
200
300
400
0%
2%
4%
6%
8%
0.61 0.6 0.59 0.58 0.57 0.56 0.55 0.54
FaultRatio(per1Mbit)
ClassificationError(%)
VCCBRAM (V)
NN Error Ratio Fault Ratio
Inherent Error rate: 2.56%

19. 19
ECC for NN Accelerator
0%
2%
4%
6%
8%
0.61 0.6 0.59 0.58 0.57 0.56 0.55 0.54
ClassificationError(%)
VCCBRAM (V)
Without ECC With ECC
Inherent NN Error: 2.56%
Area Utilization (%)
BRAM LUT FF
Without ECC 96% 3% 0.25%
With ECC 100% 12% 0.25%
BRAM Power (W)
Vnom= 1V Vmin= 0.61V Vcrash= 0.54V
Without ECC 2.4 0.31 0.198
With ECC ---- ---- 0.211
ECC efficiency to prevent NN accuracy loss
ECC area and power costs
 Pros:
 Significant accuracy loss prevention.
 Negligible power and performance overhead.
 Cons:
 Requires larger data rows/lines.
 2/18 hardware (SRAM bits) cost.

20. 20
Wrap up
 Summary and Conclusion
 Below the minimum safe voltage, undervolting faults may
appear in FPGA BRAMs. They are mostly single-bit errors.
 Built-in ECC of FPGA BRAMs has considerable efficiency to
cover undervolting faults.
 Future Works
 Selective ECC activation in FPGA BRAMs
 More optimized fault mitigation techniques
 Evaluating additional state-of-the-art applications
 ….

21. 21
Acknowledgement
https://legato-project.eu/

22. www.bsc.es
Thanks!
Any Question/Comment?
Contact:
behzad.salami@bsc.es