RISC-V Summit 2020 presentation

1. December 8-10, 2020
Virtual Event
Ivannikov Institute for System Programming
Russian Academy of Sciences
Nikita Chertok, Mikhail Chupilko, Alexander Kamkin
Online Test Program Generator for
RISC-V Microprocessors

2. MicroTESK Framework
MicroTESK Framework
Specifications
(nML)
Specifications
Translator
Test Programs
(ASM)
Test Program
Generator
Binary Code
Analyzer
Tool
Builders
Processor
Model
Coverage
Simulator
Metadata
lui a0,0xdead
ori a0,a0,0x0
lui a1,0xbeef
ori a1,a1,0xf
add t0,a0,a1
Simulator
(C/C++)
Specifications
(WhyML)
…
Ref. Simulator
(QEMU)
Specifications
Debug
1
https://forge.ispras.ru/projects/microtesk

3. op add (rd: X, rs1: X, rs2: X)
syntax = format("add %s, %s, %s",
rd.syntax, rs1.syntax, rs2.syntax)
image = format("0000000%s%s000%s0110011",
rs2.image, rs1.image, rd.image)
action = {
rd = rs1 + rs2;
}
mode X (i: card(5)) = XREG[i]
syntax = format("%s",
if i==0 then ZERO().syntax
elif i==1 then RA().syntax
elif i==2 then SP().syntax
elif i==3 then GP().syntax
elif i==4 then TP().syntax
elif i>=5 && i<=7 then
Temp(coerce(card(3), i-5)).syntax
elif i>=8 && i<=9 then
Saved(coerce(card(4), i-8)).syntax
elif i>=10 && i<=17 then
Func(coerce(card(3), i-10)).syntax
elif i>=18 && i<=27 then
Saved(coerce(card(4), i-16)).syntax
else
Temp(coerce(card(3), i-25)).syntax
image = format("%5s", i)
nML Language Overview
2
#ifdef RV64I
let XLEN = 64
...
#else
...
#endif
let MEMORY_SIZE_IN_WORDS = 2 ** (XLEN – 2)
shared mem MEM[MEMORY_SIZE_IN_WORDS, WORD]
type WORD = card(32)
type XWORD = card(XLEN)
type FLOAT32 = float(23, 8)
reg XREG [32, XWORD]
reg PC [XWORD]

4. Specifications
3
Specifications Characteristic Value
RISC-V Instruction Set Manual Vol. I: User-Level ISA (v. 2.2) 145 pages (94 pages in chapters 2-18)
Base Extension Base Extension In Total
RV32I M F C P
24 pages 44 pages 68 pages
RV32E A D B V
RV64I Q J N
2 pages 24 pages 26 pages
RV128I L T
# Specified Instructions 230 instructions
ISA Specifications Size (nML) 4 900 LOC (w/o comments)
MMU Specification Size (MMUSL) 460 LOC (Sv32, Sv39, and Sv48)

5. Offline Test Program Generation
4
Design Under Test
(RTL, FPGA)
lui a0, 0xdead
ori a0, a0, 0x0
lui a1, 0xbeef
ori a1, a1, 0xf
add t0, a0, a1
sub t1, a0, t1
add t0, t0, t1
Test Programs
(ASM)
Reference Simulator
(C/C++)
Traces (Tarmac) /
Signatures
0x2000: lui ...
0x2004: ori ...
0x2008: lui ...
0x200c: ori ...
0x2010: add ...
0x2014: sub ...
0x2018: add ...
Comparator
(Python)
0x2000: lui ...
0x2004: ori ...
0x2008: lui ...
0x200c: ori ...
0x2010: add ...
0x2014: sub ...
0x2018: bug ...
Traces (Tarmac) /
Signatures

6. What’s Inside?
?
5

7. MicroTESK Test Program Generator (Offline)
MicroTESK Test Program Generator
Specifications (nML)
Translator
Test Templates (Ruby)
lui a0, 0xdead
ori a0, a0, 0x0
lui a1, 0xbeef
ori a1, a1, 0xf
add t0, a0, a1
sub t1, a0, t1
add t0, t0, t1
Test Programs
(Assembly Code)
Processor Model
Generation Core
Specification Engineer
Verification Engineer
6
Autogen
https://forge.ispras.ru/projects/microtesk-riscv

8. Test Templates (Offline)
 Test templates similar to RISC-V Foundation’s Tests
https://github.com/riscv/riscv-tests
 Test templates similar to RISC-V Torture Test Generator
https://github.com/ucb-bar/riscv-torture 7
class MyTemplate < RiscVBaseTemplate
def run
block(:combinator => 'product',
:compositor => 'random') {
iterate {
xor x(_), x(_), x(_) do situation(...) end
lui x(_), _
}
iterate {
and x(_), x(_), x(_)
or x(_), x(_), x(_)
}
}.run
end
end
# Single Test Case
# Initialization
ori a7, a7, 0x2d7
slli a7, a7, 0xb
ori a7, a7, 0x1
slli a7, a7, 0xb
ori a7, a7, 0x3d2
ori t3, t3, 0x164
slli t3, t3, 0xb
ori t3, t3, 0x52b
slli t3, t3, 0xb
ori t3, t3, 0x24e
# Stimulus
and s4, a7, a7
xor s8, s4, t3
2 × 2 = 4 test cases
• Combinatorial Brute Force
• Randomization
• Constraint Solving
• Self Checking
Code is distributed under
Apache License, v2.0

9. Online Test Generation
Offline Test Generation
lui a0, 0xdead
ori a0, a0, 0x0
lui a1, 0xbeef
ori a1, a1, 0xf
add t0, a0, a1
sub t1, a0, t1
add t0, t0, t1
Test Program Generator
(Standalone Software)
Test Programs
(Binaries)
Design Under Test
(RTL, FPGA, IC)
DUT executes loaded
test programs
lui a0, 0xdead
ori a0, a0, 0x0
lui a1, 0xbeef
ori a1, a1, 0xf
add t0, a0, a1
sub t1, a0, t1
add t0, t0, t1
Test Program Generator
(Binary Inside DUT)
Test Programs
(Binaries)
Design Under Test
(FPGA, IC)
DUT executes
test generator
that produces
test programs
Tests programs are
loaded to memory
8

10. Online TPG Requirements
 Autonomy
 Self-generation of a variety of tests from a template
 Self-checking of tests results
 Performance
 Test massiveness prevails over test directedness
 Simple techniques (brute force, randomization, mutation, etc.)
 Robustness
 No loss of control (exception handling, timers, etc.)
 Saving and restoring generator state
9

11. Online TPG Architecture
Control
Unit
Allocator
Executor
Test Oracle
add …
sub …
...
add …
nop
sub …
Generator
Transformer
Residential Part Test Image
External Part
Control
Console
Generator
Commands
Parameters
Statistics
Coverage
Failures
.text
110011010100000…
110011011100000…
...
.data
1101111010101101
.text
110011010100000…
110010000000000…
110011011100000…
.data
1101111010101101
Test w/ mutant (QED)
Verdict
.data
1101111010101101
1011111011101111
.data
1101111010101101
1011101011101111
Test Results
Test Code
Allocation, linking, etc.
Signatures
Metadata
Allocates code in memory
Gets signatures and compares them
Requests next test case
Transfers control to
tests and returns back
Deviation
10

12. MicroTESK Test Program Generator (Online)
MicroTESK Test Program Generator
Specifications (nML)
Translator
Test Templates (Ruby)
Test Generator
(Code in C)
Processor Model
Generation Core
Specification Engineer
Verification Engineer
11
Autogen
https://forge.ispras.ru/projects/microtesk-riscv
Generator
Transformer

13. Test Templates (Online)
 Test template is translated to C language:
combinators, compositors, preparators, etc.
 Generator in C implements a certain interface:
can be obtained by other means (e.g., Autogen)
class MyTemplate < RiscVBaseTemplate
def run
block(:combinator => 'product',
:compositor => 'random') {
iterate {
xor x(_), x(_), x(_) do ... end
lui x(_), _
}
iterate {
and x(_), x(_), x(_)
or x(_), x(_), x(_)
}
}.run
end
end
// MyTemplate (simplified)
OP *iterate_0[] = { {XOR}, {LUI} };
OP *iterate_1[] = { {AND}, {OR} };
// :combinator => 'product'
for (i[0] = 0; i[0] < 2; i[0]++)
for (i[1] = 0; i[1] < 2; i[1]++) {
OP *combine[] = tpg_malloc(...)
combine[0] = iterate_0[i[0]];
combine[1] = iterate_1[i[1]];
// :compositor => 'random'
OP compose[] = tpg_malloc(...);
k = j[0] = j[1] = 0;
while (j[0] < 1 && j[1] < 1) {
while (j[s = rand() % 2] == 2);
compose[k++] = combine[s][j[s]++];
}
// choose registers, generate data,
// insert initialization code, etc.
...
// get the test’s binary image
for (int n = 0; n < N; n++) {
binary[n] = test[n].image();
}
} 12

14. Online TPG Features
 Test generation techniques
 Combinatorial brute-force generation (“perebor”)
 Randomization
 No constraint solving!
 Test result checking techniques
 Precalculated signatures returned by generator
 Equivalent (modulo inputs) transformations
 Dead code injection (NOPs between instructions, etc.)
 Quick error detection (QED)
13

15. Future Work Directions
 RISC-V specifications
 Specifying more subsets (RV128I, Q, L, B, T, J, P, V, and N)
 Online test program generation
 Error localization and diagnostics
 Test coverage tracking and test completeness assessment
 Advanced test program generation
 Model-based generation (modeling compiler transformations)
 Mutations and coverage-directed fuzzing
 Binary code equivalence checking
 Verifying optimizing compilers (including JIT and AOT) 14

16. http://www.microtesk.org
microtesk-support@ispras.ru
Thank
You!