The document discusses p4pktgen, a tool for automated test case generation for P4 programs which addresses the bugs introduced by the flexibility of programmable network devices. It utilizes symbolic execution with SMT solvers to effectively generate test cases and has successfully identified and fixed multiple issues within the P4 toolchain. Future work includes enhancing support for additional P4 features and improving test case generation techniques.

1. p4pktgen:
Automated Test Case
Generation for P4 Programs
Ajay Kharat (20191030011G)
BITS-Pilani

2. Motivation
- Traditional network devices - fixed set of capabilities
- Rise of programmable network devices in recent years
- Offers great flexibility / capability than traditional network devices
- Flexibility introduces new bugs:
- Hardware
- Toolchains
- Programs
- These bugs were previously covered by traditional network devices due to
fixed set of capabilities
- Use test cases to check whether program is behaving as intended on the
device
2

3. Problem Statement
- Programmable network devices and programmable languages (P4)
- Offers more flexibility / capability
- Uniform support across different devices
- Introduces new layers and components that can fail
- Subtle differences between toolchains
- Different interpretation of the language specification
- Need for new tools
- Keeps benefits of programmable hardware
- Keep/improve on the reliability of traditional approaches
- p4pktgen tool for automatic generation of test cases for P4 programs for
validation
3

4. Approach
and
Solution
4

5. Symbolic Execution using SMT Solvers
Analysis technique that translates program into logical formulas to
examine its behaviour on all inputs
Reasoning done by satisfiability modulo theory (SMT) solvers:
Input: Constraints
Example: (x < y) ^ (y < x - 1)
Output : A variable assignment that makes the formula true or
unsatisfiable.
5

6. Parser Translator
6

7. Evaluation of the solution
1. Effective generation of test cases for large P4 programs
For smaller programs, able to generate test cases for all paths in under 3
minutes.
7

8. 2. Generate test cases to reveal bugs in P4 toolchain
- Issues with p4c JSON backend (4 issues confirmed, current status: fixed)
- Incorrect JSON for select statement with multiple key fields
- Incorrect JSON when bit-wise negating a bitvector and casting it to a wider bitvector
- Incorrect JSON for select statements that use masks for ternary matching
- Incorrect JSON specifying maximum length of headers with variable-length fields
- Danger of bugs in P4 compilers are real
- Test cases can help reveal them
Evaluation of the solution
8

9. Related Work
- Foster et al - verification of P4 programs
Translate P4 programs to a small imperative language and then verify that user-
annotated properties hold using an SMT solver.
- Kheradmand et al - formalization of P4 programs in K framework.
Their focus is on P414 whereas Author’s approach is version-agnostic used and a
direct formalization in SMT. Test case generation is one of their side goals.
- Zeng et al - automatically generating test packets for testing and debugging
networks.
Uses header space framework to model packet processing
requires implementation of protocol and vendor-specific translations into their model
and assuming a fixed table configuration.
Author’s focuses on a single switch and generate table configurations automatically.
9

10. Future Work
- Support for more P4 features (checksums, action profiles etc)
- Branch coverage to find set of paths that covers all branches
- User-supplied constraints / assertions
- Test different P4 implementations
- Modeling egress pipeline
10