This document describes a protocol test generator that uses nested virtual machines and rollback mechanisms to perform exhaustive fuzz testing of protocol implementations. It proposes using a virtual test protocol to encapsulate test packets and control the target virtual machine. Special packets allow taking snapshots of the target VM state and rolling back to previous snapshots to repeatedly test protocol states with different fuzzed packets. The current prototype implements this approach with KVM and QEMU virtual machines to find bugs in TLS/SSL protocol implementations through fuzz testing of the handshake process.

1. National Institute of Advanced Industrial Science and Technology
Rollback MechanismRollback Mechanism
of Nested Virtual Machines
for Protocol Fuzz Testing
Kuniyasu Suzaki*, Toshiki Yagi*, Akira Tanaka*,
Y taka Oi a* Ets a Shiba ama*‡Yutaka Oiwa , Etsuya Shibayama ‡
* National Institute of Advanced Industrial Science and Technology(AIST)National Institute of Advanced Industrial Science and Technology(AIST)
‡The University of Tokyo
ACM SAC(Symposium On Applied Computing)2014 at Gyeongju, South Korea March 25

2. National Institute of Advanced Industrial Science and Technology
Motivation 1/2Motivation 1/2
• Sec re protocols (SSL/TLS SSH etc) are important for• Secure protocols (SSL/TLS, SSH, etc) are important for
Internet commerce.
l bili i f i l i f• However, vulnerabilities of implementation of secure
protocols are discovered repeatedly.
– Incidents of SSL/TLS vulnerabilities
• Insufficient entropy of random numbers [2012]
– It creates same secret-keys for SSL/TLS certificates.It creates same secret keys for SSL/TLS certificates.
– EFF offers a site to check SSL certificates.
» https://www.eff.org/observatory
• CRIME[2012]• CRIME[2012]
– It attacks to the data compression used by the TLS and SPDY.
• Lucky Thirteen[2013]
– It attacks padding oracle using timing side-channel.

3. National Institute of Advanced Industrial Science and Technology
Motivation 2/2Motivation 2/2
• The implementations of secure protocols should be verified.
• Our group increases TLS security with 2 approaches.
F l ifi i– Formal verification
• Verify important parts of protocol hander program.
– “Formal Network Packet Processing with Minimal Fuss: Invertible Syntax Descriptions atFormal Network Packet Processing with Minimal Fuss: Invertible Syntax Descriptions at
Work.” [PLPV’12]
– “Towards Formal Verification of TLS Network Packet Processing Written in C”
[PLPV’13]
– Exhaustive Protocol Fuzz Testing
• Try to check all reaction of protocol handler.
– Protocol Test Generator creates and sends many fuzzed packets to check implementation
of protocol handler.
– (Today’s Topic) Computing environment for protocol fuzz testing

4. National Institute of Advanced Industrial Science and Technology
O liOutline
M i i• Motivation
• Protocol Test Generator: fuzz testing on a protocol handler
• Why we use nested VM for protocol fuzz testing?
• Design issuesDesign issues
• Implementation
P f l i• Performance evaluation on current prototype.
• Conclusion

5. National Institute of Advanced Industrial Science and Technology
Protocol Test Generator
V if t l h d f h t l t t• Verify a protocol hander for each protocol state
– Repeat the following sequence for each protocol state
1. Creates a fuzzed packet
2. Send it to a protocol hander
3. Check the reply
• Fuzzed packets are tested on each protocol stateFuzzed packets are tested on each protocol state
many times.
• Snapshot and rollback mechanism is required.S p q

6. National Institute of Advanced Industrial Science and Technology
Sequence of Protocol Test Generator (TLS)Sequence of Protocol Test Generator (TLS)
Normal
TargetProtocol Generator
Client Hello
Take Snapshot
Client Hello
Server Hello
Server Certificate
Snapshot1
Server Hello Done
Take SnapshotTake Snapshot
Client Key Exchange
… Snapshot 2

7. National Institute of Advanced Industrial Science and Technology
Sequence of Protocol Test Generator (TLS)Sequence of Protocol Test Generator (TLS)
Normal Fuzz for Client Hello
TargetProtocol Generator TargetProtocol Generator
Client Hello
Take Snapshot
[Fuzz1] Client Hello
Rollback Snapshot1
Client Hello
Server Hello
Server Certificate
Snapshot1
[Fuzz1] Client Hello
????
Good or Bad?
Rollback1
Server Hello Done
Take Snapshot Rollback Snapshot1
Good or Bad?
Take Snapshot
Client Key Exchange
… Snapshot 2
[Fuzz2] Client Hello
????
Rollback 1
Good or Bad?
[FuzzN] Client Hello

8. National Institute of Advanced Industrial Science and Technology
Sequence of Protocol Test Generator (TLS)Sequence of Protocol Test Generator (TLS)
Normal Fuzz for Client Hello Fuzz for Client Key Exchange
TargetProtocol Generator TargetProtocol Generator TargetProtocol Generator
Client Hello
Take Snapshot
[Fuzz1] Client Hello
Rollback Snapshot1
[Fuzz1] Client Key
Rollback Snapshot2
Client Hello
Server Hello
Server Certificate
[Fuzz1] Client Hello
????
Good or Bad?
Snapshot1 Rollback
1
[Fuzz1] Client Key
Exchange
???
Rollback2
Server Hello Done
Take Snapshot Rollback Snapshot1
Good or Bad?
Rollback Snapshot2
Good or Bad?
Take Snapshot
Client Key Exchange
…
[Fuzz2] Client Hello
????
Snapshot 2 Rollback 1 Rollback 2
[Fuzz2] Client Key
Exchange
???
[FuzzN] Client Hello
Good or Bad?
???
Good or Bad?
[FuzzN] Client Key
Exchange

9. National Institute of Advanced Industrial Science and Technology
Requirement for Protocol Test GeneratorRequirement for Protocol Test Generator
1. Packet level granularity control
– Generator wants to take a snapshot just before sending a packet.
2. Hardware-level repeatability
B l biliti d b it ti i t (– Because some vulnerabilities are caused by its computing environment (e.g.,
random number)
3. Consistency between generator and target protocol hander3. Consistency between generator and target protocol hander
– Most snapshot/rollback mechanism does not care of packets on a wire.
1. We propose a special protocol to control target computing
environment.
2. We uses Virtual Machine which offers hardware-level repeatability
f h / llb kfor snapshot/rollback.
3. Design issues. (Mention in later slides)

10. National Institute of Advanced Industrial Science and Technology
Proposed Protocol (VTP: Virtual Test Protocol)Proposed Protocol (VTP: Virtual Test Protocol)
• VTP treats packets of test-target protocol as data.
T t t t k t l t d t “C l ” k t– Test-target packets are en-capsulated to “Capsule” packets
– “Capsule” packets must be de-capsulated on target computing environment.
• VTP has special packets to control target computing environment• VTP has special packets to control target computing environment
– “Take_Snapshot” and “Rollback”
Protocol Action
Capsule
Packets of secure communication are encapsulated with
capsule protocol. The flow of capsulated packets is managed
h t k h t llb k k t i i dwhen take_snapshot or rollback packet is issued.
Take_
Snapshot
Take VM snapshot image. Packets on the wire must be
managed by a certain method. Return snapshot ID.
R h i C i f
Rollback
(with ID)
Resume snapshot image. Connection of secure
communication must be operative, when a previous image is
resumed.
Target computing environment (VM) has to deal with the VTP.

11. National Institute of Advanced Industrial Science and Technology
Design Issues for VM (Computing environment)Design Issues for VM (Computing environment)
• Virtual Machine runs a protocol handler.
• En-capsulated packets by VTP must be de-capsulated on a target VM.
– Original packets must be passed to a target protocol handler on the VM.
• The control packets (take_snapshot and rollback) must be dealt by the VM.
• The VTP packets on the network must be maintained.
– We have to deal with consistent global state defied by [Chandy-Lamport, 1985].
Protocol
Generator
Target Environment
Snapshots of VM
VTP
Protocol
handler
TLS
VTP
VM deals with VTP protocol.
Packets on the network
must be maintained.
VM

12. National Institute of Advanced Industrial Science and Technology
Design choice for handling VTPDesign choice for handling VTP
• Customize VM’s network function for VTPCustomize VM s network function for VTP
– We planned to customize SLiRP of QEMU
• However VM control (snapshot/rollback) from SLiRP is strange, because it( p ) g ,
is a part of VM. It is not flexible.
• Nested VM and Proxies (external & internal)
– External proxy manages VTP control packets and takes snapshot of
External VM.
– A snapshot of External VM keeps connectivity of network between
internal proxy and a protocol handler on Internal VM.
• Pros: Easy and flexible implementation.
» It utilizes existing VM control commands. Don’t need to change VM.
• Cons: Overhead? Performance is improved by many previous researchesCons: Overhead? Performance is improved by many previous researches.

13. National Institute of Advanced Industrial Science and Technology
Design: Nested VM and Proxiesg
• External VM (a unit of snapshot)
– It is used to take snapshot of Internal VM and Internal Proxy.
• Internal VM (hardware repeatability)( p y)
– It includes protocol hander and keeps connection with Internal Proxy, when External VM is taken a snapshot.
• External Proxy
– Connects to Protocol Generator directly, and controls snapshot/rollback of External VM.
Real Machine External VMControl the target VM
• Internal Proxy
– Maintain connectivity of VTP Protocol when a snapshot image is resumed.
Real Machine External VM
Internal VM
Control the target VM
(SAVEVM, LOADVM)
Decapsulate
control commands
Internal
Ｐｒｏｘｙ openSSL
Port forward
External
Ｐｒｏｘｙ
External
Ｐｒｏｘｙ
Internal
ＰｒｏｘｙTLS
VTP
TLSTLS
VTP
External VM
Internal VM
External VM
Internal VM
Protocol
Generator “take_snapshot” packet make a
round‐trip between external
and internal proxy to confirm
no VTP packets on the wire
13
Internal VM
SSH
openSSL
Keep connectionSnapshots
no VTP packets on the wire.
Rollback
TLS

14. National Institute of Advanced Industrial Science and Technology
Handling VTP “capsule” packetHandling VTP capsule packet
• Protocol Generator en-casuplates test-target packets with “capsule” packets.
• External Proxy passes capsule packets to Internal Proxy without change.
• Internal Proxy de-capsulates VTP packets and passes original packets to protocol
hander on Internal VMhander on Internal VM.
Real Machine External VMReal Machine External VM
Internal VM
Decapsulate
control commands
Internal
Ｐｒｏｘｙ openSSL
Port forward
External
Ｐｒｏｘｙ
External
Ｐｒｏｘｙ
Internal
ＰｒｏｘｙTLS
VTP
TLSTLS
VTP
Protocol
Generator

15. National Institute of Advanced Industrial Science and Technology
Handling VTP “take snapshot” packetHandling VTP take_snapshot packet
• When a “take_snapshot” packet is issued, the packet makes a round trip between
External and Internal Proxy to confirm that there is no VTP packet on the wire
(consistent global state defied by Chandy Lamport)(consistent global state defied by Chandy-Lamport).
• After that, External Proxy sends SAVEM command to External VM to take a
snapshot.
• Connection between Internal Proxy and protocol handler is kept in a snapshot.
Real Machine External VMControl the target VMReal Machine External VM
Internal VM
g
(SAVEVM)
Decapsulate
control commands
Internal
Ｐｒｏｘｙ openSSL
Port forward
External
Ｐｒｏｘｙ
External
Ｐｒｏｘｙ
Internal
Ｐｒｏｘｙ
P t l
TLS
VTP
TLSTLS
VTP
External VM
Internal VM
External VM
Internal VM
Protocol
Generator “take_snapshot” packet make a
round‐trip between external
and internal proxy to confirm
no VTP packets on the wire.
15
Internal VM
SSH
openSSL
Keep connectionSnapshots
no VTP packets on the wire.
TLS

16. National Institute of Advanced Industrial Science and Technology
Handling VTP “rollback” packetHandling VTP rollback packet
• When a “rollback” packet is issued, the External Proxy send
LOADVM command to External VM It resumes an old imageLOADVM command to External VM. It resumes an old image.
• After that, External and Internal proxies re-establish the connection
between them, because they loses the network states when rollback., y
– Internal proxy and protocol handler do not need to care about network connection.
Real Machine External VMControl the target VMReal Machine External VM
Internal VM
g
(LOADVM)
Decapsulate
control commands
Internal
Ｐｒｏｘｙ openSSL
Port forward
External
Ｐｒｏｘｙ
External
Ｐｒｏｘｙ
Internal
Ｐｒｏｘｙ
P t l
TLS
VTP
TLSReestablish
External VM
Internal VM
External VM
Internal VM
Protocol
Generator
16
Internal VM
SSH
openSSL
Keep connection Rollback
TLS

17. National Institute of Advanced Industrial Science and Technology
OptimizationOptimization
• Most overhead of VM is caused by video emulation.
– Nested VMs must emulate the GUI of internal VM on external VM again.g
• Fortunately QEMU and KVM have “curses mode” which emulates
text user interface.
• We eliminate GUI on both internal and external VMs.
N GUI d d k h t i k– No GUI reduces memory usage and make snapshot quick.
GUIs of Nested VMs No GUI of Nested VMs
OpenSSL on Linux on
Internal VM (QEMU)
on External VM (KVM)
ith d
Internal VM
(QEMU)
OpenSSL on
Windows
Extenal Proxy
Protocol Tester
with curses mode
Protocol Tester
External Proxy
External VM (KVM)
Internal Proxy on
Linux

18. National Institute of Advanced Industrial Science and Technology
Current ImplementationCurrent Implementation
• We implement Nested VM and proxies on normalWe implement Nested VM and proxies on normal
Debian GNU Linux x86.
– External VM: KVMExternal VM: KVM
• The VM is a unit of snapshot and is NOT required hardware
repeatability.
• KVM is fast with virtualization technology of X86.
– Internal VM: QEMU
h l h d f bili ( d b• The VM emulates hardware for repeatability (e.g., random number
generator).
– Proxies are written by PerlProxies are written by Perl.
• External Proxy: 430 LOC
• Internal Proxy: 132 LOC• Internal Proxy: 132 LOC

19. National Institute of Advanced Industrial Science and Technology
Performance evaluationPerformance evaluation
• We evaluated performance on Protocol Test Generator.
– Test for hand shake of TLS (PolarSSL) takes 9 snapshots and 2,311 rollbacks.
– Effect of GUI and noGUI.
– Effct of Memory size (large 1024MB/512MB small 512MB/256MB)– Effct of Memory size (large 1024MB/512MB, small 512MB/256MB)
GUI
Mem:
NoGUI
Mem:
NoGUI
Mem:
1024/512
(sec)
1024/512
(sec)
512/256
(sec)
Setting up nested VMs 266 107 93g p
Fuzz packet generator 1,307 1,164 1,080
Nested VMs
and Proxies
Snapshot
(9 times)
57
(Av 6.33)
37
(Av 4.11)
24
(Av 2.67)
R llb k 3 135 2 197 1 286 VM overhead isRollback
(2,311times)
3,135
(Av 1.36)
2,197
(Av 0.96)
1,286
(Av 0.56)
Other 12 13 12
Total 5 043 3 622 2 587
VM overhead is
almost same to
packet generator.
Total 5,043 3,622 2,587
ThinkPad T410 (CPU Intel Core i7-M620 2.67Ghz, Memory 4GB).

20. National Institute of Advanced Industrial Science and Technology
R lt f P t l F T tiResults of Protocol Fuzz Testing
W ifi d th h d h k f 4 TLS• We verified the handshake of 4 TLS servers.
– OpenSSL, GnuTLS, CyaSSL, and PolarSSL.
• The fuzz testing found
2 bugs in CyaSSL– 2 bugs in CyaSSL
– 1 bug in PolarSSL
• We reported the results to the mailing list, and the
bugs were fixed.g

21. National Institute of Advanced Industrial Science and Technology
R l t d kRelated works
• Nested VM• Nested VM
– Migration
• Xen-Blanket [EuroSys’12] enables user level migration on CloudXen Blanket [EuroSys 12] enables user level migration on Cloud.
– Security
• CloudVisor [SOSP’11] is inserted on existing VM on multi-tenant
Cloud.
• Process level snapshot
– DMTCP: Distributed Multi Threaded Check Pointing
• http://dmtcp.sourceforge.net/
If d ’t t d t b t h d l l t bilit f• If we don’t not need to care about hardware-level repeatability for
rollback, DMTCP will be a candidate for protocol test generator.

22. National Institute of Advanced Industrial Science and Technology
Conclusion
• We propose a combination of nested VM and proxies
for protocol fuzz testing environment.
– It treats hardware-level repeatability (e.g., random number).
– It is easy and flexible to implement.
• Our method is not limited to protocol testing. It can be
applied on other areas (e.g., I/O fuzz testing, memorypp ( g , g, y
fuzz testing).
Acknowledgement
This work is supported by the National Institute of Information and Communications Technology of Japan.