Blockchain testing

1. Testing distributed decentralized systems
by Antoine Toulme, CTO @ Whiteblock

3. New requirements of distributed, decentralized systems
• Pure peer-to-peer networks
• Consensus among all nodes is critical
• Eventual, asynchronous finality
• The chain can reorganize!
• Proof of work
• Proof of stake
• Applications deployed on those systems expose their own performance bottlenecks
• Oracles
• Bridges
• New attack vectors…

4. New metrics to capture
• Transaction size
• Sender address
• Receiving address
• Sent and received timestamps
• Block information: number, hash, size
• Transaction mempool size
• Block time
• Block size
• Block mempool size
• Block transmission time (Difference of send and receive time)
• Block header validation time
• Block processing time
• Total delay
• Uncle Rate

5. Collecting metrics
• Logs
• Block data
• Prometheus
• Use data science tools like Jupyter, Grafana, the ELK stack to process.
• Each run and client implementation is different

6. Test suites
• Control group
• Network testing
• Data testing
• Compatibility testing
• Security testing
• Load testing
• Consensus testing
• More!

7. Network testing

8. Control group
• The control test will involve a test case where latency, packet loss, and bandwidth
will be configured to 0ms, 0%, and 1000 MB*, respectively. The test will run 50
transactions per second to the network in order to determine the transactional
throughput of the network. Upon completion of these tests, Whiteblock will export
the raw block data of all nodes. The results will be used as a reference point for
future tests.
• Optimal, peak performance
• “Works on my machine feel”

9. Latency testing
• The latency test will involve 3 test cases where latency will be configured and varied
from 0ms, 50ms, and 100ms. The test will run 50 transactions per second to the
network in order to determine the transactional throughput of the network. Upon
completion of these tests, Whiteblock will export the raw block data of all nodes.
• We usually see some performance degradation.
• Good benchmark as a warm up.
• Note nodes peer randomly throughout the network - this means a node in New York
may peer with a node in Tokyo.

10. Bandwidth testing
• The bandwidth test will involve 3 test cases where the bandwidth between nodes
will be configured and varied from 10 MB, 50 MB, and 100 MB. The test will run 50
transactions per second to the network in order to determine the transactional
throughput of the network. Upon completion of these tests, Whiteblock will export
the raw block data.
• We notice some blockchains are bandwidth heavy and are quite affected by this
testing.

11. Packet loss testing
• The packet loss test will involve 3 test cases where the packet loss between nodes
will be configured and varied from 0.01%, 0.1%, and 1%. The test will run 50
transactions per second to the network in order to determine the transactional
throughput of the network. Upon completion of these tests, Whiteblock will export
the raw block data of all nodes.
• This exercises the capability of the blockchain to sustain and recover incomplete
messages. This causes important issues, having to replay and request data again.
• In some scenarios, this may create situations where nodes drop peers.

12. Network stress testing
• The stress test will involve 3 test cases where latency, packet loss, and bandwidth
between nodes will be configured to 150ms, 0.01%, and 10MB respectively. The
test will run 50 transactions per second to the network in order to determine the
transactional throughput of the network. Upon completion of these tests, Whiteblock
will export the raw block data of all nodes of all nodes.
• Goal is to create something that looks like the real Internet.

13. Network topology testing
• The network topology test will involve 3 test cases where the network topologies for
all nodes will be configured in a custom manner. The cases will be a linear peering
topology, static peering to X number of nodes, and a full mesh peering
topology. The test will run 50 transactions per second to the network in order to
determine the transactional throughput of the network. Upon completion of these
tests, Whiteblock will export the raw block data.
• The block propagation time is the metric to watch here.

14. Discovery topology testing
• The network topology test will involve 3 test cases where the network topologies for
all nodes will be configured using a discovery mechanism, against a boot node. The
cases will involve 10, 20 and 30 nodes. In each case, the network should be
configured to allow a maximum number of peers of 5, 10 and 20. The test will run
50 transactions per second to the network in order to determine the transactional
throughput of the network. Upon completion of these tests, Whiteblock will export
the raw block data.
• Can the network self-organize in an optimized manner?

15. Data testing

16. Sync testing
• Whiteblock will set up a regular testnet, and identify a node which it will cut from the
rest of the network for a given duration. Whiteblock will monitor usage and functional
resources of the node while it is cut from the network. Upon reestablishing
connections, Whiteblock will check the node is able to reconnect and synchronize
with the rest of the network.
• How fast is sync?

17. Title

18. Smart contract testing
• Whiteblock will set up a regular testnet, and deploy the smart contract to it.
• An external dedicated helper will then exercise the contract by calling its endpoints
and ensuring the results of the interactions matches the expected behavior.
• Whiteblock will output the result of the testing as the state of the application after the
script has executed.
• Exercises the smart contract.
• Possibly exercises the infrastructure using the smart contract.

19. Compatibility testing
• Whiteblock will set up nodes of different implementations implementing the same
protocol.
• Whiteblock will deploy smart contracts and inject the same transactions across all
nodes.
• It will then compare the output of the result and ensure they all reach the same
output. It will also benchmark the time to process the transactions to check the
performance of the implementations.

20. Compatibility testing

21. Security testing

22. Eclipse attack testing
• The eclipse attack test will be the overtaking of a node’s peerlist with bad acting
nodes. This will be done using a static peers list to simulate the connecting to
dishonest nodes. The bad acting nodes will be on a fork of the network and will
propagate blocks that are false (different than those of the canonical chain). Upon
completion of these tests, Whiteblock will export the raw block data.
• How do you counter eclipse attacks?

23. Double spend testing
• Whiteblock will set up a regular testnet, and create two competing transactions so
that only one transaction may be successful at a time. Whiteblock will then inject
both transactions at the same time to two different nodes of the network, and check
the target account is only being credited by one transaction. One of the two
transactions should be rejected.
• The double spend attack is a classic. PoW is meant to ensure these attacks are
defeated.
• … but not all blockchains use proof of work :)

24. 51% attack testing
• Whiteblock will set up a testnet made of normal nodes and modified nodes that have
been instructed to always follow the chain built on top of a block with a specific
transaction hash.
• The network is exercised with 50 transactions per second, until the attacking
transaction is injected. Ideally, the consensus algorithm is able to withstand this
attack and reject the transaction.
• Ethereum and Bitcoin are vulnerable to 51% attacks.

25. Load testing

26. Local load testing
• Whiteblock will set up a regular testnet, and select a node to be the recipient of the
load.
• Whiteblock will inject continuously over a thousand transactions per second to the
node for a configurable period of times, in minutes. During the test, Whiteblock will
monitor resources usage on the node and check the transaction success rate over
the network.
• What will make a node fall over?

27. Distributed load testing
• Whiteblock will set up a regular testnet.
• Whiteblock will inject continuously over a thousand transactions per second to all
nodes for a configurable period of time, in minutes. During the test, Whiteblock will
monitor resources usage on nodes and check the transaction success rate over the
network.
• Good measure of stress of the network under heavy load

28. Consensus testing

29. Block time testing
• In blockchains, block times may vary depending on implementation. The block time
can affect the way a network behaves and a shorter block time will usually lead to a
higher orphan (uncle) rate in a proof-of-work network. This is important as a higher
orphan (uncle) rate equates to a lower level of security in the network until those
orphans (uncles) are pruned and the nodes are synced up to the latest canonical
block. This test will focus on measuring block times in the network and upon
completion of this test, Whiteblock will export the raw block data.

30. State transition testing
• Whiteblock will initiate one node with an existing, static state.
• It will create pre-determined conditions by which the state of the application will
mutate, such as a state transition or a new block.
• Whiteblock will output the state in a machine-readable format and compare it to an
expected output.
• Great to debug and check compatibility between clients of a new chain.

31. Finality testing
• Some blockchains support eventual finality. New blockchains based on proof of
stake support strong, deterministic finality. Whiteblock will set up a regular testnet
and observe block finalization. It will compare the times at which blocks are finalized
between clients. When applicable, Whiteblock will create situations where
finalization is difficult based on the network topology or the transaction load, and
check finalization still operates uniformly across clients.

32. MOAR testing
• Custom testing cases, such as:
• Comparisons between blockchains
• Network bad actors
• Rendez-vous nodes
• TCP hole punching tests
• UDP-based traffic
• Tor and VPN traffic
• …

33. Thank you!