Functional and scale performance tests
using zopkio
{‘Event’: ‘PyCON HK 2015’,
‘Name’: ‘Marcelo Araujo’,
‘Email’: ‘marcelo@gandi.net’

Agenda
● About me.
● Why we test software?
● Zopkio test framework.
● Q&A.
PyCON HK 2015 Marcelo Araujo <marcelo@gandi.net>

About me
PyCON HK 2015 Marcelo Araujo <marcelo@gandi.net>
● Name: Marcelo Araujo.
● Bachelor in Computer System Networking and Telecommunications.
● Post-Degree in Quality on Software Engineering.
● Python consumer since 2007.
● FreeBSD Developer since 2007 (ports and *kernel).
● DevOps at Gandi since 2015.
From To

Why we test software?
PyCON HK 2015 Marcelo Araujo <marcelo@gandi.net>
● Assure what we created does what it supposed to do.
● The behavior with one user could be different with thousand users.
● Users could do something unexpected or not planned.
● Many different devices, browsers, operating systems and so forth.
● We want assure a good software quality.
● Validation: Are we doing the right job?
● Verification: Are we doing the job right?

A typical application infra.
PyCON HK 2015 Marcelo Araujo <marcelo@gandi.net>
App Server Database
Your application starts like this!

Then, becomes like this.
PyCON HK 2015 Marcelo Araujo <marcelo@gandi.net>
App Server Master DB
Slave DB
Cache
Storage
Balancer/Proxy
API Servers
Storage Replication
VMs
Services

A question!
PyCON HK 2015 Marcelo Araujo <marcelo@gandi.net>
As a DevOps, It is not only about test software anymore!
How can I test the infrastructure?

Zopkio test framework.
PyCON HK 2015 Marcelo Araujo <marcelo@gandi.net>
● Made by Linkedin.
● It is a test framework built to support at scale performance and functional
testing.
● It can be installed via pip.
● Latest code at: https://github.com/linkedin/Zopkio
● There is documentation.
● Enough pydoc for every each class.
● They have some examples of code.
● Under Apache 2.0 License.
● They are open for pull requests.
● Active development.

Zopkio test framework.
PyCON HK 2015 Marcelo Araujo <marcelo@gandi.net>
● Zopkio provides the ability to write tests that combine performance and
functional testing across a distributed system.
● Writing tests using Zopkio should be nearly as simple as writing tests in xUnit
or Nose.
● Zopkio strongly depends on:
○ Naarad: A system analysis tool that parses and plots time series data.
○ Paramiko: Python implementation of SSHv2 protocol.

Zopkio test framework.
PyCON HK 2015 Marcelo Araujo <marcelo@gandi.net>
● What kind of tests can we perform with zopkio?
○ Functional.
■ Ensure every function produces its expected outcome.
○ Load.
■ Understand the behaviour under a specific and expected load.
○ Stress.
■ What is the behaviour beyond the normal expected load.
○ Performance.
■ Determine the speed or effectiveness of an software or service.

Zopkio test framework.
PyCON HK 2015 Marcelo Araujo <marcelo@gandi.net>
● What we need from a functional and performance test framework?
○ ASSERT: Must be able to test a condition!
○ Get data vs time and compare with what we expected.
○ Plot graphs!
○ Provisioning before run any test.
○ Parallelize tests.
○ Control the sequence of tests.
○ Collect logs.
○ Parse logs.
○ Cleanup environment after tests.
○ An interface with test result.
○ Be able to debug! Our test is a software too.

My PYHK test lab.
PyCON HK 2015 Marcelo Araujo <marcelo@gandi.net>
bare-metal machine
FreeBSD OS
Hypervisor Type 2 (bhyve)
VM1 (BSD) VM2 (BSD) VM3 (Linux) VM4 (BSD)
Zopkio
Switch
Server

What tests I will perform in this lab?
PyCON HK 2015 Marcelo Araujo <marcelo@gandi.net>
● VM1 and VM2: Check how many files inside /etc/ and apply an ASSERT.
● VM1 and VM2: ICMP to example.com and check the lantency.
● VM3 and VM4: TCP Server/Client and make a connection between them.

The basic of Zopkio.
PyCON HK 2015 Marcelo Araujo <marcelo@gandi.net>
.
|-- deployment.py
|-- naarad.cfg
|-- perf.py
|-- run.py
|-- test_suites
| |-- connection.py
| |-- machine1_etc.py
| |-- machine1_ping.py
| |-- machine2_etc.py
| `-- machine2_ping.py
|-- configs
| `-- pyhk.json
|-- scripts
| |-- ping-csv.sh
| |-- run.sh
| |-- server.py
| |-- client.py
| `-- scripts.tar
● A test suite consists of:
○ A deployment file.
○ A dynamic configuration file.
○ One or more test files.
○ A config directory.

The Zopkio framework.
PyCON HK 2015 Marcelo Araujo <marcelo@gandi.net>
● Zopkio has a main script that takes several optional arguments:
○ Run only named tests to help debug broken tests.
○ Configs overrides at execution time.
○ Log level.
○ Console log level.
● To run my test suite:
araujo@coxinha:/z/hk# zopkio run.py --nopassword --log-level DEBUG

Main run.
PyCON HK 2015 Marcelo Araujo <marcelo@gandi.net>
1 +-- 2 lines: import os------------------------------------------------------------------
3 test = {
4 "deployment_code": os.path.join(
5 os.path.dirname(
6 os.path.abspath(__file__)), "deployment.py"),
7 "test_code": [
8 os.path.join(
9 os.path.dirname(
10 os.path.abspath(__file__)), "test_suites/machine1_etc.py"),
11 os.path.join(
12 os.path.dirname(
13 os.path.abspath(__file__)), "test_suites/machine1_ping.py"),
14 os.path.join(
15 os.path.dirname(
16 os.path.abspath(__file__)), "test_suites/connection.py"),
17 os.path.join(
18 os.path.dirname(
19 os.path.abspath(__file__)), "test_suites/machine2_ping.py"),
20 os.path.join(
21 os.path.dirname(
22 os.path.abspath(__file__)), "test_suites/machine2_etc.py")],
.
|-- run.py
23 "perf_code": os.path.join(
24 os.path.dirname(os.path.abspath(__file__)), "perf.py"),
25 "configs_directory": os.path.join(
26 os.path.dirname(os.path.abspath(__file__)), "configs/")
27 }

The config file.
PyCON HK 2015 Marcelo Araujo <marcelo@gandi.net>
1 {
2 "comment": "This is the first test",
3 "tip": "How many files on my /etc",
4 "pyhk_install": "/tmp/pyhk/",
5 "pyhk_exec": "scripts/scripts.tar",
6 "pyhk_cmd": "sh run.sh",
7 "ping_cmd": "bash ping-csv.sh --add-timestamp example.com >/tmp/pyhk/ping-output.csv",
8 "tcp_server_cmd": "python server.py &",
9 "tcp_client_cmd": "python client.py >connection.log",
10 "show_all_interations":true,
11 "verify_after_each_test":true,
12 "pyhk_hostname1": "10.0.1.22",
13 "pyhk_hostname2": "10.0.1.12"
14 }
|-- configs
| `-- pyhk.json

The dynamic config.
PyCON HK 2015 Marcelo Araujo <marcelo@gandi.net>
1 +-- 4 lines: !/usr/bin/env python-----------------------------------------------
5 LOGS_DIRECTORY = "/tmp/pyhk/collected_logs/"
6 OUTPUT_DIRECTORY = "/tmp/pyhk/results/"
7 +-- 2 lines: --------------------------------------------------------------------------
9 def machine_logs():
10 return {
11 "client1": [os.path.join("/tmp/pyhk/", "connection.log")],
12 "machine1": [os.path.join("/tmp/pyhk/", "run.log")],
13 "machine2": [os.path.join("/tmp/pyhk/", "run.log")],
14 }
15 +-- 2 lines: --------------------------------------------------------------------------
17 def naarad_logs():
18 return {
19 "machine1": [os.path.join("/tmp/pyhk/", "run.csv")],
20 "machine2": [os.path.join("/tmp/pyhk/", "run.csv")],
21 "machine1": [os.path.join("/tmp/pyhk/", "ping-output.csv")],
22 "machine2": [os.path.join("/tmp/pyhk/", "ping-output.csv")],
23 }
24 +-- 2 lines: -----------------------------------------------------------------------------
26 def naarad_config():
27 return os.path.join(
28 os.path.dirname(os.path.abspath(__file__)), "naarad.cfg")
.
|-- perf.py

The deployment file.
PyCON HK 2015 Marcelo Araujo <marcelo@gandi.net>
● It is in charge of provisioning machines.
● A deployment file can contain four functions:
○ setup_suite() - Will run before any of tests.
○ setup() - Will run before each test.
○ teardown() - Will run if setup() ran successfully.
○ teardown_suite() - Will run if setup_suite() ran successfully.

The deployer.
PyCON HK 2015 Marcelo Araujo <marcelo@gandi.net>
1 +-- 3 lines: !/usr/bin/env python-------------------------------------------------------------------------------
4 import zopkio.adhoc_deployer as adhoc_deployer
5 import zopkio.runtime as runtime
6 +-- 6 lines: pyhk_deployer = None--------------------------------------------------------------------------
12 def setup_suite():
13 +-- 2 lines: print "==> Starting tests for PYHK."---------------------------------------------------------
15 runtime.set_user('root', '')
16 +-- 4 lines: global pyhk_deployer---------------------------------------------------------------------------
20 tcp_server = adhoc_deployer.SSHDeployer(
21 "server",
22 {'executable': runtime.get_active_config('pyhk_exec'),
23 'extract': True,
24 'start_command': runtime.get_active_config('tcp_server_cmd'),
25 'stop_command': "ps ax | grep '[p]ython server' | awk '{print $1}' | xargs kill -9"})
26 runtime.set_deployer("server", tcp_server)
27
28 tcp_server.install("server1",
29 {"hostname": "10.0.1.23",
30 "install_path": runtime.get_active_config('pyhk_install')})
31
32 tcp_client = adhoc_deployer.SSHDeployer(
33 "client",
34 {'executable': runtime.get_active_config('pyhk_exec'),
35 'extract': True,
36 'start_command': runtime.get_active_config('tcp_client_cmd')})
37 runtime.set_deployer("client", tcp_client)
38
39 tcp_client.install("client1",
40 {"hostname": "10.0.1.24",
41 "install_path": runtime.get_active_config('pyhk_install')})
42 +-- 20 lines: pyhk_deployer = adhoc_deployer.SSHDeployer(--------
62 def setup():
63 for process in tcp_server.get_processes():
64 tcp_server.start(process.unique_id)
65
66 def teardown_suite():
67 for process in tcp_server.get_processes():
68 tcp_server.undeploy(process.unique_id)
.
|-- deployment.py

An test file.
PyCON HK 2015 Marcelo Araujo <marcelo@gandi.net>
1 +-- 6 lines: !/usr/bin/env python-----------------------------------------------
7 import zopkio.runtime as runtime
8 import zopkio.test_utils as testutilities
9
10 test_phase = 3
11 +-- 2 lines: ------------------------------------------------------------------------
13 def test_ping_host1():
14 print "==> ping example.com (machine1)"
15 pyhk_deployer = runtime.get_deployer("pyhk")
16
17 pyhk_deployer.start(
18 "machine1",
19 configs={
20 "start_command": runtime.get_active_config('ping_cmd'),
21 "sync": True
22 })
23 +-- 2 lines: ------------------------------------------------------------------------
25 def validate_ping_host1():
26 '''
27 Send 10 icmp to example.com .
28 '''
29 hostname1_log_file = os.path.join(
30 perf.LOGS_DIRECTORY, "machine1-ping-output.csv")
31 hostname1_logs = testutilities.get_log_for_test(
32 "test_ping_host1", hostname1_log_file, "12:00:00")
33
34 # Number of icmp packages
35 size_p = len(hostname1_logs.split(',')) / 2
36
37 assert size_p == 10, "Less than 10 replies on host1"
|-- test_suites
| |-- machine1_ping.py

Plot graphs.
PyCON HK 2015 Marcelo Araujo <marcelo@gandi.net>
1 [machine1-icmp]
2 infile=/tmp/pyhk/collected_logs/machine1-ping-output.csv
3 columns=sequence latency
4 sep=,
5 qps.sla=mean<500
6 latency.sla=mean<400 p50<300
7
8 [machine2-icmp]
9 infile=/tmp/pyhk/collected_logs/machine2-ping-output.csv
10 columns=sequence latency
11 sep=,
12
13 [GRAPH]
14 graphs=machine1-icmp.sequence,machine1-icmp.latency machine2-icmp.sequence,machine2-icmp.latency
.
|-- naarad.cfg

The Zopkio GUI.
PyCON HK 2015 Marcelo Araujo <marcelo@gandi.net>
Let's take a look on my machine

The conclusion is:
PyCON HK 2015 Marcelo Araujo <marcelo@gandi.net>
We need to be able to test the infrastructure where I
will run my application.

References.
PyCON HK 2015 Marcelo Araujo <marcelo@gandi.net>
● Zopkio:
○ https://github.com/linkedin/Zopkio
● Naarad:
○ https://github.com/linkedin/naarad/
● Code for PYHK Test Lab:
○ https://github.com/araujobsd/pyhk2015
● Slides at:
○ TBD

PyCON HK 2015 Marcelo Araujo <marcelo@gandi.net>
Thank you!
marcelo@gandi.net
araujo@FreeBSD.org