Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Functional and scale performance tests
using zopkio
{‘Event’: ‘PyCON HK 2015’,
‘Name’: ‘Marcelo Araujo’,
‘Email’: ‘marcelo...
Agenda
● About me.
● Why we test software?
● Zopkio test framework.
● Q&A.
PyCON HK 2015 Marcelo Araujo <marcelo@gandi.net>
About me
PyCON HK 2015 Marcelo Araujo <marcelo@gandi.net>
● Name: Marcelo Araujo.
● Bachelor in Computer System Networking...
Why we test software?
PyCON HK 2015 Marcelo Araujo <marcelo@gandi.net>
● Assure what we created does what it supposed to d...
A typical application infra.
PyCON HK 2015 Marcelo Araujo <marcelo@gandi.net>
App Server Database
Your application starts ...
Then, becomes like this.
PyCON HK 2015 Marcelo Araujo <marcelo@gandi.net>
App Server Master DB
Slave DB
Cache
Storage
Bala...
A question!
PyCON HK 2015 Marcelo Araujo <marcelo@gandi.net>
As a DevOps, It is not only about test software anymore!
How ...
Zopkio test framework.
PyCON HK 2015 Marcelo Araujo <marcelo@gandi.net>
● Made by Linkedin.
● It is a test framework built...
Zopkio test framework.
PyCON HK 2015 Marcelo Araujo <marcelo@gandi.net>
● Zopkio provides the ability to write tests that ...
Zopkio test framework.
PyCON HK 2015 Marcelo Araujo <marcelo@gandi.net>
● What kind of tests can we perform with zopkio?
○...
Zopkio test framework.
PyCON HK 2015 Marcelo Araujo <marcelo@gandi.net>
● What we need from a functional and performance t...
My PYHK test lab.
PyCON HK 2015 Marcelo Araujo <marcelo@gandi.net>
bare-metal machine
FreeBSD OS
Hypervisor Type 2 (bhyve)...
What tests I will perform in this lab?
PyCON HK 2015 Marcelo Araujo <marcelo@gandi.net>
● VM1 and VM2: Check how many file...
The basic of Zopkio.
PyCON HK 2015 Marcelo Araujo <marcelo@gandi.net>
.
|-- deployment.py
|-- naarad.cfg
|-- perf.py
|-- r...
The Zopkio framework.
PyCON HK 2015 Marcelo Araujo <marcelo@gandi.net>
● Zopkio has a main script that takes several optio...
Main run.
PyCON HK 2015 Marcelo Araujo <marcelo@gandi.net>
1 +-- 2 lines: import os---------------------------------------...
The config file.
PyCON HK 2015 Marcelo Araujo <marcelo@gandi.net>
1 {
2 "comment": "This is the first test",
3 "tip": "How...
The dynamic config.
PyCON HK 2015 Marcelo Araujo <marcelo@gandi.net>
1 +-- 4 lines: !/usr/bin/env python------------------...
The deployment file.
PyCON HK 2015 Marcelo Araujo <marcelo@gandi.net>
● It is in charge of provisioning machines.
● A depl...
The deployer.
PyCON HK 2015 Marcelo Araujo <marcelo@gandi.net>
1 +-- 3 lines: !/usr/bin/env python------------------------...
An test file.
PyCON HK 2015 Marcelo Araujo <marcelo@gandi.net>
1 +-- 6 lines: !/usr/bin/env python------------------------...
Plot graphs.
PyCON HK 2015 Marcelo Araujo <marcelo@gandi.net>
1 [machine1-icmp]
2 infile=/tmp/pyhk/collected_logs/machine1...
The Zopkio GUI.
PyCON HK 2015 Marcelo Araujo <marcelo@gandi.net>
Let's take a look on my machine
The conclusion is:
PyCON HK 2015 Marcelo Araujo <marcelo@gandi.net>
We need to be able to test the infrastructure where I
...
References.
PyCON HK 2015 Marcelo Araujo <marcelo@gandi.net>
● Zopkio:
○ https://github.com/linkedin/Zopkio
● Naarad:
○ ht...
PyCON HK 2015 Marcelo Araujo <marcelo@gandi.net>
Thank you!
marcelo@gandi.net
araujo@FreeBSD.org
Upcoming SlideShare
Loading in …5
×

Functional and scale performance tests using zopkio

959 views

Published on

PyCon Hong Kong 2015.

Published in: Internet
  • Be the first to comment

Functional and scale performance tests using zopkio

  1. 1. Functional and scale performance tests using zopkio {‘Event’: ‘PyCON HK 2015’, ‘Name’: ‘Marcelo Araujo’, ‘Email’: ‘marcelo@gandi.net’
  2. 2. Agenda ● About me. ● Why we test software? ● Zopkio test framework. ● Q&A. PyCON HK 2015 Marcelo Araujo <marcelo@gandi.net>
  3. 3. About me PyCON HK 2015 Marcelo Araujo <marcelo@gandi.net> ● Name: Marcelo Araujo. ● Bachelor in Computer System Networking and Telecommunications. ● Post-Degree in Quality on Software Engineering. ● Python consumer since 2007. ● FreeBSD Developer since 2007 (ports and *kernel). ● DevOps at Gandi since 2015. From To
  4. 4. Why we test software? PyCON HK 2015 Marcelo Araujo <marcelo@gandi.net> ● Assure what we created does what it supposed to do. ● The behavior with one user could be different with thousand users. ● Users could do something unexpected or not planned. ● Many different devices, browsers, operating systems and so forth. ● We want assure a good software quality. ● Validation: Are we doing the right job? ● Verification: Are we doing the job right?
  5. 5. A typical application infra. PyCON HK 2015 Marcelo Araujo <marcelo@gandi.net> App Server Database Your application starts like this!
  6. 6. Then, becomes like this. PyCON HK 2015 Marcelo Araujo <marcelo@gandi.net> App Server Master DB Slave DB Cache Storage Balancer/Proxy API Servers Storage Replication VMs Services
  7. 7. A question! PyCON HK 2015 Marcelo Araujo <marcelo@gandi.net> As a DevOps, It is not only about test software anymore! How can I test the infrastructure?
  8. 8. Zopkio test framework. PyCON HK 2015 Marcelo Araujo <marcelo@gandi.net> ● Made by Linkedin. ● It is a test framework built to support at scale performance and functional testing. ● It can be installed via pip. ● Latest code at: https://github.com/linkedin/Zopkio ● There is documentation. ● Enough pydoc for every each class. ● They have some examples of code. ● Under Apache 2.0 License. ● They are open for pull requests. ● Active development.
  9. 9. Zopkio test framework. PyCON HK 2015 Marcelo Araujo <marcelo@gandi.net> ● Zopkio provides the ability to write tests that combine performance and functional testing across a distributed system. ● Writing tests using Zopkio should be nearly as simple as writing tests in xUnit or Nose. ● Zopkio strongly depends on: ○ Naarad: A system analysis tool that parses and plots time series data. ○ Paramiko: Python implementation of SSHv2 protocol.
  10. 10. Zopkio test framework. PyCON HK 2015 Marcelo Araujo <marcelo@gandi.net> ● What kind of tests can we perform with zopkio? ○ Functional. ■ Ensure every function produces its expected outcome. ○ Load. ■ Understand the behaviour under a specific and expected load. ○ Stress. ■ What is the behaviour beyond the normal expected load. ○ Performance. ■ Determine the speed or effectiveness of an software or service.
  11. 11. Zopkio test framework. PyCON HK 2015 Marcelo Araujo <marcelo@gandi.net> ● What we need from a functional and performance test framework? ○ ASSERT: Must be able to test a condition! ○ Get data vs time and compare with what we expected. ○ Plot graphs! ○ Provisioning before run any test. ○ Parallelize tests. ○ Control the sequence of tests. ○ Collect logs. ○ Parse logs. ○ Cleanup environment after tests. ○ An interface with test result. ○ Be able to debug! Our test is a software too.
  12. 12. My PYHK test lab. PyCON HK 2015 Marcelo Araujo <marcelo@gandi.net> bare-metal machine FreeBSD OS Hypervisor Type 2 (bhyve) VM1 (BSD) VM2 (BSD) VM3 (Linux) VM4 (BSD) Zopkio Switch Server
  13. 13. What tests I will perform in this lab? PyCON HK 2015 Marcelo Araujo <marcelo@gandi.net> ● VM1 and VM2: Check how many files inside /etc/ and apply an ASSERT. ● VM1 and VM2: ICMP to example.com and check the lantency. ● VM3 and VM4: TCP Server/Client and make a connection between them.
  14. 14. The basic of Zopkio. PyCON HK 2015 Marcelo Araujo <marcelo@gandi.net> . |-- deployment.py |-- naarad.cfg |-- perf.py |-- run.py |-- test_suites | |-- connection.py | |-- machine1_etc.py | |-- machine1_ping.py | |-- machine2_etc.py | `-- machine2_ping.py |-- configs | `-- pyhk.json |-- scripts | |-- ping-csv.sh | |-- run.sh | |-- server.py | |-- client.py | `-- scripts.tar ● A test suite consists of: ○ A deployment file. ○ A dynamic configuration file. ○ One or more test files. ○ A config directory.
  15. 15. The Zopkio framework. PyCON HK 2015 Marcelo Araujo <marcelo@gandi.net> ● Zopkio has a main script that takes several optional arguments: ○ Run only named tests to help debug broken tests. ○ Configs overrides at execution time. ○ Log level. ○ Console log level. ● To run my test suite: araujo@coxinha:/z/hk# zopkio run.py --nopassword --log-level DEBUG
  16. 16. Main run. PyCON HK 2015 Marcelo Araujo <marcelo@gandi.net> 1 +-- 2 lines: import os------------------------------------------------------------------ 3 test = { 4 "deployment_code": os.path.join( 5 os.path.dirname( 6 os.path.abspath(__file__)), "deployment.py"), 7 "test_code": [ 8 os.path.join( 9 os.path.dirname( 10 os.path.abspath(__file__)), "test_suites/machine1_etc.py"), 11 os.path.join( 12 os.path.dirname( 13 os.path.abspath(__file__)), "test_suites/machine1_ping.py"), 14 os.path.join( 15 os.path.dirname( 16 os.path.abspath(__file__)), "test_suites/connection.py"), 17 os.path.join( 18 os.path.dirname( 19 os.path.abspath(__file__)), "test_suites/machine2_ping.py"), 20 os.path.join( 21 os.path.dirname( 22 os.path.abspath(__file__)), "test_suites/machine2_etc.py")], . |-- run.py 23 "perf_code": os.path.join( 24 os.path.dirname(os.path.abspath(__file__)), "perf.py"), 25 "configs_directory": os.path.join( 26 os.path.dirname(os.path.abspath(__file__)), "configs/") 27 }
  17. 17. The config file. PyCON HK 2015 Marcelo Araujo <marcelo@gandi.net> 1 { 2 "comment": "This is the first test", 3 "tip": "How many files on my /etc", 4 "pyhk_install": "/tmp/pyhk/", 5 "pyhk_exec": "scripts/scripts.tar", 6 "pyhk_cmd": "sh run.sh", 7 "ping_cmd": "bash ping-csv.sh --add-timestamp example.com >/tmp/pyhk/ping-output.csv", 8 "tcp_server_cmd": "python server.py &", 9 "tcp_client_cmd": "python client.py >connection.log", 10 "show_all_interations":true, 11 "verify_after_each_test":true, 12 "pyhk_hostname1": "10.0.1.22", 13 "pyhk_hostname2": "10.0.1.12" 14 } |-- configs | `-- pyhk.json
  18. 18. The dynamic config. PyCON HK 2015 Marcelo Araujo <marcelo@gandi.net> 1 +-- 4 lines: !/usr/bin/env python----------------------------------------------- 5 LOGS_DIRECTORY = "/tmp/pyhk/collected_logs/" 6 OUTPUT_DIRECTORY = "/tmp/pyhk/results/" 7 +-- 2 lines: -------------------------------------------------------------------------- 9 def machine_logs(): 10 return { 11 "client1": [os.path.join("/tmp/pyhk/", "connection.log")], 12 "machine1": [os.path.join("/tmp/pyhk/", "run.log")], 13 "machine2": [os.path.join("/tmp/pyhk/", "run.log")], 14 } 15 +-- 2 lines: -------------------------------------------------------------------------- 17 def naarad_logs(): 18 return { 19 "machine1": [os.path.join("/tmp/pyhk/", "run.csv")], 20 "machine2": [os.path.join("/tmp/pyhk/", "run.csv")], 21 "machine1": [os.path.join("/tmp/pyhk/", "ping-output.csv")], 22 "machine2": [os.path.join("/tmp/pyhk/", "ping-output.csv")], 23 } 24 +-- 2 lines: ----------------------------------------------------------------------------- 26 def naarad_config(): 27 return os.path.join( 28 os.path.dirname(os.path.abspath(__file__)), "naarad.cfg") . |-- perf.py
  19. 19. The deployment file. PyCON HK 2015 Marcelo Araujo <marcelo@gandi.net> ● It is in charge of provisioning machines. ● A deployment file can contain four functions: ○ setup_suite() - Will run before any of tests. ○ setup() - Will run before each test. ○ teardown() - Will run if setup() ran successfully. ○ teardown_suite() - Will run if setup_suite() ran successfully.
  20. 20. The deployer. PyCON HK 2015 Marcelo Araujo <marcelo@gandi.net> 1 +-- 3 lines: !/usr/bin/env python------------------------------------------------------------------------------- 4 import zopkio.adhoc_deployer as adhoc_deployer 5 import zopkio.runtime as runtime 6 +-- 6 lines: pyhk_deployer = None-------------------------------------------------------------------------- 12 def setup_suite(): 13 +-- 2 lines: print "==> Starting tests for PYHK."--------------------------------------------------------- 15 runtime.set_user('root', '') 16 +-- 4 lines: global pyhk_deployer--------------------------------------------------------------------------- 20 tcp_server = adhoc_deployer.SSHDeployer( 21 "server", 22 {'executable': runtime.get_active_config('pyhk_exec'), 23 'extract': True, 24 'start_command': runtime.get_active_config('tcp_server_cmd'), 25 'stop_command': "ps ax | grep '[p]ython server' | awk '{print $1}' | xargs kill -9"}) 26 runtime.set_deployer("server", tcp_server) 27 28 tcp_server.install("server1", 29 {"hostname": "10.0.1.23", 30 "install_path": runtime.get_active_config('pyhk_install')}) 31 32 tcp_client = adhoc_deployer.SSHDeployer( 33 "client", 34 {'executable': runtime.get_active_config('pyhk_exec'), 35 'extract': True, 36 'start_command': runtime.get_active_config('tcp_client_cmd')}) 37 runtime.set_deployer("client", tcp_client) 38 39 tcp_client.install("client1", 40 {"hostname": "10.0.1.24", 41 "install_path": runtime.get_active_config('pyhk_install')}) 42 +-- 20 lines: pyhk_deployer = adhoc_deployer.SSHDeployer(-------- 62 def setup(): 63 for process in tcp_server.get_processes(): 64 tcp_server.start(process.unique_id) 65 66 def teardown_suite(): 67 for process in tcp_server.get_processes(): 68 tcp_server.undeploy(process.unique_id) . |-- deployment.py
  21. 21. An test file. PyCON HK 2015 Marcelo Araujo <marcelo@gandi.net> 1 +-- 6 lines: !/usr/bin/env python----------------------------------------------- 7 import zopkio.runtime as runtime 8 import zopkio.test_utils as testutilities 9 10 test_phase = 3 11 +-- 2 lines: ------------------------------------------------------------------------ 13 def test_ping_host1(): 14 print "==> ping example.com (machine1)" 15 pyhk_deployer = runtime.get_deployer("pyhk") 16 17 pyhk_deployer.start( 18 "machine1", 19 configs={ 20 "start_command": runtime.get_active_config('ping_cmd'), 21 "sync": True 22 }) 23 +-- 2 lines: ------------------------------------------------------------------------ 25 def validate_ping_host1(): 26 ''' 27 Send 10 icmp to example.com . 28 ''' 29 hostname1_log_file = os.path.join( 30 perf.LOGS_DIRECTORY, "machine1-ping-output.csv") 31 hostname1_logs = testutilities.get_log_for_test( 32 "test_ping_host1", hostname1_log_file, "12:00:00") 33 34 # Number of icmp packages 35 size_p = len(hostname1_logs.split(',')) / 2 36 37 assert size_p == 10, "Less than 10 replies on host1" |-- test_suites | |-- machine1_ping.py
  22. 22. Plot graphs. PyCON HK 2015 Marcelo Araujo <marcelo@gandi.net> 1 [machine1-icmp] 2 infile=/tmp/pyhk/collected_logs/machine1-ping-output.csv 3 columns=sequence latency 4 sep=, 5 qps.sla=mean<500 6 latency.sla=mean<400 p50<300 7 8 [machine2-icmp] 9 infile=/tmp/pyhk/collected_logs/machine2-ping-output.csv 10 columns=sequence latency 11 sep=, 12 13 [GRAPH] 14 graphs=machine1-icmp.sequence,machine1-icmp.latency machine2-icmp.sequence,machine2-icmp.latency . |-- naarad.cfg
  23. 23. The Zopkio GUI. PyCON HK 2015 Marcelo Araujo <marcelo@gandi.net> Let's take a look on my machine
  24. 24. The conclusion is: PyCON HK 2015 Marcelo Araujo <marcelo@gandi.net> We need to be able to test the infrastructure where I will run my application.
  25. 25. References. PyCON HK 2015 Marcelo Araujo <marcelo@gandi.net> ● Zopkio: ○ https://github.com/linkedin/Zopkio ● Naarad: ○ https://github.com/linkedin/naarad/ ● Code for PYHK Test Lab: ○ https://github.com/araujobsd/pyhk2015 ● Slides at: ○ TBD
  26. 26. PyCON HK 2015 Marcelo Araujo <marcelo@gandi.net> Thank you! marcelo@gandi.net araujo@FreeBSD.org

×