Microarmy - by J2 Labs


Published on

I describe a distributed load testing tool I built. It deploys some number of ec2 micros, configures them and then launches the load balancing tool in parallel against the website.


Published in: Technology
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Microarmy - by J2 Labs

  1. 1. Micro  Army:  Load  Tes1ng   By  James  Dennis  (@j2labs)   h?p://j2labs.net  
  2. 2. What  is  Micro  Army?  •  Website  Load  Tes1ng  with  EC2  micros   –  Micros  are  cheap   –  I spent $2 building the tool –  Brubeck  vs.  Tornado   •  this  is  basically  Eventlet  vs.  Tornado   –  Easily  scalable,     •  at  least  high  enough  to  slaughter  a  server  
  3. 3. Challenges  •  Arbitrary  number  of  boxes  should  be  possible   – I  call  them  “cannons”   – They  should  be  configurable  via  some  script.   •  With  a  liAle  more  work,  it  could  be  an  admin  tool  •  Parallel   – Deploying  the  cannons  must  be  parallel   •  I’m  impaEent   – The  cannons  must  fire  in  parallel  
  4. 4. Python  Modules  •  Boto:  AWS  for  Python   – Used  to  deploy  EC2  instances  •  Paramiko:  SSH   – Required  a  liAle  work   – Fairly  old   •  Not  sure  if  that’s  bad  though  •  Eventlet:  paralleliza1on  (and  other  goodness)   – GreenPiles  are  easy   – Free  nonblocking  I/O     •  Listen  up,  Tornado  users  
  5. 5. Using  It  •  Deploy   –   Hopefully  AWS  gives  us  our  boxes   – We’ll  need  a  list  of  hostnames   – SSH  to  each  host  and  setup  the  boxes   •  Upload  and  run  script   •  Must  be  parallel  or  large  numbers  hurts   – Prints  host  list   •  The  deploy  script  will  eval()  it  for  you!  
  6. 6. Using  It  •  Firing  the  cannons   – SSH  to  all  boxes  and  run  `siege`   •  siege c 200 t 10s `hostname` •  I usually test with c 500 •  Siege can handle GET and POST args (!) – Each  SSH  session  blocks  unEl  compleEon   – Eventlet  makes  this  easy  to  manage   – Quick  and  dirty  parsing  of  out  to  generate  a  CSV  
  7. 7. Show  Me  Code  
  8. 8. Micro  Army:  SeUngs   Easily  overrideable  with  local_seUngs.py  ### Create n instancesaws_access_key = ‘...aws_secret_key = ‘...ami_key = ami-ccf405a5’ # official ubuntu imageec2_ssh_username = ubuntu # ami specificsecurity_groups = [’MicroArmyGroup]key_pair_name = ’micro_army_testnum_cannons = 5placement = us-east-1a’instance_type = t1.micro’ec2_ssh_key = /some/path/ec2_testing.pem
  9. 9. Boto:  AWS     Find  an  OS  image  (official  Ubuntu)  import boto# First, get a connectionec2_conn = boto.connect_ec2(aws_access_key, aws_secret_key)# Get *all* images that match this unique AMI keyimages = ec2_conn.get_all_images(ami_key)image = images[0]
  10. 10. Boto:  AWS   Instan1ate  N  boxes  ### Create n instancesr = image.run(min_count=num_cannons, max_count=num_cannons, placement=placement, security_groups=security_groups, key_name=key_pair_name, instance_type=instance_type)
  11. 11. Boto:  AWS   Launching  is  a  li?le  verbose…  while True: # Give AWS some time to work time.sleep(5) # Aggregate our instance status info for i in r.instances: i.update() status = [i.state for i in r.instances] # Finish only if Amazon says all instances are up if status.count(running) == len(r.instances): print Done!’ # Return list of host names return [i.public_dns_name for i in r.instances]
  12. 12. Paramiko:  SSH   Everything  you  need:  to  use  build  an  SSH  abstracEon  jd@gibson : 14:42:44 : ~/Projects/microarmy/microarmy$ wc -l communications.py 61 communications.pyjd@gibson : 14:42:45 : ~/Projects/microarmy/microarmy$ grep "def" communications.pydef ssh_connect(host, port=22):def exec_command(transport, command, return_stderr=False):# And half an SFTP abstraction...def sftp_connect(transport):def put_file(transport, local_path, remote_path):def put_files(transport, paths):
  13. 13. Micro  Army:  build_cannon.sh   apt-­‐get  can  handle  everything   ### Lets get recent stuff apt-get update ### Install apt-get -y install python-dev build-essential autoconf automake libtool uuid-dev git-core mercurial python-pip siege
  14. 14. Paramiko:  SSH   Configure  each  cannon  with  a  script  def _setup_a_cannon(hostname): # Get a connection (paramiko calls it a transport) ssh_conn = ssh_connect(hostname) # copy script to cannon and make it executable script_path = env_scripts_dir + / + CANNON_INIT_SCRIPT put_file(ssh_conn, script_path, CANNON_INIT_SCRIPT) response = exec_command(ssh_conn, chmod 755 ~/%s % CANNON_INIT_SCRIPT) if response: # response would be error output return False # execute the setup script (expect this call to take a while) response = exec_command(ssh_conn, sudo ./%s % CANNON_INIT_SCRIPT) return (hostname, response)
  15. 15. Eventlet:  GreenPools   Setup  each  host  in  parallel  def setup_cannons(hostnames): print Loading cannons... , # Use eventlet’s abstraction of a collection of tasks: a GreenPile pile = eventlet.GreenPile(pool) # Spawn a coroutine in our pile for each host for hostname in hostnames: pile.spawn(_setup_a_cannon, hostname) # Iterating the pile causes eventlet to do the work responses = list(pile) return responses
  16. 16. Micro  Army:  Cannons  are  GO!   Time  to  blast  some  poor  web  server  (that  you  own)  def fire_cannon(cannon_host, target): ssh_conn = ssh_connect(cannon_host) # 200 simultaneous connections requesting data for 10 seconds remote_command = siege -c200 -t10s %s % (target) # Siege writes stats to stderr response = exec_command(ssh_conn, remote_command, return_stderr=True) return response
  17. 17. Micro  Army:  Fire  Cannons!   Same  strategy  as  before,  with  a  GreenPile  def slam_host(cannon_hosts, target): # Familiar pattern pile = eventlet.GreenPile(pool) for hostname in cannon_hosts: pile.spawn(fire_cannon, hostname, target) responses = list(pile) # Parse output from each host for CSV generation report = parse_responses(responses) return report
  18. 18. Micro  Army:  Finishing  up.   Just  the  facts,  micro.   Num_Trans,Elapsed,Tran_Rate   3679,                      9.54,              385.64   3635,                      9.48,              383.29   3535,                      9.33,              378.89  
  19. 19. Micro  Army:  Recap   • LAUNCH  some  number  of  EC2  micros     • INSTALL  siege,  etc  on  instances     • SLAM  web  host  from  micros  in  parallel     • AGGREGATE  the  results  in  a  CSV    
  20. 20. Hopefully  you  learned  how  to…  •  deploy  instances  on  EC2  with  Python  •  execute  a  script  on  many  boxes  in  parallel  •  use  SSH  from  Python   –  You  should  sEll  checkout  Fabric  (fabfile.org)  •  use  the  excellent  module,  Eventlet:   – Free  nonblocking  I/O   – Simple,  but  efficient  concurrency   – Read  the  webpage,  it  actually  gets  beAer   •  Green  threads!    
  21. 21. Try  it!   h?ps://github.com/j2labs/microarmy  h?ps://console.aws.amazon.com/ec2/home   Read  more!   h?p://eventlet.net   h?p://www.lag.net/paramiko/   h?p://boto.cloudhackers.com/  
  22. 22. Ques1ons   ?  James  Dennis  (@j2labs)   jdennis@gmail.com