Your SlideShare is downloading. ×
0
Testing with Docker
Tom Offermann
@toffermann
June 17, 2014
Introduction
‣What’s the Problem?
Testing in Varied Environments
!
‣Our Solution
Step-by-Step Explanation
Presentation Overview
‣What’s the Problem?
Testing in Varied Environments
!
‣Our Solution
Step-by-Step Explanation
Presentation Overview
What’s the Problem?
What Does New Relic Do?
What’s the Problem?
What’s the Problem?
Is Best!
What’s the Problem?
Python
versions
What’s the Problem?
Python
versions
frameworks
What’s the Problem?
Python
versions
frameworksdatabases
What’s the Problem?
Python
versions
frameworksdatabases
database
clients
What’s the Problem?
Python
versions
frameworksdatabases
database
clients
How do we test
all of these
scenarios?
‣What’s the Problem?
Testing in Varied Environments
!
‣Our Solution
Step-by-Step Explanation
Presentation Overview
‣What’s the Problem?
Testing in Varied Environments
!
‣Our Solution
Step-by-Step Explanation
Presentation Overview
Our Solution
✤ Create Docker Images
✤ Run a Test Container
✤ Packnsend
Our Solution
✤ Create Docker Images
✤ Run a Test Container
✤ Packnsend
Create Docker Images
# Dockerfile for memcached!
!
FROM ubuntu:12.04!
!
RUN apt-get update!
RUN apt-get install -y memcach...
Create Docker Images
$ tree python_agent/docker !
!
python_agent/docker!
...!
!"" mysql!
#   $"" Dockerfile!
!"" postgresq...
Our Solution
✤ Create Docker Images
✤ Run a Test Container
!
✤ Packnsend
Our Solution
✤ Create Docker Images
✤ Run a Test Container
• Install testing tools

!
✤ Packnsend
Testing Tools
x 4
Testing Tools
x 4
tox
Testing Tools
x 4
tox
virtualenv
Our Solution
✤ Create Docker Images
✤ Run a Test Container
• Install testing tools

• Copy source code

!
✤ Packnsend
Copy Source Code
Technique #1: Pull!
!
!
$ docker run test-image test.sh !
! --repo http://github.com/nr/python-agent !
! ...
Copy Source Code
Technique #1: Pull!
!
!
$ docker run test-image test.sh !
! --repo http://github.com/nr/python-agent !
! ...
Copy Source Code
Technique #2: Host-Mounted Volume!
!
!
$ docker run -v /path/to/repo:/code !
! test-image test.sh tox!
!


Copy Source Code
Technique #2: Host-Mounted Volume!
!
!
$ docker run -v /path/to/repo:/code !
! test-image test.sh tox!
!
...
Copy Source Code
Technique #3: Docker ADD to Image!
!
!
$ DATA_DIR=`mktemp -d $TMPDIR/packnsend.XXXXXXXX`!
!
$ git checkou...
Copy Source Code
Technique #3: Docker ADD to Image!
!
# Dockerfile for Test Image!
# Copy to $DATA_DIR!
FROM python-base!
...
Copy Source Code
Technique #3: Docker ADD to Image!
!
# Dockerfile for Test Image!
# Copy to $DATA_DIR!
FROM python-base!
...
Copy Source Code
Technique #3: Docker ADD to Image!
!
!
$ IMG_NAME="packnsend-`date '+%Y%m%d%H%M%S'`-$$"!
!
$ cd $DATA_DIR...
Our Solution
✤ Create Docker Images
✤ Run a Test Container
• Install testing tools

• Copy source code

• Use service cont...
Use Service Containers
Linking Docker Containers!
!
!
$ docker run -d --name db mysql!
!
$ docker run -d --link db:db —nam...
Use Service Containers
Linking Docker Containers!
!
!
DB_PORT_3306_TCP_ADDR=172.17.0.5!
DB_PORT_3306_TCP_PORT=3306!
DB_POR...
Use Service Containers
Linking Docker Containers!
!
$ docker run -d !
--name $DATA_IMAGE_NAME !
--link packnsend-gearmand:...
Our Solution
✤ Create Docker Images for Services
✤ Running a Test Container
• Install testing tools

• Copy source code

•...
Run Test Command
!
$ docker run -d !
--name $DATA_IMAGE_NAME !
--link packnsend-gearmand:gearmand !
--link packnsend-memca...
Our Solution
✤ Create Docker Images for Services
✤ Running a Test Container
!
✤ Packnsend
Packnsend
packnsend: COMMAND [arg...]!
!
Run Commands:!
run Run command on test container.!
run -i Run command interactive...
Packnsend
Demo!
Our Solution
✤ Create Docker Images for Services
✤ Running a Test Container
✤ Packnsend
• Launcher Script

!
Launcher Script
!
#!/bin/sh!
!
. /data/docker/environ!
!
"$@" > /data/out.log 2>&1!
!
STATUS=$?!
!
cp /data/out.log /tmp/o...
Launcher Script
!
#!/bin/sh!
!
. /data/docker/environ # Set Env Vars!
!
"$@" > /data/out.log 2>&1!
!
STATUS=$?!
!
cp /data...
Source Environ
# Add environment variables!
export PACKNSEND_DB_USER=db_user!
!
# Munge existing environment variables!
# ...
Launcher Script
!
#!/bin/sh!
!
. /data/docker/environ!
!
"$@" > /data/out.log 2>&1 # Log Output!
!
STATUS=$?!
!
cp /data/o...
Launcher Script
!
#!/bin/sh!
!
. /data/docker/environ!
!
"$@" > /data/out.log 2>&1!
!
STATUS=$? # Capture Exit Code!
!
cp ...
Our Solution
✤ Create Docker Images for Services
✤ Running a Test Container
✤ Packnsend
• Launcher Script

• Automatic Cle...
Automatic Cleanup
!
cleanup_containers()!
{!
! docker kill $CONTAINER_ID!
! docker rm --volumes $CONTAINER_ID!
!
! docker ...
Automatic Cleanup
!
cleanup_containers()!
{!
! docker kill $CONTAINER_ID!
! docker rm --volumes $CONTAINER_ID!
!
! docker ...
Our Solution
✤ Create Docker Images
✤ Run a Test Container
✤ Packnsend
• Launcher Script

• Automatic Cleanup

• Remote Do...
Remote Docker Daemon
!
# SSH Tunnel!
!
$ export EC2=<ec2 address>!
$ ssh -f ubuntu@$EC2 -L 14243:$EC2:2375 -N !
!
!
# Set ...
Our Solution
✤ Create Docker Images
✤ Run a Test Container
✤ Packnsend
• Launcher Script

• Automatic Cleanup

• Remote Do...
Parallel Tests
# test_commands.txt!
packnsend run tox -c tests/tox1.ini!
packnsend run tox -c tests/tox2.ini!
packnsend ru...
Conclusion
Testing with Docker
Tom Offermann
@toffermann
June 17, 2014
Upcoming SlideShare
Loading in...5
×

Testing with Docker

2,120

Published on

Presentation at the Portland Docker Meetup on June 17, 2014.

Published in: Technology

Transcript of "Testing with Docker"

  1. 1. Testing with Docker Tom Offermann @toffermann June 17, 2014
  2. 2. Introduction
  3. 3. ‣What’s the Problem? Testing in Varied Environments ! ‣Our Solution Step-by-Step Explanation Presentation Overview
  4. 4. ‣What’s the Problem? Testing in Varied Environments ! ‣Our Solution Step-by-Step Explanation Presentation Overview
  5. 5. What’s the Problem? What Does New Relic Do?
  6. 6. What’s the Problem?
  7. 7. What’s the Problem? Is Best!
  8. 8. What’s the Problem? Python versions
  9. 9. What’s the Problem? Python versions frameworks
  10. 10. What’s the Problem? Python versions frameworksdatabases
  11. 11. What’s the Problem? Python versions frameworksdatabases database clients
  12. 12. What’s the Problem? Python versions frameworksdatabases database clients How do we test all of these scenarios?
  13. 13. ‣What’s the Problem? Testing in Varied Environments ! ‣Our Solution Step-by-Step Explanation Presentation Overview
  14. 14. ‣What’s the Problem? Testing in Varied Environments ! ‣Our Solution Step-by-Step Explanation Presentation Overview
  15. 15. Our Solution ✤ Create Docker Images ✤ Run a Test Container ✤ Packnsend
  16. 16. Our Solution ✤ Create Docker Images ✤ Run a Test Container ✤ Packnsend
  17. 17. Create Docker Images # Dockerfile for memcached! ! FROM ubuntu:12.04! ! RUN apt-get update! RUN apt-get install -y memcached! ! EXPOSE 11211! ENTRYPOINT ["memcached"]! CMD ["-p", "11211"]! USER daemon!
  18. 18. Create Docker Images $ tree python_agent/docker ! ! python_agent/docker! ...! !"" mysql! #   $"" Dockerfile! !"" postgresql! #   $"" Dockerfile! ...! $"" redis!    $"" Dockerfile!
  19. 19. Our Solution ✤ Create Docker Images ✤ Run a Test Container ! ✤ Packnsend
  20. 20. Our Solution ✤ Create Docker Images ✤ Run a Test Container • Install testing tools ! ✤ Packnsend
  21. 21. Testing Tools x 4
  22. 22. Testing Tools x 4 tox
  23. 23. Testing Tools x 4 tox virtualenv
  24. 24. Our Solution ✤ Create Docker Images ✤ Run a Test Container • Install testing tools • Copy source code ! ✤ Packnsend
  25. 25. Copy Source Code Technique #1: Pull! ! ! $ docker run test-image test.sh ! ! --repo http://github.com/nr/python-agent ! ! tox -c tests/tox.ini! ! 

  26. 26. Copy Source Code Technique #1: Pull! ! ! $ docker run test-image test.sh ! ! --repo http://github.com/nr/python-agent ! ! tox -c tests/tox.ini! ! 
 Downside: Requires commit to git repository
  27. 27. Copy Source Code Technique #2: Host-Mounted Volume! ! ! $ docker run -v /path/to/repo:/code ! ! test-image test.sh tox! ! 

  28. 28. Copy Source Code Technique #2: Host-Mounted Volume! ! ! $ docker run -v /path/to/repo:/code ! ! test-image test.sh tox! ! 
 Downside: Problematic with boot2docker
  29. 29. Copy Source Code Technique #3: Docker ADD to Image! ! ! $ DATA_DIR=`mktemp -d $TMPDIR/packnsend.XXXXXXXX`! ! $ git checkout-index --prefix=$DATA_DIR/ -a!
  30. 30. Copy Source Code Technique #3: Docker ADD to Image! ! # Dockerfile for Test Image! # Copy to $DATA_DIR! FROM python-base! ! RUN chown -R guest.users /data! RUN chmod 0755 /data! USER guest! ENV HOME /home/guest! ! ADD . /data! VOLUME /data!
  31. 31. Copy Source Code Technique #3: Docker ADD to Image! ! # Dockerfile for Test Image! # Copy to $DATA_DIR! FROM python-base! ! RUN chown -R guest.users /data! RUN chmod 0755 /data! USER guest! ENV HOME /home/guest! ! ADD . /data! VOLUME /data!
  32. 32. Copy Source Code Technique #3: Docker ADD to Image! ! ! $ IMG_NAME="packnsend-`date '+%Y%m%d%H%M%S'`-$$"! ! $ cd $DATA_DIR && ! docker build --rm -t $IMG_NAME .!
  33. 33. Our Solution ✤ Create Docker Images ✤ Run a Test Container • Install testing tools • Copy source code • Use service containers ! ✤ Packnsend
  34. 34. Use Service Containers Linking Docker Containers! ! ! $ docker run -d --name db mysql! ! $ docker run -d --link db:db —name test $IMG_NAME!
  35. 35. Use Service Containers Linking Docker Containers! ! ! DB_PORT_3306_TCP_ADDR=172.17.0.5! DB_PORT_3306_TCP_PORT=3306! DB_PORT_3306_TCP_PROTO=tcp! DB_PORT_3306_TCP=tcp://172.17.0.5:3306! DB_PORT=tcp://172.17.0.5:3306!
  36. 36. Use Service Containers Linking Docker Containers! ! $ docker run -d ! --name $DATA_IMAGE_NAME ! --link packnsend-gearmand:gearmand ! --link packnsend-memcached:memcached ! --link packnsend-postgresql:postgresql ! --link packnsend-mysql:mysql ! --link packnsend-devpi:devpi ! --link packnsend-squid:squid ! --link packnsend-mongodb:mongodb ! --link packnsend-redis:redis ! $IMG_NAME!
  37. 37. Our Solution ✤ Create Docker Images for Services ✤ Running a Test Container • Install testing tools • Copy source code • Use services • Run test command ! ✤ Packnsend
  38. 38. Run Test Command ! $ docker run -d ! --name $DATA_IMAGE_NAME ! --link packnsend-gearmand:gearmand ! --link packnsend-memcached:memcached ! --link packnsend-postgresql:postgresql ! --link packnsend-mysql:mysql ! --link packnsend-devpi:devpi ! --link packnsend-squid:squid ! --link packnsend-mongodb:mongodb ! --link packnsend-redis:redis ! $DATA_IMAGE_NAME ! ! tox -c tests/tox.ini!
  39. 39. Our Solution ✤ Create Docker Images for Services ✤ Running a Test Container ! ✤ Packnsend
  40. 40. Packnsend packnsend: COMMAND [arg...]! ! Run Commands:! run Run command on test container.! run -i Run command interactively.! shell Launch a shell on test container.! ! Management Commands:! ! init Pull images, if authorized, else build.! build Build base images.! push Push base images.! pull Pull base images.! start Start base containers.! stop Stop base containers.! cleanup Delete base images.!
  41. 41. Packnsend Demo!
  42. 42. Our Solution ✤ Create Docker Images for Services ✤ Running a Test Container ✤ Packnsend • Launcher Script !
  43. 43. Launcher Script ! #!/bin/sh! ! . /data/docker/environ! ! "$@" > /data/out.log 2>&1! ! STATUS=$?! ! cp /data/out.log /tmp/out.log! ! exit `expr $STATUS`!
  44. 44. Launcher Script ! #!/bin/sh! ! . /data/docker/environ # Set Env Vars! ! "$@" > /data/out.log 2>&1! ! STATUS=$?! ! cp /data/out.log /tmp/out.log! ! exit `expr $STATUS`!
  45. 45. Source Environ # Add environment variables! export PACKNSEND_DB_USER=db_user! ! # Munge existing environment variables! # We have this form! # DEVPI_PORT=tcp://172.17.0.6:3141! ! if test -n "$DEVPI_PORT"! then! ! BASE=`echo $DEVPI_PORT | sed -e ’s/tcp/http/'`! ! PATH=packnsend/testing/+simple/! ! export PIP_INDEX_URL=$BASE/$PATH! fi
  46. 46. Launcher Script ! #!/bin/sh! ! . /data/docker/environ! ! "$@" > /data/out.log 2>&1 # Log Output! ! STATUS=$?! ! cp /data/out.log /tmp/out.log! ! exit `expr $STATUS`!
  47. 47. Launcher Script ! #!/bin/sh! ! . /data/docker/environ! ! "$@" > /data/out.log 2>&1! ! STATUS=$? # Capture Exit Code! ! cp /data/out.log /tmp/out.log! ! exit `expr $STATUS`!
  48. 48. Our Solution ✤ Create Docker Images for Services ✤ Running a Test Container ✤ Packnsend • Launcher Script • Automatic Cleanup
  49. 49. Automatic Cleanup ! cleanup_containers()! {! ! docker kill $CONTAINER_ID! ! docker rm --volumes $CONTAINER_ID! ! ! docker rmi $IMAGE_ID! ! ! rm -r $DATA_DIRECTORY! }!
  50. 50. Automatic Cleanup ! cleanup_containers()! {! ! docker kill $CONTAINER_ID! ! docker rm --volumes $CONTAINER_ID! ! ! docker rmi $IMAGE_ID! ! ! rm -r $DATA_DIRECTORY! }!
  51. 51. Our Solution ✤ Create Docker Images ✤ Run a Test Container ✤ Packnsend • Launcher Script • Automatic Cleanup • Remote Docker Daemon !
  52. 52. Remote Docker Daemon ! # SSH Tunnel! ! $ export EC2=<ec2 address>! $ ssh -f ubuntu@$EC2 -L 14243:$EC2:2375 -N ! ! ! # Set DOCKER_HOST environment variable! ! $ export DOCKER_HOST="localhost:14243"
  53. 53. Our Solution ✤ Create Docker Images ✤ Run a Test Container ✤ Packnsend • Launcher Script • Automatic Cleanup • Remote Docker Daemon • Parallel Tests !
  54. 54. Parallel Tests # test_commands.txt! packnsend run tox -c tests/tox1.ini! packnsend run tox -c tests/tox2.ini! packnsend run tox -c tests/tox3.ini! packnsend run tox -c tests/tox4.ini! packnsend run tox -c tests/tox5.ini! packnsend run tox -c tests/tox6.ini! ...! ! # Run tests in parallel! ! $ cat test_commands.txt | parallel!
  55. 55. Conclusion
  56. 56. Testing with Docker Tom Offermann @toffermann June 17, 2014
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×