Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Thinking inside a box: Dockerizing Perl
Steven Lembark
Workhorse Computing
lembark@wrkhors.com
What's on your box?
Pre-compiled perl distros are terrible.
Bulky, slow, with annoying dependencies.
Not what you need...
Example: Red Hat Enterprize 7.0
5.16 == “end of life”
­Doptimize=­O2 ­g ­pipe ­fstack­
protector­strong  ­mtune=generic  
...
Example: Red Hat Enterprize 7.0
5.16 == “end of life”
Not exactly built for speed.
­Doptimize=­O2 ­g ­pipe ­fstack­
protec...
Example: Red Hat Enterprize 7.0
5.16 == “end of life”
Not exactly built for speed.
Thread overhead, even if you don't use ...
Fresh, not frozen
perl for your architecture.
optimized.
only dependencies you use (need).
Q: How?
Try it, you'll like it...
Bad approach: virtual machines...
Try it, you'll like it...
Bad approach: virtual machines...
and version-dirs...
and recompiling for each distro, architect...
Try it, you'll like it...
False lazyness!
Tasty alternative: lxc
Essentially an LPAR – we've come full circle.
Use a process to isolate & localize code.
Share the k...
Fly in the soup
Ever try to use lxc?
Let alone finish the manpages?
Kills my appetite.
RTFM
lxc-attach.1
lxc-autostart.1
l...
Docker: MRE for linux
80/20 of lxc:
Layered filesystem + Repository + Command line.
More nutritious than exciting.
Still e...
Catch: Docker's doc is fattening!
Start with full Ubuntu image.
1.6GiB – Heart attack on a plate!
Includes X11 libs, lvm, ...
Q: How can we put docker on a healthy diet?
Start with something lighter-weight underneath?
Keep less of it in the contain...
A: There is more than one way to do it.
Build perl on a light[er] weight O/S.
Single package for perl + shared libs.
Shell...
A: There is more than one way to do it.
Copy perl on top of busybox.
Much leaner cuisine.
Decent collection of shell tools...
A: There is more than one way to do it.
Just perl
Minimal for distribution of local perl.
Lacks tools to inspect the build.
A: There is more than one way to do it.
Just the application.
No empty calories.
Requires local perl for distributed use.
Start by getting docker.
They provide shell code for the basics:
$curl -sL https://get.docker.io/ | sh;
$wget -qO- https:/...
Be yourself
Don't run as su!
Add your users to “docker” in /etc/group.
After that check that docker is running:
$ docker p...
Just a taste...
Minimal base container: busybox.
Good for experimenting:
$ docker pull busybox
Pulling repository busybox
...
Getting inside the box
/bin/sh is default entrypoint:
$ docker run -t -i busybox;
# <-- su in box, login id out.
# ping 8....
Gentoo is easy to dockerize
Common solution is a “stage-3” system.
Shell + libs + build tools.
Not much else.
About half t...
Finding a distribution
Start at the docker registry
https://registry.hub.docker.com/
Looking for stage-3 builds:
https://r...
Grabbing an O/S
Get the image:
$ docker pull jgkim/gentoo-stage3;
Run a container:
$ docker run –rm -i -t jgkim/gentoo-sta...
Building a perl container
Github has templates:
http://github.com/Perl/docker-perl
Dockerfiles like “5.020.000-64bit/Docke...
FROM buildpack-deps # parent container
RUN apt-get update && apt-get install -y curl procps # commands to pull perl
RUN mk...
Check the local arguments
$ perl -V;
...
config_args='-de -Dprefix=/opt/perl/5.20
-Doptimize=-O3 -march=native -pipe'
perl...
New Dockerfile
FROM jgkim/gentoo-stage3
MAINTAINER Steven Lembark <lembark@wrkhors.com>
WORKDIR /var/tmp/
RUN wget -O – ht...
Building Perl
The build takes input and optional repository tag.
Input is a directory not “Dockerfile”
$ cd /scratch/docke...
Checking local containers
Intermediate images labeled with “<none>”.
These were from prior tests.
This one didn't finish:
...
$ docker build --rm=false –tag=lembark/gentoo-perl .;
Step 0 : FROM rndevfx/gentoo-stage3-amd64-nomultilib
---> e9d0ce6614...
Easy fix
RUN /opt/perl/bin/h2ph -r -l /usr/inlcude;
True Lazyness
More re-cycled images:
---> Using cache
---> cb4c5cd0607e
Step 8 : RUN /opt/perl/bin/h2ph -r -l /usr/include...
And impatience
$ time docker build --rm=false --tag=lembark/perl-gentoo .
Sending build context to Docker daemon 4.096 kB
...
Docker Images:
REPOSITORY VIRTUAL SIZE
lemark/gentoo-perl 884.4 MB
jgkim/gentoo-stage3 741.2 MB
localhost:5000/lembark/bus...
Welcome to perl
“CMD” gets run with an interactive container:
$ docker run –rm -i -t lembark/gentoo-perl;
Loading DB routi...
Choose your toppings
Default for gentoo run /bin/bash <your command>.
Save typing with:
ENTRYPOINT [ “/opt/perl/bin/perl” ...
The next course
Stacked images inherit the ENTRYPOINT:
FROM lembark/gentoo-perl
CMD [ "/path/to/your/program" ]
runs
/opt/...
Test containers stack
Derive tests from the package.
Add ./t to another image.
WORKDIR [ “/path/to/your/code” ]
CMD [ "/op...
Minimizing virtual size
Cannot remove inter-RUN space.
884.4 MB includes 100MB of /var/tmp/perl-5.20.2.
Avoiding it requir...
Minimal Virtual Size: single "RUN" command
FROM jgkim/gentoo-stage3
MAINTAINER Steven Lembark <lembark@wrkhors.com>
WORKDI...
In most cases there is still a better way.
Note: You already have a local O/S.
Q: Why add another one to the container?
A:...
Copy perl on top of busybox
Build & install into /opt/perl.
/opt/perl/Dockerfile
FROM lembark/busybox_x86
COPY [ “.”, “/op...
Nice combination
Useful shell tools for "qx".
/bin/sh in case of broken build.
Smaller package:
REPOSITORY VIRTUAL SIZE
le...
Local dir with modules
/opt/perl/5.20/Dockerfile for bare copy of perl:
$ docker build –tag='lembark/perl-5.20' /opt/perl/...
Portion control for perl
What if you don't want CPAN::Reporter in Docker?
A: Keep a “docker-perl” install.
Extract stage3 ...
Nothing artificial, nothing added
Skip the Dockerfile: import a tarball.
"—change" inserts Dockerfile syntax into image.
c...
Essential ingredients
Catch, this won't run as-is: since perl needs shared libs.
In my case, from /lib64.
Q: Where to get ...
One way: Share libs
Run containers with read-only mount of /lib64:
docker run -v /lib64:/lib64:r …
Light-weight.
Fast: No ...
One way: bundle libs
“ldd” lists shared libs:
COPY them into the image with perl.
Build lib64 image from perl – order does...
Zero protein pill
Use "-v" to add /lib4 and /opt/perl.
Image: Single #! script to start the application.
Images are a few ...
One way: static perl
Build perl & modules “--static”.
No hetergenious server issues.
perl image is larger.
Best for tests:...
Result: portable, optimized, minimal perl.
Static perl with busybox: 68 MB.
Whole lot less than Ubuntu.
Not a virtual mach...
What did all of this get us?
Mainly an optimized, current perl.
With all [and only] the modules you need.
You also save th...
Summary
Docker makes lxc approachable.
You don't always need a full O/S distro in the container.
perl on busybox makes a t...
References
http://docs.docker.com/
.../builder Dockerfile directives
.../userguide What you think it is.
Upcoming SlideShare
Loading in …5
×

Docker perl build

2,414 views

Published on

The perl on most linux distros is a mess. Docker makes it easier to build and packge a local perl and applications. Problem is that Docker's manuals produce a mess of their own.

Distributing perl on top of Gentoo's stage3 distro, busybox, or nothing at all made good alternatives. This talk includes basics of setting up docker, building a local perl for it, and packaging perl or applications into images for use in containers.

Published in: Technology
  • Be the first to comment

  • Be the first to like this

Docker perl build

  1. 1. Thinking inside a box: Dockerizing Perl Steven Lembark Workhorse Computing lembark@wrkhors.com
  2. 2. What's on your box? Pre-compiled perl distros are terrible. Bulky, slow, with annoying dependencies. Not what you need...
  3. 3. Example: Red Hat Enterprize 7.0 5.16 == “end of life” ­Doptimize=­O2 ­g ­pipe ­fstack­ protector­strong  ­mtune=generic   ­Dversion=5.16.3 ­Dmyhostname=localhost  ­Dperladmin=root@localhost   ­Dusethreads ­Duseithreads  ­Ubincompat5005
  4. 4. Example: Red Hat Enterprize 7.0 5.16 == “end of life” Not exactly built for speed. ­Doptimize=­O2 ­g ­pipe ­fstack­ protector­strong  ­mtune=generic   ­Dversion=5.16.3 ­Dmyhostname=localhost  ­Dperladmin=root@localhost   ­Dusethreads ­Duseithreads  ­Ubincompat5005
  5. 5. Example: Red Hat Enterprize 7.0 5.16 == “end of life” Not exactly built for speed. Thread overhead, even if you don't use them. ­Doptimize=­O2 ­g ­pipe ­fstack­ protector­strong  ­mtune=generic   ­Dversion=5.16.3 ­Dmyhostname=localhost  ­Dperladmin=root@localhost   ­Dusethreads ­Duseithreads  ­Ubincompat5005
  6. 6. Fresh, not frozen perl for your architecture. optimized. only dependencies you use (need). Q: How?
  7. 7. Try it, you'll like it... Bad approach: virtual machines...
  8. 8. Try it, you'll like it... Bad approach: virtual machines... and version-dirs... and recompiling for each distro, architecture... and symlink hell(2)... …
  9. 9. Try it, you'll like it... False lazyness!
  10. 10. Tasty alternative: lxc Essentially an LPAR – we've come full circle. Use a process to isolate & localize code. Share the kernel. Light weight, fast startup, easy to ship.
  11. 11. Fly in the soup Ever try to use lxc? Let alone finish the manpages? Kills my appetite. RTFM lxc-attach.1 lxc-autostart.1 lxc-cgroup.1 lxc-checkconfig.1 lxc-checkpoint.1 lxc-clone.1 lxc-config.1 lxc-console.1 lxc-create.1 lxc-destroy.1 lxc-device.1 lxc-execute.1 lxc-freeze.1 lxc-info.1 lxc-ls.1 lxc-monitor.1 lxc-snapshot.1 lxc-start-ephemeral.1 lxc-start.1 lxc-stop.1 lxc-top.1 lxc-unfreeze.1 lxc-unshare.1 lxc-user-nic.1 lxc-usernet.5 lxc-usernsexec.1 lxc-wait.1 lxc.7 lxc.conf.5 lxc.container.conf.5 lxc.system.conf.5
  12. 12. Docker: MRE for linux 80/20 of lxc: Layered filesystem + Repository + Command line. More nutritious than exciting. Still evolving.
  13. 13. Catch: Docker's doc is fattening! Start with full Ubuntu image. 1.6GiB – Heart attack on a plate! Includes X11 libs, lvm, mdadm, grub, parted... perl.
  14. 14. Q: How can we put docker on a healthy diet? Start with something lighter-weight underneath? Keep less of it in the container? Well... since this is about perl:
  15. 15. A: There is more than one way to do it. Build perl on a light[er] weight O/S. Single package for perl + shared libs. Shell tools available for qx{...}. Still pretty heavy-weight.
  16. 16. A: There is more than one way to do it. Copy perl on top of busybox. Much leaner cuisine. Decent collection of shell tools. Shared lib's as layer or via -v.
  17. 17. A: There is more than one way to do it. Just perl Minimal for distribution of local perl. Lacks tools to inspect the build.
  18. 18. A: There is more than one way to do it. Just the application. No empty calories. Requires local perl for distributed use.
  19. 19. Start by getting docker. They provide shell code for the basics: $curl -sL https://get.docker.io/ | sh; $wget -qO- https://get.docker.io/ | sh; will do the deed on Fedora, Ubuntu/Debian, or Gentoo. Avoids issues with apt/yum, not much value with emerge. Need to validate kernel configs.
  20. 20. Be yourself Don't run as su! Add your users to “docker” in /etc/group. After that check that docker is running: $ docker ps; Get access to the repository: $ docker login;
  21. 21. Just a taste... Minimal base container: busybox. Good for experimenting: $ docker pull busybox Pulling repository busybox fd5373b3d938: Download complete ... f06b02872d52: Download complete
  22. 22. Getting inside the box /bin/sh is default entrypoint: $ docker run -t -i busybox; # <-- su in box, login id out. # ping 8.8.8.8; <-- network available. ... # exit; <-- exit docker process $ <-- original EUID
  23. 23. Gentoo is easy to dockerize Common solution is a “stage-3” system. Shell + libs + build tools. Not much else. About half the size of Ubuntu.
  24. 24. Finding a distribution Start at the docker registry https://registry.hub.docker.com/ Looking for stage-3 builds: https://registry.hub.docker.com/search?q=gentoo+stage3 I find: jgkim/gentoo-stage3 741.2 MB Reasonable start.
  25. 25. Grabbing an O/S Get the image: $ docker pull jgkim/gentoo-stage3; Run a container: $ docker run –rm -i -t jgkim/gentoo-stage3; # gcc --version; gcc 4.8.4 good supports “--arch=native”
  26. 26. Building a perl container Github has templates: http://github.com/Perl/docker-perl Dockerfiles like “5.020.000-64bit/Dockerfile”. git acquires “5.020.0-64bit-optimized” directory.
  27. 27. FROM buildpack-deps # parent container RUN apt-get update && apt-get install -y curl procps # commands to pull perl RUN mkdir /usr/src/perl WORKDIR /usr/src/perl # build dir within the container RUN curl -SL http://www.cpan.org/src/5.0/perl-5.20.0.tar.gz | tar -xz --strip-components=1 RUN ./Configure -Duse64bitall -des && make -j$(nproc) && TEST_JOBS=$(nproc) make test_harness && make install && make veryclean WORKDIR /root CMD ["perl5.20.0","-de0"] # /bin/sh perl5.20.0 -de0 Distro's Dockerfile
  28. 28. Check the local arguments $ perl -V; ... config_args='-de -Dprefix=/opt/perl/5.20 -Doptimize=-O3 -march=native -pipe' perl -MConfig -E 'say $Config{ config_args }'
  29. 29. New Dockerfile FROM jgkim/gentoo-stage3 MAINTAINER Steven Lembark <lembark@wrkhors.com> WORKDIR /var/tmp/ RUN wget -O – http://www.cpan.org/src/5.0/perl-5.20.2.tar.gz | gzip -dc tar | tar xf -; RUN cd perl-5.20.2 && Configure -de -Dprefix=/opt/perl -Dman1dir=none -Dman3dir=none -Doptimize='-O3 -march=native -pipe' ; RUN make -C perl-5.20.2 all test install distclean; RUN /opt/perl/bin/h2ph -r -a -l; CMD [ "/opt/perl/bin/perl", "-d", "-E", "42" ]
  30. 30. Building Perl The build takes input and optional repository tag. Input is a directory not “Dockerfile” $ cd /scratch/docker/gentoo+perl; $ docker build –tag='lembark/perl-gentoo' . ; Each step in the Dockerfile is an intermediate image.
  31. 31. Checking local containers Intermediate images labeled with “<none>”. These were from prior tests. This one didn't finish: $ docker images REPOSITORY TAG IMAGE ID CREATED VIRTUAL SIZE <none> <none> 5906d8edbb59 10 minutes ago 726.1 MB <none> <none> 10f5517fcada 23 minutes ago 726.1 MB <none> <none> 7d328e761704 26 minutes ago 725.5 MB <none> <none> 88f1e41aaed5 27 minutes ago 725.5 MB <none> <none> 28052ace7e04 28 minutes ago 725.5 MB <none> <none> 6f46836220d9 31 minutes ago 725.5 MB
  32. 32. $ docker build --rm=false –tag=lembark/gentoo-perl .; Step 0 : FROM rndevfx/gentoo-stage3-amd64-nomultilib ---> e9d0ce66148c Step 1 : MAINTAINER Steven Lembark ---> Using cache <-- recycled image ---> 41d5480e49e7 Step 2 : WORKDIR /var/tmp ---> Using cache ---> 010d3d70ced1 ... Step 7 : RUN make all test install; <-- first execution is here. ---> Running in 949c9ddfc2ef Step 8 : RUN /opt/perl/bin/h2ph -r -a -l; ---> Running in b084569e8fc3 -r and -a options are mutually exclusive INFO[1342] The command [/bin/sh -c /opt/perl/bin/h2ph -r -a -l;] returned a non-zero code oops...
  33. 33. Easy fix RUN /opt/perl/bin/h2ph -r -l /usr/inlcude;
  34. 34. True Lazyness More re-cycled images: ---> Using cache ---> cb4c5cd0607e Step 8 : RUN /opt/perl/bin/h2ph -r -l /usr/include; ---> Running in 92efee8c31e2 require '_h2ph_pre.ph'; ...
  35. 35. And impatience $ time docker build --rm=false --tag=lembark/perl-gentoo . Sending build context to Docker daemon 4.096 kB Sending build context to Docker daemon Step 0 : FROM rndevfx/gentoo-stage3-amd64-nomultilib ... Step 10 : CMD [ “/opt/perl/bin/perl” ] ---> Using cache ---> f7b83ecbe276 Successfully built f7b83ecbe276 real 0m0.241s user 0m0.019s sys 0m0.012
  36. 36. Docker Images: REPOSITORY VIRTUAL SIZE lemark/gentoo-perl 884.4 MB jgkim/gentoo-stage3 741.2 MB localhost:5000/lembark/busybox 1.9 MB Successful build: tagged image
  37. 37. Welcome to perl “CMD” gets run with an interactive container: $ docker run –rm -i -t lembark/gentoo-perl; Loading DB routines from perl5db.pl version 1.44 Editor support available. ... main::(-e:1): 0 DB<1> x $^V v5.20.2
  38. 38. Choose your toppings Default for gentoo run /bin/bash <your command>. Save typing with: ENTRYPOINT [ “/opt/perl/bin/perl” ] CMD [ "-d", "-E", "42" ] Use –entrypoint='/bin/bash' if perl fails.
  39. 39. The next course Stacked images inherit the ENTRYPOINT: FROM lembark/gentoo-perl CMD [ "/path/to/your/program" ] runs /opt/perl/bin/perl /path/to/your/program;
  40. 40. Test containers stack Derive tests from the package. Add ./t to another image. WORKDIR [ “/path/to/your/code” ] CMD [ "/opt/perl/bin/prove" ] Result: no tests in product image: docker run foo/bar; run application. docker run foo/bar-test; run base tests.
  41. 41. Minimizing virtual size Cannot remove inter-RUN space. 884.4 MB includes 100MB of /var/tmp/perl-5.20.2. Avoiding it requires a single RUN. No caching of intermediate steps. Final size 787.8 MB. Best for final construction.
  42. 42. Minimal Virtual Size: single "RUN" command FROM jgkim/gentoo-stage3 MAINTAINER Steven Lembark <lembark@wrkhors.com> WORKDIR /var/tmp/ # better yet, put this in a shell script and RUN ./build-perl! RUN wget -O – http://www.cpan.org/src/5.0/perl-5.20.2.tar.gz | gzip -dc tar | tar xf - && cd perl-5.20.2 && Configure -de -Dprefix=/opt/perl -Dman1dir=none -Dman3dir=none -Doptimize='-O3 -march=native -pipe' && make all test install distclean && cd .. && rm -rf 5.20.2 ; RUN /opt/perl/bin/h2ph -r -a -l; ENTRYPOINT [ "/opt/perl/bin/perl" ] CMD [ "-d", "-E", "42" ]
  43. 43. In most cases there is still a better way. Note: You already have a local O/S. Q: Why add another one to the container? A: Because we are all used to virtual machines. Time to reduce the calories...
  44. 44. Copy perl on top of busybox Build & install into /opt/perl. /opt/perl/Dockerfile FROM lembark/busybox_x86 COPY [ “.”, “/opt/perl” ] ENTRYPOINT [ “/opt/perl/bin/perl” ] CMD [ , “-d”, “-E”, “0” ] Check build: docker run –entrypoint='/bin/sh'
  45. 45. Nice combination Useful shell tools for "qx". /bin/sh in case of broken build. Smaller package: REPOSITORY VIRTUAL SIZE lembark/busybox-perl 67.5 MB lembark/gentoo-perl 787.8 MB jgkim/gentoo-stage3 741.2 MB localhost:5000/lembark/busybox 1.9 MB
  46. 46. Local dir with modules /opt/perl/5.20/Dockerfile for bare copy of perl: $ docker build –tag='lembark/perl-5.20' /opt/perl/5.20; FROM /lembark/busybox MAINTAINER Steven Lembark <lembark@wrkhors.com> COPY [ ".", "/opt/perl/5.20" ] WORKDIR /var/tmp ENTRYPOINT [ "/opt/perl/5.20/bin/perl" ] CMD [ "-d", "-E", "0" ]
  47. 47. Portion control for perl What if you don't want CPAN::Reporter in Docker? A: Keep a “docker-perl” install. Extract stage3 into /scratch/docker-build. Build perl “chroot /scratch/docker-build”. Single /opt/perl with vetted modules. Works for just about anything.
  48. 48. Nothing artificial, nothing added Skip the Dockerfile: import a tarball. "—change" inserts Dockerfile syntax into image. cd /opt/perl; find . | cpio -ov -Htar | docker import –change=”VOLUME /lib64” --tag=”lembark/perl-5.20.2”; Minimal base for Plack server.
  49. 49. Essential ingredients Catch, this won't run as-is: since perl needs shared libs. In my case, from /lib64. Q: Where to get them without O/S image?
  50. 50. One way: Share libs Run containers with read-only mount of /lib64: docker run -v /lib64:/lib64:r … Light-weight. Fast: No images to ship. Requires homogeneous lib's for distributed use. Or "-v /var/tmp/$$:/var/tmp"
  51. 51. One way: bundle libs “ldd” lists shared libs: COPY them into the image with perl. Build lib64 image from perl – order doesn't matter. linux-vdso.so.1 (0x00007ffdfbbcb000) libperl.so => /opt/perl/5.20/lib/5.20.2/x86_64-linux/CORE/libperl.so (0x00007f40f8868000) libnsl.so.1 => /lib64/libnsl.so.1 (0x00007f40f8650000) libdl.so.2 => /lib64/libdl.so.2 (0x00007f40f844c000) libm.so.6 => /lib64/libm.so.6 (0x00007f40f8153000) libcrypt.so.1 => /lib64/libcrypt.so.1 (0x00007f40f7f1c000) libutil.so.1 => /lib64/libutil.so.1 (0x00007f40f7d19000) libc.so.6 => /lib64/libc.so.6 (0x00007f40f7981000) /lib64/ld-linux-x86-64.so.2 (0x00007f40f8c2b000)
  52. 52. Zero protein pill Use "-v" to add /lib4 and /opt/perl. Image: Single #! script to start the application. Images are a few KB. Requires homogenous install of perl, application.
  53. 53. One way: static perl Build perl & modules “--static”. No hetergenious server issues. perl image is larger. Best for tests: no issues with underlying images.
  54. 54. Result: portable, optimized, minimal perl. Static perl with busybox: 68 MB. Whole lot less than Ubuntu. Not a virtual machine. Plack web server FROM this perl. Viola!, you're up.
  55. 55. What did all of this get us? Mainly an optimized, current perl. With all [and only] the modules you need. You also save the overhead of shipping an O/S image. Faster startup. Easier updates. Simpler deployment.
  56. 56. Summary Docker makes lxc approachable. You don't always need a full O/S distro in the container. perl on busybox makes a tasty, low-calorie alternative. Use “-v” to import /lib64 for a full meal. Even testing gets simpler: derive tests from package. Say goodby to brewing perl, managing multiple versions.
  57. 57. References http://docs.docker.com/ .../builder Dockerfile directives .../userguide What you think it is.

×