Download to read offline
![Get our code to work at all
Reproduce our results
Share our results
Test our code properly
Change our dev environments
✗
✗
✗
✗
✗
Getting development environments right is hugely important, and the
consequences of screwing it up affect almost everything we do.
“[T]he alternative [to testing with SQLite] is not
testing at all because there is limited time for
testing and setting up a proper database for this is
so much more trouble.
The choices are not good test vs bad test.
They are test-with-issues vs. no test.”
- jbb555
Hacker News, August 4th, 2015
We are failing at managing the complexity of our dev environments.
This is from a Hacker News thread on testing code against SQLite instead of
the database you actually run in production.
Clearly we have a lot of work to do here.
Your Code
Language Libraries
Language Runtime
Userland
OS / Kernel
Hardware
Let’s start thinking through the problem by imagining a simple Python app
and getting it in the right environment to run correctly. If we want to share
that code or reproduce our results on another machine, we need to make
sure that our circles of Hell are compatible all the way down.
Most of the time, in Python-land, we are dealing with the three layers at the
top. In this example…](https://image.slidesharecdn.com/devenvironmentsthenextgeneration-150816223814-lva1-app6892/75/Dev-Environments-The-Next-Generation-3-2048.jpg)
Uploaded byTravis Thieman
Dev Environments: The Next Generation
The document discusses the complexities of development environments, emphasizing the importance of isolation through tools like virtual environments (venvs) and containers. It highlights the limitations of current solutions such as virtual machines and suggests that containers, particularly through the Docker ecosystem, provide a more scalable and manageable unit of isolation. The speaker introduces 'dusty', a framework that leverages these concepts to simplify development processes and improve collaboration, while recognizing ongoing challenges and the need for community contributions to enhance these tools.
More Related Content
Similar to Dev Environments: The Next Generation
Dev Environments: The Next Generation
- 1. thieman Travis Thieman THE NEXTGENERATION DEV ENVS Talk through the outline on this slide. What is a dev environment? Current state of the world (running on local and VMs) Ways we can make this better (containers) Whether we can realize this benefit with the tools available (Dusty)
- 2. What are developmentenvironments? It’s all of this crap. From the code you’re running, to the filesystem, to the OS, to the hardware, to the bits of electricity pinging around inside the machine, all of this affects what you’re actually trying to do, which is write and run software. Your Code Language Libraries Language Runtime Userland OS / Kernel Hardware Here’s a different model for thinking about all the crap inside our machine that affects how we run and write software. There’s a *lot* going on. Here’s yet another way of looking at a typical dev env that encompasses an entire machine. Dev envs are inherently complex, and our tiny human brains have no chance of being able to comprehend them. We need to create tooling to help us do this.
- 3. Get our codeto work at all Reproduce our results Share our results Test our code properly Change our dev environments ✗ ✗ ✗ ✗ ✗ Getting development environments right is hugely important, and the consequences of screwing it up affect almost everything we do. “[T]he alternative [to testing with SQLite] is not testing at all because there is limited time for testing and setting up a proper database for this is so much more trouble. The choices are not good test vs bad test. They are test-with-issues vs. no test.” - jbb555 Hacker News, August 4th, 2015 We are failing at managing the complexity of our dev environments. This is from a Hacker News thread on testing code against SQLite instead of the database you actually run in production. Clearly we have a lot of work to do here. Your Code Language Libraries Language Runtime Userland OS / Kernel Hardware Let’s start thinking through the problem by imagining a simple Python app and getting it in the right environment to run correctly. If we want to share that code or reproduce our results on another machine, we need to make sure that our circles of Hell are compatible all the way down. Most of the time, in Python-land, we are dealing with the three layers at the top. In this example…
- 4. Your Code requests 2.6.0 Python2.7 … … … Let’s say our Python code needs a specific version of requests and Python to run. If we have these set up, our code will probably run. So we give our awesome script to our friend and tell her to install requests 2.6.0. pip install requests==2.6.0 This goes fine, but it turns out she already had requests 2.5.0 installed to run some other important app on her machine. When we install 2.6.0… pip install requests==2.6.0 requests 2.5.0 …her old version gets unceremoniously uninstalled by pip.
- 5. Your Code +Friend’s Code requests 2.5.0 Python 2.7 … … … requests 2.6.0 So we have a conflict here. Because of how Python works, these two versions cannot live harmoniously within the same set of Python libraries. So, if they refuse to get along, we’ll have to separate them. Your Code Language Libraries Language Runtime Userland OS / Kernel Hardware This is where we’re going to see the concept of isolation come into play. What if we cut our circles of Hell above the system library level. We’ll keep *one* total copy of everything below the cut, but we’ll let ourselves create multiple, isolated versions of everything above the cut. Userland OS / Kernel Hardware Your Code requests 2.6.0 Runtime Friend’s Code requests 2.5.0 Runtime This means we can have a structure that looks like this. Two separate, isolated stacks of Python components. In one, our awesome Riker code and requests 2.6.0. In the other, our friend’s app.
- 6. Hey, guess what,this is actually a thing! Venvs give us scalable isolation over our Python programs. This lets us resolve simple version conflicts, but scalable isolation has a bigger, more general impact on our development process as well. Isolation is also really useful for keeping the number of things we need to think about smaller, which helps us approach and reason about complicated tasks. Without isolation, we have to consider all parts of the system all at once, all of the time. If we have a problem in the middle of this mess, we’re going to have to wade through the whole thing to fix it. This is really hard.
- 7. With scalable isolation(like venvs!) we can break the problem down into smaller, isolated components. Now, if we have a problem in a specific area… …we can focus our cognitive effort entirely on that unit of isolation, which is much smaller and easier to reason about. Scalable isolation helps us identify problems and takes away a lot of the cognitive burden we otherwise face when trying to solve them.
- 8. Venvs are notenough • Need C extensions? • Conflicting system libraries? • A language other than Python? • Run a database? Multiple databases? However, virtualenvs don’t go far enough. We need a more general solution. Your Code Language Libraries Language Runtime Userland OS / Kernel Hardware So we need our unit of isolation to penetrate deeper into the circles of Hell. Let’s recall what this entire thing is equivalent to… …a machine, right? And one machine is one big stacked Hell metaphor. Let’s restrict ourselves to running on one physical machine.
- 9. Your Code Language Libraries LanguageRuntime Userland OS / Kernel Hardware Beyond that restriction, let’s go wild. What if we use a unit of isolation that encompasses everything BUT the hardware? Your Code Language Libraries Language Runtime Userland OS / Kernel Hardware It’d look like this, right? We’re definitely going to get some great isolation out of this. Hardware Your Code Language Libraries Language Runtime Userland OS / Kernel Your Code Language Libraries Language Runtime Userland OS / Kernel It’d look like this, right? We’re definitely going to get some great isolation out of this.
- 10. This is calleda virtual machine. It is, also, totally a thing. There’s a big ecosystem around these, because a lot of people use them for their dev environments. Up until recently, they were the latest greatest thing in dev environments. VMs have one major flaw that leads to a whole host of problems: they’re big and slow. Turns out, running a whole ‘nother kernel takes a lot of clever engineering and the end result doesn’t scale well. As a result, we *can’t* run a bunch of VMs. Maybe two, maybe three, probably not five, definitely not 100. Hardware Your Code Language Libraries Language Runtime Userland OS / Kernel Your Code Language Libraries Language Runtime Userland OS / Kernel Host Machine Guest VM This winds up with us ending up basically where we started. All of our stuff ends up in the same unit of isolation. Whether it’s on one Hell stack or in 1 VM running in parallel with our host machine, it’s effectively the same. VMs give us a way to jumpstart the process, but not much more. You end up with most of the same problems you had running locally. We need something better.
- 11. Your Code Language Libraries LanguageRuntime Userland OS / Kernel Hardware Let’s review. Venvs are lightweight, pretty simple, but they don’t offer enough isolation. Your Code Language Libraries Language Runtime Userland OS / Kernel Hardware Virtual machines offer great isolation, but they’re so heavyweight that we can’t run many of them. And, really, do we need a separate OS and kernel to run a Python app, or even a database? Generally, no. Your Code Language Libraries Language Runtime Userland OS / Kernel Hardware What if we learned something from Goldilocks and created a unit of isolation above the OS?
- 12. OS / Kernel Hardware YourCode Language Libraries Language Runtime Userland Your Code Language Libraries Language Runtime Userland We know from VMs that running multiple kernels is really hard and expensive, so what if we split the circles of Hell right above that and created a new unit of isolation? This could let us run our two apps with different library dependencies pretty easily. OS / Kernel Hardware This isolation is incredibly powerful. Now we *only* need the dependencies in that unit of isolation that let us run our code. So we can make them pretty small and MUCH easier to reason about! OS / Kernel Hardware Taking this approach to its natural conclusion, we can create a bunch of these small units of isolation for everything we need to run as part of our development environments There are numerous benefits here. Smaller, easier to reason about. Actually scalable, unlike VMs.
- 13. = This unit ofisolation already exists, and it’s called a container! If you’ve heard about the Docker project, that’s the leader in implementing containers right now. Docker provides a core engine for managing containers, and has done a lot of work around building an ecosystem around all of this. So we have this new shiny scalable unit of isolation called a container. What can we do with it?
- 14. Get our codeto work at all Reproduce our results Share our results Test our code properly Change our dev environments ? ? ? ? ? To review, let’s go back to our scary checklist from earlier. This is what we *want* our dev environments to help us do. Let’s see how containers can achieve this. First, we need to get our code to work at all. Because our containers only need to have the minimum amount of stuff to run our app, this is usually pretty simple! The isolation keeps the amount of stuff we need to think about pretty low. Let’s just throw in exactly what we need. OS / Kernel Hardware When we throw this on top of the non-isolated parts of our stack, we end up with a running container! It’s small, isolated, and runs our code just fine.
- 15. Get our codeto work at all Reproduce our results Share our results Test our code properly Change our dev environments ? ? ? ? One of the biggest value adds of the Docker ecosystem is the ability to easily reproduce and share containers. You can either… My Awesome Python Container PyGotham 2015 …take a snapshot of the container once you’ve got it the way you like. Quick and dirty.
- 16. FROM debian:jessie RUN apt-getupdate && apt-get install python ADD . /my-python-app RUN pip install -r /my-python-app/reqs.txt CMD python /my-python-app/awesome-app.py You can also formalize the instructions on how to build the container, which makes it actually reproducible. Get our code to work at all Reproduce our results Share our results Test our code properly Change our dev environments ? ? OS / Kernel Hardware Containers enable an awesome testing flow. Let’s say we’re running our Python app against a Postgres database. We take advantage of the scalable isolation of containers to create a totally separate copy of the app container running against a fresh Postgres container. We can even use a separate container with our test requirements already installed. Afterwards we can throw those temporary containers away.
- 17. Get our codeto work at all Reproduce our results Share our results Test our code properly Change our dev environments? OS / Kernel Hardware Scalable isolation is what makes it easy for us to change our development environments. It doesn’t matter how many containers we add, this Python container will still work just fine, because it’s not really affected by external forces. We could visit an apocalypse on all the other containers individually, but our Python container won’t be affected by anything that happens inside of them. Get our code to work at all Reproduce our results Share our results Test our code properly Change our dev environments
- 18. Great, so everythingis sunshine and rainbows, right? Talk over, goodbye. NOW WE GOT PROBLEMS Not quite. Using containers for everything is going to introduce some problems we didn’t have before. OS / Kernel Hardware Isolation makes normal everyday stuff like navigating through a space in a shell, copying files around, or accessing processes through ports more difficult. Maybe you’re starting out in this space over here, but your Python app is over here in a separate unit of isolation, and your database is separate from both of those.
- 19. OS / Kernel Hardware Wranglingyour vast menagerie of containers can also get overwhelming, even with relatively small stacks. When you add in test containers with short lifecycles, the problem gets worse. The solution is still *simple*, but it’s also *complicated* (as opposed to complex). Some tooling and automation could go a long way here. Another problem: how do you share your crazy setup with all these containers floating around? How do you share how to run tests on them? How do you share common tasks like a database migration that needs to be aware that it’s running in a containerized environment? Here’s another big issue…
- 20. …if you runLinux, you’re in luck. You can use containers out of the box. Did I mention this whole concept is more accurately called “Linux containers”? + + If you use something else like Mac or Windows, you’re going to need to run a Linux VM in order to utilize containers. This adds another level of isolation/ abstraction that we constantly need to navigate. This adds a lot of cognitive overhead unless we can automate it away via tooling. Docker Machine Docker Compose The Docker ecosystem provides us tools to solve some of these problems. Machine gives us a way to get up and running on non-Linux systems. Compose lets us wrangle sets of containers into a working stack in a reproducible and sharing way. But these are ultimately fairly low-level tools, and they can’t meet the specialized and complex needs of running a development environment specifically.
- 21. • Usability • Sharingcode • Restarting when code changes • Sharing tests • Navigating through containerland • Mix and Match Other dev env problems Dusty is our attempt to use containers to solve the problems with dev environments. It’s written in Python and uses Docker and existing tools from the Docker ecosystem. Dusty provides a solution which: 1. Uses containers to achieve the right level of isolation, keeping things simple. 2. Scalable. The 100th app you run in Dusty is as easy as the 1st. 3. Specialized. Dusty solves problems specific to a dev environment, like running tests against isolated database containers. • Usability • Sharing code • Restarting when code changes • Sharing tests • Navigating through containerland • Mix and Match Dusty allows to define groups of containers which can be toggled on or off separately. We can tell Dusty how to run hundreds of services, but if we only need one right now, it only bothers running the containers for that one. This mix and match lets us be efficient with our resources and scale out Dusty to support stacks of any size.
- 22. When Dusty runsyour containers, the plumbing needed to navigate the layers of isolation between your normal operating space (your local filesystem, etc) and the containers is set up for you. Your code gets seamlessly mounted into the container. You can make a request in your browser and Dusty will pipe it through the VM and into the container just the way you need. We’re developers, we think in code as often as we do in processes. To reflect this, Dusty doesn’t just know about your containers, it also knows about your repos. If you make a change in a repo, you can ask Dusty to make sure all containers using that repo get restarted to pick up the new code. This is also really easy to automate with a tool like watchdog. Dusty has first class support for the awesome container testing flow we talked about earlier. You can define a test script, give it a set of service containers to run against, and Dusty will handle rigging up the test harness for you. When it’s done, all the containers involved are cleaned up. Each time, you start fresh. This whole process takes about 5 seconds right now, and we’re still working to make it even faster.
- 23. Dusty, along withthe other tools in the Docker ecosystem, is going to help us manage away the pain of using containers and help us realize all the promise we saw earlier in the talk. Outro on technical pieces. Containers give us an opportunity to address a lot of the problems with existing local-only and VM-based dev environments. Docker and other container ecosystems are emerging and maturing, making it easier to get to a workable solution. Dusty and other projects are creating end-to-end solutions to make the dream of non-Hellish dev environments a reality. But we aren’t there yet. The night is dark, we have problems and a lot of work to do, and the Python community can help to solve this problem. Call to action for contributors and for people to start their own projects trying to solve the problem.
- 24.