What are the challenges to use Docker? Find out what we learned at New Relic software. Presented at Velocity 2016 Santa Clara (at the New Relic booth).
2. PresenterName,TitleandorDate
This document and the information herein (including any information that may
be incorporated by reference) is provided for informational purposes only and
should not be construed as an offer, commitment, promise or obligation on
behalf of New Relic, Inc. (“New Relic”) to sell securities or deliver any product,
material, code, functionality, or other feature. Any information provided hereby is
proprietary to New Relic and may not be replicated or disclosed without New
Relic’s express written permission.
Such information may contain forward-looking statements within the meaning of
federal securities laws. Any statement that is not a historical fact or refers to
expectations, projections, future plans, objectives, estimates, goals, or other
characterizations of future events is a forward-looking statement. These forward-
looking statements can often be identified as such because the context of the
statement will include words such as “believes,” “anticipates,” “expects” or words
of similar import.
Actual results may differ materially from those expressed in these forward-
looking statements, which speak only as of the date hereof, and are subject to
change at any time without notice. Existing and prospective investors, customers
and other third parties transacting business with New Relic are cautioned not to
place undue reliance on this forward-looking information. The achievement or
success of the matters covered by such forward-looking statements are based on
New Relic’s current assumptions, expectations, and beliefs and are subject to
substantial risks, uncertainties, assumptions, and changes in circumstances that
may cause the actual results, performance, or achievements to differ materially
from those expressed or implied in any forward-looking statement. Further
information on factors that could affect such forward-looking statements is
included in the filings we make with the SEC from time to time. Copies of these
documents may be obtained by visiting New Relic’s Investor Relations website at
ir.newrelic.com or the SEC’s website at www.sec.gov.
New Relic assumes no obligation and does not intend to update these forward-
looking statements, except as required by law. New Relic makes no warranties,
expressed or implied, in this document or otherwise, with respect to the
information provided.
4. Ruby Frontend + Java Backend
1) The Ruby Monolith
2) The Duolith
3) Services Architecture
200+ Services
Docker in Prod 2014
5. 5
• Based on Selenium
• Each script runs in its
own container
• Performance
• Security
A crash test dummy
for your software.
6. 6
• Docker is a Moving Target
• Only the BOLD run the current version
• CaaS announced this week
• Boot2Docker/ Docker ToolBox/ Docker for Mac(beta)
blog The Modern Developer Workstation on MacOS with Docker
• Docs are Written for Greenfield development
• No Runbook for Migrating existing apps
• Noisy Neighbors
CHALLENGES
7. A Cloud Platform (AWS, OpenStack)
Configuration Management (Chef, Puppet)
Testing (Vagrant)
Deployment Tool (pre-Swarm)
Built Our Own: Centurion
The Build Ships Containers to Docker Registry
Centurion Sends Containers to Docker Fleet
GitHub: Newrelic/Centurion
Docker is Not…
Luc Viatour / www.Lucnix.be
8. New Skills and Tools
For Ops
Library of Base Images
Solid Process for Building and Shipping
Good Internal Docs for Devs
For Devs
Learn the Tools
Let Go of the Notion of a Large and Persistent Filesystem
Make your code multithreaded
Take Ownership of the App’s Dependencies
App Dependencies
What’s New?
9. How Do You Know What’s
Happening With Your
Containers?
11. Comprehensive overview
CPU
physical memory
network activity
disk I/O utilization and
capacity
Server Monitoring
from the app
perspective.
12. Server Monitoring
from the app
perspective.
• Grouped by Docker Image
• Troubleshoot Quickly
• What Does Your
Application Cost
to Run?
13. Learn More
“Hello World” Tutorial at Docker.com
Docker In Production book
NewRelic.com/docker
Docker for Java Tutorial
github.com/docker/community/tree/master/tutorials/java
Infographic: The Life and Times of a Docker Container
Webinar: "Scalingwith Docker: New Relic's Containerization Journey"
Join Tori Wieldt, Developer Advocate at New Relic on Wednesday, April 6th@ 8 AM PT / 4 pm BST to find out how the move from monolith to a modern, scalable, software stack provides many lessons for enterprises making similar journeys.
April 6th, 2016 - 8 am PT (4 PM BST)
In this webinar, we will discuss:
New Relic's migration to containers
Is Docker ready for production?
How do you know what's happening with your containers?
We started off with a true monolith. We had a single application that ran our entire business that contained the agent, and the data collection pipeline, and the web interface. Back in ancient times, this was then divided out. And so, really, once we started getting a good clip of customers through, we had two large applications, or two small applications in the beginning.
One was this Ruby on Rails web application, which is the user interface, and another, which is a Java data collection pipeline. As our customer base grew, as our feature set grew, as our company grew, we started to have problems, which are probably familiar to many of you. And as our success continued to grow, the problems also continued to grow.
We’re going to create a service architecture,” and everyone jumped out and built lots and lots and lots of services. And as part of that, we had decided, we say, “Well you know what? I think that the services architecture thing could really take off, so we will probably have dozens of services.” And I believe our architect at the time said, “Oh, my friend, it could be 100,” and everyone said, “Oh, that’s bullshit. There’s no way it’s going to be 100.” It’s over 200 now.
So we said, “We need a way to deploy these faster.” We chose Docker. Keep in mind, we chose Docker in January of 2014, which was perhaps ambitious. But it had all of the characteristics we wanted. It isolated the experience of developing your application from the maintenance of that service. We had great efficiency and good uptime. When you really understood everything you were running, now we were saying, “Hey, operations group, instead of running two or three services, you’re going to run 50 or 70 or 200 services.” And so they needed to have a much more standardized interface on top of that to let them do that.
New Relic is one of the largest users of Docker.
Based on Selenium web browser automation
1. Docker is an API that are strapped to things in Linux kernel UFS/DeviceMapper/BTRfs
You would be very BOLD to run edge version of Docker a few dot releases from the edge
2. Now Native for Mac/Windoze
3. No playbook on how to Dockerize your DC, is in tribal knowledge form
Lots of pain…Get the workflow into everybody’s lives
Cloud Platform(AWS, OpenStack)
Docker can solve a lot of things, but sometimes you need a specific tool with more in-depth features than Docker can offer. Docker can significantly improve your organization’s ability to manage applications and your dependencies, it does not directly replace more traditional configuration management
Configuration Management (Chef, Puppet)
We use Vagrant to support testing on images that match our production environment. Since version 1.6, Vagrant natively supports Docker containers, which in some cases can serve as a substitute for a fully virtualized operating system
Docker can’t be used to automate a complex deployment process by itself. Other tools are usually still needed to stitch together the larger workflow automation.100s of deployments a day
Centurion is a tool to centrally manage configurations for fleets of Docker services and it has been instrumental in formalizing the handoff between developers and system administrators (Pre-Swarm)
Treats your Docker server farm as a deployment target (volume mounts and part mapping) use w Jenkins
Centurion: Docker Deployment Tool for repeatable deployments
The build ships container to Docker registry
Centurion sends container to Docker fleetGitHub: Newrelic/centurion
Can support Amazon S3 via external tools.
Both your Dev and your Ops sides will need to acquire new skill sets. On the Ops side you’ll need: A solid library of base images. A process for building and shipping Docker. And really good internal docs + support for developers as they uptake their share of all this stuff, so they don’t hate you. On the Dev side: You have to learn the tools. Take ownership of your app’s dependencies. Let go of certain things, like access to a large and persistent filesystem! NO LOGGING TO CONTAINER
We’ve found that converting legacy apps to run on Docker is a deep pain point. It’s great once you know all the dependencies of your app and can build containers repeatedly. But sussing them out can be a tough exercise. once your application is running in a container, there are all these things that you take for granted in your own environment that don't exist in a container. For example, that you can write it to the temp directory.
Just using Docker Scale doesn’t make you code multithreaded. Code your app to be truly redundant.
The process of containerizing your app means you configure state into environmental variables. Something that containerizes well is a web application that keeps its state in a DB, but the web app may have a bunch of localized settings on its own. you have to go through and discover each dependency that you didn't really know you had.
Containers are Isolated from each other, but it is probably more limited than you might expect. While you can put limits on resources, the default container config has them sharing CPU and memory on the host system. This means that unless you constrain them, containers can compete for resources on your production machines.
docker run can drop you into a container as root! But our Ops People use base images to force some best practices, like preventing anyone from logging into a container as root. Patch management is also an issue. When a vuln is discovered, you have to be able to upgrade your whole env fast. This is where that solid library of base images comes in.
1) Patch Docker host 2) Upgrade Dockerfiles 3) Devs have to pull updated base images to their machines
Docker 10.0 provides namespaces
that security is incrementally improving for containers as time goes on
One solution is to dedicate a nonzero UID to your containers. Onyour Docker hosts you can create a “container” userd ID of 4000 and then create the same UIS in your base images. If you then run all your contianers as UIS 4000, you improve the security of your system. Also it makes the ps output on your Docker host to easier to decipher.
The docker client already provides a rudimentary command line tool to inspect containers’ resource consumption. To look at the container stats run docker stats with the name(s) of the running container(s) for which you would like to see stats. This will present the CPU utilization for each container, the memory used and total memory available to the container. Note that if you have not limited memory for containers this command will post total memory of your host. This does not mean each of your container has access to that much memory. In addition you will also be able to see total data sent and received over the network by the container.
The docker stats command will only return a live stream of data for running containers. Stopped containers will not return any data.
Like the docker stats command, the API will continuously report a live stream of CPU, memory, I/O, and network metrics. The API provides more detail than the stats command.
Since Docker is evolving quickly, the metrics provided by these commands will likely continue to change significantly over the coming years.
containers are typically used in large numbers and have a very short half-life, so they can easily increase operational complexity by an order of magnitude.
if you have a operationally mature system, you should never have to SSH into it. Casey West, Pivotal Dev Advocate
Grouped by Image
Sorted by CPU or physical memory
Quickly know which app/image is having trouble
See the total cost of servicing your application in containers on a given server by rolling up performance to the image level.
How Container Distribution Effects Service Load
docker hello world tutorial on docker.com
Our Engineers wrote Docker Up and Running NewRelic.com/docker