Docker and Your Path to a Better Staging Environment - webinar by Gil Tayar
•
2 likes•6,527 views
** Full webinar recording here: https://youtu.be/cJqSr7ySTfo ** Staging environments are notoriously difficult to setup and maintain. Unless you have a top-notch DevOps team, staging environments are usually different from production environments, and because of that, are fraught with problems—from failing deployments, to out-of-disk-space errors, and various other errors. Even when the staging environment is great, it has one problem—there’s only one. If you want to test a feature branch, you have to “allocate time”, or alternatively install the feature branch and risk disrupting other testers. It’s time the testers took control! And build their own testing environments using Docker, Docker-Compose, and Kubernetes. In this talk, Sr. Software Architect Gil Tayar shows how to deploy an app on your local machine using Docker and Docker Compose, and run an E2E test on it. He also describes the necessary changes needed to make the application deployable in such a setup—turning it into a Twelve-Factor Application. Watch this hands-on session. and enjoy these key takeaways: Remind yourself why staging environments are problematic -- Learn what Docker is -- Quickly deploy an app that includes a frontend, backend service, and database, and run an E2E test on it -- Learn how Docker, Docker Compose and Kubernetes can help you easily build multiple ephemeral staging environments -- Enable you to help developers change their code so that it can be deployed using a Docker setup
Recommended
Recommended
More Related Content
What's hot
What's hot (20)
Similar to Docker and Your Path to a Better Staging Environment - webinar by Gil Tayar
Similar to Docker and Your Path to a Better Staging Environment - webinar by Gil Tayar (20)
More from Applitools
More from Applitools (20)
Recently uploaded
Recently uploaded (20)
Docker and Your Path to a Better Staging Environment - webinar by Gil Tayar
- 1. @giltayar Docker and the Path to a Better Staging Environment Gil Tayar (@giltayar) June 2018 https://github.com/giltayar/docker-and-the-path 1
- 3. @giltayar Staging Environments ● The environment where the application is deployed to, prior to deployment in production ● Also used prior to integrating code between teams ● Usually one environment, but can be more ● And, unfortunately… 3
- 4. @giltayar Almost, but not quite, entirely unlike production (apologies to Douglas Adams) 4
- 5. @giltayar Staging Environments ● Built using a combination of chewing gum and baling wire ● Not maintained as well as the production environment ● Not the same deployment procedure ● Not the same production infrastructure 5
- 6. @giltayar And this is where we test... 6
- 7. @giltayar@giltayar About Me ● My developer experience goes all the way back to the ‘80s. ● Am, was, and always will be a developer ● Testing the code I write is my passion ● Currently evangelist and architect @ Applitools ● We deliver Visual Testing tools: If you’re serious about testing, checkout Applitools Eyes ● Sometimes my arms bend back ● But the gum I like is coming back in style @giltayar 7
- 8. @giltayar 8
- 10. @giltayar Revolution in... ● How apps are developed ● How apps are tested ● How apps are deployed to production 10
- 11. @giltayar This Talk ● What Docker is ● How to use it with Docker-compose ● How to use it with Kubernetes ● How to build a production and staging environment with it 11
- 12. @giltayar Put your seatbelt on, it’s going to be a wild and technical ride 12
- 13. @giltayar End Goal ● Run our application in a staging environment using Docker ● Simple application: ○ MongoDB database ○ Blogging web application (frontend and backend) 13
- 14. @giltayar How would we run Mongo in a staging environment? ● Install it on one of the machines in Staging or… ● Build a VM image that includes Mongodb, and run it on AWS, VMware… 14
- 15. @giltayar Problematic ● Installing: ○ Automated install scripts are not easy ● VM image: ○ We can run two! ○ Not easy building the image ○ Start time is minutes. ○ But it’s the better option. 15
- 16. @giltayar Virtual Hardware The VM Option Blog app VM1 mongodb VM2 Operating System Operating System Hypervisor (AWS, Vmware, Hyper-V…) Virtual Hardware Hardware 16
- 17. @giltayar The Docker Option Blog app Docker Container 1 mongodb Process Isolation Process Isolation Operating System Hardware Docker Container 2 17
- 18. @giltayar Let’s Run MongoDB under Docker 18
- 19. @giltayar Dockerfile FROM ubuntu WORKDIR /data ENV DEBIAN_FRONTEND=noninteractive RUN apt-get update -y RUN apt-get install -q -y mongodb RUN mkdir -p /data/db EXPOSE 27017 CMD ["mongod"] ● An image is a frozen container ● A template for docker containers ● Set of instructions that is run in the container: ● Runs the base image (FROM) ● Executes the RUN-s (using WORKDIR and ENV) ● Adds some metadata to container (CMD, EXPOSE) ● Freezes the result into a docker image 19
- 20. @giltayar Building the docker image docker build . -t mymongo Where the files are Tag the image with a name 20
- 21. @giltayar Running a container $ docker run mymongo The image to run 21
- 22. @giltayar Yay! It’s working MongoDB Process Isolation 27017 Operating System Hardware Docker Container 1 22
- 23. @giltayar Connecting to mongo via mongo client $ mongo 23
- 24. @giltayar Why isn’t it connecting? (Docker and TCP Ports) MongoDB Process Isolation 27017 Mongo Client 27017 Operating System Hardware Docker Container 1 24
- 25. @giltayar Running a container with port mapping docker run -d -p 27017:27017 mymongo Host Port Container Port Detached 25
- 26. @giltayar This is what port mapping looks like MongoDB Process Isolation 27017 Mongo Client Operating System Hardware Docker Container 1 27017 26
- 27. @giltayar Connecting with Mongo Client $ mongo > db.users.save({username: 'giltayar', name: 'Gil Tayar'}) > db.users.find() { "_id" : ObjectId("5abf3d2bdef8a36d505d3732"), "a" : 1 } 27
- 28. @giltayar And… it’s connecting! MongoDB Process Isolation 27017 Mongo Client Operating System Hardware Docker Container 1 27017 28
- 29. @giltayar Connecting with Mongo Client $ mongo > db.users.save({username: 'giltayar', name: 'Gil Tayar'}) > db.users.find() { "_id" : ObjectId("5abf3d2bdef8a36d505d3732"), "a" : 1 } 29
- 30. @giltayar Running another Mongo container $ docker run -d -p 7017:27017 mymongo $ mongo localhost:7017 > db.users.save({username: 'giltayar', name: 'Gil Tayar'}) > db.users.find() { "_id" : ObjectId("5abf3d2bdef8a36d505d3732"), "a" : 1 } Use a different host port 30
- 31. @giltayar Two containers, different host ports MongoDB Process Isolation 27017 Mongo Client Docker Container 1 27017 MongoDB Process Isolation 27017 Docker Container 2 7017 31
- 32. @giltayar Maintenance of Docker Containers $ docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS 6e33d46a915c mymongo "mongod" Less than a second ago Up 1 second 0.0.0.0:27017->27017/tcp 44ce252702e6 mymongo "mongod" 9 seconds ago Up 11 seconds 0.0.0.0:7017->27017/tcp $ docker rm 6e33d46a915c $ docker ps CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS 44ce252702e6 mymongo "mongod" 9 seconds ago Up 11 seconds 0.0.0.0:7017->27017/tcp 32
- 33. @giltayar Let’s Run The Blog App 33
- 34. @giltayar Running the Blog App Locally $ cd ../blog-app-example $ npm install $ npm run build:frontend $ npm run dev 34
- 35. @giltayar Running the Blog App MongoDB Process Isolation 27017 Blog App Operating System Hardware Docker Container 1 27017 3000 35
- 36. @giltayar Dockerfile FROM node WORKDIR app ENV NODE_ENV=production COPY . . RUN npm install --production EXPOSE 3000 CMD ["node", "src/app.js"] ● “node” is an image that already has NodeJS installed ○ Thousands of these pre-built images in hub.docker.com. ● COPY copies from host directory into docker directory 36
- 37. @giltayar Building the Blog App $ cd ../blog-app-example $ npm run build:frontend $ docker build . -t myblog 37
- 38. @giltayar $ docker run -t -p 3000:3000 giltayar/blog-app-example ... Error: secret should be set at module.exports (.../index.js:21:42) at Object.<anonymous> (/app/src/routes/auth.js:14:13) Running the App 38
- 39. @giltayar Twelve Factor Apps ● A set of rules on how to run modern web apps ● Apply extremely well to docker apps ● Rule #3: Store config in the environment ● The blog app is a 12-factor app ● We have to pass: ○ A SECRET environment variable ○ a MONGODB_URI environment variable For more information: https://12factor.net 39
- 40. @giltayar $ docker run -t -p 3000:3000 -e SECRET=shhhh -e MONGODB_URI=localhost:27017 giltayar/blog-app-example ... Error: connect ECONNREFUSED 127.0.0.1:27017 at Object._errnoException (util.js:1003:13) Running the App Environment variable So I can Ctrl+C 40
- 41. @giltayar localhost in a container refers to the container itself MongoDB Process Isolation 27017 Docker Container 1 27017 Blog App Process Isolation 3000 Docker Container 2 3000 localhost:27017 41
- 42. @giltayar MongoDB Process Isolation 27017 mymongo Blog App Process Isolation 3000 Docker Container 2 3000 mymongo:27017 What if we could create one network? 42
- 43. @giltayar Running the App in a network $ docker network create mynet $ docker run -d --network=mynet --name=mongo mymongo $ docker run -t -p 3000:3000 -e SECRET=shhhh -e MONGODB_URI=mongo:27017 --network=mynet myblog Join the network Use this name 43
- 44. @giltayar Running the same app twice $ docker network create mynet $ docker run -d --network=mynet --name=mongo mymongo $ docker run -d -p 3000:3000 --network=mynet -e SECRET=shhhh -e MONGODB_URI=mongo:27017 myblog $ docker network create mynet2 $ docker run -d --network=mynet2 --name=mongo2 mymongo $ docker run -d -p 3001:3000 --network=mynet2 -e SECRET=shhhh -e MONGODB_URI=mongo2:27017 myblog 44
- 46. @giltayar MongoDB Process Isolation mymongo Blog App Process Isolation 3000 Docker Container 2 3000 MongoDB Process Isolation mymongo Blog App Process Isolation 3000 Docker Container 2 3001 Ephemeral Staging Environments! 46
- 47. @giltayar What if there was an even simpler way to run multiple containers? 47
- 49. @giltayar Docker Compose ● Write a file (docker-compose.yml) ● The file declares all the containers and the connections between them ● Then just “run” the docker composition Let’s do it! 49
- 50. @giltayar docker-compose.yml version: '3' services: mongo: image: mymongo blog: image: myblog environment: SECRET: shhh MONGODB_URI: mongo:27017 ports: - "3000:3000" 50
- 51. @giltayar Running it $ docker-compose up or… $ docker-compose up -d 51
- 53. @giltayar $ docker-compose stop or… $ docker-compose down Killing it 53
- 54. @giltayar Can we run two instances of a docker-compose? ● Yes, but we need to take care not to use the same host port. ● Time, alas, does not permit me to show you the details ● You can check the git repo for the full information 54
- 55. @giltayar OK, This is cool But what has this got to do with Staging? 55
- 56. @giltayar docker-compose ● Enables you to run the whole application locally ● Does not run your application in a staging environment… ● … or in production 56
- 57. @giltayar So whadda we do? 57
- 58. @giltayar 58
- 59. @giltayar 59
- 60. @giltayar Kubernetes is docker-compose for production 60
- 61. @giltayar Kubernetes ● Like Docker-compose: ○ Also uses docker containers to run services ○ Can also deploy easily using a set of declarative files ○ Easy to use ● But better: ○ Can deploy multiple services to multiple computers ○ Self-heals itself ○ Robust enough to be used in production 61
- 63. @giltayar Minikube (or… it might as well have been in the cloud) Minikube VM This laptop kubectl 63
- 64. @giltayar apiVersion: apps/v1 kind: Deployment metadata: name: myblog spec: selector: matchLabels: app: myblog replicas: 1 template: metadata: labels: app: myblog myblog.yml spec: containers: - name: myblog image: myblog ports: - containerPort: 3000 env: - name: MONGODB_URI value: 'mymongo:27017' - name: SECRET value: 'shhhhhh' 64
- 65. @giltayar Let’s “apply” the yml to the Kubernetes cluster $ kubectl apply -f myblog-1.yml deployment "myblog" created $ kubectl get pods NAME READY STATUS RESTARTS AGE myblog-56589c8f7f-xdlgz 0/1 ErrImagePull 0 5s 65
- 66. @giltayar Why the “ErrImagePull” Error? Minikube VM This laptop mymongo myblog Local docker registry ubuntu node ... Global docker registry @hub.docker.com pull 66
- 67. @giltayar Why the “ErrImagePull” Error? Minikube VM This laptop mymongo myblog Local docker registry ubuntu node giltayar/mymongo giltayar/myblog ... Global docker registry @hub.docker.com push pull 67
- 68. @giltayar So let’s push our images to the global docker registry $ docker tag myblog giltayar/myblog $ docker push giltayar/myblog $ docker tag mymongo giltayar/mymongo $ docker push giltayar/mymongo 68
- 69. @giltayar apiVersion: apps/v1 kind: Deployment metadata: name: myblog spec: selector: matchLabels: app: myblog replicas: 1 template: metadata: labels: app: myblog myblog.yml (fixed) spec: containers: - name: myblog image: giltayar/myblog ports: - containerPort: 3000 env: - name: MONGODB_URI value: 'mymongo:27017' - name: SECRET value: 'shhhhhh' 69
- 70. @giltayar Let’s reapply... $ kubectl apply -f myblog-2.yml $ kc get pods NAME READY STATUS RESTARTS AGE myblog-6b4567d975-h89g6 0/1 CrashLoopBackOff 3 3m $ kc logs myblog-6b4567d975-h89g6 ... Error: getaddrinfo ENOTFOUND mymongo mymongo:27017 70
- 71. @giltayar apiVersion: apps/v1 kind: Deployment metadata: name: mymongo spec: selector: matchLabels: app: mymongo replicas: 1 template: metadata: labels: app: mymongo mymongo.yml spec: containers: - name: mymongo image: giltayar/mymongo ports: - containerPort: 27017 71
- 72. @giltayar Let’s apply the mongo yml $ kubectl apply -f mymongo.yml $ kc get pods NAME READY STATUS RESTARTS AGE myblog-6b4567d975-h89g6 0/1 Error 6 7m mymongo-67b9bd8c7f-87j2j 1/1 Running 0 2m $ kc logs myblog-6b4567d975-h89g6 ... Error: getaddrinfo ENOTFOUND mymongo mymongo:27017 72
- 73. @giltayar Same Problem. Why? ● Because in docker-compose, the name of the service is also the way to discover it ● In kubernetes, a deployment is just a set of containers. ○ Not discoverable ● To “discover” pods, create a Kubernetes Service 73
- 74. @giltayar Pods are not discoverable mymongo myblog myblog myblog K8s Cluster 74
- 75. @giltayar We need a service mymongo myblog myblog myblog mymongo-service K8s Cluster 75
- 76. @giltayar mymongo-service.yml kind: Service apiVersion: v1 metadata: name: mymongo-service spec: selector: app: mymongo ports: - protocol: TCP targetPort: 27017 port: 27017 76
- 77. @giltayar apiVersion: apps/v1 kind: Deployment metadata: name: myblog spec: selector: matchLabels: app: myblog replicas: 1 template: metadata: labels: app: myblog myblog.yml (fixed again) spec: containers: - name: myblog image: giltayar/myblog ports: - containerPort: 3000 env: - name: MONGODB_URI value: 'mymongo-service:27017' - name: SECRET value: 'shhhhhh' 77
- 78. @giltayar And now applying... $ kubectl apply -f mymongo-service.yml service "mymongo-service" created $ kubectl apply -f myblog.yml deployment "myblog" configured $ kc get pods NAME READY STATUS RESTARTS AGE myblog-566ccc499c-sx8cf 1/1 Running 0 2m mymongo-67b9bd8c7f-87j2j 1/1 Running 0 14m 78
- 79. @giltayar Blog is running. Can we access it? ● No! It’s just a set of containers. ● We need a service. ● But this time, the service needs to be accessible from outside the cluster! 79
- 80. @giltayar We need a service mymongo myblog myblog myblog mymongo-service K8s Cluster myblog-service 80
- 81. @giltayar kind: Service apiVersion: v1 metadata: name: myblog-service spec: selector: app: myblog ports: - protocol: TCP targetPort: 3000 port: 3000 type: NodePort # this is how you enable access from outside myblog-service.yml 81
- 82. @giltayar And now applying... $ kubectl apply -f myblog.yml service "myblog-service" created $ kubectl apply -f myblog.yml deployment "myblog" configured $ kubectl get services NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) myblog-service NodePort 10.109.42.144 <none> 3000: 31489/TCP mymongo-service ClusterIP 10.103.150.100 <none> 27017/TCP 82
- 83. @giltayar And now we try it... $ minikube ip 192.168.64.8 http://192.168.64.8:31489/ 83
- 84. @giltayar And it works! ● And the same deployment procedure will work in production... 84
- 85. @giltayar $ kubectl create namespace blog1 $ kubectl -n blog1 apply -f mymongo.yml, Mymongo-service.yml, myblog.yml, myblog-service.yml Let’s create another parallel environment 85
- 86. @giltayar Summary ● The long journey… ● Running individual containers using docker ● Running environments locally using docker-compose ● Running environments in a production environment using Kubernetes 86
- 87. @giltayar This is how you build your staging environment: ● Docker images built by developers… ● … and deployed into Kubernetes clusters 87