Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Docker In Bank Unrated

1,622 views

Published on

Docker on a local machine and Docker in production — are two big differences. It's easy to play with technology but it's hard to do something real for many customers.

Half a year ago inside of Alpha Laboratory (division of Alfa-Bank) we've started building new microservices architecture for one of our pilot projects. We've almost completely changed a stack of the used technologies on a frontend and significantly changed it on a middle layer. For package and distribution we have choosen Docker. Two months ago we've deployed project to production and have opened service for clients.

In the report the following topics will be covered:
- reasons of a choice Docker;
- why Docker without other tools is not enough for a production;
- what stack of technologies we used in our solution;
- what advantages we've got;
- what problems have been faced and how we've solved them.

Published in: Software
  • Be the first to comment

Docker In Bank Unrated

  1. 1. Docker в Банке 1
  2. 2. DISCLAIMER Мнение докладчика может не совпадать с официальной позицией банка, его начальника, коллег или других специалистов. Докладчик не претендует на роль последней инстанции в вопросах использования той или иной технологии. Все представленные решения вы можете использовать на свой страх и риск. За все ваши действия ответственность несёте только вы сами. 2
  3. 3. Кто я? 9 лет > 1 года 4 года @aatarasoff aatarasoff Тарасов Александр http://developerblog.info/ 3
  4. 4. 4
  5. 5. 5
  6. 6. localhost 6
  7. 7. production 7
  8. 8. production 8
  9. 9. Production localhost 9 Some Big App Test Some Big App Some Big App Some Big App Some Big App Cluster
  10. 10. VM1 Java 6 J2EE Server Some Big App VM2 Java 6 J2EE Server Some Big App VMN Java 6 J2EE Server Some Big App 10
  11. 11. Some Big App Small App 1 Small App 3 Small App 2 11
  12. 12. Small App UI API 2 API 1 12
  13. 13. 13
  14. 14. Разные технологии 14
  15. 15. Разные процессы развёртывания 15
  16. 16. Различная конфигурация 16
  17. 17. 17
  18. 18. 18
  19. 19. 19
  20. 20. 20
  21. 21. 21
  22. 22. 22
  23. 23. 23
  24. 24. 24
  25. 25. 25
  26. 26. Унификация поставок 26
  27. 27. OS 27
  28. 28. OS Oracle Java 8 28
  29. 29. OS Oracle Java 8 Timezone Patch 29
  30. 30. OS Oracle Java 8 Timezone Patch Spring Boot Application 30
  31. 31. OS Oracle Java 8 Timezone Patch Spring Boot Application 1 Spring Boot Application 2 31
  32. 32. 32
  33. 33. FROM docker.moscow.alfaintra.net/java8 MAINTAINER aatarasov@alfabank.ru ADD payments-api.jar / CMD ["java", "-jar", "/payments-api.jar"] EXPOSE 8080 33
  34. 34. docker build -t docker.moscow.alfaintra.net/payments-api <build-dir> docker build -t docker.moscow.alfaintra.net/payments-api:0.0.1 <build-dir> 34
  35. 35. FROM docker.moscow.alfaintra.net/java8 MAINTAINER aatarasov@alfabank.ru ADD payments-api.jar / CMD ["java", "-jar", "/payments-api.jar"] EXPOSE 8080 35
  36. 36. /bin/payments-api.sh exec "$JAVACMD" "${JVM_OPTS[@]}" -classpath "$CLASSPATH" ru.alfabank.api.Application -e "JAVA_OPTS=-Xms128m -Xmx256m" 36
  37. 37. FROM docker.moscow.alfaintra.net/java8 MAINTAINER aatarasov@alfabank.ru ADD payments-api.tar / ENTRYPOINT ["/payments-api/bin/payments-api.sh"] EXPOSE 8080 37
  38. 38. 38
  39. 39. apply plugin: 'com.bmuschko.docker-remote-api' apply plugin: 'com.bmuschko.docker-java-application' apply from: "${project.projectDir}/variables.gradle" docker { url = 'unix:///var/run/docker.sock' registryCredentials { url = privateRegistryUrl } javaApplication { baseImage = 'docker.moscow.alfaintra.net/java8' maintainer = "${maintainer}" port = 8080 tag = "${dockerApplication}:${projectVersion}" } } https://github.com/bmuschko/gradle-docker-plugin gradle dockerBuildImage gradle dockerPushImage 39
  40. 40. tag = "${dockerApplication}:${projectVersion}" нельзя использовать "-" в значении свойства :( task pushToRegistry(type: DockerPushImage) { imageName = "${dockerApplication}" tag = "${jar.version}" registryCredentials = docker.registryCredentials } 40
  41. 41. OS Oracle Java 8 Java App 1 Timezone Patch Node JS Frontend App Java App 2 41
  42. 42. # install slow-changing app deps ADD package.deps.json /src/package.json RUN cd /src && npm install --production # install bower deps ADD bower.json /src/bower.json ADD .bowerrc /src/.bowerrc RUN cd /src && bower --allow-root install # install app ADD package.json /src/package.json RUN cd /src && npm install --production ADD .enb /src/.enb ADD . /src RUN cd /src && npm run make 42
  43. 43. CI Registry Target Server UI API 1 API 2 UI API 1 API 2 UI API 1 API 2 UI API 1 API 2 UI API 1 API 2 43
  44. 44. Единый API 44
  45. 45. docker pull docker run docker stop docker start docker logs docker stats 45
  46. 46. docker run docker.moscow.alfaintra.net/payments-api:0.0.1 docker run docker.moscow.alfaintra.net/customers-api:0.0.1 docker run docker.moscow.alfaintra.net/history-api:0.0.1 docker run docker.moscow.alfaintra.net/frontend-app:0.0.1 46
  47. 47. docker run -p 8080:8080 docker.moscow.alfaintra.net/payments-api:0.0.1 docker run -p 9081:8080 docker.moscow.alfaintra.net/payments-api:0.0.1 docker run -P docker.moscow.alfaintra.net/payments-api:0.0.1 docker run -P --name payments-api docker.moscow.alfaintra.net/payments-api:0.0.1 47
  48. 48. Изоляция процессов 48
  49. 49. -m=1024m --memory-swap=2048m -c=1024 --cpu-shares=512 --cpuset-cpus="1,3" 49
  50. 50. Target Server UI API 1 API 2 -m=1024m -cpu-shares=1024 -m=512m -cpu-shares=2048 -m=1024m -cpu-shares=512 50
  51. 51. Изоляция пререквизитов 51
  52. 52. VM1 VM2 Java 8 Java 7 app1 app2 52
  53. 53. VM1 VM2 Java 8 Java 7 app1 app2app1 53
  54. 54. VM1 VM2 Java 8 Java 7 app1 app2 VM3 Java 8 app1 54
  55. 55. Server1 Java 8 app1 Java 8 app1 Java 7 app2 Server2 Java 8 app1 Java 7 app2 Java 7 app2 55
  56. 56. 56
  57. 57. 57
  58. 58. 10инфраструктурных образов 58
  59. 59. gliderlabs/registrator consul template logstash kibana spring boot admin 59
  60. 60. 4 прикладных образа 60
  61. 61. 61
  62. 62. 62
  63. 63. 63
  64. 64. - hosts: log_servers remote_user: "{{docker_user}}" roles: - elastic - kibana - hosts: api remote_user: "{{docker_user}}" roles: - { role: logstash, log_type: api } - hosts: front remote_user: "{{docker_sudo}}" roles: - { role: logstash, log_type: front } [log_servers] dev_server_logs_1 dev_server_logs_2 dev_server_logs_3 [api] dev_server_api_1 dev_server_api_2 [front] dev_server_front_1 dev_server_front_2 dev_server_front_3 [log_servers] prod_server_logs_1 prod_server_logs_2 prod_server_logs_3 [api] prod_server_api_1 prod_server_api_2 [front] prod_server_front_1 prod_server_front_2 prod_server_front_3 logging_init.yml development production ansible-playbook -i development logging_init.yml ansible-playbook -i production logging_init.yml 64
  65. 65. 15 ролей 65
  66. 66. NGINX role front config role api config role docker pull docker run copy template docker restart nginx copy template docker restart nginx make volume dir chmod 66
  67. 67. 5 рабочих дней на развёртывание 67
  68. 68. 10 минут на развёртывание 68
  69. 69. 1 команда для установки ПО 69
  70. 70. 3 минуты на апдейт 70
  71. 71. 71
  72. 72. HTTP + DNS 72
  73. 73. http://consul:8500/v1/catalog/services { "consul-53": [ "udp" ], "consul-8500": [], "customers-api": [], "email": [], "logstash": [], "nginx-80": [], } 73
  74. 74. http://consul:8500/v1/catalog/service/logstash [ { "Node": "host1", "Address": "10.0.0.0", "ServiceID": "d1e985218049:logstash:5959", "ServiceName": "logstash", "ServiceTags": null, "ServiceAddress": "", "ServicePort": 5959 } ] 74
  75. 75. dig @consul logstash.service.consul ; ANSWER SECTION: logstash.service.consul. 0 IN A 10.0.0.0 logstash.service.consul. 0 IN A 10.0.0.1 75
  76. 76. Health Check curl, обратный ttl 76
  77. 77. http://consul:8500/v1/health/service/logstash "Checks": [ { "Node": "node1", "CheckID": "serfHealth", "Name": "Serf Health Status", "Status": "passing", "Notes": "", "Output": "Agent alive and reachable", "ServiceID": "", "ServiceName": "" }, { ... } ] 77
  78. 78. dig @consul logstash.service.consul ; ANSWER SECTION: logstash.service.consul. 0 IN A 10.0.0.0 logstash.service.consul. 0 IN A 10.0.0.1 78
  79. 79. Key-Value 79
  80. 80. http://consul:8500/v1/kv/config/application/property [ { "CreateIndex": 46430, "ModifyIndex": 94900, "LockIndex": 0, "Key": "config/application/property", "Flags": 0, "Value": "Y2hlY3s=" } ] 80
  81. 81. Consul Consul Consul API UI External Service Consul Client Registrator Nginx Consul Template Consul Client register Settings 81
  82. 82. Consul Consul Consul API UI Consul Client Registrator Nginx Consul Template Consul Client auto register Consul Consul Consul 82
  83. 83. Consul Consul Consul API UI Consul Client Registrator Nginx Consul Template Consul Client listen 83
  84. 84. Consul Consul Consul API UI External Service Consul Client Registrator Nginx Consul Template Consul Client update conf 84
  85. 85. {% raw %}{{range services}} upstream {{.Name}} { least_conn; {{range service .Name}} server {{.Address}}:{{.Port}} max_fails=3 fail_timeout=60 weight=1;{{else}}server localhost:60000;{{end}} } {{end}} server { server_name localhost; {{range services}} location /{{.Name}}/ { proxy_pass http://{{.Name}}/; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header Host $host; proxy_set_header X-Real-IP $remote_addr; } {{end}} }{% endraw %} 85
  86. 86. upstream api { least_conn; server server1:8080 max_fails=3 fail_timeout=60 weight=1; server server1:8081 max_fails=3 fail_timeout=60 weight=1; } server { server_name localhost; location /api/ { proxy_pass http://api/; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header Host $host; proxy_set_header X-Real-IP $remote_addr; } } 86
  87. 87. docker kill -s HUP nginx 87
  88. 88. Consul Consul Consul API UI External Service Consul Client Registrator Nginx Consul Template Consul Client dns 88
  89. 89. Consul Consul Consul API UI External Service Consul Client Registrator Nginx Consul Template Consul Client call 89
  90. 90. https://github.com/spring-cloud/spring-cloud-consul пока что только M1, используем только для KV 90
  91. 91. 91
  92. 92. etcd + confd 92
  93. 93. etcd + confd https://github.com/coreos/etcd/issues/964 93
  94. 94. Consul Consul Consul Consul Consul Consul DC1 DC2 join Consul Client Consul Client Consul Client Consul Client Consul Client Consul Client 94
  95. 95. Consul Consul Consul Consul Consul Consul DC1 DC2 Consul Client Consul Client Consul Client Consul Client Consul Client Consul Client force 95
  96. 96. Consul Consul Consul Consul Consul Consul DC1 DC2 Consul Client Consul Client Consul Client Consul Client Consul Client Consul Client KV 96
  97. 97. 97
  98. 98. Docker не решает проблемы конфигурации окружения 98
  99. 99. Через env-переменные docker run … -e "SERVICE_NAME=payments-api" … 99
  100. 100. Внешний конфигурационный файл /data/payments-api/conf/logback.groovy … logger('org.springframework', OFF) logger(‘ru.alfabank.api’, DEBUG) … docker run … -v /data/payments-api/conf:/conf … 100
  101. 101. Consul KV application.yml … protocol: http … /v1/kv/config/application/protocol /v1/kv/config/application,production/protocol /v1/kv/config/payments-api/protocol /v1/kv/config/payments-api,test/protocol 101
  102. 102. Consul DNS плохо держит нагрузку 102
  103. 103. Consul Consul Consul API UI External Service Consul Client Registrator Consul Client call Nginx Consul Template 103
  104. 104. Consul Consul Consul API UIExternal Service Consul Client Registrator Consul Client call Nginx Consul Template 104
  105. 105. Spring Consul Cloud Ribbon Zuul Netflix OSS 105
  106. 106. Настройка Consul DNS dns_config allow_stale + max_stale node_ttl / service_ttl { "dns_config": { "allow_stale" : true, "max_stale" : "5s" "service_ttl": { "*": "5s" } } } 106
  107. 107. Multicast 107
  108. 108. docker run -d -p 8080:8080 --name my-app --net=host docker.moscow.alfaintra.net/my-app 108
  109. 109. SELinux 109
  110. 110. - name: enforce selinux command: setenforce 1 - name: copy dockersock.pp for selinux copy: src=dockersock.pp dest=/root - name: execute dockersock.pp command: semodule -i /root/dockersock.pp - name: restart docker service service: name=docker state=restarted Решается правильной настройкой Policy 110
  111. 111. Registry v1 111
  112. 112. 112
  113. 113. Artifactory Pro 113
  114. 114. Борьба с логами 114
  115. 115. /var/lib/docker/containers/<id>/<id>-json.log растёт неконтролируемо :( 115
  116. 116. docker >= 1.6 docker run … --log-driver=none … docker logs … будет пустым 116
  117. 117. /var/lib/docker/containers/*/*.log { rotate {{ logrotate_count }} daily compress size={{ logrotate_size }} missingok copytruncate } logrotate.d 117
  118. 118. Можно смотреть логи через docker logs Один раз не спас :( 118
  119. 119. Memory leaks 119
  120. 120. 120
  121. 121. HDD Overhead 121
  122. 122. Просто купите больше HDD 122
  123. 123. Оптимизируйте образы RUN <command> && <command> && … Иерархия базовых образов Лёгкие образы ОС - alpine/busybox 123
  124. 124. Мониторинг 124
  125. 125. 125
  126. 126. https://github.com/docker/docker/issues/5684 126
  127. 127. docker stop cadvisor do something docker start cadvisor 127
  128. 128. 128
  129. 129. 129
  130. 130. Облачные сервисы хорошо, но мы не можем их использовать 130
  131. 131. docker run -d --restart=always -p {{service_port}}:8080 -e SERVICE_HOST={{ansible_ens192.ipv4.address}} -e SERVICE_ID={{ansible_hostname}}:{{service_name}} -e SERVICE_NAME={{image_name}} -e "SERVICE_CHECK_SCRIPT=curl {{ansible_ens192.ipv4.address}}:{{service_port}}/admin/health" -e SERVICE_CHECK_INTERVAL=5s -e SPRING_PROFILES_ACTIVE={{spring_profile}} -e "SPRING_BOOT_ADMIN_CLIENT_URL=http://{{ansible_ens192.ipv4.address}}:{{service_port}}/admin" -e SPRING_APPLICATION_NAME={{service_name}} —dns={{ansible_docker0.ipv4.address}} -e LOGGING_CONFIG=file:/data/logback/logback.groovy -e "TZ=Europe/Moscow" -v /logback:/data/logback -v /etc/localtime:/etc/localtime:ro -m={{container_memory_limit}} --name {{service_name}} {{private_registry}}/{{image_name}}:{{image_version}} 131
  132. 132. Уже очень непросто, правда? 132
  133. 133.  Ansible плохо масштабируется 133
  134. 134. Swarm - ещё очень сырой 134
  135. 135. Смотрим в сторону 135
  136. 136. 136
  137. 137. 137
  138. 138. Docker - … 138
  139. 139. Жизнеспособен 139
  140. 140. Решает много проблем, но не все 140
  141. 141. Добавляет вам новые проблемы 141
  142. 142. Спасибо! Ваши вопросы? @aatarasoff aatarasoff Тарасов Александр http://developerblog.info/ 142

×