Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Using Concourse in Production
- Lessons Learned -
Shingo Omura(@everpeace)
omura@chatwork.com
ChatWork, Inc.
Concourse Mee...
© ChatWork All rights reserved.© ChatWork All rights reserved.
Outline
● About ChatWork
● Our Context From the Point of Vi...
© ChatWork All rights reserved.
Group Chat File Sharing
Task Management Video Conference
About ChatWork ~Group Chat for Gl...
© ChatWork All rights reserved.
ChatWork is growing rapidly
● 127,000 organizations
○ number of users is not opened
● 205 ...
Our Context
From the Point of View
of Infrastructure
© ChatWork All rights reserved.
New Infrastracture Project (1/2)
● Current Infra
○ EC2 based apps, deploy servers(for capi...
© ChatWork All rights reserved.
New Infrastracture Project (2/2)
● Next Infra
○ Kubernetes and Helm with Dockerized apps
○...
Our Use Case
© ChatWork All rights reserved.
Overview of deployment system
● Concourse is deployed by concourse-aws
○ maintained by @mu...
© ChatWork All rights reserved.
Our build pipeline environment can be
split by ‘groups’
notification resource
10
© ChatWork All rights reserved.
Our build pipeline
test&build jobs
deploy jobs
rollback jobs
11
Good Parts Learned
© ChatWork All rights reserved.
Good Parts
● concourse.ci is extemely well-documented
○ You can start trying concourse in ...
© ChatWork All rights reserved.
Good Parts (cont.)
● Concourse frees us from ”plugin hell”
○ all resource is provided by d...
© ChatWork All rights reserved.
● easy to develop pipelines
○ Pipeline developed & tested in local env can be deployed dir...
© ChatWork All rights reserved.
Good Parts (cont.)
● easy to extend/custom
○ easy to develop custom resource.
■ you only n...
Pipeline tips Learned
© ChatWork All rights reserved.
Pipeline tips: summary
● Use groups for large pipeline
● Use aggregate for running in para...
© ChatWork All rights reserved.
Pipeline Tips
● Use groups for large pipeline to group many jobs
● Use aggregate for multi...
© ChatWork All rights reserved.
Pipeline Tips
● Use “[ci skip]” keyword to commit message when
Concourse commits/push to g...
© ChatWork All rights reserved.
● on_success/on_failure hook is useful for notification
Pipeline Tips
pipeline.yml
- task:...
© ChatWork All rights reserved.
● input_mapping/output_mapping is useful for shared
task definition
Pipeline Tips
pipeline...
© ChatWork All rights reserved.
● use attempts for deployment task due to intermittent
network failure
Pipeline Tips
pipel...
© ChatWork All rights reserved.
● @making’s trick is helpful for build caches(sbt, ivy, maven)
○ prepare own cache docker ...
Small Bad Parts
(expect to improve)
© ChatWork All rights reserved.
Small Bad Parts (expect to improve)
● No fine-grained authorization
(No role based aaccess...
© ChatWork All rights reserved.
Small Bad Parts (expect to improve)(cont.)
● No parameterized job
○ we would like to deplo...
© ChatWork All rights reserved.
Small Bad Parts (expect to improve)(cont.)
● No Docker Compose in task
○ the issue is now ...
Thank you for Listening!!
We’re Hiring!!!
Search “ChatWork” in Wantedly
https://www.wantedly.com/companies/chatwork/projects
Upcoming SlideShare
Loading in …5
×

Lessons Learned: Using Concourse In Production

5,317 views

Published on

Lessons Learned: Using Concourse In Production

Published in: Software
  • Hello! Get Your Professional Job-Winning Resume Here - Check our website! https://vk.cc/818RFv
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • Great presentation!
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here

Lessons Learned: Using Concourse In Production

  1. 1. Using Concourse in Production - Lessons Learned - Shingo Omura(@everpeace) omura@chatwork.com ChatWork, Inc. Concourse Meetup #5 2017/03/13
  2. 2. © ChatWork All rights reserved.© ChatWork All rights reserved. Outline ● About ChatWork ● Our Context From the Point of View of Infrastructure ● Our Use Case ● Good parts ● Pipeline Tips ● Small Bad parts (expect to improve) 2
  3. 3. © ChatWork All rights reserved. Group Chat File Sharing Task Management Video Conference About ChatWork ~Group Chat for Global Teams~ 3
  4. 4. © ChatWork All rights reserved. ChatWork is growing rapidly ● 127,000 organizations ○ number of users is not opened ● 205 countries or regions ● 6 languages supported as of 2017/02 4
  5. 5. Our Context From the Point of View of Infrastructure
  6. 6. © ChatWork All rights reserved. New Infrastracture Project (1/2) ● Current Infra ○ EC2 based apps, deploy servers(for capistorano) ○ Jenkins servers for CI/CD ● Pain points ○ Ops team doesn’t scale ■ release always have to be done with Infra team members ○ AWS env and Jenkins are hard to sandboxing ■ part of aws resouces are managed by terraform, but not all ■ deployment flow is hard to develop and testing 6
  7. 7. © ChatWork All rights reserved. New Infrastracture Project (2/2) ● Next Infra ○ Kubernetes and Helm with Dockerized apps ○ Concourse CI for CI/CD ● Benefits ○ Kubernetes accelarate DevOps ■ App team can fully manage their deployment cycle by themselves. ■ minikube is really helpful for local dev environemnt. ■ kubernetes team can focus on reliability of Kuberentes. ○ Concourse CI does too! ← Today’s Focus ■ reduces operational load ■ helps agile development of deployment/testing process ● Status ○ Using from new messaging backend (released the last december) ○ Current system is planned to migrate to this next infra 7
  8. 8. Our Use Case
  9. 9. © ChatWork All rights reserved. Overview of deployment system ● Concourse is deployed by concourse-aws ○ maintained by @mumoshu (my-colleague) and @everpeace (me) ● Branching model is Gitlab flow with Environment Branches ● chatwork-notify-resource for notification staging branch staging environment production environment master branch push im age build and deploy helm package build and deploy helm packagepush image pull image pull image notify 9
  10. 10. © ChatWork All rights reserved. Our build pipeline environment can be split by ‘groups’ notification resource 10
  11. 11. © ChatWork All rights reserved. Our build pipeline test&build jobs deploy jobs rollback jobs 11
  12. 12. Good Parts Learned
  13. 13. © ChatWork All rights reserved. Good Parts ● concourse.ci is extemely well-documented ○ You can start trying concourse in 5 min. ■ virtualbox and vagrant: just ‘vagrant up’!! ■ docker-compose support!! ○ easty to write pipelines thanks to comrehensive reference ● easy to deploy & version up (thanks to concourse-aws :-P ) ○ initial deploy: 3 steps ■ ‘build-amis.sh’ → edit ‘cluster.yml’ → ‘concourse-aws up’ ○ version up: similar 3 steps ■ ‘build-amis.sh’(new version) → edit ‘cluster.yml’(new ami) → ‘concourse-aws up’ 13
  14. 14. © ChatWork All rights reserved. Good Parts (cont.) ● Concourse frees us from ”plugin hell” ○ all resource is provided by docker image ○ task environment can be injected by docker image too ○ no need to manage backups of CI servers!! ● Multi tenancy ‘team’ support ■ multiple team can share CI server resources ■ but isolated appropriately ■ each app team can have controll in their team ● Various authentication scheme support ■ concourse need not to have user database ■ we use github authentication 14
  15. 15. © ChatWork All rights reserved. ● easy to develop pipelines ○ Pipeline developed & tested in local env can be deployed directly to production concourse ■ Concourse CI’s pipeline is stateless and reproductive ■ Concourse & Kubernetes both supports local env (minikube & concourse vagrant box) Good Parts (cont.) 15
  16. 16. © ChatWork All rights reserved. Good Parts (cont.) ● easy to extend/custom ○ easy to develop custom resource. ■ you only need to develop 3 commands(check, in, out) whose returns json objects. ■ language agnostic! you can choose your own language!! ○ easy to prepare task environment ■ when you need some task environment in which some toolkit is installed, you just push docker image to any repository and specify the image to your task definition task.yml --- platform: linux image_resource: type: docker-image source: repository: /yourown/image tag: '1.1' 16
  17. 17. Pipeline tips Learned
  18. 18. © ChatWork All rights reserved. Pipeline tips: summary ● Use groups for large pipeline ● Use aggregate for running in parallel (useful for resources) ● Use “[ci skip]” keyword to commit message when Concourse commits/push to git repo ● on_success/on_failure hook is useful for notification ● input_mapping/output_mapping is useful for shared task definition ● use attempts for deployment task due to intermittent network failure ● @making’s trick is helpful for build caches(sbt, ivy, maven) 18
  19. 19. © ChatWork All rights reserved. Pipeline Tips ● Use groups for large pipeline to group many jobs ● Use aggregate for multiple resources (useful for resources) pipeline.yml groups: - name: master jobs: - job-for-master - name: production jobs: - job-for-production pipeline.yml plan: - aggregate: - get: app-repo trigger: true - get: tool-repo - get: sbt-ivy-cache those 3 get runs in parallel 19
  20. 20. © ChatWork All rights reserved. Pipeline Tips ● Use “[ci skip]” keyword to commit message when Concourse commits/push to git repo ○ git resource skip commits with [ci skip] keywords ○ It’s really useful when ■ back merge: “merging release branch to develop branch” ● the commit is wanted to skip CI process ■ the commit bumping versions ● when using sbt, version number is embedded to repo 20
  21. 21. © ChatWork All rights reserved. ● on_success/on_failure hook is useful for notification Pipeline Tips pipeline.yml - task: deploy-write-api-to-dev-kube file: foo/task.yml on_success: task: chatwork-notification file: tasks/notify_chatwork.yml on_failure: task: chatwork-notification file: tasks/notify_chatwork.yml on_failure on_success 21
  22. 22. © ChatWork All rights reserved. ● input_mapping/output_mapping is useful for shared task definition Pipeline Tips pipeline.yml - task: test-pull-request file: pull-request/ci/tasks/unit.yml input_mapping: { repo: pull-request } - task: unit file: master/ci/tasks/unit.yml input_mapping: { repo: master } ci/tasks/unit.yml --- platform: linux image_resource: type: docker-image source: repository: yourown/toolbox inputs: - name: repo run: path: /bin/bash args: - repo/ci/tasks/unit.sh 22
  23. 23. © ChatWork All rights reserved. ● use attempts for deployment task due to intermittent network failure Pipeline Tips pipeline.yml ... - task: deploy-write-api-to-dev-kube file: ..snip../deploy-to-kube-helm.yml attempts: {{attempts}} attempts=3 23
  24. 24. © ChatWork All rights reserved. ● @making’s trick is helpful for build caches(sbt, ivy, maven) ○ prepare own cache docker image repo (anywhere) ○ archives cache files as rootfs.tar and push it directly to the image repo ○ related issue is now open: Caching directories between runs of a task #230 Pipeline Tips 24
  25. 25. Small Bad Parts (expect to improve)
  26. 26. © ChatWork All rights reserved. Small Bad Parts (expect to improve) ● No fine-grained authorization (No role based aaccess control) ○ every team member can take full controll in the team ○ ‘fly get-pipeline’ exposes all creadentials embedded in pipelines ○ We sometime want to split ■ people who can write/read pipeline ■ people who can just view logs and trigger jobs (no rights to change pipelines but can just operate the pipeline) ○ related issues are open ■ Credential management #19 ■ Individual/fine-grained access control #23 26
  27. 27. © ChatWork All rights reserved. Small Bad Parts (expect to improve)(cont.) ● No parameterized job ○ we would like to deploy specific feature branch to shared dev environment ○ How could do this with Concourse?? Any Idea?? ○ git-multibranch-resource could achive similar thing ■ branch name convention which will be deployed to shared dev env should be agreed ○ Perhaps `fly exec` prompts user input? 27
  28. 28. © ChatWork All rights reserved. Small Bad Parts (expect to improve)(cont.) ● No Docker Compose in task ○ the issue is now open: Docker Compose support in Task definitions #324 ■ integration test task with app & local db containers ● FYI: various improvements are disscued in https://github.com/concourse/design-notes/issues 28
  29. 29. Thank you for Listening!!
  30. 30. We’re Hiring!!! Search “ChatWork” in Wantedly https://www.wantedly.com/companies/chatwork/projects

×