Due to some design decision with ConcourseCI there is no sharing between JOB, so Object storage is a good way of achivig this. Minio is an OSS Object storage with a great coverage of S3 api.

1. 30/11/2016
Etourneau Gwenn
@The_shinji62
ConcourseCI
Minio

2. About me
Etourneau Gwenn
Sr Solution Architect
Bef.: Platform Architect
https://github.com/shinji62
https://twitter.com/the_shinji62

3. Pivotal
• Spring Framework
• RabbitMQ
• Concourse
• Cloud Foundry
• Pivotal Tracker
• Gemfire

4. Meetup

5. Agenda
• Concourse
• Minio
• Minio for Concourse
• Q&A

6. ConcourseCI

7. ConcourseCI
• Do not allow snowflake
• Isolation between job
• Non share volume / storage / cache

8. Jobs 1 Jobs 2
file_out file_out
X
Concourse

9. Concourse
• Concourse support S3
• S3 on premise is not easy
• RiackCS, leoFS, Swift are hard
• S3 compatibility so so…

10. Minio

11. Minio
• Open Source
• Object Storage written in Go
• Distributed ? Not really more like replication
• S3 compatible
• Data protection using Bit rots , Erasure Code
• Single go binary easy to start

12. Minio limitation
• Do not scale-out or scale back
• Not multi-tenant, user are the same for the same pool
• number of nodes =>4 and <= 16
• Still early phase (>5 commit /day on master)

13. Minio 1 disk by node

14. Minio Many disk by nodes

15. Minio and Concourse
• Support S3 native resource from concourse
Jobs 1
Jobs 2
file_out
file_out
file_out

16. Minio bosh release

17. Minio bosh release
• Deploy a pool of distributed minio in minutes
• On any IaaS (supported by bosh)

18. Minio bosh release
• Use bosh v2 links
• Easy to spin-off
• well, you need to learn bosh
a little….

19. Minio 1 disk by node

20. Demo

21. Demo
• Deploy Minio Cluster
• Set Concourse Pipeline

23. Resources
https://github.com/shinji62/minio-dist-boshrelease
https://github.com/cloudfoundry-community/minio-boshrelease
• Minio bosh release
• Minio
• https://github.com/minio/minio
• https://minio.io/

24. Resources
• ConcourseCI
• https://concourse.ci

25. Slack
https://concourseci.slack.com/
#general

26. Q & A

27. Thank You !!