SlideShare a Scribd company logo
1 of 27
Data(?)Ops with CircleCI
CircleCI Korea User Group 2nd Meetup
김진웅
About Me
김진웅 @ddiiwoong
Cloud Engineer @SK C&C
Interested in Kubernetes and Serverless(FaaS), Dev(Data)Ops, SRE, ML/DL
Today
• Data Lake, DataOps
• AWS Native CI/CD
• Why CircleCI?
• DataOps with CircleCI
• Summary
Big Data
https://blogs.gartner.com/doug-laney/big-datas-10-biggest-vision-and-strategy-questions/
https://kr.cloudera.com/products/open-source/apache-hadoop.html
Data Lake
• Centralized repository that allows you to store all your structured and
unstructured data at any scale.
• From dashboards and visualizations to big data processing, real-time
analytics, and machine learning to guide better decisions.
https://aws.amazon.com/ko/big-data/datalakes-
and-analytics/what-is-a-data-lake/
Data Lake Management
참고 - https://www.samsungsds.com/global/ko/support/insights/data_lake.html
What is DataOps
@Wikipedia
DataOps is an automated, process-oriented methodology, used by analytic
and data teams, to improve the quality and reduce the cycle time of data
analytics.
@The DataOps Manifesto
Data Science, Data Engineering, Data Management, Big Data,
Business Intelligence, or the like, through our work we have come to value in
analytics
@My point of view
데이터 중심 사고를 하는 사람들이 모여서 일을 하는 것
Dev + Ops + Data Engineer + Data Scientist
DataOps Principles
https://www.dataopsmanifesto.org/dataops-manifesto.html
1. Continually satisfy your customer - 지속적으로 고객을 만족시켜라
2. Value working analytics - 분석을 가치있게 생각하라
3. Embrace change - 변화 수용
4. It's a team sport - 다양한 역할, 기술, 도구 수용
5. Daily interactions - 매일 협력
6. Self-organize - 자기주도
7. Reduce heroism - 영웅주의를 줄여라
8. Reflect - 반성하라
9. Analytics is code - 분석은 코드다
10. Orchestrate - 결합하라
11. Make it reproducible - 재현 가능하게 만들어라
12. Disposable environments - 비용 최소화
13. Simplicity - 단순성
14. Analytics is manufacturing - 분석은 제조와 같다
15. Quality is paramount - 품질이 제일 중요
16. Monitor quality and performance - 품질 및 성능을 모니터링하라
17. Reuse - 재사용하라
18. Improve cycle times - 사이클 타임을 개선하라
DataOps Principles
• 애자일 방법론부터 시작
• 내/외부 고객 만족을 위한 끊임없는 분석 통찰력 제공
• 분석 성과를 측정하고, 변화를 추구하며, 변하는 고객 요구사항을 지속적으로 이
해해야함
DataOps 구성
• 목표를 중심으로 스스로 조직
• No Hero, Sustainable, Scalable, Process 지향
• Data, Tool, Code, Environment 모두 장악이 필요함
• Reproducible 결과물 -> 분석 Pipeline
• Cross-Functional Team(교차기능 팀)
• Dev, Architect, Ops, Data Science, Data Engineer 모두 포함
• 개발자, 운영자, 데이터전문가 (3자 협업 구도)
Our Project
Goal
• No-Ops : Remove existing management (Serverless)
• GitOps : All infra, codes, and scripts are managed in immutable state
• Automation : Communications, Approvals, SRs, Issues
Requirements
• Key Management
• IAM Role
• Access/SecretKey
• Code Repository
• Github, Bitbucket (Public Access) Account
• Code Commit
• CI
• CircleCI(Github Auth)
• Code Build
• Container Registry
• AWS ECR (Elastic Container Registry)
• Dockerhub
• CD
• Terraform, CircleCI
• CloudFormation, Code Deploy
• Notification
• Approval, SR, Issue, Collaboration
AWS Native CI/CD for Web Service
AWS Native CI/CD for Data Preparation
AWS Native CI/CD for Data Ingest
Code Repository
Github
• 다양한 Eco 3rd Party
• Private 사용 및 Collaboration 기능으로 비공개 Repo 활용
• GitOps
Container Registry
ECR
• Fully Managed
• Security (IAM) 연동
• CircleCI Orbs 제공
• EKS, ECS, Batch 연동 용이
CI/CD
Terraform
• CloudFormation : 직접 사용하는 도구보다는 백엔드로 활용하는 서비스
• 유지보수, 재사용, 모듈화 어려움
• 선언적 인프라스트럭처 관리 도구로 많이 사용하고 있는 도구
• 옵션,설정에 대한 관리 (State), 재사용, Dev-Ops 동시 확인 용이
• VPC, Security Group, IAM 관리 용이
CI/CD
CircleCI
• Fully Managed (Serverless)
• Caching, Debugging, Context
• AWS 종속성 최소화
• Git, Registry, CD영역의 확장성 고려
• AWS Console 접속 최소화
• 쉽고 단순하고 빠른 빌드 환경구성
• 소규모 프로젝트 빠르게 시작 가능
Portal Development Environment
Terraform Pipeline
Job flow
• 인프라 작업 및 IAM 계정작업
• master branch에서만 terraform apply
• CircleCI Version 2.1 기능 활용
• slack notification
• executors
• terraform plan 결과 저장 : persist_to_workspace
• terraform apply : attach_workspace
Checkout Lint Plan
Approval Apply
Master?
https://github.com/ddiiwoong/ecs-tf-template/blob/master/.circleci/config.yml
Batch Code Pipeline
Job flow
• Crawling을 위한 Batch Job(AWS ECS) Script update Pipeline 구성
• Docker build 및 AWS ECR(Registry)로 Script image Push
• AWS Batch Job Definition 변경 (Image Change)
참고 : https://ddii.dev/devops/circleci-ecs/
https://github.com/ddiiwoong/batch-cicd-demo/blob/master/.circleci/config.yml
Docker
Build & Push
S3 Upload
Slack Noti.
Approval
Deploy BatchCheckout
Portal Development Environment Pipeline
Job flow
• 서비스 Application Build Pipeline
• Docker build 및 AWS ECR(Registry)로 image Push
• image archive 및 Caching
• 특정 Tags 또는 branch에 대해서만 Build/Deploy
• Landing Page는 S3로 hosting (S3 upload)
참고 : https://yunsangjun.github.io/blog/cicd/2019/07/03/circleci.html
https://github.com/ddiiwoong/circleci-demo/blob/master/.circleci/config.yml
Build & Test
Archive &
Caching
Image Push ECS updateCheckout
S3(Dev)
Sync
Checkout Approval
S3(Prod)
Sync
ECS update
Static Hosting
In Progress
Glue Job Code Update
• ETL/ELT Job 처리를 위한 Python/Scala Code Update
• Scripts Location, Filename Update
Lambda Code Update
• https://github.com/ddiiwoong/serverless-example-monorepo-with-circleci/blob/master/.circleci/config.yml
EMR, Sagemaker Provisioning 설정 구성
• Bootstrap, Lifecycle Scripts
• Shutdown-actions
Multi-Cloud를 위한 준비
• Azure, IBM Data Pipeline Integration
• Data Migration을 위한 Crawler 구성
DevSecOps
• Scan images, Secrets Management
• https://circleci.com/integrations/devsecops/
Summary
Challenge
• Dev 동의 - Easy!
• 정보보호 동의 - Hard!!
• Secret Environment, Context (DevSecOps)
• https://circleci.com/blog/protect-secrets-with-restricted-contexts/
• Ops(TA) 동의 - Very Hard!!!
• CloudFormation vs Terraform
• S3 vs Git
• Role, Policy
Remember
• executor
• caching (persist_to_workspace)
• ECR immutable image tags
• https://aws.amazon.com/ko/about-aws/whats-new/2019/07/amazon-ecr-now-
supports-immutable-image-tags/
Q&A
@ddiiwoong
@ddiiwoong
ddiiwoong@gmail.com
https://ddii.dev

More Related Content

What's hot

Shakr - Container CI/CD with Google Cloud Platform
Shakr - Container CI/CD with Google Cloud PlatformShakr - Container CI/CD with Google Cloud Platform
Shakr - Container CI/CD with Google Cloud PlatformMinku Lee
 
Micro services infrastructure with AWS and Ansible
Micro services infrastructure with AWS and AnsibleMicro services infrastructure with AWS and Ansible
Micro services infrastructure with AWS and AnsibleBamdad Dashtban
 
"On-premises" FaaS on Kubernetes
"On-premises" FaaS on Kubernetes"On-premises" FaaS on Kubernetes
"On-premises" FaaS on KubernetesAlex Casalboni
 
Distributed Tensorflow with Kubernetes - data2day - Jakob Karalus
Distributed Tensorflow with Kubernetes - data2day - Jakob KaralusDistributed Tensorflow with Kubernetes - data2day - Jakob Karalus
Distributed Tensorflow with Kubernetes - data2day - Jakob KaralusJakob Karalus
 
Openstack Study Nova 1
Openstack Study Nova 1Openstack Study Nova 1
Openstack Study Nova 1Jinho Shin
 
Kubernetes Day 2017 - Build, Ship and Run Your APP, Production !!
Kubernetes Day 2017 - Build, Ship and Run Your APP, Production !!Kubernetes Day 2017 - Build, Ship and Run Your APP, Production !!
Kubernetes Day 2017 - Build, Ship and Run Your APP, Production !!smalltown
 
[OpenStack Days Korea 2016] Track4 - Deep Drive: k8s with Docker
[OpenStack Days Korea 2016] Track4 - Deep Drive: k8s with Docker[OpenStack Days Korea 2016] Track4 - Deep Drive: k8s with Docker
[OpenStack Days Korea 2016] Track4 - Deep Drive: k8s with DockerOpenStack Korea Community
 
[OpenInfra Days Korea 2018] Day 2 - E3-2: "핸즈온 워크샵: Kubespray, Helm, Armada를 ...
[OpenInfra Days Korea 2018] Day 2 - E3-2: "핸즈온 워크샵: Kubespray, Helm, Armada를 ...[OpenInfra Days Korea 2018] Day 2 - E3-2: "핸즈온 워크샵: Kubespray, Helm, Armada를 ...
[OpenInfra Days Korea 2018] Day 2 - E3-2: "핸즈온 워크샵: Kubespray, Helm, Armada를 ...OpenStack Korea Community
 
CDK Meetup: Rule the World through IaC
CDK Meetup: Rule the World through IaCCDK Meetup: Rule the World through IaC
CDK Meetup: Rule the World through IaCsmalltown
 
Autoscaling Kubernetes
Autoscaling KubernetesAutoscaling Kubernetes
Autoscaling Kubernetescraigbox
 
The Kubernetes Operator Pattern - ContainerConf Nov 2017
The Kubernetes Operator Pattern - ContainerConf Nov 2017The Kubernetes Operator Pattern - ContainerConf Nov 2017
The Kubernetes Operator Pattern - ContainerConf Nov 2017Jakob Karalus
 
Kubernetes day 2_jozef_halgas_pf
Kubernetes day 2_jozef_halgas_pfKubernetes day 2_jozef_halgas_pf
Kubernetes day 2_jozef_halgas_pfJuraj Hantak
 
Containers and CloudStack
Containers and CloudStackContainers and CloudStack
Containers and CloudStackShapeBlue
 
Serverless technologies with Kubernetes
Serverless technologies with KubernetesServerless technologies with Kubernetes
Serverless technologies with KubernetesProvectus
 
KubeCon EU 2016 Keynote: Pushing Kubernetes Forward
KubeCon EU 2016 Keynote: Pushing Kubernetes ForwardKubeCon EU 2016 Keynote: Pushing Kubernetes Forward
KubeCon EU 2016 Keynote: Pushing Kubernetes ForwardKubeAcademy
 
[GS네오텍] Google Kubernetes Engine
[GS네오텍]  Google Kubernetes Engine [GS네오텍]  Google Kubernetes Engine
[GS네오텍] Google Kubernetes Engine GS Neotek
 
Kubernetes Meetup: CNI, Flex Volume, and Scheduler
Kubernetes Meetup: CNI, Flex Volume, and SchedulerKubernetes Meetup: CNI, Flex Volume, and Scheduler
Kubernetes Meetup: CNI, Flex Volume, and SchedulerKatie Crimi
 

What's hot (19)

Shakr - Container CI/CD with Google Cloud Platform
Shakr - Container CI/CD with Google Cloud PlatformShakr - Container CI/CD with Google Cloud Platform
Shakr - Container CI/CD with Google Cloud Platform
 
Micro services infrastructure with AWS and Ansible
Micro services infrastructure with AWS and AnsibleMicro services infrastructure with AWS and Ansible
Micro services infrastructure with AWS and Ansible
 
Intro to kubernetes
Intro to kubernetesIntro to kubernetes
Intro to kubernetes
 
"On-premises" FaaS on Kubernetes
"On-premises" FaaS on Kubernetes"On-premises" FaaS on Kubernetes
"On-premises" FaaS on Kubernetes
 
Distributed Tensorflow with Kubernetes - data2day - Jakob Karalus
Distributed Tensorflow with Kubernetes - data2day - Jakob KaralusDistributed Tensorflow with Kubernetes - data2day - Jakob Karalus
Distributed Tensorflow with Kubernetes - data2day - Jakob Karalus
 
Openstack Study Nova 1
Openstack Study Nova 1Openstack Study Nova 1
Openstack Study Nova 1
 
Kubernetes Day 2017 - Build, Ship and Run Your APP, Production !!
Kubernetes Day 2017 - Build, Ship and Run Your APP, Production !!Kubernetes Day 2017 - Build, Ship and Run Your APP, Production !!
Kubernetes Day 2017 - Build, Ship and Run Your APP, Production !!
 
[OpenStack Days Korea 2016] Track4 - Deep Drive: k8s with Docker
[OpenStack Days Korea 2016] Track4 - Deep Drive: k8s with Docker[OpenStack Days Korea 2016] Track4 - Deep Drive: k8s with Docker
[OpenStack Days Korea 2016] Track4 - Deep Drive: k8s with Docker
 
[OpenInfra Days Korea 2018] Day 2 - E3-2: "핸즈온 워크샵: Kubespray, Helm, Armada를 ...
[OpenInfra Days Korea 2018] Day 2 - E3-2: "핸즈온 워크샵: Kubespray, Helm, Armada를 ...[OpenInfra Days Korea 2018] Day 2 - E3-2: "핸즈온 워크샵: Kubespray, Helm, Armada를 ...
[OpenInfra Days Korea 2018] Day 2 - E3-2: "핸즈온 워크샵: Kubespray, Helm, Armada를 ...
 
CDK Meetup: Rule the World through IaC
CDK Meetup: Rule the World through IaCCDK Meetup: Rule the World through IaC
CDK Meetup: Rule the World through IaC
 
Autoscaling Kubernetes
Autoscaling KubernetesAutoscaling Kubernetes
Autoscaling Kubernetes
 
The Kubernetes Operator Pattern - ContainerConf Nov 2017
The Kubernetes Operator Pattern - ContainerConf Nov 2017The Kubernetes Operator Pattern - ContainerConf Nov 2017
The Kubernetes Operator Pattern - ContainerConf Nov 2017
 
Kubernetes day 2_jozef_halgas_pf
Kubernetes day 2_jozef_halgas_pfKubernetes day 2_jozef_halgas_pf
Kubernetes day 2_jozef_halgas_pf
 
Containers and CloudStack
Containers and CloudStackContainers and CloudStack
Containers and CloudStack
 
Serverless technologies with Kubernetes
Serverless technologies with KubernetesServerless technologies with Kubernetes
Serverless technologies with Kubernetes
 
Cloud data center and openstack
Cloud data center and openstackCloud data center and openstack
Cloud data center and openstack
 
KubeCon EU 2016 Keynote: Pushing Kubernetes Forward
KubeCon EU 2016 Keynote: Pushing Kubernetes ForwardKubeCon EU 2016 Keynote: Pushing Kubernetes Forward
KubeCon EU 2016 Keynote: Pushing Kubernetes Forward
 
[GS네오텍] Google Kubernetes Engine
[GS네오텍]  Google Kubernetes Engine [GS네오텍]  Google Kubernetes Engine
[GS네오텍] Google Kubernetes Engine
 
Kubernetes Meetup: CNI, Flex Volume, and Scheduler
Kubernetes Meetup: CNI, Flex Volume, and SchedulerKubernetes Meetup: CNI, Flex Volume, and Scheduler
Kubernetes Meetup: CNI, Flex Volume, and Scheduler
 

Similar to Data(?)Ops with CircleCI

Tokyo Azure Meetup #7 - Introduction to Serverless Architectures with Azure F...
Tokyo Azure Meetup #7 - Introduction to Serverless Architectures with Azure F...Tokyo Azure Meetup #7 - Introduction to Serverless Architectures with Azure F...
Tokyo Azure Meetup #7 - Introduction to Serverless Architectures with Azure F...Tokyo Azure Meetup
 
DevOps on Oracle Cloud
DevOps on Oracle CloudDevOps on Oracle Cloud
DevOps on Oracle CloudMee Nam Lee
 
Business and IT agility through DevOps and microservice architecture powered ...
Business and IT agility through DevOps and microservice architecture powered ...Business and IT agility through DevOps and microservice architecture powered ...
Business and IT agility through DevOps and microservice architecture powered ...Lucas Jellema
 
Dutch Oracle Architects Platform - Reviewing Oracle OpenWorld 2017 and New Tr...
Dutch Oracle Architects Platform - Reviewing Oracle OpenWorld 2017 and New Tr...Dutch Oracle Architects Platform - Reviewing Oracle OpenWorld 2017 and New Tr...
Dutch Oracle Architects Platform - Reviewing Oracle OpenWorld 2017 and New Tr...Lucas Jellema
 
Stateful Interaction In Serverless Architecture With Redis: Pyounguk Cho
Stateful Interaction In Serverless Architecture With Redis: Pyounguk ChoStateful Interaction In Serverless Architecture With Redis: Pyounguk Cho
Stateful Interaction In Serverless Architecture With Redis: Pyounguk ChoRedis Labs
 
Code for Startup MVP (Ruby on Rails) Session 1
Code for Startup MVP (Ruby on Rails) Session 1Code for Startup MVP (Ruby on Rails) Session 1
Code for Startup MVP (Ruby on Rails) Session 1Henry S
 
Measure and increase developer productivity with help of Severless by Kazulki...
Measure and increase developer productivity with help of Severless by Kazulki...Measure and increase developer productivity with help of Severless by Kazulki...
Measure and increase developer productivity with help of Severless by Kazulki...Vadym Kazulkin
 
Re-Platforming Applications for the Cloud
Re-Platforming Applications for the CloudRe-Platforming Applications for the Cloud
Re-Platforming Applications for the CloudCarter Wickstrom
 
Accelerating analytics in the cloud with the Starburst Presto + Alluxio stack
Accelerating analytics in the cloud with the Starburst Presto + Alluxio stackAccelerating analytics in the cloud with the Starburst Presto + Alluxio stack
Accelerating analytics in the cloud with the Starburst Presto + Alluxio stackAlluxio, Inc.
 
VMworld 2013: Virtualizing Databases: Doing IT Right
VMworld 2013: Virtualizing Databases: Doing IT Right VMworld 2013: Virtualizing Databases: Doing IT Right
VMworld 2013: Virtualizing Databases: Doing IT Right VMworld
 
Geek Sync | Deployment and Management of Complex Azure Environments
Geek Sync | Deployment and Management of Complex Azure EnvironmentsGeek Sync | Deployment and Management of Complex Azure Environments
Geek Sync | Deployment and Management of Complex Azure EnvironmentsIDERA Software
 
Accelerating Digital Transformation: It's About Digital Enablement
Accelerating Digital Transformation:  It's About Digital EnablementAccelerating Digital Transformation:  It's About Digital Enablement
Accelerating Digital Transformation: It's About Digital EnablementJoshua Gossett
 
2014.10.22 Building Azure Solutions with Office 365
2014.10.22 Building Azure Solutions with Office 3652014.10.22 Building Azure Solutions with Office 365
2014.10.22 Building Azure Solutions with Office 365Marco Parenzan
 
SharePoint Connections Conference Amsterdam - Pitfalls and success factors of...
SharePoint Connections Conference Amsterdam - Pitfalls and success factors of...SharePoint Connections Conference Amsterdam - Pitfalls and success factors of...
SharePoint Connections Conference Amsterdam - Pitfalls and success factors of...Wilco Turnhout
 
PostgreSQL as a Strategic Tool
PostgreSQL as a Strategic ToolPostgreSQL as a Strategic Tool
PostgreSQL as a Strategic ToolEDB
 
Migrating on premises workload to azure sql database
Migrating on premises workload to azure sql databaseMigrating on premises workload to azure sql database
Migrating on premises workload to azure sql databasePARIKSHIT SAVJANI
 
Modern MySQL Monitoring and Dashboards.
Modern MySQL Monitoring and Dashboards.Modern MySQL Monitoring and Dashboards.
Modern MySQL Monitoring and Dashboards.Mydbops
 

Similar to Data(?)Ops with CircleCI (20)

Tokyo Azure Meetup #7 - Introduction to Serverless Architectures with Azure F...
Tokyo Azure Meetup #7 - Introduction to Serverless Architectures with Azure F...Tokyo Azure Meetup #7 - Introduction to Serverless Architectures with Azure F...
Tokyo Azure Meetup #7 - Introduction to Serverless Architectures with Azure F...
 
楽天が挑むDevOps
楽天が挑むDevOps楽天が挑むDevOps
楽天が挑むDevOps
 
DevOps on Oracle Cloud
DevOps on Oracle CloudDevOps on Oracle Cloud
DevOps on Oracle Cloud
 
Business and IT agility through DevOps and microservice architecture powered ...
Business and IT agility through DevOps and microservice architecture powered ...Business and IT agility through DevOps and microservice architecture powered ...
Business and IT agility through DevOps and microservice architecture powered ...
 
Dutch Oracle Architects Platform - Reviewing Oracle OpenWorld 2017 and New Tr...
Dutch Oracle Architects Platform - Reviewing Oracle OpenWorld 2017 and New Tr...Dutch Oracle Architects Platform - Reviewing Oracle OpenWorld 2017 and New Tr...
Dutch Oracle Architects Platform - Reviewing Oracle OpenWorld 2017 and New Tr...
 
Power of Azure Devops
Power of Azure DevopsPower of Azure Devops
Power of Azure Devops
 
Stateful Interaction In Serverless Architecture With Redis: Pyounguk Cho
Stateful Interaction In Serverless Architecture With Redis: Pyounguk ChoStateful Interaction In Serverless Architecture With Redis: Pyounguk Cho
Stateful Interaction In Serverless Architecture With Redis: Pyounguk Cho
 
Code for Startup MVP (Ruby on Rails) Session 1
Code for Startup MVP (Ruby on Rails) Session 1Code for Startup MVP (Ruby on Rails) Session 1
Code for Startup MVP (Ruby on Rails) Session 1
 
Measure and increase developer productivity with help of Severless by Kazulki...
Measure and increase developer productivity with help of Severless by Kazulki...Measure and increase developer productivity with help of Severless by Kazulki...
Measure and increase developer productivity with help of Severless by Kazulki...
 
Re-Platforming Applications for the Cloud
Re-Platforming Applications for the CloudRe-Platforming Applications for the Cloud
Re-Platforming Applications for the Cloud
 
Accelerating analytics in the cloud with the Starburst Presto + Alluxio stack
Accelerating analytics in the cloud with the Starburst Presto + Alluxio stackAccelerating analytics in the cloud with the Starburst Presto + Alluxio stack
Accelerating analytics in the cloud with the Starburst Presto + Alluxio stack
 
Elastic-Engineering
Elastic-EngineeringElastic-Engineering
Elastic-Engineering
 
VMworld 2013: Virtualizing Databases: Doing IT Right
VMworld 2013: Virtualizing Databases: Doing IT Right VMworld 2013: Virtualizing Databases: Doing IT Right
VMworld 2013: Virtualizing Databases: Doing IT Right
 
Geek Sync | Deployment and Management of Complex Azure Environments
Geek Sync | Deployment and Management of Complex Azure EnvironmentsGeek Sync | Deployment and Management of Complex Azure Environments
Geek Sync | Deployment and Management of Complex Azure Environments
 
Accelerating Digital Transformation: It's About Digital Enablement
Accelerating Digital Transformation:  It's About Digital EnablementAccelerating Digital Transformation:  It's About Digital Enablement
Accelerating Digital Transformation: It's About Digital Enablement
 
2014.10.22 Building Azure Solutions with Office 365
2014.10.22 Building Azure Solutions with Office 3652014.10.22 Building Azure Solutions with Office 365
2014.10.22 Building Azure Solutions with Office 365
 
SharePoint Connections Conference Amsterdam - Pitfalls and success factors of...
SharePoint Connections Conference Amsterdam - Pitfalls and success factors of...SharePoint Connections Conference Amsterdam - Pitfalls and success factors of...
SharePoint Connections Conference Amsterdam - Pitfalls and success factors of...
 
PostgreSQL as a Strategic Tool
PostgreSQL as a Strategic ToolPostgreSQL as a Strategic Tool
PostgreSQL as a Strategic Tool
 
Migrating on premises workload to azure sql database
Migrating on premises workload to azure sql databaseMigrating on premises workload to azure sql database
Migrating on premises workload to azure sql database
 
Modern MySQL Monitoring and Dashboards.
Modern MySQL Monitoring and Dashboards.Modern MySQL Monitoring and Dashboards.
Modern MySQL Monitoring and Dashboards.
 

More from Jinwoong Kim

Prometheus Project Journey
Prometheus Project JourneyPrometheus Project Journey
Prometheus Project JourneyJinwoong Kim
 
AWS기반 서버리스 데이터레이크 구축하기 - 김진웅 (SK C&C) :: AWS Community Day 2020
AWS기반 서버리스 데이터레이크 구축하기 - 김진웅 (SK C&C) :: AWS Community Day 2020AWS기반 서버리스 데이터레이크 구축하기 - 김진웅 (SK C&C) :: AWS Community Day 2020
AWS기반 서버리스 데이터레이크 구축하기 - 김진웅 (SK C&C) :: AWS Community Day 2020Jinwoong Kim
 
Knative로 서버리스 워크로드 구현
Knative로 서버리스 워크로드 구현Knative로 서버리스 워크로드 구현
Knative로 서버리스 워크로드 구현Jinwoong Kim
 
EKS workshop 살펴보기
EKS workshop 살펴보기EKS workshop 살펴보기
EKS workshop 살펴보기Jinwoong Kim
 
Spinnaker on Kubernetes
Spinnaker on KubernetesSpinnaker on Kubernetes
Spinnaker on KubernetesJinwoong Kim
 
Cloud Z 의 오픈소스 서비스 소개 및 Serverless로 게임 개발하기
Cloud Z 의 오픈소스 서비스 소개 및 Serverless로 게임 개발하기Cloud Z 의 오픈소스 서비스 소개 및 Serverless로 게임 개발하기
Cloud Z 의 오픈소스 서비스 소개 및 Serverless로 게임 개발하기Jinwoong Kim
 
Continuous Delivery with Spinnaker on K8s(kubernetes) Cluster
Continuous Delivery with Spinnaker on K8s(kubernetes) Cluster Continuous Delivery with Spinnaker on K8s(kubernetes) Cluster
Continuous Delivery with Spinnaker on K8s(kubernetes) Cluster Jinwoong Kim
 
Provisioning Dedicated Game Server on Kubernetes Cluster
Provisioning Dedicated Game Server on Kubernetes ClusterProvisioning Dedicated Game Server on Kubernetes Cluster
Provisioning Dedicated Game Server on Kubernetes ClusterJinwoong Kim
 

More from Jinwoong Kim (8)

Prometheus Project Journey
Prometheus Project JourneyPrometheus Project Journey
Prometheus Project Journey
 
AWS기반 서버리스 데이터레이크 구축하기 - 김진웅 (SK C&C) :: AWS Community Day 2020
AWS기반 서버리스 데이터레이크 구축하기 - 김진웅 (SK C&C) :: AWS Community Day 2020AWS기반 서버리스 데이터레이크 구축하기 - 김진웅 (SK C&C) :: AWS Community Day 2020
AWS기반 서버리스 데이터레이크 구축하기 - 김진웅 (SK C&C) :: AWS Community Day 2020
 
Knative로 서버리스 워크로드 구현
Knative로 서버리스 워크로드 구현Knative로 서버리스 워크로드 구현
Knative로 서버리스 워크로드 구현
 
EKS workshop 살펴보기
EKS workshop 살펴보기EKS workshop 살펴보기
EKS workshop 살펴보기
 
Spinnaker on Kubernetes
Spinnaker on KubernetesSpinnaker on Kubernetes
Spinnaker on Kubernetes
 
Cloud Z 의 오픈소스 서비스 소개 및 Serverless로 게임 개발하기
Cloud Z 의 오픈소스 서비스 소개 및 Serverless로 게임 개발하기Cloud Z 의 오픈소스 서비스 소개 및 Serverless로 게임 개발하기
Cloud Z 의 오픈소스 서비스 소개 및 Serverless로 게임 개발하기
 
Continuous Delivery with Spinnaker on K8s(kubernetes) Cluster
Continuous Delivery with Spinnaker on K8s(kubernetes) Cluster Continuous Delivery with Spinnaker on K8s(kubernetes) Cluster
Continuous Delivery with Spinnaker on K8s(kubernetes) Cluster
 
Provisioning Dedicated Game Server on Kubernetes Cluster
Provisioning Dedicated Game Server on Kubernetes ClusterProvisioning Dedicated Game Server on Kubernetes Cluster
Provisioning Dedicated Game Server on Kubernetes Cluster
 

Recently uploaded

So einfach geht modernes Roaming fuer Notes und Nomad.pdf
So einfach geht modernes Roaming fuer Notes und Nomad.pdfSo einfach geht modernes Roaming fuer Notes und Nomad.pdf
So einfach geht modernes Roaming fuer Notes und Nomad.pdfpanagenda
 
SALESFORCE EDUCATION CLOUD | FEXLE SERVICES
SALESFORCE EDUCATION CLOUD | FEXLE SERVICESSALESFORCE EDUCATION CLOUD | FEXLE SERVICES
SALESFORCE EDUCATION CLOUD | FEXLE SERVICESmohitsingh558521
 
New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024
New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024
New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024BookNet Canada
 
New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024
New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024
New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024BookNet Canada
 
Anypoint Exchange: It’s Not Just a Repo!
Anypoint Exchange: It’s Not Just a Repo!Anypoint Exchange: It’s Not Just a Repo!
Anypoint Exchange: It’s Not Just a Repo!Manik S Magar
 
DevEX - reference for building teams, processes, and platforms
DevEX - reference for building teams, processes, and platformsDevEX - reference for building teams, processes, and platforms
DevEX - reference for building teams, processes, and platformsSergiu Bodiu
 
[Webinar] SpiraTest - Setting New Standards in Quality Assurance
[Webinar] SpiraTest - Setting New Standards in Quality Assurance[Webinar] SpiraTest - Setting New Standards in Quality Assurance
[Webinar] SpiraTest - Setting New Standards in Quality AssuranceInflectra
 
Gen AI in Business - Global Trends Report 2024.pdf
Gen AI in Business - Global Trends Report 2024.pdfGen AI in Business - Global Trends Report 2024.pdf
Gen AI in Business - Global Trends Report 2024.pdfAddepto
 
The Fit for Passkeys for Employee and Consumer Sign-ins: FIDO Paris Seminar.pptx
The Fit for Passkeys for Employee and Consumer Sign-ins: FIDO Paris Seminar.pptxThe Fit for Passkeys for Employee and Consumer Sign-ins: FIDO Paris Seminar.pptx
The Fit for Passkeys for Employee and Consumer Sign-ins: FIDO Paris Seminar.pptxLoriGlavin3
 
Transcript: New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024
Transcript: New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024Transcript: New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024
Transcript: New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024BookNet Canada
 
Visualising and forecasting stocks using Dash
Visualising and forecasting stocks using DashVisualising and forecasting stocks using Dash
Visualising and forecasting stocks using Dashnarutouzumaki53779
 
How to write a Business Continuity Plan
How to write a Business Continuity PlanHow to write a Business Continuity Plan
How to write a Business Continuity PlanDatabarracks
 
Rise of the Machines: Known As Drones...
Rise of the Machines: Known As Drones...Rise of the Machines: Known As Drones...
Rise of the Machines: Known As Drones...Rick Flair
 
"ML in Production",Oleksandr Bagan
"ML in Production",Oleksandr Bagan"ML in Production",Oleksandr Bagan
"ML in Production",Oleksandr BaganFwdays
 
Enhancing User Experience - Exploring the Latest Features of Tallyman Axis Lo...
Enhancing User Experience - Exploring the Latest Features of Tallyman Axis Lo...Enhancing User Experience - Exploring the Latest Features of Tallyman Axis Lo...
Enhancing User Experience - Exploring the Latest Features of Tallyman Axis Lo...Scott Andery
 
Tampa BSides - Chef's Tour of Microsoft Security Adoption Framework (SAF)
Tampa BSides - Chef's Tour of Microsoft Security Adoption Framework (SAF)Tampa BSides - Chef's Tour of Microsoft Security Adoption Framework (SAF)
Tampa BSides - Chef's Tour of Microsoft Security Adoption Framework (SAF)Mark Simos
 
DevoxxFR 2024 Reproducible Builds with Apache Maven
DevoxxFR 2024 Reproducible Builds with Apache MavenDevoxxFR 2024 Reproducible Builds with Apache Maven
DevoxxFR 2024 Reproducible Builds with Apache MavenHervé Boutemy
 
Digital Identity is Under Attack: FIDO Paris Seminar.pptx
Digital Identity is Under Attack: FIDO Paris Seminar.pptxDigital Identity is Under Attack: FIDO Paris Seminar.pptx
Digital Identity is Under Attack: FIDO Paris Seminar.pptxLoriGlavin3
 
(How to Program) Paul Deitel, Harvey Deitel-Java How to Program, Early Object...
(How to Program) Paul Deitel, Harvey Deitel-Java How to Program, Early Object...(How to Program) Paul Deitel, Harvey Deitel-Java How to Program, Early Object...
(How to Program) Paul Deitel, Harvey Deitel-Java How to Program, Early Object...AliaaTarek5
 
Merck Moving Beyond Passwords: FIDO Paris Seminar.pptx
Merck Moving Beyond Passwords: FIDO Paris Seminar.pptxMerck Moving Beyond Passwords: FIDO Paris Seminar.pptx
Merck Moving Beyond Passwords: FIDO Paris Seminar.pptxLoriGlavin3
 

Recently uploaded (20)

So einfach geht modernes Roaming fuer Notes und Nomad.pdf
So einfach geht modernes Roaming fuer Notes und Nomad.pdfSo einfach geht modernes Roaming fuer Notes und Nomad.pdf
So einfach geht modernes Roaming fuer Notes und Nomad.pdf
 
SALESFORCE EDUCATION CLOUD | FEXLE SERVICES
SALESFORCE EDUCATION CLOUD | FEXLE SERVICESSALESFORCE EDUCATION CLOUD | FEXLE SERVICES
SALESFORCE EDUCATION CLOUD | FEXLE SERVICES
 
New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024
New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024
New from BookNet Canada for 2024: BNC CataList - Tech Forum 2024
 
New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024
New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024
New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024
 
Anypoint Exchange: It’s Not Just a Repo!
Anypoint Exchange: It’s Not Just a Repo!Anypoint Exchange: It’s Not Just a Repo!
Anypoint Exchange: It’s Not Just a Repo!
 
DevEX - reference for building teams, processes, and platforms
DevEX - reference for building teams, processes, and platformsDevEX - reference for building teams, processes, and platforms
DevEX - reference for building teams, processes, and platforms
 
[Webinar] SpiraTest - Setting New Standards in Quality Assurance
[Webinar] SpiraTest - Setting New Standards in Quality Assurance[Webinar] SpiraTest - Setting New Standards in Quality Assurance
[Webinar] SpiraTest - Setting New Standards in Quality Assurance
 
Gen AI in Business - Global Trends Report 2024.pdf
Gen AI in Business - Global Trends Report 2024.pdfGen AI in Business - Global Trends Report 2024.pdf
Gen AI in Business - Global Trends Report 2024.pdf
 
The Fit for Passkeys for Employee and Consumer Sign-ins: FIDO Paris Seminar.pptx
The Fit for Passkeys for Employee and Consumer Sign-ins: FIDO Paris Seminar.pptxThe Fit for Passkeys for Employee and Consumer Sign-ins: FIDO Paris Seminar.pptx
The Fit for Passkeys for Employee and Consumer Sign-ins: FIDO Paris Seminar.pptx
 
Transcript: New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024
Transcript: New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024Transcript: New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024
Transcript: New from BookNet Canada for 2024: Loan Stars - Tech Forum 2024
 
Visualising and forecasting stocks using Dash
Visualising and forecasting stocks using DashVisualising and forecasting stocks using Dash
Visualising and forecasting stocks using Dash
 
How to write a Business Continuity Plan
How to write a Business Continuity PlanHow to write a Business Continuity Plan
How to write a Business Continuity Plan
 
Rise of the Machines: Known As Drones...
Rise of the Machines: Known As Drones...Rise of the Machines: Known As Drones...
Rise of the Machines: Known As Drones...
 
"ML in Production",Oleksandr Bagan
"ML in Production",Oleksandr Bagan"ML in Production",Oleksandr Bagan
"ML in Production",Oleksandr Bagan
 
Enhancing User Experience - Exploring the Latest Features of Tallyman Axis Lo...
Enhancing User Experience - Exploring the Latest Features of Tallyman Axis Lo...Enhancing User Experience - Exploring the Latest Features of Tallyman Axis Lo...
Enhancing User Experience - Exploring the Latest Features of Tallyman Axis Lo...
 
Tampa BSides - Chef's Tour of Microsoft Security Adoption Framework (SAF)
Tampa BSides - Chef's Tour of Microsoft Security Adoption Framework (SAF)Tampa BSides - Chef's Tour of Microsoft Security Adoption Framework (SAF)
Tampa BSides - Chef's Tour of Microsoft Security Adoption Framework (SAF)
 
DevoxxFR 2024 Reproducible Builds with Apache Maven
DevoxxFR 2024 Reproducible Builds with Apache MavenDevoxxFR 2024 Reproducible Builds with Apache Maven
DevoxxFR 2024 Reproducible Builds with Apache Maven
 
Digital Identity is Under Attack: FIDO Paris Seminar.pptx
Digital Identity is Under Attack: FIDO Paris Seminar.pptxDigital Identity is Under Attack: FIDO Paris Seminar.pptx
Digital Identity is Under Attack: FIDO Paris Seminar.pptx
 
(How to Program) Paul Deitel, Harvey Deitel-Java How to Program, Early Object...
(How to Program) Paul Deitel, Harvey Deitel-Java How to Program, Early Object...(How to Program) Paul Deitel, Harvey Deitel-Java How to Program, Early Object...
(How to Program) Paul Deitel, Harvey Deitel-Java How to Program, Early Object...
 
Merck Moving Beyond Passwords: FIDO Paris Seminar.pptx
Merck Moving Beyond Passwords: FIDO Paris Seminar.pptxMerck Moving Beyond Passwords: FIDO Paris Seminar.pptx
Merck Moving Beyond Passwords: FIDO Paris Seminar.pptx
 

Data(?)Ops with CircleCI

  • 1. Data(?)Ops with CircleCI CircleCI Korea User Group 2nd Meetup 김진웅
  • 2. About Me 김진웅 @ddiiwoong Cloud Engineer @SK C&C Interested in Kubernetes and Serverless(FaaS), Dev(Data)Ops, SRE, ML/DL
  • 3. Today • Data Lake, DataOps • AWS Native CI/CD • Why CircleCI? • DataOps with CircleCI • Summary
  • 5. Data Lake • Centralized repository that allows you to store all your structured and unstructured data at any scale. • From dashboards and visualizations to big data processing, real-time analytics, and machine learning to guide better decisions. https://aws.amazon.com/ko/big-data/datalakes- and-analytics/what-is-a-data-lake/
  • 6. Data Lake Management 참고 - https://www.samsungsds.com/global/ko/support/insights/data_lake.html
  • 7. What is DataOps @Wikipedia DataOps is an automated, process-oriented methodology, used by analytic and data teams, to improve the quality and reduce the cycle time of data analytics. @The DataOps Manifesto Data Science, Data Engineering, Data Management, Big Data, Business Intelligence, or the like, through our work we have come to value in analytics @My point of view 데이터 중심 사고를 하는 사람들이 모여서 일을 하는 것 Dev + Ops + Data Engineer + Data Scientist
  • 8. DataOps Principles https://www.dataopsmanifesto.org/dataops-manifesto.html 1. Continually satisfy your customer - 지속적으로 고객을 만족시켜라 2. Value working analytics - 분석을 가치있게 생각하라 3. Embrace change - 변화 수용 4. It's a team sport - 다양한 역할, 기술, 도구 수용 5. Daily interactions - 매일 협력 6. Self-organize - 자기주도 7. Reduce heroism - 영웅주의를 줄여라 8. Reflect - 반성하라 9. Analytics is code - 분석은 코드다 10. Orchestrate - 결합하라 11. Make it reproducible - 재현 가능하게 만들어라 12. Disposable environments - 비용 최소화 13. Simplicity - 단순성 14. Analytics is manufacturing - 분석은 제조와 같다 15. Quality is paramount - 품질이 제일 중요 16. Monitor quality and performance - 품질 및 성능을 모니터링하라 17. Reuse - 재사용하라 18. Improve cycle times - 사이클 타임을 개선하라
  • 9. DataOps Principles • 애자일 방법론부터 시작 • 내/외부 고객 만족을 위한 끊임없는 분석 통찰력 제공 • 분석 성과를 측정하고, 변화를 추구하며, 변하는 고객 요구사항을 지속적으로 이 해해야함
  • 10. DataOps 구성 • 목표를 중심으로 스스로 조직 • No Hero, Sustainable, Scalable, Process 지향 • Data, Tool, Code, Environment 모두 장악이 필요함 • Reproducible 결과물 -> 분석 Pipeline • Cross-Functional Team(교차기능 팀) • Dev, Architect, Ops, Data Science, Data Engineer 모두 포함 • 개발자, 운영자, 데이터전문가 (3자 협업 구도)
  • 12. Goal • No-Ops : Remove existing management (Serverless) • GitOps : All infra, codes, and scripts are managed in immutable state • Automation : Communications, Approvals, SRs, Issues
  • 13. Requirements • Key Management • IAM Role • Access/SecretKey • Code Repository • Github, Bitbucket (Public Access) Account • Code Commit • CI • CircleCI(Github Auth) • Code Build • Container Registry • AWS ECR (Elastic Container Registry) • Dockerhub • CD • Terraform, CircleCI • CloudFormation, Code Deploy • Notification • Approval, SR, Issue, Collaboration
  • 14. AWS Native CI/CD for Web Service
  • 15. AWS Native CI/CD for Data Preparation
  • 16. AWS Native CI/CD for Data Ingest
  • 17. Code Repository Github • 다양한 Eco 3rd Party • Private 사용 및 Collaboration 기능으로 비공개 Repo 활용 • GitOps
  • 18. Container Registry ECR • Fully Managed • Security (IAM) 연동 • CircleCI Orbs 제공 • EKS, ECS, Batch 연동 용이
  • 19. CI/CD Terraform • CloudFormation : 직접 사용하는 도구보다는 백엔드로 활용하는 서비스 • 유지보수, 재사용, 모듈화 어려움 • 선언적 인프라스트럭처 관리 도구로 많이 사용하고 있는 도구 • 옵션,설정에 대한 관리 (State), 재사용, Dev-Ops 동시 확인 용이 • VPC, Security Group, IAM 관리 용이
  • 20. CI/CD CircleCI • Fully Managed (Serverless) • Caching, Debugging, Context • AWS 종속성 최소화 • Git, Registry, CD영역의 확장성 고려 • AWS Console 접속 최소화 • 쉽고 단순하고 빠른 빌드 환경구성 • 소규모 프로젝트 빠르게 시작 가능
  • 22. Terraform Pipeline Job flow • 인프라 작업 및 IAM 계정작업 • master branch에서만 terraform apply • CircleCI Version 2.1 기능 활용 • slack notification • executors • terraform plan 결과 저장 : persist_to_workspace • terraform apply : attach_workspace Checkout Lint Plan Approval Apply Master? https://github.com/ddiiwoong/ecs-tf-template/blob/master/.circleci/config.yml
  • 23. Batch Code Pipeline Job flow • Crawling을 위한 Batch Job(AWS ECS) Script update Pipeline 구성 • Docker build 및 AWS ECR(Registry)로 Script image Push • AWS Batch Job Definition 변경 (Image Change) 참고 : https://ddii.dev/devops/circleci-ecs/ https://github.com/ddiiwoong/batch-cicd-demo/blob/master/.circleci/config.yml Docker Build & Push S3 Upload Slack Noti. Approval Deploy BatchCheckout
  • 24. Portal Development Environment Pipeline Job flow • 서비스 Application Build Pipeline • Docker build 및 AWS ECR(Registry)로 image Push • image archive 및 Caching • 특정 Tags 또는 branch에 대해서만 Build/Deploy • Landing Page는 S3로 hosting (S3 upload) 참고 : https://yunsangjun.github.io/blog/cicd/2019/07/03/circleci.html https://github.com/ddiiwoong/circleci-demo/blob/master/.circleci/config.yml Build & Test Archive & Caching Image Push ECS updateCheckout S3(Dev) Sync Checkout Approval S3(Prod) Sync ECS update Static Hosting
  • 25. In Progress Glue Job Code Update • ETL/ELT Job 처리를 위한 Python/Scala Code Update • Scripts Location, Filename Update Lambda Code Update • https://github.com/ddiiwoong/serverless-example-monorepo-with-circleci/blob/master/.circleci/config.yml EMR, Sagemaker Provisioning 설정 구성 • Bootstrap, Lifecycle Scripts • Shutdown-actions Multi-Cloud를 위한 준비 • Azure, IBM Data Pipeline Integration • Data Migration을 위한 Crawler 구성 DevSecOps • Scan images, Secrets Management • https://circleci.com/integrations/devsecops/
  • 26. Summary Challenge • Dev 동의 - Easy! • 정보보호 동의 - Hard!! • Secret Environment, Context (DevSecOps) • https://circleci.com/blog/protect-secrets-with-restricted-contexts/ • Ops(TA) 동의 - Very Hard!!! • CloudFormation vs Terraform • S3 vs Git • Role, Policy Remember • executor • caching (persist_to_workspace) • ECR immutable image tags • https://aws.amazon.com/ko/about-aws/whats-new/2019/07/amazon-ecr-now- supports-immutable-image-tags/