Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

[OpenStack Days Korea 2016] Track1 - 카카오는 오픈스택 기반으로 어떻게 5000VM을 운영하고 있을까?

40,419 views

Published on

하현 / 카카오

Published in: Technology
  • Follow the link, new dating source: ❶❶❶ http://bit.ly/2F90ZZC ❶❶❶
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • Sex in your area is here: ❶❶❶ http://bit.ly/2F90ZZC ❶❶❶
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • DOWNLOAD THAT BOOKS INTO AVAILABLE FORMAT (2019 Update) ......................................................................................................................... ......................................................................................................................... Download Full PDF EBOOK here { http://bit.ly/2m6jJ5M } ......................................................................................................................... Download Full EPUB Ebook here { http://bit.ly/2m6jJ5M } ......................................................................................................................... Download Full doc Ebook here { http://bit.ly/2m6jJ5M } ......................................................................................................................... Download PDF EBOOK here { http://bit.ly/2m6jJ5M } ......................................................................................................................... Download EPUB Ebook here { http://bit.ly/2m6jJ5M } ......................................................................................................................... Download doc Ebook here { http://bit.ly/2m6jJ5M } ......................................................................................................................... ......................................................................................................................... ................................................................................................................................... eBook is an electronic version of a traditional print book that can be read by using a personal computer or by using an eBook reader. (An eBook reader can be a software application for use on a computer such as Microsoft's free Reader application, or a book-sized computer that is used solely as a reading device such as Nuvomedia's Rocket eBook.) Users can purchase an eBook on diskette or CD, but the most popular method of getting an eBook is to purchase a downloadable file of the eBook (or other reading material) from a Web site (such as Barnes and Noble) to be read from the user's computer or reading device. Generally, an eBook can be downloaded in five minutes or less ......................................................................................................................... .............. Browse by Genre Available eBooks .............................................................................................................................. Art, Biography, Business, Chick Lit, Children's, Christian, Classics, Comics, Contemporary, Cookbooks, Manga, Memoir, Music, Mystery, Non Fiction, Paranormal, Philosophy, Poetry, Psychology, Religion, Romance, Science, Science Fiction, Self Help, Suspense, Spirituality, Sports, Thriller, Travel, Young Adult, Crime, Ebooks, Fantasy, Fiction, Graphic Novels, Historical Fiction, History, Horror, Humor And Comedy, ......................................................................................................................... ......................................................................................................................... .....BEST SELLER FOR EBOOK RECOMMEND............................................................. ......................................................................................................................... Blowout: Corrupted Democracy, Rogue State Russia, and the Richest, Most Destructive Industry on Earth,-- The Ride of a Lifetime: Lessons Learned from 15 Years as CEO of the Walt Disney Company,-- Call Sign Chaos: Learning to Lead,-- StrengthsFinder 2.0,-- Stillness Is the Key,-- She Said: Breaking the Sexual Harassment Story That Helped Ignite a Movement,-- Atomic Habits: An Easy & Proven Way to Build Good Habits & Break Bad Ones,-- Everything Is Figureoutable,-- What It Takes: Lessons in the Pursuit of Excellence,-- Rich Dad Poor Dad: What the Rich Teach Their Kids About Money That the Poor and Middle Class Do Not!,-- The Total Money Makeover: Classic Edition: A Proven Plan for Financial Fitness,-- Shut Up and Listen!: Hard Business Truths that Will Help You Succeed, ......................................................................................................................... .........................................................................................................................
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • Hello! Get Your Professional Job-Winning Resume Here - Check our website! https://vk.cc/818RFv
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
  • 감사합니다.
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here

[OpenStack Days Korea 2016] Track1 - 카카오는 오픈스택 기반으로 어떻게 5000VM을 운영하고 있을까?

  1. 1. “ 카카오는 오픈스택 기반으로 5000VM을 어떻게 운영하고 있을까? ” 하 현 (Raymon.ha) raymon.ha@kakaocorp.com
  2. 2. 하 현 ( Raymon.ha ) 클라우드 엔지니어 2012년 1월부터 현재까지 약 4년 간 오픈스택 기반 클라우드플랫폼 개발 및 운영 틈틈이, OpenStack Bug report, Fix, Blueprint Contribution 활동 前) LG CNS 클라우드플랫폼팀 - Swift 기반 오브젝트 스토리지 서비스 구축 - 오픈스택 기반 Private/Public Cloud 개발/운영 現) 카카오 클라우드컴퓨팅셀(Cloud Computing Cell) 소속 - Cinder backend Storage CEPH 구축 및 운영 - 오픈스택 기반 Private Cloud 개발/운영 - 가장 최근에는 Juno 버전을 Kilo버전으로 업그레이드 진행중 발표자 소개
  3. 3. 1. Intro
  4. 4. Requirements 개발자曰: 원래 이런건 다 되는거 아닌가요? - 빠른 개발 주기, Time to market이 중요함 - 많은 서비스 런칭 - 개발자가 원할때 빠르게 서버 제공 - 인프라의 이슈로 개발이 늦어져서는 안됨 - 서버나 네트워크는 잘 모르겠다. 알아서 해주면 좋겠다. 출처) kakaocorp.com 서비스 등록 기준
  5. 5. Kakao Private Cloud Legacy Operation vs Cloud Operation How to Develop + Operation with OpenStack DevOps Style Management 이런 이야기를 하려고 합니다
  6. 6. 2. Current Status
  7. 7. Current Usage (2015년 12월 기준) Created : 약 2900 VM Deleted : 약 2000 VM Running : 약 900 VM  평균 매 8분마다 1개 VM 이 생성되거나 삭제됨 Compute Node: 약 700대 VM : 약 5000대 Ceph : 약 200TB 출처) 카카오 infra portal Keystone tenants : 약 1000개 Keystone users : 약 1300 명
  8. 8. 사용량 예측 현재 추세대로라면, 2016년 말 VM 수 약 1만대 규모로 증가 * 3개월 단위로 물리머신 100대 신규 증설
  9. 9. OpenStack Global Survey 출처) Openstack Survey, 2015 (https://www.openstack.org/assets/survey/Public-User-Survey-Report.pdf) How many physical compute nodes to OpenStack clouds have? How many instances in an OpenStack deployment? 10,000 to 99,999 7%
  10. 10. 3. Overview of Kakao Private Cloud
  11. 11. Kakao Private Cloud  오픈스택 기반의 카카오 Private Cloud Krane : Crane [krein] : 명사 [기계] 기중기, 크레인 (Kakao + Crane) Spec : OS: ubuntu 14.04 OpenStack version: stable/juno (2014.2.4) Hypervisor: only KVM Project: nova, glance, cinder, neutron, keystone, horizon, trove, heat
  12. 12. OpenStack Upgrade History Folsom 2012. 11 Grizzly 2013. 4 Havana 2013. 10 Icehouse 2014. 4 Juno 2014. 10 Kilo 2015. 4 Liberty 2015. 10 Icehouse 2014. 5 Ubuntu 14.04 2014. 11 Juno 2015. 4 Kilo 2016. 2 (진행중) Havana 2014. 1 Grizzly 2013. 8 (… Test Lab)
  13. 13. KaKao Krane – Openstack 아키텍처 Global Controller - shared Region : busan_v1 horizon keystone Swift-API nova nova-compute Compute Node nova-compute Compute Node nova-compute Compute Node Load Balancer Ceph Ceph Kage neutron-agents neutron-agents neutron-agents rabbitmq DB rabbitmq Region : seoul_v1 rabbitmq Region : seoul_v2 rabbitmq Load Balancer Load BalancerLoad Balancer glance cinder DB DBDB Inhouse-sso 사내 DNS neutron heat trove nova glance neutron heat trove nova glance cinder neutron heat trove
  14. 14. Global Controller - shared Region : busan_v1 horizon keystone Swift-API nova nova-compute Compute Node nova-compute Compute Node nova-compute Compute Node Load Balancer Ceph Ceph Kage neutron-agents neutron-agents neutron-agents rabbitmq DB rabbitmq Region : seoul_v1 rabbitmq Region : seoul_v2 rabbitmq Load Balancer Load BalancerLoad Balancer glance cinder DB DBDB Inhouse-sso 사내 DNS neutron heat trove nova glance neutron heat trove nova glance cinder neutron heat trove KaKao Krane – Openstack 아키텍처
  15. 15. Global Controller - shared Region : busan_v1 horizon keystone Swift-API nova nova-compute Compute Node nova-compute Compute Node nova-compute Compute Node Load Balancer Ceph Ceph Kage neutron-agents neutron-agents neutron-agents rabbitmq DB rabbitmq Region : seoul_v1 rabbitmq Region : seoul_v2 rabbitmq Load Balancer Load BalancerLoad Balancer glance cinder DB DBDB Inhouse-sso 사내 DNS neutron heat trove nova glance neutron heat trove nova glance cinder neutron heat trove KaKao Krane – Openstack 아키텍처
  16. 16. Global Controller - shared Region : busan_v1 horizon keystone Swift-API nova nova-compute Compute Node nova-compute Compute Node nova-compute Compute Node Load Balancer Ceph Ceph Kage neutron-agents neutron-agents neutron-agents rabbitmq DB rabbitmq Region : seoul_v1 rabbitmq Region : seoul_v2 rabbitmq Load Balancer Load BalancerLoad Balancer glance cinder DB DBDB Inhouse-sso 사내 DNS neutron heat trove nova glance neutron heat trove nova glance cinder neutron heat trove KaKao Krane – Openstack 아키텍처
  17. 17. Global Controller - shared Region : busan_v1 horizon keystone Swift-API nova nova-compute Compute Node nova-compute Compute Node nova-compute Compute Node Load Balancer Ceph Ceph Kage neutron-agents neutron-agents neutron-agents rabbitmq DB rabbitmq Region : seoul_v1 rabbitmq Region : seoul_v2 rabbitmq Load Balancer Load BalancerLoad Balancer glance cinder DB DBDB Inhouse-sso 사내 DNS neutron heat trove nova glance neutron heat trove nova glance cinder neutron heat trove KaKao Krane – Openstack 아키텍처
  18. 18. “2명” 카카오 클라우드컴퓨팅셀 헬게이트 오픈스택 운영 인원 raymon.haal.l
  19. 19. 포기할 순…
  20. 20. “ Automation “ = reduce operation cost
  21. 21. But, as you know, OpenStack Automation is “Really hard work” https://www.openstack.org/summit/tokyo-2015/videos/presentation/ntt- communications-automate-deployment-and-benchmark-for-your-openstack-with- chef-cobbler-and-rally (1분 30초) *100
  22. 22. Why? Complex system of many components and OSS OpenStack is fast moving 테스트 환경 vs 서비스 환경 하드웨어 자동화 레거시 시스템 OpenStack is one of the system within IDC
  23. 23. 4. Develop & Operation
  24. 24. User/Project Inhouse sso plugin 모듈 개발 - 카카오 사내 인증 시스템 사용자 인증을 거쳐 자동으로 OpenStack keystone user 생성 프로젝트는 개별 사용자가 직접 생성 - 신규 생성 및 기존 프로젝트 참여 [identity] driver = kakao.openstack.keystone.inhouse_sso_auth.Identity auto_create_user = true default_project = demo member_role = Member email_postfix = @kakaocorp.com system_users = admin,nova,quantum,neutron,cinder,glance, heat,ceilometer,swift,docker-registry, heat_domain_admin,sahara keystone.conf
  25. 25. Domk OpenStack 내에서 생성된 VM의 hostname을 사내 DNS 서비스에 도메인 네임 등록하도록 자동화 오픈스택의 Designate 프로젝트와 유사한 기능 구현
  26. 26. 운영 tool http://stacktach.9rum.cc/ OpenStack Error Alert RabbitMQ queue count 모니터링 Notification 모니터링 VM 생성 갯수 모니터링 자원 여유량 모니터링
  27. 27. Pool Compute node 를 Resource Pool 로 관리 Production Pool 발주 투입 Compute nodes Standby Physical host HW 검수 OS 셋팅 Openstack 설치Service Compute nodes L 기준으로 node 증설 Pool 부족 시, 신규 발주
  28. 28. Continuous Integration / Continuous Deployment 그 시작은, Kfield (started by andrew.kong)
  29. 29. 5. Kfield
  30. 30. What is Kfield? OpenStack Build, Test, Deploy 자동화 개발자 개별 개발 환경 배포에 사용 Production 배포에 사용 Chef cookbooks + ansible playbooks 내부 OpenStack mirrored repo를 통해 OpenStack code 배포
  31. 31. Kfield details kvm Libvirt LB DB/MQ compute02 swift Vagrant up GitHub Openstack mirror repo Dev#1, individual test node Kfield git clone /home/kfield compute02controller copy
  32. 32. Kfield Demo Running time: 약 6분
  33. 33. Develop Process Kfield Your git repo Github 1. fork Your local dev 2. Git clone Kfield/master Your/master Your/master 3. branch feature/CCC-001 Your/feature/CCC-001 Your/feature/CCC-001 4. Develop any feature Run unit test 5. Git commit jenkins Fail? Succes? Github Code Review 7. Pull Request Good? Bad? Request Fix 6. Build Test 8. Merge
  34. 34. CI/CD pipline develop to production Staging ProductionDevelop Build Chef-server Web hook Commit Deploy Upload Deploy OpenStack mirror repo Version Control Kfield Run Test Slave Clone
  35. 35. 5. 앞으로는 . . .
  36. 36. 우리의 Cloud는? VM, Network 와 같은 구현물(implementation)에 최우선을 둠 OpenStack 서비스에 문제가 있어도 VM, Network와 같은 구현물에는 영향을 끼치면 안됨 API 기반 (= programmable) 인스턴스는 휘발성 절대 죽지않는 시스템? 시스템이 죽더라도 서비스가 유지되는 아키텍처 Reduce process, remove complexity
  37. 37. Self-Service Concept 출처) Charlie.choe, CCC Monitoring/Alerting/Logging System Monitoring/Alerting/Logging System CMDB
  38. 38. “졸라리 헤엄치고 있어. 산다는게 그런거다. 장난 아냐 임마 ” "백조가 아주 물위에선 폼나고 우아하게 떠있지. 근데 너 물속은 어떤지 알아?” (영화 넘버3. 한석규 대사)
  39. 39. Q & A

×