자동화된 인프라 구축
Building an Automated Infrastructure
2009년 https://groups.google.com/forum/#!forum/sysadminstudy 에서 발표했던 자료입니다.
출처 : http://wiki.tunelinux.pe.kr/pages/viewpage.action?pageId=884766
2. 자료소개
• 이 자료는 Building an Automated
Infrastructure 자료 및 "Web Ops 2.0:
Achieving Fully Automated Provisioning"
를 기반으로 일부 수정한 자료임
• 해당 자료는 마지막 부분 참고자료에서 확인
3. 개요
• 시스템운영 – 현재의 상황
• “ 자동화된 인프라스트럭쳐”란 ?
• 자동화 단계
• 질문과 답
4. • 수작업 (OS 설치 , OS 설정 , 애플리
케이션 배포 및 설정 )
• 서버에 직접 접속하여 시스템 관리작업
을 함
• 시스템관리에 대한 프로세스 , 정책이
부족
• 능숙한 SM 이 필요하며 하위 관리자에게
위임하기 어려움
시스템운영 - 현재의 상황
9. 자동화 단계 – OS 설치
수작업
• 각 시스템 수동설치
자동화
• 자동화된 설치 시스템 이용 (PXE)
• kickstart, Jumpstart, Windows
RIS/WDS(Linux 도 설치 가능함 ) 등
• 표준화 , 인증된 하드웨어 사용 및 절차 필요
10. 자동화 단계 - DNS
수작업
• 각 시스템의
/etc/hosts 업데이트
자동화
• DNS 서버 설치
• 설정 관리
Adding a new Hostname
0
10
20
30
40
50
60
6 10 20 30 40 50
Number of Systems
Minutes
DNS
Manual
11. 자동화 단계 – 서버 인벤토리
수작업
• 엑셀파일 , 위키
자동화
• iClassify
• LDAP
• ControlTier
• 직접 만들기
12. 자동화 단계 - 계정관리
수작업
• 각 계정 수동 추가
자동화
• LDAP 또는 AD 이용
• 설정 관리
• LDAP 지원하는 애플
리케이션 다양함
Adding a new User
0
10
20
30
40
50
60
6 10 20 30 40 50
Number of Systems
Minutes
Automated
Manual
13. 자동화 단계 - 버전관리
수작업
• 파일 복사 , FTP, http 이용
자동화
• 버전관리 툴 이용
• CVS, Subversion, 기타
• 개발뿐만 아니라 운영에서도 버전관리 필수
14. 자동화 단계 - 설정관리
수작업
• 버전 관리 시스템
• 위키
• SSH 루프 이용
설정 파일을 복사한 후
체크
작업한 내용을 기록해야
함
20. 자동화 단계 – 애플리케이션 배
포
수작업
• 수작업으로 애플리케
이션 설치
자동화
• Capistrano
• ControlTier
• 시스템 인벤토리 툴에
통합
Odds of a non-code Mistake
during a Deploy
0%
2%
4%
6%
8%
10%
12%
1 2 5 10
Number of Manual Steps
Mistakes
21. 다시 확인
• 자동화된 인프라스트럭쳐의 의미 :
– “ 사람의 개입 없이 대규모 사업을 운영하는데
필요한 기본 서비스를 갖추는 것”
– “ 고양이가 사용할 수 있는 시스템”
• 자동화된 인프라스트럭쳐
– 시간 절약
– 효율성 늘림
– 확장성 가능함
– 경제적인 충격을 줄임
– 유연성 개선
22. 다시 확인
• 자동화된 인프라스트럭쳐의 의미 :
– “ 사람의 개입 없이 대규모 사업을 운영하는데
필요한 기본 서비스를 갖추는 것”
– “ 고양이가 사용할 수 있는 시스템”
• 자동화된 인프라스트럭쳐
– 시간 절약
– 효율성 늘림
– 확장성 가능함
– 경제적인 충격을 줄임
– 유연성 개선
23. 참고자료
• Building an Automated Infrastructure (O’REILY Velocity 2008) :
http://en.oreilly.com/velocity2008/public/schedule/detail/2238
• Achieving Fully Automated Provisioning :
http://www.dtosolutions.com/storage/downloads/FullyAutomatedProvision
• Reliable, Repeatable, Reproducible Infrastructure
http://sysadmin.miniconf.org/presentations08.html#02
• ControlTier http://wiki.controltier.org/wiki/ControlTier
• Continuous integration
http://en.wikipedia.org/wiki/Continuous_integration
• Automating Linux and Unix System Administration Second Edition
• The Practice of System and Network Administration
(2/E)
• http://groups.google.com/group/sysadminstudy
• http://tunelinux.pe.kr/
13 years as a Systems Administrator
From garages to public companies
How many people are systems administators?
How many people are software developers?
How many people conisder themsleves primarily concerned with business?
Overview
Why it’s important
Talk about how to do it, and why it’s good, with monkeynews
review
Q & A
Overview
Why it’s important
Talk about how to do it, and why it’s good, with monkeynews
review
Q & A
These are the steps to go from the napkin to actual deployment.
OS Install - Get an operating system up and on a network
DNS - Give your new system a name
Server Inventory - Have a place where you keep track of each system, and what it does
Identity Management - Grant your users access and privileges to your new servers
Version Control - Keep track of the changes to your application code, and ideally, your infrastructure too
Configuration Management - Keep track of how each system is configured, and update it when you make changes
Monitoring - Watch your new systems for signs of trouble
Trending - Make graphs and charts of important metrics, so that you can tell if the infrastructure is behaving well, and predict future capacity
Application Deployment - Actually put your application on the infrastructure, and update it
Show of hands - how many people in this room can refer to every server in their infrastructure by name? How many people only have to go to one place to update the list of what hosts you have, and their IP addresses?
The graph tells the story, I think. The time it takes to install and configure DNS is negligible, even if you have never touched it before. It’s worth your time, even if you only think about keeping everything in sync.
Lots of good DNS tools, djbdns, bind, maradns. If you hate DNS, that’s fine - you can remove it entirely as long as you have configuraton management in place to update your /etc/hosts files or equivilant. The point is: have one place, centrally managed, that is canonical for the names of your severs.
Show of hands - how many people in this room can refer to every server in their infrastructure by name? How many people only have to go to one place to update the list of what hosts you have, and their IP addresses?
The graph tells the story, I think. The time it takes to install and configure DNS is negligible, even if you have never touched it before. It’s worth your time, even if you only think about keeping everything in sync.
Lots of good DNS tools, djbdns, bind, maradns. If you hate DNS, that’s fine - you can remove it entirely as long as you have configuraton management in place to update your /etc/hosts files or equivilant. The point is: have one place, centrally managed, that is canonical for the names of your severs.
Now that you have servers up, and they have names everyone can see, you need to keep track of the servers you have, and what they do.
This may seem obvious, but I bet 90% of the startups I encounter, and 80% of the large companies, can’t tell you even *how many* servers they have with any degree of reliabaility, much less what each one is doing at any given time. (Even if they have DNS!)
iClassify is a tool we created for doing just this job. It is a small agent that runs on each system, and repors to a centralized web service about the system it’s running on. You can then tag hosts, del.ici.ous style, and search the inventory with a full text search engine (Solr, for the curious.) I’ll talk more about it later. Also, Trusera, a client of ours, graciously let us use their actual infrastructure for these screenshots. Thanks, Trusera. :)
LDAP often already exists for Identity Management in many infrastructures, and as long as you don’t need a lot of complex data, it’s a good place to put your host information.
Lots of people have written databases that do this sort of thing. Use whatever suits you -- but I have to say, making the systems report themselves to the inventory system is a huge, huge win.
All that, and we still don’t have users everywhere yet. MonkeyNews is a small company right now, only two people, and six servers. But you still have to figure out who has access to which servers, and what privileges they have.
The 수작업 way to do this is to add each user on every system.
The 자동화 way is to use a centralized service, such as LDAP or Active Directory.
This graph should look familiar, because it has the exact same automation bonus as DNS does. When you have 5 servers, the 5 minutes it takes seems like no big deal. But that’s 5 minutes for *any user change*.
Password change? 5 minutes.
And the curve is linear.. As you add more servers, you have to add users everywhere, and it takes longer and longer.
Centralize your identity management infrastructure.
Have one user name and password.
Having a central place to track changes to code and infrastructure, with blame and history
Not really an “자동화” vs “수작업” thing - you just don’t have a choice :)
Using version control is a requirement of at least two future steps
Subversion, Git, Mercurial, CVS Perforce
Just pick one you like and use it religiously
Server Classification says what a thing ought to be, Configuration Management makes it so.
Everything up to deploying your application specific code on all of your servers
This means everything that isn’t done for you at OS installation
자동화 configuration management is the heart of having an 자동화 infrastructure
Instead of doing things by hand and keeping track of them
You express how the infrastructure should behave as code
Cfengine is the grand old academic dean of Unix/Linux configuration management
Puppet is, in my opinion, the current state of the art
Bcfg2 I have never used, but some folks dig it - XML based config files
Vertebra a new entry here?
Let me show you what I mean with a puppet example that everyone can relate to, managing /etc/sudoers
Easy to adapt to wide variation between systems
Incredible time savings
Always current
Monitoring, for our purposes, is the act of watching the system for conditions that we want to be notified about. Things like “is this service running”, or “did I make enough money in the last hour”.
In a 수작업 world, you would configure each server (and service) to be monitored by hand
In an 자동화 one, you would configure each class of server one time, and let the automation do the rest
Only edit the config files once for each kind of system
Tools like Nagios can be 자동화 with configruation management, tools like Hyperic and OpenNMS have there own discovery mechanisms
“The process of extrapolating metrics to make future capacity forecasts.”
Charts and Graphs
Has a similar configuration burden as Monitoring, and an identical solution
Modern web applications send a ton of email, so make it easy to do
Most linux distributions will send lots of email, so make it easy on them too
Monitoring will want to send email
Use your configuration management system and system inventory to automate the configuration on a per-server basis
Or do it by hand
Application Deployment can be easy, or it can be hard
It can be time consuming, or it can be very quick
The key predictor of either is the number of steps you have to take to deploy
If the number is 0%, and a deploy either succeeds or fails, you have a 0% chance of a deploy related production impacting outage
As the number of steps increases, so does your odds of screwing it up
Someday, Vertebra!