Infrastructure is development

1,531 views
1,413 views

Published on

Talk I gave on infrastructure as development at the 2009 Red Hat Summit. Slides are rapid-fire. It was only a 45 minute talk.

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
1,531
On SlideShare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
31
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Infrastructure is development

  1. 1. Infrastructure is Development Michael Stahnke (stahnma) 02-SEP-2009
  2. 2. The ideas presented today are not representative of my employer, business success, past jobs and do not offer endorsement to any particular products brands or companies. Heck, a lot of this stuff isn't even my idea to begin with. http://www.flickr.com/photos/sea-turtle/3049443478/ http://www.flickr.com/photos/jjze/726682393/
  3. 3. <ul>“ There are only two hard problems in Computer Science: cache invalidation and naming things.” <ul>--Phil Karlton </ul></ul>
  4. 4. Who Am I?
  5. 5. Who Am I?
  6. 6. Who Am I?
  7. 7. Who Am I?
  8. 8. Who Am I?
  9. 9. Who Am I? <Insert Large Company Name Here>
  10. 10. Where Am I? <ul><li>http://stahnma.fedorapeople.org
  11. 11. http://www.stahnkage.com
  12. 12. @stahnma on Twitter
  13. 13. @stahnma on identi.ca
  14. 14. stahnma on github
  15. 15. [email_address]
  16. 16. stahnma on Freenode IRC </li></ul>
  17. 17. Where Am I? <ul>If you google stahnma , it's probably me. </ul>
  18. 18. Baseline <ul>Infrastructure – the collection of all components that make up the non-external facing IT realm. Primarily, in this case, servers connected to storage and network. It can include network/SAN devices in some cases. </ul>
  19. 19. What's Coming Up <ul>The Tao of the Agile Infrastructure </ul>
  20. 20. What's Coming Up <ul>Inherit Problems with System Administration </ul>
  21. 21. What's Coming Up <ul>Some solutions to problems with System Administration </ul>
  22. 22. Three Domains <ul>1. Infrastructure Practices </ul>
  23. 23. Three Domains <ul>1. Infrastructure Practices 2. People </ul>
  24. 24. Three Domains <ul>1. Infrastructure Practices 2. People 3. Technology Choices </ul>
  25. 25. The Problem
  26. 26. System Administration isn't a science
  27. 27. You can't get a degree in System Administration
  28. 28. Heterogeneous Systems
  29. 29. No Clear Job Description
  30. 30. Expectations not Clear
  31. 31. System Admins are @ holes
  32. 32. Can't keep up with demand
  33. 33. Admins fight management http://www.flickr.com/photos/jjze/726682393/
  34. 34. Solutions
  35. 35. Are delivered Solutions
  36. 36. Through Ideas Solutions
  37. 37. Put together by Good People Solutions
  38. 38. Using some awesome technologies Solutions
  39. 39. Topic 2 Infrastructure Practices
  40. 40. Why is your infrastructure special?
  41. 41. Steal Ideas http://www.flickr.com/photos/purplemattfish/3508761485/
  42. 42. Open your infrastructure
  43. 43. Opening your infrastructure includes your issues
  44. 44. Open your infrastructure and your solutions
  45. 45. Well, what do you mean?
  46. 46. To Have a High Performing Team : <ul><li>You must know </li><ul><ul><li>what you manage. </li></ul></ul></ul>
  47. 47. What do you have? <ul><li>Asset Database
  48. 48. LDAP Directory
  49. 49. Hardware Management Tools
  50. 50. Power Management Tools
  51. 51. Monitoring Tools
  52. 52. Provisioning Tools
  53. 53. Storage Management Tools
  54. 54. Backup Tools
  55. 55. Policy Engines
  56. 56. Patch Tools
  57. 57. Security Scanning Tools
  58. 58. Virtualization Management Tools
  59. 59. Log Management Tools </li></ul>
  60. 60. Common Example: <ul><li>Asset Database – Who knows?
  61. 61. LDAP Directory – RHDS, 389, SunOne, AD, OpenLDAP
  62. 62. Hardware Management Tools – IBM Director, DRAC
  63. 63. Power Management Tools – APC
  64. 64. Monitoring Tools – Nagios, Tivoli, OpenView
  65. 65. Provisioning Tools – Cobbler, Vmware,
  66. 66. Storage Management Tools – IBM Whatever, Some Custom Stuff
  67. 67. Backup Tools – Netbackup, Tivoli, Networker, Tar, Gzip, Rsync, Cron
  68. 68. Policy Engines – Cfengine, Puppet, Scripts
  69. 69. Patch Tools – RHN, NIM, Custom Repos
  70. 70. Security Scanning Tools – Lots of stuff
  71. 71. Virtualization Management Tools – vCenter, Virt-Manager, Spacewalk, RHN
  72. 72. Log Management Tools – Syslog Server
  73. 73. DNS – Bind </li></ul>
  74. 75. What now? <ul>A. You can cry about it </ul>
  75. 76. What now? <ul>A. You can cry about it B. You can remove data sources </ul>
  76. 77. What now? <ul>A. You can cry about it B. You can remove data sources C. You can integrate/federate them </ul>
  77. 78. What now? <ul>A. You can cry about it B. You can remove data sources C. You can integrate/federate them D. All of the Above </ul>
  78. 79. What now? <ul>A. You can cry about it B. You can remove data sources C. You can integrate/federate them D. All of the Above </ul>The correct answers are both B and C; however it is very likely you will encounter A, so the likely answer is, in fact, D.
  79. 80. Infrastructure is Development
  80. 81. The Infrastructure is the Application
  81. 82. Application == Infrastructure
  82. 83. Infrastructure Goals <ul><li>Deliver results to the business </li></ul>
  83. 84. Infrastructure Goals <ul><li>Deliver results to the business
  84. 85. Make the infrastructure an investment, not a cost </li></ul>
  85. 86. Infrastructure Direction <ul><li>Solving Problems </li></ul>
  86. 87. Infrastructure Direction <ul><li>Solving Problems
  87. 88. Automation </li></ul>
  88. 89. Infrastructure Direction <ul><li>Solving Problems
  89. 90. Automation
  90. 91. The right mix of people </li></ul>
  91. 92. Infrastructure Direction <ul><li>Solving Problems
  92. 93. Automation
  93. 94. The right mix of people
  94. 95. The right decision processes </li></ul>
  95. 96. Infrastructure Direction <ul><li>Solving Problems
  96. 97. Automation
  97. 98. The right mix of people
  98. 99. The right decision processes
  99. 100. Vision </li></ul>
  100. 101. Infrastructure Direction <ul><li>Solving Problems
  101. 102. Automation
  102. 103. The right mix of people
  103. 104. The right decision processes
  104. 105. Vision
  105. 106. Known State </li></ul>
  106. 107. Where do we start?
  107. 108. Axiom 1 <ul>Reuse before building or purchasing </ul>
  108. 109. Your Infrastructure isn't a secret
  109. 110. Your Infrastructure isn't the secret sauce
  110. 111. Your Infrastructure isn't differentiating
  111. 112. Your Infrastructure isn't a secret <ul><li>Everybody has servers, a network, some storage
  112. 113. Somebody has probably solved this problem
  113. 114. Check some common places: </li><ul><li>Google
  114. 115. IRC
  115. 116. Sourceforge
  116. 117. Ohloh.net
  117. 118. Amazon book selection </li></ul><li>What solutions do you see? </li></ul>
  118. 119. So when I say re-use??? <ul>Reuse the code and tools you have </ul>
  119. 120. So when I say re-use??? <ul>Enable feature you are currently not utilizing </ul>
  120. 121. So when I say re-use??? <ul>Search for a quality Open Source project </ul>
  121. 122. So when I say re-use??? <ul>Find other organizations successes on an open infrastructure </ul>
  122. 123. You can't reuse? <ul>Ask why. </ul>
  123. 124. You can't reuse? <ul>Is your problem new? </ul>
  124. 125. You can't reuse? <ul>Is your problem special? </ul>
  125. 126. You can't reuse? <ul>Ok, you can look at building or purchasing... </ul>http://www.flickr.com:80/photos/iambrad/289462494/
  126. 127. I have/need a purchased proprietary solution for Problem X
  127. 128. I have/need a purchased proprietary solution for Problem X <ul>So do I, and I'm sorry. </ul>http://www.stahnkage.com/outage
  128. 129. Software/Tool Selection
  129. 130. Purchased Software Selection Criteria <ul>There a few things to evaluate </ul>
  130. 131. Purchased Software Selection Criteria <ul>There a few things to evaluate <li>Price </li></ul>
  131. 132. Purchased Software Selection Criteria <ul>There a few things to evaluate <li>Price
  132. 133. Performance </li></ul>
  133. 134. Purchased Software Selection Criteria <ul>There a few things to evaluate <li>Price
  134. 135. Performance
  135. 136. Functionality </li></ul>
  136. 137. Purchased Software Selection Criteria <ul>There a few things to evaluate <li>Price
  137. 138. Performance
  138. 139. Functionality
  139. 140. But really.... </li></ul>
  140. 141. Meatcloud Manifesto
  141. 142. Cloud Computing is all the rage
  142. 143. Cloud Computing is all the rage <ul>Scaling through people is not </ul>
  143. 144. <ul>The GUI is for what some user interface designer thought you wanted to do. The CLI is for what you actually need to get done. </ul>
  144. 145. <ul>The GUI is for what some user interface designer thought you wanted to do. The CLI is for what you actually need to get done. <ul><ul><ul><li>-- Mike Stahnke </li></ul></ul></ul></ul>
  145. 146. <ul>The GUI is for what some user interface designer thought you wanted to do. The CLI is for what you actually need to get done. <ul><ul><ul><li>-- Mike Stahnke </li></ul></ul></ul></ul>
  146. 147. Axiom 2 <ul>Don't Leverage the Meatcloud </ul>
  147. 148. Choose your technologies wisely
  148. 149. Very wisely
  149. 150. Rules for Software Evaluation <ul><li>Do not implement any product that does not provide an API. </li></ul>
  150. 151. Rules for Software Evaluation <ul><li>Do not implement any product that does not provide an API.
  151. 152. The provided API must have all functionality that the application provides. </li></ul>
  152. 153. Rules for Software Evaluation <ul><li>Do not implement any product that does not provide an API.
  153. 154. The provided API must have all functionality that the application provides.
  154. 155. The provided API must be tailored to more than one language and platform. </li></ul>
  155. 156. Rules for Software Evaluation <ul><li>Do not implement any product that does not provide an API.
  156. 157. The provided API must have all functionality that the application provides.
  157. 158. The provided API must be tailored to more than one language and platform.
  158. 159. Source code counts as an API, and may be restricted to one language or platform. </li></ul>
  159. 160. Rules for Software Evaluation <ul><li>Do not implement any product that does not provide an API.
  160. 161. The provided API must have all functionality that the application provides.
  161. 162. The provided API must be tailored to more than one language and platform.
  162. 163. Source code counts as an API, and may be restricted to one language or platform.
  163. 164. The API must include functional examples and not require someone to be an expert on the product to use. </li></ul>
  164. 165. Rules for Software Evaluation <ul><li>Do not use any product with configurations that are not machine parseable and machine writable </li></ul>
  165. 166. Rules for Software Evaluation <ul><li>Do not use any product with configurations that are not machine parseable and machine writable
  166. 167. All data stored in the product must be machine readable and writable by applications other than the product itself. </li></ul>
  167. 168. Rules for Software Evaluation <ul><li>Do not use any product with configurations that are not machine parseable and machine writable
  168. 169. All data stored in the product must be machine readable and writable by applications other than the product itself.
  169. 170. Writing work-arounds to cover the deficiencies in a product should be less work than writing the product’s designed functionality. </li></ul>
  170. 171. Rules for Software Evaluation <ul><li>Do not use any product with configurations that are not machine parseable and machine writable
  171. 172. All data stored in the product must be machine readable and writable by applications other than the product itself.
  172. 173. Writing work-arounds to cover the deficiencies in a product should be less work than writing the product’s designed functionality. </li></ul>
  173. 174. But I don't have automation specialists
  174. 175. That's a problem.
  175. 176. Do you have developers?
  176. 178. What's the difference between development and automation?
  177. 179. Process?
  178. 180. Requirements Gathering?
  179. 181. Cost?
  180. 182. Testing?
  181. 183. Egos?
  182. 184. Funky Lava Lamps?
  183. 185. Design Patterns?
  184. 186. Code portability?
  185. 187. What's the difference between development and automation?
  186. 188. <ul>Developers write code for use by somebody else </ul>http://www.flickr.com:80/photos/davesag/71441707/
  187. 189. <ul>System Admins write code for use by themselves, and hopefully somebody else </ul>http://www.flickr.com:80/photos/herzogbr/2274372747/
  188. 190. People make the Infrastructure Agile
  189. 191. Attract Talent People
  190. 192. Retain Talent People
  191. 193. People <ul>Hire rock stars </ul>
  192. 194. People <ul>Treat them like rock stars </ul>
  193. 195. People <ul>Observe their organizational behavior </ul>
  194. 196. People <ul>“...organizations which design systems ... are constrained to produce designs which are copies of the communication structures of these organizations.” <ul>-- Conway's Law (1968) </ul></ul>
  195. 197. People <ul>Test their knowledge </ul>
  196. 198. <ul>Hiring Questions </ul>
  197. 199. You want rock stars remember?
  198. 200. <ul>Some helpful hiring questions </ul><ul>Describe your home network setup. </ul>
  199. 201. <ul>Some helpful hiring questions </ul><ul>What cool technology projects didn't make your resume? </ul>
  200. 202. <ul>Some helpful hiring questions </ul><ul>Do you know what version control is? </ul>
  201. 203. <ul>Some helpful hiring questions </ul><ul>Do you know what version control is? Are you beyond CVS? </ul>
  202. 204. <ul>Some helpful hiring questions </ul><ul>Security or Availability? </ul>
  203. 205. <ul>Some helpful hiring questions </ul><ul>Apt or Yum? </ul>
  204. 206. <ul>Some helpful hiring questions </ul><ul>Provide three uses for a towel. </ul>
  205. 207. <ul>Some helpful hiring questions </ul><ul>Pirates or Ninjas? </ul>
  206. 208. <ul>Some helpful hiring questions </ul><ul>Icanhascheezburger or ihasahotdog ? </ul>
  207. 209. Two Types of People <ul>Breadth vs Depth </ul>
  208. 210. Breadth vs Depth <ul>Breadth <li>Systems Automation People. </li></ul>
  209. 211. Breadth vs Depth <ul>Breadth <li>Systems Automation People.
  210. 212. Big Picture thinkers. </li></ul>
  211. 213. Breadth vs Depth <ul>Breadth <li>Systems Automation People.
  212. 214. Big Picture thinkers.
  213. 215. Visionaries. </li></ul>
  214. 216. Breadth vs Depth <ul>Breadth <li>Systems Automation People.
  215. 217. Big Picture thinkers.
  216. 218. Visionaries.
  217. 219. Holistic View. </li></ul>
  218. 220. Breadth vs Depth <ul>Depth <li>Solve the problem for this exact situation. </li></ul>
  219. 221. Breadth vs Depth <ul>Depth <li>Solve the problem for this exact situation.
  220. 222. Tune like it's going out of style. </li></ul>
  221. 223. Breadth vs Depth <ul>Depth <li>Solve the problem for this exact situation.
  222. 224. Tune like it's going out of style.
  223. 225. Maximum ROI. </li></ul>
  224. 226. Breadth vs Depth <ul>Depth <li>Solve the problem for this exact situation.
  225. 227. Tune like it's going out of style.
  226. 228. Maximum ROI.
  227. 229. Deep understanding of the technology. </li></ul>
  228. 230. Breadth vs Depth <ul>Turns out, you need both. </ul>
  229. 231. People <ul>Specialist vs Generalist </ul>
  230. 232. People <ul>Specialist vs Generalist.^H^H^H^H^... not so much. </ul>
  231. 233. How many people is the right amount?
  232. 234. 50:1?
  233. 235. 100:1?
  234. 236. 200:1?
  235. 237. 2:1?
  236. 238. Gartner Says:
  237. 239. Gartner Says: “Yes”
  238. 240. There is one rule for Server/Admin Ratio:
  239. 242. Axiom 2 <ul>Don't Leverage the Meatcloud </ul>
  240. 243. We've Covered Technology Selection
  241. 244. We've Covered People Decisions
  242. 245. Let's cover Community
  243. 246. Axiom 3 <ul>Decouple your Infrastructure. </ul>
  244. 247. Code for the generic case
  245. 248. Stop reinventing the wheel, srsly.
  246. 249. Moving Forward <ul><ul><ul><li>Can you implement an Open Strategy to solve this issue? </li></ul></ul></ul>
  247. 250. Moving Forward <ul><ul><ul><li>If you have to solve it yourself, can it be Open for others? </li></ul></ul></ul>
  248. 251. Moving Forward <ul>Have an Open Infrastructure </ul>
  249. 252. Moving Forward <ul>Driving on hexagonal wheels isn't fun </ul>http://www.flickr.com:80/photos/andrewmbutler/2762480367/
  250. 253. Moving Forward <ul>Driving on hexagonal wheels isn't fun </ul>http://www.flickr.com:80/photos/andrewmbutler/2762480367/ Ask this guy.
  251. 254. Moving Forward <ul>Driving on hexagonal wheels isn't fun Quit reinventing the wheel...poorly </ul>
  252. 255. Steal it all <ul>Practical Examples of Open Infrastructure </ul>
  253. 256. Steal it all <ul>Practical Examples The Fedora Infrastructure Project <ul><ul><li>https://fedoraproject.org/wiki/Infrastructure </li></ul></ul></ul>
  254. 257. Steal it all <ul>Practical Examples The Community Services Infrastructure Standards <ul><ul><li>http://infrastructure.fedoraproject.org/csi/free-software-policy/en-US/html-single/ </li></ul></ul></ul>https://fedorahosted.org/csi/
  255. 258. De-coupled Infrastructure Benefits <ul><li>Commonly accepted solutions to problem </li></ul>http://www.flickr.com:80/photos/thomas-merton/255204957
  256. 259. De-coupled Infrastructure Benefits <ul><li>Commonly accepted solutions to problem
  257. 260. Portability of solutions </li></ul>http://www.flickr.com:80/photos/thomas-merton/255204957
  258. 261. De-coupled Infrastructure Benefits <ul><li>Commonly accepted solutions to problem
  259. 262. Portability of solutions </li></ul>http://www.flickr.com:80/photos/thomas-merton/255204957
  260. 263. De-coupled Infrastructure Benefits <ul><li>Commonly accepted solutions to problem
  261. 264. Portability of solutions
  262. 265. Ability to hire knowledgeable individuals </li></ul>http://www.flickr.com:80/photos/thomas-merton/255204957
  263. 266. De-coupled Infrastructure Benefits <ul><li>Commonly accepted solutions to problem
  264. 267. Portability of solutions
  265. 268. Ability to hire knowledgeable individuals
  266. 269. Able to retain rock stars </li></ul>http://www.flickr.com:80/photos/thomas-merton/255204957
  267. 270. De-coupled Infrastructure Benefits <ul><li>Commonly accepted solutions to problem
  268. 271. Portability of solutions
  269. 272. Ability to hire knowledgeable individuals
  270. 273. Able to retain rock stars
  271. 274. Community built around tools and support </li></ul>http://www.flickr.com:80/photos/thomas-merton/255204957
  272. 275. 3 Axioms <ul>1. Reuse before building or purchasing 2. Don't leverage the meatcloud 3. Decouple your infrastructure </ul>
  273. 277. Your Time is Valuable
  274. 278. Waste less of it
  275. 279. Your team's time is valuable
  276. 280. Manage it
  277. 281. Your Time is Valuable <ul>Identify tasks on which the team spends the most time. Commonly <ul><li>(Growth) Deployment
  278. 282. Account Management
  279. 283. Ad-Hoc File Transfer type activity
  280. 284. Patches </li></ul></ul>http://www.flickr.com:80/photos/denaldo/3326738675/
  281. 285. Time Evaluation <ul><li>Evaluate your team's time spent on “ Displacement Activities ” </li><ul><li>Fund Raising
  282. 286. Parties/Showers
  283. 287. Volunteer Stuff
  284. 288. Charity </li></ul><li>This is normally done because the person can't actually meet the requirements of $DAYJOB
  285. 289. See Also: People </li></ul>
  286. 290. Pick a Task <ul><li>Focus on one thing
  287. 291. Focus on one thing only
  288. 292. Seriously, this will help </li></ul>
  289. 293. You're ready to automate.
  290. 295. OK, maybe you're not ready to automate.
  291. 296. Don't automate bad process. http://www.flickr.com:80/photos/houseofcards/94935538/in/photostream/
  292. 297. You get bad results.....faster.
  293. 298. Bad processes deliver bad results.
  294. 299. You don't need a panflute.
  295. 301. It starts with requirements
  296. 302. Pick a Task <ul><li>Let's start with system Deployment </li><ul><li>You have some great options for tools
  297. 303. Do you need a tool?
  298. 304. Do you need it automated?
  299. 305. Do you need a pan-flute? </li></ul></ul>
  300. 306. Process Evaluation <ul>Who consumes the process? </ul>
  301. 307. Process Evaluation <ul>What are the outputs? </ul>
  302. 308. Process Evaluation <ul>What are the inputs? </ul>
  303. 309. Process Evaluation <ul>What's unknown? </ul>
  304. 310. Process Evaluation <ul>What's that rare case that you should just leave out? </ul>
  305. 311. Process Evaluation <ul>What's the gotcha you didn't put in because it was hard? </ul>
  306. 312. Gather Requirements <ul>Ask your process consumers about their needs </ul>
  307. 313. Gather Requirements <ul>Ask them to think about outputs, not the process </ul>
  308. 314. Gather Requirements <ul>How do they provide the information? </ul>
  309. 315. Requirements <ul>Formal Specification Document? </ul>
  310. 316. Requirements <ul>Formal Specification Document? <ul><li>Awesome. </li></ul></ul>
  311. 317. Requirements <ul>Formal Specification Document? <ul><li>Awesome. </li><ul><li>I doubt it </li></ul></ul></ul>
  312. 318. Requirements <ul>Hundreds of tickets, IM conversations, phone calls, hallway conversations and just plain old complaining? </ul>
  313. 319. Requirements <ul>Hundreds of tickets, IM conversations, phone calls, hallway conversations and just plain old complaining? <ul><li>Yeah, I thought so. </li></ul></ul>
  314. 320. Requirements <ul>Let's Get Agile. </ul>
  315. 321. Requirements <ul>Let's Get Agile. Development is agile. </ul>
  316. 322. Requirements <ul>Let's Get Agile. Development Infrastructure is agile. </ul>
  317. 323. Requirements <ul>Stories </ul>
  318. 324. Requirements <ul>Use Cases </ul>
  319. 325. Requirements <ul>Test Cases </ul>
  320. 326. Requirements <ul>Stores, Use Cases, Test Case --> Track them </ul>
  321. 327. Requirements <ul>Use a tracker, </ul>
  322. 328. Requirements <ul>Use a tracker, </ul>
  323. 329. Requirements <ul>Use a tracker, use a wiki, </ul>
  324. 330. Requirements <ul>Use a tracker, use a wiki, use a notebook, </ul>
  325. 331. Requirements <ul>Use a tracker, use a wiki, use a notebook, a whiteboard </ul>
  326. 332. Requirements http://www.flickr.com/photos/usonian/106005435/
  327. 333. Requirements <ul>Use a tracker, http://www.pivotaltracker.com </ul>http://pivotaltracker.com
  328. 334. Example Case <ul><li>Application team needs an instance
  329. 335. 2 CPUs
  330. 336. 4 GB RAM
  331. 337. 100G Disk Space
  332. 338. RHEL 5 </li></ul>http://www.flickr.com:80/photos/jimfrazier/1480809458/
  333. 339. Requirements <ul><li>Requirements Story: </li><ul><li>An application team should be able to request capacity. </li></ul></ul>
  334. 340. Requirements <ul><li>Requirements Story: </li><ul><li>An application team should be able to request capacity.
  335. 341. An application team should be able to see the status of their requests. </li></ul></ul>
  336. 342. Requirements <ul><li>Requirements Story: </li><ul><li>An application team should be able to request capacity.
  337. 343. An application team should be able to see the status of their requests.
  338. 344. A capacity planner should approve the request </li></ul></ul>
  339. 345. Requirements <ul><li>Requirements Story: </li><ul><li>An application team should be able to request capacity.
  340. 346. An application team should be able to see the status of their requests.
  341. 347. A capacity planner should approve the request
  342. 348. A system should have multipath from RHEL 5.3 installed </li></ul></ul>
  343. 349. Requirements <ul><li>Requirements Story: </li><ul><li>An application team should be able to request capacity.
  344. 350. An application team should be able to see the status of their requests.
  345. 351. A capacity planner should approve the request
  346. 352. A system should have multipath from RHEL 5.3 installed
  347. 353. A physical system should have a bonded network connection </li></ul></ul>
  348. 354. Design <ul>Review Stories </ul>
  349. 355. Design <ul>Look for common abstractions and workflows </ul>
  350. 356. Design <ul>Abstract the process as an interface. </ul>
  351. 357. Design <ul>Review the process </ul>
  352. 358. Design <ul>Run through the process a couple times manually </ul>
  353. 359. Design <ul>Track issues. </ul>
  354. 360. Design <ul>Do not automate JUNK </ul>
  355. 361. Design <ul>Break work into small units </ul>
  356. 362. Design <ul>Don't be afraid to say 'good enough for version 1' </ul>
  357. 363. Design <ul>Don't be afraid to say 'good enough for version 1' <ul><li>Limit Scope. </li></ul></ul>
  358. 364. Design <ul>Progress over Perfection </ul>
  359. 365. Design <ul>A good enough system today is better than a perfect one that hasn't been invented. </ul>
  360. 366. Design <ul><li>You will iterate again.
  361. 367. You will iterate again. </li></ul>
  362. 368. Design <ul>What points of your process require measurement? </ul>
  363. 369. Design <ul>Can that be automated? </ul>
  364. 370. Design <ul>Can you test for it? </ul>
  365. 371. Design Your Tests <ul>Am I testing directly against my stories? (requirements) </ul>
  366. 372. Design Your Tests <ul>Does it have the right version of Multipath? </ul>
  367. 373. Design Your Tests <ul>Is the network connection bonded? </ul>
  368. 374. Testing
  369. 375. Testing <ul>Provides quality measurements </ul>
  370. 376. Testing <ul>Provides metrics for reporting. </ul>
  371. 377. Testing <ul>Your management team will love it </ul>
  372. 378. Testing <ul>You will have facts. </ul>
  373. 379. Testing <ul>You will have facts. Your team will love it. </ul>
  374. 380. Specifications for Testing <ul>Write Simple Tests </ul>
  375. 381. Specifications for Testing <ul>Unit Test your Configuration </ul>
  376. 382. Specifications for Testing <ul>Watch it FAIL. </ul>
  377. 383. Specifications for Testing <ul>Watch it FAIL. Good tests FAIL often. </ul>
  378. 384. Specifications for Testing http://www.flickr.com/photos/fireflythegreat/2845637227/
  379. 385. Specifications for Testing <ul>Write a policy/method/procedure/function to fix it </ul>
  380. 386. Specifications for Testing <ul>Watch it PASS </ul>
  381. 387. Specifications for Testing <ul>Store Results. </ul>
  382. 388. Test Case <ul><li>Network configuration should be bonded. </li></ul>
  383. 389. Test Case <ul><li>Network configuration should be bonded.
  384. 390. Can I script this? </li></ul>
  385. 391. Test Case <ul><li>Network configuration should be bonded.
  386. 392. Can I script this? </li><ul><li>Given enough Time and Money, it's all scriptable. </li></ul></ul>
  387. 393. Test Case <ul><li>Network configuration should be bonded.
  388. 394. Can I script this? </li><ul><li>Given enough Time and Money, it's all scriptable. </li></ul><li>Can I store the results? </li></ul>
  389. 395. Test Case <ul><li>Network configuration should be bonded.
  390. 396. Can I script this? </li><ul><li>Given enough Time and Money, it's all scriptable. </li></ul><li>Can I store the results? </li><ul><li>Yes </li></ul></ul>
  391. 397. Test Case <ul><li>Network configuration should be bonded.
  392. 398. Can I script this? </li><ul><li>Given enough Time and Money, it's all scriptable. </li></ul><li>Can I store the results? </li><ul><li>Yes </li></ul><li>How Often should I test this? </li></ul>
  393. 399. Test Case #!/bin/bash . config_vars.sh rc=0 # First, if we a VM exit all good if ( lspci | grep -i vmware &> /dev/null ) ; then rc=0 else if ( ! ifconfig -a | grep -i bond0 &> /dev/null ) ; then echo &quot;FAIL: Network interfaces not bonded.&quot; rc=1 fi fi exit $rc
  394. 400. Test Case #!/bin/bash . config_vars.sh rc=0 # First, if we a VM exit all good if ( lspci | grep -i vmware &> /dev/null ) ; then rc=0 else if ( ! ifconfig -a | grep -i bond0 &> /dev/null ) ; then echo &quot;FAIL: Network interfaces not bonded.&quot; rc=1 fi fi exit $rc Site Specific Data
  395. 401. Test Case #!/bin/bash . config_vars.sh rc=0 # First, if we a VM exit all good if ( lspci | grep -i vmware &> /dev/null ) ; then rc=0 else if ( ! ifconfig -a | grep -i bond0 &> /dev/null ) ; then echo &quot;FAIL: Network interfaces not bonded.&quot; rc=1 fi fi exit $rc Site Specific Data Decoupled Logic
  396. 402. Test Case I have results, where do I put them?
  397. 403. Pick One
  398. 404. Example Storage <ul>Ruby using RHN API </ul>http://www.redhat.com/spacewalk
  399. 405. Implementation <ul>Think about re-use <ul><li>Inventory Queries
  400. 406. Asking for/updating configuration info
  401. 407. Authentication Modules </li></ul></ul>
  402. 408. Implementation <ul>Are existing tools close to what you need? </ul>
  403. 409. Implementation <ul>Can you reuse existing code? </ul>
  404. 410. Test Again <ul>After implementation, test again. </ul>
  405. 411. Test Again <ul>After implementation, test again. <ul><li>It's automated right? </li></ul></ul>
  406. 412. Test Again <ul>After implementation, test again. <ul><li>It's automated right? </li><ul><li>So it's quick and painless. </li></ul></ul></ul>
  407. 413. Close the loop <ul>Maintain state of your process </ul>
  408. 414. Close the loop <ul>Maintain state of your components </ul>
  409. 415. Close the loop <ul>If the version of package X changes, and you have a policy for package X, report. </ul>
  410. 416. Close the loop <ul>If the version of package X changes, and you have a policy for package X, report^H^H^H^H^H fix it. </ul>
  411. 417. Close the loop <ul>If don't have a network bond, report it or fix it. </ul>
  412. 418. Maintain <ul>Most fun step </ul>
  413. 419. Deployment Example <ul><li>Host is RHEL 5.3
  414. 420. Host has the correct Version of Multipath
  415. 421. Host should have 4 GB RAM
  416. 422. Host should have 2 CPUs
  417. 423. Host must have a network bond if it's physica </li></ul>
  418. 424. Maintain <ul>Offers most improvement </ul>
  419. 425. Maintain <ul>Customers offer ideas </ul>
  420. 426. Maintain <ul>Other admins complain (offer ideas) </ul>
  421. 427. Maintain <ul>Iterate </ul>
  422. 428. Maintain <ul>Lather, rinse, repeat. </ul>
  423. 429. CI <ul>You need CI. </ul>
  424. 430. Continuous Integration <ul>CI is used in the best development shops in the world </ul>
  425. 431. Continuous Integration <ul>Run upon commits </ul>
  426. 432. Continuous Integration <ul>You are using a VCS right? </ul>
  427. 433. Continuous Integration <ul>You are using a VCS right? <ul><ul><li>Please </li></ul></ul></ul>
  428. 434. Continuous Integration <ul>Track Failures </ul>
  429. 435. Continuous Integration <ul>Any new bugs, require Tests </ul>
  430. 436. Continuous Integration <ul><li>Automate Builds
  431. 437. Self-test the Build
  432. 438. Commit early, commit often
  433. 439. Have a test environment
  434. 440. Report Progress for everybody (laconica?)
  435. 441. Automate Deployment – (usage packages) </li></ul>
  436. 442. Can you have a 'release' of infrastructure? <ul><li>Yes
  437. 443. Plan change-sets
  438. 444. Plan your work-life balance
  439. 445. Work with setting expectations for consumers of your infrastructure
  440. 446. Work on hot/rolling releases
  441. 447. Communicate like a politician, except tell the truth </li></ul>
  442. 448. Some common areas for improvement <ul>A few things to avoid when crafting solutions. </ul>
  443. 449. Things to Avoid <ul>Meatcloud </ul>
  444. 450. Things to Avoid <ul>Automation of Junk </ul>
  445. 451. Things to Avoid <ul>Talking about IT rather than doing IT </ul>
  446. 452. Things to Avoid <ul>Thinking there is one solution </ul>
  447. 453. Things to Avoid <ul>UberTools – Silver Bullets </ul>
  448. 454. Things to Avoid <ul>New Hotness </ul>
  449. 455. Things to Avoid <ul>NIH Syndrome </ul>
  450. 456. Axiom 1 <ul>Reuse before building or purchasing </ul>
  451. 457. Axiom 2 <ul>Don't Leverage the Meatcloud </ul>
  452. 458. Axiom 3 <ul>Decouple your Infrastructure. </ul>
  453. 459. Go From This:
  454. 460. To This:
  455. 463. Sources <ul><li>http://stochasticresonance.wordpress.com/2009/04/01/meatcloud-manifesto/
  456. 464. http://www.boingboing.net/2007/08/22/panflute-flowchart.html
  457. 465. http://gilesbowkett.blogspot.com/2007_05_01_archive.html
  458. 466. http://www.verber.com/mark/sysadm/how-many-admins.html </li></ul>
  459. 467. Suggested Reading <ul><li>http://meetronome.com
  460. 468. http://gist.github.com/161265
  461. 469. https://fedoraproject.org/wiki/Infrastructure
  462. 470. https://fedorahosted.org/csi/
  463. 471. Ship It – Pragmatic Press
  464. 472. Automating Unix and Linux Administration – Apress
  465. 473. Pro Unix Administration – Apress
  466. 474. Pulling Strings with Puppet - Apress
  467. 475. Release IT – Pragmatic Press </li></ul>
  468. 476. More Reading <ul><li>http://groups.google.com/group/agile-system-administration
  469. 477. http://www.melconway.com/research/committees.html
  470. 478. http://status.net/?source=laconica
  471. 479. http://reductivelabs.com/trac/puppet/
  472. 480. https://fedorahosted.org/cobbler/
  473. 481. http://www.mockobjects.com/book/ </li></ul>
  474. 482. Infrastructure is Development

×