Rugged Dev: Building Reliability and Security Into Software


Published on

Presented at InnoTech Austin on October 20, 2011. For details on InnoTech, visit

Published in: Technology
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Rugged Dev: Building Reliability and Security Into Software

  1. 1. The Rugged Way in the Cloud–Building Reliabilityand Security Into Software James Wickett 1
  2. 2. 2
  3. 3. @wickett• Operations and Security for software delivered on the cloud• National Instruments, R&D• Certs: CISSP, GSEC, GCFW, CCSK• Tags: OWASP, Cloud, DevOps, Ruby• Blogger at• I do stuff for LASCON (• Twitter: @wickett 3
  4. 4. Cloud @ NIWe built a DevOps team to rapidly delivernew SaaS products and product functionalityusing cloud hosting and services (IaaS, PaaS,SaaS) as the platform and operations, usingmodel driven automation, as a keydifferentiating element.With this approach we have deliveredmultiple major products to market quicklywith a very small staffing and financial outlay. 4
  5. 5. National Instruments• 30 years old; 5000+ employees around the world, half in Austin, mostly engineers; $873M in 2010• Hardware and software for data acquisition, embedded design, instrument control, and test• LabVIEW is our graphical dataflow programming language used by scientists and engineers in many fields 5
  6. 6. From toys to black holes 6
  7. 7. NI’s Cloud Products• LabVIEW Web UI Builder• FPGA Compile Cloud• more to come... 7
  8. 8. 8
  9. 9. 9
  10. 10. 10
  11. 11. FPGA Compile Cloud• LabVIEW FPGA compiles take hours and consume extensive system resources; compilers are getting larger and more complex• Implemented on Amazon - EC2, Java/Linux,C#/.NET/Windows, and LabVIEW FPGA• Also an on premise product, the “Compile Farm” 11
  12. 12. Using the FPGACompile Cloud 12
  13. 13. BuildingRuggedIn 13
  14. 14. Am I healthy? 14
  15. 15. Am I healthy?• Latest and greatest research• Justification to insurance companies• Measurement and testing as available• Point in time snapshot 15
  16. 16. Am I secure? 16
  17. 17. Am I secure?• Latest and greatest vulnerabilities• Justification of budget for tools• Measurement and testing as available• Point in time snapshot 17
  18. 18. People, Process, Tech 18
  19. 19. It’s not our problem anymore 19
  20. 20. If you want to build a ship, dontdrum up people together to collectwood and dont assign them tasksand work, but rather teach them tolong for the endless immensity ofthe sea- Antoine Jean-Baptiste Marie Roger de Saint Exupéry 20
  21. 21. Twitter SurveyWhat is one word that youwould use to describe ‘ITSecurity’ people? 21
  22. 22. unicorns paranoid prepared Tenacious HAWT! smart masochisticdemented jaded smart sisyphean omnium-gatherum facebored passionate weird drunk compassionate 22
  23. 23. Us vs. Them• Security professionals often degrade developers• Developers don’t get security people• There is interest across the isle, but often ruined by negative language 23
  24. 24. Why do you see the speck that is in yourbrotherʼs eye, but do not notice the log that is inyour own eye? - Jesus 24
  25. 25. Adverse conditionsneed Rugged solutions 25
  26. 26. Adversity fueled innovation• NASA in Space• Military hard drives• ATMs in Europe 26
  27. 27. Chip and PIN ATM 27
  28. 28. The Internets is Mean• Latency• Distribution• Anonymity• Varied protocols• People 28
  29. 29. Systems are complex• “How Complex Systems Fail”• Failure at multiple layers• Synonyms in other industries• Defense in Depth 29
  30. 30. Software needs to meet adversity 30
  31. 31. Intro to Rugged by analogy 31
  32. 32. Current Software 32
  33. 33. Rugged Software 33
  34. 34. Current Software 34
  35. 35. Rugged Software 35
  36. 36. Current Software 36
  37. 37. Rugged Software 37
  38. 38. Current Software 38
  39. 39. Rugged Software 39
  40. 40. Current Software 40
  41. 41. Rugged Software 41
  42. 42. Current Software 42
  43. 43. Rugged Software 43
  44. 44. 44
  45. 45. Rugged Software Manifesto 45
  46. 46. I am rugged... and more importantly,my code is rugged. 46
  47. 47. I recognize that software has becomea foundation of our modern world. 47
  48. 48. I recognize the awesomeresponsibility that comes with thisfoundational role. 48
  49. 49. I recognize that my code will be usedin ways I cannot anticipate, in ways itwas not designed, and for longerthan it was ever intended. 49
  50. 50. I recognize that my code will beattacked by talented and persistentadversaries who threaten ourphysical, economic, and nationalsecurity. 50
  51. 51. I recognize these things - and Ichoose to be rugged. 51
  52. 52. I am rugged because I refuse to be asource of vulnerability or weakness. 52
  53. 53. I am rugged because I assure mycode will support its mission. 53
  54. 54. I am rugged because my code canface these challenges and persist inspite of them. 54
  55. 55. I am rugged, not because it is easy,but because it is necessary... and Iam up for the challenge. 55
  56. 56. Rugged-ities • Availability • Survivability • Defensibility • Security • Longevity • Portability 56
  57. 57. Security vs. Rugged• Absence of • Verification of Events quality• Cost • Benefit• Negative • Positive• FUD • Known values• Toxic • Affirming 57
  58. 58. Rugged Survival Guide • Defensible Infrastructure • Operational Discipline • Situational Awareness • CountermeasuresOn YouTube: “PCI Zombies” 58
  59. 59. Security as a Feature• SaaF is possible, but hard for most products• Tough to measure• Hiding among other features 59
  60. 60. Rugged as a Feature• RaaF addresses to customer felt needs• Values that people covet• Buyers want it 60
  61. 61. Qualities of Rugged Software• Availability - Speed and performance• Longevity, Long-standing, persistent - Time• Scalable, Portable• Maintainable and Defensible - Topology Map• Resilient in the face of failures• Reliable - Time, Load 61
  62. 62. Measuring Ruggedness• Physical: Heat, Cold, Friction, Time, Quantity of use, Type of use• Software: Concurrency, Transactions, Speed, Serial Load, Input handling, Entropy, Lines of Code 62
  63. 63. Measuring Frameworks• Measured by lack of incidents and quantifying risk and vulns• OWASP / CVE tracking• Common Vuln Scoring System (CVSS)• Mitre Common Weakness Enumeration (CWE)• Common Weakness Scoring System (CWSS) 63
  64. 64. Supply and ______ 64
  65. 65. Marketing Possibilities• Positive: Rugged Rating System • 3rd party verification of Ruggedness • Self Attestation• Negative: warning signs• Buyers Bill of Rights 65
  66. 66. Measuring Rugged 66
  67. 67. 3rd Party Warnings 67
  68. 68. Self Attestation 68
  69. 69. Implicit vs. Explicit 69
  70. 70. Explicit Requirements• Customers Demand• 20% Use Cases• Most Vocal• Failure results in loss of customers but not all customers 70
  71. 71. Implicit Requirements• Customers Assume• 80% of use cases• Unsaid and Unspoken• Most basic and expected features• Failure results in a loss of most customers 71
  72. 72. Is Security Explicit or Implicit? 72
  73. 73. Is Rugged Explicit or Implicit? 73
  74. 74. 74
  75. 75. RuggedImplementations 75
  76. 76. build aruggedteam 76
  77. 77. People and Process• Sit near the developers... DevOpsSec• Track security flaws or bugs in the same bug tracking system• Train to automate• Involve team with vendors• Measurement over time and clear communication 77
  78. 78. OPSEC Framework• Know your system and people• Make security better in small steps• Add layers of security without overcompensating• Use a weekly, iteration-based approach to security 78
  79. 79. 79
  80. 80. ProgrammableInfrastructureEnvironment 80
  81. 81. Configuration Management• Infrastructure as Code (IaC)• Model driven deployment• Version control everything• PIE (Programmable Infrastructure Enviroment)• Know Your Environment if you want to make it defensible 81
  82. 82. What is PIE?• a a framework to define, provision, monitor, and control cloud-based systems• written in Java, uses SSH as transport, currently supports Amazon AWS (Linux and Windows)• takes an XML-based model from source control and creates a full running system• to define, provision, monitor, and control cloud-based systems 82
  83. 83. PIE ingredients• model driven automation• infrastructure as code• DevOps• dynamic scaling• agility• security in the model 83
  84. 84. 84
  85. 85. The Model• XML descriptions of the system as ‘specs’• system (top level)• environment (instance of a system)• role (“tier” within a system)• image (specific base box config)• service (specific software or application)• commands (for various levels)• templates (files to be parsed) 85
  86. 86. 86
  87. 87. 87
  88. 88. The Registry• uses Apache Zookeeper (part of Hadoop project)• the registry contains information about the running system• specific addressing scheme:• /fcc/test1/external-services/2/tomcat• [/<system>/<environment>/<role>/<instance>/<service>] pie registry.register /fcc/test1/external-services/2 pie registry.bind /fcc/test1 pie registry.list /fcc/test1 88
  89. 89. Control • create, terminate, start, stop instances using the AWS API • enforce scaling policy • execute remote commandspie control.create /fcc/test1/external-services/2pie control.stop /fcc/test1/external-services/2pie control.enforce /fcc/test1pie control.remote.service.restart /fcc/test1/external-services/2/external-tomcatpie control.remote.execute /fcc/test1/external-services/2 –i exe[0]=“ls –l /etc/init.d” 89
  90. 90. Provisioning • deploy services and apps • two-phase for fast deploys • update config files and parse templatespie provision.deploy.stage /fcc/test1/external-services/2 –i pack[0]=lvdotcom-authpie /fcc/test1/external-services/2 –i pack[0]=lvdotcom-authpie provision.remote.updateConfig /fcc/test1 90
  91. 91. Monitoring• integrated with third party SaaS monitoring provider Cloudkick• systems register with Cloudkick as they come online and immediately have appropriate monitors applied based on tags set from the model 91
  92. 92. 92
  93. 93. Logging• logging in the cloud using splunk• logging agents are deployed in the model and they are given the config from registry and the model as they come online 93
  94. 94. Rugged Results• repeatable – no manual errors• reviewable – model in source control• rapid – bring up, install, configure, and test dozens of systems in a morning• resilient – automated reconfiguration to swap servers (throw away infrastructure)• rugged by design 94
  95. 95. buildthe newDMZ 95
  96. 96. What’s a DMZ?• Demilitarized Zone• Physical and logical divisions between assets• Military history• Control what goes in and what goes out 96
  97. 97. Control your environment• Make every service a DMZ• Cloud environment• 3-tier web architecture• Allow automated provisioning 97
  98. 98. Traditional 3-Tier Web Architecture Firewall Web Web Web DMZ 1 Firewall Middle Tier Middle Tier DMZ 2 Firewall DB LDAP DMZ 3 98
  99. 99. Rugged Architecturefirewall firewall firewall Web Web Web DMZ x3 firewall firewall Middle Tier Middle Tier DMZ x2 firewall firewall DB LDAP DMZ x3 99
  100. 100. firewall firewall firewall Web Web Web Repeatable firewall firewall Verifiable Middle Tier Middle Tier Prod/Dev/Test Matching firewall firewall Controlled DB LDAP Automatedfirewall firewall firewall firewall firewall firewall Web Web Web Web Web Web firewall firewall firewall firewall Middle Tier Middle Tier Middle Tier Middle Tier firewall firewall firewall firewall DB LDAP DB LDAP 100
  101. 101. firewall firewall firewall firewall firewall firewall firewall firewall firewall Web Web Web Web Web Web Web Web Web firewall firewall firewall firewall firewall firewall Middle Tier Middle Tier Middle Tier Middle Tier Middle Tier Middle Tier firewall firewall firewall firewall firewall firewall DB LDAP DB LDAP DB LDAPfirewall firewall firewall firewall firewall firewall firewall firewall firewall Web Web Web Web Web Web Web Web Web firewall firewall firewall firewall firewall firewall Middle Tier Middle Tier Middle Tier Middle Tier Middle Tier Middle Tier firewall firewall firewall firewall firewall firewall DB LDAP DB LDAP DB LDAPfirewall firewall firewall firewall firewall firewall firewall firewall firewall Web Web Web Web Web Web Web Web Web firewall firewall firewall firewall firewall firewall Middle Tier Middle Tier Middle Tier Middle Tier Middle Tier Middle Tier firewall firewall firewall firewall firewall firewall DB LDAP DB LDAP DB LDAP 101
  102. 102. Rugged 3-Tier Architecture Benefits• Control• Config Management• Reproducible and Automated• Data can’t traverse environments accidentally• Dev and Test Tier accurate 102
  103. 103. OWASP Secure Coding Quick Reference Guide• Checklist format that can be added to into your sprints• Helps development team find common security flaws• Topics include: Input Validation, Output Encoding, Auth, Session Management, Memory Management, ...• 103
  104. 104. Rugged Next Steps• Use Rugged language• Know your systems• Automate, track results, repeat• Begin weekly OPSEC in your org• Attend LASCON ( 104
  105. 105. Rugged Resources 105
  106. 106. h"ps://­‐so4ware 106
  107. 107. Recommended Reading 107