Free Yourself with CloudFoundry: A Private Cloud Experience
Upcoming SlideShare
Loading in...5
×
 

Like this? Share it with your network

Share

Free Yourself with CloudFoundry: A Private Cloud Experience

on

  • 888 views

Speakers: Mike Heath, Shawn Nielsen and Mike Youngstrom ...

Speakers: Mike Heath, Shawn Nielsen and Mike Youngstrom
Cloud Foundry makes managing and deploying applications incredibly simple. However, deploying Cloud Foundry itself can be a challenging task. We will be sharing what we learned deploying Cloud Foundry and what it took to win over our organization.
Learn from our experiences deploying Cloud Foundry with BOSH and integrating with our existing enterprise infrastructure. We will discuss:
Developing and customizing Cloud Foundry while staying in sync with the open source repositories Building custom Cloud Foundry services using Java and Spring Improved gathering application diagnostics by simplifying JMX and remote debugging support in Cloud Foundry Enhanced security and auditability with application level firewalls
Come learn from our successes as well as our mistakes.

Statistics

Views

Total Views
888
Views on SlideShare
883
Embed Views
5

Actions

Likes
4
Downloads
39
Comments
0

2 Embeds 5

https://twitter.com 4
http://192.168.33.10 1

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

Free Yourself with CloudFoundry: A Private Cloud Experience Presentation Transcript

  • 1. Free Yourself with Cloud Foundry A Private Cloud Experience Mike Heath, Shawn Nielsen, Mike Youngstrom © 2013 SpringOne 2GX. All rights reserved. Do not distribute without permission.
  • 2. Disclaimer This presentation does not represent the views, opinions, policies, nor direction of The Church of Jesus Christ of Latter-day Saints. These views are the sole views of the presenters involved with the presentation. We take full responsibility for content presented and any errors or incorrect perception of representation.
  • 3. Description / Overview • This presentation is about our experience integrating Cloud Foundry into our organization. • It is not a tutorial on how to deploy CF.
  • 4. Presentation Roadmap • • • • • • • Why we chose Cloud Foundry? Cloud Foundry APIs in a UI Gap analysis Develop and deploy Authentication & diagnostics Custom services Application level firewalls
  • 5. Business Problems • Slowness in provisioning time • 100s of small/medium apps • Difficult to manage infrastructure for so many apps • More interested in fault tolerance over scale for most apps
  • 6. The PaaS Team • 2 ½ Developers • ½ Manager • 3 Operations personnel • Passion for efficiency
  • 7. Current State • Currently in beta – Moving to enterprise ready in Q4 • Good feedback so far • Applications – 4 production – 14 development
  • 8. Why we went with PaaS • • • • • Developer productivity Fault tolerant and scalable Simplification of infrastructure Consistent deployments across runtimes Self-service
  • 9. Why Cloud Foundry?
  • 10. Other Reasons for Cloud Foundry • Infrastructure agnostic – Avoids vendor lock-in • Public/private • Cloud on cloud – PaaS running on an IaaS cloud
  • 11. Other Reasons We Love Cloud Foundry • Open source – Ability to adapt Cloud Foundry to our needs – Community involvement • Architecture – Good architecture sells itself
  • 12. Overview of Environment • • • • • • vSphere F5 (load balancer) Oracle LDAP Proxy based SSO Primarily Java+Spring shop – Starting to see some NodeJS
  • 13. Cloud Foundry Adoption Challenges
  • 14. Cloud Foundry Adoption Challenges • Convincing the systems engineers – VM centric world to app centric world
  • 15. Network Zoning and Firewalls • Network firewalls • Transition to host / app firewalls
  • 16. Network Firewalls
  • 17. Host / App Level Firewalls
  • 18. Cost Challenges • Current costs and billing models – Multi-year bill-out • How do you know what you’re going to use in 4 years? You don’t!
  • 19. Quota and Utilization Bill-back Know your bill $25 per GB RAM per month Know your usage
  • 20. Trade-in Model
  • 21. Organizational Efficiencies
  • 22. Presentation Roadmap • • • • • • • Why we chose Cloud Foundry? Cloud Foundry APIs in a UI Gap analysis Develop and deploy Authentication & diagnostics Custom services Application level firewalls
  • 23. Billing and Usage Through CC APIs CloudController API: url: "/v2/quota_definitions/36673f76-c617-4ae8-94b9-7adccb747ced” entity: { name: "Enterprise Management Organization - Quota“ non_basic_services_allowed: false total_services: 100 memory_limit: 40960 trial_db_allowed: false }
  • 24. Custom UI • Just like everyone else…we have our own UI. • Why? – No standard UI available. Though several community UIs on their way. – No way to do access management through CF. Why Not? – Management of quotas for organizations – Give management something visual to look at
  • 25. Cloud Foundry APIs CC URL: /v2/organization/<guid> entity: { name: “MySpringOrganization" quota_definition_url: "/v2/quota_definitions/<guid> spaces_url: "/v2/organizations/<guid>/spaces” domains_url: "/v2/organizations/<guid>/domains” users_url: "/v2/organizations/<guid>/users” managers_url: "/v2/organizations/<guid>/managers” auditors_url: "/v2/organizations/<guid>/auditors” app_events_url: "/v2/organizations/<guid>/app_events” }
  • 26. s
  • 27. Key Performance Indicator (KPI) Data
  • 28. Cloud Foundry APIs CC URL: /v2/apps/<guid>/stats Quota Live Usage state: "RUNNING" stats: { uptime: 984643 mem_quota: 536870912 disk_quota: 1073741824 fds_quota: 16384 } usage: { time: "2013-09-10 02:05:27” cpu: 0.0007454006633748 mem: 211116032 disk: 116498432 }
  • 29. Operations Center
  • 30. UI Demo
  • 31. Presentation Roadmap • • • • • • • Why we chose Cloud Foundry? Cloud Foundry APIs in a UI Gap analysis Develop and deploy Authentication & diagnostics Custom services Application level firewalls
  • 32. Gaps
  • 33. Authentication
  • 34. Diagnostics
  • 35. Enterprise Services and Legacy
  • 36. Presentation Roadmap • • • • • • • Why we chose Cloud Foundry? Cloud Foundry APIs in a UI Gap analysis Develop and deploy Authentication & diagnostics Custom services Application level firewalls
  • 37. Deployment • Use BOSH – Fork cf-release – Proxy? • Environments: – X Dev – 1 Test – 1 Prod • Customers only use prod • Break deployments into pools – Core – DEA/Router-X
  • 38. Development • Dev is prod but smaller – – – – Use BOSH vSphere F5 SSL – kind of • Develop on component? – `bosh stop` existing component – Configure local component in place • Other options – Vagrant? – Warden CPI?
  • 39. Presentation Roadmap • • • • • • • Why we chose Cloud Foundry? Cloud Foundry APIs in a UI Gap analysis Develop and deploy Authentication & diagnostics Custom services Application level firewalls
  • 40. Customizing Authentication • UAA & Login – Java Spring + Spring Security • Customized UAA – Added UAA to src release – Added spring config • Login Server – May re-visit
  • 41. Buildpack Customization • Vital extension point • Support for legacy artifacts • Just fork and tweak – – – – Proxies Oracle support for Node Pre-processed `npm install` Add JMX support
  • 42. Problem: Diagnostics • Need: – – – – Thread dumps Heap dumps Remote debug APM • Current story? Not good – Logging – Console port??? – Debug port???
  • 43. Solution: Caldecott? • Use console and debug ports • Caldecott – TCP over HTTP proxy – Inter-app communication • Diagnostic – Router/gateway/server
  • 44. Solution: Diagnostics Server Authorize Request UAA/Cloud Controller Proxy to DEA Request Proxy DEA Forward to Warden on Debug or Console Port Diagnostic Server Dev Workstation Warden
  • 45. Demo Remote Debug
  • 46. JMX • JMX is great – – – – Heap dumps Thread dumps Basic profiling Management operations • JMX over RMI lacking – JMXMP to the rescue • Configure JMX in buildpack – Bind to `console` port • Created `cf` plugin to launch Visualvm
  • 47. JMX Demo
  • 48. Diagnostics Summary • Caldecott – Doesn’t work for us • Diagnostic Server • Safe customization? – Removed ports
  • 49. Presentation Roadmap • • • • • • • Why we chose Cloud Foundry? Cloud Foundry APIs in a UI Gap analysis Develop and deploy Authentication & diagnostics Custom services Application level firewalls
  • 50. Custom Services • Integrate with existing systems – LDAP, NFS/CIFS, Oracle, SSO, Web Services, etc. • New ‘user-provided’ service is not adequate – Will still work for most people – We need more than a set of key/value pairs
  • 51. Custom Services Built on Java and Spring • We’re a Java shop. • Java is more enterprise friendly. – Client libraries for existing systems • Our team consists of Java/Spring developers.
  • 52. Custom Service Framework Features • NATS Client • Cloud Foundry utilities – Type-safe NATS messaging – PID file support for working with Monit/BOSH – YAML config • Cloud Controller client for invoking service APIs • Service Broker • Open source – https://github.com/cloudfoundry-community/java-nats – https://github.com/cloudfoundry-community/cf-java-component
  • 53. NATS Java Client • NATS is the distributed pub/sub messaging system used by Cloud Foundry. • NATS support is essential for customizing Cloud Foundry. • Java client – Built using Netty – Integrates with Spring • Generic NATS client, nothing Cloud Foundry specific
  • 54. NATS Client Sample Code // Connecting NATS to server Nats nats = new NatsConnector() .addHost("nats://localhost") .connect(); // Simple subscriber nats.subscribe("foo", (message) -> { System.out.println("Received: " + message); }); // Simple publisher nats.publish("foo", "Hello world!");
  • 55. NATS Client Spring Integration <nats:nats> <nats:url>nats://localhost:4222</nats:url> </nats:nats> @EnableNatsAnnotations @Configuration public class Configuration { @Subscribe("foo") public void onMessage(Message message) { System.out.println(message); } }
  • 56. Spring Boot/NATS Example • NATS to HTTP Gateway – Using Spring Boot – Single Java source file – Less-than 75 lines of Java (including import statements.) https://github.com/mheath/spring-boot-nats-example
  • 57. Cloud Foundry Specific NATS (cf-nats) • Simplifies using NATS with Cloud Foundry • Type-safe NATS messaging – Component discovery – Router registration – Staging notifications • Still a work in progress https://github.com/cloudfoundry-community/cf-java-component/tree/master/cf-nats
  • 58. PID File • Support for creating .pid file at Spring context initialization • Useful for working with Monit – BOSH uses Monit <cf:pid-file resource=“file:/var/run/component/my-cf-component.pid” />
  • 59. YAML Configuration • Existing Cloud Foundry components use YAML for configuration. • YAML is cool. • We provide Spring integration for using YAML. – XML configuration for loading YAML as properties. – Java configuration for using YAML is a Spring PropertySource.
  • 60. Service Broker Framework • Enables creating custom services • Creating service broker requires implementing a single Java interface • Simple Cloud Controller client for invoking service APIs • Provides APIs for automatically registering with Cloud Controller
  • 61. Example Service Broker • Built on Spring Boot • http://github.com/cloudfoundry-community/java-servicebroker-example
  • 62. Example Service Broker Demo
  • 63. Oracle Service Demo
  • 64. Custom Oracle Service • Prompts for: service name, schema/user, password • Produces credentials: "credentials“ : { "schema":"CF_DEV2", "ldap“ : {"host":“fake-ldap.lds.org", "port":389, "context":"cn=OracleContext…"}, "firewall":[{"port":1234,"host":“oracle-scan-host”}…], "descriptor":"(DESCRIPTION=(ADDRESS_LIST=…", "alias":“DB-DEV2", "service":“service134", "addresses":[{"host":“oracle-scan-host","port":"1234"}], "jdbcUrl":"jdbc:oracle:thin:@ldap://fake-ldap/DB-DEV2,cn=OracleContext,…", "password":"super-secret-password"}}
  • 65. Custom Service Creation Workflow Holds service data (host: ‘foo’, port 1234) cf plugin sends data to broker (host: ‘foo’, port: 1234) Service Broker `cf` client Broker correlates service data Cloud Controller tells broker originally sent by cf plugin and to create service instance. returns service credentials to Cloud Controller. cf creates service cf plugin prompts for service data Cloud Controller
  • 66. Presentation Roadmap • • • • • • • Why we chose Cloud Foundry? Cloud Foundry APIs in a UI Gap analysis Develop and deploy Authentication & diagnostics Custom services Application level firewalls
  • 67. Firewall Requirements • Proxy based SSO requires control over incoming requests • Minimize attack vectors from compromised/malicious apps – Limit access to console and debug ports • Allow access to high risk network zones • Auditability – What network resources does each app have access to? • Minimize customization of Cloud Foundry
  • 68. Proxy Based Single Sign-On (SSO)
  • 69. Warden • Used by DEA to securely host applications • Applications run with a “Warden Container” • Warden Containers isolate resources – Network – Disk – Memory • Think of a Warden Container as a “lightweight VM.”
  • 70. Controlling Incoming Requests • Load Balancer and Cloud Foundry routers easily secured. • Securing requests to Warden Containers more difficult – Need to allow incoming requests from routers – Disallow outgoing requests directly to routers • Cloud Foundry scales dynamically – Routers can be added/removed from the system at any time – Changing static configuration of all DEAs when router is added or removed is unacceptable
  • 71. Custom DEA Incoming Firewall • Configure Warden to drop all incoming and outgoing packets • Customized DEA to track Cloud Foundry routers – Routers already broadcast presence via NATS – Use existing NATS messages to build router registry • Warden uses `iptables` to isolate network resources – `iptables` are Linux kernel level network rules – Our firewalls piggybacks on Warden’s `iptables` rules
  • 72. Custom DEA Outbound Firewall • Service credentials embed firewall rules • Use Warden API to open firewall holes • Facilitated by custom services "credentials" : { ... "firewall" : [{ "network" : "10.118.50.0/24", "port" : 8080 }] }
  • 73. Securing Outbound HTTP • Problems – Single IP address can host multiple web sites/applications – Public web services often have large pool of IP addresses that can change over time • Solution – HTTP proxy for hostname validation • Custom service firewall rules enables access to the HTTP proxy "credentials" : { ... "firewall" : [{ “http" : “http://www.google.com" }] }
  • 74. Http Proxy Firewall Workflow DEA Internet Warden Proxies HTTP requests HTTP Proxy Network Firewall Intranet
  • 75. Summary • • • • • • Cloud Foundry is awesome! Cloud Controller APIs make creating a UI easy. Cloud Foundry authorization customization is maturing. Diagnostics need work. Our solution was promising. You can create services in Java. We isolate our applications in app firewalls
  • 76. Questions?
  • 77. Citation references • • • • • • • • • • • • • • • • • Cloud picture (slide 3) Copyright 2007, Karin Dalziel. Creative Commons licensed. Snail picture (slide 4) Copyright 2013, Eirien (Ilweranta). Create Commons licensed. Wrench photo (slide 5) Copyright 2009, zzpza. Creative Commons licensed. Building photo (slide 10) Copyright 2010, Ricardo Diaz. Creative Commons licensed. Lightening road screen do not enter picture (slide 12) Copyright 2008, Bobby. Creative Commons licensed. Gap (slide 28) Copyright 2013, Upupa4me. Creative Commons licensed. “Key Note” (slide 29) Copyright 2009, William Neuheisel. Creative Commons licensed. “Angry Computer Support Worker Baning His Fists On His Desk” (slide 30) Copyright 2010, jfcherry. Creative Commons licensed. “24 Hour Service” (slide 31) Copyright 2008, Natalie Maynor. Creative Commons licensed. “Launch of Atlas V TDRS-K from Cape Canaveral AFS” (slide 32) Copyright 2013 NASA Goddard Space Flight Center. Creative Commons licsened. “American Architect, 1909” (Slide 33) Copyright 2011 REVIVALthedigest. Creative Commons licensed. “Keys” (Slide 34) Copyright 2005 mmarchin. Create Commons licensed. “Lego building” (Slide 35) Copyright 2006 Matt Bateman. Creative Commons licensed. “Heavy Metal: TDK MA-R90 Cassette Tape” (Slide 39) Copyright 2012 Scott Schiller. Creative Commons licensed. “Tunnel” (Slide 41) Copyrigh 2010 Dushan Hanuska. Creative Commons licensed. “Roadside Service” (Slide 42) Copyright 2012 Pam Morris. Creative Commons licensed. “Typical Prison Guard” (Slide 60) Copyright 2009 Son of Groucho. Creative Commons licensed.