Improving DevOps through Cloud Automation and Management - Real-World Rocket Science with Chef, Ostrato & OpenWhere
•Download as PPTX, PDF•
1 like•991 views
Explore how DevOps processes can be made more efficient through improved service delivery and cloud automation. Check out this real-world example to see how Chef and Ostrato helped OpenWhere, a geospatial analytics startup, compete in the hyper-competitive defense marketplace. Chef allows enterprises like OpenWhere to automate infrastructure deployments to accelerate and simplify the development process. Ostrato’s cloud management platform enables enterprises to control costs and institute governance in hybrid cloud environments.
Recommended
More Related Content
What's hot
What's hot (20)
Viewers also liked
Viewers also liked (20)
Similar to Improving DevOps through Cloud Automation and Management - Real-World Rocket Science with Chef, Ostrato & OpenWhere
Similar to Improving DevOps through Cloud Automation and Management - Real-World Rocket Science with Chef, Ostrato & OpenWhere (20)
Recently uploaded
Recently uploaded (20)
Improving DevOps through Cloud Automation and Management - Real-World Rocket Science with Chef, Ostrato & OpenWhere
- 1. Real-World Rocket Science with Chef and Ostrato
- 2. Nicolas Rycar, Automation Engineer
- 3. 1990 1991 1992 1993 1994 1995 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 2014 2015 1980 1981 1982 1983 1984 1985 1986 1987 1988 1989 Computing Resources Have Been Increasing Exponentially Virtual Nodes Physical Hardware 20 40 60 80 100 120 Millions 1980 Mainframe 1990 Client/Server 2010+ Web-Scale 2000 Datacenter MillionsofServers
- 4. Increased Size Leads to Operational Complexity WEB SERVERS APPLICATION SERVERS DATABASE ADD 1 SERVER 20+ Changes 12+ New dependences
- 5. How does Chef work? • Ensures desired state by continually testing and repairing individual resources in the system • You compose policies using a series of simple declarations • The Chef client fetches those policies from a central server and applies them to the local machine • The state of the machine is recorded and sent back to a database, where it is indexed for search, reporting, and audit.
- 6. Chef Client Checks In Node Chef Server
- 7. Policy is Stored on a Central Server Node Chef Server "recipe[ntp::client]" "recipe[users]" "role[webserver]"
- 8. Chef Client Pulls New Policy and Applies It Chef Server "recipe[ntp::client]" "recipe[users]" "role[webserver]"
- 9. The Chef Software Platform Chef Development Kit Cookbook and Policy Authoring Test-Driven Infrastructure Chef Server Management Console Analytics Platform High Availability and Replication Chef Client Nodes Data Center The Cloud
- 10. Dale Wickizer, Chief Technology Officer
- 11. Private Cloud Business Group A Business Group B IT Top Challenges: - Governance - Orchestration - Controlling Costs - Continuous Delivery Market Problem
- 12. Our Design Philosophy • Build a powerful, cloud service management platform: • Seamless operations across public & private clouds • Simple-to-use • Open Source • Deliver immediate business value – Strong, global policies – Rich product features – Role-based Access Controls (RBAC) • Great user experience • User-specific marketplaces (multi-tenant) • Same intuitive actions and workflows, regardless of CSP
- 13. Self-Service Portals Governance Engine Cloud Operations Our Solution Ostrato cloudSM API Abstraction Layer
- 14. Self-Service Portals Governance Engine Cloud Operations Our Differentiation Ostrato cloudSM API Abstraction Layer
- 15. What is Ostrato cloudSM? GET /parking_calendars 200 OK [ { "name": "Schedule A", "id": <id>, "calendar_url": <url>, "times": { With GUI With API C O N T R O L One Pane to Govern Cloud Services
- 16. Automation & Governance in DevOps • Organizations struggle to combine dev & QA processes with IT operations (a.k.a,“DevOps”) • Business problem: Move application changes to production faster, without sacrificing: • Quality • Governance • Reporting & Visibility • Cost Controls • Security Customer Expectation Continuous Delivery
- 18. Real DevOps Is A Lot More Complicated ..
- 19. Key: Policy-Driven Automation Ostrato Chef Self-Service Marketplaces • No Scripting • Template-driven Strong Governance • RBAC-driven Global Policies • Workflow Approval CSP-independent Provisioning • Fast & Repeatable • API-Driven Configuration Management: • Powerful • Scalable
- 20. Andrew Heifetz, Chief Cloud Officer
- 21. Who is Andrew? • Chief Cloud Officer for OpenWhere • 20+ years serving Fortune 500, Public Sector, and high growth new ventures across multiple sectors including telecommunications, media and entertainment, remote sensing, defense, and intelligence. • Held Senior Management Consulting Roles at Ernst & Young's Center for Technology Enablement and leadership at various start-up companies. • Aerospace Engineer
- 22. 24x7 video streaming from the International SpaceStation: Ten months from whiteboard to Initial Operations
- 23. Ground Terminal 1 CUSTOMER SUPPORT CENTER Ground Terminal 2 Ground Terminal n MISSION OPERATION CENTER Cloud Data Center 2 Cloud Data Center 1 NETWORK OPERATIONS CENTER Typical ground system environment has high degree of operational complexity
- 24. Typical ground system has a high degree of operational complexity • 100’s - 1000’s of servers • 4 Types of databases clusters • 2 HPC clusters • 17 VLANS • 7 internal firewalls • Hardened Windows and Linux Images • 9 Major COTS Packages • 15 Custom applications in five languages • 3 NAS devices – Petabyte level storage • Multiple locations including public and private Clouds
- 25. Systems Engineering and Program Management require multiple environments throughout the mission... • Need multiple, concurrent environments - Large systems require multiple copies of the environment to support concurrent activities. The different environments can include functional testing, pre-integration (component-to component testing), system integration, training, performance, user acceptance, training, and production simulation. • Support Out of cycle, ad-hoc testing needs – Emergency production fixes, critical security patches, and other mission events can trigger activities that require on-demand environments to support these ad-hoc test requirements. • Mimic production environment – test environments should be as close to the production configuration as possible in order to validate nonfunctional requirements like high availability, recovery, performance, etc.
- 26. ...but have limited time & budget for support.
- 27. And then the mission can change at a moments notice
- 28. AWS, Chef, and Ostrato made it possible to accelerate the development life cycle. • Too expensive to maintain multiple environments • Over 50% difference in configurations between environments • Resources are focused on production, alternative environments are secondary 1. Use AWS for low cost, on- demand infrastructure 2. Use Chef to capture environment configurations as software 3. Use Ostrato to provide self-service and cost management Traditional Approach OpenWhere Approach
- 29. Our approach required all three capabilities to meet the requirements while reducing costs & schedule Infrastructure ConfigurationInfrastructure Provisioning Orchestration & Governance
- 30. Version Control Continuous Integration Chef Server Ostrato Virtual Private Cloud Create Initial System from marketplace using HEAT/Cloud Formation Templates Virtual servers include Chef client which checks in 1 2 Developer checks in recipe Continuous integration and unit tests Deploy to Chef Server 3 4 5 The three components working together
- 31. Use Purpose Built Environments Fixed, Static Environments – Average 50% variance between production & lower environments – Supporting environments have variable demand, low overall utilization, & minimal support Dynamic, on-demand environments – Built and scaled for specific purposes – Chef ensures no infrastructure variance between environments – Ostrato was used to orchestrate the lifecycle of the environments Fixed Environments Development Staging Integration Demonstration Test /QA Training Production Etc. Dynamic Environments Development for Sprint 11 (3 weeks) User Test for User Story US 217 (14 hours) Regression Test for Defect 42 (1 hour) Performance Test for release 2.3.1 (4 hours) Training for release 2.3.2 (8 hours/day) Production for release 2.3.0 (1 month)
- 32. Create Programmatic Bill of Material The entire system is captured as a software code. This allows the infrastructure to be version controlled and replicated like any other software asset.
- 33. Provide Self Service for the entire System (not just servers) Single server is not a viable unit of work “In today’s distributed compute environment, developers can’t develop on local workstations.” Teams want self-service to full systems, not servers “I want an entire system not 12 servers, 2 subnets, database, NAT, load balancer, etc” Teams aren’t good about clean up, so need guard rails (governance)
- 34. Summary • 1st Program where infrastructure wasn’t a bottleneck • Create parallel environments – De-conflicts development activities – Reduces schedule pressure – Increases agility • Need all three capabilities to be successful – AWS: Cloud Infrastructure – Chef: Infrastructure Configuration Management – Ostrato: Orchestration and governance
- 35. Q & A Dale Wickizer CTO, Ostrato dwickizer@ostrato.com @dalewickizer Nicolas Rycar Automation Engineer, Chef rycar@chef.io @rycar Andrew Heifetz Chief Cloud Officer, OpenWhere aheifetz@openwhere.com @andyheifetz Contact Information
Editor's Notes
- Brock – give brief introduction, then pass off to Nick
- Frame the problem - growing infrastructure - fewer resources - can't watch everything - surface problems faster
- ----- Meeting Notes (9/15/14 13:28) ----- - infrastructure more dynamic - higher rate of change requires more data to allow for safety
- When the chef-server run on the node it asks the chef server what policy should I follow, or what is my run list
- Chef server looks at the run_list and sends the appropriate cookbooks to the node
- Chef-client then executes the resources on the run list All the heavy lifting is done on the node!! the server is quite light weight! This is super important, especially for customers who work at scale. Yahoo now has 140,000 nodes converging against a single Chef server HA instance. Facebook regularly converges between 10-15k nodes at a time. We are also the only company in the market space to offer a hosted version of our product; part of the reason we’re able to do this is because our technology scales so well!
- Read this diagram from left to right, bottom to top, as follows: Systems Administrators and Software Developers write Cookbooks and Policy describing how to build, deploy, and manage infrastructure and applications as code They use best-of-breed tools to test their changes before publishing them, ensuring quality throughout the entire technology stack They publish to a Chef Server, which stores the cookbooks and policy and scalably distributes them Less technical users can set policy and configure infrastructure and applications through our web based management console - not everyone is an author The Chef Server can be made highly available within a single failure domain, and multiple failure domains (such as between data centers) can be replicated Activity on the Chef Server is tracked by our analytics platform - every API call and user interaction is stored Chef Client Nodes (a Server, VM, or Container) retrieve policy from the Chef Server, configure themselves appropriately, and submit information about their state to the chef server, where they are indexed for search All information about what changed on a node is stored in our analytics platform - every resource, recipe, and policy application, for every run WE COVER THE WHOLE EXPERIENCE - FROM DEVELOPER LAPTOP TO PRODUCTION SERVER
- Brock – introduce Dale
- The good news is cloud adoption is taking off like crazy. The bad news is cloud adoption is taking off like crazy. People from different groups within an organization are consuming cloud, but not through their CIO organization. That’s resulting in all sorts of challenges in terms of governance and controlling costs, as well as the challenge of doing test and deployment automation across diverse cloud environments.
- To tackle that problem, our approach was to build a powerful cloud service management platform, that is simple-to-use, leverages Open Source tools and provides seamless operations across cloud providers. We wanted to focus on governance to deliver a rich set of global policies that deliver immediate business value. We also wanted to provide a great user experience: User-specific marketplaces (based on RBAC) We want 80-90% of the workflows to be the same, and only grudgingly do we want to do anything CSP-specific Where we can overcome CSP limitations, we will; where we can provide innovations the CSPs never dreamed of, we will Put the most common cloud service management tasks at your fingertips on our Cloud Services Page, so you don’t waste time Strive to make reporting and cost analysis clear and actionable Most of all, our multi-tenant environments must be secure
- The result is cloudSM, our service management platform. Think of it as a cross-Cloud governance and service management abstraction layer. Security and policies are driven through our governance engine on the backend. We are interfaced to the top cloud service providers and to enterprise automation applications, such as Chef, through our orchestration engine. Self-service portals for users are provided through our web client, via a REST API.
- We view integration into the various cloud providers as “table stakes”. Our differentiation comes in our self-service portals, which are customized for each user, our API abstraction layer, which allows the platform to be used an automated fashion, and through our innovative global policies. By “global” we mean exact same policies work across all cloud service providers.
- We open up the same REST API allowing customers to interface with our server directly, so that we can help them automate their processes.
- One important area for us is that of DevOps, as companies strive to deliver applications and revisions to those applications must faster, without sacrificing quality, cost controls, security and visibility into the process. [Click]. Increasingly they are being driven by the expectations of their customers for continuous delivery.
- There is a lot of buzz around continuous delivery. In the perfect world, development, QA & IT teams come together as one team, enabled by the latest technology to dramatically shorten their development/deployment cycles.
- Real DevOps, however, is often much more complicated. Technology: Great enabler, but …Governance and process cannot be ignored. They, too, also need to be addressed. Thinking needs to change.
- Reducing this complexity is where Chef and Ostrato can bring a great deal value. Continuous delivery can be securely governed using the Ostrato’s global policy engine. Ostrato’s service templates can be used to drive a high degree of automation in provisioning without the need for scripting. Likewise, Chef also can enforce policies from the outside, as well as provide sophisticated provisioning and extremely powerful and scalable configuration management in accordance with policies provided. The two together can take what is normally 9 or 10 step continuous delivery pipeline, often involving a lot of scripting and shrink it down a 5-step process requiring very little scripting. The combination of the two technologies is quite formidable.
- Brock – introduce Andy
- Brock – moderate Q&A for all presenters.