Agile Development at GoGrid with Pallet and JClouds

Complex Infrastructure Made Easy™Agile Development at GoGrid with Pallet and JClouds Presented to JClouds Meetup @ Twitter’s HQ Toni Batchelli, Consultant and Entrepreneur wannabe tbatchelli@gmail.com, @tbatchelli, @disclojure February 2, 2011 02/01/11

GoGrid’s Challenges GoGrid is a cloud provider, building their own software Developing and testing their cloud management software is complex. They need to create and maintain realistic environments in order to load them with their latest software builds ~10 different servers types ~20 different services

The problem? Current time to setup a development/QA environment is 2 days • Mostly manual work Building the environments manually is very error prone A high level of interconnectivity between services

Goal To be able to create development andtesting environments dynamically, in a customizable way, and on cloud infrastructure A Cloud in a Cloud

Additional Challenges Environments need to be verified as fully functional before deploying software on them Must support multiple locations Must be able to scale dynamically Must support concurrent development • Different feature branches Must manage/version non-software components in a lockstep with the software: • Network configuration • System-level configurations • Application-level configurations

tools 02/01/11

Abstraction layer on resource managementIf you code your infrastructure managementcode to the jclouds API, you can port yourcode to all cloud providersJVM-basedProvides access to GoGrid’s infrastructure byabstracting the communication GoGrid’sCloud API

Uses JClouds to create and manage nodesProvisions nodes with software andconfigurationVerifies node and service correct operationOperates the services in the nodesNodes are grouped by groups (formerlyknown as ‘tags’). All nodes in a group areequivalent

ClojurePallet is built on ClojureClojure is a JVM-based lisp with superbintegration with Java and concurrencyCan use any existing Java library (Tomcat, jdbc,etc...)Provides a high level of abstraction, which iswhat is needed in this caseGrowing community: there are 7 books on Clojureeither published or in preparation

how? 02/01/11

GoGrid Cloud Infrastructure Service ServiceGoGrid Cloud API Service Service Service Service Service Service Service Service Service

GoGrid Cloud Infrastructure Service Service GoGrid Cloud APIjclouds Service Service Service Service Service Service Service Service Service

GoGrid Cloud Infrastructuregimme Service Servicenodes! GoGrid Cloud API jclouds Service Service Service Service Service Service Service Service Service

GoGrid Cloud Infrastructure Service ServiceGoGrid Cloud API Service Service Service Service Service Service Service Service Service

GoGrid Cloud Infrastructure Service Service et GoGrid Cloud APIpall Service Service Servicepallet Service Service Service Service Service Service

GoGrid Cloud Infrastructuredo stuff! Service Service et GoGrid Cloud API pall Service Service Service pallet Service Service Service Service Service Service

GoGrid Cloud InfrastructureGoGrid Cloud API

GoGrid Cloud Infrastructure GoGrid Cloud APIjclouds

GoGrid Cloud Infrastructure make mea dev cloud! GoGrid Cloud API jclouds

GoGrid Cloud Infrastructure make me Service Servicea dev cloud! GoGrid Cloud API jclouds Service Service Service Service Service Service Service Service Service

GoGrid Cloud Infrastructure install cloud Service Serviceinfrastructure! et GoGrid Cloud API pall Service Service Service pallet Service Service Service Service Service Service

GoGrid Cloud Infrastructureis all working Service Service fine? et GoGrid Cloud API pall Service Service Service pallet Service Service Service Service Service Service

GoGrid Cloud Infrastructureis all working Service Service fine? et GoGrid Cloud API pall Service Service Service pallet Service Service Service OK! Service Service Service

GoGrid Cloud Infrastructure deploy Service Servicenew build! et GoGrid Cloud API pall Service Service Service pallet Service Service Service Service Service Service

GoGrid Cloud Infrastructure deploy GoGrid Cloud Infrastructure Service Servicenew build! et GoGrid Cloud API pall GoGrid Cloud API Service Service Service pallet Service Service Service Service Service Service

GoGrid Cloud Infrastructure GoGrid Cloud Infrastructure Service ServiceGoGrid Cloud API GoGrid Cloud API Service Service Service Service Service Service Service Service Service

GoGrid Cloud Infrastructureawesome! GoGrid Cloud Infrastructure Service Service GoGrid Cloud API GoGrid Cloud API Service Service Service Service Service Service Service Service Service

GoGrid Cloud Infrastructure GoGrid Cloud Infrastructure Service Service GoGrid Cloud API GoGrid Cloud API Service Service Servicejclouds Service Service Service Service Service Service

GoGrid Cloud Infrastructure let’s test: GoGrid Cloud Infrastructure Service Servicegimme nodes! GoGrid Cloud API GoGrid Cloud API Service Service Service jclouds Service Service Service Service Service Service

GoGrid Cloud Infrastructure GoGrid Cloud Infrastructure Service Service GoGrid Cloud API GoGrid Cloud API Service Service Servicejclouds Service Service Service Service Service Service

GoGrid Cloud InfrastructureI’m done, GoGrid Cloud Infrastructure Service Service thanks! GoGrid Cloud API GoGrid Cloud API Service Service Service jclouds Service Service Service Service Service Service

GoGrid Cloud InfrastructureI’m done, thanks! GoGrid Cloud API jclouds

GoGrid Cloud InfrastructureGoGrid Cloud API

environment GoGrid Cloud Infrastructurefor branch A? GoGrid Cloud API

environment GoGrid Cloud Infrastructurefor branch A? GoGrid Cloud Infrastructure branch A GoGrid Cloud API GoGrid Cloud API

environment GoGrid Cloud Infrastructurefor branch A? GoGrid Cloud Infrastructure branch A environment GoGrid Cloud API GoGrid Cloud APIfor branch B?

environment GoGrid Cloud Infrastructurefor branch A? GoGrid Cloud Infrastructure branch A GoGrid Cloud Infrastructure branch B environment GoGrid Cloud API GoGrid Cloud APIfor branch B? GoGrid Cloud API

environment GoGrid Cloud Infrastructurefor branch A? GoGrid Cloud Infrastructure branch A GoGrid Cloud Infrastructure branch B environment GoGrid Cloud API GoGrid Cloud APIfor branch B? GoGrid Cloud API environmentfor QA please?

environment GoGrid Cloud Infrastructurefor branch A? GoGrid Cloud Infrastructure branch A GoGrid Cloud Infrastructure branch B GoGrid Cloud Infrastructure QA environment GoGrid Cloud API GoGrid Cloud APIfor branch B? GoGrid Cloud API GoGrid Cloud API environmentfor QA please?

development 02/01/11

GoGrid’s Development Process Create a feature branch Create an environment (cloud infrastructure) for this branch Verify new environment (end-to-end tests) Automatically build and deploy the new software every time new code is committed to the branch. Destroy environment and free resources when done

An Environment Usually one environment for active development branch Each environment is specified by: • The development branch it tracks • Service types and node count for each service type • Network configuration • External services/DB provisioning

Development Environments Each development environment tracks a development feature branch Each environment has a node with Hudson CI, configured to track the environment’s branch When new code is pushed to the development branch, the environment’s Hudson server builds the software and updates the environment with it.

Agile Development at GoGrid with Pallet and JClouds

by GoGrid Cloud Hosting

Accessibility

Categories

Upload Details

Usage Rights

1 Embed 364

Statistics