Playing God in the clouds...
SJUG Aug 2010 @Atlassian



    Michael Neale
    michael.neale@gmail.com
    twitter.com/michaelneale
    www.michaelneale.net
           Copyright © 2010 Opscode, Inc - All Rights Reserved   1
Tiny bit of background...



         Copyright © 2010 Opscode, Inc - All Rights Reserved   2
#define cloud



   Copyright © 2010 Opscode, Inc - All Rights Reserved   3
SaaS
 PaaS
IaaS <--

(NIST)
 Copyright © 2010 Opscode, Inc - All Rights Reserved   4
IaaS:
 Illusion of infinite
       servers
Low level resources
Requires Sysadmin/
       Devops
       Copyright © 2010 Opscode, Inc - All Rights Reserved   5
IaaS - fire data centre



        Copyright © 2010 Opscode, Inc - All Rights Reserved   6
PaaS - fire the
 sysadmins


    Copyright © 2010 Opscode, Inc - All Rights Reserved   7
SaaS - fire everone else



        Copyright © 2010 Opscode, Inc - All Rights Reserved   8
#define devops



    Copyright © 2010 Opscode, Inc - All Rights Reserved   9
Infrastructure as code
     Useful tools:
        jclouds
          FOG
      deltacloud
     puppet/chef
        Copyright © 2010 Opscode, Inc - All Rights Reserved   10
What Does Cloud
Provisioning Look Like?
        Copyright © 2010 Opscode, Inc - All Rights Reserved   11
Two Answers
	

       Copyright © 2010 Opscode, Inc - All Rights Reserved   12
You building a platform
 (building an image?)

        Copyright © 2010 Opscode, Inc - All Rights Reserved   13
You deploying it


     Copyright © 2010 Opscode, Inc - All Rights Reserved   14
Build your platform
Chef, puppet, by hand...
     snapshotting



         Copyright © 2010 Opscode, Inc - All Rights Reserved   16
Deploy to a (IaaS)
             platform




http://www.byteonic.com/2009/overview-of-java-support-in-google-app-engine/
Command line, web UI,
      or API



        Copyright © 2010 Opscode, Inc - All Rights Reserved   18
cloud + devops =
better, faster, more

       Copyright © 2010 Opscode, Inc - All Rights Reserved   19
“Enable the reconstruction of the
business from nothing but a source
code repository, an application data
   backup, and “... a credit card

             Copyright © 2010 Opscode, Inc - All Rights Reserved   20
Cloud Provisioning is
       Step 1




       Copyright © 2010 Opscode, Inc - All Rights Reserved   21
Cloud Provisioning is
       Step 1
    Provisioning



       Copyright © 2010 Opscode, Inc - All Rights Reserved   21
Cloud Provisioning is
       Step 1
    Provisioning
    Configuration


       Copyright © 2010 Opscode, Inc - All Rights Reserved   21
Cloud Provisioning is
       Step 1
    Provisioning
    Configuration
 Systems Integration
       Copyright © 2010 Opscode, Inc - All Rights Reserved   21
Cloudy
environments
Cloudy
    environments

• automate EVERYTHING
Cloudy
    environments

• automate EVERYTHING
• seed/snapshotting
Cloudy
     environments

• automate EVERYTHING
• seed/snapshotting
• safely update configuration
github jclouds/

              runs on the JVM (including       )

(with-compute-service [compute]
  (run-node "webserver"))
                                       pis pis
                                    d a ud a
                                 lou clo
                              o c t of
                       k- in t -ou
                    loc lock
                zero zero



                nodes = compute.runNodesWithTag(“webserver”, 1);
jclouds

open source
feels like java (and clojure)
portability between clouds
deal with web complexity
unit testability
thread-safe and scalable
Cross cloud API
abstractions:




         Copyright © 2010 Opscode, Inc - All Rights Reserved   26
Cross cloud API
abstractions:
Generic binary storage
 BlobStore




              Copyright © 2010 Opscode, Inc - All Rights Reserved   26
Cross cloud API
abstractions:
Generic binary storage
 BlobStore
Generic service provisioning
 ComputeService




              Copyright © 2010 Opscode, Inc - All Rights Reserved   26
Cross cloud API
abstractions:
Generic binary storage
 BlobStore
Generic service provisioning
 ComputeService

but you have low level access when
need it


              Copyright © 2010 Opscode, Inc - All Rights Reserved   26
jclouds-blobstore             github jclouds/jclouds


blobstore = new BlobStoreContextFactory().createContext(
                    “s3”, identity, credential).getAsyncBlobStore();

blob = blobstore.newBlob(key);
blob.getMetadata().getUserMetadata().put("date", now());
blob.setPayload(snapshot());
blob.getPayload().setContentMD5(snapshotMD5());

blobstore.putBlob(containerName, blob)
             .addListener(publishAMQP(key),sameThreadExecutor());
jclouds-compute               github jclouds/jclouds


service = new ComputeServiceContextFactory().createContext(
                   “ec2”, identity, credential).getComputeService();

template = service.templateBuilder().fastest().build();

template.getOptions().runScript(installGemsAndRunChef);

nodes = service.runNodesWithTag(“webserver”, 5, template);
jclouds                            github jclouds/jclouds


service = new ComputeServiceContextFactory().createContext(
            “rimuhosting”, user, password ).getComputeService();

template = service.templateBuilder().any().biggest().build();

template.getOptions().installPrivateKey(privateRSA)
                     .authorizePublicKey(publicRSA)
                     .runScript(installGemsAndRunChef);

nodes = service.runNodesWithTag(“webserver”, 5, template);
Common demoniator




       Copyright © 2010 Opscode, Inc - All Rights Reserved   30
Common demoniator

 BlobStore: OK




           Copyright © 2010 Opscode, Inc - All Rights Reserved   30
Common demoniator

 BlobStore: OK

 ComputeService:
 Not all clouds created equal, you may need to dive down
 and “lock in” just a tiny bit.. I have !




                   Copyright © 2010 Opscode, Inc - All Rights Reserved   30
Common demoniator

  BlobStore: OK

  ComputeService:
   Not all clouds created equal, you may need to dive down
   and “lock in” just a tiny bit.. I have !



Still early days:




                     Copyright © 2010 Opscode, Inc - All Rights Reserved   30
Common demoniator

 BlobStore: OK

 ComputeService:
  Not all clouds created equal, you may need to dive down
  and “lock in” just a tiny bit.. I have !



Still early days:
No Volume abstractions, yes, no LBs, firewalls etc



                    Copyright © 2010 Opscode, Inc - All Rights Reserved   30
To avoid lockin:




                   Copyright © 2010 Opscode, Inc - All Rights Reserved   31
To avoid lockin:

IaaS Cloud primitives (my opinion on low level)
 Volume management
 Server provisioning **
 Blob storage / Bulk data**
 Firewall/access control
 Image construction ***


                   Copyright © 2010 Opscode, Inc - All Rights Reserved   31
To avoid lockin:

IaaS Cloud primitives (my opinion on low level)
 Volume management
 Server provisioning **
 Blob storage / Bulk data**
 Firewall/access control
 Image construction ***
** This is the currently successfully abstracted
ones - generic enough via various apis

                   Copyright © 2010 Opscode, Inc - All Rights Reserved   31
To avoid lockin:

IaaS Cloud primitives (my opinion on low level)
 Volume management
 Server provisioning **
 Blob storage / Bulk data**
 Firewall/access control
 Image construction ***
** This is the currently successfully abstracted
ones - generic enough via various apis
*** plenty of “traditional” tools help here - chef !
                     Copyright © 2010 Opscode, Inc - All Rights Reserved   31
Bright future




          Copyright © 2010 Opscode, Inc - All Rights Reserved   32
Bright future

 Eucalyptus
 In house “private clouds”




                  Copyright © 2010 Opscode, Inc - All Rights Reserved   32
Bright future

 Eucalyptus
 In house “private clouds”


 OpenStack
 Rackspace/NASA - comparable to EC2 in
 scalecapability




                  Copyright © 2010 Opscode, Inc - All Rights Reserved   32
Bright future

 Eucalyptus
 In house “private clouds”


 OpenStack
 Rackspace/NASA - comparable to EC2 in
 scalecapability




                  Copyright © 2010 Opscode, Inc - All Rights Reserved   32
Copyright © 2010 Opscode, Inc - All Rights Reserved   33
Thanks !

twitter.com/michaelneale
http://www.jclouds.org/
http://deltacloud.org/
http://www.openstack.org/




           Copyright © 2010 Opscode, Inc - All Rights Reserved   33
Thanks !

twitter.com/michaelneale
http://www.jclouds.org/
http://deltacloud.org/
http://www.openstack.org/




           Copyright © 2010 Opscode, Inc - All Rights Reserved   33

Sjug aug 2010_cloud

  • 1.
    Playing God inthe clouds... SJUG Aug 2010 @Atlassian Michael Neale michael.neale@gmail.com twitter.com/michaelneale www.michaelneale.net Copyright © 2010 Opscode, Inc - All Rights Reserved 1
  • 2.
    Tiny bit ofbackground... Copyright © 2010 Opscode, Inc - All Rights Reserved 2
  • 3.
    #define cloud Copyright © 2010 Opscode, Inc - All Rights Reserved 3
  • 4.
    SaaS PaaS IaaS <-- (NIST) Copyright © 2010 Opscode, Inc - All Rights Reserved 4
  • 5.
    IaaS: Illusion ofinfinite servers Low level resources Requires Sysadmin/ Devops Copyright © 2010 Opscode, Inc - All Rights Reserved 5
  • 6.
    IaaS - firedata centre Copyright © 2010 Opscode, Inc - All Rights Reserved 6
  • 7.
    PaaS - firethe sysadmins Copyright © 2010 Opscode, Inc - All Rights Reserved 7
  • 8.
    SaaS - fireeverone else Copyright © 2010 Opscode, Inc - All Rights Reserved 8
  • 9.
    #define devops Copyright © 2010 Opscode, Inc - All Rights Reserved 9
  • 10.
    Infrastructure as code Useful tools: jclouds FOG deltacloud puppet/chef Copyright © 2010 Opscode, Inc - All Rights Reserved 10
  • 11.
    What Does Cloud ProvisioningLook Like? Copyright © 2010 Opscode, Inc - All Rights Reserved 11
  • 12.
    Two Answers Copyright © 2010 Opscode, Inc - All Rights Reserved 12
  • 13.
    You building aplatform (building an image?) Copyright © 2010 Opscode, Inc - All Rights Reserved 13
  • 14.
    You deploying it Copyright © 2010 Opscode, Inc - All Rights Reserved 14
  • 15.
  • 16.
    Chef, puppet, byhand... snapshotting Copyright © 2010 Opscode, Inc - All Rights Reserved 16
  • 17.
    Deploy to a(IaaS) platform http://www.byteonic.com/2009/overview-of-java-support-in-google-app-engine/
  • 18.
    Command line, webUI, or API Copyright © 2010 Opscode, Inc - All Rights Reserved 18
  • 19.
    cloud + devops= better, faster, more Copyright © 2010 Opscode, Inc - All Rights Reserved 19
  • 20.
    “Enable the reconstructionof the business from nothing but a source code repository, an application data backup, and “... a credit card Copyright © 2010 Opscode, Inc - All Rights Reserved 20
  • 21.
    Cloud Provisioning is Step 1 Copyright © 2010 Opscode, Inc - All Rights Reserved 21
  • 22.
    Cloud Provisioning is Step 1 Provisioning Copyright © 2010 Opscode, Inc - All Rights Reserved 21
  • 23.
    Cloud Provisioning is Step 1 Provisioning Configuration Copyright © 2010 Opscode, Inc - All Rights Reserved 21
  • 24.
    Cloud Provisioning is Step 1 Provisioning Configuration Systems Integration Copyright © 2010 Opscode, Inc - All Rights Reserved 21
  • 26.
  • 27.
    Cloudy environments • automate EVERYTHING
  • 28.
    Cloudy environments • automate EVERYTHING • seed/snapshotting
  • 29.
    Cloudy environments • automate EVERYTHING • seed/snapshotting • safely update configuration
  • 30.
    github jclouds/ runs on the JVM (including ) (with-compute-service [compute] (run-node "webserver")) pis pis d a ud a lou clo o c t of k- in t -ou loc lock zero zero nodes = compute.runNodesWithTag(“webserver”, 1);
  • 31.
    jclouds open source feels likejava (and clojure) portability between clouds deal with web complexity unit testability thread-safe and scalable
  • 32.
    Cross cloud API abstractions: Copyright © 2010 Opscode, Inc - All Rights Reserved 26
  • 33.
    Cross cloud API abstractions: Genericbinary storage BlobStore Copyright © 2010 Opscode, Inc - All Rights Reserved 26
  • 34.
    Cross cloud API abstractions: Genericbinary storage BlobStore Generic service provisioning ComputeService Copyright © 2010 Opscode, Inc - All Rights Reserved 26
  • 35.
    Cross cloud API abstractions: Genericbinary storage BlobStore Generic service provisioning ComputeService but you have low level access when need it Copyright © 2010 Opscode, Inc - All Rights Reserved 26
  • 36.
    jclouds-blobstore github jclouds/jclouds blobstore = new BlobStoreContextFactory().createContext( “s3”, identity, credential).getAsyncBlobStore(); blob = blobstore.newBlob(key); blob.getMetadata().getUserMetadata().put("date", now()); blob.setPayload(snapshot()); blob.getPayload().setContentMD5(snapshotMD5()); blobstore.putBlob(containerName, blob) .addListener(publishAMQP(key),sameThreadExecutor());
  • 37.
    jclouds-compute github jclouds/jclouds service = new ComputeServiceContextFactory().createContext( “ec2”, identity, credential).getComputeService(); template = service.templateBuilder().fastest().build(); template.getOptions().runScript(installGemsAndRunChef); nodes = service.runNodesWithTag(“webserver”, 5, template);
  • 38.
    jclouds github jclouds/jclouds service = new ComputeServiceContextFactory().createContext( “rimuhosting”, user, password ).getComputeService(); template = service.templateBuilder().any().biggest().build(); template.getOptions().installPrivateKey(privateRSA) .authorizePublicKey(publicRSA) .runScript(installGemsAndRunChef); nodes = service.runNodesWithTag(“webserver”, 5, template);
  • 39.
    Common demoniator Copyright © 2010 Opscode, Inc - All Rights Reserved 30
  • 40.
    Common demoniator BlobStore:OK Copyright © 2010 Opscode, Inc - All Rights Reserved 30
  • 41.
    Common demoniator BlobStore:OK ComputeService: Not all clouds created equal, you may need to dive down and “lock in” just a tiny bit.. I have ! Copyright © 2010 Opscode, Inc - All Rights Reserved 30
  • 42.
    Common demoniator BlobStore: OK ComputeService: Not all clouds created equal, you may need to dive down and “lock in” just a tiny bit.. I have ! Still early days: Copyright © 2010 Opscode, Inc - All Rights Reserved 30
  • 43.
    Common demoniator BlobStore:OK ComputeService: Not all clouds created equal, you may need to dive down and “lock in” just a tiny bit.. I have ! Still early days: No Volume abstractions, yes, no LBs, firewalls etc Copyright © 2010 Opscode, Inc - All Rights Reserved 30
  • 44.
    To avoid lockin: Copyright © 2010 Opscode, Inc - All Rights Reserved 31
  • 45.
    To avoid lockin: IaaSCloud primitives (my opinion on low level) Volume management Server provisioning ** Blob storage / Bulk data** Firewall/access control Image construction *** Copyright © 2010 Opscode, Inc - All Rights Reserved 31
  • 46.
    To avoid lockin: IaaSCloud primitives (my opinion on low level) Volume management Server provisioning ** Blob storage / Bulk data** Firewall/access control Image construction *** ** This is the currently successfully abstracted ones - generic enough via various apis Copyright © 2010 Opscode, Inc - All Rights Reserved 31
  • 47.
    To avoid lockin: IaaSCloud primitives (my opinion on low level) Volume management Server provisioning ** Blob storage / Bulk data** Firewall/access control Image construction *** ** This is the currently successfully abstracted ones - generic enough via various apis *** plenty of “traditional” tools help here - chef ! Copyright © 2010 Opscode, Inc - All Rights Reserved 31
  • 48.
    Bright future Copyright © 2010 Opscode, Inc - All Rights Reserved 32
  • 49.
    Bright future Eucalyptus In house “private clouds” Copyright © 2010 Opscode, Inc - All Rights Reserved 32
  • 50.
    Bright future Eucalyptus In house “private clouds” OpenStack Rackspace/NASA - comparable to EC2 in scalecapability Copyright © 2010 Opscode, Inc - All Rights Reserved 32
  • 51.
    Bright future Eucalyptus In house “private clouds” OpenStack Rackspace/NASA - comparable to EC2 in scalecapability Copyright © 2010 Opscode, Inc - All Rights Reserved 32
  • 52.
    Copyright © 2010Opscode, Inc - All Rights Reserved 33
  • 53.
  • 54.

Editor's Notes

  • #16 choose your products and standards used
  • #21 Adam in his chapter of the Web Operations book: &quot;Enable the reconstruction of the business from nothing but a source code repository, an application data backup, and bare-metal resources&quot;
  • #30 works in google app engine high performance thread safe enterprise ready
  • #31 focused on semantic portability across clouds I want an image running ubuntu and don&amp;#x2019;t want to know the id absolute portability where possible, but expose vendor apis where needed
  • #32 focused on semantic portability across clouds I want an image running ubuntu and don&amp;#x2019;t want to know the id absolute portability where possible, but expose vendor apis where needed
  • #33 focused on semantic portability across clouds I want an image running ubuntu and don&amp;#x2019;t want to know the id absolute portability where possible, but expose vendor apis where needed
  • #36 focused on semantic portability across clouds I want an image running ubuntu and don&amp;#x2019;t want to know the id absolute portability where possible, but expose vendor apis where needed
  • #37 focused on semantic portability across clouds I want an image running ubuntu and don&amp;#x2019;t want to know the id absolute portability where possible, but expose vendor apis where needed
  • #38 focused on semantic portability across clouds I want an image running ubuntu and don&amp;#x2019;t want to know the id absolute portability where possible, but expose vendor apis where needed
  • #39 focused on semantic portability across clouds I want an image running ubuntu and don&amp;#x2019;t want to know the id absolute portability where possible, but expose vendor apis where needed
  • #40 focused on semantic portability across clouds I want an image running ubuntu and don&amp;#x2019;t want to know the id absolute portability where possible, but expose vendor apis where needed
  • #44 focused on semantic portability across clouds I want an image running ubuntu and don&amp;#x2019;t want to know the id absolute portability where possible, but expose vendor apis where needed
  • #45 focused on semantic portability across clouds I want an image running ubuntu and don&amp;#x2019;t want to know the id absolute portability where possible, but expose vendor apis where needed
  • #46 focused on semantic portability across clouds I want an image running ubuntu and don&amp;#x2019;t want to know the id absolute portability where possible, but expose vendor apis where needed
  • #47 focused on semantic portability across clouds I want an image running ubuntu and don&amp;#x2019;t want to know the id absolute portability where possible, but expose vendor apis where needed
  • #48 focused on semantic portability across clouds I want an image running ubuntu and don&amp;#x2019;t want to know the id absolute portability where possible, but expose vendor apis where needed