• Share
  • Email
  • Embed
  • Like
  • Save
  • Private Content
2012 02 pre_hbs_grid_overview_ianstokesrees_pt2
 

2012 02 pre_hbs_grid_overview_ianstokesrees_pt2

on

  • 1,541 views

 

Statistics

Views

Total Views
1,541
Views on SlideShare
1,541
Embed Views
0

Actions

Likes
0
Downloads
4
Comments
0

0 Embeds 0

No embeds

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

    2012 02 pre_hbs_grid_overview_ianstokesrees_pt2 2012 02 pre_hbs_grid_overview_ianstokesrees_pt2 Presentation Transcript

    • Functional  MRIGrid Overview - Ian Stokes-Rees ijstokes@seas.harvard.edu
    • Next  Generation  SequencingGrid Overview - Ian Stokes-Rees ijstokes@seas.harvard.edu
    • Scienti6ic  Research  Today • International  collaborations • IT  becomes  embedded  into  research  process:  data,  results,   analysis,  visualization • Crossing  institutional  and  national  boundaries • Computational  techniques  increasingly   important • ...  and  computationally  intensive  techniques  as  well • requires  use  of  high  performance  computing  systems • Data  volumes  are  growing  fast • hard  to  share • hard  to  manage • ScientiBic  software  often  difBicult  to  use • or  to  use  properly • Web  based  tools  increasingly  important • but  often  lack  disconnect  from  persisted  and  shared  resultsGrid Overview - Ian Stokes-Rees ijstokes@seas.harvard.edu
    • Required: Collaborative  environment  for  compute  and  data  intensive  science
    • http://www.xsede.orgGrid Overview - Ian Stokes-Rees ijstokes@seas.harvard.edu
    • • 200,000  hour  allocations  “easy” • millions  of  hours  possible • any  US-­‐based  researcher  can  apply • allocation  holder  can  delegate • access  to  ~dozen  of   supercomputing  centers • command  line  access • standard  batch  systems  like  PBS,  LSF,  SGE • web-­‐based  interaction • build  your  own  Science  Gateway • XSEDE  for  processing  behind  the  scenesGrid Overview - Ian Stokes-Rees ijstokes@seas.harvard.edu
    • Open  Science  Grid http://opensciencegrid.org • US  National   Cyberinfrastructure • Primarily  used  for  high   energy  physics  computing • 80  sites • 100,000  job  slots 5,073,293  hours • 1,500,000  hours  per  day ~570  years • PB  scale  aggregate  storage • 1  PB  transferred  each  day • Virtual  Organization-­‐basedGrid Overview - Ian Stokes-Rees ijstokes@seas.harvard.edu
    • Grid Overview - Ian Stokes-Rees ijstokes@seas.harvard.edu
    • Simpli6ied  Grid  ArchitectureGrid Overview - Ian Stokes-Rees ijstokes@seas.harvard.edu
    • Grid  Architectural  Details • Resources • Information • Uniform  compute  clusters • LDAP  based  most  common  (not   • Managed  via  batch  queues optimized  for  writes) • Local  scratch  disk • Domain  speciBic  layer • Sometimes  high  perf.  network   • Open  problem! (e.g.  InBiniBand) • Fabric • Behind  NAT  and  Birewall • In  most  cases,  assume  functioning   • No  shell  access Internet • Data • Some  sites  part  of  experimental   private  networks • Tape-­‐backed  mass  storage • Disk  arrays  (100s  TB  to  PB) • Security • High  bandwidth  (multi-­‐stream)   • Typically  underpinned  by  X.509   transfer  protocols Public  Key  Infrastructure • File  catalogs • Same  standards  as  SSL/TLS  and   • Meta-­‐data “server  certs”  for  “https” • Replica  managementGrid Overview - Ian Stokes-Rees ijstokes@hkl.hms.harvard.edu
    • OSG  Components  (I) • Centralized • X.509  CertiBicate  Authority:  Energy  Science  Network  CA  @  LBL • Accounting:  Gratia  logging  system  to  track  usage  (CPU,  Network,  Disk) • Status:  LDAP  directory  with  details  of  each  participating  system • Support:  Central  clearing  house  for  support  tickets • Software:  distribution  system,  update  testing,  bug  reporting  and  Bixing • Communication:  Wikis,  docs,  mailing  lists,  workshops,  conferences,  etc. • Per  Site • Compute  Element/Gatekeeper  (CE/GK):  access  point  for  external  users,  acts   as  frontend  for  any  cluster.    Globus  GRAM  +  local  batch  system • Storage  Element  (SE):  grid-­‐accessible  storage  system,  GridFTP-­‐based  +  SRM • Worker  Nodes  (WN):  cluster  nodes  with  grid  software  stack • User  Interface  (UI):  access  point  for  local  users  to  interact  with  remote  grid • Access  Control:  GUMS  +  PRIMA  for  ACLs  to  local  system  by  grid  identities • Admin  contact:  need  a  local  expert  (or  two!)Grid Overview - Ian Stokes-Rees ijstokes@seas.harvard.edu
    • OSG  Components  (II) • Per  Virtual  Organization  (user  community) • VO  Management  System  (VOMS):  to  organize  and  register  users • Registration  Authority  (RA):  to  validate  community  users  with  X.509  issuer • User  Interface  system  (UI):  provide  gateway  to  OSG  for  users • Support  Contact:  users  are  supported  by  their  VO  representatives • Per  User • X.509  user  certiBicate  (although  I’d  like  to  hide  that  part) • Induction:  unless  it  is  through  a  portal,  grid  computing  is  not  shared  Bile   system  batch  computing!    Many  more  failure  modes  and  gotchas.Grid Overview - Ian Stokes-Rees ijstokes@seas.harvard.edu
    • Grid  Opportunities • New  compute  intensive  workBlows • think  big:  tens  or  hundreds  of  thousands  of  hours  Binished  in  1-­‐2  days • sharing  resources  for  efBicient  and  large  scale  utilization • Data  intensive  problems • we  mirror  20  GB  of  data  to  30  computing  centers • Data  movement,  management,  and  archive • Federated  identity  and  user  management • labs,  collaborations  or  ad-­‐hoc  groups • role-­‐based  access  control  (RBAC)  and  IdM • Collaborative  environment • Web-­‐based  access  to  applicationsGrid Overview - Ian Stokes-Rees ijstokes@seas.harvard.edu
    • Protein  Structure  DeterminationGrid Overview - Ian Stokes-Rees ijstokes@seas.harvard.edu
    • Grid Overview - Ian Stokes-Rees ijstokes@seas.harvard.edu
    • Typical  Layered  Environment Fortran bin • Command  line  application  (e.g.  Fortran) • Friendly  application  API  wrapper Python API Map- • Batch  execution  wrapper  for  N-­‐iterations Multi-exec wrapper Reduce • Results  extraction  and  aggregation Result aggregator • Grid  job  management  wrapper Grid management • Web  interface Web interface • forms,  views,  static  HTML  results • GOAL  eliminate  shell  scripts • often  found  as  “glue”  language  between  layersGrid Overview - Ian Stokes-Rees ijstokes@seas.harvard.edu
    • Web  Portals  for  Collaborative,   Multi-­‐disciplinary  Research......  which  leverage  capabilities  of  federated   grid  computing  environments
    • The  Browser  as  the   Universal  Interface • If  it  isn’t  already  obvious  to  you • Any  interactive  application  developed  today  should  be  web-­‐based  with  a   RESTful  interface  (if  at  all  possible) • A  rich  set  of  tools  and  techniques • AJAX,  HTML4/5,  CSS,  and  JavaScript • Dynamic  content  negotiation • HTTP  headers,  caching,  security,  sessions/cookies • Scalable,  replicable,  centralized,  multi-­‐threaded,   multi-­‐user • Alternatives • Command  Line  (CLI):  great  for  scriptable  jobs • GUI  toolkits:  necessary  for  applications  with  high  graphics  or  I/O  demandsGrid Overview - Ian Stokes-Rees ijstokes@seas.harvard.edu
    • What  is  a  Science  Portal? • A  web-­‐based  gateway  to  resources  and  data • simpliBied  access • centralized  access • uniBied  access  (CGI,  Perl,  Python,  PHP,  static  HTML,  static  Biles,  etc.) • Attempt  to  provide  uniform  access  to  a  range  of   services  and  resources • Data  access  via  HTTP • Leverage  brilliance  of  Apache  HTTPD  and   associated  modulesGrid Overview - Ian Stokes-Rees ijstokes@seas.harvard.edu
    • SBGrid  Science  Portal  Objectives A.   Extensible  infrastructure  to  facilitate   development  and  deployment  of  novel   computational  workBlows   B. Web-­‐accessible  environment  for  collaborative,   compute  and  data  intensive  scienceGrid Overview - Ian Stokes-Rees ijstokes@seas.harvard.edu
    • XSEDE SBGrid User NERSC Community Open Science Grid National Federated Cyberinfrastructure Odyssey Facilitate  interface   between  community   and  cyberinfrastructure Orchestra EC2Grid Overview - Ian Stokes-Rees ijstokes@seas.harvard.edu
    • Grid Overview - Ian Stokes-Rees ijstokes@seas.harvard.edu
    • Results  Visualization  and  AnalysisGrid Overview - Ian Stokes-Rees ijstokes@seas.harvard.edu
    • Data  Access
    • User  access  to  results  dataGrid Overview - Ian Stokes-Rees ijstokes@seas.harvard.edu
    • Grid Overview - Ian Stokes-Rees ijstokes@seas.harvard.edu
    • Experimental  Data  Access • Collaboration • Access  Control • Identity  Management • Data  Management • High  Performance  Data  Movement • Multi-­‐modal  AccessGrid Overview - Ian Stokes-Rees ijstokes@seas.harvard.edu
    • Data  Model • Data  Tiers • VO-­wide:  all  sites,  admin  managed,  very  stable • User  project:  all  sites,  user  managed,  1-­‐10  weeks,  1-­‐3  GB • User  static:  all  sites,  user  managed,  indeBinite,  10  MB • Job  set:  all  sites,  infrastructure  managed,  1-­‐10  days,  0.1-­‐1  GB • Job:  direct  to  worker  node,  infrastructure  managed,  1  day,  <10  MB • Job  indirect:  to  worker  node  via  UCSD,  infrastructure  managed,  1   day,  <10  GBGrid Overview - Ian Stokes-Rees ijstokes@seas.harvard.edu
    • About  2PB  with 100  front  end   servers  for  high   bandwidth  parallel   Bile  transferGrid Overview - Ian Stokes-Rees ijstokes@seas.harvard.edu
    • Globus  Online:  High  Performance   Reliable  3rd  Party  File  Transfer GUMS DN  to  user  mapping CertiBicate  Authority VOMS root  of  trust VO  membership portal cluster Globus  Online Bile  transfer  service lab file data collection facility serverGrid Overview - Ian Stokes-Rees desktop laptop ijstokes@seas.harvard.edu
    • Architecture • SBGrid • manages  all  user  account  creation  and  credential  mgmt • hosts  MyProxy,  VOMS,  GridFTP,  and  user  interfaces • Facility • knows  about  lab  groups • e.g.  “Harrison”,  “Sliz” • delegates  knowledge  of  group  membership  to  SBGrid  VOMS • facility  can  poll  VOMS  for  list  of  current  members • uses  X.509  for  user  identiBication • deploys  GridFTP  server • Lab  group • designates  group  manager  that  adds/removes  individuals • deploys  GridFTP  server  or  Globus  Connect  client • Individual • username/password  to  access  facility  and  lab  storage • Globus  Connect  for  personal  GridFTP  server  to  laptop • Globus  Online  web  interface  to  “drive”  transfersGrid Overview - Ian Stokes-Rees ijstokes@seas.harvard.edu
    • Grid Overview - Ian Stokes-Rees ijstokes@seas.harvard.edu
    • Objective • Easy  to  use  high  performance  data  mgmt   environment • Fast  Bile  transfer • facility-­‐to-­‐lab,  facility-­‐to-­‐individual,  lab-­‐to-­‐individual • Reduced  administrative  overhead • Better  data  curationGrid Overview - Ian Stokes-Rees ijstokes@seas.harvard.edu
    • Ryan,  a  postdoc  in  the   Frank  Lab  at  Columbia Access  NRAMM  facilities   securely  and  transfer  data   back  to  home  institute automated  X.509 check  SBGrid  for   application group   Ryan’s   membership /data/columbia/frank facility file server transfer  data  to  lab veriBication  in  Frank  Lab,  so   of   lab  membership access  to  Biles grant   SBGrid Ryan  initiate  tfransfer  at   applies   or  an   request  access Science account  at  the  SBGrid   NRAMM to  NRAMM Portal Science  Portal facility using  credential   notify  user  of   held  by  SBGrid lab file completion desktop serverautomated   /nfs/data/rsmithGlobus  Online  application use  Globus  Online   to  manage transfer  from   /Users/Ryan NRAMM  back  to  lab laptop
    • Challenges • Access  control • visibility • policies • Provenance • data  origin • history • Meta-­‐data • attributes • searchingGrid Overview - Ian Stokes-Rees ijstokes@seas.harvard.edu
    • User  Credentials
    • Uni6ied  Account  Management Hierarchical  LDAP  database user  basics passwords Standard  schemas Relational  DB user  custom  proBiles institutions lab  groups Custom  schemasGrid Overview - Ian Stokes-Rees ijstokes@seas.harvard.edu
    • X.509  Digital  Certi6icates✦ Analogy  to  a  passport: • Application  form • Sponsor’s  attestation • Consular  services • veriBication  of  application,  sponsor,  and  accompanying   identiBication  and  eligibility  documents • Passport  issuing  ofBice✦ Portable,  digital  passport • Bixed  and  secure  user  identiBiers • name,  email,  home  institution • signed  by  widely  trusted  issuer • time  limited • ISO  standard
    • U1 U1 U1Addressing  Certi6icate  Problems /. -..)"*& 012*%2! 3%"! )"*"!4&" ,"!&5":14(! !"#$"%&%()*"+,"!& U1 !"&$!*&!4,5(*)*$67"! *289:4)"*&% !";("<!"#$"%& R1time ;"!(9:$%"!"=()(7(=(&: ,2*>!6"=()(7(=(&: S1 411!2;","!& R2 %()*,"!& *289:4;4(=47(=(&: !"&!(";","!& U2a "?12!&%()*"+ ,"!&5":14(!
    • VO  (Group)  Membership   Registration !")*# !"#$%&(# *+,(-,.# /-0.# +.0-0(:#50.:#:,#.0?>0-:#&0&90.-<+#93#@A# .0?>0-:#!"#8.,>+-#4(%#.,70-# U2b (,123#4%&(# V1 ;0.23#>-0.#07897:3# 5,(6.&#07897:3# S2time 4++.,;0#&0&90.-<+=# 8.,>+-=#4(%#.,70-# 4%%#@A# V2 :,#!")*# (,123# .0?>0-:#!")*#$B# .0:>.(#!")*#$B# 4%%#$B#:,# +.,C3#50.:#
    • () AB)! !"#$%& *+",-"# .-/# ;< #/>:/-$+"#$%&%66":,$ =3!#"I3 ;<=* )@81, 0/#176%?",/8%1&-/,$ /8%1&0/#17/@ U1 4/,/#%$/ !"#$%&(%)*+% #/>:/-$-14,/@6/#$ 6/#$9/3+%1# ,,#""%+#0% 1*$/2% #/$:#,$#%691,4,:85/# ,"?23%4/,$- 0/#123/&14151&1$3 A1a 6#/%$/ 6",7#8/&14151&1$3 &"6%&%66$ S1* %++#"0/6/#$time -14,6/#$ ,"?23%0%1&%51&1$3 A1b -/$#/$#1/0%&-/#1%&,:85/# -/$;<#14C$- %66":,$#/%@3,"?76%?", +"#$%&&"41, U2* #/>:/-$-14,/@6/#?76%$/ #/$:#,-14,/@6/#?76%$/ DE+%1#-14,/@6/#$ 1,$"!F(*GDH7&/ #/41-$/#+#"I36/#$ HE6#/%$/&"6%& J1$C=3!#"I3 !"#$%&(%)*+% +#"I36/#$ ,,#""%-#.#$/#.% $#"*!$,#"%
    • Process  and  Design  Improvements ✦ Single  web-­‐form  application • includes  e-­‐mail  veriBicationn ✦ Centralized  and  connected  credential  management • FreeIPA  LDAP  -­‐  user  directory  and  credential  store • VOMS  -­‐  lab,  institution,  and  collaboration  afBiliations • MyProxy  -­‐  X.509  credential  store ✦ Overlap  administrative  roles • system  admin • registration  agent  for  certiBicate  authority  (approve  X.509   request) • VO  administrator  to  register  group  afBiliations ✦ Automation
    • Security
    • Access  Control • Need  a  strong  Identity  Management  environment • individuals:  identity  tokens  and  identiBiers • groups:  membership  lists • Active  Directory/CIFS  (Windows),  Open  Directory  (Apple),  FreeIPA  (Unix)  all  LDAP-­‐ based • Need  to  manage  and  communicate  Access  Control  policies • institutionally  driven • user  driven • Need  Authorization  System • Policy  Enforcement  Point  (shell  login,  data  access,  web  access,  start  application) • Policy  Decision  Point  (store  policies  and  understand  relationship  of  identity  token     and  policy)Grid Overview - Ian Stokes-Rees ijstokes@seas.harvard.edu
    • Access  Control • What  is  a  user? • .htaccess  and  .htpasswd • local  system  user  (NIS  or  /etc/passwd) • portal  framework  user  (proprietary  DB  schema) • grid  user  (X.509  DN) • What  are  we  securing  access  to? • Web  pages? • URLs? • Data? • SpeciBic  operations? • Meta  Data? • What  kind  of  policies  do  we  enable? • Simplify  to  READ  WRITE  EXECUTE  LIST  ADMINGrid Overview - Ian Stokes-Rees ijstokes@seas.harvard.edu
    • Grid Overview - Ian Stokes-Rees ijstokes@seas.harvard.edu
    • Grid Overview - Ian Stokes-Rees ijstokes@seas.harvard.edu
    • Grid Overview - Ian Stokes-Rees ijstokes@seas.harvard.edu
    • Grid Overview - Ian Stokes-Rees ijstokes@seas.harvard.edu
    • Grid Overview - Ian Stokes-Rees ijstokes@seas.harvard.edu
    • Architecture  Diagrams
    • Service  Architecture GlobusOnline UC San Diego @Argonne GUMS User GUMS GridFTP + glideinWMS data Hadoop factory Open Science Grid computations MyProxy @NCSA, UIUC monitoring interfaces data computation ID mgmt Ganglia scp Condor FreeIPA Apache DOEGrids CA Nagios GridFTP Cycle Server @Lawrence GridSite LDAP RSV SRM VDT Berkley Labs Django VOMS Globus pacct WebDAV Sage Math GUMS glideinWMS Gratia Accting R-Studio GACL @FermiLab file SQL shell CLI server DB cluster Monitoring SBGrid Science Portal @ Harvard Medical School @IndianaGrid Overview - Ian Stokes-Rees ijstokes@seas.harvard.edu
    • Grid Overview - Ian Stokes-Rees ijstokes@seas.harvard.edu
    • SummaryGrid Overview - Ian Stokes-Rees ijstokes@seas.harvard.edu
    • Acknowledgements  &  Questions • Piotr  Sliz • Principle  Investigator,  head  of  SBGrid Please  contact  me   with  any  questions: • SBGrid  Science  Portal • Ian  Stokes-­‐Rees • Daniel  O’Donovan,  Meghan  Porter-­‐Mahoney • ijstokes@hkl.hms.harvard.edu • SBGrid  System  Administrators • ijstokes@spmetric.com • Ian  Levesque,  Peter  Doherty,  Steve  Jahl • Globus  Online  Team Look  at  our  work • Steve  Tueke,  Ian  Foster,  Rachana   • portal.sbgrid.org Ananthakrishnan,  Raj  Kettimuthu   • www.sbgrid.org • Ruth  Pordes • www.opensciencegrid.org • Director  of  OSG,  for  championing  SBGridGrid Overview - Ian Stokes-Rees ijstokes@seas.harvard.edu
    • Extra  SlidesGrid Overview - Ian Stokes-Rees ijstokes@seas.harvard.edu
    • Existing  Security   Infrastructure • X.509  certiBicates • Department  of  Energy  CA • Regional/Institutional  RAs  (SBGrid  is  an  RA) • X.509  proxy  certiBicate  system • Users  self-­‐sign  a  short-­‐lived  passwordless  proxy  certiBicate  used  for  “portable”   and  “automated”  grid  processing  identity  token • Similarities  to  Kerberos  tokens • Virtual  Organizations  (VO)  for  deBinitions  of  roles,   groups,  attrs • Attribute  CertiBicates • Users  can  (attempt)  to  fetch  ACs  from  the  VO  to  be  attached  to  proxy  certs • POSIX-­‐like  Bile  access  control  (Grid  ACL)  Grid Overview - Ian Stokes-Rees ijstokes@seas.harvard.edu
    • Grid Overview - Ian Stokes-Rees ijstokes@seas.harvard.edu
    • Data  Management quota du  scan tmpwatch conventions workBlow  integration Data  Movement scp  (users) rsync  (VO-­‐wide) grid-­‐ftp  (UCSD) curl  (WNs) cp  (NFS) htcp  (secure  web)Grid Overview - Ian Stokes-Rees ijstokes@seas.harvard.edu
    • 4.  pull  6iles  from UCSD  to  WNs 5.  pull  6iles  from 3.  Auto-­replicate local  NSF  to  WNs 6.  pull  6iles  from SBGrid  to  WNs red  -­  push  6iles green  -­  pull  6iles 2.  replicate  gold  standard 7.  job  results  copied   back  to  SBGrid 8a.  large  job  results   copied  to  UCSD 8b.  later  pulled  to   1.  user  6ile  upload SBGridGrid Overview - Ian Stokes-Rees ijstokes@seas.harvard.edu
    • “weak” solution 2nx5q2 Log Likelihood GainMHC-­‐TCR:  2VLJ “strong” solution 1im3a2 Translation Z scoreGrid Overview - Ian Stokes-Rees ijstokes@seas.harvard.edu
    • • NEBioGrid  Django  Portal • PyGACL Interactive  dynamic  web  portal  for   Python  representation  of  GACL  model   workBlow  deBinition,  submission,   and  API  to  work  with  GACL  Biles monitoring,  and  access  control • osg_wrap • NEBioGrid  Web  Portal Swiss  army  knife  OSG  wrapper  script  to   GridSite  based  web  portal  for  Bile-­‐system   handle  Bile  staging,  parameter  sweep,   level  access  (raw  job  output),  meta-­‐data   DAG,  results  aggregation,  monitoring tagging,  X.509  access  control/sharing,   • sbanalysis CGI data  analysis  and  graphing  tools  for   • PyCCP4 structural  biology  data  sets Python  wrappers  around  CCP4   • osg.monitoring structural  biology  applications tools  to  enhance  monitoring  of  job  set   • PyCondor and  remote  OSG  site  status Python  wrappers  around  common   • shex Condor  operations Write  bash  scripts  in  Python:  replicate   enhanced  Condor  log  analysis commands,  syntax,  behavior • PyOSG • xcon6ig Python  wrappers  around  common  OSG   Universal  conBiguration operationsGrid Overview - Ian Stokes-Rees ijstokes@hkl.hms.harvard.edu
    • 10k  grid  jobsExample  Job  Set approx  30k  CPU  hours 99.7%  success  rate evicted - red 24  wall  clock  hours completed - green held - orange MIT 5292 UWisc 1173 1077 120 1657 3 662 Cornell 840 20 Buffalo 720 628 ND 76 407 47 421 Caltech 190 FNAL 1409 237 12 24 79 4 47 UNL 6 1159 3 HMS 60 20 Purdue 349 10,000 jobs 52 17 39 UCR RENCI local queue remote queue SPRACE 1216 running 316 248Grid Overview - Ian Stokes-Rees 24 hours ijstokes@seas.harvard.edu
    • Job  LifelinesGrid Overview - Ian Stokes-Rees ijstokes@seas.harvard.edu
    • REST • Don’t  try  to  read  too  much  into  the  name • REpresentational  State  Transfer:  coined  by  Roy  Fielding,  co-­‐author  of   HTTP  protocol  and  contributor  to  original  Apache  httpd  server • Idea • The  web  is  the  worlds  largest  asynchronous,  distributed,  parallel   computational  system • Resources  are  “hidden”  but  representations  are  accessible  via  URLs • Representations  can  be  manipulated  via  HTTP  operations  GET  PUT  POST   HEAD  DELETE  and  associated  state • State  transitions  are  initiated  by  software  or  by  humans • Implication • Clean  URLs  (e.g.  Flickr)Grid Overview - Ian Stokes-Rees ijstokes@seas.harvard.edu
    • Grid Overview - Ian Stokes-Rees ijstokes@seas.harvard.edu
    • Cloud  Computing: Industry  solution  to  the  Grid • Virtualization  has  taken  off  in  the  past  5  years • VMWare,  Xen,  VirtualPC,  VirtualBox,  QEMU,  etc. • Builds  on  ideas  from  VMS  (i.e.  old) • (Good)  System  administrators  are  hard  to  come  by • And  operating  a  large  data  center  is  costly • Internet  boom  means  there  are  companies  that  have  Bigured  out   how  to  do  this  really  well • Google,  Amazon,  Yahoo,  Microsoft,  etc. • Outsource  IT  infrastructure!    Outsource  software  hosting! • Amazon  EC2,  Microsoft  Azure,  RightScale,  Force.com,  Google  Apps • Over  simpliBied: • You  can’t  install  a  cloud • You  can’t  buy  a  gridGrid Overview - Ian Stokes-Rees ijstokes@seas.harvard.edu
    • Is  “Cloud”  the  new  “Grid”? • Grid  is  about  mechanisms  for  federated,   distributed,  heterogeneous  shared  compute  and   storage  resources • standards  and  software • Cloud  is  about  on-­‐demand  provisioning  of   compute  and  storage  resources • services No  one  buys  a  grid.    No  one  installs  a  cloud.Grid Overview - Ian Stokes-Rees ijstokes@seas.harvard.edu
    • The  interesting  thing  about  Cloud  Computing  is  that   we’ve  rede7ined  Cloud  Computing  to  include   everything  that  we  already  do.  .  .  .  I  don’t  understand   what  we  would  do  differently  in  the  light  of  Cloud   Computing  other  than  change  the  wording  of  some  of   our  ads. Larry  Ellison,  Oracle  CEO,  quoted  in  the  Wall  Street  Journal,  September  26,  2008*   *http://blogs.wsj.com/biztech/2008/09/25/larry-­‐ellisons-­‐brilliant-­‐anti-­‐cloud-­‐computing-­‐rant/Grid Overview - Ian Stokes-Rees ijstokes@seas.harvard.edu
    • When  is  cloud  computing   interesting? • My  deBinition  of  “cloud  computing” • Dynamic  compute  and  storage  infrastructure  provisioning  in  a  scalable  manner  providing   uniform  interfaces  to  virtualized  resources • The  underlying  resources  could  be •  “in-­‐house”  using  licensed/purchased  software/hardware • “external”  hosted  by  a  service/infrastructure  provider • Consider  using  cloud  computing  if • You  have  operational  problems/constraints  in  your  current  data  center • You  need  to  dynamically  scale  (up  or  down)  access  to  services  and  data • You  want  fast  provisioning,  lots  of  bandwidth,  and  low  latency • Organizationally  you  can  live  with  outsourcing  responsibility  for  (some  of)  your  data  and   applications • Consider  providing  cloud  computing  services  if • You  have  an  ace  team  efBiciently  running  your  existing  data  center • You  have  lots  of  experience  with  virtualization • You  have  a  speciBic  application/domain  that  could  beneBit  from  being  tied  to  a  large  compute   farm  or  disk  array  with  great  Internet  connectivityGrid Overview - Ian Stokes-Rees ijstokes@seas.harvard.edu