Your SlideShare is downloading. ×
0
!                                   !2011   12   21   33       @
• • • •         I/O• •    	                     2
X as a Service 		                	          	                                   3
Software as a Service                            Google Apps,         (SaaS)	                                  Salesforce....
IaaS                        	                                                         	                                   ...
6
•  HPC                                        	 •                        	                               	  AIST Super Clu...
AIST               	                        	                        	                                  F-32       10     ...
AIST                                     	                               	             	        	            	            ...
20,000                                                  	 300                     20,000                       	ACM/IEEE S...
!              NEB+Hybrid QM/MD                 !   GridRPC + MPI                 !                        (1193 CPUs)    ...
Top500   	              	                                  	                          P32	              19 	 458
AIST Green Cloud Server (AGC)	!   2010 3!      !   Dell PowerEdge M1000e ×8!      !   Dell PowerEdge M610 ×16×8 =128      ...
AIST Super Cloud Server (ASC)	!   2011 6!      !   Dell PowerEdge M1000e ×12         	!      !   Dell PowerEdge M610 ×16×1...
1/10	                                          AIST             AIST                AIST                                  ...
ASC                                                       	•      –  Pay as you go     – •                                ...
•  AIST Super Cluster•  AIST Green Cloud     AIST Super Cloud                                   1~2   –  T2K        • •  I...
18
e.g., ASC 	                          19
•      –  VMWare ESXi Xen KVM Hyper-V•      –                       OS•      –  Eucalyptus OpenStack CloudStack OpenNebula...
•  IaaS  –  VM  –  VM  –           	           	                                	      Instance	   Instance	      Instance...
•  Rocks: http://www.rocksclusters.org/   –      UCSD   –  CentOS (RHEL)                       Rolls•  Eucalyptus http://w...
Rocks	              Eucalyptus	 OpenStack	 OpenNebula	       OS               	              RHEL5          1	            ...
OpenNebula	•                  (Complutense University of     Madrid)•                          Apache License 2.0•  C12G L...
OpenNebula                                                      	                                                 CLI or S...
OpenNebula   	•  CUI  VM  ID	HostID	•  Sunstone       –  Web   GUI                              26
Contextualization                                         	•  VM                       VM     VM•      –                  ...
Contextualization                                                         	                VM                             ...
MAC_PREFIX:IP                                       	•  OpenNebula                   VM                   IP    MAC       ...
•      –           •               OS           OS     –      – •      –      –                    VLAN          •  OpenNe...
•  VM     –  OS• • •      –       VM   SDSC•                        P2V•  H/W     – •      – •                 ASC        ...
•      –  VMM•  QEMU Xen libvirt     –  Linux               libvirt   OpenStack     –  QEMU    -cpu host•      –      –   ...
I/O   	           33
…•      – •      – •      –              IO     –           •              …     –         	                             34
• •                 ←•      DB   HPC                         	                              35
IO                                           	   IO emulation	                         PCI passthrough                    ...
AIST Green Cloud	 AGC                       1                          16 HPC                                             ...
MPI Point-to-Point                                                            	                     10000                 ...
NPB BT-MZ:                                              	                                                            	    ...
Bloss:                             	                          Bloss:                                –  MPI OpenMP         ...
•                               HPC•      –         "InfiniBand PCI                           HPC                  ", SACS...
PCI                                                       	•  PCI                                           NG•  PCI      ...
GesutOS	         bond0	   eth0         eth1  (virtio)	   (igbvf)	   tap0	       Host OS	             Host OS	             ...
GesutOS	                           (qemu) device_del vf0	         bond0	   eth0         eth1  (virtio)	   (igbvf)	   tap0	...
(qemu) migrate -d tcp:x.x.x.x:y	GesutOS	                              GesutOS	         bond0	   eth0  (virtio)	           ...
(qemu) device_add pci-assign,host=05:10.0,id=vf0	           GesutOS	                                         bond0	       ...
MPI                                                                     	Guest OS	              rank 1	                   ...
48
GridARS                                                                                                                   ...
PRAGMA Grid/Clouds   UZHSwitzerland             CNIC      JLU                   AIST                        China    China...
Put	  all	  together	                                                  Store	  VM	  images	  in	  Gfarm	  systems	        ...
•  2011     –  OpenNebula        OSS              OpenStack•      –  PCI                 I/O     – • •                    ...
•      –      – •      –      – •      –      –           •  c.f.,   rough consensus and running code 	                   ...
Upcoming SlideShare
Loading in...5
×

産総研におけるプライベートクラウドへの取り組み

4,235

Published on

第33回グリッド協議会ワークショップ発表。

Published in: Technology, Business
2 Comments
10 Likes
Statistics
Notes
No Downloads
Views
Total Views
4,235
On Slideshare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
52
Comments
2
Likes
10
Embeds 0
No embeds

No notes for slide

Transcript of "産総研におけるプライベートクラウドへの取り組み"

  1. 1. ! !2011 12 21 33 @
  2. 2. • • • •  I/O• •  2
  3. 3. X as a Service 3
  4. 4. Software as a Service Google Apps, (SaaS) Salesforce.com Platform as a Service Web Google App Engine, (PaaS) Windows Azure Infrastructure as a Service (IaaS) Amazon EC2 4
  5. 5. IaaS •  •  Pay-as-you-go •  “ ” •  Web •  •  •  IT •  •  Cloud bursting 5
  6. 6. 6
  7. 7. •  HPC •  AIST Super Cluster AIST Super Cloud2004 3 2010 2 AIST Green Cloud 2011 6 2010 3 7
  8. 8. AIST F-32 10 P-32 M-64 79
  9. 9. AIST 
 
 
 

  10. 10. 20,000 300 20,000 ACM/IEEE Supercomputing 2005 Best Paper Award “Full Electron Calculation Beyond 20,000 Atoms: Ground Electronic State of Photosynthetic Proteins”
  11. 11. !  NEB+Hybrid QM/MD !   GridRPC + MPI !  (1193 CPUs) !  58 5000 QM QM QM NCSA 64×4 CPUs for QM QM QM SDSC 64×2 CPUs for QM Purdue 64×4 CPUs for QM USC 64×1 CPUs for QM QM QM QM QM NCSA MPI USC MPI RPC RPC Purdue AIST SDSC QM QM QM QM QM QM MD MD RPC QM QM RPC MPI MPI NEB scheduler AIST F32 41×1 CPUs for MD Number of Execution AIST F32 64×3 CPUs for QM MD MD AIST P32 64×4 CPUs for QM MPIsystem energy end-1 end-2 reaction 0 Elapsed time
  12. 12. Top500 P32 19 458
  13. 13. AIST Green Cloud Server (AGC) !   2010 3!  !   Dell PowerEdge M1000e ×8!  !   Dell PowerEdge M610 ×16×8 =128 !  Intel Xeon E5540 × 2 2.53GHz, 8MB ) !   48GB !   300GB HDD ×2!  ! Mellanox M3601Q FI Infiniband I/F 4X QDR 32port!  !   Gigabit Ethernet
  14. 14. AIST Super Cloud Server (ASC) !   2011 6!  !   Dell PowerEdge M1000e ×12 !  !   Dell PowerEdge M610 ×16×12 =192 !  Intel Xeon E5620 × 2 2.4GHz, 12MB ) !   24GB !   600GB HDD ×2!  !   10GB Ethernet!  !   Gigabit Ethernet! 
  15. 15. 1/10 AIST AIST AIST ASC AGC ASC 2004 2009 2011 (M ) 1500 76 66 (M /5 ) 400 28 43 1408 128 192 2816 1024 1536 (TFlops) 14.6 10.4 14.6 (KW) 800 63 86 (KW) 460 40 48 (Gflops/KW) 18.25 165.08 169.77 (Gflops/KW) 31.74 260 304.17 /Tflops) 1 760 450 15
  16. 16. ASC •  –  Pay as you go – •  200 BMM BMM KVM 150 KVM OpenNebula 100 KVM OS: Scientific Linux 6.0 OS: CentOS 5.6 CentOS 4.9 openSUSE 11.4 50 BMM CentOS 5.6 openSUSE 11.4 0 16
  17. 17. •  AIST Super Cluster•  AIST Green Cloud AIST Super Cloud 1~2 –  T2K • •  IT 17
  18. 18. 18
  19. 19. e.g., ASC 19
  20. 20. •  –  VMWare ESXi Xen KVM Hyper-V•  –  OS•  –  Eucalyptus OpenStack CloudStack OpenNebula Nimbus Wakame Rocks Condor VMWare vSphere… OSS 20
  21. 21. •  IaaS –  VM –  VM –  Instance Instance Instance Instance 21
  22. 22. •  Rocks: http://www.rocksclusters.org/ –  UCSD –  CentOS (RHEL) Rolls•  Eucalyptus http://www.eucalyptus.com/ –  UCSB Amazon EC2/S3•  OpenStack: http://openstack.org/ –  NASA Rackspace •  NASA Eucalyptus•  OpenNebula http://opennebula.org/ –  OpenNebula OpenStack 22
  23. 23. Rocks Eucalyptus OpenStack OpenNebula OS RHEL5 1 VMM Xen Xen KVM Xen KVM VMWare Hyper-V VM VLAN /home libvirt VMM VM 23
  24. 24. OpenNebula •  (Complutense University of Madrid)•  Apache License 2.0•  C12G Labs http://opennebula.org/about:about (FP7) 24
  25. 25. OpenNebula CLI or Sunstone GUI global network frontend Sche ONED VM host 1 VM host 2 duler VMM VMM /srv/cloud /srv/cloud SSH /srv/cloud SSH |-- one `-- images VM local network VM NFS scp 25
  26. 26. OpenNebula •  CUI VM ID HostID •  Sunstone –  Web GUI 26
  27. 27. Contextualization •  VM VM VM•  –  –  root –  SSH VM oned –  /etc/hosts /etc/resolv. conf –  NFS from OpenNebula Documentation 27
  28. 28. Contextualization VM context.sh (test.one) � � � � � � � � � �� � � � � � � � � � % onevm create test.one � � � 28
  29. 29. MAC_PREFIX:IP •  OpenNebula VM IP MAC MAC IP•  VM MAC IP NIC % onevnet show 1 (snip) LEASES INFORMATION LEASE=[IP=192.168.57.209, MAC=02:00:c0:a8:39:d1, USED=0, VID=-1] IP ”02:00” IP 4 octet 16 29
  30. 30. •  –  •  OS OS –  – •  –  –  VLAN •  OpenNebula 3.0 –  PCI I/O –  30
  31. 31. •  VM –  OS• • •  –  VM SDSC•  P2V•  H/W – •  – •  ASC 31
  32. 32. •  –  VMM•  QEMU Xen libvirt –  Linux libvirt OpenStack –  QEMU -cpu host•  –  –  – •  I/O – • •  32
  33. 33. I/O 33
  34. 34. …•  – •  – •  –  IO –  •  … –  34
  35. 35. • •  ←•  DB HPC 35
  36. 36. IO IO emulation PCI passthrough SR-IOVVM1 VM2 VM1 VM2 VM1 VM2 Guest OS Guest OS Guest OS … … … Guest Physical Physical driver driver driver VMM VMM VMM vSwitch Physical driver NIC NIC NIC Switch (VEB) IO emulation PCI passthrough SR-IOV VM sharing ✔ ✖ ✔ Performance ✖ ✔ ✔ 36
  37. 37. AIST Green Cloud AGC 1 16 HPC Compute node Dell PowerEdge M610 Host machine environment CPU Intel quad-core Xeon E5540/2.53GHz x2 OS Debian 6.0.1 Chipset Intel 5520 Linux kernel 2.6.32-5-amd64 Memory 48 GB DDR3 KVM 0.12.50 InfiniBand Mellanox ConnectX (MT26428) Compiler gcc/gfortran 4.4.5 MPI Open MPI 1.4.2 Blade switch VM environment InfiniBand Mellanox M3601Q (QDR 16 ports) VCPU 8 Memory 45 GB 1 1 VM 37
  38. 38. MPI Point-to-Point 10000 (higher is better) 1000Bandwidth [MB/sec] 100 PCI KVM 10 Bare Metal Bare Metal KVM 1 1 10 100 1k 10k 100k 1M 10M 100M 1G Message size [byte] Bare Metal: 38
  39. 39. NPB BT-MZ: (higher is better) 300 100Performance [Gop/s total] 250 Degradation of PE: Parallel efficiency [%] 80 KVM: 2%, EC2: 14% 200 Bare Metal 60 150 KVM Amazon EC2 40 100 Bare Metal (PE) KVM (PE) 20 50 Amazon EC2 (PE) 0 0 1 2 4 8 16 Number of nodes 39
  40. 40. Bloss: Bloss: –  MPI OpenMP 120 100Parallel Efficiency [%] 80 60 Degradation of PE: KVM: 8%, EC2: 22% 40 20 Bare Metal KVM Amazon EC2 Ideal 0 1 2 4 8 16 Number of nodes 40
  41. 41. •  HPC•  –  "InfiniBand PCI HPC ", SACSIS2011, pp.109-116, 2011 5 . •  KVM Xen PVM NAB-MZ Bloss –  "HPC ", ACS37 . •  NUMA KVM Xen HVM –  Takano, et al., "Toward a practical "HPC Cloud": Performance tuning of a virtualized InfiniBand cluster", CUTE2011, December 2011. •  VMM HPC Challenge benchmark 41
  42. 42. PCI •  PCI NG•  PCI Bonding –  PCI NIC –  NIC IO NIC active-standby bonding –  S•  SR-IOV NIC VF IO PV 1 NIC 42
  43. 43. GesutOS bond0 eth0 eth1 (virtio) (igbvf) tap0 Host OS Host OS tap0 br0 br0 eth0 eth0 (igb) (igb) SR-IOV NIC SR-IOV NIC 43
  44. 44. GesutOS (qemu) device_del vf0 bond0 eth0 eth1 (virtio) (igbvf) tap0 Host OS Host OS tap0 br0 br0 eth0 eth0 (igb) (igb) SR-IOV NIC SR-IOV NIC 44
  45. 45. (qemu) migrate -d tcp:x.x.x.x:y GesutOS GesutOS bond0 eth0 (virtio) $ qemu -incoming tcp:0:y ... tap0 Host OS Host OS tap0 br0 br0 eth0 eth0 (igb) (igb) SR-IOV NIC SR-IOV NIC 45
  46. 46. (qemu) device_add pci-assign,host=05:10.0,id=vf0 GesutOS bond0 eth0 eth1 (virtio) (igbvf) tap0 Host OS Host OS tap0 br0 br0 eth0 eth0 (igb) (igb) SR-IOV NIC SR-IOV NIC 46
  47. 47. MPI Guest OS rank 1 → bond0 eth0 eth1 (virtio) (igbvf) tap0 192.168.0.1 tap0 192.168.0.2 192.168.0.3 br0 rank 0 br0 eth0 eth0 (igb) (igb) SR-IOV NIC SR-IOV NIC NIC 192.168.0.0/24 47
  48. 48. 48
  49. 49. GridARS •  RMS   –    –  GRC  :     User –  RM  (CRM/NRM/SRM)  :     Domain  0 •    ID GRC DMS/A  AEM   –    –  CRM   GRC DMS/A GRC DMS/A •    CRM Domain  2 CRM DMS   NRM DMC/C NRM DMC/C –    –  DMS/A  :     CRM DMC/C DMC/C GRS DMC/A SRM SRM –  DMS/C  :     CRM DMC/C Domain  1 Domain  3 49
  50. 50. PRAGMA Grid/Clouds UZHSwitzerland CNIC JLU AIST China China KISTI OsakaU IndianaU KMU UTsukuba SDSC USA LZU Korea Japan China USA ASGC HKU NCHC UoHyd HongKong Taiwan India ASTI NECTEC Philippines CeNAT-ITCR KU HCMUT Costa Rica Thailand HUT IOIT-Hanoi UValle MIMOS IOIT-HCM Colombia USM Vietnam Malaysia MU BESTGrid UChile Australia New Zealand Chile 26 institutions in 17 countries/regions, 23 compute sites, 10VM sites 50
  51. 51. Put  all  together   Store  VM  images  in  Gfarm  systems   gFC   Run  vm-­‐deploy  scripts  at  PRAGMA  Sites   gFC  VM  Image   Copy  VM  images  on  Demand  from  gFarm   VM  Image  copied  from     CondorgFarm   slave gFS   Modify/start  VM  instances  at  PRAGMA  sites   Master copied  from     gFarm   slave SDSC  (USA)   Manage  jobs  with  Condor   AIST  (Japan)   Rocks  Xen   OpenNebula  KVM   gFS   GFARM  Grid  File   gFC   System  (Japan)   gFC  VM  Image   AIST  QuickQuake  +  Condor   VM  Image  copied  from     copied  from    gFarm   slave gFS   NCHC  FmoRf   gFS   gFarm   slave gFS   UCSD  Autodock  +  Condor   gFS   NCHC  (Taiwan)   IU  (USA)  OpenNebula  KVM   AIST  Web  Map  Service  +  Condor   Rocks  Xen   AIST  Geogrid  +  Bloss   AIST  HotSpot  +  Condor     gFC   gFS   gFS   gFC  VM  Image   VM  Image  copied  from     copied  from    gFarm   slave gFS   gFS   gFarm   slave LZU  (China)   =  VM  deploy  Script   Osaka  (Japan)   Rocks  KVM   gFC   =  Grid  Farm  Client   Rocks  Xen   gFS   =  Grid  Farm  Server   51
  52. 52. •  2011 –  OpenNebula OSS OpenStack•  –  PCI I/O – • •  52
  53. 53. •  –  – •  –  – •  –  –  •  c.f., rough consensus and running code 53
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×