D-Grid Infrastructure
Stefan Freitag
Dortmund University of Technology
6th International Workshop on Grid Computing for
Co...
Outline
●
Projects & Infrastructure
●
Lessons learned & Future Challenges
2004 2007 20102009 20112008 2012
Gap project
20062005
Integration project
Service layer
Academic projects
Public/ private ...
Gap projects
Service layer
projects
Commercial community
projects
Academic community
projects
Integration project
Project ...
Representation (?)
Academic service and
resource providers
Academic research
communities
D – Grid GmbH
Industrial service ...
Academic projects
Astronomy
Climate Research
High Energy Physics
Medical Science
Humanities
Aircraft Industry
Financial Business
Automobile Industry
Building & Construction Industry
Geospatial Information Systems
M...
Commercial Projects
Logistics
Media Technology
Chemical Industry
Numerical Simulation
Medical Engineering
Plasma Technolog...
Service Projects
Backup & Recovery
Sustainable Grid Infrastructure
SLA4D-Grid
Networks of Knowledge in the Grid
Gap & Service Projects
VO management
D-MON
DGSI
GapSLC
GIDS
D-Grid Infrastructure
Situation in 2005
High energy physicists
● advanced knowledge (CERN experiments)
● gLite middleware
Other communities
● no...
Leere Fusszeile
Supported Middlewares
Compute middleware
●
gLite
●
UNICORE
●
Globus Toolkit
Storage middleware
●
dCache
●
...
Leere Fusszeile
Software Stack @ D-Grid Resource
Leere Fusszeile
D-Grid Reference System
● Template for a D-Grid compute and storage resource
● Targeted at small and mediu...
D-Grid Resources
●
> 30.000 CPU cores
●
> 5 PByte of storage
●
In 2009 >= 140 Mio.
CPUh consumed by
users
●
Accounting
●
D...
Usage of D-Grid Resources
Towards Sustainability
Technology is not really a problem!
Focus on existing concepts
● Support for different Grid middlewares
● Virtualization
B...
Heterogeneous Provider Community
Some large hardware providers play a vital role in D-Grid
● Development of a business mod...
There are not enough users!
Prominent figures of the various disciplines have not yet joined
● Imbalance between computer ...
Grid computing is a solution, isn’t it?
Funding agencies cannot be wrong!
●
More than 100 Mio € in D-Grid
Industry cannot ...
Due to the weather conditions...
Cloud Computing!
Photo by: Paolo Malgari
Cloud Computing in D-Grid ?
Situation in 2009
● Utilization of compute resources ≥ 140 Mio CPUh
● only around 50% of the a...
Attractivity of Cloud Interface I
Cloud
● Register once via web. Specify name, credit card
information,... (few minutes)
G...
Attractivity of Cloud Interface II
Cloud Computing in D-Grid
●
Extension of the
software stack
●
What to add?
●
Platform
virtualization
●
Compute cloud
middl...
D-Grid Cloud Prototype
Located at Dortmund University of Technology
Services
● Compute middlewares
– gLite 3.2
– UNICORE 6...
OpenNebula
Statement on homepage
● Standard-based open-source toolkit to build private, public and
hybrid clouds. [...] pr...
Prototyp Setup
Open Issues
Technical level
●
User management
●
Authorization
●
Accounting and Billing
●
Monitoring
●
Information system
→...
User Management
D-Grid
● VO central user management
● Resources query VOMRS servers and generate a mapping
● VO member → l...
Lessons learned
Reference System Acceptance
Only a few sites
●
Follow the recommended setup
●
Use the technical documentation
Why should a...
Legal Issues – Dual Use
Dual-use technology: a technology which can be used for
both peaceful and military aims.
Most indu...
Legal Issues – Liability
With the development of business models, the liability of single
VO members and of the whole VO i...
Avoid Parallel Universes
Before D-Grid German HEPs used WLCG resources
In 2010 German HEPs still use WLCG resources
● and ...
Future Challenges
Future Challenge: D-Grid/ NGI-DE
EGI was still EGEE as D-Grid started
With the end of EGEE-III, NGI-DE was established
Sin...
New Service: D-Mon
Summary
Projects in D-Grid
D-Grid Infrastructure
●
Software stack, resources
●
Prototypic integration of a Cloud middlewar...
Thank you for your attention!
If you found this talk interesting, slides are
available at
http://www.slideshare.net/stefan...
Upcoming SlideShare
Loading in...5
×

D-Grid Infrastructure

939

Published on

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
939
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
10
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

D-Grid Infrastructure

  1. 1. D-Grid Infrastructure Stefan Freitag Dortmund University of Technology 6th International Workshop on Grid Computing for Complex Problems Bratislava, November 2010
  2. 2. Outline ● Projects & Infrastructure ● Lessons learned & Future Challenges
  3. 3. 2004 2007 20102009 20112008 2012 Gap project 20062005 Integration project Service layer Academic projects Public/ private projects D-Grid GmbH Working Groups Pre-project Phase 1 Phase 2 Phase 3 D-Grid Projects
  4. 4. Gap projects Service layer projects Commercial community projects Academic community projects Integration project Project structure of D-Grid
  5. 5. Representation (?) Academic service and resource providers Academic research communities D – Grid GmbH Industrial service and resource providers Organizational issues Legal issues International issues Public Relation GAUSS-Allianz WissGrid Representation structure in D-Grid
  6. 6. Academic projects Astronomy Climate Research High Energy Physics Medical Science Humanities
  7. 7. Aircraft Industry Financial Business Automobile Industry Building & Construction Industry Geospatial Information Systems Medical Services Meteorology for regenerative energy industry Commercial Projects
  8. 8. Commercial Projects Logistics Media Technology Chemical Industry Numerical Simulation Medical Engineering Plasma Technology Dynamic Service Value Networks
  9. 9. Service Projects Backup & Recovery Sustainable Grid Infrastructure SLA4D-Grid Networks of Knowledge in the Grid
  10. 10. Gap & Service Projects VO management D-MON DGSI GapSLC GIDS
  11. 11. D-Grid Infrastructure
  12. 12. Situation in 2005 High energy physicists ● advanced knowledge (CERN experiments) ● gLite middleware Other communities ● not really any prior knowledge How to make all communities happy? ● Let them choose! ● Community requirements still unclear ● Offer a set of commonly used middlewares
  13. 13. Leere Fusszeile Supported Middlewares Compute middleware ● gLite ● UNICORE ● Globus Toolkit Storage middleware ● dCache ● OGSA-DAI
  14. 14. Leere Fusszeile Software Stack @ D-Grid Resource
  15. 15. Leere Fusszeile D-Grid Reference System ● Template for a D-Grid compute and storage resource ● Targeted at small and medium resource centers ● Technical documentation & software packages available to system administrators ● http://dgiref.d-grid.de/wiki/Introduction ● Validation of packages and documentation by installation “from scratch” ● Release Cycle: six months ● Reference system is only a recommendation ● Support available
  16. 16. D-Grid Resources ● > 30.000 CPU cores ● > 5 PByte of storage ● In 2009 >= 140 Mio. CPUh consumed by users ● Accounting ● DGAS/ manually
  17. 17. Usage of D-Grid Resources
  18. 18. Towards Sustainability
  19. 19. Technology is not really a problem! Focus on existing concepts ● Support for different Grid middlewares ● Virtualization Bridging the gap to services and applications ● Workflow engines ● Commercial software Addressing new requirements ● Dynamic virtual organizations Cooperation between different projects ● Scheduling
  20. 20. Heterogeneous Provider Community Some large hardware providers play a vital role in D-Grid ● Development of a business model for basic services Some providers used D-Grid to acquire new hardware ● Investments to support a transitional phase to Grid systems No known concepts to provide IaaS type of services ● Obstacle for commercial users ● Competition to cloud computing?
  21. 21. There are not enough users! Prominent figures of the various disciplines have not yet joined ● Imbalance between computer science and application science Most users are not willing to provide the required additional effort. ● Potential lack of focus in the application area Industrial users are pushed off by bureaucratic obstacles ● Use of D-Grid hardware for commercial purpose
  22. 22. Grid computing is a solution, isn’t it? Funding agencies cannot be wrong! ● More than 100 Mio € in D-Grid Industry cannot be wrong! ● Enterprise Grids ● Cloud Computing Science cannot be wrong! ● LHC production Grid
  23. 23. Due to the weather conditions... Cloud Computing! Photo by: Paolo Malgari
  24. 24. Cloud Computing in D-Grid ? Situation in 2009 ● Utilization of compute resources ≥ 140 Mio CPUh ● only around 50% of the available CPUh ½ of D-Grid compute resources idling ● Turn them off → GreenIT ● Not really acceptable for a sustainable operation → Introduce compute cloud interface as new pillar Why ? ● Grid is too complex and hence not attractive to SMEs
  25. 25. Attractivity of Cloud Interface I Cloud ● Register once via web. Specify name, credit card information,... (few minutes) Grid ● Application for X.509 certificate (1-2 days) Contact Certification authority Contact Registration authority (f2f required) ● Application for VO membership (hours - days) ● Updating of the authorization information on the Grid resources (hours - days)
  26. 26. Attractivity of Cloud Interface II
  27. 27. Cloud Computing in D-Grid ● Extension of the software stack ● What to add? ● Platform virtualization ● Compute cloud middleware
  28. 28. D-Grid Cloud Prototype Located at Dortmund University of Technology Services ● Compute middlewares – gLite 3.2 – UNICORE 6 – Globus Toolkit 4.0.8 ● Storage middlewares ● dCache 1.9.5 ● OGSA-DAI WSRF 2.2 Hardware ● 2048 CPU cores, 108 TByte storage All services deployed in Xen virtual machines
  29. 29. OpenNebula Statement on homepage ● Standard-based open-source toolkit to build private, public and hybrid clouds. [...] provides flexible architecture, interfaces and components that fit into any existing data center. Supports Xen, KVM and VMware Interfaces with Amazon EC2 and ElasticHosts Other users ● CERN (cloud infrastructure consisting of 3,200 cores) ● NIKHEF, Netherlands
  30. 30. Prototyp Setup
  31. 31. Open Issues Technical level ● User management ● Authorization ● Accounting and Billing ● Monitoring ● Information system → Interfaces to all D-Grid core services
  32. 32. User Management D-Grid ● VO central user management ● Resources query VOMRS servers and generate a mapping ● VO member → local user ID ● Attribute-based authorization Cloud (OpenNebula) ● Local user management, ● No Attribute-based authorization Can the concept of VOs and Cloud middleware go together?
  33. 33. Lessons learned
  34. 34. Reference System Acceptance Only a few sites ● Follow the recommended setup ● Use the technical documentation Why should a resource install all middlewares? ● System administrators are not qualified to support all middlewares ● Lack of interest at resource provider level Migration/ Upgrade path unspecified
  35. 35. Legal Issues – Dual Use Dual-use technology: a technology which can be used for both peaceful and military aims. Most industrial countries have export controls on certain types of designated dual-use technologies […] These controls restrict the export of certain commodities and technologies without the permission of the government. (Wikipedia) ● Is Grid a dual-use technology? ● Export embargo for dual-use technologies to Iran ● Technical solution for a legal requirement
  36. 36. Legal Issues – Liability With the development of business models, the liability of single VO members and of the whole VO is of interest What is the legal status of a virtual organization? ● Company constituted under Civil law → Joint and several liability ● In joint and several agreements, the liability for default is enforceable against all of the signatories as a group[...], or against any one of them as an individual at the choice of the enforcing party (www.businessdictionary.com) Can business models and virtual organizations co-exist?
  37. 37. Avoid Parallel Universes Before D-Grid German HEPs used WLCG resources In 2010 German HEPs still use WLCG resources ● and not a single one uses D-Grid resources Why? ● Accessibility to compute resources, data and experiment software is limited by the boundaries of a virtual organization ● In WLCG VOs for ALICE, ATLAS, CMS and LHCb were already existing ● In D-Grid a HEP-CG VO was established with the aim to absorb all HEPs independent from their origin ● Even if HEPs joined the HEP-CG VO, they were not able to access the experiment software
  38. 38. Future Challenges
  39. 39. Future Challenge: D-Grid/ NGI-DE EGI was still EGEE as D-Grid started With the end of EGEE-III, NGI-DE was established Since then D-Grid and NGI-DE co-exist in Germany Merge to only one initiative ● Avoid competition ● Combine endeavors for creating a sustainable infrastructure ● Removal of duplicity of service (GRRS/ GOC DB) ● Integration of D-Grid services (D-Mon) ● Operations concept (1:1 mapping, pool accounts)
  40. 40. New Service: D-Mon
  41. 41. Summary Projects in D-Grid D-Grid Infrastructure ● Software stack, resources ● Prototypic integration of a Cloud middleware Lessons learned Legal issues & Future challenges
  42. 42. Thank you for your attention! If you found this talk interesting, slides are available at http://www.slideshare.net/stefanfreitag
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×