Your SlideShare is downloading. ×
0
PyCon 2012Grid Job management   Felix Lee, ASGC                      1
About ASGCAcademia Sinica Grid & Cloud                               2
Something we might     need to know..• LHC• WLCG• Grid Computing                          3
LHC experiment• LHC – The Large Hadron Collider.    • It was built by European Organization for Nuclear      Research (CER...
WLCG• World-wide LHC Computing Grid• Its a distributed computing infrastructure to provide the  production and analysis en...
Grid Computing• Its one of distributed computing.• Base on federal resources.• It connects loosely-coupled computers by th...
What we do• ASGC is WLCG(World-wide LHC Computing  Grid) Tier 1 operation center since 2005• ASGC is also conducting Asia ...
Python for us                8
Python in WLCG & Grid• Its widely used for high level integration.• Clear code, clear syntax...• Totally open source.• Fas...
Python in WLCG & Grid• Work flow & Job Management.• Data Management.• Information system.• Monitoring.• HEP applications  ...
Computing system in    WLCG/Grid• They are all integrated/implemented by Python    •   WMAgent:          • Workload Manage...
Python in ASGC• Work flow & Job Management    •   GAP 1.0 (base on DIANE)    •   PanDA, collaborating with Atlas• Monitori...
GStat 2.0            13
PanDAThe Integrated GridComputing System        with      Python                      14
Work flow & Job    management• A typical Grid workflow                            15
PanDA• PanDA   •   Production and Distributed Analysis system.   •   Designed and developed by Atlas       experiment.   •...
PanDA diagram                17
PanDA Server • PanDA server design         •   Apache-based         •   Communication via HTTP/HTTPs         •   Multi-pro...
PanDA Client• PanDA client     •     Pickle module of python and native curl.     •     Client require python 2.3 or highe...
PanDA screen shot                    20
Thanks for your attention!      felix@twgrid.org
Upcoming SlideShare
Loading in...5
×

Grid Job Management

909

Published on

by 李宏德 (Felix Lee)

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
909
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
13
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Transcript of "Grid Job Management"

  1. 1. PyCon 2012Grid Job management Felix Lee, ASGC 1
  2. 2. About ASGCAcademia Sinica Grid & Cloud 2
  3. 3. Something we might need to know..• LHC• WLCG• Grid Computing 3
  4. 4. LHC experiment• LHC – The Large Hadron Collider. • It was built by European Organization for Nuclear Research (CERN) • 27KM tunnel in circumference, as deep as 175M 4
  5. 5. WLCG• World-wide LHC Computing Grid• Its a distributed computing infrastructure to provide the production and analysis environment for LHC experiment.• Currently, there are 11 tier1, 140 tier2 and several small tier3 in the world.• There are 269299 CPU cores, 183PB storage capacity in the world. 5
  6. 6. Grid Computing• Its one of distributed computing.• Base on federal resources.• It connects loosely-coupled computers by the Internet to be super virtual computer. 6
  7. 7. What we do• ASGC is WLCG(World-wide LHC Computing Grid) Tier 1 operation center since 2005• ASGC is also conducting Asia Pacific regional e-Science collaborations, development and infrastructure operation.• Developing new generation distributed computing infrastructure and technologies. 7
  8. 8. Python for us 8
  9. 9. Python in WLCG & Grid• Its widely used for high level integration.• Clear code, clear syntax...• Totally open source.• Fast and flexible implementing. • Its script. • No need to be complied. • Plenty of mathematic and science modules. 9
  10. 10. Python in WLCG & Grid• Work flow & Job Management.• Data Management.• Information system.• Monitoring.• HEP applications • Data processing. • Data analysis. 10
  11. 11. Computing system in WLCG/Grid• They are all integrated/implemented by Python • WMAgent: • Workload Manager Agent. • GRAB: • CMS Remote Analysis Builder. • PanDA: • Production and Distributed Analysis system. • DIRAC: • Distributed Infrastructure with Remote Agent Control • AliEn: • Alice Environment • DIANE: • Distributed Analysis Environment 11
  12. 12. Python in ASGC• Work flow & Job Management • GAP 1.0 (base on DIANE) • PanDA, collaborating with Atlas• Monitoring and information • GSTAT 2.0, Nagios plugin.• Integration of Grid & Cloud. • Virtual worker node on demand. • Virtual machine catalog service.• Deployment and automation. 12
  13. 13. GStat 2.0 13
  14. 14. PanDAThe Integrated GridComputing System with Python 14
  15. 15. Work flow & Job management• A typical Grid workflow 15
  16. 16. PanDA• PanDA • Production and Distributed Analysis system. • Designed and developed by Atlas experiment. • Its data driven and pull model computing. • Including workflow, resource matchmaking and job management. • We are now working with Atlas to improve and deploy it for eScience users. 16
  17. 17. PanDA diagram 17
  18. 18. PanDA Server • PanDA server design • Apache-based • Communication via HTTP/HTTPs • Multi-process • Global info in the memory resident database Apache Child process MySQL API HTTP/HTTPS DBClient Python interpreter Python interpreter DQ2 18
  19. 19. PanDA Client• PanDA client • Pickle module of python and native curl. • Client require python 2.3 or higher, curl and grid-proxy • Simple, light-weight. PanDA UserIF Client Request Pyhon Serialize (HTTPS) mod_python Obj (cPlckle) Pyhon deserialize Pyhon (cPlckle) mod_deflate Obj Response Obj (HTTPS) 19
  20. 20. PanDA screen shot 20
  21. 21. Thanks for your attention! felix@twgrid.org
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×