Esi web2.0 may2010


Published on

The Influence and Impact of Web 2.0 on e-Research Infrastructure, Applications and Users.

  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Esi web2.0 may2010

  1. 1. Exploiting Web 2.0 for Scientific Simulation<br />Gabrielle Allen<br />Department of Computer Science<br />Center for Computation & Technology<br />Louisiana State University<br />Allen,Loeffler,Radke,Schnetter, Seidel,Integrating Web 2.0 Technologies with Scientific Simulation Codes for Real-Time Collaboration, IEEE International Conference on Cluster Computing (Cluster 2009), Workshop on The Impact and Influence of Web 2.0 on e-Research Infrastructure, Services and Applications.<br />
  2. 2. Gravitational Wave Physics<br />Models<br />Analysis & Insight<br />Observations<br />Petascale problems: Full 3D general relativistic models of binary systems, supernova, gamma-ray bursts<br />
  3. 3. Understanding Gravity<br />Data and Collaboration increasing<br />Log(Data)<br />Galileo<br />Smarr<br />LSU PRAC<br />
  4. 4. Cactus Framework<br />Component-based HPC framework: <br />Freely available, modular, portable, manageable environment for collaboratively developing parallel, multi-dimensional simulation codes<br />Enabling applications: <br />Numerical Relativity/Astrophysics, CFD, Coastal, Reservoir Engineering, Quantum Gravity, …<br />Finite difference, AMR, FE/FV, multipatch, …<br /><ul><li>Cutting edge CS:
  5. 5. Grid computing, petascale, accelerators, steering, remote viz, application driver
  6. 6. Active user & developer communities:
  7. 7. 12 year pedigree, led from LSU. Over $10M support : NSF, EU, DOD, DOE, NASA, Microsoft, MPG, LSU, NCSA.</li></li></ul><li>Cactus Structure<br />Plug-In “Thorns”<br />(components)<br />remote steering<br />extensibleAPIs<br />ANSI C<br />driver<br />Fortran/C/C++<br />parameters<br />input/output<br />scheduling<br />equations of state<br />interpolation<br />Core “Flesh”<br />errorhandling<br />SOR solver<br />Your Physics !!<br />makesystem<br />wave evolvers<br />Computational<br />Tools !!<br />gridvariables<br />multigrid<br />black holes<br />coordinates<br />boundaryconditions<br />
  8. 8. Cactus Structure<br />Web 2.0 <br />Plug-In “Thorns”<br />(components)<br />remote steering<br />extensibleAPIs<br />ANSI C<br />driver<br />Fortran/C/C++<br />parameters<br />input/output<br />scheduling<br />equations of state<br />interpolation<br />Core “Flesh”<br />errorhandling<br />SOR solver<br />Your Physics !!<br />makesystem<br />wave evolvers<br />Computational<br />Tools !!<br />gridvariables<br />multigrid<br />black holes<br />coordinates<br />boundaryconditions<br />
  9. 9. Cactus Application Environment<br />Individual research groups<br />Domain specific shared infrastructure<br />Flesh: APIs, information, orchestration<br />Adaptive mesh refinement, parallel I/O, interaction, …<br />
  10. 10. Einstein Toolkit<br />Based on Cactus Framework<br />Over 130 open, community developed Cactus modules<br />Building a consortium of users<br />Governance and software development<br />Members<br />40 listed on web page<br />10 different groups<br />US, Japan, Mexico, Spain, Germany, Canada<br />300 science publications, 50 student theses<br />
  11. 11. Typical Black Hole Simulations<br />At LSU …<br />300 Cactus thorns<br />10,000 potential parameters<br />20 different supercomputers<br />100-2000 cores<br />Days/weeks to run (checkpoint/restart)<br />GBs to TBs of data (HDF5, ASCII, jpeg)<br />
  12. 12. Collaborative Technologies<br />Technologies to share simulation-related information developed in our group from the early 1990s<br />Essential to support the scientific research<br />Review historical evolution of these technologies<br />Show how Web 2.0 provides new tools to enable old scenarios<br />
  13. 13. Web-based Mail Lists<br />Mosaic web browser (1993, NCSA)<br />Seidel’s group at NCSA worry about content<br /><br />Collaborative Cork Board (CoCoBoard) (Mid 90’s)<br />Researchers have web-based “project pages”<br />Could attach images!! (usually 1-D plots of results)<br />Used till late 90’s<br />Currently<br />Project based private wikis: parameter/output files, figures<br />Organize material for weekly project conference calls<br />Cons: network to access/edit wiki, editing slow<br />
  14. 14. CoCoBoard<br />
  15. 15. Simulation Web Interfaces<br />Thorn “httpd”<br />First collaborative tool fundamentally integrated into Cactus<br />Werner Benger (1999), visiting NCSA from Germany (7 hr time difference and email)<br />Used socket library developed for remote viz (John Shalf & TIKSL project)<br />Thorn “HTTPD” in standard toolkit (2000)<br />Simulation status, variables, timing, viewport, output files, parameter steering, etc<br />Thorns can include their own web content<br />
  16. 16. Issues<br />Authorization to web pages (username/password in parameter file is insecure and awkward, newer version uses https and can also use X.509)<br />Browsers can display images in certain formats, a Visualization thorn uses gnuplot to include e.g. performance with time, physical parameters<br />Problem deploying on compute nodes where web server cannot be directly accessed (port forwarding, filewalls)<br />How to find and track the simulations, publicize existence to a collaboration?<br />
  17. 17. Cactus HTTPD Simulation Page<br />
  18. 18. Cactus HTTPD Viewport<br />
  19. 19. Simulation Reports and Email<br />Readable report automatically generated for each simulation (computation and physics)<br />Prototyped 2001 but not used (?)<br />How to collect reports in one place?<br />Mail Thorn (sendmail)<br />Email reliable and fault tolerant (spool)<br />Supercomputers do not allow mail to be sent from compute nodes.<br />
  20. 20. GridLab Visualization Service<br />BryggUllmer (2004)<br />
  21. 21. Announcing and Grid Portals<br />Collaborations need reliable, live information about long running simulations.<br />NSF Astrophysics Simulation Collaboratory (ASC), 1999<br />Grid Portal provided centralized, collaborative interface to submit, monitor and archive simulations<br />Java, JSP, Javascript with back-end data base, contributed to GridSphere design (GridLab)<br />JavaCOG to submit jobs and basic monitoring.<br />ASC Portal (2002)<br />
  22. 22. Announcing Simulation Info<br />Publish (application provided) simulation information<br />Thorn Announce, in prototype Cactus Worm scenario (2001)<br />Message from Flesh/Thorn info<br />Transport: XML-RPC to remote socket (portal)<br />Issues<br />Job IDs<br />Security, mapping users <br />Cumbersome user set parameters (portal location, visibility of job, notification needs)<br />Announcing to ASC Portal (2002)<br />
  23. 23. Notification<br />Portal notification service<br />Portal users configure at portal, simulations configure in parameter file<br />Email, SMS, Instant Message<br />Initial experiments generated large telecom bills!<br />Cool and useful, but lots of work (FTE) to develop and modify portal service, difficult to configure. <br />
  24. 24. Web 2.0 Technologies<br />Use for collaborative, simulation-level messaging and information archiving<br />Reliable, persistent, well-documented, user-configurable, cheap, well supported, good APIs<br />
  25. 25. Twitter<br />March 2006<br />Real-time short messaging system. Users send and receive each others updates (tweets). Wide range of devices and rudimentary social networking. <br />Receivers can filter messages they see and specify how they receive them<br />Twitter API (e.g. post a new Twitter message from a user)<br />Free<br />
  26. 26. Thorn Twitter<br />Uses libcurl<br />Cactus parameters for twitter username/password<br />Twitter API: statuses/update<br />At LSU “numrel” group account<br />Messages when simulation starts and at different stages<br />
  27. 27. Flickr<br />2004, image hosting website for digital photographs (and now videos). Bought by Yahoo (2005).<br />Professional account ($25/yr) for unlimited use<br />Web service API for uploading and manipulating images<br />Group images into Sets and Collections<br />Tags, title, description, metadata from EXIF headers<br />Social networking: users can comment on images, flag them, order by popularity, etc. Public/Private/Friends/Family. Blogs.<br />RSS field allows quick previewing.<br />
  28. 28. Thorn Flickr<br />Send images from running simulation<br />Uses: flickcurl, libcurl, libxml2, openssl<br />Authentication more complex (api key, shared secret)<br />Thorn uploads images that are generated by Cactus (and known to I/O layer), e.g. IoJpeg<br />Each simulation given its own Flickr set<br />
  29. 29. Future Work<br />Extend capabilities, production testing<br />Common authentication mechanism<br />Social networking model (individual/shared accounts)<br />Development of common tags, more metadata etc<br />Storing videos (Flickr, YouTube, Vimeo)<br />Advantage for scientists presenting<br />Lots of other possibilities: DropBox to publish files across a collaboration, WordPress for simulation reports/blogs, FaceBook to replace grid portals and aggregate services, Cloud computing APIs for “grid” scenarios, … <br />
  30. 30. Einstein Toolkit<br />Trying to establish a community for computational relativity:<br />Wiki for community documentation<br />Blog for community posting<br /><br />
  31. 31. Conclusions<br />Started as a fun project (undergrad)<br />Web 2.0 provide reliable delivery, storage, access, and flexible collaborative features<br />Can use Web 2.0 to easily prototype new interactive and collaborative scenarios (have really missed this)<br />Small groups and individuals can do this too!!<br />Target standard of ease-of-use for cyberinfrastructure development <br />For real use need unified authentication, clear policies on data, site versions<br />