Most scientific computation and analyses create important transient data and preliminary results. Quick and effective assessments of these data are necessary for efficient use of researchers time and computation resources, but this process is complicated when a large collaborating team is geographically dispersed and/or some team members do not have direct access to the computation resource and output data. Current methods for sharing and assessing transient data and preliminary results are cumbersome, labor intensive, and largely unsupported by useful tools and procedures. Each research team is forced to create their own ad hoc procedures to push results from system to system, and user to user, to guide the next step in their research. The SeedMe platform provides a web-based cyberinfrastructure to enable easy sharing and streaming of transient data and preliminary results directly from computing resources to a variety of platforms, from mobile devices to workstations, and make it possible to quickly and conveniently view and assess results and provide an essential yet missing component in computing infrastructure.
1. Your results from disk to device
Stream Encode Explore and Disseminate My Experiments
SeedMe name inspired by
Seed : proliferate and grow
2. National Science Foundation
This material is based upon work supported by the National Science
Foundation under Grant No. OCI-1235505
Acknowledgements
Kristen Levy, Mahidhar Tatineni, Michael Dwyer, Tom Hutton, & Doug Weimer, SDSC
Apple Inc. : Provided test hardware/software on loan during conceptualization phase
Team
Amit Chourasia
Mona Wong
David Nadeau
Andrew Ferbert
Michael Norman
3. Presentation Overview
• Situate context & define problem
• Introduce SeedMe
• Sample use cases
• Sample end user interaction
• Q & A
6. Direct Access
Direct Web
Sharing
Not All
Members
Have Access
----
Need Exact
Location +
Permissions
Inflexible
Security
Policy
Derived Products
(Preliminary Results)
(Transient Content)
Many customers
9. Accessibility Irony
1993
“NCSA Mosaic, or
simply Mosaic, is the
web browser credited
with popularizing the
World Wide Web.”
Mosaic (Web browser), Wikipedia
http://en.wikipedia.org/wiki/Mosaic_(w
eb_browser)
1990
“The web was originally
conceived and developed
to meet the demand for
automatic information-
sharing between
scientists in universities
and institutes around the
world.”
The birth of the web. CERN
http://home.web.cern.ch/topics/birth-web
Mission Dashboard at NASA – JPL
Voyager space crafts have been sharing
information since 1977
2007
Mobile devices can
share content instantly
2014
Computation simulations cannot
share information with us
• No place to push information
(no infrastructure)
• Lack of easy tools
10. Pitfalls in sharing derived content
Download
Upload
Download
(Round Trip + 1)
Video
Encoding
Complexity
Missing Easy
Automation
How To
Describe &
Discuss
Content
Replication
&
Scalability
12. Your results from disk to device
SeedMe offers you to
• Seamlessly seed (upload) your research content
• Share and access your content
• Generate dashboards for your simulation or expt.
• Describe your content (everything is not a file)
• Discuss your work, right next to your content
Features and Benefits
Ubiquitous
Access your results anywhere on any device
Collaborate
Discuss & keep your colleagues updated with latest results at one spot
Near Real Time
Track progress from your simulation or instrument
Share and Reuse
Share your research results and reusable content widely
Sequence to Video
Convert sequence of images to videos at desired frame rate
Automate
Easily integrate your scripts, workflow or app using SeedMe tools
Security
All your communications with SeedMe are encrypted
Privacy
Control access to your content with private, group or public privileges
14. SeedMe: How it Works
Create
Collection
SignUp
1 2
Create
Collection
Web Browser Command Line or REST
(Automate scripts, workflow or app)
SignUp
1 2
Set
Metadata
Create
Collection
Set
Privacy &
Sharing
Upload
Content
Web Browser Command Line or REST
(Automate scripts, workflow or app)
SignUp
1 52 3 4
Set
Metadata
Create
Collection
Set
Privacy &
Sharing
Upload
Content
Update as
desired
Web Browser Command Line or REST
(Automate scripts, workflow or app)
SignUp
1 52 3 4
Set
Metadata
Create
Collection
Set
Privacy &
Sharing
Upload
Content
View or
Download
on Web
Update as
desired
Web Browser Command Line or REST
(Automate scripts, workflow or app)
SignUp
Phone or Tablet or Computer
(Web Browser)
1 52 3 4
15. SeedMe User Interactions
• Create Collection ( CLI, API, Web Browser)
• Update Collection ( CLI, API, Web Browser)
• View Collection (Web Browser only)
18. Use Case: Sequence to Video
Sequences of images are
generated by
Visualizations
Confocal scans
Time lapse recording
Requirements
Frame rate
High Quality Encoding
Universal playback
Automation
seedme.cmd -t “Seq Collection” # Collection title
-sp "sample/sequences/steam" # Seq Path
-st "sequence title" # Seq Title
-sd "Desc of sequence" # Seq Description
-sr "5" # Seq frame Rate
-se # Trigger Seq Encode
Command line example, but you can upload content simply via Web Browser
19. Use Case: Create Dashboards
Results from simulations,
instruments, analysis
Files
Plots
Sequences
Videos
Requirements
Managed Sharing
Universal access
Automation
seedme.cmd -t “Quick Start” # Collection title
-privacy group # Privacy
-email nobody@sdsc.edu # Share
-notify # Notify (not automatic)
-pp "sample/plots/node.png" # Plot Path
-sp "sample/sequences/steam" # Seq Path
-vp "sample/videos/air.mp4" # Video Path
-fp "sample/files/doc.pdf” # File Path
Command line example, but you can upload content simply via Web Browser
20. Use Case: Monitor
Computation state
Progress %
Progress parameters
Last file created
Requirements
Sharing
Universal access
Automation
seedme.cmd -t “Progress Tracking” # Create new Collection
seedme.cmd -update 29643 # Update collection ID
-tic "step 1" # Ticker Text
-tic "step 2" # Ticker Text
-fp "sample/files/doc.pdf” # File Path
Command line example, but you can upload content simply via Web Browser
21. Use Case: Share & Reuse
Disseminate
Share and reuse content
IPython notebook
State files from vis tools
R scripts
Matlab scripts
Applications
Exploring :
‘Save to SeedMe’
Kepler Workflow
VisIt software
YT software
Integrate SeedMe Python Module or write your own REST client
22. Demo: Web Browser
Types of Collections: My Shared Public
http://www.seedme.org/collections
You may Add, Edit and Notify collaborators
Add a new collection
https://www.seedme.org/collections/add
View a collection with monitoring info
https://www.seedme.org/collection/14924
View a collection with variety of content
https://www.seedme.org/collection/16567
23. seedme.cmd -title "SeedMe Quick Start"
seedme.cmd -update 29643 # Update
-fp "sample/files/doc.pdf" # File Path
-pp "sample/plots/node.png" # Plot Path
-sp "sample/sequences/steam" # Seq Dir
Path
-vp "sample/videos/air.mp4" # Video Path
seedme.cmd -update 29643
-privacy group
-email abc@sdsc.edu
-email xyz@sdsc.edu
seedme.cmd -update 29643 -notify
Demo: Command Line Interaction
Create a new
collection
Add image seq.
+ create video
from it
Setup sharing
Notify Group
24. Getting Started
Interaction
• Web Browser
• Standalone executable
(command line)
• Python module with API
(demo.py)
• Curl
Information
• Quick Start Guide
• Use Cases
• Documentation
• Tips for C, Fortran codes
• Blog
Each iteration in Computation cycle requires creation of succinct and consumable derived products
Derived products are created by a single user?
But usually we have more than one person interested in assessing the derived products
Problems
Problems
What are pitfalls in sharing derived products currently
Download/Upload: Manual operation (baby sit), Slow Network, Large Content
Movie Encoding: Complex & Cumbersome
Web Setup: Laborious and requires management
Replication: For each instance and each group
Scalable: from content perspective and ubiquity (desktop to smart phones)
All this requires too much plumbing
Existing tools are not really suitable for HPC research oriented needs
New content in SeedMe is called Collection
Metadata: All text, key/value pairs, tags, credits, expiration
Ticker: ephemeral text stream like stocks or tweets. Example: simulation progress, simulation state, convergence
Plots: Images, charts, documents
Image sequences: set of images
Movies