SlideShare a Scribd company logo
Singularity Registry HPC
Collaborative Container Modules
Vanessa Sochat PhD
Computer Scientist
Lawrence Livermore National Lab
A formula for collaboration?
Identify an actual need that crosses communities
Singularity Registry HPC
How do we provide containers?
The Problem:
● Public registries (Docker Hub, Quay.io) or self deployed (Singularity
Registry Server, Docker Registry)
● Each user separately pulling means a lot of redundancy
● Pulling “latest” means we aren’t aware of versioning
● It’s not impossible to be reproducible, but it’s harder, and best practices
are hard to follow!
How do we provide containers?
● HPC admins can provide a shared collection of containers
● Users could also install a collection, if not provided by an admin
● It’s done so in a way so that using the containers is reproducible.
Ideally:
A formula for collaboration?
Identify an actual need that crosses communities
Have an idea that fits into a current way of thinking
Singularity and LMOD? 🤔
Hmm 🤔
Hmm 🤔
What is wrong with current solutions?
● Not easy enough to collaborate on
● Not modular enough
● Not enough granularity in defining container versions or interactions
Reproducibility
01
Define exact tags
and digests to install
and load.
Reproducibility
01
Define exact tags
and digests to install
and load.
Ease of use
02
Install, load, and use.
I don’t want to know
it’s a container!
Reproducibility
01
Define exact tags
and digests to install
and load.
Ease of use
02
Install, load, and use.
I don’t want to know
it’s a container!
Abstraction
03
I largely don’t want to
be aware that I’m
interacting with
containers.
Reproducibility
01
Define exact tags
and digests to install
and load.
Ease of use
02
Install, load, and use.
I don’t want to know
it’s a container!
Abstraction
03
I largely don’t want to
be aware that I’m
interacting with
containers.
Automation
04
Find new container
tags and digests for
me. Let me check for
updates and decide.
How does it work?
$ git clone https://github.com/singularityhub/singularity-hpc
$ cd singularity-hpc
$ pip install -e .
$ which shpc
$ git clone https://github.com/singularityhub/singularity-hpc
$ cd singularity-hpc
$ pip install -e .
$ which shpc
https://singularityhub.github.io/singularity-hpc/
https://singularityhub.github.io/singularity-hpc/
biocontainers
bids
docker
spack
jupyter
nvidia
...
Singularity Registry HPC
singularity exec --home ${HOME} --bind ${HOME}/.local:/home/joyvan/.local
<container> jupyter notebook --no-browser --port=$(shuf -i 20000-65000) --ip 0.0.0.0
run-notebook
I want to run a notebook!
What does the filesystem registry
look like?
Singularity Registry HPC
Singularity Registry HPC
Singularity Registry HPC
An shpc registry entry
An shpc registry entry
An shpc registry entry
An shpc registry entry
An shpc registry entry
An shpc registry entry
Automated Updates with “Binoc”
@alecbcs
Automated Updates with “Binoc”
Automated Testing of Containers
A formula for collaboration?
Identify an actual need that crosses communities
Have an idea that fits into a current way of thinking
Create a contributor friendly learning environment
Singularity Registry HPC
A formula for collaboration?
Identify an actual need that crosses communities
Have an idea that fits into a current way of thinking
Create a contributor friendly learning environment
What do I do now?
Singularity Registry HPC
Research Software Engineers
National Lab Lenny
“We have the biggest
supercomputers!”
Academic Alicia
“We have a research
computing team too.”
Industry Ignacio
“We use cloud services
and anywhere we can
find GPUs!”
National Lab Lenny
“We have the biggest
supercomputers!”
Academic Alicia
“We have a research
computing team too.”
Industry Ignacio
“We use cloud services
and anywhere we can
find GPUs!”
How do we work together?
🤔
Singularity Registry HPC
Singularity Registry HPC
Singularity Registry HPC
What are the benefits of collaboration?
Why should we work together?
● Collaboration leads to better ideas and approaches.
● Convergence on best practices (behavior and software) for research
software engineering
● Learn and grow from one another.
What are the drawbacks of collaboration?
Why shouldn’t we (or can’t we) work together?
● Costs of effort and time.
● Concerns about intellectual property or resulting profits from work
● It requires a security clearance
● I can’t afford it.
How are we doing?
LLNL Open Source
1. Identify an actual need that crosses communities → Assess repository state and define statement of need
2. Have an idea for a way to fix it.
3. Create documentation, branding, and tutorials
4. Tell people about it, get them interested.
Singularity Registry HPC
Singularity Registry HPC
Singularity Registry HPC
How are we doing?
● 765 repos
● 36 % have more external than internal contributors
● 61 % have more internal than external contributors.
● 3 % have equal numbers
● Top ten internal/external contributors has overlap of 1
How are we doing?
1. We have a few projects that are excelling in attracting external contributors
2. The most “popular” projects at the lab are possibly not contributor friendly.
3. What about the long tail?
spack (package manager)
mfem
zfs
https://vsoch.github.io/llnl-contributors/
Can we do better? 🤔
A Plan of Action!
Contributor CI
1. BASELINE: Collect metrics to measure contributions
2. CFA: Contributor Friendliness Assessment
3. ACTION: Timestamps of proactive action
A set of tools and procedures to assess contributions
A Call to Action!
Why should I care?
- It’s hard to work together across siloed RSE communities.
- Let’s look closely at our institutions and get better at that.
- Identify software developed in your lab or institution.
- Make a assessment of external vs. internal contributions
- Contributor Friendliness Assessment
- Improve the scores.
Why should I care?
- No established best practice for an HPC registry
- Little involvement of RSE/HPC communities with OCI
What do you think?
Thank you!
@vsoch GitHub and Twitter

More Related Content

What's hot

Aligning Continuous Integration Deployment: Automated Validation of OpenStack...
Aligning Continuous Integration Deployment: Automated Validation of OpenStack...Aligning Continuous Integration Deployment: Automated Validation of OpenStack...
Aligning Continuous Integration Deployment: Automated Validation of OpenStack...
Atlassian
 
Cloud expo-east-2015
Cloud expo-east-2015Cloud expo-east-2015
Cloud expo-east-2015
argvader
 
JupyterHub tutorial at JupyterCon
JupyterHub tutorial at JupyterConJupyterHub tutorial at JupyterCon
JupyterHub tutorial at JupyterCon
Carol Willing
 
Porting your favourite cmdline tool to Android
Porting your favourite cmdline tool to AndroidPorting your favourite cmdline tool to Android
Porting your favourite cmdline tool to Android
Vlatko Kosturjak
 
Ripping web accessible .git files
Ripping web accessible .git filesRipping web accessible .git files
Ripping web accessible .git files
Vlatko Kosturjak
 
PyLadies Seattle - Lessons in Interactive Visualizations
PyLadies Seattle - Lessons in Interactive VisualizationsPyLadies Seattle - Lessons in Interactive Visualizations
PyLadies Seattle - Lessons in Interactive Visualizations
Amanda Casari
 
Data science bootcamp day2
Data science bootcamp day2Data science bootcamp day2
Data science bootcamp day2
Chetan Khatri
 
Wonderful world of (distributed) SCM or VCS
Wonderful world of (distributed) SCM or VCSWonderful world of (distributed) SCM or VCS
Wonderful world of (distributed) SCM or VCS
Vlatko Kosturjak
 
Inside GitHub with Chris Wanstrath
Inside GitHub with Chris WanstrathInside GitHub with Chris Wanstrath
Inside GitHub with Chris Wanstrath
SV Ruby on Rails Meetup
 
Open shift
Open shiftOpen shift
Open shift
marcolof
 
DevOps Practice in Nonprofit - Abdurrachman Mappuji
DevOps Practice in Nonprofit - Abdurrachman MappujiDevOps Practice in Nonprofit - Abdurrachman Mappuji
DevOps Practice in Nonprofit - Abdurrachman Mappuji
DevOpsDaysJKT
 
Puppet at GitHub
Puppet at GitHubPuppet at GitHub
Puppet at GitHub
Puppet
 

What's hot (12)

Aligning Continuous Integration Deployment: Automated Validation of OpenStack...
Aligning Continuous Integration Deployment: Automated Validation of OpenStack...Aligning Continuous Integration Deployment: Automated Validation of OpenStack...
Aligning Continuous Integration Deployment: Automated Validation of OpenStack...
 
Cloud expo-east-2015
Cloud expo-east-2015Cloud expo-east-2015
Cloud expo-east-2015
 
JupyterHub tutorial at JupyterCon
JupyterHub tutorial at JupyterConJupyterHub tutorial at JupyterCon
JupyterHub tutorial at JupyterCon
 
Porting your favourite cmdline tool to Android
Porting your favourite cmdline tool to AndroidPorting your favourite cmdline tool to Android
Porting your favourite cmdline tool to Android
 
Ripping web accessible .git files
Ripping web accessible .git filesRipping web accessible .git files
Ripping web accessible .git files
 
PyLadies Seattle - Lessons in Interactive Visualizations
PyLadies Seattle - Lessons in Interactive VisualizationsPyLadies Seattle - Lessons in Interactive Visualizations
PyLadies Seattle - Lessons in Interactive Visualizations
 
Data science bootcamp day2
Data science bootcamp day2Data science bootcamp day2
Data science bootcamp day2
 
Wonderful world of (distributed) SCM or VCS
Wonderful world of (distributed) SCM or VCSWonderful world of (distributed) SCM or VCS
Wonderful world of (distributed) SCM or VCS
 
Inside GitHub with Chris Wanstrath
Inside GitHub with Chris WanstrathInside GitHub with Chris Wanstrath
Inside GitHub with Chris Wanstrath
 
Open shift
Open shiftOpen shift
Open shift
 
DevOps Practice in Nonprofit - Abdurrachman Mappuji
DevOps Practice in Nonprofit - Abdurrachman MappujiDevOps Practice in Nonprofit - Abdurrachman Mappuji
DevOps Practice in Nonprofit - Abdurrachman Mappuji
 
Puppet at GitHub
Puppet at GitHubPuppet at GitHub
Puppet at GitHub
 

Similar to Singularity Registry HPC

Repertoire of contributions hacktoberfest
Repertoire of contributions hacktoberfestRepertoire of contributions hacktoberfest
Repertoire of contributions hacktoberfest
DSCVSSUT
 
Introduction to Crab - Python Framework for Building Recommender Systems
Introduction to Crab - Python Framework for Building Recommender SystemsIntroduction to Crab - Python Framework for Building Recommender Systems
Introduction to Crab - Python Framework for Building Recommender Systems
Marcel Caraciolo
 
DevOps - A Purpose for an Institution.pdf
DevOps - A Purpose for an Institution.pdfDevOps - A Purpose for an Institution.pdf
DevOps - A Purpose for an Institution.pdf
Vishwas N
 
Value streammapping cascadiait2014-mceniry
Value streammapping cascadiait2014-mceniryValue streammapping cascadiait2014-mceniry
Value streammapping cascadiait2014-mceniry
Chris McEniry
 
Containers for Science and High-Performance Computing
Containers for Science and High-Performance ComputingContainers for Science and High-Performance Computing
Containers for Science and High-Performance Computing
Dmitry Spodarets
 
Hacktoberfest 2020 - Open source for beginners
Hacktoberfest 2020 - Open source for beginnersHacktoberfest 2020 - Open source for beginners
Hacktoberfest 2020 - Open source for beginners
DeepikaRana30
 
Building software: the lessons from open source
Building software: the lessons from open sourceBuilding software: the lessons from open source
Building software: the lessons from open source
Arnaud Porterie
 
Software Carpentry and the Hydrological Sciences @ AGU 2013
Software Carpentry and the Hydrological Sciences @ AGU 2013Software Carpentry and the Hydrological Sciences @ AGU 2013
Software Carpentry and the Hydrological Sciences @ AGU 2013
Aron Ahmadia
 
How Intuit is overhauling legacy engineering practices at scale with innersource
How Intuit is overhauling legacy engineering practices at scale with innersourceHow Intuit is overhauling legacy engineering practices at scale with innersource
How Intuit is overhauling legacy engineering practices at scale with innersource
Aliza Carpio
 
Release management with NuGet/Chocolatey/JIRA
Release management with NuGet/Chocolatey/JIRARelease management with NuGet/Chocolatey/JIRA
Release management with NuGet/Chocolatey/JIRA
Yaroslav Serhieiev
 
Understanding Research 2.0 from a Socio-technical Perspective
Understanding Research 2.0 from a Socio-technical PerspectiveUnderstanding Research 2.0 from a Socio-technical Perspective
Understanding Research 2.0 from a Socio-technical Perspective
Yuwei Lin
 
 Towards Reproducible Data Analysis Using Cloud and Container Technologies
 Towards Reproducible Data Analysis Using Cloud and Container Technologies Towards Reproducible Data Analysis Using Cloud and Container Technologies
 Towards Reproducible Data Analysis Using Cloud and Container Technologies
inside-BigData.com
 
How To Contribute Drupalcon
How To Contribute   DrupalconHow To Contribute   Drupalcon
How To Contribute Drupalcon
guestc9344e
 
How to get started with Site Reliability Engineering
How to get started with Site Reliability EngineeringHow to get started with Site Reliability Engineering
How to get started with Site Reliability Engineering
Andrew Kirkpatrick
 
DevOps culture
DevOps cultureDevOps culture
DevOps culture
PappKrisztin2
 
GitOps Core Concepts & Ways of Structuring Your Repos
GitOps Core Concepts & Ways of Structuring Your ReposGitOps Core Concepts & Ways of Structuring Your Repos
GitOps Core Concepts & Ways of Structuring Your Repos
Weaveworks
 
Contributing to an Open Source Project 101
Contributing to an Open Source Project 101Contributing to an Open Source Project 101
Contributing to an Open Source Project 101
POSSCON
 
Let's talk FOSS!
Let's talk FOSS!Let's talk FOSS!
Let's talk FOSS!
AditiSaxena72
 
Cloud Foundry Summit 2015: Diego Update
Cloud Foundry Summit 2015: Diego UpdateCloud Foundry Summit 2015: Diego Update
Cloud Foundry Summit 2015: Diego Update
VMware Tanzu
 
Docker for business: How I stopped worrying about antipatterns
Docker for business: How I stopped worrying about antipatternsDocker for business: How I stopped worrying about antipatterns
Docker for business: How I stopped worrying about antipatterns
Lenses.io
 

Similar to Singularity Registry HPC (20)

Repertoire of contributions hacktoberfest
Repertoire of contributions hacktoberfestRepertoire of contributions hacktoberfest
Repertoire of contributions hacktoberfest
 
Introduction to Crab - Python Framework for Building Recommender Systems
Introduction to Crab - Python Framework for Building Recommender SystemsIntroduction to Crab - Python Framework for Building Recommender Systems
Introduction to Crab - Python Framework for Building Recommender Systems
 
DevOps - A Purpose for an Institution.pdf
DevOps - A Purpose for an Institution.pdfDevOps - A Purpose for an Institution.pdf
DevOps - A Purpose for an Institution.pdf
 
Value streammapping cascadiait2014-mceniry
Value streammapping cascadiait2014-mceniryValue streammapping cascadiait2014-mceniry
Value streammapping cascadiait2014-mceniry
 
Containers for Science and High-Performance Computing
Containers for Science and High-Performance ComputingContainers for Science and High-Performance Computing
Containers for Science and High-Performance Computing
 
Hacktoberfest 2020 - Open source for beginners
Hacktoberfest 2020 - Open source for beginnersHacktoberfest 2020 - Open source for beginners
Hacktoberfest 2020 - Open source for beginners
 
Building software: the lessons from open source
Building software: the lessons from open sourceBuilding software: the lessons from open source
Building software: the lessons from open source
 
Software Carpentry and the Hydrological Sciences @ AGU 2013
Software Carpentry and the Hydrological Sciences @ AGU 2013Software Carpentry and the Hydrological Sciences @ AGU 2013
Software Carpentry and the Hydrological Sciences @ AGU 2013
 
How Intuit is overhauling legacy engineering practices at scale with innersource
How Intuit is overhauling legacy engineering practices at scale with innersourceHow Intuit is overhauling legacy engineering practices at scale with innersource
How Intuit is overhauling legacy engineering practices at scale with innersource
 
Release management with NuGet/Chocolatey/JIRA
Release management with NuGet/Chocolatey/JIRARelease management with NuGet/Chocolatey/JIRA
Release management with NuGet/Chocolatey/JIRA
 
Understanding Research 2.0 from a Socio-technical Perspective
Understanding Research 2.0 from a Socio-technical PerspectiveUnderstanding Research 2.0 from a Socio-technical Perspective
Understanding Research 2.0 from a Socio-technical Perspective
 
 Towards Reproducible Data Analysis Using Cloud and Container Technologies
 Towards Reproducible Data Analysis Using Cloud and Container Technologies Towards Reproducible Data Analysis Using Cloud and Container Technologies
 Towards Reproducible Data Analysis Using Cloud and Container Technologies
 
How To Contribute Drupalcon
How To Contribute   DrupalconHow To Contribute   Drupalcon
How To Contribute Drupalcon
 
How to get started with Site Reliability Engineering
How to get started with Site Reliability EngineeringHow to get started with Site Reliability Engineering
How to get started with Site Reliability Engineering
 
DevOps culture
DevOps cultureDevOps culture
DevOps culture
 
GitOps Core Concepts & Ways of Structuring Your Repos
GitOps Core Concepts & Ways of Structuring Your ReposGitOps Core Concepts & Ways of Structuring Your Repos
GitOps Core Concepts & Ways of Structuring Your Repos
 
Contributing to an Open Source Project 101
Contributing to an Open Source Project 101Contributing to an Open Source Project 101
Contributing to an Open Source Project 101
 
Let's talk FOSS!
Let's talk FOSS!Let's talk FOSS!
Let's talk FOSS!
 
Cloud Foundry Summit 2015: Diego Update
Cloud Foundry Summit 2015: Diego UpdateCloud Foundry Summit 2015: Diego Update
Cloud Foundry Summit 2015: Diego Update
 
Docker for business: How I stopped worrying about antipatterns
Docker for business: How I stopped worrying about antipatternsDocker for business: How I stopped worrying about antipatterns
Docker for business: How I stopped worrying about antipatterns
 

More from Vanessa S

The Stories We Tell Ourselves
The Stories We Tell OurselvesThe Stories We Tell Ourselves
The Stories We Tell Ourselves
Vanessa S
 
Research Software Engineering at Stanford University
Research Software Engineering at Stanford UniversityResearch Software Engineering at Stanford University
Research Software Engineering at Stanford University
Vanessa S
 
Research Software Engineering at Stanford
Research Software Engineering at StanfordResearch Software Engineering at Stanford
Research Software Engineering at Stanford
Vanessa S
 
Adding An Operator to Airflow: A Contributor Overflow Exception
Adding An Operator to Airflow: A Contributor Overflow ExceptionAdding An Operator to Airflow: A Contributor Overflow Exception
Adding An Operator to Airflow: A Contributor Overflow Exception
Vanessa S
 
The Research Software Encyclopedia
The Research Software EncyclopediaThe Research Software Encyclopedia
The Research Software Encyclopedia
Vanessa S
 
The Scientific Filesystem
The Scientific FilesystemThe Scientific Filesystem
The Scientific Filesystem
Vanessa S
 
Singularity Containers for Scientific Compute
Singularity Containers for Scientific ComputeSingularity Containers for Scientific Compute
Singularity Containers for Scientific Compute
Vanessa S
 
Laboratory of NeuroGenetics QA (8/2010)
Laboratory of NeuroGenetics QA (8/2010)Laboratory of NeuroGenetics QA (8/2010)
Laboratory of NeuroGenetics QA (8/2010)
Vanessa S
 
PEARC17: Reproducibility and Containers: The Perfect Sandwich
PEARC17: Reproducibility and Containers: The Perfect SandwichPEARC17: Reproducibility and Containers: The Perfect Sandwich
PEARC17: Reproducibility and Containers: The Perfect Sandwich
Vanessa S
 
Building Tools for Neuroimaging
Building Tools for NeuroimagingBuilding Tools for Neuroimaging
Building Tools for Neuroimaging
Vanessa S
 
Brain Maps like Mine
Brain Maps like MineBrain Maps like Mine
Brain Maps like Mine
Vanessa S
 
Qualifying Exam Presentation
Qualifying Exam PresentationQualifying Exam Presentation
Qualifying Exam Presentation
Vanessa S
 
Pre-Proposal Presentation
Pre-Proposal PresentationPre-Proposal Presentation
Pre-Proposal Presentation
Vanessa S
 
Independent component analysis
Independent component analysisIndependent component analysis
Independent component analysis
Vanessa S
 
Subnetworks in Schizophrenia, fMRI
Subnetworks in Schizophrenia, fMRISubnetworks in Schizophrenia, fMRI
Subnetworks in Schizophrenia, fMRI
Vanessa S
 
Research in Progress April 2014
Research in Progress April 2014Research in Progress April 2014
Research in Progress April 2014
Vanessa S
 
Research in Progress Presentation
Research in Progress PresentationResearch in Progress Presentation
Research in Progress Presentation
Vanessa S
 
Introduction to Machine Learning Lecture
Introduction to Machine Learning LectureIntroduction to Machine Learning Lecture
Introduction to Machine Learning Lecture
Vanessa S
 
Introduction to Neuroimaging Informatics
Introduction to Neuroimaging InformaticsIntroduction to Neuroimaging Informatics
Introduction to Neuroimaging Informatics
Vanessa S
 
ISIS Clustering Functional Connectivity
ISIS Clustering Functional ConnectivityISIS Clustering Functional Connectivity
ISIS Clustering Functional Connectivity
Vanessa S
 

More from Vanessa S (20)

The Stories We Tell Ourselves
The Stories We Tell OurselvesThe Stories We Tell Ourselves
The Stories We Tell Ourselves
 
Research Software Engineering at Stanford University
Research Software Engineering at Stanford UniversityResearch Software Engineering at Stanford University
Research Software Engineering at Stanford University
 
Research Software Engineering at Stanford
Research Software Engineering at StanfordResearch Software Engineering at Stanford
Research Software Engineering at Stanford
 
Adding An Operator to Airflow: A Contributor Overflow Exception
Adding An Operator to Airflow: A Contributor Overflow ExceptionAdding An Operator to Airflow: A Contributor Overflow Exception
Adding An Operator to Airflow: A Contributor Overflow Exception
 
The Research Software Encyclopedia
The Research Software EncyclopediaThe Research Software Encyclopedia
The Research Software Encyclopedia
 
The Scientific Filesystem
The Scientific FilesystemThe Scientific Filesystem
The Scientific Filesystem
 
Singularity Containers for Scientific Compute
Singularity Containers for Scientific ComputeSingularity Containers for Scientific Compute
Singularity Containers for Scientific Compute
 
Laboratory of NeuroGenetics QA (8/2010)
Laboratory of NeuroGenetics QA (8/2010)Laboratory of NeuroGenetics QA (8/2010)
Laboratory of NeuroGenetics QA (8/2010)
 
PEARC17: Reproducibility and Containers: The Perfect Sandwich
PEARC17: Reproducibility and Containers: The Perfect SandwichPEARC17: Reproducibility and Containers: The Perfect Sandwich
PEARC17: Reproducibility and Containers: The Perfect Sandwich
 
Building Tools for Neuroimaging
Building Tools for NeuroimagingBuilding Tools for Neuroimaging
Building Tools for Neuroimaging
 
Brain Maps like Mine
Brain Maps like MineBrain Maps like Mine
Brain Maps like Mine
 
Qualifying Exam Presentation
Qualifying Exam PresentationQualifying Exam Presentation
Qualifying Exam Presentation
 
Pre-Proposal Presentation
Pre-Proposal PresentationPre-Proposal Presentation
Pre-Proposal Presentation
 
Independent component analysis
Independent component analysisIndependent component analysis
Independent component analysis
 
Subnetworks in Schizophrenia, fMRI
Subnetworks in Schizophrenia, fMRISubnetworks in Schizophrenia, fMRI
Subnetworks in Schizophrenia, fMRI
 
Research in Progress April 2014
Research in Progress April 2014Research in Progress April 2014
Research in Progress April 2014
 
Research in Progress Presentation
Research in Progress PresentationResearch in Progress Presentation
Research in Progress Presentation
 
Introduction to Machine Learning Lecture
Introduction to Machine Learning LectureIntroduction to Machine Learning Lecture
Introduction to Machine Learning Lecture
 
Introduction to Neuroimaging Informatics
Introduction to Neuroimaging InformaticsIntroduction to Neuroimaging Informatics
Introduction to Neuroimaging Informatics
 
ISIS Clustering Functional Connectivity
ISIS Clustering Functional ConnectivityISIS Clustering Functional Connectivity
ISIS Clustering Functional Connectivity
 

Recently uploaded

CHAPTER-8 COMPONENTS OF COMPUTER SYSTEM CLASS 9 CBSE
CHAPTER-8 COMPONENTS OF COMPUTER SYSTEM CLASS 9 CBSECHAPTER-8 COMPONENTS OF COMPUTER SYSTEM CLASS 9 CBSE
CHAPTER-8 COMPONENTS OF COMPUTER SYSTEM CLASS 9 CBSE
kumarjarun2010
 
Vertex AI Agent Builder - GDG Alicante - Julio 2024
Vertex AI Agent Builder - GDG Alicante - Julio 2024Vertex AI Agent Builder - GDG Alicante - Julio 2024
Vertex AI Agent Builder - GDG Alicante - Julio 2024
Nicolás Lopéz
 
Vulnerability Management: A Comprehensive Overview
Vulnerability Management: A Comprehensive OverviewVulnerability Management: A Comprehensive Overview
Vulnerability Management: A Comprehensive Overview
Steven Carlson
 
July Patch Tuesday
July Patch TuesdayJuly Patch Tuesday
July Patch Tuesday
Ivanti
 
Premium Girls Call Mumbai 9920725232 Unlimited Short Providing Girls Service ...
Premium Girls Call Mumbai 9920725232 Unlimited Short Providing Girls Service ...Premium Girls Call Mumbai 9920725232 Unlimited Short Providing Girls Service ...
Premium Girls Call Mumbai 9920725232 Unlimited Short Providing Girls Service ...
shanihomely
 
leewayhertz.com-AI agents for healthcare Applications benefits and implementa...
leewayhertz.com-AI agents for healthcare Applications benefits and implementa...leewayhertz.com-AI agents for healthcare Applications benefits and implementa...
leewayhertz.com-AI agents for healthcare Applications benefits and implementa...
alexjohnson7307
 
Active Inference is a veryyyyyyyyyyyyyyyyyyyyyyyy
Active Inference is a veryyyyyyyyyyyyyyyyyyyyyyyyActive Inference is a veryyyyyyyyyyyyyyyyyyyyyyyy
Active Inference is a veryyyyyyyyyyyyyyyyyyyyyyyy
RaminGhanbari2
 
(CISOPlatform Summit & SACON 2024) Digital Personal Data Protection Act.pdf
(CISOPlatform Summit & SACON 2024) Digital Personal Data Protection Act.pdf(CISOPlatform Summit & SACON 2024) Digital Personal Data Protection Act.pdf
(CISOPlatform Summit & SACON 2024) Digital Personal Data Protection Act.pdf
Priyanka Aash
 
Choose our Linux Web Hosting for a seamless and successful online presence
Choose our Linux Web Hosting for a seamless and successful online presenceChoose our Linux Web Hosting for a seamless and successful online presence
Choose our Linux Web Hosting for a seamless and successful online presence
rajancomputerfbd
 
IPLOOK Remote-Sensing Satellite Solution
IPLOOK Remote-Sensing Satellite SolutionIPLOOK Remote-Sensing Satellite Solution
IPLOOK Remote-Sensing Satellite Solution
IPLOOK Networks
 
[Talk] Moving Beyond Spaghetti Infrastructure [AOTB] 2024-07-04.pdf
[Talk] Moving Beyond Spaghetti Infrastructure [AOTB] 2024-07-04.pdf[Talk] Moving Beyond Spaghetti Infrastructure [AOTB] 2024-07-04.pdf
[Talk] Moving Beyond Spaghetti Infrastructure [AOTB] 2024-07-04.pdf
Kief Morris
 
Litestack talk at Brighton 2024 (Unleashing the power of SQLite for Ruby apps)
Litestack talk at Brighton 2024 (Unleashing the power of SQLite for Ruby apps)Litestack talk at Brighton 2024 (Unleashing the power of SQLite for Ruby apps)
Litestack talk at Brighton 2024 (Unleashing the power of SQLite for Ruby apps)
Muhammad Ali
 
Three New Criminal Laws in India 1 July 2024
Three New Criminal Laws in India 1 July 2024Three New Criminal Laws in India 1 July 2024
Three New Criminal Laws in India 1 July 2024
aakash malhotra
 
WhatsApp Spy Online Trackers and Monitoring Apps
WhatsApp Spy Online Trackers and Monitoring AppsWhatsApp Spy Online Trackers and Monitoring Apps
WhatsApp Spy Online Trackers and Monitoring Apps
HackersList
 
How to Build a Profitable IoT Product.pptx
How to Build a Profitable IoT Product.pptxHow to Build a Profitable IoT Product.pptx
How to Build a Profitable IoT Product.pptx
Adam Dunkels
 
Girls Call Churchgate 9910780858 Provide Best And Top Girl Service And No1 in...
Girls Call Churchgate 9910780858 Provide Best And Top Girl Service And No1 in...Girls Call Churchgate 9910780858 Provide Best And Top Girl Service And No1 in...
Girls Call Churchgate 9910780858 Provide Best And Top Girl Service And No1 in...
maigasapphire
 
Using LLM Agents with Llama 3, LangGraph and Milvus
Using LLM Agents with Llama 3, LangGraph and MilvusUsing LLM Agents with Llama 3, LangGraph and Milvus
Using LLM Agents with Llama 3, LangGraph and Milvus
Zilliz
 
Evolution of iPaaS - simplify IT workloads to provide a unified view of data...
Evolution of iPaaS - simplify IT workloads to provide a unified view of  data...Evolution of iPaaS - simplify IT workloads to provide a unified view of  data...
Evolution of iPaaS - simplify IT workloads to provide a unified view of data...
Torry Harris
 
Use Cases & Benefits of RPA in Manufacturing in 2024.pptx
Use Cases & Benefits of RPA in Manufacturing in 2024.pptxUse Cases & Benefits of RPA in Manufacturing in 2024.pptx
Use Cases & Benefits of RPA in Manufacturing in 2024.pptx
SynapseIndia
 
Google I/O Extended Harare Merged Slides
Google I/O Extended Harare Merged SlidesGoogle I/O Extended Harare Merged Slides
Google I/O Extended Harare Merged Slides
Google Developer Group - Harare
 

Recently uploaded (20)

CHAPTER-8 COMPONENTS OF COMPUTER SYSTEM CLASS 9 CBSE
CHAPTER-8 COMPONENTS OF COMPUTER SYSTEM CLASS 9 CBSECHAPTER-8 COMPONENTS OF COMPUTER SYSTEM CLASS 9 CBSE
CHAPTER-8 COMPONENTS OF COMPUTER SYSTEM CLASS 9 CBSE
 
Vertex AI Agent Builder - GDG Alicante - Julio 2024
Vertex AI Agent Builder - GDG Alicante - Julio 2024Vertex AI Agent Builder - GDG Alicante - Julio 2024
Vertex AI Agent Builder - GDG Alicante - Julio 2024
 
Vulnerability Management: A Comprehensive Overview
Vulnerability Management: A Comprehensive OverviewVulnerability Management: A Comprehensive Overview
Vulnerability Management: A Comprehensive Overview
 
July Patch Tuesday
July Patch TuesdayJuly Patch Tuesday
July Patch Tuesday
 
Premium Girls Call Mumbai 9920725232 Unlimited Short Providing Girls Service ...
Premium Girls Call Mumbai 9920725232 Unlimited Short Providing Girls Service ...Premium Girls Call Mumbai 9920725232 Unlimited Short Providing Girls Service ...
Premium Girls Call Mumbai 9920725232 Unlimited Short Providing Girls Service ...
 
leewayhertz.com-AI agents for healthcare Applications benefits and implementa...
leewayhertz.com-AI agents for healthcare Applications benefits and implementa...leewayhertz.com-AI agents for healthcare Applications benefits and implementa...
leewayhertz.com-AI agents for healthcare Applications benefits and implementa...
 
Active Inference is a veryyyyyyyyyyyyyyyyyyyyyyyy
Active Inference is a veryyyyyyyyyyyyyyyyyyyyyyyyActive Inference is a veryyyyyyyyyyyyyyyyyyyyyyyy
Active Inference is a veryyyyyyyyyyyyyyyyyyyyyyyy
 
(CISOPlatform Summit & SACON 2024) Digital Personal Data Protection Act.pdf
(CISOPlatform Summit & SACON 2024) Digital Personal Data Protection Act.pdf(CISOPlatform Summit & SACON 2024) Digital Personal Data Protection Act.pdf
(CISOPlatform Summit & SACON 2024) Digital Personal Data Protection Act.pdf
 
Choose our Linux Web Hosting for a seamless and successful online presence
Choose our Linux Web Hosting for a seamless and successful online presenceChoose our Linux Web Hosting for a seamless and successful online presence
Choose our Linux Web Hosting for a seamless and successful online presence
 
IPLOOK Remote-Sensing Satellite Solution
IPLOOK Remote-Sensing Satellite SolutionIPLOOK Remote-Sensing Satellite Solution
IPLOOK Remote-Sensing Satellite Solution
 
[Talk] Moving Beyond Spaghetti Infrastructure [AOTB] 2024-07-04.pdf
[Talk] Moving Beyond Spaghetti Infrastructure [AOTB] 2024-07-04.pdf[Talk] Moving Beyond Spaghetti Infrastructure [AOTB] 2024-07-04.pdf
[Talk] Moving Beyond Spaghetti Infrastructure [AOTB] 2024-07-04.pdf
 
Litestack talk at Brighton 2024 (Unleashing the power of SQLite for Ruby apps)
Litestack talk at Brighton 2024 (Unleashing the power of SQLite for Ruby apps)Litestack talk at Brighton 2024 (Unleashing the power of SQLite for Ruby apps)
Litestack talk at Brighton 2024 (Unleashing the power of SQLite for Ruby apps)
 
Three New Criminal Laws in India 1 July 2024
Three New Criminal Laws in India 1 July 2024Three New Criminal Laws in India 1 July 2024
Three New Criminal Laws in India 1 July 2024
 
WhatsApp Spy Online Trackers and Monitoring Apps
WhatsApp Spy Online Trackers and Monitoring AppsWhatsApp Spy Online Trackers and Monitoring Apps
WhatsApp Spy Online Trackers and Monitoring Apps
 
How to Build a Profitable IoT Product.pptx
How to Build a Profitable IoT Product.pptxHow to Build a Profitable IoT Product.pptx
How to Build a Profitable IoT Product.pptx
 
Girls Call Churchgate 9910780858 Provide Best And Top Girl Service And No1 in...
Girls Call Churchgate 9910780858 Provide Best And Top Girl Service And No1 in...Girls Call Churchgate 9910780858 Provide Best And Top Girl Service And No1 in...
Girls Call Churchgate 9910780858 Provide Best And Top Girl Service And No1 in...
 
Using LLM Agents with Llama 3, LangGraph and Milvus
Using LLM Agents with Llama 3, LangGraph and MilvusUsing LLM Agents with Llama 3, LangGraph and Milvus
Using LLM Agents with Llama 3, LangGraph and Milvus
 
Evolution of iPaaS - simplify IT workloads to provide a unified view of data...
Evolution of iPaaS - simplify IT workloads to provide a unified view of  data...Evolution of iPaaS - simplify IT workloads to provide a unified view of  data...
Evolution of iPaaS - simplify IT workloads to provide a unified view of data...
 
Use Cases & Benefits of RPA in Manufacturing in 2024.pptx
Use Cases & Benefits of RPA in Manufacturing in 2024.pptxUse Cases & Benefits of RPA in Manufacturing in 2024.pptx
Use Cases & Benefits of RPA in Manufacturing in 2024.pptx
 
Google I/O Extended Harare Merged Slides
Google I/O Extended Harare Merged SlidesGoogle I/O Extended Harare Merged Slides
Google I/O Extended Harare Merged Slides
 

Singularity Registry HPC

  • 1. Singularity Registry HPC Collaborative Container Modules Vanessa Sochat PhD Computer Scientist Lawrence Livermore National Lab
  • 2. A formula for collaboration? Identify an actual need that crosses communities
  • 4. How do we provide containers? The Problem: ● Public registries (Docker Hub, Quay.io) or self deployed (Singularity Registry Server, Docker Registry) ● Each user separately pulling means a lot of redundancy ● Pulling “latest” means we aren’t aware of versioning ● It’s not impossible to be reproducible, but it’s harder, and best practices are hard to follow!
  • 5. How do we provide containers? ● HPC admins can provide a shared collection of containers ● Users could also install a collection, if not provided by an admin ● It’s done so in a way so that using the containers is reproducible. Ideally:
  • 6. A formula for collaboration? Identify an actual need that crosses communities Have an idea that fits into a current way of thinking
  • 10. What is wrong with current solutions? ● Not easy enough to collaborate on ● Not modular enough ● Not enough granularity in defining container versions or interactions
  • 11. Reproducibility 01 Define exact tags and digests to install and load.
  • 12. Reproducibility 01 Define exact tags and digests to install and load. Ease of use 02 Install, load, and use. I don’t want to know it’s a container!
  • 13. Reproducibility 01 Define exact tags and digests to install and load. Ease of use 02 Install, load, and use. I don’t want to know it’s a container! Abstraction 03 I largely don’t want to be aware that I’m interacting with containers.
  • 14. Reproducibility 01 Define exact tags and digests to install and load. Ease of use 02 Install, load, and use. I don’t want to know it’s a container! Abstraction 03 I largely don’t want to be aware that I’m interacting with containers. Automation 04 Find new container tags and digests for me. Let me check for updates and decide.
  • 15. How does it work?
  • 16. $ git clone https://github.com/singularityhub/singularity-hpc $ cd singularity-hpc $ pip install -e . $ which shpc
  • 17. $ git clone https://github.com/singularityhub/singularity-hpc $ cd singularity-hpc $ pip install -e . $ which shpc
  • 21. singularity exec --home ${HOME} --bind ${HOME}/.local:/home/joyvan/.local <container> jupyter notebook --no-browser --port=$(shuf -i 20000-65000) --ip 0.0.0.0 run-notebook I want to run a notebook!
  • 22. What does the filesystem registry look like?
  • 32. Automated Updates with “Binoc” @alecbcs
  • 33. Automated Updates with “Binoc”
  • 34. Automated Testing of Containers
  • 35. A formula for collaboration? Identify an actual need that crosses communities Have an idea that fits into a current way of thinking Create a contributor friendly learning environment
  • 37. A formula for collaboration? Identify an actual need that crosses communities Have an idea that fits into a current way of thinking Create a contributor friendly learning environment
  • 38. What do I do now?
  • 40. Research Software Engineers National Lab Lenny “We have the biggest supercomputers!” Academic Alicia “We have a research computing team too.” Industry Ignacio “We use cloud services and anywhere we can find GPUs!”
  • 41. National Lab Lenny “We have the biggest supercomputers!” Academic Alicia “We have a research computing team too.” Industry Ignacio “We use cloud services and anywhere we can find GPUs!” How do we work together? 🤔
  • 45. What are the benefits of collaboration? Why should we work together? ● Collaboration leads to better ideas and approaches. ● Convergence on best practices (behavior and software) for research software engineering ● Learn and grow from one another.
  • 46. What are the drawbacks of collaboration? Why shouldn’t we (or can’t we) work together? ● Costs of effort and time. ● Concerns about intellectual property or resulting profits from work ● It requires a security clearance ● I can’t afford it.
  • 47. How are we doing?
  • 48. LLNL Open Source 1. Identify an actual need that crosses communities → Assess repository state and define statement of need 2. Have an idea for a way to fix it. 3. Create documentation, branding, and tutorials 4. Tell people about it, get them interested.
  • 52. How are we doing? ● 765 repos ● 36 % have more external than internal contributors ● 61 % have more internal than external contributors. ● 3 % have equal numbers ● Top ten internal/external contributors has overlap of 1
  • 53. How are we doing? 1. We have a few projects that are excelling in attracting external contributors 2. The most “popular” projects at the lab are possibly not contributor friendly. 3. What about the long tail?
  • 55. Can we do better? 🤔
  • 56. A Plan of Action!
  • 57. Contributor CI 1. BASELINE: Collect metrics to measure contributions 2. CFA: Contributor Friendliness Assessment 3. ACTION: Timestamps of proactive action A set of tools and procedures to assess contributions
  • 58. A Call to Action!
  • 59. Why should I care? - It’s hard to work together across siloed RSE communities. - Let’s look closely at our institutions and get better at that. - Identify software developed in your lab or institution. - Make a assessment of external vs. internal contributions - Contributor Friendliness Assessment - Improve the scores.
  • 60. Why should I care? - No established best practice for an HPC registry - Little involvement of RSE/HPC communities with OCI
  • 61. What do you think?