The document presents an overview of the Open Science Grid (OSG) services and operations shared by Igor Sfiligoi during a workshop at the Great Plains Network Annual Meeting 2019. It highlights the independence of participating sites, the role of OSG in providing standardized services, and the operational challenges such as software installation and troubleshooting. Key technologies discussed include the compute element for job submission, CVMFS for software distribution, and GridFTP for data transfer, with an emphasis on caching solutions like StashCache and XRootD for efficiency.

1. Hosted OSG services
Presented by Igor Sfiligoi, UCSD
Workshop at the Great Plains Network Annual Meeting 2019

2. Outline
• Reminder of the why
• List of services OSG
• Current Kubernetes support status
• Looking forward

3. What are
OSG services
Each site is independent
• Has its own policies
• Has much freedom
regarding the technology it uses
OSG provides the abstraction layer
• So all sites look the same to the users
OSG standardized to a set of services
• Who perform the abstraction magic
• And are known to work well in dHTC env.

4. Who operates
the services?
OSG provided just the software and documentation
• Each site was expected to install the services
• Operate the services,
• And keep them up to date
Know-how turned out to be a big problem
• Most software OSG-specific
• OSG spent a lot of time teaching admins and
providing troubleshooting support
OSG operating those services is actually cheaper
• Problems usually few and far in between
• Easier to actually fix problems
than teach people how to do it
• Economies of scale - Very little difference
between instances

5. OSG services
Site services:
• Compute Element (CE)
• Site Squid in support of CVMFS
• Upload - GridFTP
• Download - XRootD
Nationwide services:
• CDN - StashCache

6. Compute Element (CE)
Provides an interface into the site’s batch system
Implemented as HTCondor with x.509 authentication
• HTCondor JobRouter converts to target batch system (supports HTCondor, pbs, slurm)
Two conceptual variants
• Local submission to the site’s batch system
• Remote submission to the site’s batch system over shh
https://opensciencegrid.org/docs/compute-element/htcondor-ce-overview/

7. Compute Element (CE)
Local submission needs co-location
• Both due to policy/firewall limits and efficiency considerations
• This is the traditional deployment model
• Provides most flexibility
Remote submission can happen from anywhere
• All that is needed is a password-less ssh login into the site
• But has relatively high overhead
• OSG is currently operating a few CEs like these

8. A word about CVMFS
OSG encourages users to distribute their software via CVMFS
• No need to distribute whole packages to sites
• Only the individual files actually accessed get transferred to the execute nodes
CVMFS is a virtual global file system
• Full POSIX Read Only access, all files look like they are locally mounted
• Directory listing available and cheap
Makes heavy use of caching
• Locally on the execute node
• Site-wide using a Squid cache https://cernvm.cern.ch/portal/filesystem

9. Frontier Squid
A properly configured Squid cache will
drastically reduce CVMFS WAN traffic
OSG recommends the Frontier Squid package
• Essentially a helper wrapper around the standard
Squid service
Must be run on-site to benefit
http://frontier.cern.ch

10. Bringing data home - GridFTP
• And that output has to be delivered back home ASAP
Most compute jobs
will produce some output
• Small files better handled by the user’s dHTC system (e.g. HTCondor)
GridFTP is a great tool for transferring
large amounts of data over WAN
• Load balanced, essential when a site supports many users
OSG also provides support for
multi-node GridFTP setups
• POSIX access behind the scenesHas to be co-located with the storage
https://opensciencegrid.org/docs/data/gridftp/
https://opensciencegrid.org/docs/data/load-balanced-gridftp/

11. Bringing data home - GridFTP
• And that output has to be delivered back home ASAP
Most compute jobs
will produce some output
• Small files better handled by the user’s dHTC system (e.g. HTCondor)
GridFTP is a great tool for transferring
large amounts of data over WAN
• Load balanced, essential when a site supports many users
OSG also provides support for
multi-node GridFTP setups
• POSIX access behind the scenesHas to be co-located with the storage
https://opensciencegrid.org/docs/data/gridftp/
https://opensciencegrid.org/docs/data/load-balanced-gridftp/
Note:
Do not confuse it with Globus Connect!
Also, while Globus originally developed GridFTP,
it does not support it anymore.
It is now maintained by the
Grid Community Forum.
https://gridcf.org

12. Input data handling
Compute jobs also need input data
Very often the same input data used by many compute jobs
• Caching can greatly improve efficiency there
• Two technologies cache friendly: HTTP and XRootD
Unique/Rarely used data also exist
• Small files usually handled by the user’s HTC system (e.g. HTCondor)
• Large files can be downloaded by GridFTP
• But HTTP and XRootD work just fine there, too

13. Input data handling
Compute jobs also need input data
Very often the same input data used by many compute jobs
• Caching can greatly improve efficiency there
• Two technologies cache friendly: HTTP and XRootD
Unique/Rarely used data also exist
• Small files usually handled by the user’s HTC system (e.g. HTCondor)
• Large files can be downloaded by GridFTP
• But HTTP and XRootD work just fine there, too
OSG currently
recommends XRootD
for large data
Exploring
effectiveness
of HTTP

14. XRootD as a StashCache Origin
XRootD can be used to effectively export data hosted by a site
• Read Only access
• Can be public or authenticated
Public XRootD data can be exported through CVMFS
• So users can use standard POSIX API to access the data
• Although there is a bit of delay between files being available on disk and being visible in CVMFS
• OSG itself provides a service like this for some of its users
• Authenticated CVMFS access also possible, but gets complicated fast
Also known as StashCache or XCache Origin
https://opensciencegrid.org/docs/data/xrootd/install-standalone/
https://opensciencegrid.org/docs/data/stashcache/install-origin/

15. StashCache
XRootD natively supports caching
• It was actually designed as a caching service
• If a requested file is not available on local disk, it will ask upstream for it
StashCache is the name OSG uses to describe XRootD used as a cache
Two obvious deployment strategies
• Locally at a site, to minimize WAN traffic
• In the network backbone to create a CDN system
The hierarchy can be
as deep as needed
GeoIP used to
pick the closest

16. All of the above could
be operated by OSG

17. Software on execute nodes
• Mentioned before, used for user input
• FUSE based
• Has to be installed by sysadmin
on every execute node
CVMFS
• Allows for running user-provided containers
• Gives no privileged access to users
• But must be install by sysadmin
on every execute node
Singularity

18. Software on execute nodes
• Mentioned before, used for user input
• FUSE based
• Has to be installed by sysadmin
on every execute node
CVMFS
• Allows for running user-provided containers
• Gives no privileged access to users
• But must be install by sysadmin
on every execute node
Singularity
NFS version exists,
but comes with
many drawbacks
No
workarounds
NFS version
could be
operated by
OSG

19. Containerization
status

20. Deployed
services
StashCache services were the first
• Kubernetes obvious choice
for the backbone
• A couple Origins also up and running
Frontier-Squid and Local CE deployed
in support of PRP compute
• Hybrid support, since I am supported by
both PRP and OSG grants

21. Other services
coming
Hosted CE available but not in use
• Waiting for a willing site to support
GridFTP planned soon
• Just a matter of needs and internal priorities
Considering NFS-based CVMFS
• Will likely happen

22. Now back to
hands-on

23. Acknowledgents
This work was partially funded by US
National Science Foundation (NSF)
awards OAC-1826967 and
OAC-1841530.