The document discusses using the Scalable Communication Network (SCON) library to create overlay networks among resource manager daemons. SCON can be used to create reliable broadcast networks for event notification, scalable collectives for initial setup, and high-speed fabric networks. The Parallel Environment Message Passing Interface (PMIx) could use a SCON to reliably notify interested parties like other jobs of errors. SCON allows daemons to join existing networks and provides APIs for point-to-point, broadcast, allgather, and barrier operations over the overlay networks.

1. RM SCON
Annapurna Dasari
Intel Inc.

2. RM Overlay Networks
• An overlay network of the resident RM daemons (ie
slurmd/orcmd overlay network). Spans over the
daemons in the allocation, lifetime – single/mulitple
job executions session start - session end.
• An overlay network of the job step daemons
responsible for launch and management of the MPI
processes.
– One participant per node.
– Spans across all step daemons where the job is executing.
– Lifetime: job launch – job termination.
• Embedded overlay networks that use the RMs native
communication.

3. Additional RM Overlay Networks
• The RM may create an additional special purpose
overlay network outside of its own integrated
communication system for:
– reliable broadcast for event notification, job launch
– scalable collectives - for initial wire-up, oob request
for job information by debuggers or other tools
– High speed fabrics – RM daemons can speed up
communications by using an overlay network that can
send messages on a high speed fabric and it also has
the required hooks for taking advantages of the high
speed fabrics capabilities (can provide required QoS).

4. Additional RM Overlay Networks
• RMs can create an adhoc overlay network on
demand for a specific purpose or they can
offload all further communications to the
newly created overlay network.
• The SCON library provides interface to create,
send, recv point to point messages and
broadcast, all gather and barrier capabilities.

5. PMIx Event Notification SCON
• PMIx provides a capability that allows MPI applications
to request the RM to notify error events of relevance
to them. They can also request the RM to propagate
events detected by them to their peers.
• The RM local PMIx server is required to provide reliable
notification of the error events to all interested parties
(process could belong to other jobs).
• The RM local PMIx server could create a SCON among
the requested parties and send the event notification
broadcast on the SCON.

6. PMIx Event Notification SCON
local RM
PMIx Server
local RM
PMIx Server
local RM
PMIx Server
local RM
PMIx Server
local RM
PMIx Server
local RM
PMIx Server
local RM
PMIx Server
local RM
PMIx Server

7. Event Notification SCON Creation
• The SCON participants consists of
– All local RM (PMIx server) daemons for the specific job
plus
– All local RM daemons of the job to which process of the
specified job is connected to. The parent process spawning
the job and any explicit connections requested via PMIx
connect.
• The SCON can be created during PMIx server
initialization or on demand.
• RM PMIx server daemons may join and leave the SCON
according to the PMIx connect/disconnect requests?

8. Creating a RM SCON
• The participant daemons create SCON by
calling the create API.
• Participant list is provided by the daemon.
• Specify info keys for fabric selection, topology
• Request reliable broadcast capability during
create by specifying the info key.
• Topology is a tree topology with rank 0
daemon at root.

9. Creating a RM SCON
• SCON library will in response wire up the
SCON according to the topology and complete
the create operation when all participants
join.
• SCON is operational. Typical operations
include
– broadcast event notification
– Allgather/barrier for fence operations.
– Pt to pt send/recv?

10. Join a RM SCON
• Additional RM daemons may join an existing RM SCON
in response to connect requests.
• The joining daemon must specify the existing SCON
information to the library where it wants to join.
• The library would join the new member using internal
xcast and allgather operations among all members.
• Should we provide new scon_join API or enable join by
specifying the join directive in create info keys?
• Do we need to support a join?
• Do existing participants know outside of the SCON
about the new member? How do RMs handle connect
requests today?

11. Reliable xcast on RM SCON
• The notifying daemon broadcasts the event
notification message to all members by calling
scon_xcast API
• Is it required/nice to have/not required to know
when
– The xcast has completed
– The xcast completion status with detailed error
information.
• What would the RM daemon do if the xcast
cannot be delivered to some of the participant
daemons?

12. Allgather/Barrier on RM SCON
• Perform barrier operation by calling the
scon_barrier API.
• Allgather API to be added.
• Brainstorm:
– Reliable allgather/barrier? – being able to
synchronize among the live participants.
– allgather/barrier timeout – each process releases
itself from the barrier upon timeout?