A low-level introduction to libfabric/OFI. Presented at the OpenFabrics Alliance Developer Workshop 2015

1. A Taste of OFI
Sean Hefty
#OFADevWorkshop

2. Taste of OFI
March 15 – 18, 2015 #OFADevWorkshop 2
Elegant and intellectually satisfying
with subtle undertones?
Selected analysis of the Open
Fabrics Interfaces architecture
and implementation

3. Development
Requirement
analysis
Iterative design and
implementation
Deployment
March 15 – 18, 2015 #OFADevWorkshop 3
Rough
conceptual
model
Quarterly
release cycle
Collective feedback
from OFIWG
~200 requirements
MPI, PGAS, SHMEM,
DBMS, sockets, …
Input from wide
variety of devices

4. Application Requirements
March 15 – 18, 2015 #OFADevWorkshop 4
Give us a high-level interface!
Give us a low-level interface!
And this was just the MPI developers!
Try talking to the government!

5. Design
• Enable simple, basic usage
• Move functionality under OFI
• Advanced application constructs
• Expose abstract HW capabilities
March 15 – 18, 2015 #OFADevWorkshop 5
Implementation
Agnostic
GURU
EASY
Range of usage models

6. Architecture
March 15 – 18, 2015 #OFADevWorkshop 6
Open Fabrics Interfaces (OFI)
Communication
Services
Connection
Management
Address
Vectors
Completion
Services
Event
Queues
Counters
Data Transfer
Services
Message
Queues
Tag
Matching
RMA
Atomics
Control
Services
Discovery
TriggeredOperations
MPI SHMEM PGAS
OFI Enabled Applications
. . .
fi_info

7. Fabric Information
Endpoint Types
• MSG
– Reliable connected
• DGRAM
– Datagram
• RDM
– Reliable datagram
messages
– Reliable unconnected
Capabilities
• Message queue
– FIFO
• RMA
• Tagged messages
– Sends match with specific
receive buffers
• Atomics
March 15 – 18, 2015 #OFADevWorkshop 7
EASY
Select desired endpoint
type and capabilities

8. March 15 – 18, 2015 #OFADevWorkshop 8
Fabric InformationEASY
RDM Message
Queue
OFI Enabled Applications
. . .
RDM Message
Queue
RDM Message
Queue
DGRAM Message Queue
RDM Message Queue
Common Implementation
App nApp 2App 1

9. Fabric Information
March 15 – 18, 2015 #OFADevWorkshop 9
GURU
• Capabilities
– Application desired features and permissions
– Primary capabilities
• Must be requested by application
– Secondary capabilities
• May be requested by application
• May be offered by provider
• Attributes
– Defines the limits and behavior of selected interfaces
– Negotiated
• Mode
– Provider request on application

10. Threading Options
Fully thread safe
Identify resource usage constraints
needed for lockless access
Example:
thread 1: {endpoint 1, CQ 1}
thread 2: {endpoint 2, CQs 2-3}
March 15 – 18, 2015 #OFADevWorkshop 10
GURU
EASY
Is synchronization
needed?

11. Progress
Automatic – submit and forget
Manual – application thread used to
complete request
Timeouts and retries, ack processing, atomic
operations, data placement, etc.
March 15 – 18, 2015 #OFADevWorkshop 11
GURU
EASY
What thread context does a
request use to complete?

12. Resource Management
Enabled – prevent overrunning local and
remote queues, including completion
queues
Disabled – application responsible for
preventing overruns
March 15 – 18, 2015 #OFADevWorkshop 12
GURU
EASY
Flow control
and queuing
App: behavior that can be relied on.
Provider: requirements and potential optimizations
not restrictions.

13. Ordering
Strict – requests are processed and
completed in order
Relaxed – enable out of order processing
Dynamic routing, optimized completion
processing, parallel data transfers, optimized
retry algorithms, etc.
March 15 – 18, 2015 #OFADevWorkshop 13
GURU
EASY
Sequence of request and
completion processing

14. Mode Bits
None! – provider does all the work!
Application support may improve performance
– Local MR – must register buffers for local
operations (send/receives)
– Context – app provides ‘scratch space’ for
provider to track request (full or partial onload
models)
– Buffer prefix – app provides space for network
headers (usnic, IB GRH)
March 15 – 18, 2015 #OFADevWorkshop 14
GURU
EASY
Provider hints on how it
should best be used

15. Architecture
March 15 – 18, 2015 #OFADevWorkshop 15
Open Fabrics Interfaces (OFI)
Communication
Services
Connection
Management
Address
Vectors
Completion
Services
Event
Queues
Counters
Data Transfer
Services
Message
Queues
Tag
Matching
RMA
Atomics
Control
Services
Discovery
TriggeredOperations
MPI SHMEM PGAS
OFI Enabled Applications
. . .
fi_info

16. Endpoints
March 15 – 18, 2015 #OFADevWorkshop 16
EASY
Sequence of request and
completion processing
transmit
Endpoint
receive completions
Conceptually similar
to a socket or QP
Addressable
communication portal

17. Shared Tx/Rx Contexts
March 15 – 18, 2015 #OFADevWorkshop 17
transmit
Endpoint
receive
GURU
Endpoint Endpoint Endpoint
Enable resource manager to
direct use of HW resources
Number of
endpoints greater
than available
resources
Map to command
queues or HW
limits (caching)

18. Scalable Endpoints
- Multi-threading
- Ordering
- Progress
- Completions
March 15 – 18, 2015 #OFADevWorkshop 18
transmit
GURU
Endpoint
transmittransmittransmit receive receive receivereceive
Multiple Tx/Rx
contexts per endpoint
Incoming requests may
be able to target a
specific receive context

19. OFI Is a Full Meal Deal
• Select a main dish and a side
• OFI tells you today’s specials
• You select the ingredients and
tell OFI how to assemble them
March 15 – 18, 2015 #OFADevWorkshop 19
GURU
EASY
We treat you right!
Hello, my name is:
OFI
flair

20. OFI 1.0
• Framework – libfabric
– Interfaces & data structures definitions
– ‘Spec’ = man pages
• Functional implementation
– Quickly enable hardware and fabrics
• Portions layer over vendor interfaces
– Allow for application development
– Amount and quality of support is provider specific
March 15 – 18, 2015 #OFADevWorkshop 20
Important to distinguish between architecture
and direction versus current implementations

21. OFI 1.0 Providers
• Sockets
– Implement all interfaces
and functionality
– App. development &
debug
• Verbs
– Targets any verbs HW
• Not optimized for a
specific device
– Only common verbs
functionality supported
• PSM
– Targets non-verbs HW
– Expands capabilities
beyond lower software
driver
• USNIC
– Targets non-verbs HW
– Cisco will address
March 15 – 18, 2015 #OFADevWorkshop 21
Input from verbs derivative
and non-verbs providers
also fed into OFI design

22. OFI 1.x
• Address other requirements
– Multicast
– Virtualization
– Features cut from 1.0 release
• Expand and optimize providers
– Native providers
– Additional hardware
March 15 – 18, 2015 #OFADevWorkshop 22

23. #OFADevWorkshop
Thank You