Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

A Taste of Open Fabrics Interfaces

A low-level introduction to libfabric/OFI. Presented at the OpenFabrics Alliance Developer Workshop 2015

  • Be the first to comment

A Taste of Open Fabrics Interfaces

  1. 1. A Taste of OFI Sean Hefty #OFADevWorkshop
  2. 2. Taste of OFI March 15 – 18, 2015 #OFADevWorkshop 2 Elegant and intellectually satisfying with subtle undertones? Selected analysis of the Open Fabrics Interfaces architecture and implementation
  3. 3. Development Requirement analysis Iterative design and implementation Deployment March 15 – 18, 2015 #OFADevWorkshop 3 Rough conceptual model Quarterly release cycle Collective feedback from OFIWG ~200 requirements MPI, PGAS, SHMEM, DBMS, sockets, … Input from wide variety of devices
  4. 4. Application Requirements March 15 – 18, 2015 #OFADevWorkshop 4 Give us a high-level interface! Give us a low-level interface! And this was just the MPI developers! Try talking to the government!
  5. 5. Design • Enable simple, basic usage • Move functionality under OFI • Advanced application constructs • Expose abstract HW capabilities March 15 – 18, 2015 #OFADevWorkshop 5 Implementation Agnostic GURU EASY Range of usage models
  6. 6. Architecture March 15 – 18, 2015 #OFADevWorkshop 6 Open Fabrics Interfaces (OFI) Communication Services Connection Management Address Vectors Completion Services Event Queues Counters Data Transfer Services Message Queues Tag Matching RMA Atomics Control Services Discovery TriggeredOperations MPI SHMEM PGAS OFI Enabled Applications . . . fi_info
  7. 7. Fabric Information Endpoint Types • MSG – Reliable connected • DGRAM – Datagram • RDM – Reliable datagram messages – Reliable unconnected Capabilities • Message queue – FIFO • RMA • Tagged messages – Sends match with specific receive buffers • Atomics March 15 – 18, 2015 #OFADevWorkshop 7 EASY Select desired endpoint type and capabilities
  8. 8. March 15 – 18, 2015 #OFADevWorkshop 8 Fabric InformationEASY RDM Message Queue OFI Enabled Applications . . . RDM Message Queue RDM Message Queue DGRAM Message Queue RDM Message Queue Common Implementation App nApp 2App 1
  9. 9. Fabric Information March 15 – 18, 2015 #OFADevWorkshop 9 GURU • Capabilities – Application desired features and permissions – Primary capabilities • Must be requested by application – Secondary capabilities • May be requested by application • May be offered by provider • Attributes – Defines the limits and behavior of selected interfaces – Negotiated • Mode – Provider request on application
  10. 10. Threading Options Fully thread safe Identify resource usage constraints needed for lockless access Example: thread 1: {endpoint 1, CQ 1} thread 2: {endpoint 2, CQs 2-3} March 15 – 18, 2015 #OFADevWorkshop 10 GURU EASY Is synchronization needed?
  11. 11. Progress Automatic – submit and forget Manual – application thread used to complete request Timeouts and retries, ack processing, atomic operations, data placement, etc. March 15 – 18, 2015 #OFADevWorkshop 11 GURU EASY What thread context does a request use to complete?
  12. 12. Resource Management Enabled – prevent overrunning local and remote queues, including completion queues Disabled – application responsible for preventing overruns March 15 – 18, 2015 #OFADevWorkshop 12 GURU EASY Flow control and queuing App: behavior that can be relied on. Provider: requirements and potential optimizations not restrictions.
  13. 13. Ordering Strict – requests are processed and completed in order Relaxed – enable out of order processing Dynamic routing, optimized completion processing, parallel data transfers, optimized retry algorithms, etc. March 15 – 18, 2015 #OFADevWorkshop 13 GURU EASY Sequence of request and completion processing
  14. 14. Mode Bits None! – provider does all the work! Application support may improve performance – Local MR – must register buffers for local operations (send/receives) – Context – app provides ‘scratch space’ for provider to track request (full or partial onload models) – Buffer prefix – app provides space for network headers (usnic, IB GRH) March 15 – 18, 2015 #OFADevWorkshop 14 GURU EASY Provider hints on how it should best be used
  15. 15. Architecture March 15 – 18, 2015 #OFADevWorkshop 15 Open Fabrics Interfaces (OFI) Communication Services Connection Management Address Vectors Completion Services Event Queues Counters Data Transfer Services Message Queues Tag Matching RMA Atomics Control Services Discovery TriggeredOperations MPI SHMEM PGAS OFI Enabled Applications . . . fi_info
  16. 16. Endpoints March 15 – 18, 2015 #OFADevWorkshop 16 EASY Sequence of request and completion processing transmit Endpoint receive completions Conceptually similar to a socket or QP Addressable communication portal
  17. 17. Shared Tx/Rx Contexts March 15 – 18, 2015 #OFADevWorkshop 17 transmit Endpoint receive GURU Endpoint Endpoint Endpoint Enable resource manager to direct use of HW resources Number of endpoints greater than available resources Map to command queues or HW limits (caching)
  18. 18. Scalable Endpoints - Multi-threading - Ordering - Progress - Completions March 15 – 18, 2015 #OFADevWorkshop 18 transmit GURU Endpoint transmittransmittransmit receive receive receivereceive Multiple Tx/Rx contexts per endpoint Incoming requests may be able to target a specific receive context
  19. 19. OFI Is a Full Meal Deal • Select a main dish and a side • OFI tells you today’s specials • You select the ingredients and tell OFI how to assemble them March 15 – 18, 2015 #OFADevWorkshop 19 GURU EASY We treat you right! Hello, my name is: OFI flair
  20. 20. OFI 1.0 • Framework – libfabric – Interfaces & data structures definitions – ‘Spec’ = man pages • Functional implementation – Quickly enable hardware and fabrics • Portions layer over vendor interfaces – Allow for application development – Amount and quality of support is provider specific March 15 – 18, 2015 #OFADevWorkshop 20 Important to distinguish between architecture and direction versus current implementations
  21. 21. OFI 1.0 Providers • Sockets – Implement all interfaces and functionality – App. development & debug • Verbs – Targets any verbs HW • Not optimized for a specific device – Only common verbs functionality supported • PSM – Targets non-verbs HW – Expands capabilities beyond lower software driver • USNIC – Targets non-verbs HW – Cisco will address March 15 – 18, 2015 #OFADevWorkshop 21 Input from verbs derivative and non-verbs providers also fed into OFI design
  22. 22. OFI 1.x • Address other requirements – Multicast – Virtualization – Features cut from 1.0 release • Expand and optimize providers – Native providers – Additional hardware March 15 – 18, 2015 #OFADevWorkshop 22
  23. 23. #OFADevWorkshop Thank You

×