Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
HSA QUEUEING
APU13 - NOVEMBER 2013
IAN BRATT,
PRINCIPAL ENGINEER
ARM
HSA QUEUEING, MOTIVATION
MOTIVATION (TODAY’S PICTURE)
Application
Transfer
buffer to GPU

OS

GPU

Copy/Map
Memory

Queue Job
Schedule Job
Start Jo...
HSA QUEUEING: REQUIREMENTS
REQUIREMENTS
! 

Requires four mechanisms to enable lower overhead job dispatch.
!  Shared Virtual Memory
!  System Cohere...
SHARED VIRTUAL MEMORY
SHARED VIRTUAL MEMORY (TODAY)
! 

Multiple Virtual memory address spaces

PHYSICAL MEMORY
PHYSICAL MEMORY

VIRTUAL MEMORY1...
SHARED VIRTUAL MEMORY (HSA)
! 

Common Virtual Memory for all HSA agents

PHYSICAL MEMORY
PHYSICAL MEMORY

VIRTUAL MEMORY
...
SHARED VIRTUAL MEMORY
! 

! 

Advantages
!  No mapping tricks, no copying back-and-forth between different PA
addresses
! ...
GETTING THERE …
Application
Transfer
buffer to GPU

OS

GPU

Copy/Map
Memory

Queue Job
Schedule Job
Start Job

Finish Job...
SHARED VIRTUAL MEMORY
! 

Specifics
!  Minimum supported VA width is 48b for 64b systems, and 32b for
32b systems.
!  HSA ...
CACHE COHERENCY
CACHE COHERENCY DOMAINS (1/3)

! 

Data accesses to global memory segment from all HSA Agents shall be
coherent without th...
CACHE COHERENCY DOMAINS (2/3)
! 

! 

Advantages
!  Composability
!  Reduced SW complexity when communicating between agen...
GETTING CLOSER …
Application
Transfer
buffer to GPU

OS

GPU

Copy/Map
Memory

Queue Job
Schedule Job
Start Job

Finish Jo...
CACHE COHERENCY DOMAINS (3/3)
! 

Specifics
!  No requirement for instruction memory accesses to be
coherent
!  Only appli...
SIGNALING
SIGNALING (1/3)
! 

HSA agents support the ability to use signaling objects
!  All creation/destruction signaling objects ...
SIGNALING (2/3)
! 

! 

Advantages
!  Enables asynchronous interrupts between HSA agents,
without involving the kernel
!  ...
ALMOST THERE…
Application
Transfer
buffer to GPU

OS

GPU

Copy/Map
Memory

Queue Job
Schedule Job
Start Job

Finish Job
S...
SIGNALING (3/3)
! 

Specifics
!  Only supported within a PASID
!  Supported wait conditions are =, !=, < and >=
!  Wait op...
USER MODE QUEUEING
ONE BLOCK LEFT
Application
Transfer
buffer to GPU

OS

GPU

Copy/Map
Memory

Queue Job
Schedule Job
Start Job

Finish Job
...
USER MODE QUEUEING (1/3)
! 

User mode Queueing
! 

Enables user space applications to directly, without OS intervention, ...
USER MODE QUEUEING (2/3)
! 

Advantages
! 

Avoid involving the kernel/driver when dispatching work for an Agent.

! 

Low...
SUCCESS!
Application
Transfer
buffer to GPU

OS

GPU

Copy/Map
Memory

Queue Job
Schedule Job
Start Job

Finish Job
Schedu...
SUCCESS!
Application

OS

GPU

Queue Job

Start Job

Finish Job

© Copyright 2012 HSA Foundation. All Rights Reserved.

27
ARCHITECTED QUEUEING
LANGUAGE, QUEUES
ARCHITECTED QUEUEING LANGUAGE
! 

HSA Queues look just like standard
shared memory queues, supporting
multi-producer, sing...
ARCHITECTED QUEUEING LANGUAGE
! 

Packets are read and dispatched for execution from the queue in order, but
may complete ...
POTENTIAL MULTI-PRODUCER ALGORITHM
// Read the current queue write offset
tmp_WriteOffset = hsa_WriteOffset();
// wait unt...
POTENTIAL CONSUMER ALGORITHM
// spin while empty (could also perform low-power wait on doorbell)
while (BaseAddress[hsa_Re...
ARCHITECTED QUEUEING
LANGUAGE, PACKETS
DISPATCH PACKET
! 

Packets come in two main types (Dispatch and Barrier), with architected
layouts

! 

Dispatch packet i...
DISPATCH PACKET
Start
Offset
(Bytes)

Format

Field Name

Description

0

uint16_t header

2

uint16_t dimensions

4

uint...
AGENT_DISPATCH PACKET
Start
Offset
(Bytes)

Format

0

uint16_t

header

2

uint16_t

type

4

48

uint32_t
uint64_t
uint6...
BARRIER PACKET
! 

! 

! 

Used for specifying dependences between packets
HAS agent will not launch any further packets f...
BARRIER PACKET
Start Offset
(Bytes)

Format

0

uint16_t

header

Packet header, see 2.8.1 Packet header (p. 16).

2

uint...
DEPENDENCES
! 

! 

A user may never assume more than one packet is being executed by an HSA
agent at a time.
Implications...
HSA QUEUEING, PACKET EXECUTION
PACKET EXECUTION
! 

Launch phase
! 

Initiated when launch conditions are met
! 
! 

! 

All preceeding packets in the qu...
PACKET EXECUTION

Pkt1
Launch

Pkt1
Execute
Pkt2
Launch

Pkt3 stays in Launch
phase until all packets
have completed.

Pkt...
PUTTING IT ALL TOGETHER (FFT)
Packet 1

Packet 3

Packet 5

Packet 2

Packet 4

Packet 6

X[0]
X[1]
X[2]
X[3]
X[4]
X[5]
X[...
PUTTING IT ALL TOGETHER
AQL Pseudo Code
// Send the packets to do the first stage.
aql_dispatch(pkt1);
aql_dispatch(pkt2);...
SUMMARY

© Copyright 2012 HSA Foundation. All Rights Reserved.

45
REQUIREMENTS
! 

HAS Requires four mechanisms to enable lower overhead job
dispatch.
!  Shared Virtual Memory
!  System Co...
QUESTIONS?

© Copyright 2012 HSA Foundation. All Rights Reserved.

47
Upcoming SlideShare
Loading in …5
×

HSA-4122, "HSA Queuing Mode," by Ian Bratt

1,436 views

Published on

Presentation HSA-4122, "HSA Queuing Mode," by Ian Bratt at the AMD Developer Summit (APU13) Nov. 11-13, 2013.

Published in: Technology, Education
  • Be the first to comment

HSA-4122, "HSA Queuing Mode," by Ian Bratt

  1. 1. HSA QUEUEING APU13 - NOVEMBER 2013 IAN BRATT, PRINCIPAL ENGINEER ARM
  2. 2. HSA QUEUEING, MOTIVATION
  3. 3. MOTIVATION (TODAY’S PICTURE) Application Transfer buffer to GPU OS GPU Copy/Map Memory Queue Job Schedule Job Start Job Finish Job Schedule Application Get Buffer Copy/Map Memory © Copyright 2012 HSA Foundation. All Rights Reserved. 3
  4. 4. HSA QUEUEING: REQUIREMENTS
  5. 5. REQUIREMENTS !  Requires four mechanisms to enable lower overhead job dispatch. !  Shared Virtual Memory !  System Coherency !  Signaling !  User mode queueing © Copyright 2012 HSA Foundation. All Rights Reserved. 5
  6. 6. SHARED VIRTUAL MEMORY
  7. 7. SHARED VIRTUAL MEMORY (TODAY) !  Multiple Virtual memory address spaces PHYSICAL MEMORY PHYSICAL MEMORY VIRTUAL MEMORY1 CPU0 VA1->PA1 © Copyright 2012 HSA Foundation. All Rights Reserved. VIRTUAL MEMORY2 GPU VA2->PA1 7
  8. 8. SHARED VIRTUAL MEMORY (HSA) !  Common Virtual Memory for all HSA agents PHYSICAL MEMORY PHYSICAL MEMORY VIRTUAL MEMORY CPU0 VA->PA © Copyright 2012 HSA Foundation. All Rights Reserved. GPU VA->PA 8
  9. 9. SHARED VIRTUAL MEMORY !  !  Advantages !  No mapping tricks, no copying back-and-forth between different PA addresses !  Send pointers (not data) back and forth between HSA agents. Implications !  Common Page Tables (and common interpretation of architectural semantics such as shareability, protection, etc). !  Common mechanisms for address translation (and servicing address translation faults) !  Concept of a process address space (PASID) to allow multiple, per process virtual address spaces within the system. © Copyright 2012 HSA Foundation. All Rights Reserved. 9
  10. 10. GETTING THERE … Application Transfer buffer to GPU OS GPU Copy/Map Memory Queue Job Schedule Job Start Job Finish Job Schedule Application Get Buffer Copy/Map Memory © Copyright 2012 HSA Foundation. All Rights Reserved. 10
  11. 11. SHARED VIRTUAL MEMORY !  Specifics !  Minimum supported VA width is 48b for 64b systems, and 32b for 32b systems. !  HSA agents may reserve VA ranges for internal use via system software. !  All HSA agents other than the host unit must use the lowest privelege level !  If present, read/write access flags for page tables must be maintained by all agents. !  Read/write permissions apply to all HSA agents, equally. © Copyright 2012 HSA Foundation. All Rights Reserved. 11
  12. 12. CACHE COHERENCY
  13. 13. CACHE COHERENCY DOMAINS (1/3) !  Data accesses to global memory segment from all HSA Agents shall be coherent without the need for explicit cache maintenance. © Copyright 2012 HSA Foundation. All Rights Reserved. 13
  14. 14. CACHE COHERENCY DOMAINS (2/3) !  !  Advantages !  Composability !  Reduced SW complexity when communicating between agents !  Lower barrier to entry when porting software Implications !  Hardware coherency support between all HSA agents !  Can take many forms !  Stand alone Snoop Filters / Directories !  Combined L3/Filters !  Snoop-based systems (no filter) !  Etc … © Copyright 2012 HSA Foundation. All Rights Reserved. 14
  15. 15. GETTING CLOSER … Application Transfer buffer to GPU OS GPU Copy/Map Memory Queue Job Schedule Job Start Job Finish Job Schedule Application Get Buffer Copy/Map Memory © Copyright 2012 HSA Foundation. All Rights Reserved. 15
  16. 16. CACHE COHERENCY DOMAINS (3/3) !  Specifics !  No requirement for instruction memory accesses to be coherent !  Only applies to the Primary memory type. !  No requirement for HSA agents to maintain coherency to any memory location where the HSA agents do not specify the same memory attributes !  Read-only image data is required to remain static during the execution of an HSA kernel. !  No double mapping (via different attributes) in order to modify. Must remain static © Copyright 2012 HSA Foundation. All Rights Reserved. 16
  17. 17. SIGNALING
  18. 18. SIGNALING (1/3) !  HSA agents support the ability to use signaling objects !  All creation/destruction signaling objects occurs via HSA runtime APIs !  From an HSA Agent you can directly access signaling objects. !  Signaling a signal object (this will wake up HSA agents waiting upon the object) !  Query current object !  Wait on the current object (various conditions supported). © Copyright 2012 HSA Foundation. All Rights Reserved. 18
  19. 19. SIGNALING (2/3) !  !  Advantages !  Enables asynchronous interrupts between HSA agents, without involving the kernel !  Common idiom for work offload !  Low power waiting Implications !  Runtime support required !  Commonly implemented on top of cache coherency flows © Copyright 2012 HSA Foundation. All Rights Reserved. 19
  20. 20. ALMOST THERE… Application Transfer buffer to GPU OS GPU Copy/Map Memory Queue Job Schedule Job Start Job Finish Job Schedule Application Get Buffer Copy/Map Memory © Copyright 2012 HSA Foundation. All Rights Reserved. 20
  21. 21. SIGNALING (3/3) !  Specifics !  Only supported within a PASID !  Supported wait conditions are =, !=, < and >= !  Wait operations may return sporadically (no guarantee against false positives) !  Programmer must test. !  Wait operations have a maximum duration before returning. !  The HSAIL atomic operations are supported on signal objects. !  Signal objects are opaque © Copyright 2012 HSA Foundation. All Rights Reserved. 21
  22. 22. USER MODE QUEUEING
  23. 23. ONE BLOCK LEFT Application Transfer buffer to GPU OS GPU Copy/Map Memory Queue Job Schedule Job Start Job Finish Job Schedule Application Get Buffer Copy/Map Memory © Copyright 2012 HSA Foundation. All Rights Reserved. 23
  24. 24. USER MODE QUEUEING (1/3) !  User mode Queueing !  Enables user space applications to directly, without OS intervention, enqueue jobs (“Dispatch Packets”) for HSA agents. !  Queues are created/destroyed via calls to the HSA runtime. !  One (or many) agents enqueue packets, a single agent dequeues packets. !  Requires coherency and shared virtual memory. © Copyright 2012 HSA Foundation. All Rights Reserved. 24
  25. 25. USER MODE QUEUEING (2/3) !  Advantages !  Avoid involving the kernel/driver when dispatching work for an Agent. !  Lower latency job dispatch enables finer granularity of offload !  !  Standard memory protection mechanisms may be used to protect communication with the consuming agent. Implications !  Packet formats/fields are Architected – standard across vendors! !  !  !  Guaranteed backward compatibility Packets are enqueued/dequeued via an Architected protocol (all via memory accesses and signalling) More on this later…… © Copyright 2012 HSA Foundation. All Rights Reserved. 25
  26. 26. SUCCESS! Application Transfer buffer to GPU OS GPU Copy/Map Memory Queue Job Schedule Job Start Job Finish Job Schedule Application Get Buffer Copy/Map Memory © Copyright 2012 HSA Foundation. All Rights Reserved. 26
  27. 27. SUCCESS! Application OS GPU Queue Job Start Job Finish Job © Copyright 2012 HSA Foundation. All Rights Reserved. 27
  28. 28. ARCHITECTED QUEUEING LANGUAGE, QUEUES
  29. 29. ARCHITECTED QUEUEING LANGUAGE !  HSA Queues look just like standard shared memory queues, supporting multi-producer, single-consumer !  !  Producer Producer Support is allowed for single-producer, single-consumer Queues consist of storage, read/write indices, ID, etc. Write Index Packets Read Index !  !  !  Queues are created/destroyed via calls to the HSA runtime “Packets” are placed in queues directly from user mode, via an architected protocol Storage in coherent, shared memory Consumer Packet format is architected © Copyright 2012 HSA Foundation. All Rights Reserved. 29
  30. 30. ARCHITECTED QUEUEING LANGUAGE !  Packets are read and dispatched for execution from the queue in order, but may complete in any order. !  !  There is no guarantee that more than one packet will be processed in parallel at a time There may be many queues. A single agent may also consume from several queues. !  A packet processing agent may also enqueue packets. !  Once a packet is enqueued, the producer signals the doorbell !  !  !  Consumers are not required to wait on the doorbell – the consumer could instead be polling. The doorbell is not the synchronization mechanism (the shared memory updates to the read/write indices ensure the synchronization). Modifications to the read/write indices occur via HSA intrinsics, to support portability across different platforms. © Copyright 2012 HSA Foundation. All Rights Reserved. 30
  31. 31. POTENTIAL MULTI-PRODUCER ALGORITHM // Read the current queue write offset tmp_WriteOffset = hsa_WriteOffset(); // wait until the queue is no longer full. while(tmp_WriteOffset == hsa_ReadOffset() + Size) {} // Atomically bump the WriteOffset if (hsa_WriteOffset.compare_exchange(tmp_WriteOffset, tmp_WriteOffset + 1){ // calculate index uint32_t index = tmp_WriteOffset & (Size -1); // copy over the packet, the format field is INVALID BaseAddress[index] = pkt; // Update format field with release semantics BaseAddress[index].hdr.format.store(DISPATCH, std::memory_order_release); // ring doorbell, with release semantics (could also amortize over multiple packets) hsa_ring_doorbell(tmp_WriteOffset+1); } © Copyright 2012 HSA Foundation. All Rights Reserved. 31
  32. 32. POTENTIAL CONSUMER ALGORITHM // spin while empty (could also perform low-power wait on doorbell) while (BaseAddress[hsa_ReadOffset() & (Size - 1)].hdr.format == INVALID) { } // calculate the index uint32_t index = hsa_ReadOffset() & (Size - 1); // copy over the packet pkt = BaseAddress[index]; // set the format field to invalid BaseAddress[index].hdr.format.store(INVALID, std::memory_order_relaxed); // Update the readoffset using HSA intrinsic hsa_ReadOffset.store(hsa_ReadOffset()); © Copyright 2012 HSA Foundation. All Rights Reserved. 32
  33. 33. ARCHITECTED QUEUEING LANGUAGE, PACKETS
  34. 34. DISPATCH PACKET !  Packets come in two main types (Dispatch and Barrier), with architected layouts !  Dispatch packet is the most common type of packet !  Contains !  !  Pointer to the arguments !  WorkGroupSize (x,y,z) !  gridSize(x,y,z) !  And more…… !  !  Pointer to the kernel dispatch_agent packet more general Packets contain an additional “barrier” flag. When the barrier flag is set, no other packets will be launched until all previously launched packets from this queue have completed. © Copyright 2012 HSA Foundation. All Rights Reserved. 34
  35. 35. DISPATCH PACKET Start Offset (Bytes) Format Field Name Description 0 uint16_t header 2 uint16_t dimensions 4 uint16_t workgroupSize.x Packet header Number of dimensions specified in gridSize. Valid values are 1, 2, or 3. x dimension of work-group (measured in work-items). 6 uint16_t workgroupSize.y y dimension of work-group (measured in work-items). 8 12 uint16_t workgroupSize.z uint16_t reserved2 uint32_t gridSize.x z dimension of work-group (measured in work-items). Must be 0. x dimension of grid (measured in work-items). 16 uint32_t gridSize.y y dimension of grid (measured in work-items). 20 uint32_t gridSize.z privateSegmentSizeB uint32_t ytes groupSegmentSizeBy uint32_t tes z dimension of grid (measured in work-items). Total size in bytes of private memory allocation request (per work-item). Total size in bytes of group memory allocation request (per work-group). Address of an object in memory that includes an implementation-defined executable ISA image for the kernel. Address of memory containing kernel arguments. Must be 0. Address of HSA signaling object used to indicate completion of the job. 10 24 28 32 uint64_t kernelObjectAddress 40 48 uint64_t kernargAddress uint64_t reserved3 56 uint64_t completionSignal © Copyright 2012 HSA Foundation. All Rights Reserved. 35
  36. 36. AGENT_DISPATCH PACKET Start Offset (Bytes) Format 0 uint16_t header 2 uint16_t type 4 48 uint32_t uint64_t uint64_t uint64_t uint64_t uint64_t uint64_t reserved2 returnLocation arg[0] arg[1] arg[2] arg[3] reserved3 56 uint64_t 8 16 24 32 40 Field Name Description Packet header The function to be performed by the destination Agent. The type value is split into the following ranges: •  0x0000:0x3FFF – Vendor specific •  0x4000:0x7FFF – HSA runtime •  0x8000:0xFFFF – User registered function Must be 0. Pointer to location to store the function return value in. 64-bit direct or indirect arguments. Must be 0. Address of HSA signaling object used to indicate completionSignal completion of the job. © Copyright 2012 HSA Foundation. All Rights Reserved. 36
  37. 37. BARRIER PACKET !  !  !  Used for specifying dependences between packets HAS agent will not launch any further packets from this queue until the barrier packet signal conditions are met Used for specifying dependences on packets dispatched from any queue. !  !  Execution phase completes only when all of the dependent signals (up to five) have been signaled (with the value of 0). Or if an error has occurred in one of the packets upon which we have a dependence. © Copyright 2012 HSA Foundation. All Rights Reserved. 37
  38. 38. BARRIER PACKET Start Offset (Bytes) Format 0 uint16_t header Packet header, see 2.8.1 Packet header (p. 16). 2 uint16_t reserved2 Must be 0. 4 uint32_t reserved3 Must be 0. 8 uint64_t depSignal0 16 uint64_t depSignal1 24 uint64_t depSignal2 32 uint64_t depSignal3 40 uint64_t depSignal4 48 uint64_t reserved4 Must be 0. 56 uint64_t completionSignal Address of HSA signaling object used to indicate completion of the job. Field Name © Copyright 2012 HSA Foundation. All Rights Reserved. Description Address of dependent signaling objects to be evaluated by the packet processor. 38
  39. 39. DEPENDENCES !  !  A user may never assume more than one packet is being executed by an HSA agent at a time. Implications: !  !  !  Packets can’t poll on shared memory values which will be set by packets issued from other queues, unless the user has ensured the proper ordering. To ensure all previous packets from a queue have been completed, use the Barrier bit. To ensure specific packets from any queue have completed, use the Barrier packet. © Copyright 2012 HSA Foundation. All Rights Reserved. 39
  40. 40. HSA QUEUEING, PACKET EXECUTION
  41. 41. PACKET EXECUTION !  Launch phase !  Initiated when launch conditions are met !  !  !  All preceeding packets in the queue must have exited launch phase If the barrier bit in the packet header is set, then all preceding packets in the queue must have exited completion phase Active phase !  !  !  Execute the packet Barrier packets remain in Active phase until conditions are met. Completion phase !  First step is memory release fence – make results visible. !  CompletionSignal field is then signaled with a decrementing atomic. © Copyright 2012 HSA Foundation. All Rights Reserved. 41
  42. 42. PACKET EXECUTION Pkt1 Launch Pkt1 Execute Pkt2 Launch Pkt3 stays in Launch phase until all packets have completed. Pkt1 Complete Pkt2 Execute Pkt2 Complete Pkt3 Launch (barrier=1) Pkt3 Execute Time © Copyright 2012 HSA Foundation. All Rights Reserved. 42
  43. 43. PUTTING IT ALL TOGETHER (FFT) Packet 1 Packet 3 Packet 5 Packet 2 Packet 4 Packet 6 X[0] X[1] X[2] X[3] X[4] X[5] X[6] X[7] Barrier © Copyright 2012 HSA Foundation. All Rights Reserved. Time Barrier 43
  44. 44. PUTTING IT ALL TOGETHER AQL Pseudo Code // Send the packets to do the first stage. aql_dispatch(pkt1); aql_dispatch(pkt2); // Send the next two packets, setting the barrier bit so we // know packets 1 &2 will be complete before 3 and 4 are // launched. aql_dispatch_with _barrier_bit(pkt3); aql_dispatch(pkt4); // Same as above (make sure 3 & 4 are done before issuing 5 //& 6) aql_dispatch_with_barrier_bit(pkt5); aql_dispatch(pkt6); // This packet will notify us when 5 & 6 are complete) aql_dispatch_with_barrier_bit(finish_pkt); © Copyright 2012 HSA Foundation. All Rights Reserved. 44
  45. 45. SUMMARY © Copyright 2012 HSA Foundation. All Rights Reserved. 45
  46. 46. REQUIREMENTS !  HAS Requires four mechanisms to enable lower overhead job dispatch. !  Shared Virtual Memory !  System Coherency !  Signaling !  User mode queueing © Copyright 2012 HSA Foundation. All Rights Reserved. 46
  47. 47. QUESTIONS? © Copyright 2012 HSA Foundation. All Rights Reserved. 47

×