This document discusses various communication patterns in parallel and distributed systems including one-to-all broadcast, all-to-one reduction, all-to-all broadcast and reduction, all-reduce, prefix-sum, scatter, gather, and circular shift operations. It describes how to perform these operations efficiently using techniques like recursive doubling and message splitting. Improving the performance of common communication operations can be done by splitting large messages into smaller parts and combining different patterns like scatter, all-to-all, and gather.

1. Basic
Communication
Prof. Shashikant V. Athawale
Assistant Professor | Computer Engineering
Department | AISSMS College of Engineering,
Kennedy Road, Pune , MH, India - 411001

2. Contents
❖ Operations- One-to-All Broadcast and All-to-One
Reduction
❖ All-to-All Broadcast and Reduction
❖ All-Reduce and Prefix-Sum Operations
❖ Scatter and Gather
❖ All-to-All Personalized Communication
❖ Circular Shift
❖ Improving the Speed of Some Communication
Operations.

3. Circular Shift
❖ It’s a particular permutation.
❖ Circular q-shift: Node i sends data to node (i+q) mod p
(in a set of p nodes).
❖ Useful in some matrix operations and pattern matching.

4. One to All Broadcast And All to One Reduction
❖ Simplest way is to send p − 1 messages from the source
to the other p − 1 processors – this is not very efficient.
❖ Use recursive doubling: source sends a message to a
selected processor. We now have two independent
problems derined over halves of machines.
❖ Reduction can be performed in an identical fashion by
inverting the process.

5. One to All Broadcast

6. One to All Broadcast
❖ One-to-all broadcast on an eight-node ring.
❖ Node 0 is the source of the broadcast. Each message
transfer step is shown by a numbered, dotted arrow from
the source of the message to its destination.
❖ The number on an arrow indicates the time step during
which the message is transferred.

7. All to One Reduction
❖ Reduction on an eight-node ring with node 0 as the
destination of the reduction.

8. All to All Broadcast

9. All to All Broadcast And All to All REduction

10. All Reduce Operations
❖ Each node starts with a buffer of size m.
❖ The final result is the same combination of all buffers on
every node.
❖ Same as all-to-one reduce + one-to-all broadcast.

11. Prefix Sum Operations

12. Scatter And Gather
❖ Scatter : A node sends a unique message to every other
node – unique per node.
❖ Gather : Dual operation but the target node does not
combine the messages into one.

13. All to All Personalised Communication
❖ Each node sends a distinct message to every other node.

14. Improving The Performance of
Communication Operation
❖ So far messages of size m were not split.
❖ If we split them into p parts:
➢ One-to-all broadcast = scatter + all-to-all broadcast of
messages of size m/p.
➢ All-to-one reduction = all-to-all reduce + gather of
messages of size m/p.
➢ All-reduce = all-to-all reduction + all-to-all broadcast
of messages of size m/p.