The document describes PUSH, a distributed shell for Unix pipelines that runs across computer clusters. PUSH allows users to compose and run multi-stage data analysis pipelines in a distributed manner similar to traditional Unix pipelines. It uses a fan-out/fan-in model to distribute data across nodes. The authors provide background on Unix pipelines and the PUSH concept. They discuss the prototype implementation and status as well as future work.

1. Eric Van Hensbergen & Noah Evans - IBM Research
11 August 2009
PUSH
a DISC Shell
© 2009 IBM Corporation
Monday, August 10, 2009

2. Noah Paul Evans
IBM Research Austin Intern -> Bell Labs Antwerp RSM
Monday, August 10, 2009

3. PUSH: a DISC Shell
Wisdom
“This is the Unix philosophy. Write programs that do one thing and
do it well. Write programs to work together.” - Doug McIlroy
3 © 2009 IBM Corporation
Monday, August 10, 2009

4. PUSH: a DISC Shell
UNIX Pipelines
cat file | sort -n -r | uniq | more
Source: If applicable, describe source origin
4 © 2009 IBM Corporation
Monday, August 10, 2009

5. PUSH: a DISC Shell
PUSH Concept
ls |< cat | sort -n -r | uniq >| sort -n -r | more
Source: If applicable, describe source origin
5 © 2009 IBM Corporation
Monday, August 10, 2009

6. PUSH: a DISC Shell
PUSH Structure
!"#$$%
,-.# ,-.#
&'(()*+
,-.# !"#$$% ,-.#
&'(()*+
,-.# !"#$$% ,-.#
&'(()*+
!"#$$% !"#$$%
,-.# /0$1-.$#2'3 4#(0$1-.$#2'3 ,-.#
&'(()*+ &'(()*+
,-.# !"#$$% ,-.#
&'(()*+
,-.# !"#$$% ,-.#
&'(()*+
!"#$$%
,-.# &'(()*+ ,-.#
6 © 2009 IBM Corporation
Monday, August 10, 2009

7. PUSH: a DISC Shell
Composable
!"#$%&'
!"#(#
!"#$%&'
!"#$%& !"#(#
!"#(#
 stage1 |< stage2 |< stage3 >| stage4 >| stage5
!"#$%&'
!"#(#
!"#$%&'
!"#(#
!"#$%&'
!"#$%& !"#$%& !"#(# !"#(# !"#(#
!"#$%&'
!"#(#
!"#$%&'
!"#(#
!"#$%&'
!"#$%& !"#(# !"#(#
!"#$%&'
!"#(#
7 © 2009 IBM Corporation
Monday, August 10, 2009

8. PUSH: a DISC Shell
Operators
 Fan Out ( |<[n] )
–[n] specifies maximum degree of fan-out
–default will fan each record out to a new core (up to the
maximum number of cores)
–parsing and distribution strategy determined by a module
specified via environment variable OFS
–default module splits records based on newline
 Fan In ( >| )
8 © 2009 IBM Corporation
Monday, August 10, 2009

9. PUSH: a DISC Shell
Status
 Prototype built using Inferno and MASH shell
 Deployed to local Linux cluster, Amazon EC2, and BlueGene
via Kittyhawk (which runs a cloud on BlueGene hardware)
 Currently building out underlying execution model to support
wide range of cluster environments and provide better
distribution and control
 Future Work - Alternate Distribution Models
–Separate Distribution Model from Record Parsing Module
–Broadcast and other MPI-style Collective Operations (?)
–Adapt to changes in underlying resources and/or failure
–Apply to heterogenous systems (Cell, GPUs, multi-ISA)
9 © 2009 IBM Corporation
Monday, August 10, 2009

10. PUSH: a DISC Shell
Thanks
http://code.google.com/p/push
http://www.research.ibm.com/hare
This work has been supported by the Department of Energy
Of Office of Science Operating and Runtime Systems for
Extreme Scale Scientific Computation project under contract
#DE-FG02-08ER25851.
10 © 2009 IBM Corporation
Monday, August 10, 2009