Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Jumping Bean 
Map Reduce With Bash 
(the power of the Unix philosophy)
Jumping Bean 
About Me 
● Solutions integrator at Jumping Bean 
– Developer & Trainer 
– Technologies 
● Java 
● PHP 
● HT...
Jumping Bean 
Map/Reduce with Bash 
● Purpose of this presentation is: 
– to demonstrate the power and flexibility of the ...
Unix Philosophy 
“is a set of cultural norms and philosophical 
approaches to developing small yet capable 
Jumping Bean 
...
Jumping Bean 
Unix Philosophy 
“Early Unix developers were important in bringing 
the concepts of modularity and reusabili...
Jumping Bean 
Business Problem 
● Nuclear Engineering department needs to run monte-carlo methods 
on data to calculate so...
Jumping Bean 
Business Problem 
● As-Is system 
– Professor uses laptop and desktop, 
– Manually starts application with s...
Jumping Bean 
Business Problem 
● Unknowns 
– How 900 node cluster set up i.e using any cluster software or virtualisation...
Jumping Bean 
What is Map/Reduce? 
● Programming model for 
– Processing and generating large datasets, 
– Using a paralle...
Jumping Bean 
Map/Reduce Steps 
● Map – Master node takes large dataset and 
distributes it to compute nodes to perform 
a...
Jumping Bean 
What we need 
● Controller node functions 
– to distribute data to nodes, 
– execute calculation functions 
...
Jumping Bean 
Controller Functions 
● How to distribute files to a node (map), execute calculations & gather 
results (red...
Management Node Functions 
● Use parallel ssh to distribute scripts from 
management node to compute nodes, 
● Using Ansib...
Jumping Bean 
Compute Node Functions 
● Basically bash scirpt - How to parallelise single threaded 
application to use mul...
Jumping Bean 
Compute/Controller Node 
● At end of compute node process either 
compute node pings controller node, 
● Con...
Jumping Bean 
Security 
● Each student should have separate account 
– Linux mutli-user system. User home directory for st...
Jumping Bean 
Security 
● Limit the root user 
– Linux capabilities 
● setcap, capsh,pscap 
● Disable root account – grant...
Jumping Bean 
Resources 
● Parallel SSH, 
● Xargs, 
● GNU parallel, 
● cgroups, 
● namespaces, 
● Linux capabilities 
● Tw...
Upcoming SlideShare
Loading in …5
×

Map Reduce with Bash - An Example of the Unix Philosophy in Action

4,448 views

Published on

How www.JumpingBean.co.za used simple bash and Linux commands to build a map/reduce solution for a University's physics department that could be maintained by lecturers and students using existing skills.

A stunning example of how the architectural approach of Unix enables users to design simple solutions to their complex processes.

The presentation discusses some of the command line utilities used such as GNU parallels and xargs as well how cgroups, namespaces and Linux capabilities can be combined to create a bash based map/reduce framework.

Published in: Software

Map Reduce with Bash - An Example of the Unix Philosophy in Action

  1. 1. Jumping Bean Map Reduce With Bash (the power of the Unix philosophy)
  2. 2. Jumping Bean About Me ● Solutions integrator at Jumping Bean – Developer & Trainer – Technologies ● Java ● PHP ● HTML5/Javascript ● Linux – What I am planning to do: ● The Internet of things
  3. 3. Jumping Bean Map/Reduce with Bash ● Purpose of this presentation is: – to demonstrate the power and flexibility of the Unix philosophy, – what awesome solutions can be created by using simple bash script and userland tools, – cool utilities and tools ● The purpose is not: – to suggest that Map/Reduce is best done with bash – best given constraint – see business problem
  4. 4. Unix Philosophy “is a set of cultural norms and philosophical approaches to developing small yet capable Jumping Bean software” - Wikipedia
  5. 5. Jumping Bean Unix Philosophy “Early Unix developers were important in bringing the concepts of modularity and reusability into software engineering practice, spawning a 'software tools' movement” - Wikipedia
  6. 6. Jumping Bean Business Problem ● Nuclear Engineering department needs to run monte-carlo methods on data to calculate something to do with core temperature of nuclear reactors :), ● Post-grad students need to run analysis as part of their course work, ● Analysis can take days or weeks to run, ● University has invested in 900 node cluster, ● Cluster used for research when not used by students ● Tool used for analysis is – written in Fortran. – single threaded, ● No money for fancy-pants solution
  7. 7. Jumping Bean Business Problem ● As-Is system – Professor uses laptop and desktop, – Manually starts application with simple script, – Start script x number of times where x=number of cores, – Waits for days, – Manually checks progress, – Not scalable to 900 nodes!
  8. 8. Jumping Bean Business Problem ● Unknowns – How 900 node cluster set up i.e using any cluster software or virtualisation? ● Open Stack? ● Open Nebula? ● KVM? – Tools available to IT department – I.e how they do deploys, monitoring, user management etc ● Requirements – independence from IT department or experts for help, – Student & lecturer IT skills is limited to Fortran & some bash scripting skills, – Due to security concerns prevent IT staff from gaining access to research, ● Keep it simple – Proof of concept
  9. 9. Jumping Bean What is Map/Reduce? ● Programming model for – Processing and generating large datasets, – Using a parallel distribution algorithm, – On a cluster or set of distributed nodes ● Popularised by Google and the advent of cloud computing ● Apache Hadoop – full blown map/reduce framework. Used to analyse your social media data, “understand the customer” and by numerous agencies with 3 letter acronyms. – “Really we only trying to help you know yourself better”
  10. 10. Jumping Bean Map/Reduce Steps ● Map – Master node takes large dataset and distributes it to compute nodes to perform analysis on. The compute nodes return a result, ● Reduce – Gather the results of the compute nodes and aggregate results into final answer
  11. 11. Jumping Bean What we need ● Controller node functions – to distribute data to nodes, – execute calculation functions – collect results ● Management node functions – distribute application and scripts to compute nodes, ● Compute node functions – Scripts to run the single threaded application in parallel on multi core processors ● Security Requirements – Prevent system administrators from gaining access to core application , script or data
  12. 12. Jumping Bean Controller Functions ● How to distribute files to a node (map), execute calculations & gather results (reduce)? – Use split to split input files, – Use ssh to distribute files, execute processes, ● How to do this to multiple (900) nodes? – Use parallel ssh (pssh), paralle scp, ● Issues: – Copying public key to 900 machines? – Give each student their own account? ● Solution – Set up ldap authentication (password based) or – Include controller nodes root public key in compute node image, distribute 2ndary keys via scripts using pssh – Fancy pants – chef, ansible
  13. 13. Management Node Functions ● Use parallel ssh to distribute scripts from management node to compute nodes, ● Using Ansible or Chef could be a next evolutionary step to automate system maintenance Jumping Bean
  14. 14. Jumping Bean Compute Node Functions ● Basically bash scirpt - How to parallelise single threaded application to use multiple cores on modern CPUs? ● xargs – pass through list of input files, – -n set each iteration to run on one input file – -P set number of processes to start in parallel – Script waits for completion of processing & check output ● GNU parallels – Can run commands in parallel using 1 or more hosts – More options for target input placement {}, string replacement – Can pass output as input to another process
  15. 15. Jumping Bean Compute/Controller Node ● At end of compute node process either compute node pings controller node, ● Controller node waits for pssh to return to carry out next step. I.e – reduce process or start next script with output from 1st being input to 2nd step, ● Check for errors and reschedule failed computes,
  16. 16. Jumping Bean Security ● Each student should have separate account – Linux mutli-user system. User home directory for storing files and results ● Each user should be limited in resource usage – Simple ● ulimit ● psacct – Advanced ● Cgroups ● Namespaces ● Students can execute but not read bash script file, special permissions – Use sudo or – Linux capabilities ● setcap – eg setcap "cap_kill=+ep" script.sh
  17. 17. Jumping Bean Security ● Limit the root user – Linux capabilities ● setcap, capsh,pscap ● Disable root account – grant CAP_SYS_ADMIN as needed, ● /etc/security/capabilities.conf
  18. 18. Jumping Bean Resources ● Parallel SSH, ● Xargs, ● GNU parallel, ● cgroups, ● namespaces, ● Linux capabilities ● Twitter - @mxc4 ● Gplus – Mark Clarke ● Jumping Bean ● Cyber Connect ● Jozi Linux User Grou p ● Jozi Java User Group ● Maker Labs

×