How to integrate Spark, Python, PySpark and C/C++ using pipe

1. Big Data, Spark and Python:
C/C++ Integration
Moshe Kaplan
moshe.kaplan@brightaqua.com

2. © All rights reserved: Moshe Kaplan
http://top-performance.blogspot.com
C++ Integration using PIPE
Compile Code:
/usr/bin/gcc -o /tmp/simple /tmp/simple.c
Copy Executable to central location (S3)
Distribute code to all nodes
Use pipe for integration
Clean the old code
2
>>> names = sc.parallelize(["Don", "Betty", "Sally"])
>>> piped = names.pipe("/tmp/simple")
>>> piped.collect()

3. © All rights reserved: Moshe Kaplan
http://top-performance.blogspot.com
General Concept: PySpark Internals
3

4. © All rights reserved: Moshe Kaplan
http://top-performance.blogspot.com
import os
import shutil
num_worker_nodes = 1
def copyFile(filepath):
shutil.copyfile("/dbfs%s" % filepath, filepath)
os.system("chmod u+x %s" % filepath)
sc.parallelize(range(0, 2 * (1 + num_worker_nodes)))
.map(lambda s: copyFile("/tmp/simple"))
.count()
C++: Copy code to nodes
4

5. © All rights reserved: Moshe Kaplan
http://top-performance.blogspot.com
Thank You !
Moshe Kaplan
moshe.kaplan@brightaqua.com