Lightning talk slides from the May 2015 CloudCamp "unconference" focused on "Big Data and Cloud" "Spark: A Quick Ignition" - Matthew Kemp, Architect of Things at Signal @mattkemp About CloudCamp: the event features short lightning talks, an "unpanel" with audience participation and questions, and small breakout clusters around beers and pizza. Hosted by Cohesive Networks at TechNexus.

1. Spark: A Quick
Ignition
Matthew Kemp

2. Provides distributed processing
Main unit of abstraction is the RDD
Can be used with frameworks like Mesos or Yarn
Supports Java, Python and Scala
https://spark.apache.org/
What is Spark?

3. Can be created from…
Files or HDFS
In memory iterable
Cassandra or SQL tables
Transformations
Lazily create a new RDD from an existing one
Actions
Usually return a value, force computation of
RDD
Resilient Distributed Dataset

4. Some examples:
filter
map
flatMap
distinct
union
intersection
join
reduceByKey
Transformations

5. Some examples:
reduce
collect
take
count
foreach
saveAsTextFile
Actions

6. Sample Text
Spark Example
Spark Shell
Shell Example
Gists

7. Example: Word Count
flatMap()input
reduceBy
Key()
map() outputmap()

8. #!/bin/python
regex = re.compile('[%s]' % re.escape(string.punctuation))
def word_count(sc, in_file_name, out_file_name):
sc.textFile(in_file_name)
.map(lambda line: regex.sub(' ', line).strip().lower())
.flatMap(lambda line: [
(word, 1) for word in line.split()
])
.reduceByKey(lambda a, b: a + b)
.map(lambda (word, count): '%s,%s' % (word, count))
.saveAsTextFile(out_file_name)
Example: Word Count

9. #!/bin/python
regex = re.compile('[%s]' % re.escape(string.punctuation))
def word_count(sc, in_file_name, out_file_name):
sc.textFile(in_file_name)
.map(lambda line: regex.sub(' ', line))
.map(lambda line: line.strip())
.map(lambda line: line.lower())
.flatMap(lambda line: line.split())
.map(lambda word: (word, 1))
.reduceByKey(lambda a, b: a + b)
.map(lambda (word, count): '%s,%s' % (word, count))
.saveAsTextFile(out_file_name)
Example: Alternate Word Count

10. $ pyspark
...
Using Python version 2.7.2 (default)
SparkContext available as sc.
>>> from word_count import word_count
>>> word_count(sc, 'text.txt', 'text_counts')
Running the Example

11. a,23
able,1
about,6
above,1
accept,1
accuse,1
ago,2
alarm,2
all,7
although,1
always,2
an,1
The Results From Spark
and,26
anger,1
another,1
any,2
anyone,1
arches,1
are,1
arm,1
armour,1
as,7
assistant,2
...

12. #!/bin/bash
text=$(cat ${1} | tr "[:punct:]" " " |
tr "[:upper:]" "[:lower:]")
parsed=(${text})
for w in ${parsed[@]}; do echo ${w}; done | sort | uniq -c
A (Bad) Shell Version

13. 23 a
1 able
6 about
1 above
1 accept
1 accuse
2 ago
2 alarm
7 all
1 although
2 always
1 an
The Results From the Shell
26 and
1 anger
1 another
2 any
1 anyone
1 arches
1 are
1 arm
1 armour
7 as
2 assistant
...

14. Our Use Case
distinct()
3rd party
3rd party
distinct()
join()
join()
union() distinct() foreach()
1st party

15. Questions?

16. Contact Info
mkemp@signal.co
@mattkemp
/in/matthewkemp