Take a look at Apache Spark from parallel processing perspective

1. Spark in Action
from parallel processing perspective
ณัฐวุฒิ หนูไพโรจน์ (อาจารย์เป้ ง)
natawut.n@chula.ac.th
http://natawutn.wordpress.com
@natawutn

2. “ผมอยากให้มีภาพว่าอายุไม่ใช่อุปสรรคสาหรับ
การเขียนโค้ด ต่างประเทศยังเขียนกันผมหงอก”
ผู้จัดงาน Code Mania#11 ท่านหนึ่งได้กล่าวไว้

3. = Parallel Programming Reborn

4. Very Small Big Data Example

5. Storage Requirements for 90 days = 39,000,000,000 events (6.5TB)

6. #เรามาถึงจุดนี้ได้อย่างไร

7. MapReduce

8. Source: https://www.mssqltips.com/sqlservertip/3222/big-data-basics--part-5--introduction-to-
mapreduce/

9. Hadoop Shortcomings
× Too simplified
× Not interactive
× Pessimistic

10. Along come Apache Spark!
• Opensource Big Data Framework
from UC Berkeley
• In-Memory Analytics
• > 800 contributors
• Largest Cluster = 8,000+ nodes (Tencent)

11. Source: M. Zaharia, “New Directions for Spark in 2015”, Spark Summit East 2015, 18 March 2015.

13. Let’s get to Action!

14. Action#1 – Wifi Radius Log
• Size = 336.6 MB
• Sample Data
2016-01-01T00:00:08.000+07:00 acs1121-cen59-1
CSCOacs_Passed_Authentications 0137798608 4 0 2016-
01-01 00:00:08.198 +07:00 0113947311 5200 NOTICE
Passed-Authentication: Authentication succeeded,
ACSVersion=acs-5.6.0.22-B.225, ConfigVe...
2016-01-01T00:00:13.000+07:00 acs1121-cen59-1
CSCOacs_RADIUS_Accounting 0137798618 2 0 2016-01-01
00:00:13.644 +07:00 0113947536 3000 NOTICE Radius-
Accounting: RADIUS Accounting start request, ACSV...

15. Action#1 – Wifi Radius Log
val loglines = sc.textFile("wifi-radius.log")
val successlogin = loglines.filter(
line => line.contains("Authentication succeeded"))
val logintime = successlogin.map(
line => line.split(" ")(0))
val logintimepair = logintime.map(
ltime => (ltime.substring(11,13), 1))
val loginByHour = logintimepair.reduceByKey(
(x,y) => x + y).sortByKey()
loginByHour.collect().foreach(println)

16. For Spark: RDD is everything!

20. Action#2 – Thai Monitoring Corpus
• Size = 40 x 200 MB
• Sample Data
{"ngram": 8, "totallen": 39, "wordseg": "}อยาก}ไป~นอน}ที่~นี่}<s>}วิว}
สวย}<s>}รอ}ชม}ค่า}<s>", "text": "อยากไปนอนที่นี่ วิวสวยยยย <img
class="img-in-emotion" title="อมยิ้ม02" alt="อมยิ้ม02“
src="http://ptcdn.info/emoticons/smiley/smiley02.png"/><br />nรอชม
ค่าาาา", "timestamp": 1392286781000, "id": 21935913, "faillen": 0, "refid":
"31650001"}
{"ngram": 6, "totallen": 26, "wordseg": "ตาม}มา~เที่ยว}ด้วย~กัน}ค่ะ}คุณ}
วา}<s>}", …

21. Action#2 – Thai Monitoring Corpus
val data = sqlContext.read.json("gs://tmc-data/*")
val d = data.withColumn("date",
from_unixtime(data("timestamp")/1000, "yyyy-MM-dd"))
val failre = "<[Ff][^>]+>[^<]+</[Ff][^>]+>".r
val nonthaire = "[u0021-u007F]+".r
def prepareText(x:Any):String = {
var text = x.asInstanceOf[String]
text = text.replace("n"," ")
text = text.replace("<s>"," ")
text = text.replace("~","")
text = text.replace("}"," ")
text = failre.replaceAllIn(text, " ")
text = nonthaire.replaceAllIn(text, " ")
return text
}
val dTmp = d.select(d("date"), d("wordseg"))
val dataMap = dTmp.map(item => (item(0), prepareText(item(1))))
val baseGramRDD = dataMap.flatMapValues(v => v.split("( )+"))
val baseGramCountRDD = baseGramRDD.countByValue()

22. Source: P.Zecevic and M.Bonaci, Spark in Action, Manning Publications, Summer 2016 (est.)

23. Why I like Spark?
• RDD: Simple but Elegant Design
• Very consistent abstraction
• Extensible with minimal overheads
• One feature added to base RDD,
all other modules get benefits

24. Big Data + Spark = #คู่จิ้นฟินเวอร์