Deadly Code! (seriously) Blocking & Hyper Context Switching Patternchibochibo
ScalaMatsuri 2017
In Scala, it’s relatively simple to write asynchronous processing, but you might overlook the bigger picture happening on the frameworks (such as Play and Slick).In this session, I will illustrate seemingly elegant code that turns out to bring about total destruction.
Is spark streaming based on reactive streams?chibochibo
What is Reactive Streams? What if a subscriber was very slow in asynchronous stream processing? In this talk we will briefly introduce you to the basics of Reactive Streams and the importance of back-pressure and then explain the architecture (implementations) of Spark Streaming back-pressure.
Since 2.10.0 Scala includes macros. I was talk about something like the following:
* What do macros benefit to us?
* What can we do with macros?
* Actual use cases
8. (例)Word Count
val conf = new SparkConf().setAppName("sample batch")
val spark = new SparkContext(conf)
// 処理を定義
val rdd = spark.textFile("s3n://foo/bar/a.log")
.flatMap(_ split " ")
.map(_ -> 1)
.reduceByKey(_ + _)
// 実行
rdd foreach println
スペースで分割
文字とカウントのタプルに変換
文字ごとに集計
まるでコレクション操作のよう!!
9. Sparkの変革
2009年 2013年 2014年 2016年
ASFへ寄贈
UC BerkeleyのAMPLabで
開発スタート
1.0リリース
2.0
予定
▲DataFrame Since 1.3
▼Dataset Since 1.6
10. DataFrame Since 1.3
● built on SQL
● Dynamically typed
val sqlContext = new SQLContext(...)
val df: DataFrame = sqlContext.read.json("people.json")
df.filter(df("age") < 20)
11. Dataset Since 1.6
● Statically typed
val sqlContext = new SQLContext(...)
val ds: Dataset[Person] = sqlContext.read.json("people.json")
.as[Person]
ds.filter(_.age < 20)
case class Person(
name: String, age: Int)