This document summarizes the NEScala2016 conference and unconference. The conference on March 4th included 11 talks on Scala topics like Scalaz streams and Spark. The following day's unconference had 26 talks across 2-5 tracks and covered additional Scala topics such as macros, logical programming, Akka streams, and type parameters vs members. Resources from the events were also listed.

1. NEScala2016Roundup
by @hunglin at @VideoBlocks
1

2. About
● A media company - Creative Content Everyone Can Afford
● Since 2011, we saved more than $5B for our customers
● We reached 1M clips of inventory + $1M marketplace
sales faster than any company in history
● 15 engineers (total 60+ employees)
2

3. AboutMe
● Use Scala for about 5 years
● Not using Scala day to day for the past 9 months
● Scala developer, not sorcerer.
● Data Handyman @ Videoblocks - We Heart Data
● codingphilosophy.com
3

4. 4

5. ● On 3/4 - 3/5
● Since 2011
● At Philadelphia this year (last year in Boston)
● 3/4: conference - 1 track, 11 talks
● 3/5: unconference - 2-5 tracks, 26 talks
● pretty academia as usual
● $60 - order of magnitude cheaper, and better
NEScala2016
5
You have to be there!

6. Day1:Conference
6

7. 7

8. Scalaz-StreamMasterclass
● https://github.com/runarorama/ircz
● monad is just a fancy way of saying "accumulate the
effects"
● monoid is just a fancy way of saying "collect the
elements"
8

9. 9

10. SparkRDD
● compile-time type safe
● api is based on Scala collection (good and bad)
10
def collect(): Array[T]
def collect[U](f: PartialFunction[T,U])(implicit arg0: ClassTag[U]): RDD[U]
//val rdd = sc.textFile("hdfs:/user/bmc/wikipedia‐pagecounts.gz")
//val parsedRDD = rdd.flatMap { line=>
// line.split("""s+""") match {
// case Array(project,_,numRequests,_)=>Some((project,numRequests))
// case_=>None
// }}
parsedRDD.filter { case (project,numRequest) => project=="en" }.
reduceByKey(_+_).
take(100).
Foreach{ case (project,requests) => println(s"$project:$requests") }

11. SparkDataFrame
● DSL
● Optimized by catalyst
● lost type safety
11
val df = parsedRDD.toDF("project", "numRequests")
df.groupBy($"page").
agg(sum($"numRequests").as("count")).
limit(100).
show(100)
scala>:typedf.collect()
Array[org.apache.spark.sql.Row]

12. SparkDataset
● DSL + RDD api
● Type safety
● Optimized
● Use Tungsten, faster (20x) and smaller (4x)
12
val df = sqlContext.read.json("people.json")
case class Person(name: String, age: Long)
val ds: Dataset[Person] = df.as[Person]
ds.groupBy(_.name).count()
experimental

13. 13

14. MacrosforMortals
● illTyped from Shapeless
● AST is pretty hard to deal with
● quasiquotes: val tree = q"class C"
14

15. 15

16. Logical Programming in Scala
https://github.com/stewSquared/ukanren 16

17. 17

18. MonadicLogging?
● Monad to accumulate effects (log)
● Write log of the same thread together
● Write log with logic
○ if (latency > 10ms) { mlog.toStdOut }
● Maybe too complicated and defeat the purpose of
logging?
○ Async + JVM crash, how to know what's going?
18

19. 19

20. StreamFrameworkComparison
● Apache Flink
● Apache Spark
● Google Cloud Dataflow
● Apache Samza
20

21. 21

22. AsyncTest
● Do we need it?
● It is required by Scala.js
22

23. 23

24. CBT:CommunityBuildTool
● https://github.com/cvogt/cbt
● Side project from Chris Vogt
● Write Scala code to build, instead of the DSL (black
magic) of SBT
24

25. 25

26. Modularity
● We shouldn't need to Mock if we modularize our code
correctly
● https://adriaanm.github.io/files/higher.pdf
26

27. 27

28. DataPipelinewithAkkaStream
● Websocket as emitter
● Kafka as queue
● Akka Stream as process engine
28

29. 29

30. TypeParameterv.s.TypeMember
● Sometimes compiler has issues with type parameter
○ trait Foo[F <: Foo[T]]
30
trait Foo[+T]
trait Foo {
Type T <: Exception
}

31. Day2:Unconference
31

32. 32

33. 33

34. Rapture-GuavaofScala
34
● single point of failure is GREAT in code
● config through import
● Rapture JSON supports 8 JSON backend
● syntax error of html or json are checked by compiler
● use gitter to report bug to Jon

35. Scala.js
● All Scala code are supported, but not all libraries
● union type
● D3, react, autowire
35

36. SI-2712
36
Scala Issue: Implement higher-order unification for type
constructor inference.
object Test {
def meh[M[_], A](x: M[A]): M[A] = x
meh{(x: Int) => x} // should solve ?M = [X] X => X and ?A = Int ...
}

37. 37

38. Resources
38
● Runar's blog: http://blog.higher-order.com/
● Slides of Brendan's talk: https://speakerdeck.com/bwmcadams/nescala-16-scala-
macros-for-mortals-or-how-i-learned-to-stop-worrying-and-mumbling-wtf
● Logic programming: https://github.com/stewSquared/ukanren
● Akka stream: https://github.com/pkinsky/akka-streams-example
● Type member vs type parameter: http://www.artima.com/weblogs/viewpost.jsp?
thread=270195
● Good blog of category theory: http://bartoszmilewski.com/2014/10/28/category-
theory-for-programmers-the-preface/
● Unconference grid: https://goo.gl/ei5ijy
● Rapture (The Grava of Scala) https://github.com/propensive/rapture
● Another javascript library in shapeless: https://github.com/travisbrown/circe
● Typesafe RPC: https://github.com/lihaoyi/autowire
● https://issues.scala-lang.org/browse/SI-2712