Spark in the BigData dark

1. Spark in the BigData dark

2. 6+ years in HighLoad and BigData
3+ years as Team / Tech Lead
Java, Scala, Javascript, PHP
About me

3. Hadoop components

4. Nano History

5. Apache Hadoop
Pros:
• Batch operations
• Scalability
• User defined methods
Cons:
• The problem must be resolved in context of a single
job
• Filesystem based

6. Nano History

7. Tez, Pig, Hive, etc
Pros:
• Batch operations
• Over Hadoop
• Faster then MapReduce
• DAG
Cons:
• Filesystem based

8. Nano History

9. What is in-memory?
• In-memory compute grid
• In-memory data grid

10. In-memory compute grid

11. In-memory data grid

12. HDD vs MEMORY?
• Memory speed is in nanoseconds
• 10GbE Network speed is in microseconds (~50)
• Flash speed is in microseconds (between 20-500+)
• Disk speed is in milliseconds (between 4-7)

13. Spark in-memory model

14. Apache Spark
Pros:
• In memory operations up to 100x times faster then
Hadoop MapReduce
• On disc operations up to 10x times faster then Hadoop
MapReduce
• In-memory
• Batch operations & near real time
• Interactive
• Not bound to hadoop
• Easy to start for developers

15. Really fast?

16. Is Spark popular?
HazelcastApache Spark Apache Hadoop

17. Is it popular?

18. The most active project

19. Who use Spark?

20. Languages

21. Libraries

22. RDD
• Resilient == fault-tolerant
• Distributed == compute in parallel
• Dataset​ == collection

23. How create RDD
• parallelize
• external dataset: filesystem, HDFS, HBase, etc

24. Lazy RDD
• map
• filter
• flatMap
• mapPartitions
• mapPartitionsWithIndex
• union
• intersection
• distinct
• groupByKey
• reduceByKey
• join
• collect
• count
• first
• take(n)
• reduce
• countByKey
• foreach
• takeOrdered
• takeSample
• saveAsTextFile
• saveAsSequenceFile
• saveAsObjectFile
Transformations Actions

25. Example

26. DataFrame
• Distributed collection of data organized into named
columns
• SQL like syntax
• Catalyst Optimizer

27. • Catalyst Optimizer

28. DataFrame vs RDD

29. RDD

30. Cluster Overview

31. Cluster managers
• Standalone
• Apache Mesos
• Hadoop YARN

32. DEMO
• Standalone
• 1.5G dataset
• 2G RAM executor

33. DEMO
• Standalone
• 1.5G dataset
• 2G RAM executor

34. DEMO 2
https://goo.gl/xbnANN
levandovskiy.s@gmail.com

35. Reference list
https://spark.apache.org
https://databricks.com/blog
http://hadoop.apache.org/docs/current
http://www.gridgain.com
https://www.google.com/trends
http://blog.revolutionanalytics.com/2013/12/apache-spark.html
http://0xdata.com/blog/2014/09/Sparkling-Water/
http://hadoop.apache.org/docs/current/hadoop-yarn/hadoop-yarn-site/YARN.html
https://spark.apache.org/docs/1.3.1/job-scheduling.html
https://databricks.com/blog/2015/02/17/introducing-dataframes-in-spark-for-large-scale-data-science.html
https://databricks.com/blog/2015/04/13/deep-dive-into-spark-sqls-catalyst-optimizer.html
http://aryannava.com/2014/02/19/apache-hadoop-ecosystem/
http://www.gridgain.com/in-memory-compute-grid-explained/
http://gridgain.blogspot.com/2012/11/gridgain-and-hadoop-differences-and.html
http://blog.infinio.com/relative-speeds-from-ram-to-flash-to-disk

36. Thank You!