Overview of Apache Spark Streaming

3. SPARK STREAMING OVERVIEW

4. SPARK STREAMING OVERVIEW

6. SPARK STREAMING OVERVIEW

7. SPARK STREAMING OVERVIEW

8. Spark
SQL
Spark
Streaming
MLlib
(machine
learning)
GraphX
(graph)
SPARK STREAMING OVERVIEW

9. SPARK STREAMING OVERVIEW

10. 
SPARK STREAMING OVERVIEW

13. •
•
SPARK STREAMING OVERVIEW

14. •
•
SPARK STREAMING OVERVIEW

16. SPARK STREAMING OVERVIEW

17. •
•
•
•
•
•
•
SPARK STREAMING OVERVIEW

19. •
•
•
• Kafka provides seamless integration between information of producers and consumers without blocking the producers of the
information, and without letting producers know who the final consumers are.
• Each consumer keeps control of its own offset (read)
• On demand topic creation
SPARK STREAMING OVERVIEW

20. • ETL and ELT, wide catalog of sources and sinks
• Flexible design of topologies and agent deployment strategies.
• Data transformation, thanks to interceptors.
•
•
SPARK STREAMING OVERVIEW

21. readClob
readCSV
readLine
readMultiLine
readAvro
readJson
addCurrentTime
addLocalHost
geoIP
findReplace
Split
generateUUID
decompress
If
extractJsonPaths
detectMimeType
xquery
extractURIComponents
xslt
Grok (regular
expressions)
exec
spooling
logger
SPARK STREAMING OVERVIEW

23. •
•
•
•
•
•
•
•
SPARK STREAMING OVERVIEW

25. SPARK STREAMING OVERVIEW

26. CASSANDRA
Kafka
STRATIO DEEP
STRATIO DEEP
•
•
•
•
•
•
•
SPARK STREAMING OVERVIEW

28. •
•
•
•
•
Shark
(SQL)
Spark
Streaming
Mllib
(machine
learning)
GraphX
(graph)
SPARK STREAMING OVERVIEW

29. RDD, what is that?
SPARK STREAMING OVERVIEW

30. RDD, what is that?
SPARK STREAMING OVERVIEW

31. SPARK STREAMING OVERVIEW

32. •
•
SPARK STREAMING OVERVIEW

33. •
•
•
•
SPARK STREAMING OVERVIEW

34. SPARK STREAMING OVERVIEW

35. SPARK STREAMING OVERVIEW

36. SPARK STREAMING OVERVIEW

37. SPARK STREAMING OVERVIEW

38. ?
SPARK STREAMING OVERVIEW

39. Spark Streaming: Overall view
SPARK STREAMING OVERVIEW

40. SPARK STREAMING OVERVIEW
Spark Streaming: Overall view

41. Discretized Stream or DStream.
SPARK STREAMING OVERVIEW

42. Discretized Stream or DStream.
SPARK STREAMING OVERVIEW

43. Discretized Stream or DStream.
SPARK STREAMING OVERVIEW

44. Overall view
SPARK STREAMING OVERVIEW

45. Input DStreams and Receivers.
• Basic (distributed with Spark Streaming).
• Advanced (available as dependency).
SPARK STREAMING OVERVIEW

46. Basic sources
• File Stream.
• Sockets.
• Actors (Akka).
• Queue RDDs (Testing).
SPARK STREAMING OVERVIEW

47. Advanced sources
SPARK STREAMING OVERVIEW

48. Do It Yourself
• Code onStart()
• Code onStop()
• Code receive()
• Custom Receiver ready!
SPARK STREAMING OVERVIEW

49. • map(func), flatMap(func),
filter(func), count()
• repartition(numPartitions)
• union(otherStream)
• reduce(func),countByValue
(), reduceByKey(func,
[numTasks])
• join(otherStream,
[numTasks]), cogroup
(otherStream, [numTasks])
• transform(func)
• updateStateByKey(func)
• window(windowLength,
slideInterval)
• countByWindow(windowLength
, slideInterval)
• reduceByWindow(func,
windowLength, slideInterval)
• reduceByKeyAndWindow(func,
windowLength, slideInterval,
[numTasks])
• countByValueAndWindow(wind
owLength, slideInterval,
[numTasks])
• print()
• foreachRDD(func)
• saveAsObjectFiles(prefix,
[suffix])
• saveAsTextFiles(prefix,
[suffix])
• saveAsHadoopFiles(prefix,
[suffix])
SPARK STREAMING OVERVIEW

50. •
•
•
•
•
•
•
•
•
•
•
SPARK STREAMING OVERVIEW

51. •
•
SPARK STREAMING OVERVIEW

52. SPARK STREAMING OVERVIEW

53. SPARK STREAMING OVERVIEW

54. SPARK STREAMING OVERVIEW

55. SPARK STREAMING OVERVIEW

56. SPARK STREAMING OVERVIEW

57. •
•
SPARK STREAMING OVERVIEW

58. •
•
•
•
•
•
SPARK STREAMING OVERVIEW

59. SPARK STREAMING OVERVIEW

60. SPARK STREAMING OVERVIEW

61. •
•
•
•
•
SPARK STREAMING OVERVIEW

62. SPARK STREAMING OVERVIEW

63. • Stateful transformations (updateStateByKey,
reduceByKeyAndWindow).
• As fault-tolerance mechanism, when driver crashes.
HDFS is mandatory if you are going to use operations that requires checkpointing.
SPARK STREAMING OVERVIEW

64. Configuration parameters
• spark.streaming.receiver.maxRate
• spark.streaming.concurrentJobs
• spark.streaming.receiver.writeAheadLogs.enable
• spark.streaming.unpersist
SPARK STREAMING OVERVIEW

66. each node has mutable state and
for each record they have to update
state & send new records
SPARK STREAMING OVERVIEW

67. •
•
•
SPARK STREAMING OVERVIEW

68. SPARK STREAMING OVERVIEW

69. •
•
SPARK STREAMING OVERVIEW

70. SPARK STREAMING OVERVIEW

71. •
•
•
•
•
•
•
SPARK STREAMING OVERVIEW

72. SPARK STREAMING OVERVIEW

73. SPARK STREAMING OVERVIEW

74. SPARK STREAMING OVERVIEW

75. •
•
•
•
•
SPARK STREAMING OVERVIEW