This document discusses batch processing versus stream processing. It defines batch processing as processing bounded data sets without interruption, while stream processing handles unbounded asynchronous data. Common use cases like ETL, data ingestion, and connected devices are examined in terms of whether batch or stream processing is more appropriate. The document concludes that both approaches have merits, and combining them provides the most power, with stream processing platforms like Spring XD and Cloud Dataflow offering the best of both worlds.

1. SPRINGONE2GX
WASHINGTON, DC
Unless otherwise indicated, these slides are © 2013-2015 Pivotal Software, Inc. and licensed under a Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/
Who Needs Batch?
By Michael Minella and Gunnar Hillert
@michaelminella, @ghillert

3. Michael Minella
Twitter: @michaelminella
Podcast: http://javaOffHeap.com
or @OffHeap
Website: http://spring.io
Gunnar Hillert
Twitter: @ghillert
Website: http://blog.hillert.com
Conference: http://devnexus.com

5. https://github.com/ghillert/s1-2gx2015-who-needs-batch

6. W E H AV E A N
ANNOUNCEMENT

7. WE HAVE A
DISEASE

8. OBJECTIVIUS SHINIUM
SYNDROMUS

9. Shiny Object Syndrom
SHINY OBJECT
SYNDROME

10. IT MAKES US
INNOVATE

11. ROAD TO
RECOVERY

12. COOL KIDS ARE DOING
STREAM PROCESSING

13. Stream Analytics
Cloud Dataflow

16. LET’S TAKE A
CLOSER LOOK

18. BATCH AND STREAM
DEFINITIONS

19. LOOK AT COMMON
USE CASES

20. CONCLUSIONS

21. DEFINITION:
STREAM PROCESSING

22. A system for processing an unbounded
set of data in an asynchronous manor.

23. DEFINITION:
BATCH PROCESSING

24. DEPENDS ON
WHO YOU ASK

25. “Batch … applications run … as special
cases of stream processing applications”
- Apache Flink Documentation

26. Batch processing is the processing of
a bounded data set without interruption
or interaction

27. KEY
DIFFERENCES

28. 1
BOUNDED VS
UNBOUNDED

29. 2
SYNCHRONOUS VS
ASYNCHRONOUS

30. 3
STATEFUL VS
STATELESS

31. STATE OF
STREAMING

32. Stream Analytics
Cloud Dataflow

33. Stream Analytics
Cloud Dataflow

34. STATE OF
STREAMING

35. JSR-352

36. WHY THE
END OF BATCH?

37. LATENCY

38. WHAT DOES THE
END OF BATCH MEAN?

39. LIMIT
LATENCY

40. “ENTERPRISE GRADE”

41. JSR-352

42. LOOK AT COMMON
USE CASES

43. E.T.L.

44. DATABASE TO HDFS

45. DEMO

46. STREAMING CAN BE USEFUL
IN INGESTION USE CASES

47. TWITTER TO HDFS

48. DATA SCIENCE

49. BATCH TRAINING
PREDICTIVE MODELS

50. CONNECTED
CAR

51. transformerhttp filter hdfs
type-transformer
python
Gemfire
REST
Hadoop
Gemfire

52. STREAMING
APROXIMIATIONS EXIST
≈

53. BREADTH AND ACCURACY
DON’T MATCH BATCH
≠

54. WORKFLOW
ORCHESTRATION

56. forEachDir
start
getDirectoryInfo
join
dirSubProcss dirSubProcss…

57. Condition 1
start
getDirAgeAndNumberOfFiles
Ingest
Condition 2
sendReminderEmail
ArchiveEnd

58. DEMO

59. HOW IS THIS DONE
VIA STREAMING?

60. NON-INTERACTIVE
PROCESSING

61. SAME AS ETL
PROCESSING

62. PORT SCANNER

63. DEMO

64. STREAMING
DOESN’T MAKE SENSE

65. STREAMING
DOES MAKE SENSE

66. WHERE LATENCY
IS A PRIORITY

67. DATA LOSS
IS ACCEPTABLE

68. UNBOUNDED
DATASET

69. BUT DOES NOT
IN OTHER USE CASES

70. HIGHLY
COMPLEX

71. ERROR HANDLING
NOT AS ROBUST

72. DATA GUARANTEES
NOT AS ROBUST

73. WHERE BATCH
IS BETTER

74. FINITE
DATASET

75. ROBUST
ERROR HANDLING

76. RESOURCES
ARE A CONCERN

77. NOT EITHER OR

78. COMBINING THE TWO
PROVIDES THE MOST POWER

79. E.T.L.v2

80. mailhttp mysql2hdfs
Database to HDFS v2

81. DEMO

83. BATCH HAS LASTED
BECAUSE IT MAKSE SENSE

84. SPRING XD/DATA FLOW
BEST OF BOTH WORLDS

85. ENTERPRISE GRADE
STREAM PROCESSING

86. BEST BATCH OPTION
ON THE JVM

88. Unless otherwise indicated, these slides are © 2013-2015 Pivotal Software, Inc. and licensed under a
Creative Commons Attribution-NonCommercial license: http://creativecommons.org/licenses/by-nc/3.0/ 88
Check out Spring Cloud Data Flow on https://spring.io
Apache Spark for Big Data Processing – Salon N-P Up next!
High Performance Stream Processing – Salon N-P 8:30AM Tomorrow
Spring Integration Extensions Ecosystem – Here! 12:45 Tomorrow
Learn More. Stay Connected.
@springcentral Spring.io/video