Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Data Processing with Microservices - Michael Minella

123 views

Published on

Data Processing with Microservices by Michael Minella at SpringOne Tour 2019

Published in: Software
  • Be the first to comment

  • Be the first to like this

Data Processing with Microservices - Michael Minella

  1. 1. DATA PROCESSING WITH MICROSERVICES
  2. 2. @michaelmine lla @OffHe ap
  3. 3. github.com/mminella/data- microservices
  4. 4. LAY OF THE LAND
  5. 5. SPRING FRAMEWORK
  6. 6. SPRING INTEGRATION
  7. 7. SPRING BATCH
  8. 8. SPRING CLOUD STREAM
  9. 9. SPRING CLOUD TASK
  10. 10. SPRING CLOUD DATA FLOW
  11. 11. WHAT ARE WE GOING TO TALK ABOUT?
  12. 12. DATA INTEGRATION
  13. 13. EVOLUTI ON
  14. 14. MONOLIT HS
  15. 15. MULTIPLE INTEGRATION FLOWS & BATCH JOBS
  16. 16. LARGE COMPLEX ARTIFAC TS
  17. 17. DIFFIC ULT TO TEST
  18. 18. HARD TO GET TO PRODUCTIO N
  19. 19. BIG DATA PLATFOR MS
  20. 20. GOOD WITH LARGE VOLUMES OF DATA
  21. 21. BAD WITH SMALL VOLUMES OF DATA
  22. 22. LARGE LEARNING CURVE
  23. 23. LITTLE/NO REUSE BETWEEN SOLUTIONS
  24. 24. HEAVY FOOTPRINT IN YOUR CODE
  25. 25. HARD TO TEST
  26. 26. HARD TO GET TO PRODUCTIO N
  27. 27. CATEGORY CAN BE USEFUL JUST NOT FOR EVERYTHING
  28. 28. DATA MICROSERVI CES
  29. 29. CLOUD NATI VE
  30. 30. DEVELOPED AND TESTED IN ISOLATIO N
  31. 31. INDEPENDANT LY SCALAB LE
  32. 32. FAMILIAR DEVELOPMENT MODEL
  33. 33. SMALL AND SIMPLE TO TEST
  34. 34. OPERATIONALLY EASY TO GOVER N
  35. 35. TWO MAIN PROCESSING MODELS
  36. 36. STREA MS
  37. 37. TAS KS
  38. 38. DATA INTEGRATION AS A SERVICE
  39. 39. COMPOSTIO N OF MICROSERVI CES
  40. 40. STREAMS AND FLOWS BECOME YOUR “APP”
  41. 41. OPERATIONAL AND ORCHESTRATION COVERAGE
  42. 42. SPRI NG
  43. 43. Spring Spring Integration Spring Batch Spring Cloud Stream Spring Cloud Task
  44. 44. SPRING CLOUD STREA M
  45. 45. MESSAG E DRIV EN
  46. 46. transformertime log
  47. 47. transformertime log
  48. 48. @EnableBinding(Source.class)
  49. 49. @EnableBinding(Processor.class)
  50. 50. @EnableBinding(Sink.class)
  51. 51. USE CASES
  52. 52. DATA INGESTION 1
  53. 53. DATA TRANSFORMATION 2
  54. 54. REAL-TIME DATA PROCESSING 3
  55. 55. IO T 4
  56. 56. Demo DEM O
  57. 57. SPRING CLOUD TAS K
  58. 58. MICROSERVIC ES THAT END
  59. 59. TASK REPOSITORY 1
  60. 60. BATCH INTEGRATION 2
  61. 61. STREAM INTEGRATION 3
  62. 62. @EnableTask
  63. 63. USE CASES
  64. 64. BATCH JOBS 1
  65. 65. ONE OFF PROCESSES 2
  66. 66. ETL PROCESSES 3
  67. 67. DATA SCIENCE 4
  68. 68. FINITE PROCESSING 5
  69. 69. Demo DEM O
  70. 70. SPRING CLOUD DATA FLOW
  71. 71. WHAT IS SPRING CLOUD DATA FLOW?
  72. 72. DATA FLOW ORCHESTR ATES
  73. 73. TAKES ADVANTAGE OF MODERN PLATFORMS
  74. 74. CLOUDFOUN DRY 1
  75. 75. KUBERNE TES 2
  76. 76. LOC AL 3
  77. 77. http | filter | hdfs // DSL
  78. 78. http | filter | hdfs // DSL
  79. 79. NO INSTALLED CONTAIN
  80. 80. EVERYTHIN G IS SPRING
  81. 81. A BOOT APP FOR THE
  82. 82. A BOOT APP FOR THE
  83. 83. ALL BOOT APPS FOR OTB
  84. 84. Data Flow Server shell app app app browser
  85. 85. Data Flow Server shell twitter log field-value-counter browser
  86. 86. twitter log field-value-counter
  87. 87. Stream Demo DEM O
  88. 88. BUT WAIT, THERE’S MORE!
  89. 89. TASK/STREAM/B ATCH INTEGRAT ION 1
  90. 90. PARTITIONED STREAMS 2
  91. 91. SCALING STREAMS & TASKS 3
  92. 92. Learn More. Stay Connected. Spring Cloud Data Flow Project: cloud.spring.io/spring-cloud-dataflow/ Github: github.com/spring-cloud/spring-cloud-dataflow Spring Cloud Stream Project: cloud.spring.io/spring-cloud-stream/ Github: github.com/spring-cloud/spring-cloud-stream Spring Cloud Task Project: cloud.spring.io/spring-cloud-task/ Github: github.com/spring-cloud/spring-cloud-task Pivotal: pivotal.io Twitter: twitter.com/springcentral YouTube: spring.io/video LinkedIn: spring.io/linkedin Google Plus: spring.io/gplus
  93. 93. QUESTIO NS?
  94. 94. Th End

×