Submit Search
Upload
Data Microservices with Spring Cloud Stream, Task, and Data Flow #jsug #springday
•
20 likes
•
22,436 views
Toshiaki Maki
Follow
Spring Cloud Data Flow #jsug #springday
Read less
Read more
Technology
Report
Share
Report
Share
1 of 117
Download now
Download to read offline
Recommended
Apache Kafka 0.11 の Exactly Once Semantics
Apache Kafka 0.11 の Exactly Once Semantics
Yoshiyasu SAEKI
Spring Cloud Data Flow の紹介 #streamctjp
Spring Cloud Data Flow の紹介 #streamctjp
Yahoo!デベロッパーネットワーク
脱RESTful API設計の提案
脱RESTful API設計の提案
樽八 仲川
決済システムの内製化への旅 - SpringとPCFで作るクラウドネイティブなシステム開発 #jsug #sf_h1
決済システムの内製化への旅 - SpringとPCFで作るクラウドネイティブなシステム開発 #jsug #sf_h1
Toshiaki Maki
DockerとPodmanの比較
DockerとPodmanの比較
Akihiro Suda
マイクロにしすぎた結果がこれだよ!
マイクロにしすぎた結果がこれだよ!
mosa siru
MySQLアーキテクチャ図解講座
MySQLアーキテクチャ図解講座
Mikiya Okuno
Maven基礎
Maven基礎
Toshio Takiguchi
Recommended
Apache Kafka 0.11 の Exactly Once Semantics
Apache Kafka 0.11 の Exactly Once Semantics
Yoshiyasu SAEKI
Spring Cloud Data Flow の紹介 #streamctjp
Spring Cloud Data Flow の紹介 #streamctjp
Yahoo!デベロッパーネットワーク
脱RESTful API設計の提案
脱RESTful API設計の提案
樽八 仲川
決済システムの内製化への旅 - SpringとPCFで作るクラウドネイティブなシステム開発 #jsug #sf_h1
決済システムの内製化への旅 - SpringとPCFで作るクラウドネイティブなシステム開発 #jsug #sf_h1
Toshiaki Maki
DockerとPodmanの比較
DockerとPodmanの比較
Akihiro Suda
マイクロにしすぎた結果がこれだよ!
マイクロにしすぎた結果がこれだよ!
mosa siru
MySQLアーキテクチャ図解講座
MySQLアーキテクチャ図解講座
Mikiya Okuno
Maven基礎
Maven基礎
Toshio Takiguchi
Kubernetes Controller for Pull Request Based Environment
Kubernetes Controller for Pull Request Based Environment
Vishal Banthia
マイクロサービス化デザインパターン - #AWSDevDay Tokyo 2018
マイクロサービス化デザインパターン - #AWSDevDay Tokyo 2018
Yusuke Suzuki
At least onceってぶっちゃけ問題の先送りだったよね #kafkajp
At least onceってぶっちゃけ問題の先送りだったよね #kafkajp
Yahoo!デベロッパーネットワーク
え、まって。その並列分散処理、Kafkaのしくみでもできるの? Apache Kafkaの機能を利用した大規模ストリームデータの並列分散処理
え、まって。その並列分散処理、Kafkaのしくみでもできるの? Apache Kafkaの機能を利用した大規模ストリームデータの並列分散処理
NTT DATA Technology & Innovation
ソーシャルゲーム案件におけるDB分割のPHP実装
ソーシャルゲーム案件におけるDB分割のPHP実装
infinite_loop
Micrometer/Prometheusによる大規模システムモニタリング #jsug #sf_26
Micrometer/Prometheusによる大規模システムモニタリング #jsug #sf_26
Yahoo!デベロッパーネットワーク
モノリスからマイクロサービスへの移行 ~ストラングラーパターンの検証~(Spring Fest 2020講演資料)
モノリスからマイクロサービスへの移行 ~ストラングラーパターンの検証~(Spring Fest 2020講演資料)
NTT DATA Technology & Innovation
Event Driven Microservices with Spring Cloud Stream #jjug_ccc #ccc_ab3
Event Driven Microservices with Spring Cloud Stream #jjug_ccc #ccc_ab3
Toshiaki Maki
Goss入門
Goss入門
ShuyaMotouchi1
GKE に飛んでくるトラフィックを 自由自在に操る力 | 第 10 回 Google Cloud INSIDE Games & Apps Online
GKE に飛んでくるトラフィックを 自由自在に操る力 | 第 10 回 Google Cloud INSIDE Games & Apps Online
Google Cloud Platform - Japan
Application Load Balancer
Application Load Balancer
Amazon Web Services Japan
OpenAPI 3.0でmicroserviceのAPI定義を試みてハマった話
OpenAPI 3.0でmicroserviceのAPI定義を試みてハマった話
Daichi Koike
今からでも遅くないDBマイグレーション - Flyway と SchemaSpy の紹介 -
今からでも遅くないDBマイグレーション - Flyway と SchemaSpy の紹介 -
onozaty
BuildKitによる高速でセキュアなイメージビルド
BuildKitによる高速でセキュアなイメージビルド
Akihiro Suda
Spring Boot の Web アプリケーションを Docker に載せて AWS ECS で動かしている話
Spring Boot の Web アプリケーションを Docker に載せて AWS ECS で動かしている話
JustSystems Corporation
マイクロサービスバックエンドAPIのためのRESTとgRPC
マイクロサービスバックエンドAPIのためのRESTとgRPC
disc99_
イミュータブルデータモデル(入門編)
イミュータブルデータモデル(入門編)
Yoshitaka Kawashima
Spring Boot × Vue.jsでSPAを作る
Spring Boot × Vue.jsでSPAを作る
Go Miyasaka
macOSの仮想化技術について ~Virtualization-rs Rust bindings for virtualization.framework ~
macOSの仮想化技術について ~Virtualization-rs Rust bindings for virtualization.framework ~
NTT Communications Technology Development
Mavenの真実とウソ
Mavenの真実とウソ
Yoshitaka Kawashima
SpringOne 2GX 2014 参加報告 & Spring 4.1について #jsug
SpringOne 2GX 2014 参加報告 & Spring 4.1について #jsug
Toshiaki Maki
Spring Day 2016 - Web API アクセス制御の最適解
Spring Day 2016 - Web API アクセス制御の最適解
都元ダイスケ Miyamoto
More Related Content
What's hot
Kubernetes Controller for Pull Request Based Environment
Kubernetes Controller for Pull Request Based Environment
Vishal Banthia
マイクロサービス化デザインパターン - #AWSDevDay Tokyo 2018
マイクロサービス化デザインパターン - #AWSDevDay Tokyo 2018
Yusuke Suzuki
At least onceってぶっちゃけ問題の先送りだったよね #kafkajp
At least onceってぶっちゃけ問題の先送りだったよね #kafkajp
Yahoo!デベロッパーネットワーク
え、まって。その並列分散処理、Kafkaのしくみでもできるの? Apache Kafkaの機能を利用した大規模ストリームデータの並列分散処理
え、まって。その並列分散処理、Kafkaのしくみでもできるの? Apache Kafkaの機能を利用した大規模ストリームデータの並列分散処理
NTT DATA Technology & Innovation
ソーシャルゲーム案件におけるDB分割のPHP実装
ソーシャルゲーム案件におけるDB分割のPHP実装
infinite_loop
Micrometer/Prometheusによる大規模システムモニタリング #jsug #sf_26
Micrometer/Prometheusによる大規模システムモニタリング #jsug #sf_26
Yahoo!デベロッパーネットワーク
モノリスからマイクロサービスへの移行 ~ストラングラーパターンの検証~(Spring Fest 2020講演資料)
モノリスからマイクロサービスへの移行 ~ストラングラーパターンの検証~(Spring Fest 2020講演資料)
NTT DATA Technology & Innovation
Event Driven Microservices with Spring Cloud Stream #jjug_ccc #ccc_ab3
Event Driven Microservices with Spring Cloud Stream #jjug_ccc #ccc_ab3
Toshiaki Maki
Goss入門
Goss入門
ShuyaMotouchi1
GKE に飛んでくるトラフィックを 自由自在に操る力 | 第 10 回 Google Cloud INSIDE Games & Apps Online
GKE に飛んでくるトラフィックを 自由自在に操る力 | 第 10 回 Google Cloud INSIDE Games & Apps Online
Google Cloud Platform - Japan
Application Load Balancer
Application Load Balancer
Amazon Web Services Japan
OpenAPI 3.0でmicroserviceのAPI定義を試みてハマった話
OpenAPI 3.0でmicroserviceのAPI定義を試みてハマった話
Daichi Koike
今からでも遅くないDBマイグレーション - Flyway と SchemaSpy の紹介 -
今からでも遅くないDBマイグレーション - Flyway と SchemaSpy の紹介 -
onozaty
BuildKitによる高速でセキュアなイメージビルド
BuildKitによる高速でセキュアなイメージビルド
Akihiro Suda
Spring Boot の Web アプリケーションを Docker に載せて AWS ECS で動かしている話
Spring Boot の Web アプリケーションを Docker に載せて AWS ECS で動かしている話
JustSystems Corporation
マイクロサービスバックエンドAPIのためのRESTとgRPC
マイクロサービスバックエンドAPIのためのRESTとgRPC
disc99_
イミュータブルデータモデル(入門編)
イミュータブルデータモデル(入門編)
Yoshitaka Kawashima
Spring Boot × Vue.jsでSPAを作る
Spring Boot × Vue.jsでSPAを作る
Go Miyasaka
macOSの仮想化技術について ~Virtualization-rs Rust bindings for virtualization.framework ~
macOSの仮想化技術について ~Virtualization-rs Rust bindings for virtualization.framework ~
NTT Communications Technology Development
Mavenの真実とウソ
Mavenの真実とウソ
Yoshitaka Kawashima
What's hot
(20)
Kubernetes Controller for Pull Request Based Environment
Kubernetes Controller for Pull Request Based Environment
マイクロサービス化デザインパターン - #AWSDevDay Tokyo 2018
マイクロサービス化デザインパターン - #AWSDevDay Tokyo 2018
At least onceってぶっちゃけ問題の先送りだったよね #kafkajp
At least onceってぶっちゃけ問題の先送りだったよね #kafkajp
え、まって。その並列分散処理、Kafkaのしくみでもできるの? Apache Kafkaの機能を利用した大規模ストリームデータの並列分散処理
え、まって。その並列分散処理、Kafkaのしくみでもできるの? Apache Kafkaの機能を利用した大規模ストリームデータの並列分散処理
ソーシャルゲーム案件におけるDB分割のPHP実装
ソーシャルゲーム案件におけるDB分割のPHP実装
Micrometer/Prometheusによる大規模システムモニタリング #jsug #sf_26
Micrometer/Prometheusによる大規模システムモニタリング #jsug #sf_26
モノリスからマイクロサービスへの移行 ~ストラングラーパターンの検証~(Spring Fest 2020講演資料)
モノリスからマイクロサービスへの移行 ~ストラングラーパターンの検証~(Spring Fest 2020講演資料)
Event Driven Microservices with Spring Cloud Stream #jjug_ccc #ccc_ab3
Event Driven Microservices with Spring Cloud Stream #jjug_ccc #ccc_ab3
Goss入門
Goss入門
GKE に飛んでくるトラフィックを 自由自在に操る力 | 第 10 回 Google Cloud INSIDE Games & Apps Online
GKE に飛んでくるトラフィックを 自由自在に操る力 | 第 10 回 Google Cloud INSIDE Games & Apps Online
Application Load Balancer
Application Load Balancer
OpenAPI 3.0でmicroserviceのAPI定義を試みてハマった話
OpenAPI 3.0でmicroserviceのAPI定義を試みてハマった話
今からでも遅くないDBマイグレーション - Flyway と SchemaSpy の紹介 -
今からでも遅くないDBマイグレーション - Flyway と SchemaSpy の紹介 -
BuildKitによる高速でセキュアなイメージビルド
BuildKitによる高速でセキュアなイメージビルド
Spring Boot の Web アプリケーションを Docker に載せて AWS ECS で動かしている話
Spring Boot の Web アプリケーションを Docker に載せて AWS ECS で動かしている話
マイクロサービスバックエンドAPIのためのRESTとgRPC
マイクロサービスバックエンドAPIのためのRESTとgRPC
イミュータブルデータモデル(入門編)
イミュータブルデータモデル(入門編)
Spring Boot × Vue.jsでSPAを作る
Spring Boot × Vue.jsでSPAを作る
macOSの仮想化技術について ~Virtualization-rs Rust bindings for virtualization.framework ~
macOSの仮想化技術について ~Virtualization-rs Rust bindings for virtualization.framework ~
Mavenの真実とウソ
Mavenの真実とウソ
Viewers also liked
SpringOne 2GX 2014 参加報告 & Spring 4.1について #jsug
SpringOne 2GX 2014 参加報告 & Spring 4.1について #jsug
Toshiaki Maki
Spring Day 2016 - Web API アクセス制御の最適解
Spring Day 2016 - Web API アクセス制御の最適解
都元ダイスケ Miyamoto
Lineにおけるspring frameworkの活用
Lineにおけるspring frameworkの活用
Tokuhiro Matsuno
Spring Day 2016 springの現在過去未来
Spring Day 2016 springの現在過去未来
Yuichi Hasegawa
Spring bootで学ぶ初めてのwebアプリ開発
Spring bootで学ぶ初めてのwebアプリ開発
terahide
アメブロの大規模システム刷新と それを支えるSpring
アメブロの大規模システム刷新と それを支えるSpring
Takuya Hattori
Spring 5に備えるリアクティブプログラミング入門
Spring 5に備えるリアクティブプログラミング入門
Takuya Iwatsuka
Springを使ったwebアプリにリファクタリングしよう
Springを使ったwebアプリにリファクタリングしよう
土岐 孝平
Application Re-Architecture Technology ~ StrutsからSpring MVCへ ~
Application Re-Architecture Technology ~ StrutsからSpring MVCへ ~
Yuichi Hasegawa
Grailsでドメイン駆動設計を実践する時の勘所
Grailsでドメイン駆動設計を実践する時の勘所
Takuma Watabiki
Cloud Foundry x Wagby
Cloud Foundry x Wagby
Yoshinori Nie
楽天トラベルとSpring(Spring Day 2016)
楽天トラベルとSpring(Spring Day 2016)
Rakuten Group, Inc.
ビッグじゃなくても使えるSpark Streaming
ビッグじゃなくても使えるSpark Streaming
chibochibo
Distributed Tracing Velocity2016
Distributed Tracing Velocity2016
Reshmi Krishna
Distributed tracing - get a grasp on your production
Distributed tracing - get a grasp on your production
nklmish
Spring CloudとZipkinを利用した分散トレーシング
Spring CloudとZipkinを利用した分散トレーシング
Rakuten Group, Inc.
形態素解析
形態素解析
Works Applications
WalB: Real-time and Incremental Backup System for Block Devices
WalB: Real-time and Incremental Backup System for Block Devices
uchan_nos
3000社の業務データ絞り込みを支える技術
3000社の業務データ絞り込みを支える技術
Ryo Mitoma
離れた場所でも最高のチームワークを実現する方法 ーサイボウズ開発チームのリモートワーク事例ー
離れた場所でも最高のチームワークを実現する方法 ーサイボウズ開発チームのリモートワーク事例ー
Teppei Sato
Viewers also liked
(20)
SpringOne 2GX 2014 参加報告 & Spring 4.1について #jsug
SpringOne 2GX 2014 参加報告 & Spring 4.1について #jsug
Spring Day 2016 - Web API アクセス制御の最適解
Spring Day 2016 - Web API アクセス制御の最適解
Lineにおけるspring frameworkの活用
Lineにおけるspring frameworkの活用
Spring Day 2016 springの現在過去未来
Spring Day 2016 springの現在過去未来
Spring bootで学ぶ初めてのwebアプリ開発
Spring bootで学ぶ初めてのwebアプリ開発
アメブロの大規模システム刷新と それを支えるSpring
アメブロの大規模システム刷新と それを支えるSpring
Spring 5に備えるリアクティブプログラミング入門
Spring 5に備えるリアクティブプログラミング入門
Springを使ったwebアプリにリファクタリングしよう
Springを使ったwebアプリにリファクタリングしよう
Application Re-Architecture Technology ~ StrutsからSpring MVCへ ~
Application Re-Architecture Technology ~ StrutsからSpring MVCへ ~
Grailsでドメイン駆動設計を実践する時の勘所
Grailsでドメイン駆動設計を実践する時の勘所
Cloud Foundry x Wagby
Cloud Foundry x Wagby
楽天トラベルとSpring(Spring Day 2016)
楽天トラベルとSpring(Spring Day 2016)
ビッグじゃなくても使えるSpark Streaming
ビッグじゃなくても使えるSpark Streaming
Distributed Tracing Velocity2016
Distributed Tracing Velocity2016
Distributed tracing - get a grasp on your production
Distributed tracing - get a grasp on your production
Spring CloudとZipkinを利用した分散トレーシング
Spring CloudとZipkinを利用した分散トレーシング
形態素解析
形態素解析
WalB: Real-time and Incremental Backup System for Block Devices
WalB: Real-time and Incremental Backup System for Block Devices
3000社の業務データ絞り込みを支える技術
3000社の業務データ絞り込みを支える技術
離れた場所でも最高のチームワークを実現する方法 ーサイボウズ開発チームのリモートワーク事例ー
離れた場所でも最高のチームワークを実現する方法 ーサイボウズ開発チームのリモートワーク事例ー
Similar to Data Microservices with Spring Cloud Stream, Task, and Data Flow #jsug #springday
Spring Cloud Servicesの紹介 #pcf_tokyo
Spring Cloud Servicesの紹介 #pcf_tokyo
Toshiaki Maki
Meetup: Streaming Data Pipeline Development
Meetup: Streaming Data Pipeline Development
Timothy Spann
DBA Fundamentals Group: Continuous SQL with Kafka and Flink
DBA Fundamentals Group: Continuous SQL with Kafka and Flink
Timothy Spann
2024 February 28 - NYC - Meetup Unlocking Financial Data with Real-Time Pipel...
2024 February 28 - NYC - Meetup Unlocking Financial Data with Real-Time Pipel...
Timothy Spann
Spring Framework 5.0による Reactive Web Application #JavaDayTokyo
Spring Framework 5.0による Reactive Web Application #JavaDayTokyo
Toshiaki Maki
Delivering the power of data using Spring Cloud DataFlow and DataStax Enterpr...
Delivering the power of data using Spring Cloud DataFlow and DataStax Enterpr...
VMware Tanzu
Using the SDACK Architecture on Security Event Inspection by Yu-Lun Chen and ...
Using the SDACK Architecture on Security Event Inspection by Yu-Lun Chen and ...
Docker, Inc.
Hadoop application architectures - using Customer 360 as an example
Hadoop application architectures - using Customer 360 as an example
hadooparchbook
Real-time Analysis of Data Processing Pipelines with Spring Cloud Data Flow a...
Real-time Analysis of Data Processing Pipelines with Spring Cloud Data Flow a...
VMware Tanzu
Episode 3: Kubernetes and Big Data Services
Episode 3: Kubernetes and Big Data Services
Mesosphere Inc.
Azure HDlnsight에서 R 및 Spark를 이용하여 확장 가능한 머신러닝
Azure HDlnsight에서 R 및 Spark를 이용하여 확장 가능한 머신러닝
OSS On Azure
Big Data Analytics with Spark
Big Data Analytics with Spark
DataStax Academy
Monitoring as Code: Getting to Monitoring-Driven Development - DEV314 - re:In...
Monitoring as Code: Getting to Monitoring-Driven Development - DEV314 - re:In...
Amazon Web Services
The Heart of the Data Mesh Beats in Real-Time with Apache Kafka
The Heart of the Data Mesh Beats in Real-Time with Apache Kafka
Kai Wähner
Analyzing and processing streaming data with Amazon EMR - ADB204 - New York A...
Analyzing and processing streaming data with Amazon EMR - ADB204 - New York A...
Amazon Web Services
Data Science at Scale on MPP databases - Use Cases & Open Source Tools
Data Science at Scale on MPP databases - Use Cases & Open Source Tools
Esther Vasiete
Fraud Detection for Israel BigThings Meetup
Fraud Detection for Israel BigThings Meetup
Gwen (Chen) Shapira
DevOps Days Tel Aviv - Serverless Architecture
DevOps Days Tel Aviv - Serverless Architecture
Antons Kranga
Spark Streaming with Azure Databricks
Spark Streaming with Azure Databricks
Dustin Vannoy
Spark Summit EU talk by Christos Erotocritou
Spark Summit EU talk by Christos Erotocritou
Spark Summit
Similar to Data Microservices with Spring Cloud Stream, Task, and Data Flow #jsug #springday
(20)
Spring Cloud Servicesの紹介 #pcf_tokyo
Spring Cloud Servicesの紹介 #pcf_tokyo
Meetup: Streaming Data Pipeline Development
Meetup: Streaming Data Pipeline Development
DBA Fundamentals Group: Continuous SQL with Kafka and Flink
DBA Fundamentals Group: Continuous SQL with Kafka and Flink
2024 February 28 - NYC - Meetup Unlocking Financial Data with Real-Time Pipel...
2024 February 28 - NYC - Meetup Unlocking Financial Data with Real-Time Pipel...
Spring Framework 5.0による Reactive Web Application #JavaDayTokyo
Spring Framework 5.0による Reactive Web Application #JavaDayTokyo
Delivering the power of data using Spring Cloud DataFlow and DataStax Enterpr...
Delivering the power of data using Spring Cloud DataFlow and DataStax Enterpr...
Using the SDACK Architecture on Security Event Inspection by Yu-Lun Chen and ...
Using the SDACK Architecture on Security Event Inspection by Yu-Lun Chen and ...
Hadoop application architectures - using Customer 360 as an example
Hadoop application architectures - using Customer 360 as an example
Real-time Analysis of Data Processing Pipelines with Spring Cloud Data Flow a...
Real-time Analysis of Data Processing Pipelines with Spring Cloud Data Flow a...
Episode 3: Kubernetes and Big Data Services
Episode 3: Kubernetes and Big Data Services
Azure HDlnsight에서 R 및 Spark를 이용하여 확장 가능한 머신러닝
Azure HDlnsight에서 R 및 Spark를 이용하여 확장 가능한 머신러닝
Big Data Analytics with Spark
Big Data Analytics with Spark
Monitoring as Code: Getting to Monitoring-Driven Development - DEV314 - re:In...
Monitoring as Code: Getting to Monitoring-Driven Development - DEV314 - re:In...
The Heart of the Data Mesh Beats in Real-Time with Apache Kafka
The Heart of the Data Mesh Beats in Real-Time with Apache Kafka
Analyzing and processing streaming data with Amazon EMR - ADB204 - New York A...
Analyzing and processing streaming data with Amazon EMR - ADB204 - New York A...
Data Science at Scale on MPP databases - Use Cases & Open Source Tools
Data Science at Scale on MPP databases - Use Cases & Open Source Tools
Fraud Detection for Israel BigThings Meetup
Fraud Detection for Israel BigThings Meetup
DevOps Days Tel Aviv - Serverless Architecture
DevOps Days Tel Aviv - Serverless Architecture
Spark Streaming with Azure Databricks
Spark Streaming with Azure Databricks
Spark Summit EU talk by Christos Erotocritou
Spark Summit EU talk by Christos Erotocritou
More from Toshiaki Maki
From Spring Boot 2.2 to Spring Boot 2.3 #jsug
From Spring Boot 2.2 to Spring Boot 2.3 #jsug
Toshiaki Maki
Concourse x Spinnaker #concourse_tokyo
Concourse x Spinnaker #concourse_tokyo
Toshiaki Maki
Serverless with Spring Cloud Function, Knative and riff #SpringOneTour #s1t
Serverless with Spring Cloud Function, Knative and riff #SpringOneTour #s1t
Toshiaki Maki
Spring Boot Actuator 2.0 & Micrometer #jjug_ccc #ccc_a1
Spring Boot Actuator 2.0 & Micrometer #jjug_ccc #ccc_a1
Toshiaki Maki
Spring Boot Actuator 2.0 & Micrometer
Spring Boot Actuator 2.0 & Micrometer
Toshiaki Maki
Open Service Broker APIとKubernetes Service Catalog #k8sjp
Open Service Broker APIとKubernetes Service Catalog #k8sjp
Toshiaki Maki
Spring Cloud Function & Project riff #jsug
Spring Cloud Function & Project riff #jsug
Toshiaki Maki
Introduction to Spring WebFlux #jsug #sf_a1
Introduction to Spring WebFlux #jsug #sf_a1
Toshiaki Maki
BOSH / CF Deployment in modern ways #cf_tokyo
BOSH / CF Deployment in modern ways #cf_tokyo
Toshiaki Maki
Why PCF is the best platform for Spring Boot
Why PCF is the best platform for Spring Boot
Toshiaki Maki
Zipkin Components #zipkin_jp
Zipkin Components #zipkin_jp
Toshiaki Maki
マイクロサービスに必要な技術要素はすべてSpring Cloudにある #DO07
マイクロサービスに必要な技術要素はすべてSpring Cloudにある #DO07
Toshiaki Maki
実例で学ぶ、明日から使えるSpring Boot Tips #jsug
実例で学ぶ、明日から使えるSpring Boot Tips #jsug
Toshiaki Maki
Spring ❤️ Kotlin #jjug
Spring ❤️ Kotlin #jjug
Toshiaki Maki
Managing your Docker image continuously with Concourse CI
Managing your Docker image continuously with Concourse CI
Toshiaki Maki
Short Lived Tasks in Cloud Foundry #cfdtokyo
Short Lived Tasks in Cloud Foundry #cfdtokyo
Toshiaki Maki
今すぐ始めるCloud Foundry #hackt #hackt_k
今すぐ始めるCloud Foundry #hackt #hackt_k
Toshiaki Maki
Team Support in Concourse CI 2.0 #concourse_tokyo
Team Support in Concourse CI 2.0 #concourse_tokyo
Toshiaki Maki
From Zero to Hero with REST and OAuth2 #jjug
From Zero to Hero with REST and OAuth2 #jjug
Toshiaki Maki
Consumer Driven Contractsで REST API/マイクロサービスをテスト #m3tech
Consumer Driven Contractsで REST API/マイクロサービスをテスト #m3tech
Toshiaki Maki
More from Toshiaki Maki
(20)
From Spring Boot 2.2 to Spring Boot 2.3 #jsug
From Spring Boot 2.2 to Spring Boot 2.3 #jsug
Concourse x Spinnaker #concourse_tokyo
Concourse x Spinnaker #concourse_tokyo
Serverless with Spring Cloud Function, Knative and riff #SpringOneTour #s1t
Serverless with Spring Cloud Function, Knative and riff #SpringOneTour #s1t
Spring Boot Actuator 2.0 & Micrometer #jjug_ccc #ccc_a1
Spring Boot Actuator 2.0 & Micrometer #jjug_ccc #ccc_a1
Spring Boot Actuator 2.0 & Micrometer
Spring Boot Actuator 2.0 & Micrometer
Open Service Broker APIとKubernetes Service Catalog #k8sjp
Open Service Broker APIとKubernetes Service Catalog #k8sjp
Spring Cloud Function & Project riff #jsug
Spring Cloud Function & Project riff #jsug
Introduction to Spring WebFlux #jsug #sf_a1
Introduction to Spring WebFlux #jsug #sf_a1
BOSH / CF Deployment in modern ways #cf_tokyo
BOSH / CF Deployment in modern ways #cf_tokyo
Why PCF is the best platform for Spring Boot
Why PCF is the best platform for Spring Boot
Zipkin Components #zipkin_jp
Zipkin Components #zipkin_jp
マイクロサービスに必要な技術要素はすべてSpring Cloudにある #DO07
マイクロサービスに必要な技術要素はすべてSpring Cloudにある #DO07
実例で学ぶ、明日から使えるSpring Boot Tips #jsug
実例で学ぶ、明日から使えるSpring Boot Tips #jsug
Spring ❤️ Kotlin #jjug
Spring ❤️ Kotlin #jjug
Managing your Docker image continuously with Concourse CI
Managing your Docker image continuously with Concourse CI
Short Lived Tasks in Cloud Foundry #cfdtokyo
Short Lived Tasks in Cloud Foundry #cfdtokyo
今すぐ始めるCloud Foundry #hackt #hackt_k
今すぐ始めるCloud Foundry #hackt #hackt_k
Team Support in Concourse CI 2.0 #concourse_tokyo
Team Support in Concourse CI 2.0 #concourse_tokyo
From Zero to Hero with REST and OAuth2 #jjug
From Zero to Hero with REST and OAuth2 #jjug
Consumer Driven Contractsで REST API/マイクロサービスをテスト #m3tech
Consumer Driven Contractsで REST API/マイクロサービスをテスト #m3tech
Recently uploaded
Intro in Product Management - Коротко про професію продакт менеджера
Intro in Product Management - Коротко про професію продакт менеджера
Mark Opanasiuk
How we scaled to 80K users by doing nothing!.pdf
How we scaled to 80K users by doing nothing!.pdf
Srushith Repakula
ASRock Industrial FDO Solutions in Action for Industrial Edge AI _ Kenny at A...
ASRock Industrial FDO Solutions in Action for Industrial Edge AI _ Kenny at A...
FIDO Alliance
Demystifying gRPC in .Net by John Staveley
Demystifying gRPC in .Net by John Staveley
John Staveley
10 Differences between Sales Cloud and CPQ, Blanka Doktorová
10 Differences between Sales Cloud and CPQ, Blanka Doktorová
CzechDreamin
Oauth 2.0 Introduction and Flows with MuleSoft
Oauth 2.0 Introduction and Flows with MuleSoft
shyamraj55
The Metaverse: Are We There Yet?
The Metaverse: Are We There Yet?
Mark Billinghurst
Google I/O Extended 2024 Warsaw
Google I/O Extended 2024 Warsaw
GDSC PJATK
TEST BANK For, Information Technology Project Management 9th Edition Kathy Sc...
TEST BANK For, Information Technology Project Management 9th Edition Kathy Sc...
marcuskenyatta275
Powerful Start- the Key to Project Success, Barbara Laskowska
Powerful Start- the Key to Project Success, Barbara Laskowska
CzechDreamin
The Value of Certifying Products for FDO _ Paul at FIDO Alliance.pdf
The Value of Certifying Products for FDO _ Paul at FIDO Alliance.pdf
FIDO Alliance
AI presentation and introduction - Retrieval Augmented Generation RAG 101
AI presentation and introduction - Retrieval Augmented Generation RAG 101
vincent683379
Secure Zero Touch enabled Edge compute with Dell NativeEdge via FDO _ Brad at...
Secure Zero Touch enabled Edge compute with Dell NativeEdge via FDO _ Brad at...
FIDO Alliance
IESVE for Early Stage Design and Planning
IESVE for Early Stage Design and Planning
IES VE
Structuring Teams and Portfolios for Success
Structuring Teams and Portfolios for Success
UXDXConf
Unpacking Value Delivery - Agile Oxford Meetup - May 2024.pptx
Unpacking Value Delivery - Agile Oxford Meetup - May 2024.pptx
David Michel
A Business-Centric Approach to Design System Strategy
A Business-Centric Approach to Design System Strategy
UXDXConf
THE BEST IPTV in GERMANY for 2024: IPTVreel
THE BEST IPTV in GERMANY for 2024: IPTVreel
reely ones
Future Visions: Predictions to Guide and Time Tech Innovation, Peter Udo Diehl
Future Visions: Predictions to Guide and Time Tech Innovation, Peter Udo Diehl
Peter Udo Diehl
What's New in Teams Calling, Meetings and Devices April 2024
What's New in Teams Calling, Meetings and Devices April 2024
Stephanie Beckett
Recently uploaded
(20)
Intro in Product Management - Коротко про професію продакт менеджера
Intro in Product Management - Коротко про професію продакт менеджера
How we scaled to 80K users by doing nothing!.pdf
How we scaled to 80K users by doing nothing!.pdf
ASRock Industrial FDO Solutions in Action for Industrial Edge AI _ Kenny at A...
ASRock Industrial FDO Solutions in Action for Industrial Edge AI _ Kenny at A...
Demystifying gRPC in .Net by John Staveley
Demystifying gRPC in .Net by John Staveley
10 Differences between Sales Cloud and CPQ, Blanka Doktorová
10 Differences between Sales Cloud and CPQ, Blanka Doktorová
Oauth 2.0 Introduction and Flows with MuleSoft
Oauth 2.0 Introduction and Flows with MuleSoft
The Metaverse: Are We There Yet?
The Metaverse: Are We There Yet?
Google I/O Extended 2024 Warsaw
Google I/O Extended 2024 Warsaw
TEST BANK For, Information Technology Project Management 9th Edition Kathy Sc...
TEST BANK For, Information Technology Project Management 9th Edition Kathy Sc...
Powerful Start- the Key to Project Success, Barbara Laskowska
Powerful Start- the Key to Project Success, Barbara Laskowska
The Value of Certifying Products for FDO _ Paul at FIDO Alliance.pdf
The Value of Certifying Products for FDO _ Paul at FIDO Alliance.pdf
AI presentation and introduction - Retrieval Augmented Generation RAG 101
AI presentation and introduction - Retrieval Augmented Generation RAG 101
Secure Zero Touch enabled Edge compute with Dell NativeEdge via FDO _ Brad at...
Secure Zero Touch enabled Edge compute with Dell NativeEdge via FDO _ Brad at...
IESVE for Early Stage Design and Planning
IESVE for Early Stage Design and Planning
Structuring Teams and Portfolios for Success
Structuring Teams and Portfolios for Success
Unpacking Value Delivery - Agile Oxford Meetup - May 2024.pptx
Unpacking Value Delivery - Agile Oxford Meetup - May 2024.pptx
A Business-Centric Approach to Design System Strategy
A Business-Centric Approach to Design System Strategy
THE BEST IPTV in GERMANY for 2024: IPTVreel
THE BEST IPTV in GERMANY for 2024: IPTVreel
Future Visions: Predictions to Guide and Time Tech Innovation, Peter Udo Diehl
Future Visions: Predictions to Guide and Time Tech Innovation, Peter Udo Diehl
What's New in Teams Calling, Meetings and Devices April 2024
What's New in Teams Calling, Meetings and Devices April 2024
Data Microservices with Spring Cloud Stream, Task, and Data Flow #jsug #springday
1.
‹#›© 2016 Pivotal
Software, Inc. All rights reserved. ‹#›© 2016 Pivotal Software, Inc. All rights reserved. Data Microservices with Spring Cloud Stream, Task, and Data Flow Toshiaki Maki (@making) 2016-11-18 Spring Day 2016 #springday #jsug
2.
© 2016 Pivotal
Software, Inc. All rights reserved. Who am I ? • Toshiaki Maki (@making) http://blog.ik.am • Sr. Solutions Architect @Pivotal • Spring Framework enthusiast bit.ly/hajiboot2
3.
© 2016 Pivotal
Software, Inc. All rights reserved. Input Motivation HTTP TCP S3 JDBC Rabbit JMS Twitter Syslog Output HDFS Cassandra HAWQ Greenplum JDBC S3 TCP Gemfire ❓❔❓ • Real Time Analysis • Data Ingestion • ETL Process
4.
© 2016 Pivotal
Software, Inc. All rights reserved. Spring XD (eXtreme Data)
5.
© 2016 Pivotal
Software, Inc. All rights reserved. Spring XD (eXtreme Data) 🤔
6.
© 2016 Pivotal
Software, Inc. All rights reserved. Spring XD Architecture Container Container gpfdist Cassandra jms http ZooKeeper Message Broker XD Admin stream1 = http | cassandra stream2 = jms | gpfdist On Metal/VMs
7.
© 2016 Pivotal
Software, Inc. All rights reserved. Spring XD Architecture Container Container gpfdist Cassandra jms http ZooKeeper Message Broker XD Admin stream1 = http | cassandra stream2 = jms | gpfdist On Metal/VMs 😇
8.
© 2016 Pivotal
Software, Inc. All rights reserved. Spring XD Architecture Container Container gpfdist Cassandra jms http ZooKeeper Message Broker XD Admin stream1 = http | cassandra stream2 = jms | gpfdist On Metal/VMs
9.
© 2016 Pivotal
Software, Inc. All rights reserved. Modern Platforms Cloud Foundry YARN Kubernetes Mesos Docker Swarm Nomad OpenShift
10.
© 2016 Pivotal
Software, Inc. All rights reserved. Cloud Native Redesign •From multiple modules embedded in a container to standalone executable applications •From our own runtime to delegating to existing modern platforms
11.
© 2016 Pivotal
Software, Inc. All rights reserved. Data Microservices $ cat book.txt | tr ' ' '¥ ' | tr '[:upper:]' '[:lower:]' | tr -d '[:punct:]' | grep -v '[^a-z]‘ | sort | uniq -c | sort -rn | head
12.
© 2016 Pivotal
Software, Inc. All rights reserved. Data Microservices $ cat book.txt | tr ' ' '¥ ' | tr '[:upper:]' '[:lower:]' | tr -d '[:punct:]' | grep -v '[^a-z]‘ | sort | uniq -c | sort -rn | head
13.
© 2016 Pivotal
Software, Inc. All rights reserved. Data Microservices $ cat book.txt | tr ' ' '¥ ' | tr '[:upper:]' '[:lower:]' | tr -d '[:punct:]' | grep -v '[^a-z]‘ | sort | uniq -c | sort -rn | head Microservice for each data processing
14.
© 2016 Pivotal
Software, Inc. All rights reserved. Data Microservices $ cat book.txt | tr ' ' '¥ ' | tr '[:upper:]' '[:lower:]' | tr -d '[:punct:]' | grep -v '[^a-z]‘ | sort | uniq -c | sort -rn | head Microservice for each data processing
15.
© 2016 Pivotal
Software, Inc. All rights reserved. Data Microservices $ cat book.txt | tr ' ' '¥ ' | tr '[:upper:]' '[:lower:]' | tr -d '[:punct:]' | grep -v '[^a-z]‘ | sort | uniq -c | sort -rn | head Microservice for each data processing bound with Message Brokers
16.
© 2016 Pivotal
Software, Inc. All rights reserved. Data Microservices $ cat book.txt | tr ' ' '¥ ' | tr '[:upper:]' '[:lower:]' | tr -d '[:punct:]' | grep -v '[^a-z]‘ | sort | uniq -c | sort -rn | head Microservice for each data processing bound with Message Brokers
17.
© 2016 Pivotal
Software, Inc. All rights reserved. Data Microservices $ cat book.txt | tr ' ' '¥ ' | tr '[:upper:]' '[:lower:]' | tr -d '[:punct:]' | grep -v '[^a-z]‘ | sort | uniq -c | sort -rn | head Microservice for each data processing bound with Message Brokers on the modern platform
18.
© 2016 Pivotal
Software, Inc. All rights reserved. Data Microservices $ cat book.txt | tr ' ' '¥ ' | tr '[:upper:]' '[:lower:]' | tr -d '[:punct:]' | grep -v '[^a-z]‘ | sort | uniq -c | sort -rn | head Microservice for each data processing bound with Message Brokers on the modern platform
19.
© 2016 Pivotal
Software, Inc. All rights reserved. Data Microservices $ cat book.txt | tr ' ' '¥ ' | tr '[:upper:]' '[:lower:]' | tr -d '[:punct:]' | grep -v '[^a-z]‘ | sort | uniq -c | sort -rn | head Microservice for each data processing bound with Message Brokers on the modern platform
20.
© 2016 Pivotal
Software, Inc. All rights reserved. Spring Cloud Data Flow • Microservices-based Distributed Data Pipelines •Long Lived Stream Applications •Short Lived Task Applications
21.
© 2016 Pivotal
Software, Inc. All rights reserved. Spring Cloud Data Flow • Microservices-based Distributed Data Pipelines •Long Lived Stream Applications •Short Lived Task Applications Spring Cloud Stream
22.
© 2016 Pivotal
Software, Inc. All rights reserved. Spring Cloud Data Flow • Microservices-based Distributed Data Pipelines •Long Lived Stream Applications •Short Lived Task Applications Spring Cloud Stream Spring Cloud Task
23.
© 2016 Pivotal
Software, Inc. All rights reserved. Spring Cloud Data Flow • Microservices-based Distributed Data Pipelines •Long Lived Stream Applications •Short Lived Task Applications Spring Cloud Stream Spring Cloud Task Orchestration Layer
24.
© 2016 Pivotal
Software, Inc. All rights reserved. Structure Spring Cloud Data Flow Spring Cloud Deployer (SPI) Spring Cloud Stream Spring Cloud Task Spring Integration Spring Boot Spring Batch
25.
© 2016 Pivotal
Software, Inc. All rights reserved. Structure Spring Cloud Data Flow Spring Cloud Deployer (SPI) Spring Cloud Stream Spring Cloud Task Spring Integration Spring Boot Spring Batch • Spring Cloud Deployer Local • Spring Cloud Deployer Cloud Foundry • Spring Cloud Deployer Yarn • Spring Cloud Deployer Kubernetes • Spring Cloud Deployer Mesos
26.
© 2016 Pivotal
Software, Inc. All rights reserved. SCDF Deployment Platform Spring Cloud Data Flow Server Deployer SPI REST API SCDFShell SCDF Flo
27.
© 2016 Pivotal
Software, Inc. All rights reserved. Spring Cloud Data Flow Architecture gpfdist cassandra jms http stream1 = http | cassandra stream2 = jms | gpfdist Message Broker Data Flow Server DB Platform Runtime
28.
© 2016 Pivotal
Software, Inc. All rights reserved. Spring Cloud Data Flow Architecture gpfdist cassandra jms http stream1 = http | cassandra stream2 = jms | gpfdist Message Broker Data Flow Server DB Platform Runtime
29.
‹#›© 2016 Pivotal
Software, Inc. All rights reserved. Spring Cloud Stream
30.
© 2016 Pivotal
Software, Inc. All rights reserved. Spring Cloud Stream • Event-driven microservice framework • Built on battle-tested components (Spring Boot / Spring Integration) • Opinionated primitives for streaming applications • Persistent Pub/Sub • Consumer Groups • Partitioning Support • Pluggable messaging middleware bindings source | processor | sink
31.
© 2016 Pivotal
Software, Inc. All rights reserved. JJUG CCC in 2 weeks
32.
© 2016 Pivotal
Software, Inc. All rights reserved. Spring Cloud Stream Applications Twitter Stream Cassandra java -jar twittersource.jar --server.port=8080 --consumerKey=XYZ --consumerSecret=ABC --spring.cloud.stream.bindings. output.destination=ingest Source Sink java -jar cassandrasink.jar --server.port=8081 --spring.cassandra.keyspace=tweet --spring.cloud.stream.bindings. input.destination=ingest
33.
© 2016 Pivotal
Software, Inc. All rights reserved. Spring Cloud Stream Applications Twitter Stream Cassandra java -jar twittersource.jar --server.port=8080 --consumerKey=XYZ --consumerSecret=ABC --spring.cloud.stream.bindings. output.destination=ingest Source Sink java -jar cassandrasink.jar --server.port=8081 --spring.cassandra.keyspace=tweet --spring.cloud.stream.bindings. input.destination=ingest Twitter Stream Cassandraingest
34.
© 2016 Pivotal
Software, Inc. All rights reserved. Message Binders • @EnableBinding • Binder Implementations • Production-Ready • Rabbit MQ • Apache Kafka • Experimental • JMS • Google PubSub
35.
© 2016 Pivotal
Software, Inc. All rights reserved. Programming Model (Sink) @SpringBootApplication @EnableBinding(Sink.class) public class DemoSinkApp { @StreamListener(Sink.INPUT) void receive(Message<String> message) { System.out.println("Received " + message); } public static void main(String[] args) { SpringApplication.run(DemoSinkApp.class, args); } }
36.
© 2016 Pivotal
Software, Inc. All rights reserved. Sink Properties (Sink) spring.cloud.stream.bindings.input.destination=demo-strm demo- strm input
37.
© 2016 Pivotal
Software, Inc. All rights reserved. Programming Model (Source) @SpringBootApplication @RestController @EnableBinding(Source.class) public class DemoSourceApp { @Autowired Source source; @GetMapping void send(@RequestParam String text) { source.output() .send(MessageBuilder.withPayload(text).build()); } public static void main(String[] args) { SpringApplication.run(DemoSourceApp.class, args); } }
38.
© 2016 Pivotal
Software, Inc. All rights reserved. Programming Model (Source) @SpringBootApplication @RestController @EnableBinding(Source.class) public class DemoSourceApp { @Bean @InboundChannelAdapter(channel = Source.OUTPUT, poller = @Poller(fixedDelay = "1000", maxMessagesPerPoll = "1")) MessageSource<String> source() { return () -> MessageBuilder.withPayload("Hi").build()); } public static void main(String[] args) { SpringApplication.run(DemoSourceApp.class, args); }}
39.
© 2016 Pivotal
Software, Inc. All rights reserved. Properties (Source) spring.cloud.stream.bindings.output.destination=demo-strm demo- strm Source output
40.
© 2016 Pivotal
Software, Inc. All rights reserved. Binder (RabbitMQ) <dependency> <groupId>org.springframework.cloud</groupId> <artifactId>spring-cloud-starter-stream-rabbit</artifactId> </dependency> demo- strm
41.
© 2016 Pivotal
Software, Inc. All rights reserved. Binder (Apache Kafka) <dependency> <groupId>org.springframework.cloud</groupId> <artifactId>spring-cloud-starter-stream-kafka</artifactId> </dependency> demo- strm
42.
© 2016 Pivotal
Software, Inc. All rights reserved. Sink Pipeline demo- strm input Source output source | sink
43.
© 2016 Pivotal
Software, Inc. All rights reserved. Programming Model (Processor) @SpringBootApplication @EnableBinding(Processor.class) public class DemoProcessorApp { @StreamListener(Processor.INPUT) @SendTo(Processor.OUTPUT) void receive(String text) { return "[[" + text + "]]"; } public static void main(String[] args) { SpringApplication.run(DemoProcessorApp.class, args); } }
44.
© 2016 Pivotal
Software, Inc. All rights reserved. Properties (Processor) spring.cloud.stream.bindings.output.destination=my-source spring.cloud.stream.bindings.input.destination=my-source spring.cloud.stream.bindings.output.destination=my-proc spring.cloud.stream.bindings.input.destination=my-proc Source Processor Sink
45.
© 2016 Pivotal
Software, Inc. All rights reserved. Pipeline my- source Source output source | processor | sink Processor output input my-proc Sink input
46.
© 2016 Pivotal
Software, Inc. All rights reserved. Reactive API Support @SpringBootApplication @EnableBinding(Processor.class) public class DemoProcessorRxApp { @StreamListener @Output(Processor.OUTPUT) public Flux<String> receive(@Input(Processor.INPUT) Flux<String> stream) { return stream.map(text -> "[[" + text + "]]"); } public static void main(String[] args) { SpringApplication.run(DemoProcessorRxApp.class, args); } }
47.
© 2016 Pivotal
Software, Inc. All rights reserved. Reactive API Support @StreamListener @Output(Processor.OUTPUT) public Flux<AverageData> receive(@Input(Processor.INPUT) Flux<SensorData> stream) { return stream.window(Duration.ofSecond(20), Duration.ofSecond(10)) .flatMap(win -> win.groupBy(sensor -> sensor.id)) .flatMap(group -> calcAverage(group)); }
48.
© 2016 Pivotal
Software, Inc. All rights reserved. Spring Cloud Stream App Starters Source Processor Sink file ftp gemfire gemfire-cq http jdbc jms load-generator loggregator mail mongodb rabbit s3 sftp syslog tcp tcp-client time trigger triggertask twitterstream bridge filter groovy-filter groovy- transform httpclient pommel scriptable- transform splitter tcp-client transform aggregate- counter cassandra counter field-value- counter file ftp gemfire gpfdist hfs hdfs-dataset jdbc log rabbit redis-pubsub router s3 sftp task-launcher- local task-lancher- yarn tcp throughput web socket http://cloud.spring.io/spring-cloud-stream-app-starters/
49.
© 2016 Pivotal
Software, Inc. All rights reserved. Stream Orchestration in SCDF ingest = twitterstream | cassandra dataflow:> stream create --name=ingest --definition="twitterstream | cassandra" dataflow:> stream deploy --name=ingest |
50.
© 2016 Pivotal
Software, Inc. All rights reserved. Import apps dataflow:> app import --uri http://bit.ly/1-0-4- GA-stream-applications-rabbit-maven rabbit maven kafka docker stream-applications- http://cloud.spring.io/spring-cloud-stream-app-starters/
51.
© 2016 Pivotal
Software, Inc. All rights reserved. Register your own apps dataflow:> app register --name foo --type source --uri http://example.com/foo-source-1.0.jar dataflow:> app register --name foo --type source --uri maven://com.example:foo-source:1.0 dataflow:> app register --name foo --type source --uri docker:myapps/foo-source:1.0 dataflow:> app register --name foo --type source --uri file:///tmp/foo-source:1.0
52.
© 2016 Pivotal
Software, Inc. All rights reserved. Instance Count dataflow:> stream create --name=s1 --definition="http | work | hdfs" dataflow:> stream deploy --name=s1 -- properties="app.http.count=2,app.work.count=3,app .hdfs.count=4"
53.
© 2016 Pivotal
Software, Inc. All rights reserved. Instance Count http work hdfs http work work hdfs hdfs hdfs LoadBalancer
54.
© 2016 Pivotal
Software, Inc. All rights reserved. Resource Management dataflow:> stream deploy --name=s1 -- properties="app.work.spring.cloud.deployer.cloudf oundry.memory=2048"
55.
© 2016 Pivotal
Software, Inc. All rights reserved. Resource Management http http work work work hdfs hdfs hdfs hdfs
56.
© 2016 Pivotal
Software, Inc. All rights reserved. Spring Flo http://projects.spring.io/spring-flo/
57.
© 2016 Pivotal
Software, Inc. All rights reserved. Stream Core Features •Persistent Pub-Sub •Consumer Group •Partitioning Support
58.
© 2016 Pivotal
Software, Inc. All rights reserved. Persistent Pub-Sub HTTP Average Top Ns1.http s1.ave Message Broker
59.
© 2016 Pivotal
Software, Inc. All rights reserved. Persistent Pub-Sub HTTP Average HDFS Top Ns1.http s1.ave Message Broker
60.
© 2016 Pivotal
Software, Inc. All rights reserved. Persistent Pub-Sub HTTP Average HDFS Top N Fault Detection s1.http s1.ave Message Broker
61.
© 2016 Pivotal
Software, Inc. All rights reserved. Persistent Pub-Sub HTTP Average HDFS Top N Fault Detection s1.http s1.ave Message Broker {"id":1, "temperature":38}
62.
© 2016 Pivotal
Software, Inc. All rights reserved. Persistent Pub-Sub HTTP Average HDFS Top N Fault Detection s1.http s1.ave Message Broker {"id":1, "temperature":38}{"id":1, "temperature":38}
63.
© 2016 Pivotal
Software, Inc. All rights reserved. Persistent Pub-Sub HTTP Average HDFS Top N Fault Detection s1.http s1.ave Message Broker {"id":1, "temperature":38} {"id":1, "temperature":38} {"id":1, "temperature":38}
64.
© 2016 Pivotal
Software, Inc. All rights reserved. Persistent Pub-Sub HTTP Average HDFS Top N Fault Detection s1.http s1.ave Message Broker {"id":1, "temperature": {"id":1, "temperature":38} {"id":1, "temperature":38}
65.
© 2016 Pivotal
Software, Inc. All rights reserved. HTTP Average HDFS Top N Fault Detection s1.http s1.ave Message Broker Average Average HDFS HDFS
66.
© 2016 Pivotal
Software, Inc. All rights reserved. HTTP Average HDFS Top N Fault Detection s1.http s1.ave Message Broker {"id":1, "temperature":38} Average Average HDFS HDFS
67.
© 2016 Pivotal
Software, Inc. All rights reserved. HTTP Average HDFS Top N Fault Detection s1.http s1.ave Message Broker {"id":1, "temperature":38}{"id":1, "temperature":38} Average Average HDFS HDFS
68.
© 2016 Pivotal
Software, Inc. All rights reserved. HTTP Average HDFS Top N Fault Detection s1.http s1.ave Message Broker {"id":1, "temperature":38} {"id":1, "temperature":38} Average Average HDFS HDFS {"id":1, "temperature":38} {"id":1, "temperature":38} {"id":1, "temperature":38} {"id":1, "temperature":38}
69.
© 2016 Pivotal
Software, Inc. All rights reserved. HTTP Average HDFS Top N Fault Detection s1.http s1.ave Message Broker {"id":1, "temperature":38} {"id":1, "temperature":38} Average Average HDFS HDFS {"id":1, "temperature":38} {"id":1, "temperature":38} {"id":1, "temperature":38} {"id":1, "temperature":38} 😩
70.
© 2016 Pivotal
Software, Inc. All rights reserved. HTTP Average HDFS Top N Fault Detection s1.http s1.ave Message Broker {"id":1, "temperature":38} {"id":1, "temperature":38} Average Average HDFS HDFS {"id":1, "temperature":38} {"id":1, "temperature":38} {"id":1, "temperature":38} {"id":1, "temperature":38} 😩 Consumer Group
71.
© 2016 Pivotal
Software, Inc. All rights reserved. HTTP Average HDFS Top N Fault Detection s1.http s1.ave Message Broker {"id":1, "temperature":38} {"id":1, "temperature":38} Average Average HDFS HDFS {"id":1, "temperature":38} {"id":1, "temperature":38} {"id":1, "temperature":38} {"id":1, "temperature":38} 😩 spring.cloud.stream.bindings.<channelName>.group=ave Spring Cloud Stream group is configured by default 😁 Spring Cloud Data Flow Consumer Group
72.
© 2016 Pivotal
Software, Inc. All rights reserved. Consumer Group HTTP Average HDFS Top N Fault Detection s1.http s1.ave Message Broker Average Average HDFS HDFS group=ave group=hdfs
73.
© 2016 Pivotal
Software, Inc. All rights reserved. Consumer Group HTTP Average HDFS Top N Fault Detection s1.http s1.ave Message Broker Average Average HDFS HDFS group=ave group=hdfs {"id":1, "temperature":38}
74.
© 2016 Pivotal
Software, Inc. All rights reserved. Consumer Group HTTP Average HDFS Top N Fault Detection s1.http s1.ave Message Broker Average Average HDFS HDFS group=ave group=hdfs {"id":1, "temperature":38}{"id":1, "temperature":38}
75.
© 2016 Pivotal
Software, Inc. All rights reserved. Consumer Group HTTP Average HDFS Top N Fault Detection s1.http s1.ave Message Broker Average Average HDFS HDFS group=ave group=hdfs {"id":1, "temperature":38} {"id":1, "temperature":38}
76.
© 2016 Pivotal
Software, Inc. All rights reserved. Consumer Group HTTP Average HDFS Top N Fault Detection s1.http s1.ave Message Broker Average Average HDFS HDFS group=ave group=hdfs {"id":1, "temperature":38} {"id":1, "temperature":38} 🤓
77.
© 2016 Pivotal
Software, Inc. All rights reserved. Consumer Group HTTP Average HDFS Top N Fault Detection s1.http s1.ave Message Broker Average Average HDFS HDFS group=ave group=hdfs {"id":1, "temperature":38} {"id":1, "temperature":38} 🤓consumer group subscriptions are durable 😁
78.
© 2016 Pivotal
Software, Inc. All rights reserved. HTTP Average s1.http Average
79.
© 2016 Pivotal
Software, Inc. All rights reserved. HTTP Average s1.http Average {"id":1, "temperature":38} {"id":2, "temperature":41} {"id":2, "temperature":42} {"id":1, "temperature":37}
80.
© 2016 Pivotal
Software, Inc. All rights reserved. HTTP Average s1.http Average {"id":1, "temperature":38} {"id":2, "temperature":41} {"id":2, "temperature":42} {"id":1, "temperature":37}
81.
© 2016 Pivotal
Software, Inc. All rights reserved. HTTP Average s1.http Average {"id":1, "temperature":38} {"id":2, "temperature":41} {"id":2, "temperature":42} {"id":1, "temperature":37}
82.
© 2016 Pivotal
Software, Inc. All rights reserved. HTTP Average s1.http Average {"id":1, "temperature":38} {"id":2, "temperature":41} {"id":2, "temperature":42} {"id":1, "temperature":37}
83.
© 2016 Pivotal
Software, Inc. All rights reserved. HTTP Average s1.http Average {"id":1, "temperature":38} {"id":2, "temperature":41} {"id":2, "temperature":42} {"id":1, "temperature":37}
84.
© 2016 Pivotal
Software, Inc. All rights reserved. HTTP Average s1.http Average {"id":1, "temperature":38} {"id":2, "temperature":41} {"id":2, "temperature":42} {"id":1, "temperature":37} 😩
85.
© 2016 Pivotal
Software, Inc. All rights reserved. HTTP Average s1.http Average {"id":1, "temperature":38} {"id":2, "temperature":41} {"id":2, "temperature":42} {"id":1, "temperature":37} 😩 Partitioning Support(Stateful Stream)
86.
© 2016 Pivotal
Software, Inc. All rights reserved. HTTP Average s1.http Average {"id":1, "temperature":38} {"id":2, "temperature":41} {"id":2, "temperature":42} {"id":1, "temperature":37} 😩stream deploy --name=xxxx --properties= "app.http.producer.partitionKeyExpression=payload.id" spring.cloud.stream.bindings.<channelName>.producer.par titionKeyExpression=payload.id Spring Cloud Stream Spring Cloud Data Flow Partitioning Support(Stateful Stream)
87.
© 2016 Pivotal
Software, Inc. All rights reserved. Partitioning Support(Stateful Stream) HTTP Average s1.http Average
88.
© 2016 Pivotal
Software, Inc. All rights reserved. Partitioning Support(Stateful Stream) HTTP Average s1.http Average {"id":1, "temperature":38} {"id":2, "temperature":41} {"id":2, "temperature":42} {"id":1, "temperature":37}
89.
© 2016 Pivotal
Software, Inc. All rights reserved. Partitioning Support(Stateful Stream) HTTP Average s1.http Average {"id":1, "temperature":38} {"id":2, "temperature":41} {"id":2, "temperature":42} {"id":1, "temperature":37}
90.
© 2016 Pivotal
Software, Inc. All rights reserved. Partitioning Support(Stateful Stream) HTTP Average s1.http Average {"id":1, "temperature":38} {"id":2, "temperature":41} {"id":2, "temperature":42} {"id":1, "temperature":37}
91.
© 2016 Pivotal
Software, Inc. All rights reserved. Partitioning Support(Stateful Stream) HTTP Average s1.http Average {"id":1, "temperature":38} {"id":2, "temperature":41} {"id":2, "temperature":42} {"id":1, "temperature":37}
92.
© 2016 Pivotal
Software, Inc. All rights reserved. Partitioning Support(Stateful Stream) HTTP Average s1.http Average {"id":1, "temperature":38} {"id":2, "temperature":41} {"id":2, "temperature":42} {"id":1, "temperature":37}
93.
© 2016 Pivotal
Software, Inc. All rights reserved. Partitioning Support(Stateful Stream) HTTP Average s1.http Average {"id":1, "temperature":38} {"id":2, "temperature":41} {"id":2, "temperature":42} {"id":1, "temperature":37} 🤓
94.
© 2016 Pivotal
Software, Inc. All rights reserved. Simple Real Time Analytics tweets = twitterstream | hdfs analytics = :ingest.twitterstream > field-value-counter --fieldName=lang HTTP s1.http HDFS COUNTER Data Flow Server REST API
95.
‹#›© 2016 Pivotal
Software, Inc. All rights reserved. Demo (Twitter Stream!)
96.
© 2016 Pivotal
Software, Inc. All rights reserved.
97.
‹#›© 2016 Pivotal
Software, Inc. All rights reserved. Spring Cloud Task
98.
© 2016 Pivotal
Software, Inc. All rights reserved. Spring Cloud Task • Spring Boot based framework for short lived task • Spring Boot's CommandLineRunner or Spring Batch • Result of each process persists in Task Repository • Well Integrated with Spring Batch (Job Repository)
99.
© 2016 Pivotal
Software, Inc. All rights reserved. Programming Model (CLR) @SpringBootApplication @EnableTask public class DemoTaskApp { @Bean CommandLineRunner clr() { return args -> System.out.println("Task!"); } public static void main(String[] args) { SpringApplication.run(DemoTaskApp.class, args); } } spring.application.name=hello
100.
© 2016 Pivotal
Software, Inc. All rights reserved. Task Execution List Task Name ID Start Time End Time Exit Code hello 3 Tue Nov 15 16:49:08 JST 2016 Tue Nov 15 16:49:08 JST 2016 0 hello 2 Tue Nov 15 16:48:54 JST 2016 Tue Nov 15 16:48:55 JST 2016 0 hello 1 Tue Nov 15 16:48:23 JST 2016 Tue Nov 15 16:48:23 JST 2016 0
101.
© 2016 Pivotal
Software, Inc. All rights reserved. Programming Model (Spring Batch) @SpringBootApplication @EnableBatchProcessing @EnableTask public class DemoBatchApp { @Autowired JobBuilderFactory jobBuilderFactory; @Autowired StepBuilderFactory stepBuilderFactory; @Bean Step step1() { /* ... */ } @Bean Step step2() { /* ... */ } @Bean Job job() { return jobBuilderFactory.get("job") .start(step1()).next(step2()).build()} public static void main(String[] args) { SpringApplication.run(DemoTaskApp.class, args); }} https://github.com/making/cf-spring-batch-demo spring.application.name=hello-batch
102.
© 2016 Pivotal
Software, Inc. All rights reserved. Task Execution List Task Name ID Start Time End Time Exit Code hello- batch 5 Tue Nov 15 17:28:49 JST 2016 Tue Nov 15 17:28:49 JST 2016 0 hello- batch 4 Tue Nov 15 17:27:56 JST 2016 Tue Nov 15 17:27:57 JST 2016 0
103.
© 2016 Pivotal
Software, Inc. All rights reserved. Job Execution List ID Task ID Start Time Step Execution Count Definition Status 2 5 Tue Nov 15 17:28:49 JST 2016 2 Created 1 4 Tue Nov 15 17:27:56 JST 2016 2 Created
104.
© 2016 Pivotal
Software, Inc. All rights reserved. Step Execution List ID Step Name Job Exec Id Start Time End Time Status 1 step1 1 Tue Nov 15 17:28:49 JST 2016 Tue Nov 15 17:28:49 JST 2016 COMPLETED 2 step2 1 Tue Nov 15 17:27:56 JST 2016 Tue Nov 15 17:27:57 JST 2016 COMPLETED
105.
© 2016 Pivotal
Software, Inc. All rights reserved. Task Orchestration in SCDF >task create hello --definition="timestamp --format=¥"yyyy¥"" >task launch hello timestamp Data Flow Server DB Task Name Start Time End Time Exit Code Exit Message Last Updated Time Parameters Message Broker
106.
© 2016 Pivotal
Software, Inc. All rights reserved. Task Execution List
107.
© 2016 Pivotal
Software, Inc. All rights reserved. Job Execution List
108.
© 2016 Pivotal
Software, Inc. All rights reserved. Job Execution Details
109.
© 2016 Pivotal
Software, Inc. All rights reserved. Spring Cloud Data Flow -> Task http | tasklaunchrequest --uri=... | task-lancher timestamp Data Flow Server DB Message Broker
110.
‹#›© 2016 Pivotal
Software, Inc. All rights reserved. Demo (Scalable Tasks)
111.
© 2016 Pivotal
Software, Inc. All rights reserved. Pivotal Cloud Foundry PDF PDF PDF PDF S3 Spring Cloud Data Flow Task Task Task Task PDF PDF Spring Cloud Task Spring Batch index index source sink
112.
‹#›© 2016 Pivotal
Software, Inc. All rights reserved. Getting Started with SCDF on Cloud Foundry
113.
© 2016 Pivotal
Software, Inc. All rights reserved. Install PCF Dev • https://docs.pivotal.io/pcf-dev • Cloud Foundry on your laptop • Included • Redis / RabbitMQ / MySQL • Spring Cloud Services • Install with cf dev start
114.
© 2016 Pivotal
Software, Inc. All rights reserved. cf create-service p-mysql 512mb df-mysql cf create-service p-rabbitmq standard df-rebbitmq cf create-service p-redis shared-vm df-redis cf push dataflow-server -p spring-cloud-dataflow- server-cloudfoundry-1.0.1.RELEASE.jar Deploy SCDF Server on PCF Dev
115.
© 2016 Pivotal
Software, Inc. All rights reserved. Tutorial • 📗 https://blog.ik.am/entries/396 • 💎 https://github.com/making-demo-scdf/spring-cloud- dataflow-cookbook
116.
© 2016 Pivotal
Software, Inc. All rights reserved. Upcoming Features • Some ‘porting’ from XD • Batch Job DSL + Designer • Role based access • Looking forward • Spring Cloud Sleuth • JavaDSL • In-place application version upgrades with Spinnaker • Application Groups • Polyglot • Expanded analytics with Redis and Python/R ecosystem • More provided apps/tasks •SCDF Tile for Pivotal Cloud Foundry
117.
© 2016 Pivotal
Software, Inc. All rights reserved. Thanks! • References • http://www.slideshare.net/SpringCentral/data-microservices-in-t cloud ✨ • https://cloud.spring.io/spring-cloud-dataflow/ • http://cloud.spring.io/spring-cloud-stream/ • http://cloud.spring.io/spring-cloud-stream-app-starters/ • https://github.com/spring-cloud/spring-cloud-task/ • http://cloud.spring.io/spring-cloud-dataflow-server-cloudfoundry
Download now