SlideShare a Scribd company logo
1 of 25
Download to read offline
/ 25
閃亮的火花
Gil @ Spark.TW
⼆零⼀七年⼗⼆⽉⼆⼗七⽇
https://www.marketingcloud.com/blog/meet-spark-inspiration-information/
1
/ 25
2
/ 25
outline
• Introduction to spark
• Introduction to shiny
• shiny X spark
• Case I: Pokemon GO
• Case II:Audience Map
3
/ 25
spark
4
/ 25 https://spark.apache.org/
5
/ 25
spark stack
sparkR
pyspark
https://www.safaribooksonline.com/library/view/data-analytics-with/9781491913734/ch04.html
Storage
Cluster
Manager
Processing
Engine
Access &
Interfaces
6
/ 25
Key Advantages of Spark & R
+
Fast!
Flexible
Scalable
Statistical!
Interactive
Packages
https://spark-summit.org/2014/wp-content/uploads/2014/07/SparkR-SparkSummit.pdf
7
/ 25
shiny
8
/ 25 https://shiny.rstudio.com/
(official website before 2017/09)
9
/ 25 https://shiny.rstudio.com/
(official website now)
10
/ 25
about shiny…
• 新的 framework,觀望⼀一下...
• google trend
• https://trends.google.com.tw/trends/explore?
date=all&q=R%20shiny,d3.js
• production
• https://shiny.rstudio.com/gallery/see-more.html
11
/ 25
about shiny…
• like MVC
• demo
http://13.115.248.137:3838/kmeans/
model ⬌ function.R
view ⬌ ui.R
control ⬌ server.R
12
/ 25
shiny vs. tableau
shiny tableau
pros
• free
• data science 

packages / algorithms
• scale out
• easy to production
• easy to maintain
cons
• hard to production
• hard to maintain
• not free
• hard to import 

packages / algorithms
• hard to scale out
13
/ 25
shiny vs. javascript+php
• javascript chart framework
• like ds.js, highchart.js, plotly.js, chart.js, echarts.js, …
• R 有相關套件 →	 1+ 1 > 2
• 分⼯工
• ⼀一⼈人團隊:data science + front-end →		不太可能、⽣生命有限,學
海海無涯
• 跨部⾨門合作、找前端⼯工程師
14
/ 25
shiny X spark
15
/ 25
shiny X spark
• 既然 shiny 可以⽤用 R 做 web UI、⽽而 spark ⼜又有 sparkR API 可以⽤用
• front-end to back-end
• 為了了視覺化⼀一些想法,省去⼤大量量的 ETL (AWS S3 → MYSQL)
• 快速地獲得資訊、探索性數據分析
• ⼀一般 BI tool
• 不易易 connect unstructured DB
• 無法 trigger job
16
/ 25
Pokemon GO
17
Interactive Maps
Web Framework
Compute Engine
+
Where is the
Dragonite nest ?
+
/ 25
data pipeline brief
structure DB
AWS RDS
ETL
BI tool
raw log
AWS S3
ETL
profile tables
AWS S3
AWS Redshift
data warehouse
AWS Athena
18
/ 25
AWS Redshift
data warehouse
AWS RDS
AWS S3 AWS S3
19
/ 25
“standing on the
shoulders of giants”
20
/ 25
data query ➡ dump ➡ visualization
cache
data query ➡ cache ➡ visualization
• local files
• mysql
• schema
• device id level
• UU monitor
21
/ 25
limitation & plan B (?)
R object
files ?
rpy2 ?
(convert python object to R)
user
API
22
/ 25
23
Control
View
Model
/ 25
24
/ 25
thanks
25

More Related Content

Similar to Shiny Spark (閃亮的火花) (20171227 - Spark.TW 3rd Anniversary Sharing)

An Update on Scaling Data Science Applications with SparkR in 2018 with Heiko...
An Update on Scaling Data Science Applications with SparkR in 2018 with Heiko...An Update on Scaling Data Science Applications with SparkR in 2018 with Heiko...
An Update on Scaling Data Science Applications with SparkR in 2018 with Heiko...
Databricks
 

Similar to Shiny Spark (閃亮的火花) (20171227 - Spark.TW 3rd Anniversary Sharing) (20)

Using Spark Part Time
Using Spark Part TimeUsing Spark Part Time
Using Spark Part Time
 
Drupal Integrated DevOps Workflow
Drupal Integrated DevOps WorkflowDrupal Integrated DevOps Workflow
Drupal Integrated DevOps Workflow
 
Spark tutorial
Spark tutorialSpark tutorial
Spark tutorial
 
Concourse x Spinnaker #concourse_tokyo
Concourse x Spinnaker #concourse_tokyoConcourse x Spinnaker #concourse_tokyo
Concourse x Spinnaker #concourse_tokyo
 
JEEConf 2015 - Introduction to real-time big data with Apache Spark
JEEConf 2015 - Introduction to real-time big data with Apache SparkJEEConf 2015 - Introduction to real-time big data with Apache Spark
JEEConf 2015 - Introduction to real-time big data with Apache Spark
 
An Update on Scaling Data Science Applications with SparkR in 2018 with Heiko...
An Update on Scaling Data Science Applications with SparkR in 2018 with Heiko...An Update on Scaling Data Science Applications with SparkR in 2018 with Heiko...
An Update on Scaling Data Science Applications with SparkR in 2018 with Heiko...
 
5 reasons why spark is in demand!
5 reasons why spark is in demand!5 reasons why spark is in demand!
5 reasons why spark is in demand!
 
End-to-End Data Pipelines with Apache Spark
End-to-End Data Pipelines with Apache SparkEnd-to-End Data Pipelines with Apache Spark
End-to-End Data Pipelines with Apache Spark
 
Spark - Migration Story
Spark - Migration Story Spark - Migration Story
Spark - Migration Story
 
Reducing Pager Fatigue Using a Serverless ML Bot
Reducing Pager Fatigue Using a Serverless ML BotReducing Pager Fatigue Using a Serverless ML Bot
Reducing Pager Fatigue Using a Serverless ML Bot
 
Flink sql for continuous sql etl apps & Apache NiFi devops
Flink sql for continuous sql etl apps & Apache NiFi devopsFlink sql for continuous sql etl apps & Apache NiFi devops
Flink sql for continuous sql etl apps & Apache NiFi devops
 
Big Data Analytics in R using sparklyr
Big Data Analytics in R using sparklyrBig Data Analytics in R using sparklyr
Big Data Analytics in R using sparklyr
 
Facilitating Possibility: Appreciative Inquiry as a Tool for Content Strategy
Facilitating Possibility: Appreciative Inquiry as a Tool for Content StrategyFacilitating Possibility: Appreciative Inquiry as a Tool for Content Strategy
Facilitating Possibility: Appreciative Inquiry as a Tool for Content Strategy
 
Sydney Apache Spark Meetup - Spark Natural Language Processing
Sydney Apache Spark Meetup - Spark Natural Language ProcessingSydney Apache Spark Meetup - Spark Natural Language Processing
Sydney Apache Spark Meetup - Spark Natural Language Processing
 
5 things one must know about spark!
5 things one must know about spark!5 things one must know about spark!
5 things one must know about spark!
 
SWONtech News for May, 2012
SWONtech News for May, 2012SWONtech News for May, 2012
SWONtech News for May, 2012
 
Chals 2014 dfn
Chals 2014 dfnChals 2014 dfn
Chals 2014 dfn
 
Openbar 2 - Leuven - Faros - Invisible Infrastructure
Openbar 2 - Leuven - Faros - Invisible InfrastructureOpenbar 2 - Leuven - Faros - Invisible Infrastructure
Openbar 2 - Leuven - Faros - Invisible Infrastructure
 
20190220 Oracle Blockchain Day公開資料
20190220 Oracle Blockchain Day公開資料20190220 Oracle Blockchain Day公開資料
20190220 Oracle Blockchain Day公開資料
 
Optimizing your SparkML pipelines using the latest features in Spark 2.3
Optimizing your SparkML pipelines using the latest features in Spark 2.3Optimizing your SparkML pipelines using the latest features in Spark 2.3
Optimizing your SparkML pipelines using the latest features in Spark 2.3
 

Recently uploaded

Audience Researchndfhcvnfgvgbhujhgfv.pptx
Audience Researchndfhcvnfgvgbhujhgfv.pptxAudience Researchndfhcvnfgvgbhujhgfv.pptx
Audience Researchndfhcvnfgvgbhujhgfv.pptx
Stephen266013
 
sourabh vyas1222222222222222222244444444
sourabh vyas1222222222222222222244444444sourabh vyas1222222222222222222244444444
sourabh vyas1222222222222222222244444444
saurabvyas476
 
1:1原版定制伦敦政治经济学院毕业证(LSE毕业证)成绩单学位证书留信学历认证
1:1原版定制伦敦政治经济学院毕业证(LSE毕业证)成绩单学位证书留信学历认证1:1原版定制伦敦政治经济学院毕业证(LSE毕业证)成绩单学位证书留信学历认证
1:1原版定制伦敦政治经济学院毕业证(LSE毕业证)成绩单学位证书留信学历认证
dq9vz1isj
 
NO1 Best Kala Jadu Expert Specialist In Germany Kala Jadu Expert Specialist I...
NO1 Best Kala Jadu Expert Specialist In Germany Kala Jadu Expert Specialist I...NO1 Best Kala Jadu Expert Specialist In Germany Kala Jadu Expert Specialist I...
NO1 Best Kala Jadu Expert Specialist In Germany Kala Jadu Expert Specialist I...
Amil baba
 
obat aborsi Banjarmasin wa 082135199655 jual obat aborsi cytotec asli di Ban...
obat aborsi Banjarmasin wa 082135199655 jual obat aborsi cytotec asli di  Ban...obat aborsi Banjarmasin wa 082135199655 jual obat aborsi cytotec asli di  Ban...
obat aborsi Banjarmasin wa 082135199655 jual obat aborsi cytotec asli di Ban...
siskavia95
 
如何办理(WashU毕业证书)圣路易斯华盛顿大学毕业证成绩单本科硕士学位证留信学历认证
如何办理(WashU毕业证书)圣路易斯华盛顿大学毕业证成绩单本科硕士学位证留信学历认证如何办理(WashU毕业证书)圣路易斯华盛顿大学毕业证成绩单本科硕士学位证留信学历认证
如何办理(WashU毕业证书)圣路易斯华盛顿大学毕业证成绩单本科硕士学位证留信学历认证
acoha1
 
Displacement, Velocity, Acceleration, and Second Derivatives
Displacement, Velocity, Acceleration, and Second DerivativesDisplacement, Velocity, Acceleration, and Second Derivatives
Displacement, Velocity, Acceleration, and Second Derivatives
23050636
 
原件一样(UWO毕业证书)西安大略大学毕业证成绩单留信学历认证
原件一样(UWO毕业证书)西安大略大学毕业证成绩单留信学历认证原件一样(UWO毕业证书)西安大略大学毕业证成绩单留信学历认证
原件一样(UWO毕业证书)西安大略大学毕业证成绩单留信学历认证
pwgnohujw
 

Recently uploaded (20)

Formulas dax para power bI de microsoft.pdf
Formulas dax para power bI de microsoft.pdfFormulas dax para power bI de microsoft.pdf
Formulas dax para power bI de microsoft.pdf
 
Audience Researchndfhcvnfgvgbhujhgfv.pptx
Audience Researchndfhcvnfgvgbhujhgfv.pptxAudience Researchndfhcvnfgvgbhujhgfv.pptx
Audience Researchndfhcvnfgvgbhujhgfv.pptx
 
Identify Customer Segments to Create Customer Offers for Each Segment - Appli...
Identify Customer Segments to Create Customer Offers for Each Segment - Appli...Identify Customer Segments to Create Customer Offers for Each Segment - Appli...
Identify Customer Segments to Create Customer Offers for Each Segment - Appli...
 
Northern New England Tableau User Group (TUG) May 2024
Northern New England Tableau User Group (TUG) May 2024Northern New England Tableau User Group (TUG) May 2024
Northern New England Tableau User Group (TUG) May 2024
 
sourabh vyas1222222222222222222244444444
sourabh vyas1222222222222222222244444444sourabh vyas1222222222222222222244444444
sourabh vyas1222222222222222222244444444
 
Aggregations - The Elasticsearch "GROUP BY"
Aggregations - The Elasticsearch "GROUP BY"Aggregations - The Elasticsearch "GROUP BY"
Aggregations - The Elasticsearch "GROUP BY"
 
1:1原版定制伦敦政治经济学院毕业证(LSE毕业证)成绩单学位证书留信学历认证
1:1原版定制伦敦政治经济学院毕业证(LSE毕业证)成绩单学位证书留信学历认证1:1原版定制伦敦政治经济学院毕业证(LSE毕业证)成绩单学位证书留信学历认证
1:1原版定制伦敦政治经济学院毕业证(LSE毕业证)成绩单学位证书留信学历认证
 
Genuine love spell caster )! ,+27834335081) Ex lover back permanently in At...
Genuine love spell caster )! ,+27834335081)   Ex lover back permanently in At...Genuine love spell caster )! ,+27834335081)   Ex lover back permanently in At...
Genuine love spell caster )! ,+27834335081) Ex lover back permanently in At...
 
NO1 Best Kala Jadu Expert Specialist In Germany Kala Jadu Expert Specialist I...
NO1 Best Kala Jadu Expert Specialist In Germany Kala Jadu Expert Specialist I...NO1 Best Kala Jadu Expert Specialist In Germany Kala Jadu Expert Specialist I...
NO1 Best Kala Jadu Expert Specialist In Germany Kala Jadu Expert Specialist I...
 
How to Transform Clinical Trial Management with Advanced Data Analytics
How to Transform Clinical Trial Management with Advanced Data AnalyticsHow to Transform Clinical Trial Management with Advanced Data Analytics
How to Transform Clinical Trial Management with Advanced Data Analytics
 
obat aborsi Banjarmasin wa 082135199655 jual obat aborsi cytotec asli di Ban...
obat aborsi Banjarmasin wa 082135199655 jual obat aborsi cytotec asli di  Ban...obat aborsi Banjarmasin wa 082135199655 jual obat aborsi cytotec asli di  Ban...
obat aborsi Banjarmasin wa 082135199655 jual obat aborsi cytotec asli di Ban...
 
Jual Obat Aborsi Bandung (Asli No.1) Wa 082134680322 Klinik Obat Penggugur Ka...
Jual Obat Aborsi Bandung (Asli No.1) Wa 082134680322 Klinik Obat Penggugur Ka...Jual Obat Aborsi Bandung (Asli No.1) Wa 082134680322 Klinik Obat Penggugur Ka...
Jual Obat Aborsi Bandung (Asli No.1) Wa 082134680322 Klinik Obat Penggugur Ka...
 
Seven tools of quality control.slideshare
Seven tools of quality control.slideshareSeven tools of quality control.slideshare
Seven tools of quality control.slideshare
 
Identify Rules that Predict Patient’s Heart Disease - An Application of Decis...
Identify Rules that Predict Patient’s Heart Disease - An Application of Decis...Identify Rules that Predict Patient’s Heart Disease - An Application of Decis...
Identify Rules that Predict Patient’s Heart Disease - An Application of Decis...
 
如何办理(WashU毕业证书)圣路易斯华盛顿大学毕业证成绩单本科硕士学位证留信学历认证
如何办理(WashU毕业证书)圣路易斯华盛顿大学毕业证成绩单本科硕士学位证留信学历认证如何办理(WashU毕业证书)圣路易斯华盛顿大学毕业证成绩单本科硕士学位证留信学历认证
如何办理(WashU毕业证书)圣路易斯华盛顿大学毕业证成绩单本科硕士学位证留信学历认证
 
Displacement, Velocity, Acceleration, and Second Derivatives
Displacement, Velocity, Acceleration, and Second DerivativesDisplacement, Velocity, Acceleration, and Second Derivatives
Displacement, Velocity, Acceleration, and Second Derivatives
 
社内勉強会資料_Object Recognition as Next Token Prediction
社内勉強会資料_Object Recognition as Next Token Prediction社内勉強会資料_Object Recognition as Next Token Prediction
社内勉強会資料_Object Recognition as Next Token Prediction
 
原件一样(UWO毕业证书)西安大略大学毕业证成绩单留信学历认证
原件一样(UWO毕业证书)西安大略大学毕业证成绩单留信学历认证原件一样(UWO毕业证书)西安大略大学毕业证成绩单留信学历认证
原件一样(UWO毕业证书)西安大略大学毕业证成绩单留信学历认证
 
NOAM AAUG Adobe Summit 2024: Summit Slam Dunks
NOAM AAUG Adobe Summit 2024: Summit Slam DunksNOAM AAUG Adobe Summit 2024: Summit Slam Dunks
NOAM AAUG Adobe Summit 2024: Summit Slam Dunks
 
Credit Card Fraud Detection: Safeguarding Transactions in the Digital Age
Credit Card Fraud Detection: Safeguarding Transactions in the Digital AgeCredit Card Fraud Detection: Safeguarding Transactions in the Digital Age
Credit Card Fraud Detection: Safeguarding Transactions in the Digital Age
 

Shiny Spark (閃亮的火花) (20171227 - Spark.TW 3rd Anniversary Sharing)