Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Lambda Architecture using
Google Cloud plus Apps
collect your log at a large scale scope
Simon Su
Hadoop Conf 2015
https://www.facebook.com/groups/GCPUG.TW/
https://plus.google.com/u/0/communities/116100913832589966421
[您知道”GCPUG”要怎麼唸嗎?為...
var simon = {/** I am at GCPUG.TW **/};
simon.aboutme = 'http://about.me/peihsinsu';
simon.nodejs = ‘http://opennodes.arec...
今天講的不是這個ㄛ….
Lambda Architecture是一個大資料分析的理想模式~
即時 完整
● 存得起來的,叫做Storage (儲存)
● 看得到的,叫做Data (資料)
● 看得懂的,叫做Information (資訊)
● 用得出來的,才能夠叫做Intelligent (智慧)
您需要大資料?!
和沛科技 CEO & Foun...
Pain point?
● 沒有資料@@
→ 蒐集資料的工程太複雜
● 真的要存...
→ 資料太大、太多
● 存好了,查太難@@...
→ 分析工具學習門檻高
● 查出來,結果不是要的 ...
→ 較難回饋查詢
● 終於是我要的...
→ 不...
告訴你,在Cloud上可以輕鬆收集分析大資料...
大資料蒐集的資料傳輸
<source>
type forward
port 24224
tag mytag
</source>
<match mytag.**>
type forward
<server>
host 192.168.0.1
port 24224
</...
大資料蒐集的分析工具
select top(title), count(*)
from publicdata:samples.wikipedia
Scanning 1 TB in 1 sec
takes 5,000 disks
select
repository.language as lang,
count(*) as total
from publicdata:samples.github_nested
where
repository.language is n...
BigQuery - User Defined Function (UDF)
大資料的呈現工具
免錢最好~
大資料的分析架構 - Lambda Architecture
Cloud
Logging
fluentd / client
(out:cloud logging)
streaming insert
(hot data flow)
bq load (cold data flow)
batch
query
S...
準備HTTP Data Endpoint
gcloud compute --project "m-plaza" instances create "instance-1" 
--zone "asia-east1-b" 
--machine-ty...
<match **>
type copy
<store>
type https_json
use_https true
buffer_path /tmp/buffer
buffer_chunk_limit 256m
buffer_queue_l...
準備一個Report Server
相關設定可以參考: https://github.com/GoogleCloudPlatform/lambda-dashboard
while [ true ]
do
curl -X POST 
-d "json={"metric":"cpu","usage":`w | grep load | awk -F',' '{print $3}' | awk
'{print $3}...
查看我們的Dashboard
有一天... 如果資料再長大....
Google還有哪些雲端服務可以幫上忙?
Cloud Monitor Cloud LoggingCloud Storage Cloud DataflowCloud Pub/Sub
Cloud
Logging
fluentd / client
(out:cloud
logging)
fluentd / client
(out:secure
forward)
direct import
batch
import
soluti...
Custom Metrics
https://github.com/gcpug-tw/gcp-demo/blob/master/demo-cloudmonitor/stock-monitor.md
<match **>
type exec
command /opt/node/bin/node /path/to/your/google-metrics-loader.js
format json
time_format %Y-%m-%d %H...
考慮自動擴展
Query Engine
Q&A
MiTAC - Cloud Team 徵才
FrontEnd engineer
● Skills: HTML (must), Node.js (must) Javascript (must)
Python (better)
● Experien...
Qnap - Cloud Solution Team 徵才
FrontEnd engineer
● Skills: HTML, Javascript, Node.js or Python, Love to
Learn!
● Experience...
GCPUG.TW Call for Speaker
● 凡對Google Cloud上的應用有使用心得想要分享、想要吐槽、想要
尋求大家幫忙解決的,歡迎來 GCPUG.TW給個Session,跟大家分享
一下在使用Google Cloud Pl...
Lambda Architecture using Google Cloud plus Apps
Lambda Architecture using Google Cloud plus Apps
Lambda Architecture using Google Cloud plus Apps
Lambda Architecture using Google Cloud plus Apps
Upcoming SlideShare
Loading in …5
×

Lambda Architecture using Google Cloud plus Apps

1,170 views

Published on

This is a demo that use apps script to demo the lambda dashboard. The apps script publish a endpoint and client using fluentd to post data to apps script and also bigquery. Then, you can see the realtime and batch query in the same view.

Published in: Technology
  • Be the first to comment

Lambda Architecture using Google Cloud plus Apps

  1. 1. Lambda Architecture using Google Cloud plus Apps collect your log at a large scale scope Simon Su Hadoop Conf 2015
  2. 2. https://www.facebook.com/groups/GCPUG.TW/ https://plus.google.com/u/0/communities/116100913832589966421 [您知道”GCPUG”要怎麼唸嗎?為什麼會有一隻狗在 Logo裡面呢?] Google Cloud Platform User Group的縮寫是GCPUG GCPUG直接唸成G.C.P.U.G?當然可以! 但它也可以分開來,唸成 G.C. PUG喔~ Pug,指的是巴哥犬,所以 GCPUG的Logo中間才會有一隻可愛的巴哥犬喲。 下次聽到別人說G.C. PUG 的時候,您就可以大聲 說:「我也是G.C. PUG社團成員!」
  3. 3. var simon = {/** I am at GCPUG.TW **/}; simon.aboutme = 'http://about.me/peihsinsu'; simon.nodejs = ‘http://opennodes.arecord.us'; simon.googleshare = 'http://gappsnews.blogspot.tw' simon.nodejsblog = ‘http://nodejs-in-example.blogspot.tw'; simon.blog = ‘http://peihsinsu.blogspot.com'; simon.slideshare = ‘http://slideshare.net/peihsinsu/'; simon.email = ‘simonsu.mail@gmail.com’; simon.say(‘Good luck to everybody!'); 哇細項~
  4. 4. 今天講的不是這個ㄛ….
  5. 5. Lambda Architecture是一個大資料分析的理想模式~ 即時 完整
  6. 6. ● 存得起來的,叫做Storage (儲存) ● 看得到的,叫做Data (資料) ● 看得懂的,叫做Information (資訊) ● 用得出來的,才能夠叫做Intelligent (智慧) 您需要大資料?! 和沛科技 CEO & Founder - 翟本喬
  7. 7. Pain point? ● 沒有資料@@ → 蒐集資料的工程太複雜 ● 真的要存... → 資料太大、太多 ● 存好了,查太難@@... → 分析工具學習門檻高 ● 查出來,結果不是要的 ... → 較難回饋查詢 ● 終於是我要的... → 不過,過了市場需求時間
  8. 8. 告訴你,在Cloud上可以輕鬆收集分析大資料...
  9. 9. 大資料蒐集的資料傳輸
  10. 10. <source> type forward port 24224 tag mytag </source> <match mytag.**> type forward <server> host 192.168.0.1 port 24224 </server> </match> <filter mytag.access> type record_transformer <record> host_param "#{Socket.gethostname}" </record> </filter> Input -> filter 1 -> ... -> filter N -> Output
  11. 11. 大資料蒐集的分析工具 select top(title), count(*) from publicdata:samples.wikipedia Scanning 1 TB in 1 sec takes 5,000 disks
  12. 12. select repository.language as lang, count(*) as total from publicdata:samples.github_nested where repository.language is not null group by lang order by total desc limit 100 BigQuery所支援友善的查詢方式
  13. 13. BigQuery - User Defined Function (UDF)
  14. 14. 大資料的呈現工具
  15. 15. 免錢最好~
  16. 16. 大資料的分析架構 - Lambda Architecture
  17. 17. Cloud Logging fluentd / client (out:cloud logging) streaming insert (hot data flow) bq load (cold data flow) batch query Syslog Apache Log SNMP HTTP(s)... http / json post
  18. 18. 準備HTTP Data Endpoint gcloud compute --project "m-plaza" instances create "instance-1" --zone "asia-east1-b" --machine-type "n1-standard-1" --network "default" --metadata "startup-script-url=https://dl.google.com/cloudagents/install-logging-agent.sh" --image "https://www.googleapis.com/compute/v1/projects/debian-cloud/global/images/backports- debian-7-wheezy-v20150915" --boot-disk-size "10" --boot-disk-type "pd-standard" --boot-disk-device-name "instance-1"
  19. 19. <match **> type copy <store> type https_json use_https true buffer_path /tmp/buffer buffer_chunk_limit 256m buffer_queue_limit 128 flush_interval 3s endpoint https://script.google.com/macros/s/AK...Vyl/exec </store> <store> type google_cloud buffer_chunk_limit 512K flush_interval 5s max_retry_wait 300 disable_retry_limit </store> </match> 準備你的Fluentd Config <source> type http port 9880 </source> 啟動http listener 啟動http json輸出、 cloud loggin輸出
  20. 20. 準備一個Report Server 相關設定可以參考: https://github.com/GoogleCloudPlatform/lambda-dashboard
  21. 21. while [ true ] do curl -X POST -d "json={"metric":"cpu","usage":`w | grep load | awk -F',' '{print $3}' | awk '{print $3}'`}" http://localhost:9880/test5.tag.cpu sleep 3; done 測試
  22. 22. 查看我們的Dashboard
  23. 23. 有一天... 如果資料再長大....
  24. 24. Google還有哪些雲端服務可以幫上忙? Cloud Monitor Cloud LoggingCloud Storage Cloud DataflowCloud Pub/Sub
  25. 25. Cloud Logging fluentd / client (out:cloud logging) fluentd / client (out:secure forward) direct import batch import solution 1: for lightware data solution 2: for big data batch query solution 3: for realtime dashboard Query Engine - Unknow format log data - Archive purpose google-fluentd gateway (in:secure forward) Custom Metrics Plugin - Well known format log data - Detail report use - Frequency query Google Cloud Platform On-premise Other Implemented Client - Customized program (mobile, web, IoT...)
  26. 26. Custom Metrics https://github.com/gcpug-tw/gcp-demo/blob/master/demo-cloudmonitor/stock-monitor.md
  27. 27. <match **> type exec command /opt/node/bin/node /path/to/your/google-metrics-loader.js format json time_format %Y-%m-%d %H:%M:%S flush_interval 5s buffer_path /tmp/ </match> Sample config metric.writeTimeSeries( project_id, metricid, value, {"custom.cloudmonitoring.googleapis.com/cpu" : cpuUsageValue}, null, null, function(err, req, doc){ if(err) console.log('ERROR:', err); console.log(doc); });
  28. 28. 考慮自動擴展
  29. 29. Query Engine
  30. 30. Q&A
  31. 31. MiTAC - Cloud Team 徵才 FrontEnd engineer ● Skills: HTML (must), Node.js (must) Javascript (must) Python (better) ● Experience: 3 years related experience Backend engineer ● Skills: Shell script (must), Node.js or Python, RESTful API build up experience, Google Cloud Platform user. ● Experience: 1-3 years related experience 工商服務時間 請聯絡:cklouie@mitac.com.tw
  32. 32. Qnap - Cloud Solution Team 徵才 FrontEnd engineer ● Skills: HTML, Javascript, Node.js or Python, Love to Learn! ● Experience: 2 years related experience or more Backend engineer ● Skills: Shell script (must), Node.js or Python, RESTful API build up experience, Storage or Network knowledge. Love to Learn! ● Experience: 1-3 years related experience or more Billing engineer ● Skills: Billing system related knowledge, Java or Python, ● Experience: 2-5 years related experience or more 工商服務時間 請聯絡:peihsinsu@qnap.com
  33. 33. GCPUG.TW Call for Speaker ● 凡對Google Cloud上的應用有使用心得想要分享、想要吐槽、想要 尋求大家幫忙解決的,歡迎來 GCPUG.TW給個Session,跟大家分享 一下在使用Google Cloud Platform的點點滴滴。 ● 現場報名,直接送您 T-shirt! 工商服務時間 請聯絡:simonsu.mail@gmail.com

×