1.
Lambda Architecture using
Google Cloud plus Apps
collect your log at a large scale scope
Simon Su
Hadoop Conf 2015

2.
https://www.facebook.com/groups/GCPUG.TW/
https://plus.google.com/u/0/communities/116100913832589966421
[您知道”GCPUG”要怎麼唸嗎？為什麼會有一隻狗在 Logo裡面呢？]
Google Cloud Platform User Group的縮寫是GCPUG
GCPUG直接唸成G.C.P.U.G？當然可以！
但它也可以分開來，唸成 G.C. PUG喔～
Pug，指的是巴哥犬，所以 GCPUG的Logo中間才會有一隻可愛的巴哥犬喲。
下次聽到別人說G.C. PUG 的時候，您就可以大聲 說：「我也是G.C. PUG社團成員！」

3.
var simon = {/** I am at GCPUG.TW **/};
simon.aboutme = 'http://about.me/peihsinsu';
simon.nodejs = ‘http://opennodes.arecord.us';
simon.googleshare = 'http://gappsnews.blogspot.tw'
simon.nodejsblog = ‘http://nodejs-in-example.blogspot.tw';
simon.blog = ‘http://peihsinsu.blogspot.com';
simon.slideshare = ‘http://slideshare.net/peihsinsu/';
simon.email = ‘simonsu.mail@gmail.com’;
simon.say(‘Good luck to everybody!');
哇細項～

4.
今天講的不是這個ㄛ….

5.
Lambda Architecture是一個大資料分析的理想模式～
即時 完整

6.
● 存得起來的,叫做Storage (儲存)
● 看得到的,叫做Data (資料)
● 看得懂的,叫做Information (資訊)
● 用得出來的,才能夠叫做Intelligent (智慧)
您需要大資料？！
和沛科技 CEO & Founder - 翟本喬

7.
Pain point?
● 沒有資料＠＠
→ 蒐集資料的工程太複雜
● 真的要存...
→ 資料太大、太多
● 存好了，查太難＠＠...
→ 分析工具學習門檻高
● 查出來，結果不是要的 ...
→ 較難回饋查詢
● 終於是我要的...
→ 不過，過了市場需求時間

8.
告訴你，在Cloud上可以輕鬆收集分析大資料...

9.
大資料蒐集的資料傳輸

10.
<source>
type forward
port 24224
tag mytag
</source>
<match mytag.**>
type forward
<server>
host 192.168.0.1
port 24224
</server>
</match>
<filter mytag.access>
type record_transformer
<record>
host_param "#{Socket.gethostname}"
</record>
</filter>
Input -> filter 1 -> ... -> filter N -> Output

11.
大資料蒐集的分析工具
select top(title), count(*)
from publicdata:samples.wikipedia
Scanning 1 TB in 1 sec
takes 5,000 disks

12.
select
repository.language as lang,
count(*) as total
from publicdata:samples.github_nested
where
repository.language is not null
group by lang
order by total desc limit 100
BigQuery所支援友善的查詢方式

13.
BigQuery - User Defined Function (UDF)

14.
大資料的呈現工具

15.
免錢最好～

16.
大資料的分析架構 - Lambda Architecture

17.
Cloud
Logging
fluentd / client
(out:cloud logging)
streaming insert
(hot data flow)
bq load (cold data flow)
batch
query
Syslog Apache Log SNMP HTTP(s)...
http / json
post

18.
準備HTTP Data Endpoint
gcloud compute --project "m-plaza" instances create "instance-1"
--zone "asia-east1-b"
--machine-type "n1-standard-1"
--network "default"
--metadata "startup-script-url=https://dl.google.com/cloudagents/install-logging-agent.sh"
--image "https://www.googleapis.com/compute/v1/projects/debian-cloud/global/images/backports-
debian-7-wheezy-v20150915"
--boot-disk-size "10"
--boot-disk-type "pd-standard"
--boot-disk-device-name "instance-1"

19.
<match **>
type copy
<store>
type https_json
use_https true
buffer_path /tmp/buffer
buffer_chunk_limit 256m
buffer_queue_limit 128
flush_interval 3s
endpoint https://script.google.com/macros/s/AK...Vyl/exec
</store>
<store>
type google_cloud
buffer_chunk_limit 512K
flush_interval 5s
max_retry_wait 300
disable_retry_limit
</store>
</match>
準備你的Fluentd Config
<source>
type http
port 9880
</source>
啟動http listener
啟動http json輸出、
cloud loggin輸出

20.
準備一個Report Server
相關設定可以參考： https://github.com/GoogleCloudPlatform/lambda-dashboard

21.
while [ true ]
do
curl -X POST
-d "json={"metric":"cpu","usage":`w | grep load | awk -F',' '{print $3}' | awk
'{print $3}'`}"
http://localhost:9880/test5.tag.cpu
sleep 3;
done
測試

22.
查看我們的Dashboard

23.
有一天... 如果資料再長大....

24.
Google還有哪些雲端服務可以幫上忙？
Cloud Monitor Cloud LoggingCloud Storage Cloud DataflowCloud Pub/Sub

25.
Cloud
Logging
fluentd / client
(out:cloud
logging)
fluentd / client
(out:secure
forward)
direct import
batch
import
solution 1: for lightware data solution 2: for big data batch query solution 3: for realtime dashboard
Query
Engine
- Unknow format log
data
- Archive purpose
google-fluentd
gateway
(in:secure forward)
Custom Metrics
Plugin
- Well known format
log data
- Detail report use
- Frequency query
Google Cloud Platform
On-premise
Other
Implemented
Client
- Customized program
(mobile, web, IoT...)

26.
Custom Metrics
https://github.com/gcpug-tw/gcp-demo/blob/master/demo-cloudmonitor/stock-monitor.md

27.
<match **>
type exec
command /opt/node/bin/node /path/to/your/google-metrics-loader.js
format json
time_format %Y-%m-%d %H:%M:%S
flush_interval 5s
buffer_path /tmp/
</match>
Sample config
metric.writeTimeSeries(
project_id,
metricid,
value,
{"custom.cloudmonitoring.googleapis.com/cpu" : cpuUsageValue},
null,
null,
function(err, req, doc){
if(err) console.log('ERROR:', err);
console.log(doc);
});

28.
考慮自動擴展

29.
Query Engine

30.
Q&A

31.
MiTAC - Cloud Team 徵才
FrontEnd engineer
● Skills: HTML (must), Node.js (must) Javascript (must)
Python (better)
● Experience: 3 years related experience
Backend engineer
● Skills: Shell script (must), Node.js or Python, RESTful
API build up experience, Google Cloud Platform user.
● Experience: 1-3 years related experience
工商服務時間
請聯絡：cklouie@mitac.com.tw

32.
Qnap - Cloud Solution Team 徵才
FrontEnd engineer
● Skills: HTML, Javascript, Node.js or Python, Love to
Learn!
● Experience: 2 years related experience or more
Backend engineer
● Skills: Shell script (must), Node.js or Python, RESTful
API build up experience, Storage or Network knowledge.
Love to Learn!
● Experience: 1-3 years related experience or more
Billing engineer
● Skills: Billing system related knowledge, Java or Python,
● Experience: 2-5 years related experience or more
工商服務時間
請聯絡：peihsinsu@qnap.com

33.
GCPUG.TW Call for Speaker
● 凡對Google Cloud上的應用有使用心得想要分享、想要吐槽、想要
尋求大家幫忙解決的，歡迎來 GCPUG.TW給個Session，跟大家分享
一下在使用Google Cloud Platform的點點滴滴。
● 現場報名，直接送您 T-shirt!
工商服務時間
請聯絡：simonsu.mail@gmail.com