● 專案開通:https://sites.google.com/a/mitac.com.tw/google-cloud-platform/getting-start
● Billing Enable:https://sites.google.c...
Simon @ MiCloud
We are….
http://opennodes.arecord.us
今天講的是.... Google BigQuery
今天用的是.... Node.js
今天要做的是...
Tools
Req?
Web
App
Service
時代不同,面對資料的態度也不同...
翟本喬:
● 存得起來的,叫做Storage (儲存)
● 看得到的,叫做Data (資料)
● 看得懂的,叫做Information (資訊)
● 用得出來的,才能夠叫做Intelligent (智慧)
沒有不該儲存的資料...
● POS data
● Log data
● User behavior
● Data Warehouse
● Transactions
例如: 看似無用的Log...
WHO DO WHATWHEN Questions...
所以... 我們需要的是
● 一個存取的方式
● 越簡單越好...
● 輕量...
● 快速...
天下武功,唯快不破...
Tools
Auth
Req?
Web
App
Service
Authenticate with Oauth2.0
$ npm search oauth
google-api-utility模組開發歷程
● 透過初始化設定之後,即可
以直接進行api呼叫動作
○ 設定檔抽離
○ 結合request模組進行api調用
Google Service Account
web server application
service account
v.s.
Service Owner在Google的設定
Prepare Authentications
$ openssl pkcs12 -in privatekey.p12 -out privatekey.pem -nocerts
$ openssl rsa -in privatekey.pem -out key.pem
Generate ke...
google-api-utility module
基本資訊
● https://github.com/peihsinsu/google-api-utility
安裝
● npm install google-api-utility
操作
● ...
var auth = require('google-api-utility')
auth.init({
scope: 'https://www.googleapis.com/auth/bigquery https://www.googleap...
使用範例 - 呼叫BigQuery
var request = auth.request;
var bqurl = 'https://www.googleapis.com/bigquery/v2/projects/%s/datasets';
r...
Tools
Auth
Req?
API
Web
App
Service
Google API Explore https://developers.google.com/apis-explorer/
Google API Explore - Query
Google API Explore - Auth
Operation Scope
Google API Explore - Response
Tools
Auth
Req?
API
SDK Web
App
Service
Idea...
● bigquery.init({...configurations...})
● bigquery.dataset.list(....)
● bigquery.table.load(..., callback)
bigquery module
基本資訊
● https://github.com/peihsinsu/bigquery
安裝
● npm install bigquery
操作
● bigquery.init(config)
● bigque...
重新包裝 - bigquery模組
var bq = require('bigquery')
, prjId = 'your-bigquery-project-id';
bq.init({
client_secret: '/path/to/cl...
Source Code...
var util = require('util')
, auth = require('google-api-utility')
, request = auth.request
, _ = require('u...
An interest demo...
Google provided node.js tool
Operation with googleapis
var googleapis = require('googleapis');
var jwt = new googleapis.auth.JWT(
'429100748......hv3@d...
Operation with googleapis
jwt.authorize(function(err, tokens) {
googleapis.discover('bigquery', 'v2').execute(function(e,c...
Full Code
http://goo.gl/LD4RN4
Big query meet node.js
Upcoming SlideShare
Loading in …5
×

Big query meet node.js

577 views

Published on

0 Comments
4 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
577
On SlideShare
0
From Embeds
0
Number of Embeds
17
Actions
Shares
0
Downloads
6
Comments
0
Likes
4
Embeds 0
No embeds

No notes for slide

Big query meet node.js

  1. 1. ● 專案開通:https://sites.google.com/a/mitac.com.tw/google-cloud-platform/getting-start ● Billing Enable:https://sites.google.com/a/mitac.com.tw/google-cloud-platform/getting- start/enable-billing ● 安裝Google Cloud SDK: ○ Windows: https://sites.google.com/a/mitac.com.tw/google-cloud-platform/getting- start/windows-install-google-cloud-sdk ○ Others: https://developers.google.com/cloud/sdk/ ● 18 : 45 - 19 : 00 報到 ● 19 : 00 - 19 : 30 當 BigQuery 碰上 Node.js ● 19 : 30 - 19 : 40 休息 ● 19 : 40 - 21 : 00 BigQuery 資料上傳實作 準備... 基地台名稱 - MiTAC Guest 連線帳號 - gcp 連線密碼 - mitacmitac http://goo.gl/32E4LZ
  2. 2. Simon @ MiCloud
  3. 3. We are….
  4. 4. http://opennodes.arecord.us
  5. 5. 今天講的是.... Google BigQuery
  6. 6. 今天用的是.... Node.js
  7. 7. 今天要做的是...
  8. 8. Tools Req? Web App Service
  9. 9. 時代不同,面對資料的態度也不同... 翟本喬: ● 存得起來的,叫做Storage (儲存) ● 看得到的,叫做Data (資料) ● 看得懂的,叫做Information (資訊) ● 用得出來的,才能夠叫做Intelligent (智慧)
  10. 10. 沒有不該儲存的資料... ● POS data ● Log data ● User behavior ● Data Warehouse ● Transactions
  11. 11. 例如: 看似無用的Log... WHO DO WHATWHEN Questions...
  12. 12. 所以... 我們需要的是 ● 一個存取的方式 ● 越簡單越好... ● 輕量... ● 快速... 天下武功,唯快不破...
  13. 13. Tools Auth Req? Web App Service
  14. 14. Authenticate with Oauth2.0
  15. 15. $ npm search oauth
  16. 16. google-api-utility模組開發歷程 ● 透過初始化設定之後,即可 以直接進行api呼叫動作 ○ 設定檔抽離 ○ 結合request模組進行api調用
  17. 17. Google Service Account web server application service account v.s.
  18. 18. Service Owner在Google的設定
  19. 19. Prepare Authentications
  20. 20. $ openssl pkcs12 -in privatekey.p12 -out privatekey.pem -nocerts $ openssl rsa -in privatekey.pem -out key.pem Generate key.pem $ openssl pkcs12 -in privatekey.p12 -nodes -nocerts > key.pem or
  21. 21. google-api-utility module 基本資訊 ● https://github.com/peihsinsu/google-api-utility 安裝 ● npm install google-api-utility 操作 ● apiutil.init(config) ● apiutil.request(options, callback)
  22. 22. var auth = require('google-api-utility') auth.init({ scope: 'https://www.googleapis.com/auth/bigquery https://www.googleapis. com/auth/cloud-platform', client_secret: '/path-to-client_secret.json', key_pem: '/path-to-key.pem' }); 使用範例 - 初始化 此處需要綁定所欲呼叫的 API相關授 權之Scope位置 設定client_secret.json與相關pem檔 案位置,供jwt運算使用
  23. 23. 使用範例 - 呼叫BigQuery var request = auth.request; var bqurl = 'https://www.googleapis.com/bigquery/v2/projects/%s/datasets'; request({ url: util.format(bqurl, project), method: 'GET' }, function(err, req, doc){ // implements }); 結合原request模組之function,供api呼叫使用 同原request模組操作方式
  24. 24. Tools Auth Req? API Web App Service
  25. 25. Google API Explore https://developers.google.com/apis-explorer/
  26. 26. Google API Explore - Query
  27. 27. Google API Explore - Auth Operation Scope
  28. 28. Google API Explore - Response
  29. 29. Tools Auth Req? API SDK Web App Service
  30. 30. Idea... ● bigquery.init({...configurations...}) ● bigquery.dataset.list(....) ● bigquery.table.load(..., callback)
  31. 31. bigquery module 基本資訊 ● https://github.com/peihsinsu/bigquery 安裝 ● npm install bigquery 操作 ● bigquery.init(config) ● bigquery.[category].[operation](options, callback)
  32. 32. 重新包裝 - bigquery模組 var bq = require('bigquery') , prjId = 'your-bigquery-project-id'; bq.init({ client_secret: '/path/to/client_secret.json', key_pem: '/path-to-key.pem' }); bq.dataset.list(prjId, function(e,r,d){ if(e) console.log(e); console.log(JSON.stringify(d)); }); 操作時,透過bq呼叫job之下的function做操作 bigquery模組可參考:https://github.com/peihsinsu/bigquery
  33. 33. Source Code... var util = require('util') , auth = require('google-api-utility') , request = auth.request , _ = require('underscore') exports.init = auth.init; exports.job = { token: '', listds : function(project, cb){ var bqurl = 'https://www.googleapis.com/bigquery/v2/projects/%s/datasets'; request({ url: util.format(bqurl, project), method: 'GET' }, cb?cb:auth.commonCb); }, … (skip) } 封裝相同類別的api在一起,ex: job相關的放在job物件中
  34. 34. An interest demo...
  35. 35. Google provided node.js tool
  36. 36. Operation with googleapis var googleapis = require('googleapis'); var jwt = new googleapis.auth.JWT( '429100748......hv3@developer.gserviceaccount.com', '/path/to/key.pem', null, [ 'https://www.googleapis.com/auth/bigquery', 'https://www.googleapis.com/auth/cloud-platform' ]);
  37. 37. Operation with googleapis jwt.authorize(function(err, tokens) { googleapis.discover('bigquery', 'v2').execute(function(e,client) { if(e) console.log(e); else client.bigquery.datasets.list(param).withAuthClient(jwt).execute(function(err, response) { if(err) console.log(err); console.log(JSON.stringify(response)); }); }); });
  38. 38. Full Code
  39. 39. http://goo.gl/LD4RN4

×