This document discusses using Google Cloud services like BigQuery and Auth to access and analyze data. It introduces BigQuery and describes how to authenticate using OAuth 2.0 and service accounts. It then shows how to simplify access using Node.js modules that wrap Google APIs and provide a more intuitive interface for querying and loading data from BigQuery. Code samples are provided for initializing authentication, making API requests, and interacting with BigQuery datasets and jobs. The document emphasizes building tools that are lightweight, fast, and easy to use for working with data and APIs.

1. Simon @ MiCloud

2. We are...

3. 前情提要
很久很久以前...

4. 時代不同，⾯面對資料的態度也不同...
● 存得起來的，叫做Storage (儲存)!
● 看得到的，叫做Data (資料)!
● 看得懂的，叫做Information (資訊)!
● ⽤用得出來的，才能夠叫做Intelligent
(智慧)
和沛科技 CEO & Founder - 翟本喬

5. 資料...就是應該被儲存...
● POS data
● Log data
● User behavior
● Data Warehouse
● Transactions

6. 貌似正常的⼈人

7. 看似無意義的log

8. 接下來，我們要開幹了！

9. Tools
Req?
Web
App
Service

10. 絕世武功秘笈…

12. 我們需要的是
● ⼀一個存取的⽅方式
● 越簡單越好...
● 輕量...
● 快速...
天下武功，唯快不破...

13. Google BigQuery - 秘笈

14. Node.js - 武功
select repository language from
372211 259329 223470

16. 合體！！！
絕世武功秘笈

17. But...⼈人⽣生最厲害就是這個BUT！
Hello Google! Login Required :)
WTF… =_=

18. Node.js & Oauth & Google
Hello Google! What do u want =_=
Authenticate with
Oauth2.0

19. Tools
Auth
Req?
Web
App
Service

20. $ npm search oauth

21. google-api-utility模組開發歷程
透過初始化設定之後，
即可以直接進⾏行api呼叫動作
!
1. 設定檔抽離
2. 結合request模組進⾏行API呼叫

22. google-api-utility module
基本資訊
● https://github.com/peihsinsu/google-api-utility
安裝
● npm install google-api-utility
操作
● apiutil.init(config)
● apiutil.request(options, callback)

23. Google Oauth2
Web Server Application Service Account
Request token
Authorization
code
Exchange code
for token
Token response
Use token to
call Google API
Your App Google
Servers
User login !
& consent
Use
r
Create and
sign JWT
Use JWT to
request token
Token response
Use token to
call Google API
Server
App
Google
Servers

24. Service Account
Configures
Authenticate can be cache
and auto refresh
Others… Request…
Request… Request…
Create and
sign JWT
Use JWT to
request token
Token response
Use token to
call Google API
Server
App
Google
Servers

25. 免不了的⿇麻煩....

26. Service Owner在Google的設定

27. Prepare Authentications

28. $ openssl pkcs12 -in privatekey.p12 -out privatekey.pem -nocerts
$ openssl rsa -in privatekey.pem -out key.pem
Generate key.pem
$ openssl pkcs12 -in privatekey.p12 -nodes -nocerts > key.pem
OR

29. var auth = require('google-api-utility')
auth.init({
scope: 'https://www.googleapis.com/auth/bigquery https://
www.googleapis.com/auth/cloud-platform',
client_secret: '/path-to-client_secret.json',
key_pem: '/path-to-key.pem'
});
使⽤用範例 - 初始化
此處需要綁定所欲呼叫的API
相關授權之Scope位置
設定client_secret.json
與相關pem檔案位置，供jwt
運算使⽤用

30. 使⽤用範例 - 呼叫BigQuery
var request = auth.request;
var bqurl = 'https://www.googleapis.com/bigquery/v2/projects/%s/
datasets';request({
url: util.format(bqurl, project),
method: 'GET'
}, function(err, req, doc){
// implements
});
同原request模組操作⽅方式
結合原request模組之
function，供api呼叫使⽤用

31. Tools
Auth
Req?
API
Web
App
Service

32. Google API Explore HTTPS://developers.google.com/apis-explorer/

33. Google API Explore - Query

34. Google API Explore - Auth
Operation Scope

35. Google API Explore - Response

36. Tools
Auth
Req?
API
SDK Web
App
Service

37. Idea...
● bigquery.init({...configurations...})
● bigquery.[what].[do](...)
- bigquery.dataset.list(....)
- bigquery.table.load(..., callback)

38. bigquery module
基本資訊
● https://github.com/peihsinsu/bigquery
安裝
● npm install bigquery
操作
● bigquery.init(config)
● bigquery.[category].[operation](options, callback)

39. 重新包裝 - bigquery模組
var bq = require('bigquery')
, prjId = 'your-bigquery-project-id';
bq.init({
client_secret: '/path/to/client_secret.json',
key_pem: '/path-to-key.pem'
});
bq.dataset.list(prjId, function(e,r,d){
if(e) console.log(e);
console.log(JSON.stringify(d));
});
bigquery模組可參考：https://github.com/peihsinsu/bigquery
透過bq呼叫某個操作之下的function

40. Source Code...
var util = require('util')
, auth = require('google-api-utility')
, request = auth.request
, _ = require('underscore')
exports.init = auth.init;
exports.job = {
token: '',
listds : function(project, cb){
var bqurl = 'https://www.googleapis.com/bigquery/v2/projects/%s/datasets';
request({
url: util.format(bqurl, project),
method: 'GET'
}, cb?cb:auth.commonCb);
}, … (skip)
}
封裝相同類別的api在⼀一起!
ex: job相關的放在job物件中

41. NOT ONLY Node.js Love JavaScript…..

43. The fancy integrate with Sheet

44. More and more chart….

45. Charting in Apps Script

47. Google provided node.js tool

48. Operation with googleapis
var googleapis = require('googleapis');
!
var jwt = new googleapis.auth.JWT(
'429100748......hv3@developer.gserviceaccount.com',
'/path/to/key.pem',
null,
[
'https://www.googleapis.com/auth/bigquery',
'https://www.googleapis.com/auth/cloud-platform'
]);

49. Operation with googleapis
jwt.authorize(function(err, tokens) {
googleapis.discover('bigquery', 'v2').execute(function(e,client) {
if(e)
console.log(e);
else
client.bigquery.datasets.list(param).withAuthClient(jwt)
.execute(
function(err, response) {
if(err) console.log(err);
console.log(JSON.stringify(response));
});
});
});

50. Full Code

51. HTTP://OpenNodes.arecord.us
OR
Sample
Copy
Run!

52. http://micloud.tw
http://about.me/peihsinsu