Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Case Study on AWS JavaScript SDK and DynamoDB

1,491 views

Published on

Case study on how to manipulate AWS DynamoDB as well as IAM / STS with their JavaScript SDK in the Browser.

I keep notes in memo and comments a lot, so please download and read it if you're really interested. It's a powerpoint, and if that's a problem, please let me know. I'll try convert it to PDF or some other open / free formats.

Licensed under CC-BY / MIT (demo project).

Published in: Software, Technology

Case Study on AWS JavaScript SDK and DynamoDB

  1. 1. Cliff Chao-kuan Lu <clifflu@gmail.com> A Case Study:
  2. 2.  以超連結 (hyperlink) 代替引用 (attribution)  引用外部內容均 ◦ 已取得授權,或 ◦ 包含原始連結,並在合理範圍內引用  本文件原創內容以 CC-BY 3.0 釋出 2014/5/19 CC-BY 3.0, Cliff Chao-kuan Lu <clifflu@gmail.com> 2
  3. 3.  Full-stack Web Developer  AWS Solutions Architect  Nerd 2014/5/19 CC-BY 3.0, Cliff Chao-kuan Lu <clifflu@gmail.com> 3 about.me/clifflu
  4. 4.  Full-stack Web Developer  AWS Solutions Architect  Nerd 2014/5/19 CC-BY 3.0, Cliff Chao-kuan Lu <clifflu@gmail.com> 4 about.me/clifflu Level Up !! Professional
  5. 5.  Intro ◦ AWS JS SDK ◦ IAM ◦ DynamoDB  Case Study ◦ Headless Poller 2014/5/19 CC-BY 3.0, Cliff Chao-kuan Lu <clifflu@gmail.com> 5
  6. 6. 2014/5/19 CC-BY 3.0, Cliff Chao-kuan Lu <clifflu@gmail.com> 6
  7. 7. 這名字好長 2014/5/19 CC-BY 3.0, Cliff Chao-kuan Lu <clifflu@gmail.com> 7
  8. 8.  很新 2014/5/19 CC-BY 3.0, Cliff Chao-kuan Lu <clifflu@gmail.com> 8
  9. 9.  官網  起自 AWS SDK for Node.js  2.0 更名並支援 Browser 2014/5/19 CC-BY 3.0, Cliff Chao-kuan Lu <clifflu@gmail.com> 9
  10. 10.  率先支援 ◦ DynamoDB ◦ S3 ◦ SNS ◦ SQS  猛烈 rc 中 2014/5/19 CC-BY 3.0, Cliff Chao-kuan Lu <clifflu@gmail.com> 10
  11. 11. AutoScaling, CloudFormation, CloudFront, Cloudsearch, CloudTrail, CloudWatch, DataPipeline, DirectConnect, DynamoDB, EC2, ElastiCache, ElasticBeanstalk, ElasticTranscoder, ELB, EMR, Glacier, IAM, ImportExport, Kinesis, OpsWorks, RDS, Redshift, Route53, S3, SES, SimpleDB, SNS, SQS, StorageGateway, STS, Support, SWF 2014/5/19 CC-BY 3.0, Cliff Chao-kuan Lu <clifflu@gmail.com> 11
  12. 12. 2014/5/19 CC-BY 3.0, Cliff Chao-kuan Lu <clifflu@gmail.com> 12
  13. 13. 2014/5/19 CC-BY 3.0, Cliff Chao-kuan Lu <clifflu@gmail.com> 13
  14. 14. 2014/5/19 CC-BY 3.0, Cliff Chao-kuan Lu <clifflu@gmail.com> 14 國防布
  15. 15.  SDK for ◦ PHP, Python, Node.js ◦ Java, .NET, ◦ Ruby  AWS SDK for Android & iOS Dec. 8th, 2010 2014/5/19 CC-BY 3.0, Cliff Chao-kuan Lu <clifflu@gmail.com> 15
  16. 16. 2014/5/19 CC-BY 3.0, Cliff Chao-kuan Lu <clifflu@gmail.com> 16
  17. 17.  Browser 直通 AWS ◦ 減低對 API Server 的需求 2014/5/19 CC-BY 3.0, Cliff Chao-kuan Lu <clifflu@gmail.com> 17
  18. 18.  機房、機器、網路、電力很難搞 交給 Amazon Web Services 正好  那 EC2 Instance 呢? 2014/5/19 CC-BY 3.0, Cliff Chao-kuan Lu <clifflu@gmail.com> 18
  19. 19.  EC2 各種麻煩 ◦ Load Balancing ◦ Types ◦ Contracts ◦ AutoScaling  Parameters  Pattern 2014/5/19 CC-BY 3.0, Cliff Chao-kuan Lu <clifflu@gmail.com> 19
  20. 20.  太多 Gotcha  每層服務、各層之間都要考慮  複雜度可能變 M x N 2014/5/19 CC-BY 3.0, Cliff Chao-kuan Lu <clifflu@gmail.com> 20
  21. 21. 2014/5/19 CC-BY 3.0, Cliff Chao-kuan Lu <clifflu@gmail.com> 21 example.com ELB Route53 CF S3 Static Content Shared Env Auto Scaling group AMI AZ 2 Web Servers S1 Secondary S2 Secondary Config AZ 1 Web Servers S1 Primary S2 Primary Config AZ 3 Web Servers Config + Arbitor mongod
  22. 22.  將麻煩留給 AWS ◦ 第三方與服務端授權 ◦ Scaling / HA 2014/5/19 CC-BY 3.0, Cliff Chao-kuan Lu <clifflu@gmail.com> 22
  23. 23.  促成符合 SOA Pattern 之架構 2014/5/19 CC-BY 3.0, Cliff Chao-kuan Lu <clifflu@gmail.com> 23 Web Page HTML CSS JS Authentic ate & Authorize Services 1. Auth Request 3. Authorized Identity 2. Access Token
  24. 24.  容易整合其他服務  IAM  STS: Security Token Service  WIF: Web Identity Federation  DynamoDB, S3, …  自有服務,SOA 嘛 O.o/ 2014/5/19 CC-BY 3.0, Cliff Chao-kuan Lu <clifflu@gmail.com> 24
  25. 25.  減低將 Access / Secret Token 打入 源碼的可能性 2014/5/19 CC-BY 3.0, Cliff Chao-kuan Lu <clifflu@gmail.com> 25
  26. 26.  API  IAM  Services 2014/5/19 CC-BY 3.0, Cliff Chao-kuan Lu <clifflu@gmail.com> 26
  27. 27.  RESTful, SOAP (deprecated)  Dev Tools 是好朋友 2014/5/19 CC-BY 3.0, Cliff Chao-kuan Lu <clifflu@gmail.com> 27
  28. 28.  安全乃第一要務  不然會變成礦工 ˇˇ 2014/5/19 CC-BY 3.0, Cliff Chao-kuan Lu <clifflu@gmail.com> 28
  29. 29. AWS 權限樞紐 2014/5/19 CC-BY 3.0, Cliff Chao-kuan Lu <clifflu@gmail.com> 29
  30. 30.  Root User  Group / IAM User  Roles ◦ AWS SVC ◦ X-Account ◦ IdP  Web  SAML 2014/5/19 CC-BY 3.0, Cliff Chao-kuan Lu <clifflu@gmail.com> 30
  31. 31.  Password  MFA (Multi-factor authentication)  Access / Secret Key Pair  X.509 certificate  3rd Party ◦ SAML ◦ Web Identity Federation 2014/5/19 CC-BY 3.0, Cliff Chao-kuan Lu <clifflu@gmail.com> 31
  32. 32. 2014/5/19 CC-BY 3.0, Cliff Chao-kuan Lu <clifflu@gmail.com> 32
  33. 33. 2014/5/19 CC-BY 3.0, Cliff Chao-kuan Lu <clifflu@gmail.com> 33  Effect: Deny | Allow  Action: ◦ 允許呼叫的 API  Resource: ◦ arn  Principal ◦ 授權端限制  Condition ◦ 其他限制
  34. 34.  for rule in rules: ◦ Explicit Deny -> Deny ◦ Explicit Allow -> Allow ◦ Default Deny 2014/5/19 CC-BY 3.0, Cliff Chao-kuan Lu <clifflu@gmail.com> 34
  35. 35. 2014/5/19 CC-BY 3.0, Cliff Chao-kuan Lu <clifflu@gmail.com> 35  格式 ◦ 冒號分隔 ◦ 首二節固定為 arn:aws ◦ Service ◦ Region ◦ Account ◦ Resource Identifier
  36. 36.  給路人甲的 ◦ Access / Secret Key Pair ◦ Management Console 登入權限 2014/5/19 CC-BY 3.0, Cliff Chao-kuan Lu <clifflu@gmail.com> 36
  37. 37. 2014/5/19 CC-BY 3.0, Cliff Chao-kuan Lu <clifflu@gmail.com> 37 不用怕被關
  38. 38. 2014/5/19 CC-BY 3.0, Cliff Chao-kuan Lu <clifflu@gmail.com> 38 Facebook, Google 可也
  39. 39.  只接受下列驗證機制 ◦ 表三家:Amazon, Facebook, Google ◦ SAML  說明列表  IAM Partners 2014/5/19 CC-BY 3.0, Cliff Chao-kuan Lu <clifflu@gmail.com> 39
  40. 40.  Trust Relationships ◦ Identity Provider ◦ Client ID  Permissions 2014/5/19 CC-BY 3.0, Cliff Chao-kuan Lu <clifflu@gmail.com> 40
  41. 41.  用戶可透過第三方驗證與 IAM:STS, WIF 授予調用 AWS API 之權限 2014/5/19 CC-BY 3.0, Cliff Chao-kuan Lu <clifflu@gmail.com> 41
  42. 42.  WIF Playground  Login with amazon  AWS Documentation ◦ Using IAM ◦ Using STS ◦ SDK for JavaScript 2014/5/19 CC-BY 3.0, Cliff Chao-kuan Lu <clifflu@gmail.com> 42
  43. 43. Managed NoSQL Service 2014/5/19 CC-BY 3.0, Cliff Chao-kuan Lu <clifflu@gmail.com> 43
  44. 44.  Managed NoSQL Service  取代 SimpleDB  三本柱 ◦ Scalable ◦ Available ◦ Fast 2014/5/19 CC-BY 3.0, Cliff Chao-kuan Lu <clifflu@gmail.com> 44
  45. 45.  Scalar ◦ Number {“N”: “300”} ◦ String {“S”: “300”} ◦ Binary {“B”: “BASE64”}  Multi-valued ◦ Number Set {“NS”: [“1”,”2”,”3.14”]} ◦ String Set {“SS”: [“A”,”b”]} ◦ Binary Set {“BS”: [“BASE64”]} 2014/5/19 CC-BY 3.0, Cliff Chao-kuan Lu <clifflu@gmail.com> 45
  46. 46.  Document size: 64 KB (UTF-8)  5 LSIs / 5 GSIs per Table  Min throughput: 1  Hash Key: 2 KB  Range Key: 1KB  BatchGetItem: 1MB or 100 items  BatchWriteItem: 1MB or 25 items 2014/5/19 CC-BY 3.0, Cliff Chao-kuan Lu <clifflu@gmail.com> 46
  47. 47.  Data Storage: ◦ 約是 S3 10x  Provisioned Throughputs ◦ Reads:4kb  循序可合併  Eventually Consistent 消耗減半 ◦ Writes : 1kb 2014/5/19 CC-BY 3.0, Cliff Chao-kuan Lu <clifflu@gmail.com> 47
  48. 48.  Hash Key  [opt] Range Key 2014/5/19 CC-BY 3.0, Cliff Chao-kuan Lu <clifflu@gmail.com> 48 {“hash”: “a”, “range”: “123”, …} {“hash”: “a”, “range”: “223”, …} {“hash”: “a”, “range”: “321”, …} {“hash”: “b”, “range”: “3”, …} {“hash”: “b”, “range”: “22”, …} {“hash”: “b”, “range”: “321”, …}
  49. 49. 2014/5/19 CC-BY 3.0, Cliff Chao-kuan Lu <clifflu@gmail.com> 49 {“hash”: “a”, “range”: “123”, …} {“hash”: “a”, “range”: “223”, …} {“hash”: “a”, “range”: “321”, …} {“hash”: “b”, “range”: “3”, …} {“hash”: “b”, “range”: “22”, …} {“hash”: “b”, “range”: “321”, …} SortedSharded
  50. 50.  Index Name  Hash Key  Range Key  Projection  Shared Throughputs 2014/5/19 CC-BY 3.0, Cliff Chao-kuan Lu <clifflu@gmail.com> 50
  51. 51.  Index Name / Hash / Range Key ◦ 比照 LSI ◦ 不要求 uniqueness  有自己的 Throughputs  Eventual Consistency  Projected Attributes 2014/5/19 CC-BY 3.0, Cliff Chao-kuan Lu <clifflu@gmail.com> 51
  52. 52.  針對特定 Index (PK, LSI or GSI) 查詢  支援 Condition  支援 Filter  高效 2014/5/19 CC-BY 3.0, Cliff Chao-kuan Lu <clifflu@gmail.com> 52
  53. 53.  現適用於 scan 及 query  消耗 throughput 不變  減低 DynamoDB <-> Caller 傳輸  boto 僅 boto.dynamodb2 支援 2014/5/19 CC-BY 3.0, Cliff Chao-kuan Lu <clifflu@gmail.com> 53
  54. 54.  … 2014/5/19 CC-BY 3.0, Cliff Chao-kuan Lu <clifflu@gmail.com> 54
  55. 55.  Upsert  需包含 Primary Key  Update: PATCH  Put: POST (PUT)  支援 Conditional Operation 2014/5/19 CC-BY 3.0, Cliff Chao-kuan Lu <clifflu@gmail.com> 55
  56. 56.  支援 Conditional Operation  用於刪除 Item  若要刪除 Attribute,需使用 updateItem 2014/5/19 CC-BY 3.0, Cliff Chao-kuan Lu <clifflu@gmail.com> 56
  57. 57.  “Expected”  可實現 MVCC pattern  配合 Document Atomicity 可模擬 transactional behavior (2-Phase Commit) 2014/5/19 CC-BY 3.0, Cliff Chao-kuan Lu <clifflu@gmail.com> 57
  58. 58.  updateItem ◦ AttributeUpdates  Value  Action:  PUT  DELETE  ADD 2014/5/19 CC-BY 3.0, Cliff Chao-kuan Lu <clifflu@gmail.com> 58
  59. 59. 2014/5/19 CC-BY 3.0, Cliff Chao-kuan Lu <clifflu@gmail.com> 59  Item  Attribute
  60. 60. 2014/5/19 CC-BY 3.0, Cliff Chao-kuan Lu <clifflu@gmail.com> 60  Item ◦ “dynamodb:LeadingKeys”: [“xxx”]  Attribute ◦ “dynamodb:Attributes”: [“xxx”,”yyy”]
  61. 61.  While True: Item.save(expect=…)  1unit for Read / Write 2014/5/19 CC-BY 3.0, Cliff Chao-kuan Lu <clifflu@gmail.com> 61
  62. 62.  Stats from DynamoDB 2014/5/19 CC-BY 3.0, Cliff Chao-kuan Lu <clifflu@gmail.com> 62
  63. 63.  CloudWatch 顯示低階數據  `Expect` 不消耗 Read Unit  允許 short burst 2014/5/19 CC-BY 3.0, Cliff Chao-kuan Lu <clifflu@gmail.com> 63
  64. 64.  了解 DynamoDB 的 ◦ 設計目標 ◦ 調校  Index  Throughput ◦ 操作 ◦ 計費 2014/5/19 CC-BY 3.0, Cliff Chao-kuan Lu <clifflu@gmail.com> 64
  65. 65. 終於到正題了 !? 2014/5/19 CC-BY 3.0, Cliff Chao-kuan Lu <clifflu@gmail.com> 65
  66. 66.  Phil: clifflu 你要不要講五月小聚  Henry: 聽說 Cloudflare + wordpress 會爆炸  clifflu: 好啊,就講這個吧 2014/5/19 CC-BY 3.0, Cliff Chao-kuan Lu <clifflu@gmail.com> 66
  67. 67.  秒殺  WP 看 location.href 重導頁面  講不滿三十分鐘 ˇˇ  只好調出備用題目 2014/5/19 CC-BY 3.0, Cliff Chao-kuan Lu <clifflu@gmail.com> 67
  68. 68.  EC2 各種麻煩 ◦ Load Balancing ◦ Types ◦ Contracts ◦ AutoScaling  Parameters  Pattern 2014/5/19 CC-BY 3.0, Cliff Chao-kuan Lu <clifflu@gmail.com> 68
  69. 69. 2014/5/19 CC-BY 3.0, Cliff Chao-kuan Lu <clifflu@gmail.com> 69
  70. 70.  使用服務 ◦ IAM, S3 (DynamoDB), CF  Octopress / Jekyll !? Clone Ruby 好像很遜 >///< 2014/5/19 CC-BY 3.0, Cliff Chao-kuan Lu <clifflu@gmail.com> 70
  71. 71.  最新功能  最冷門功能  最好 Manual 沒有 2014/5/19 CC-BY 3.0, Cliff Chao-kuan Lu <clifflu@gmail.com> 71
  72. 72.  前端 ◦ Angular.js 潮 ◦ OAuth 勁  後端 ◦ 從缺,帥  資料 ◦ DynamoDB,猛 2014/5/19 CC-BY 3.0, Cliff Chao-kuan Lu <clifflu@gmail.com> 72
  73. 73.  前端 ◦ Angular.js 潮 ◦ OAuth 勁  後端 ◦ 從缺,帥  資料 ◦ DynamoDB,猛 2014/5/19 CC-BY 3.0, Cliff Chao-kuan Lu <clifflu@gmail.com> 73 CloudFront
  74. 74.  申請 APP ◦ Amazon ◦ Facebook ◦ Google  將 access_token 透過 STS 轉換為 Access / Secret Key Pair 2014/5/19 CC-BY 3.0, Cliff Chao-kuan Lu <clifflu@gmail.com> 74
  75. 75.  欄位:uid, q_id, o_id  Primary Key ◦ (uid, q_id) : unique 確保每人每題一票 ◦ 選 uid 為 hash key  Authentication  Cardinality 2014/5/19 CC-BY 3.0, Cliff Chao-kuan Lu <clifflu@gmail.com> 75
  76. 76.  updateItem (省略 callback) 2014/5/19 CC-BY 3.0, Cliff Chao-kuan Lu <clifflu@gmail.com> 76 Primary Key Upsert
  77. 77.  updateItem 2014/5/19 CC-BY 3.0, Cliff Chao-kuan Lu <clifflu@gmail.com> 77
  78. 78.  正解:Worker,但霸氣不足  需求: ◦ uid 不外漏 ◦ affordable ◦ 快 2014/5/19 CC-BY 3.0, Cliff Chao-kuan Lu <clifflu@gmail.com> 78
  79. 79.  建立 (q_id, o_id) 之 GSI  禁止讀取 uid 欄位  取出列數即為總票數  循序讀取,節省 read capacity 2014/5/19 CC-BY 3.0, Cliff Chao-kuan Lu <clifflu@gmail.com> 79
  80. 80.  LSI / GSI 必定包含 Primary Key  透過 Query / Scan 取得 Item 時,必 須允許讀取 primary key 2014/5/19 CC-BY 3.0, Cliff Chao-kuan Lu <clifflu@gmail.com> 80
  81. 81. 2014/5/19 CC-BY 3.0, Cliff Chao-kuan Lu <clifflu@gmail.com> 81
  82. 82.  運用 “select”: “COUNT”  只計算票數,不取 Item body  循序讀取 2014/5/19 CC-BY 3.0, Cliff Chao-kuan Lu <clifflu@gmail.com> 82
  83. 83.  Document 沒寫  Boto 沒使用 ◦ dynamodb 實做了 query.count(),透過 取回 item 記數 ◦ dynamodb2 未有類似功能 2014/5/19 CC-BY 3.0, Cliff Chao-kuan Lu <clifflu@gmail.com> 83
  84. 84.  GSI Read Capacity, 1.5 !!?? 2014/5/19 CC-BY 3.0, Cliff Chao-kuan Lu <clifflu@gmail.com> 84
  85. 85. 2014/5/19 CC-BY 3.0, Cliff Chao-kuan Lu <clifflu@gmail.com> 85
  86. 86. 2014/5/19 CC-BY 3.0, Cliff Chao-kuan Lu <clifflu@gmail.com> 86
  87. 87. 2014/5/19 CC-BY 3.0, Cliff Chao-kuan Lu <clifflu@gmail.com> 87
  88. 88. 2014/5/19 CC-BY 3.0, Cliff Chao-kuan Lu <clifflu@gmail.com> 88 Throughputs Read Write Table 1 5 GSI 19 5
  89. 89. 2014/5/19 CC-BY 3.0, Cliff Chao-kuan Lu <clifflu@gmail.com> 89 Throughputs Read Write Table 1 5 GSI 19 5 Total #Reads 8kb data 消耗 1 投票消耗 GSI ~ Table ~6.6 USD/mo
  90. 90.  改用 Batched Query ? ◦ 不支援 select COUNT  優化流程 ◦ 資料更新 ◦ 後台計票 ◦ 操作介面 ◦ PR please  2014/5/19 CC-BY 3.0, Cliff Chao-kuan Lu <clifflu@gmail.com> 90
  91. 91. 今天沒有 bonus session 2014/5/19 CC-BY 3.0, Cliff Chao-kuan Lu <clifflu@gmail.com> 91
  92. 92. 回家記得念 Manual 2014/5/19 CC-BY 3.0, Cliff Chao-kuan Lu <clifflu@gmail.com> 92
  93. 93.  /clifflu/headless-poller  簡報下載:  AWS Doc ◦ IAM ◦ STS ◦ DynamoDB API 2014/5/19 CC-BY 3.0, Cliff Chao-kuan Lu <clifflu@gmail.com> 93

×