Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

How to build a scalable SNS via Polling & Push

4,174 views

Published on

Agenda
● Variety of sync mechanisms
● Polling & Push
● RESTful API Design
● API Blueprint

Published in: Technology

How to build a scalable SNS via Polling & Push

  1. 1. How to build a scalable SNS via Polling & Push Kewang 三竹資訊
  2. 2. 2 Who I am ● 王慕羣 ● Java / Node.js / AngularJS ● SQL-like / HBase ● Mentor: NIU CSIE GitHub: kewang Google Play: kewang Facebook: kewangtw Linkedin: kewangtw Slideshare: kewang Mail: cpckewang@gmail.com
  3. 3. Who Mitake is 三竹資訊
  4. 4. Who Mitake is 三竹資訊 大家都唸 Mitake
  5. 5. Who Mitake is 三竹資訊 大家都唸 Mitake ,但我們公司都唸 Mitake
  6. 6. Who Mitake is 三竹資訊 Mitake 不唸作 MiTAC 啊!!!
  7. 7. Who Mitake is 三竹資訊
  8. 8. Who Mitake is 三竹資訊 ● 簡訊平台
  9. 9. Who Mitake is 三竹資訊 ● 簡訊平台 ● 行動下單:
  10. 10. Who Mitake is 三竹資訊 ● 簡訊平台 ● 行動下單:
  11. 11. Who Mitake is 三竹資訊 ● 簡訊平台 ● 行動下單:不計其數
  12. 12. Who Mitake is 三竹資訊 ● 簡訊平台 ● 行動下單:不計其數 ● 行動銀行:
  13. 13. Who Mitake is 三竹資訊 ● 簡訊平台 ● 行動下單:不計其數 ● 行動銀行:臺銀、土銀、富邦、台新、聯邦、臺 企銀、遠銀、華南、澳盛、郵局、合庫、渣打 ... 等 18 家
  14. 14. Who Mitake is 三竹資訊 ● 簡訊平台 ● 行動下單:不計其數 ● 行動銀行:臺銀、土銀、富邦、台新、聯邦、臺 企銀、遠銀、華南、澳盛、郵局、合庫、渣打 ... 等 18 家 ● 產壽險:
  15. 15. Who Mitake is 三竹資訊 ● 簡訊平台 ● 行動下單:不計其數 ● 行動銀行:臺銀、土銀、富邦、台新、聯邦、臺 企銀、遠銀、華南、澳盛、郵局、合庫、渣打 ... 等 18 家 ● 產壽險:全球、明台、新光、新安東京、富邦 ... 等
  16. 16. Who Mitake is 三竹資訊 ● 簡訊平台 ● 行動下單:不計其數 ● 行動銀行:臺銀、土銀、富邦、台新、聯邦、臺企 銀、遠銀、華南、澳盛、郵局、合庫、渣打 ... 等 18 家 ● 產壽險:全球、明台、新光、新安東京、富邦 ... 等 ● 其他:
  17. 17. Who Mitake is 三竹資訊 ● 簡訊平台 ● 行動下單:不計其數 ● 行動銀行:臺銀、土銀、富邦、台新、聯邦、臺企 銀、遠銀、華南、澳盛、郵局、合庫、渣打 ... 等 18 家 ● 產壽險:全球、明台、新光、新安東京、富邦 ... 等 ● 其他: udn 買東西、手機逛週年慶、財政園地、證交 所、綜所稅申報 ... 等
  18. 18. 18 System Architecture
  19. 19. 19 System Architecture
  20. 20. 20 System Architecture (Backend)
  21. 21. 21 System Architecture (Backend) HadoopCon 2014
  22. 22. 22 System Architecture (Frontend)
  23. 23. 23 Agenda ● Variety of sync mechanisms ● Polling & Push ● RESTful API Design ● API Blueprint
  24. 24. 24 Variety of sync mechanisms
  25. 25. 25 Variety of sync mechanisms ● Polling ● Comet ● Long Polling ● WebSocket ● Engine.IO (Socket.IO)
  26. 26. 26 Polling send scheduling request
  27. 27. 27 Polling client server
  28. 28. 28 Polling client server T0 T1 T2
  29. 29. 29 Polling client server T0 T1 T2 T4 T5 T6
  30. 30. 30 Polling client server T0 T1 T2 T4 T5 T6 T8 T9 T10
  31. 31. 31 Polling ● Pros – Easy to implement ● Cons – No efficiency
  32. 32. 32 Comet a never died HTTP request
  33. 33. 33 Comet client server
  34. 34. 34 Comet client server T0 T1
  35. 35. 35 Comet client server T0 T1
  36. 36. 36 Comet client server T0 T1 T3 T2
  37. 37. 37 Comet client server T0 T1 T3 T6 T2 T5
  38. 38. 38 Comet client server T0 T1 T3 T6 T8 T2 T5 T7
  39. 39. 39 Comet client server T0 T1 T3 T6 T8 T12 T2 T5 T7 T11
  40. 40. 40 Comet ● Pros – Save more network traffic ● Cons – Blocking IO issue – Always get server response, can't send another request
  41. 41. 41 Long Polling send a long time request repeatedly when response received
  42. 42. 42 Long Polling client server
  43. 43. 43 Long Polling client server T0 T1
  44. 44. 44 Long Polling client server T0 T1
  45. 45. 45 Long Polling client server T0 T1 T4 T3
  46. 46. 46 Long Polling client server T0 T1 T4 T3 T5
  47. 47. 47 Long Polling client server T0 T1 T4 T3 T5
  48. 48. 48 Long Polling client server T0 T1 T4 T3 T5 T21 T20
  49. 49. 49 Long Polling ● Pros – Save a little network traffic – Can send another request ● Cons – I don't know
  50. 50. 50 WebSocket FDX communications channels over a single TCP connection
  51. 51. 51 WebSocket client server
  52. 52. 52 WebSocket client server
  53. 53. 53 WebSocket client server T0 T1 T2
  54. 54. 54 WebSocket client server T0 T1 T2 T3 T4 T5
  55. 55. 55 WebSocket client server T0 T1 T2 T3 T4 T5 T9 T10 T11
  56. 56. 56 WebSocket client server T0 T1 T2 T3 T4 T5 T17 T18 T19 T9 T10 T11
  57. 57. 57 WebSocket ● Pros – Bidirectional communication – Save more network traffic ● Cons – Proxy & Firewall issue
  58. 58. 58 Engine.IO communication layer for Socket.IO
  59. 59. 59 Engine.IO - Socket.IO history 0.1~0.6.2 0.6.3~0.6.17 0.7.0~0.9.17 1.0.0~now websocket ✓ ✓ ✓ ✓ server-events ✓ flashsocket ✓ ✓ ✓ htmlfile ✓ ✓ xhr-multipart ✓ ✓ xhr-polling ✓ ✓ ✓ jsonp-polling ✓ polling ✓
  60. 60. 60 Engine.IO - Overview
  61. 61. 61 Engine.IO - Overview ● Ensure the most reliable realtime communication
  62. 62. 62 Engine.IO - Overview ● Ensure the most reliable realtime communication ● Always establishes a long-polling connection first
  63. 63. 63 Engine.IO - Overview ● Ensure the most reliable realtime communication ● Always establishes a long-polling connection first – then tries to upgrade to better transports that are "tested" on the side
  64. 64. 64 Engine.IO - Upgrade transport seamlessly
  65. 65. 65 Engine.IO - Upgrade transport seamlessly ● Switches from polling to another transport in between polling cycles
  66. 66. 66 Engine.IO - Upgrade transport seamlessly ● Switches from polling to another transport in between polling cycles ● To ensure no messages are lost, the upgrade packet will only be sent once all the buffers of the existing transport are flushed and the transport is considered paused
  67. 67. 67 Engine.IO client server
  68. 68. 68 Engine.IO client server T0 T1
  69. 69. 69 Engine.IO client server T0 T1
  70. 70. 70 Engine.IO client server T0 T1
  71. 71. 71 Engine.IO client server T0 T1 T6 T5 T7
  72. 72. 72 Engine.IO client server T0 T1 T11 T6 T5 T7 T10
  73. 73. 73 Engine.IO client server T0 T1 T11 T6 T5 T7 T10
  74. 74. 74 Engine.IO client server T0 T1 T11 T6 T12 T13 T14 T5 T7 T10
  75. 75. 75 Engine.IO ● Pros – Auto-switch different network scenario ● Cons – Use particular protocol at server & client
  76. 76. 76 Polling & Push
  77. 77. 77 Why we don't use WebSocket ● Server – Many corporate proxies block WebSocket traffic ● Client – Many personal firewalls and antivirus softwares block WebSocket traffic
  78. 78. 78 Push != Notification
  79. 79. 79 Push != Notification
  80. 80. 80 Push != Notification Push Notification Notification Push Notification Notification
  81. 81. 81 Push - Overview ● Non-IM – Breaking news notification – Remittance notification – Call off work notification – Email notification – ......etc. ● IM – Message notification
  82. 82. 82 How it works?
  83. 83. 83 When unsafe APIs done - Server 1.Store affected devices' ID & API ID to database 2.Send Push to affected devices
  84. 84. 84 When unsafe APIs done - Client ● Send an unified GET request – GET /datas
  85. 85. 85 When unsafe APIs done - Client ● Send an unified GET request – GET /datas ● Push off – apply frequency rapidly to achieve realtime, but waste resources
  86. 86. 86 When unsafe APIs done - Client ● Send an unified GET request – GET /datas ● Push off – apply frequency rapidly to achieve realtime, but waste resources ● Push on – apply frequency long to save resources – when Push comes, send request to get newest data to achieve realtime
  87. 87. 87 Push - Push off client server
  88. 88. 88 Push - Push off client server T0 T1 T2
  89. 89. 89 Push - Push off client server T0 T1 T2 T4 T5 T6
  90. 90. 90 Push - Push off client server T0 T1 T2 T4 T5 T6 T8 T9 T10
  91. 91. 91 Push - Push on client server SNS
  92. 92. 92 Push - Push on client server T0 T1 T2 SNS
  93. 93. 93 Push - Push on client server T0 T1 T2 T22 T23 T24 SNS
  94. 94. 94 Push - Push on client server T0 T1 T2 T22 T23 T24 SNS T30
  95. 95. 95 Push - Push on client server T0 T1 T2 T22 T23 T24 T31 T32 T33 SNS T30
  96. 96. 96 Fallback ● Don't polling everything ● Every GET APIs provide UI operation
  97. 97. 97 Fallback
  98. 98. 98 RESTful API design
  99. 99. 99 RESTful API Design ● HTTP method ● Common examples ● Is Graph API RESTful? ● Actual example ● Internal & Open API
  100. 100. 100 GET - Retrieve a resource ● A request to the server should never change the resource state
  101. 101. 101 GET - Retrieve a resource ● A request to the server should never change the resource state ● Incidental side effects are OK – Like logging
  102. 102. 102 GET - Retrieve a resource ● A request to the server should never change the resource state ● Incidental side effects are OK – Like logging ● Common response code – 200 OK
  103. 103. 103 GET - Retrieve a resource ● A request to the server should never change the resource state ● Incidental side effects are OK – Like logging ● Common response code – 200 OK – 301 Moved Permanently
  104. 104. 104 GET - Retrieve a resource ● A request to the server should never change the resource state ● Incidental side effects are OK – Like logging ● Common response code – 200 OK – 301 Moved Permanently – 404 Not Found (related to DELETE) – 410 Gone (related to DELETE)
  105. 105. 105 DELETE - Remove a resource ● A request to the server should destroy the resource & never refer to it again
  106. 106. 106 DELETE - Remove a resource ● A request to the server should destroy the resource & never refer to it again ● Common response code – 200 OK – 202 Accepted – 204 No Content
  107. 107. 107 POST
  108. 108. 108 POST ● POST-to-append
  109. 109. 109 POST ● POST-to-append ● Overloaded POST
  110. 110. 110 POST-to-append ● A request to the server should create a new resource
  111. 111. 111 POST-to-append ● A request to the server should create a new resource ● Common response code – 201 Created
  112. 112. 112 POST-to-append ● A request to the server should create a new resource ● Common response code – 201 Created (plus "Location" header)
  113. 113. 113 POST-to-append ● A request to the server should create a new resource ● Common response code – 201 Created (plus "Location" header) – 202 Accepted
  114. 114. 114 Overloaded POST ● Providing a block of data, such as the result of submitting a form, to a data-handling process
  115. 115. 115 Overloaded POST ● Providing a block of data, such as the result of submitting a form, to a data-handling process ● "Data-handling process" can be anything
  116. 116. 116 Overloaded POST ● Providing a block of data, such as the result of submitting a form, to a data-handling process ● "Data-handling process" can be anything – POST /users/sort
  117. 117. 117 Overloaded POST ● Providing a block of data, such as the result of submitting a form, to a data-handling process ● "Data-handling process" can be anything – POST /users/sort – POST /login
  118. 118. 118 Overloaded POST ● Providing a block of data, such as the result of submitting a form, to a data-handling process ● "Data-handling process" can be anything – POST /users/sort – POST /login – ...etc.
  119. 119. 119 PUT - Update a resource ● A request to the server should modify the resource
  120. 120. 120 PUT - Update a resource ● A request to the server should modify the resource ● Common response code – 200 OK – 204 No Content
  121. 121. 121 Idempotent & Safe
  122. 122. 122 Idempotent & Safe ● Idempotent – A HTTP method can be called many times without different outcomes
  123. 123. 123 Idempotent & Safe ● Idempotent – A HTTP method can be called many times without different outcomes ● Safe – Do not modify resources
  124. 124. 124 Idempotent & Safe Idempotent Safe GET yes yes DELETE yes no POST no no PUT yes no
  125. 125. 125 Common examples
  126. 126. 126 GET /users
  127. 127. 127 GET /users ● Retrieve all users' brief
  128. 128. 128 GET /users ● Retrieve all users' brief { "users": [ { "name": "kewang", "slogan": "Hello World" }, { "name": "Tony Stark", "slogan": "I'm Iron Man" }, { "name": "America captain", "slogan": "U.S.A." } ] }
  129. 129. 129 GET /users/:user_id
  130. 130. 130 GET /users/:user_id ● Retrieve a specific user information
  131. 131. 131 GET /users/:user_id ● Retrieve a specific user information { "name": "kewang", "slogan": "Hello World", "birthday": "19831108", "sex": 0, "email": [ "cpckewang@gmail.com", "kewang@mitake.com.tw" ], "company": "mitake" }
  132. 132. 132 GET /users/:user_id/avatars
  133. 133. 133 GET /users/:user_id/avatars ● Retrieve a specific user's all avatars' URL
  134. 134. 134 GET /users/:user_id/avatars ● Retrieve a specific user's all avatars' URL { "avatars": [ { "url": "http://www.s3.com/abc.png", "created": 1413385358 }, { "url": "http://www.s3.com/def.png", "created": 1401239876 }, { "url": "http://www.s3.com/ghi.png", "created": 1348303844 } ] }
  135. 135. 135 GET /users/:user_id/avatar
  136. 136. 136 GET /users/:user_id/avatar ● Retrieve a specific user's avatar URL
  137. 137. 137 GET /users/:user_id/avatar ● Retrieve a specific user's newest avatar URL
  138. 138. 138 GET /users/:user_id/avatar ● Retrieve a specific user's newest avatar URL { "url": "http://www.s3.com/abc.png" }
  139. 139. 139 Is Graph API RESTful? Facebook powered
  140. 140. 140 Graph API - Overview
  141. 141. Graph API - Overview ● Everything is a "node" has a unique ID
  142. 142. 142 Graph API - Overview ● Everything is a "node" has a unique ID ● Two nodes connect each other with a "edge"
  143. 143. 143 Graph API - Overview ● Everything is a "node" has a unique ID ● Two nodes connect each other with a "edge" ● A node information contains some "fields"
  144. 144. 144 Graph API - Are two people friends? ● GET /{user-a-id}/friends/{user-b-id}
  145. 145. 145 Graph API - Does someone like a Page? ● GET /{user-id}/likes/{page-id}
  146. 146. 146 Graph API - Publishing new Status Updates ● POST /{user-id}/feed ● POST /{page-id}/feed
  147. 147. 147 Graph API - Uploading Photos ● POST /{user-id}/photos ● POST /{page-id}/photos ● POST /{album-id}/photos
  148. 148. 148 So, Graph API is RESTful?
  149. 149. 149 Graph API isn't RESTful Deprecating the REST API
  150. 150. 150 An actual example Kick a user from system
  151. 151. 151 V1 - Kick a user from system
  152. 152. 152 V1 - Kick a user from system ● Kick by myself
  153. 153. 153 V1 - Kick a user from system ● Kick by myself – DELETE /me
  154. 154. 154 V1 - Kick a user from system ● Kick by myself – DELETE /me ● Kick somebody
  155. 155. 155 V1 - Kick a user from system ● Kick by myself – DELETE /me ● Kick somebody – DELETE /:user_id
  156. 156. 156 V2 - Restore a user to system
  157. 157. 157 V2 - Restore a user to system ● PUT /:user_id
  158. 158. 158 V2 - Restore a user to system ● PUT /:user_id – Restore somebody to system
  159. 159. 159 V2 - Restore a user to system ● PUT /:user_id – Restore somebody to system – Change somebody information
  160. 160. 160 V3 - Update a user status
  161. 161. 161 V3 - Update a user status ● PUT /:user_id/status
  162. 162. 162 V3 - Update a user status ● PUT /:user_id/status – Kick somebody from system
  163. 163. 163 V3 - Update a user status ● PUT /:user_id/status – Kick somebody from system – Restore somebody to system
  164. 164. 164 Internal & Open API
  165. 165. 165 Internal API design guideline
  166. 166. 166 Internal API design guideline ● Minimize data size
  167. 167. 167 Internal API design guideline ● Minimize data size ● Merge operations
  168. 168. 168 Internal API design guideline ● Minimize data size ● Merge operations ● Field names are short & convoluted
  169. 169. 169 Internal API design guideline ● Minimize data size ● Merge operations ● Field names are short & convoluted ● Customize request & response for official apps
  170. 170. 170 Open API design guideline
  171. 171. 171 Open API design guideline ● Minimize data size
  172. 172. 172 Open API design guideline ● Minimize data size ● Field names are simple & clear
  173. 173. 173 Open API design guideline ● Minimize data size ● Field names are simple & clear ● Build request & response generally
  174. 174. 174 Differences - GET /users Internal API { "us": [{ "id": "2d3f1a", "nm": "Kewang", "el": "kkk@mail.com" }, { "id": "9a4f57", "nm": "Hans", "el": "hhh@mail.com" }] } Open API { "users": [{ "id": "2d3f1a", "name": "Kewang" }, { "id": "9a4f57", "name": "Hans" }] }
  175. 175. 175 API Blueprint
  176. 176. 176 API Blueprint ● Introduction ● Aglio ● API-Mock ● Postman
  177. 177. 177 Introduction ● Web API Language ● Pure Markdown ● Design for Humans ● Understandable by Machines ● Powerful Tooling ● Easy Lifecycle
  178. 178. 178 Hello World
  179. 179. 179 Complex example
  180. 180. 180 Aglio - API Blueprint renderer
  181. 181. 181 Aglio - Complex example
  182. 182. 182 API-Mock & Postman
  183. 183. 183 Live DEMO
  184. 184. 184 References
  185. 185. 185 References
  186. 186. 186 References ● Browser 與 Server 持續同步的作法介紹 ● Comet (programming) ● Engine.IO: the realtime engine ● Engine.IO Protocol ● API Design Optimized for Mobile Platform ● HTTP GET with request body ● Does Google treat 404 and 410 status code differen tly
  187. 187. 187
  188. 188. 188

×