Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

萬事萬物皆是 LOG - 系統架構也來點科普

489 views

Published on

請搭配逐字稿使用

https://medium.com/@poga/caf3029359f8

Published in: Technology
  • Be the first to comment

  • Be the first to like this

萬事萬物皆是 LOG - 系統架構也來點科普

  1. 1. 萬事萬物皆是 LOG Poga @ COSCUP 2016.08.20
  2. 2. LOG If you want maintainable, understandable software system
  3. 3. 什麼是 LOG
  4. 4. LOG 的性質 ● 按照時間出現 ● 出現過的就不會改變(append-only) ● 目的:記錄(某個時間)發生了(某件事) ○ 常常用來作為 debug 推理的依據
  5. 5. Things based on LOG ● Database ● Version Control ● Distributed System and Distributed Consensus ● Synchronization ● Replication ● Messaging ● UI Framework ● ...And almost EVERYTHING
  6. 6. 聽完就理解了軟體架構的真理(誤)
  7. 7. POGA Software Developer, g0v.tw, KKTIX(前)
  8. 8. 資料庫 ID Username Balance 231 poga 100 7212 moon 12314 834 et 927 ... ... ...Database
  9. 9. ● 大量讀寫資料,長期儲存 ● 確保資料正確性(relation, validation, schema, ACID) ● 支援多種高效查詢(SQL, index) 如果你從沒用過資料庫...
  10. 10. 寫入資料時當機?
  11. 11. 如何避免寫入資料時當機導致資料消失? 寫入: ID Username Balance 231 poga 100 ... 2016.08.20 13:40:05.0012621 Write {ID:231, Username:poga, Balance: 100} ... 真正寫入: Validation, Update Index… etc.
  12. 12. ... 2016.08.20 13:40:05.0012621 Write {ID:231, Username:poga, Balance: 100} ... 時間 發生了什麼事
  13. 13. WRITE-AHEAD LOG
  14. 14. LOG ● Fast and simple RECAP
  15. 15. 資料庫 ID Username Balance 231 poga 100 7212 moon 12314 834 et 927 ... ... ...Database
  16. 16. ● 單一 Database 擴充性有其極限 ● 多台 Database,分散負載 ○ 需要確保多台 Database 之間資料同步 Replication
  17. 17. 確保多台資料庫間資料同步?
  18. 18. 寫入一台資料庫 寫入: ID Username Balance 231 poga 100 ... 2016.08.20 13:40:05.0012621 Write {ID:231, Username:poga, Balance: 100} ... 真正寫入: Validation, Update Index… etc.
  19. 19. 一次寫入多台資料庫 寫入: ID Username Balance 231 poga 100 ... 2016.08.20 13:40:05.0012621 Write {ID:231, Username:poga, Balance: 100} ... 真正寫入: Validation, Update Index… etc. ... 2016.08.20 13:40:05.0012621 Write {ID:231, Username:poga, Balance: 100} ... 真正寫入: Validation, Update Index… etc. DB1 DB2
  20. 20. LOG-SHIPPING
  21. 21. LOG ● Fast and simple ● Can be used to share “current state” RECAP
  22. 22. 微服務 Microservice
  23. 23. Monolith v.s. Microservice
  24. 24. Monolith ● 所有邏輯集中在單一系統中,共 享儲存運算資源 ● 容易開發 ● 不易擴充
  25. 25. Microservice ● 將邏輯按照 Domain 切割,有 獨立的儲存與運算資源 ● 容易擴充,團隊間分責明確 ● 門檻較高:不易維護,需要進階 系統維護能力
  26. 26. 微服務間怎麼互相溝通?
  27. 27. Event Stream
  28. 28. EVENT STREAM = LOG
  29. 29. LOG ● Fast and simple ● Can be used to share “current state” ● LOG are easy to parse/understand, Universal Interface RECAP
  30. 30. 前端架構 Flux/Redux
  31. 31. ● Many, many states ● Constantly changing state ○ User Interaction, Animation, Network ● Hard to reproduce bug and debug UI State
  32. 32. 前端介面狀態多變,如何維護?
  33. 33. Flux/Redux ● Unidirectional Data Flow ○ Append only ○ Handle action one by one, reasonable ● Immutable State ○ Produce new state based on previous state and action ● Everything else can be stateless(pure)
  34. 34. UNIDIRECTIONAL = LOG
  35. 35. LOG ● Fast and simple ● Can be used to share “current state” ● Text are easy to parse/understand, Universal Interface ● Append-only, easy to reason, everything else can be stateless RECAP
  36. 36. 區塊鏈 Blockchain
  37. 37. ● 2016.08.20 13:31:23, A 給 B 100 bitcoin ● State: A 跟 B 身上剩下多少 bitcoin ● Double Spend: ○ A 把 bitcoin 交給 B 後,不能再把同樣的 bitcoin 交給 C Transaction
  38. 38. 如何讓全世界都認知交易的成立? a.k.a. Avoid Double-Spending
  39. 39. ● 如果 ○ 所有參與者都知道/可以驗證所有人的交易記錄 ○ 既有的交易紀錄無法被竄改 ○ 便沒有偽造空間 ● Blockchain ○ 將所有的交易記錄用 block 封裝,每個 block 替之前的交易記錄加密驗證 ○ 想要偽造一筆交易記錄 = 必須偽造整個 blockchain 上的交易記錄 ○ 所有人都基於相同的 blockchain 驗證交易 Distributed Consensus
  40. 40. LOG AGREEMENT = CONSENSUS Raft: A Consensus Algorithm for Replicated Logs Apache Mesos - The Mesos Replicated Log (PAXOS)
  41. 41. LOG ● Fast and simple ● Can be used to share “current state” ● Text are easy to parse/understand, Universal Interface ● Append-only, Immutable, easy to reason, everything else can be stateless ● Replicate log = consensus RECAP
  42. 42. 大數據 Data Pipeline
  43. 43. ● > PB ● Data coming from everywhere ○ User Input ○ Business Data ○ Tracking ○ External Data ● ETL(Extract-Transform-Load) ● Analysis, Machine Learning… etc BIG Data
  44. 44. 如何處理雜亂、無序、大量的資料?
  45. 45. Data Pipeline
  46. 46. APPEND ONLY LOG = HIGH SCALABILITY
  47. 47. LOG ● Fast and simple ● Can be used to share “current state” ● Text are easy to parse/understand, Universal Interface ● Append-only, Immutable, easy to reason, everything else can be stateless ● Replicate log = consensus ● High Scalability RECAP
  48. 48. LOG = DETERMINISM
  49. 49. 相同的過程 = 相同的結果 DETERMINISM
  50. 50. Make your system deterministic ● 容易理解 ● 容易DEBUG ● 容易同步 ● 容易儲存 ● 容易擴展
  51. 51. LOG = DETERMINISM = 穩定的系統
  52. 52. 設計系統架構時,先試著用 LOG 表示你的系統
  53. 53. Thanks!

×