The document discusses how logs (LOG) can be used as the basis for many software systems and applications. It describes logs as sequentially recorded events that provide an immutable record of what happened over time. It then gives examples of how databases, version control systems, distributed systems, messaging, and many other areas are based on logs and benefit from their append-only, immutable nature. Logs allow sharing state, reaching consensus, and building deterministic, scalable, and maintainable systems.

1. 萬事萬物皆是 LOG
Poga @ COSCUP 2016.08.20

2. LOG
If you want maintainable, understandable software system

3. 什麼是 LOG

4. LOG 的性質
● 按照時間出現
● 出現過的就不會改變（append-only）
● 目的：記錄（某個時間）發生了（某件事）
○ 常常用來作為 debug 推理的依據

5. Things based on LOG
● Database
● Version Control
● Distributed System and Distributed Consensus
● Synchronization
● Replication
● Messaging
● UI Framework
● ...And almost EVERYTHING

6. 聽完就理解了軟體架構的真理（誤）

7. POGA
Software Developer, g0v.tw, KKTIX(前)

8. 資料庫
ID Username Balance
231 poga 100
7212 moon 12314
834 et 927
... ... ...Database

9. ● 大量讀寫資料，長期儲存
● 確保資料正確性（relation, validation, schema, ACID）
● 支援多種高效查詢（SQL, index）
如果你從沒用過資料庫...

10. 寫入資料時當機？

11. 如何避免寫入資料時當機導致資料消失？
寫入：
ID Username Balance
231 poga 100
...
2016.08.20 13:40:05.0012621 Write {ID:231, Username:poga, Balance: 100}
...
真正寫入： Validation, Update Index… etc.

12. ...
2016.08.20 13:40:05.0012621 Write {ID:231, Username:poga, Balance: 100}
...
時間 發生了什麼事

13. WRITE-AHEAD LOG

14. LOG
● Fast and simple
RECAP

15. 資料庫
ID Username Balance
231 poga 100
7212 moon 12314
834 et 927
... ... ...Database

16. ● 單一 Database 擴充性有其極限
● 多台 Database，分散負載
○ 需要確保多台 Database 之間資料同步
Replication

17. 確保多台資料庫間資料同步？

18. 寫入一台資料庫
寫入：
ID Username Balance
231 poga 100
...
2016.08.20 13:40:05.0012621 Write {ID:231, Username:poga, Balance: 100}
...
真正寫入： Validation, Update Index… etc.

19. 一次寫入多台資料庫
寫入：
ID Username Balance
231 poga 100
...
2016.08.20 13:40:05.0012621 Write
{ID:231, Username:poga, Balance: 100}
...
真正寫入： Validation, Update Index… etc.
...
2016.08.20 13:40:05.0012621 Write {ID:231,
Username:poga, Balance: 100}
...
真正寫入： Validation, Update Index… etc.
DB1 DB2

20. LOG-SHIPPING

21. LOG
● Fast and simple
● Can be used to share “current state”
RECAP

22. 微服務
Microservice

23. Monolith v.s. Microservice

24. Monolith
● 所有邏輯集中在單一系統中，共
享儲存運算資源
● 容易開發
● 不易擴充

25. Microservice
● 將邏輯按照 Domain 切割，有
獨立的儲存與運算資源
● 容易擴充，團隊間分責明確
● 門檻較高：不易維護，需要進階
系統維護能力

26. 微服務間怎麼互相溝通？

27. Event Stream

28. EVENT STREAM = LOG

29. LOG
● Fast and simple
● Can be used to share “current state”
● LOG are easy to parse/understand, Universal Interface
RECAP

30. 前端架構
Flux/Redux

31. ● Many, many states
● Constantly changing state
○ User Interaction, Animation, Network
● Hard to reproduce bug and debug
UI State

32. 前端介面狀態多變，如何維護？

33. Flux/Redux
● Unidirectional Data Flow
○ Append only
○ Handle action one by one, reasonable
● Immutable State
○ Produce new state based on previous state and action
● Everything else can be stateless(pure)

34. UNIDIRECTIONAL = LOG

35. LOG
● Fast and simple
● Can be used to share “current state”
● Text are easy to parse/understand, Universal Interface
● Append-only, easy to reason, everything else can be
stateless
RECAP

36. 區塊鏈
Blockchain

37. ● 2016.08.20 13:31:23, A 給 B 100 bitcoin
● State: A 跟 B 身上剩下多少 bitcoin
● Double Spend:
○ A 把 bitcoin 交給 B 後，不能再把同樣的 bitcoin 交給 C
Transaction

38. 如何讓全世界都認知交易的成立？
a.k.a. Avoid Double-Spending

39. ● 如果
○ 所有參與者都知道/可以驗證所有人的交易記錄
○ 既有的交易紀錄無法被竄改
○ 便沒有偽造空間
● Blockchain
○ 將所有的交易記錄用 block 封裝，每個 block 替之前的交易記錄加密驗證
○ 想要偽造一筆交易記錄 = 必須偽造整個 blockchain 上的交易記錄
○ 所有人都基於相同的 blockchain 驗證交易
Distributed Consensus

40. LOG AGREEMENT = CONSENSUS
Raft: A Consensus Algorithm for Replicated Logs
Apache Mesos - The Mesos Replicated Log (PAXOS)

41. LOG
● Fast and simple
● Can be used to share “current state”
● Text are easy to parse/understand, Universal Interface
● Append-only, Immutable, easy to reason, everything else
can be stateless
● Replicate log = consensus
RECAP

42. 大數據
Data Pipeline

43. ● > PB
● Data coming from everywhere
○ User Input
○ Business Data
○ Tracking
○ External Data
● ETL(Extract-Transform-Load)
● Analysis, Machine Learning… etc
BIG Data

44. 如何處理雜亂、無序、大量的資料？

45. Data Pipeline

46. APPEND ONLY LOG = HIGH SCALABILITY

47. LOG
● Fast and simple
● Can be used to share “current state”
● Text are easy to parse/understand, Universal Interface
● Append-only, Immutable, easy to reason, everything else
can be stateless
● Replicate log = consensus
● High Scalability
RECAP

48. LOG = DETERMINISM

49. 相同的過程 = 相同的結果
DETERMINISM

50. Make your system deterministic
● 容易理解
● 容易DEBUG
● 容易同步
● 容易儲存
● 容易擴展

51. LOG = DETERMINISM = 穩定的系統

52. 設計系統架構時，先試著用 LOG 表示你的系統

53. Thanks!