The document discusses MogileFS, an open-source distributed object storage solution known for its high availability and horizontal scalability. It highlights its ease of use, command-line tools, and production experience from companies like KKBox, along with its robust reliability features. Additionally, it addresses configuration examples, modern durable write enhancements, and analysis on system durability and data loss probabilities.

1. MogileFS
簡約可靠的儲存方案
TWJUG Meetup Nov. 2016
kaif@kaif (member of mogilefs-moji)

2. Outline
• Mogilefs
• Moji
• State of the art in mogilefs reliability

3. Quick facts
“Open source distributed object storage”
– a.k.a. cloud storage, soft defined storage…
• 高可用、水平擴展
• 檔案多副本儲存、修復
• 簡單的架構、容易使用
• 眾多應用實績

4. Brad Fitzpatrick
• Golang
• OpenID
• LiveJournal
– Memcached
– MogileFS
– …

5. Simplicity

6. Easy-to-use
• Command line tool
• Config file

7. Easy-to-use
• Admin tool

8. client
tracker
store
mysql
create_open
domain=toast&class=triple&debug_profile=0&fid=
0&multi_dest=1&key=qoo3
OK
path_1=http://127.0.0.20:7500/dev2/0/000/000/000
0000014.fid&path_3=http://127.0.0.25:7500/dev3/0
/000/000/0000000014.fid&devid_1=2&devid_3=3&
fid=14&path_2=http://127.0.0.25:7500/dev4/0/000/
000/0000000014.fid&dev_count=3&devid_2=4
store
store
tracker
tracker
PUT /dev208/0/068/050/0068050934.fid HTTP/1.0
Content-length: 9
some data
200 OK
1. Create open
3. Create close
2. Write data (webdav)
create_close
domain=toast&fid=14&devid=2&path=http://127.
0.0.20:7500/dev2/0/000/000/0000000014.fid&size=
1048576&key=qoo3&devid_2=3&path_2=http://12
7.0.0.25:7500/dev3/0/000/000/0000000014.fid&mu
ti_dest=1

9. Availability

10. 1WNR, memcached…
Scalability

11. 使用者見證

12. KKBOX

13. KKBOX
• 超過3,000 萬首歌（檔案）
• 儲存伺服器超過 75 台
• 總硬碟超過 2,300 顆
• 總儲存空間超過 10 PB
• 使用 8 個機櫃
(KKBOX 的音樂檔案儲存技術
Posted on August 2, 2016 by Chris Yuan)

14. My production experience
• 檔案量：KKBOX*10*N
• Node數：10^2*N
• 複雜的workload（備份、串流、物聯網、
web、log…orz）
• Java ♥

15. Moji
• A file-like MogileFS client for Java developers
• Production-ready features
– Connection pooling, load balancing, fault-tolerant…
• Quality
– Spring friendly, integration tests, well documented,actively
developing…
https://github.com/mogilefs-moji/moji

16. Configuration
• Using plain-old-Java
• Using the Spring framework
SpringMojiBean moji = new SpringMojiBean();
moji.setAddressesCsv("192.168.0.1:7001,192.168.0.2:7001");
moji.setDomain("testdomain");
moji.initialise();
moji.setTestOnBorrow(true);
moji.tracker.address=192.168.0.1:7001,192.168.0.2:7001
moji.domain=testdomain
<import resource="moji-context.xml" />

17. Usage
• Create/update a remote file
• Download a remote file
MojiFile rickRoll = moji.getFile("rick-astley");
moji.copyToMogile(new File("never-gonna-give-you-up.mp3"), rickRoll);
rickRoll.copyToFile(new File("foo-fighters.mp3"));

18. Usage
• IO stream
MojiFile fooFighters = moji.getFile("stacked-actors");
InputStream stream = null;
try {
stream = fooFighters.getInputStream();
// Do something streamy
// stream.read();
} finally {
stream.close();
}
OutputStream stream = null;
try {
stream = fooFighters.getOutputStream();
// Do something streamy
// stream.write(...);
stream.flush();
} finally {
stream.close();
}

19. • Setup environment manually
– MogileFS
– Maven dependency
Call to action!
• Quickstart feat.
docker run -d --name mogile-node jeffutter/mogile-node
docker run -it --link mogile-node:mogile-node hrchu/mogile-moji
<dependency>
<groupId>fm.last</groupId>
<artifactId>moji</artifactId>
<version>2.0.0</version>
</dependency>
https://code.google.com/p/mogilefs/wiki/QuickStartGuide

20. 來講一些 關於
可靠度的事

21. Mogilefs的可靠度對策
• Single copy ACK
• Multiple host replication policy
• MD5 checksum
• Basic health disk check
• Multiple zone plugin
• Reaper/fsck

22. 從此檔案們就過著
幸福快樂的日子~
… ?

23. 強化可靠度可能方向
• Mutiple sites
• Scrubber
• Modern durable write

24. Multiple Sites
• MogileFS::Network plugin
• 不同機房配置不同網段
• Zone對應網段設定
• Replication policy

25. Multiple Sites
• Given a network of: 10.10.0.0/16
• All of your machines are configured to have a netmask of
10.10.0.0/16 . When assigning IP addresses to machines, pick them
from 10.10.5.0/24
• 設定IP
– web1: 10.10.5.1 (netmask 255.255.0.0 or /16)
– web2: 10.10.5.2
– tracker1: 10.10.5.3
– tracker2: 10.10.5.4
– storage node 1: 10.10.5.5
– storage node 2: 10.10.5.6
– storage node 3: 10.10.8.1
• MogileFS zones, you configure:
– near=10.10.5.0/24 far=10.10.8.0/24
web1
tracker1
node1 node2
near
tracker2
node3
far
web2

26. Scrubber
• Make use of routine FSCK as scrubber
• Modified Algorithm
– Remove exhaustive search
– Improve performance in large scale
https://github.com/mogilefs/MogileFS-
Network/blob/master/lib/MogileFS/ReplicationPolicy/HostsPerNetwork.pm#L84
mogadm fsck status |grep " Yes " ||
(mogadm fsck reset; mogadm fsck clearlog; mogadm fsck start)
>/var/log/mogadm.fsck 2>&1

27. Modern durable write
• AS-IS
client
tracker
store
mysql
store store
tracker
tracker
4. Write other copies asynchronously
Assume that a file should have at least three replicas
in the system to fit the durability requirement

28. Modern durable write
client
tracker
store
mysql
2. Write at least two copies
before ACK
store store
tracker
tracker
4. Write other copies
asynchronously
• TO-BE
Assume that a file should have at least three replicas
in the system to fit the durability requirement
mogilefs-moji#25
mogilefs/MogileFS-Server#39

29. Analysis
• Disk failure pattern
– MTTF?
– poisson distribution?
• Mark-out: 發現錯誤的空窗期
• Rep latency: 非同步複製的空窗期
• 硬碟大小，檔案大小也會影響計算結果

30. Analysis
• Combinatorial analysis model
– Assume that each disk fails independently
– Assume that after x hours of operation each block
has P(xi) = p
– Probability of failure q = 1 - p.
– 對replication來說是一個naive的公式：1 – qn

31. Analysis
• 若考慮
– Non-Recoverable Errors (NREs)
– drive failure events are poisson
– site failures (e.g. due to regional disasters)
– rep latency, mark-out time
– …
• Analysis of system durability is commonly
done with Markov models

32. Analysis
• Example of durable write
– Assume mean disk life is 500K hrs
– 2 replicas, no NRE
249960
249980
250000
250020
250040
250060
250080
1 0.041666667 0.020833333 0.013888889
diff disk life 5
diff disk life 5
Diff of MTTDL in hr
mu
複製速率越低, durable
write的改善幅度越大

33. Analysis
• Example of probability of data loss
0.000000E+00
1.000000E-05
2.000000E-05
3.000000E-05
4.000000E-05
5.000000E-05
6.000000E-05
7.000000E-05
8.000000E-05
1 2 3 4 5 6 7 8 9 10 11 12 13 14
P of data loss 72
P of data loss 48
P of data loss 24
P of data loss 1

34. Recap
儲存之於架構
 案場需求決定儲存架構抉擇
 在考量機敏資料、業主需求、成本或是legacy的情境，
mogilefs或許會是合適的儲存架構選擇～
關於Mogilefs，我想說的是…
 簡單可擴展的非結構化儲存系統
 Java stack建議搭配moji服用
 如果事業做很大有富爸爸，能找
specialist/consulting，ceph/swift會是更先進複雜
的選擇！

35. Thank you~
【關於我】
https://kaif.io/u/kaif
https://github.com/hrchu
petertc.chu@gmail.com
【關於moji】
https://github.com/mogilefs-moji/moji