Submit Search
Upload
Durability Simulator Design for OpenStack Swift
•
1 like
•
522 views
Kota Tsuyuzaki
Follow
Slides for OpenStack 2014 Paris Design Summit.
Read less
Read more
Software
Report
Share
Report
Share
1 of 20
Download now
Download to read offline
Recommended
Global EC Cluster Updates (OpenStack Mitaka Swift Design Summit)
Global EC Cluster Updates (OpenStack Mitaka Swift Design Summit)
Kota Tsuyuzaki
More Efficient Object Replication in OpenStack Summit Juno
More Efficient Object Replication in OpenStack Summit Juno
Kota Tsuyuzaki
NFV Infrastructure Manager with High Performance Software Switch Lagopus
NFV Infrastructure Manager with High Performance Software Switch Lagopus
Hirofumi Ichihara
Contiki OS Research Projects Guidance
Contiki OS Research Projects Guidance
Network Simulation Tools
OpenContrail Experience tcp cloud OpenStack Summit Tokyo
OpenContrail Experience tcp cloud OpenStack Summit Tokyo
Jakub Pavlik
NTTs Journey with Openstack-final
NTTs Journey with Openstack-final
shintaro mizuno
CFI 2015 - Flow-centric Visibility Tools for OF@TEIN
CFI 2015 - Flow-centric Visibility Tools for OF@TEIN
GIST (Gwangju Institute of Science and Technology)
Building the IoT - Coding Serbia 2015
Building the IoT - Coding Serbia 2015
Benjamin Cabé
Recommended
Global EC Cluster Updates (OpenStack Mitaka Swift Design Summit)
Global EC Cluster Updates (OpenStack Mitaka Swift Design Summit)
Kota Tsuyuzaki
More Efficient Object Replication in OpenStack Summit Juno
More Efficient Object Replication in OpenStack Summit Juno
Kota Tsuyuzaki
NFV Infrastructure Manager with High Performance Software Switch Lagopus
NFV Infrastructure Manager with High Performance Software Switch Lagopus
Hirofumi Ichihara
Contiki OS Research Projects Guidance
Contiki OS Research Projects Guidance
Network Simulation Tools
OpenContrail Experience tcp cloud OpenStack Summit Tokyo
OpenContrail Experience tcp cloud OpenStack Summit Tokyo
Jakub Pavlik
NTTs Journey with Openstack-final
NTTs Journey with Openstack-final
shintaro mizuno
CFI 2015 - Flow-centric Visibility Tools for OF@TEIN
CFI 2015 - Flow-centric Visibility Tools for OF@TEIN
GIST (Gwangju Institute of Science and Technology)
Building the IoT - Coding Serbia 2015
Building the IoT - Coding Serbia 2015
Benjamin Cabé
OpenContrail Implementations
OpenContrail Implementations
Jakub Pavlik
OpenPOWER ADG key note
OpenPOWER ADG key note
Ganesan Narayanasamy
Introduction of Okinawa Open Laboratory Testbed, OpenStack and SDN Technology...
Introduction of Okinawa Open Laboratory Testbed, OpenStack and SDN Technology...
Takashi Torii
OpenStack Ottawa Q3 Meetup September 26th 2017
OpenStack Ottawa Q3 Meetup September 26th 2017
Stacy Véronneau
Greenplum: Building a Postgres Fabric for Large-Scale Analytical Computation ...
Greenplum: Building a Postgres Fabric for Large-Scale Analytical Computation ...
VMware Tanzu
CERN User Story
CERN User Story
Tim Bell
Collect, summarize and notify of OpenStack's log
Collect, summarize and notify of OpenStack's log
NTT Communications Technology Development
Containers on Baremetal and Preemptible VMs at CERN and SKA
Containers on Baremetal and Preemptible VMs at CERN and SKA
Belmiro Moreira
Future semantic segmentation with convolutional LSTM
Future semantic segmentation with convolutional LSTM
Kyuri Kim
OpenStack Summit Tokyo - Know-how of Challlenging Deploy/Operation NTT DOCOMO...
OpenStack Summit Tokyo - Know-how of Challlenging Deploy/Operation NTT DOCOMO...
Masaaki Nakagawa
Mike Bartley - Innovations for Testing Parallel Software - EuroSTAR 2012
Mike Bartley - Innovations for Testing Parallel Software - EuroSTAR 2012
TEST Huddle
BIRTE-13-Kawashima
BIRTE-13-Kawashima
Hideyuki Kawashima
Understanding performance aspects of etcd and Raft
Understanding performance aspects of etcd and Raft
Hitoshi Mitake
Modeling the IoT with TitanDB and Cassandra
Modeling the IoT with TitanDB and Cassandra
twilmes
HeapStats: Troubleshooting with Serviceability and the New Runtime Monitoring...
HeapStats: Troubleshooting with Serviceability and the New Runtime Monitoring...
Yuji Kubota
Tsinghua University: Two Exemplary Applications in China
Tsinghua University: Two Exemplary Applications in China
DataStax Academy
Icbai 2018 ver_1
Icbai 2018 ver_1
BlackhatGAURAV
Crash course on data streaming (with examples using Apache Flink)
Crash course on data streaming (with examples using Apache Flink)
Vincenzo Gulisano
IRJET - Predicting the Maximum Computational Power of Microprocessors using M...
IRJET - Predicting the Maximum Computational Power of Microprocessors using M...
IRJET Journal
The Pill for Your Migration Hell
The Pill for Your Migration Hell
Databricks
Transport SDN & OpenDaylight Use Cases in Korea
Transport SDN & OpenDaylight Use Cases in Korea
Justin Park
Online learning for low-latency streaming
Online learning for low-latency streaming
Theo Karagkioules
More Related Content
What's hot
OpenContrail Implementations
OpenContrail Implementations
Jakub Pavlik
OpenPOWER ADG key note
OpenPOWER ADG key note
Ganesan Narayanasamy
Introduction of Okinawa Open Laboratory Testbed, OpenStack and SDN Technology...
Introduction of Okinawa Open Laboratory Testbed, OpenStack and SDN Technology...
Takashi Torii
OpenStack Ottawa Q3 Meetup September 26th 2017
OpenStack Ottawa Q3 Meetup September 26th 2017
Stacy Véronneau
Greenplum: Building a Postgres Fabric for Large-Scale Analytical Computation ...
Greenplum: Building a Postgres Fabric for Large-Scale Analytical Computation ...
VMware Tanzu
CERN User Story
CERN User Story
Tim Bell
Collect, summarize and notify of OpenStack's log
Collect, summarize and notify of OpenStack's log
NTT Communications Technology Development
Containers on Baremetal and Preemptible VMs at CERN and SKA
Containers on Baremetal and Preemptible VMs at CERN and SKA
Belmiro Moreira
What's hot
(8)
OpenContrail Implementations
OpenContrail Implementations
OpenPOWER ADG key note
OpenPOWER ADG key note
Introduction of Okinawa Open Laboratory Testbed, OpenStack and SDN Technology...
Introduction of Okinawa Open Laboratory Testbed, OpenStack and SDN Technology...
OpenStack Ottawa Q3 Meetup September 26th 2017
OpenStack Ottawa Q3 Meetup September 26th 2017
Greenplum: Building a Postgres Fabric for Large-Scale Analytical Computation ...
Greenplum: Building a Postgres Fabric for Large-Scale Analytical Computation ...
CERN User Story
CERN User Story
Collect, summarize and notify of OpenStack's log
Collect, summarize and notify of OpenStack's log
Containers on Baremetal and Preemptible VMs at CERN and SKA
Containers on Baremetal and Preemptible VMs at CERN and SKA
Similar to Durability Simulator Design for OpenStack Swift
Future semantic segmentation with convolutional LSTM
Future semantic segmentation with convolutional LSTM
Kyuri Kim
OpenStack Summit Tokyo - Know-how of Challlenging Deploy/Operation NTT DOCOMO...
OpenStack Summit Tokyo - Know-how of Challlenging Deploy/Operation NTT DOCOMO...
Masaaki Nakagawa
Mike Bartley - Innovations for Testing Parallel Software - EuroSTAR 2012
Mike Bartley - Innovations for Testing Parallel Software - EuroSTAR 2012
TEST Huddle
BIRTE-13-Kawashima
BIRTE-13-Kawashima
Hideyuki Kawashima
Understanding performance aspects of etcd and Raft
Understanding performance aspects of etcd and Raft
Hitoshi Mitake
Modeling the IoT with TitanDB and Cassandra
Modeling the IoT with TitanDB and Cassandra
twilmes
HeapStats: Troubleshooting with Serviceability and the New Runtime Monitoring...
HeapStats: Troubleshooting with Serviceability and the New Runtime Monitoring...
Yuji Kubota
Tsinghua University: Two Exemplary Applications in China
Tsinghua University: Two Exemplary Applications in China
DataStax Academy
Icbai 2018 ver_1
Icbai 2018 ver_1
BlackhatGAURAV
Crash course on data streaming (with examples using Apache Flink)
Crash course on data streaming (with examples using Apache Flink)
Vincenzo Gulisano
IRJET - Predicting the Maximum Computational Power of Microprocessors using M...
IRJET - Predicting the Maximum Computational Power of Microprocessors using M...
IRJET Journal
The Pill for Your Migration Hell
The Pill for Your Migration Hell
Databricks
Transport SDN & OpenDaylight Use Cases in Korea
Transport SDN & OpenDaylight Use Cases in Korea
Justin Park
Online learning for low-latency streaming
Online learning for low-latency streaming
Theo Karagkioules
Globecom 2015: Adaptive Raptor Carousel for 802.11
Globecom 2015: Adaptive Raptor Carousel for 802.11
Andrew Nix
The Role of Inter-Controller Traffic in SDN Controllers Placement
The Role of Inter-Controller Traffic in SDN Controllers Placement
Paolo Giaccone
OpenTelemetry For Architects
OpenTelemetry For Architects
Kevin Brockhoff
Digital_system_design_A (1).ppt
Digital_system_design_A (1).ppt
BUCHUPALLIVIMALAREDD2
Basic Tutorial for Robotic Arm
Basic Tutorial for Robotic Arm
Yu Wei Chen
Speeding up information extraction programs: a holistic optimizer and a learn...
Speeding up information extraction programs: a holistic optimizer and a learn...
INRIA-OAK
Similar to Durability Simulator Design for OpenStack Swift
(20)
Future semantic segmentation with convolutional LSTM
Future semantic segmentation with convolutional LSTM
OpenStack Summit Tokyo - Know-how of Challlenging Deploy/Operation NTT DOCOMO...
OpenStack Summit Tokyo - Know-how of Challlenging Deploy/Operation NTT DOCOMO...
Mike Bartley - Innovations for Testing Parallel Software - EuroSTAR 2012
Mike Bartley - Innovations for Testing Parallel Software - EuroSTAR 2012
BIRTE-13-Kawashima
BIRTE-13-Kawashima
Understanding performance aspects of etcd and Raft
Understanding performance aspects of etcd and Raft
Modeling the IoT with TitanDB and Cassandra
Modeling the IoT with TitanDB and Cassandra
HeapStats: Troubleshooting with Serviceability and the New Runtime Monitoring...
HeapStats: Troubleshooting with Serviceability and the New Runtime Monitoring...
Tsinghua University: Two Exemplary Applications in China
Tsinghua University: Two Exemplary Applications in China
Icbai 2018 ver_1
Icbai 2018 ver_1
Crash course on data streaming (with examples using Apache Flink)
Crash course on data streaming (with examples using Apache Flink)
IRJET - Predicting the Maximum Computational Power of Microprocessors using M...
IRJET - Predicting the Maximum Computational Power of Microprocessors using M...
The Pill for Your Migration Hell
The Pill for Your Migration Hell
Transport SDN & OpenDaylight Use Cases in Korea
Transport SDN & OpenDaylight Use Cases in Korea
Online learning for low-latency streaming
Online learning for low-latency streaming
Globecom 2015: Adaptive Raptor Carousel for 802.11
Globecom 2015: Adaptive Raptor Carousel for 802.11
The Role of Inter-Controller Traffic in SDN Controllers Placement
The Role of Inter-Controller Traffic in SDN Controllers Placement
OpenTelemetry For Architects
OpenTelemetry For Architects
Digital_system_design_A (1).ppt
Digital_system_design_A (1).ppt
Basic Tutorial for Robotic Arm
Basic Tutorial for Robotic Arm
Speeding up information extraction programs: a holistic optimizer and a learn...
Speeding up information extraction programs: a holistic optimizer and a learn...
More from Kota Tsuyuzaki
Storlets Project Update for Train
Storlets Project Update for Train
Kota Tsuyuzaki
Project Updates Storlets Denver 2019
Project Updates Storlets Denver 2019
Kota Tsuyuzaki
NVIDIA/deepopsを触ってみた話
NVIDIA/deepopsを触ってみた話
Kota Tsuyuzaki
OpenStack Swift Introduction 2019
OpenStack Swift Introduction 2019
Kota Tsuyuzaki
Case Study: Large Scale Deployment for Machine Learning with Highspeed Storage
Case Study: Large Scale Deployment for Machine Learning with Highspeed Storage
Kota Tsuyuzaki
OpenStack Summit Storlets Project Update Queens
OpenStack Summit Storlets Project Update Queens
Kota Tsuyuzaki
OpenStack Swiftの最新機能とStorlets
OpenStack Swiftの最新機能とStorlets
Kota Tsuyuzaki
Using Storlets/Docker For Large Scale Image Processing
Using Storlets/Docker For Large Scale Image Processing
Kota Tsuyuzaki
OpenStack Swift紹介
OpenStack Swift紹介
Kota Tsuyuzaki
OpenStack Summit Vancouver Swift 報告
OpenStack Summit Vancouver Swift 報告
Kota Tsuyuzaki
Container Listing Update (Liberty Swift Design Summit)
Container Listing Update (Liberty Swift Design Summit)
Kota Tsuyuzaki
日本OpenStackユーザ会 Atlantaサミット報告会 Swift関連報告
日本OpenStackユーザ会 Atlantaサミット報告会 Swift関連報告
Kota Tsuyuzaki
More from Kota Tsuyuzaki
(12)
Storlets Project Update for Train
Storlets Project Update for Train
Project Updates Storlets Denver 2019
Project Updates Storlets Denver 2019
NVIDIA/deepopsを触ってみた話
NVIDIA/deepopsを触ってみた話
OpenStack Swift Introduction 2019
OpenStack Swift Introduction 2019
Case Study: Large Scale Deployment for Machine Learning with Highspeed Storage
Case Study: Large Scale Deployment for Machine Learning with Highspeed Storage
OpenStack Summit Storlets Project Update Queens
OpenStack Summit Storlets Project Update Queens
OpenStack Swiftの最新機能とStorlets
OpenStack Swiftの最新機能とStorlets
Using Storlets/Docker For Large Scale Image Processing
Using Storlets/Docker For Large Scale Image Processing
OpenStack Swift紹介
OpenStack Swift紹介
OpenStack Summit Vancouver Swift 報告
OpenStack Summit Vancouver Swift 報告
Container Listing Update (Liberty Swift Design Summit)
Container Listing Update (Liberty Swift Design Summit)
日本OpenStackユーザ会 Atlantaサミット報告会 Swift関連報告
日本OpenStackユーザ会 Atlantaサミット報告会 Swift関連報告
Recently uploaded
Project Based Learning (A.I).pptx detail explanation
Project Based Learning (A.I).pptx detail explanation
kaushalgiri8080
Russian Call Girls in Karol Bagh Aasnvi ➡️ 8264348440 💋📞 Independent Escort S...
Russian Call Girls in Karol Bagh Aasnvi ➡️ 8264348440 💋📞 Independent Escort S...
soniya singh
Advancing Engineering with AI through the Next Generation of Strategic Projec...
Advancing Engineering with AI through the Next Generation of Strategic Projec...
OnePlan Solutions
Implementing Zero Trust strategy with Azure
Implementing Zero Trust strategy with Azure
Dinusha Kumarasiri
What is Fashion PLM and Why Do You Need It
What is Fashion PLM and Why Do You Need It
Wave PLM
Try MyIntelliAccount Cloud Accounting Software As A Service Solution Risk Fre...
Try MyIntelliAccount Cloud Accounting Software As A Service Solution Risk Fre...
MyIntelliSource, Inc.
Call Girls In Mukherjee Nagar 📱 9999965857 🤩 Delhi 🫦 HOT AND SEXY VVIP 🍎 SE...
Call Girls In Mukherjee Nagar 📱 9999965857 🤩 Delhi 🫦 HOT AND SEXY VVIP 🍎 SE...
Call Girls In Delhi Whatsup 9873940964 Enjoy Unlimited Pleasure
why an Opensea Clone Script might be your perfect match.pdf
why an Opensea Clone Script might be your perfect match.pdf
joe51371421
cybersecurity notes for mca students for learning
cybersecurity notes for mca students for learning
VitsRangannavar
ODSC - Batch to Stream workshop - integration of Apache Spark, Cassandra, Pos...
ODSC - Batch to Stream workshop - integration of Apache Spark, Cassandra, Pos...
Christina Lin
Unit 1.1 Excite Part 1, class 9, cbse...
Unit 1.1 Excite Part 1, class 9, cbse...
aditisharan08
Asset Management Software - Infographic
Asset Management Software - Infographic
Hr365.us smith
Short Story: Unveiling the Reasoning Abilities of Large Language Models by Ke...
Short Story: Unveiling the Reasoning Abilities of Large Language Models by Ke...
kellynguyen01
(Genuine) Escort Service Lucknow | Starting ₹,5K To @25k with A/C 🧑🏽❤️🧑🏻 89...
(Genuine) Escort Service Lucknow | Starting ₹,5K To @25k with A/C 🧑🏽❤️🧑🏻 89...
gurkirankumar98700
EY_Graph Database Powered Sustainability
EY_Graph Database Powered Sustainability
Neo4j
Call Girls in Naraina Delhi 💯Call Us 🔝8264348440🔝
Call Girls in Naraina Delhi 💯Call Us 🔝8264348440🔝
soniya singh
办理学位证(UQ文凭证书)昆士兰大学毕业证成绩单原版一模一样
办理学位证(UQ文凭证书)昆士兰大学毕业证成绩单原版一模一样
umasea
buds n tech IT solutions
buds n tech IT solutions
monugehlot87
The Essentials of Digital Experience Monitoring_ A Comprehensive Guide.pdf
The Essentials of Digital Experience Monitoring_ A Comprehensive Guide.pdf
kalichargn70th171
What is Binary Language? Computer Number Systems
What is Binary Language? Computer Number Systems
JheuzeDellosa
Recently uploaded
(20)
Project Based Learning (A.I).pptx detail explanation
Project Based Learning (A.I).pptx detail explanation
Russian Call Girls in Karol Bagh Aasnvi ➡️ 8264348440 💋📞 Independent Escort S...
Russian Call Girls in Karol Bagh Aasnvi ➡️ 8264348440 💋📞 Independent Escort S...
Advancing Engineering with AI through the Next Generation of Strategic Projec...
Advancing Engineering with AI through the Next Generation of Strategic Projec...
Implementing Zero Trust strategy with Azure
Implementing Zero Trust strategy with Azure
What is Fashion PLM and Why Do You Need It
What is Fashion PLM and Why Do You Need It
Try MyIntelliAccount Cloud Accounting Software As A Service Solution Risk Fre...
Try MyIntelliAccount Cloud Accounting Software As A Service Solution Risk Fre...
Call Girls In Mukherjee Nagar 📱 9999965857 🤩 Delhi 🫦 HOT AND SEXY VVIP 🍎 SE...
Call Girls In Mukherjee Nagar 📱 9999965857 🤩 Delhi 🫦 HOT AND SEXY VVIP 🍎 SE...
why an Opensea Clone Script might be your perfect match.pdf
why an Opensea Clone Script might be your perfect match.pdf
cybersecurity notes for mca students for learning
cybersecurity notes for mca students for learning
ODSC - Batch to Stream workshop - integration of Apache Spark, Cassandra, Pos...
ODSC - Batch to Stream workshop - integration of Apache Spark, Cassandra, Pos...
Unit 1.1 Excite Part 1, class 9, cbse...
Unit 1.1 Excite Part 1, class 9, cbse...
Asset Management Software - Infographic
Asset Management Software - Infographic
Short Story: Unveiling the Reasoning Abilities of Large Language Models by Ke...
Short Story: Unveiling the Reasoning Abilities of Large Language Models by Ke...
(Genuine) Escort Service Lucknow | Starting ₹,5K To @25k with A/C 🧑🏽❤️🧑🏻 89...
(Genuine) Escort Service Lucknow | Starting ₹,5K To @25k with A/C 🧑🏽❤️🧑🏻 89...
EY_Graph Database Powered Sustainability
EY_Graph Database Powered Sustainability
Call Girls in Naraina Delhi 💯Call Us 🔝8264348440🔝
Call Girls in Naraina Delhi 💯Call Us 🔝8264348440🔝
办理学位证(UQ文凭证书)昆士兰大学毕业证成绩单原版一模一样
办理学位证(UQ文凭证书)昆士兰大学毕业证成绩单原版一模一样
buds n tech IT solutions
buds n tech IT solutions
The Essentials of Digital Experience Monitoring_ A Comprehensive Guide.pdf
The Essentials of Digital Experience Monitoring_ A Comprehensive Guide.pdf
What is Binary Language? Computer Number Systems
What is Binary Language? Computer Number Systems
Durability Simulator Design for OpenStack Swift
1.
Copyright©2014 NTT Corp.
All Rights Reserved. Durability Simulator Design for OpenStack Swift (Interactive Durability Calculation Tools) Kota Tsuyuzaki [IRC: kota_] tsuyuzaki.kota@lab.ntt.co.jp NTT Software Innovation Center Copyright(c)2009-2014 NTT CORPORATION. All Rights Reserved.
2.
2 Copyright©2014 NTT
Corp. All Rights Reserved. NTT Confidential •Goal & Benefits •How to calculate? •Demo Outline Etherpad: https://etherpad.openstack.org/p/kilo-swift-durability-simulator
3.
3 Copyright©2014 NTT
Corp. All Rights Reserved. NTT Confidential Issue User I wanna build a durable object storage system by using OpenStack Swift. I wanna know also the durability to confirm it will be enough for our SLA.
4.
4 Copyright©2014 NTT
Corp. All Rights Reserved. NTT Confidential Issue User Provider A Provider B Provider C Hey, guys. Could you tell me the Swift system architecture and its storage durability you support. OpenStack Providers
5.
5 Copyright©2014 NTT
Corp. All Rights Reserved. NTT Confidential Issue User Provider A Provider B Provider C A: 7-9s durability with 3 copies B: 9-9s durability with 3 copies C: 11-9s durability with 3 copies WHAT’S HAPPEN!? WHICH IS CORRECT? OpenStack Providers
6.
6 Copyright©2014 NTT
Corp. All Rights Reserved. NTT Confidential •Goal •Building durability calculation tools supported (or recommended) by Swift community •Enabling to get the calculation result easily from both specs of system component HWs and swift configures. (e.g. # of disks, size of each disk, # of partitions) •Benefits •Swift Administrators (almost beginners) can find their own system durability easily •Enable to standardize the calculation definition among Swift providers •Swift Users can choose the policy for their use case (Replica? EC? Which # of parities are best for you?) Goal & Benefits
7.
7 Copyright©2014 NTT
Corp. All Rights Reserved. NTT Confidential How to calculate the durability?
8.
8 Copyright©2014 NTT
Corp. All Rights Reserved. NTT Confidential For Replica Case
9.
9 Copyright©2014 NTT
Corp. All Rights Reserved. NTT Confidential •Calculation Using Markov Model (Markov Process) •2 Replica -> k = 1, m = 1 •i.e. Data Lost with 2 Fragments •3 Replica -> k = 1, m = 2 •i.e. Data Lost with 3 Fragments •Reference: •[1]: "Reliability Mechanisms for Very Large Storage Systems" •http://www.ssrc.ucsc.edu/Papers/xin-mss03.pdf How to Calculate EC Durability? [1]
10.
10 Copyright©2014 NTT
Corp. All Rights Reserved. NTT Confidential •Redundancy Set[1]: •Definition •A block group composed of data blocks or object and their associated replicas or parity blocks. A single redundancy set will typically contain 1MB to 1TB, though we expect that redundancy sets will be at least 1GB to minimize bookkeeping overhead and reduce the likelihood that two redundancy sets will be stored on the same set of object storage system. •Assuming a Reduandancy Set as a Partition Consideration for Swift’s Partition Ring MD5*(URL) = index partitions idx Copy 1 Copy 2 Copy 3 0 1 5 7 … … … … 8 3 2 6 Partition table from part to device id. From [1]
11.
11 Copyright©2014 NTT
Corp. All Rights Reserved. NTT Confidential •Definition: •Absorbing State: The end state in the state transition model. •P: Transition Probability Matrix Markov Process (1) Absorbing State Temporary State P=푄푈 푂퐼 ퟏ−ퟐ흁ퟐ흁ퟎ 풗ퟏ−(흁+풗)흁 ퟎퟎퟏ Q: Transition Probability Matrix among Temporary State U: Probability Matrix from Temporary State into Absorbing State O: Zero Matrix、I: Identity Matrix State0 State1 State2
12.
12 Copyright©2014 NTT
Corp. All Rights Reserved. NTT Confidential •Time (t) Limitation of State Transition Matrix (P) shows average # of state transition (M) from initial state to absorbing state •MTTDL (Time to be absorbing state) calculated from sum of each rows in MN Markov Process (2) 퐥퐢퐦 풕→∞ 푷풕=ퟎ푴푼 ퟎ푰 M = (I-Q)-1 MTTDLrs = M ퟏ ⋮ ퟏ P= ퟏ−ퟐ흁ퟐ흁ퟎ 풗ퟏ−(흁+풗)흁 ퟎퟎퟏ ퟏ ퟐ흁 흁+풗 흁 ퟐ 풗 흁 ퟐ State Transition Matrix for 2 replica M MTTDLrs ퟏ ퟐ흁ퟐ ퟑ흁+풗 ퟐ흁+풗 Durability = 1 – N/ MTTDLrs Probability for Data Lost Durability 1 - 2푵흁ퟐ ퟏ ퟑ흁+풗 ퟏ ퟐ흁+풗
13.
13 Copyright©2014 NTT
Corp. All Rights Reserved. NTT Confidential For EC Case
14.
14 Copyright©2014 NTT
Corp. All Rights Reserved. NTT Confidential •Object Size(bytes): n •# of Sliced Raw Objects: k •# of Parities: m •Total # of Fragments: k + m •Fragment Size(bytes): n / k (+ checksum) •Total Stored Size (bytes): Fragment Size * (k + m) Erasure Code Definition object Data fragment Data fragment parity fragment parity fragment … … k m encode decode Terminology Reference: http://specs.openstack.org/openstack/ swift-specs/specs/swift/erasure_coding.html
15.
15 Copyright©2014 NTT
Corp. All Rights Reserved. NTT Confidential •Basic Idea •Expansion of Durability Calculation for Replica Model •Calculation Using Markov Model (Markov Process) •Replica Model based on Markov Process: •2 Replica -> k = 1, m = 1 •i.e. Data Lost with 2 Fragments •3 Replica -> k = 1, m = 2 •i.e. Data Lost with 3 Fragments How to Calculate EC Durability? [1] ※ Markov Process works to calculate the durability with matrix calculation. [3]
16.
16 Copyright©2014 NTT
Corp. All Rights Reserved. NTT Confidential •Algorithms •State: Status (exists or lost) for All fragments •Each state is transferred by constant probability •μ = Disk Failure Rate, v = Fragments Repair Rate •Each Rate related to # of Fragments •E.g. RAID related to # of Devices •Extract States to m + 1 (i.e. data lost) Durability Calculation Algorithms 0 1 m-1 m … m+1 state transitions for “m” parities EC D = # of Devices (RAID5) N = k + m (N fragments located in the system) -Nμ v Nμ -(N-1)μ-v (N-m)μ mv (N-(m-1))μ -(N-(m-1))μ-mv
17.
17 Copyright©2014 NTT
Corp. All Rights Reserved. NTT Confidential Demo
18.
18 Copyright©2014 NTT
Corp. All Rights Reserved. NTT Confidential Demo
19.
19 Copyright©2014 NTT
Corp. All Rights Reserved. NTT Confidential Demo
20.
20 Copyright©2014 NTT
Corp. All Rights Reserved. NTT Confidential Kota Tsuyuzaki [IRC: kota_] tsuyuzaki.kota@lab.ntt.co.jp NTT Software Innovation Center Questions? Etherpad: https://etherpad.openstack.org/p/kilo-swift-durability-simulator
Download now