Submit Search
Upload
Durability Simulator Design for OpenStack Swift
•
1 like
•
522 views
Kota Tsuyuzaki
Follow
Slides for OpenStack 2014 Paris Design Summit.
Read less
Read more
Software
Report
Share
Report
Share
1 of 20
Download now
Download to read offline
Recommended
Global EC Cluster Updates (OpenStack Mitaka Swift Design Summit)
Global EC Cluster Updates (OpenStack Mitaka Swift Design Summit)
Kota Tsuyuzaki
More Efficient Object Replication in OpenStack Summit Juno
More Efficient Object Replication in OpenStack Summit Juno
Kota Tsuyuzaki
NFV Infrastructure Manager with High Performance Software Switch Lagopus
NFV Infrastructure Manager with High Performance Software Switch Lagopus
Hirofumi Ichihara
Contiki OS Research Projects Guidance
Contiki OS Research Projects Guidance
Network Simulation Tools
OpenContrail Experience tcp cloud OpenStack Summit Tokyo
OpenContrail Experience tcp cloud OpenStack Summit Tokyo
Jakub Pavlik
NTTs Journey with Openstack-final
NTTs Journey with Openstack-final
shintaro mizuno
CFI 2015 - Flow-centric Visibility Tools for OF@TEIN
CFI 2015 - Flow-centric Visibility Tools for OF@TEIN
GIST (Gwangju Institute of Science and Technology)
Building the IoT - Coding Serbia 2015
Building the IoT - Coding Serbia 2015
Benjamin Cabé
Recommended
Global EC Cluster Updates (OpenStack Mitaka Swift Design Summit)
Global EC Cluster Updates (OpenStack Mitaka Swift Design Summit)
Kota Tsuyuzaki
More Efficient Object Replication in OpenStack Summit Juno
More Efficient Object Replication in OpenStack Summit Juno
Kota Tsuyuzaki
NFV Infrastructure Manager with High Performance Software Switch Lagopus
NFV Infrastructure Manager with High Performance Software Switch Lagopus
Hirofumi Ichihara
Contiki OS Research Projects Guidance
Contiki OS Research Projects Guidance
Network Simulation Tools
OpenContrail Experience tcp cloud OpenStack Summit Tokyo
OpenContrail Experience tcp cloud OpenStack Summit Tokyo
Jakub Pavlik
NTTs Journey with Openstack-final
NTTs Journey with Openstack-final
shintaro mizuno
CFI 2015 - Flow-centric Visibility Tools for OF@TEIN
CFI 2015 - Flow-centric Visibility Tools for OF@TEIN
GIST (Gwangju Institute of Science and Technology)
Building the IoT - Coding Serbia 2015
Building the IoT - Coding Serbia 2015
Benjamin Cabé
OpenContrail Implementations
OpenContrail Implementations
Jakub Pavlik
OpenPOWER ADG key note
OpenPOWER ADG key note
Ganesan Narayanasamy
Introduction of Okinawa Open Laboratory Testbed, OpenStack and SDN Technology...
Introduction of Okinawa Open Laboratory Testbed, OpenStack and SDN Technology...
Takashi Torii
OpenStack Ottawa Q3 Meetup September 26th 2017
OpenStack Ottawa Q3 Meetup September 26th 2017
Stacy Véronneau
Greenplum: Building a Postgres Fabric for Large-Scale Analytical Computation ...
Greenplum: Building a Postgres Fabric for Large-Scale Analytical Computation ...
VMware Tanzu
CERN User Story
CERN User Story
Tim Bell
Collect, summarize and notify of OpenStack's log
Collect, summarize and notify of OpenStack's log
NTT Communications Technology Development
Containers on Baremetal and Preemptible VMs at CERN and SKA
Containers on Baremetal and Preemptible VMs at CERN and SKA
Belmiro Moreira
Future semantic segmentation with convolutional LSTM
Future semantic segmentation with convolutional LSTM
Kyuri Kim
OpenStack Summit Tokyo - Know-how of Challlenging Deploy/Operation NTT DOCOMO...
OpenStack Summit Tokyo - Know-how of Challlenging Deploy/Operation NTT DOCOMO...
Masaaki Nakagawa
Mike Bartley - Innovations for Testing Parallel Software - EuroSTAR 2012
Mike Bartley - Innovations for Testing Parallel Software - EuroSTAR 2012
TEST Huddle
BIRTE-13-Kawashima
BIRTE-13-Kawashima
Hideyuki Kawashima
Understanding performance aspects of etcd and Raft
Understanding performance aspects of etcd and Raft
Hitoshi Mitake
Modeling the IoT with TitanDB and Cassandra
Modeling the IoT with TitanDB and Cassandra
twilmes
HeapStats: Troubleshooting with Serviceability and the New Runtime Monitoring...
HeapStats: Troubleshooting with Serviceability and the New Runtime Monitoring...
Yuji Kubota
Tsinghua University: Two Exemplary Applications in China
Tsinghua University: Two Exemplary Applications in China
DataStax Academy
Icbai 2018 ver_1
Icbai 2018 ver_1
BlackhatGAURAV
Crash course on data streaming (with examples using Apache Flink)
Crash course on data streaming (with examples using Apache Flink)
Vincenzo Gulisano
IRJET - Predicting the Maximum Computational Power of Microprocessors using M...
IRJET - Predicting the Maximum Computational Power of Microprocessors using M...
IRJET Journal
The Pill for Your Migration Hell
The Pill for Your Migration Hell
Databricks
Transport SDN & OpenDaylight Use Cases in Korea
Transport SDN & OpenDaylight Use Cases in Korea
Justin Park
Online learning for low-latency streaming
Online learning for low-latency streaming
Theo Karagkioules
More Related Content
What's hot
OpenContrail Implementations
OpenContrail Implementations
Jakub Pavlik
OpenPOWER ADG key note
OpenPOWER ADG key note
Ganesan Narayanasamy
Introduction of Okinawa Open Laboratory Testbed, OpenStack and SDN Technology...
Introduction of Okinawa Open Laboratory Testbed, OpenStack and SDN Technology...
Takashi Torii
OpenStack Ottawa Q3 Meetup September 26th 2017
OpenStack Ottawa Q3 Meetup September 26th 2017
Stacy Véronneau
Greenplum: Building a Postgres Fabric for Large-Scale Analytical Computation ...
Greenplum: Building a Postgres Fabric for Large-Scale Analytical Computation ...
VMware Tanzu
CERN User Story
CERN User Story
Tim Bell
Collect, summarize and notify of OpenStack's log
Collect, summarize and notify of OpenStack's log
NTT Communications Technology Development
Containers on Baremetal and Preemptible VMs at CERN and SKA
Containers on Baremetal and Preemptible VMs at CERN and SKA
Belmiro Moreira
What's hot
(8)
OpenContrail Implementations
OpenContrail Implementations
OpenPOWER ADG key note
OpenPOWER ADG key note
Introduction of Okinawa Open Laboratory Testbed, OpenStack and SDN Technology...
Introduction of Okinawa Open Laboratory Testbed, OpenStack and SDN Technology...
OpenStack Ottawa Q3 Meetup September 26th 2017
OpenStack Ottawa Q3 Meetup September 26th 2017
Greenplum: Building a Postgres Fabric for Large-Scale Analytical Computation ...
Greenplum: Building a Postgres Fabric for Large-Scale Analytical Computation ...
CERN User Story
CERN User Story
Collect, summarize and notify of OpenStack's log
Collect, summarize and notify of OpenStack's log
Containers on Baremetal and Preemptible VMs at CERN and SKA
Containers on Baremetal and Preemptible VMs at CERN and SKA
Similar to Durability Simulator Design for OpenStack Swift
Future semantic segmentation with convolutional LSTM
Future semantic segmentation with convolutional LSTM
Kyuri Kim
OpenStack Summit Tokyo - Know-how of Challlenging Deploy/Operation NTT DOCOMO...
OpenStack Summit Tokyo - Know-how of Challlenging Deploy/Operation NTT DOCOMO...
Masaaki Nakagawa
Mike Bartley - Innovations for Testing Parallel Software - EuroSTAR 2012
Mike Bartley - Innovations for Testing Parallel Software - EuroSTAR 2012
TEST Huddle
BIRTE-13-Kawashima
BIRTE-13-Kawashima
Hideyuki Kawashima
Understanding performance aspects of etcd and Raft
Understanding performance aspects of etcd and Raft
Hitoshi Mitake
Modeling the IoT with TitanDB and Cassandra
Modeling the IoT with TitanDB and Cassandra
twilmes
HeapStats: Troubleshooting with Serviceability and the New Runtime Monitoring...
HeapStats: Troubleshooting with Serviceability and the New Runtime Monitoring...
Yuji Kubota
Tsinghua University: Two Exemplary Applications in China
Tsinghua University: Two Exemplary Applications in China
DataStax Academy
Icbai 2018 ver_1
Icbai 2018 ver_1
BlackhatGAURAV
Crash course on data streaming (with examples using Apache Flink)
Crash course on data streaming (with examples using Apache Flink)
Vincenzo Gulisano
IRJET - Predicting the Maximum Computational Power of Microprocessors using M...
IRJET - Predicting the Maximum Computational Power of Microprocessors using M...
IRJET Journal
The Pill for Your Migration Hell
The Pill for Your Migration Hell
Databricks
Transport SDN & OpenDaylight Use Cases in Korea
Transport SDN & OpenDaylight Use Cases in Korea
Justin Park
Online learning for low-latency streaming
Online learning for low-latency streaming
Theo Karagkioules
Globecom 2015: Adaptive Raptor Carousel for 802.11
Globecom 2015: Adaptive Raptor Carousel for 802.11
Andrew Nix
The Role of Inter-Controller Traffic in SDN Controllers Placement
The Role of Inter-Controller Traffic in SDN Controllers Placement
Paolo Giaccone
OpenTelemetry For Architects
OpenTelemetry For Architects
Kevin Brockhoff
Digital_system_design_A (1).ppt
Digital_system_design_A (1).ppt
BUCHUPALLIVIMALAREDD2
Basic Tutorial for Robotic Arm
Basic Tutorial for Robotic Arm
Yu Wei Chen
Speeding up information extraction programs: a holistic optimizer and a learn...
Speeding up information extraction programs: a holistic optimizer and a learn...
INRIA-OAK
Similar to Durability Simulator Design for OpenStack Swift
(20)
Future semantic segmentation with convolutional LSTM
Future semantic segmentation with convolutional LSTM
OpenStack Summit Tokyo - Know-how of Challlenging Deploy/Operation NTT DOCOMO...
OpenStack Summit Tokyo - Know-how of Challlenging Deploy/Operation NTT DOCOMO...
Mike Bartley - Innovations for Testing Parallel Software - EuroSTAR 2012
Mike Bartley - Innovations for Testing Parallel Software - EuroSTAR 2012
BIRTE-13-Kawashima
BIRTE-13-Kawashima
Understanding performance aspects of etcd and Raft
Understanding performance aspects of etcd and Raft
Modeling the IoT with TitanDB and Cassandra
Modeling the IoT with TitanDB and Cassandra
HeapStats: Troubleshooting with Serviceability and the New Runtime Monitoring...
HeapStats: Troubleshooting with Serviceability and the New Runtime Monitoring...
Tsinghua University: Two Exemplary Applications in China
Tsinghua University: Two Exemplary Applications in China
Icbai 2018 ver_1
Icbai 2018 ver_1
Crash course on data streaming (with examples using Apache Flink)
Crash course on data streaming (with examples using Apache Flink)
IRJET - Predicting the Maximum Computational Power of Microprocessors using M...
IRJET - Predicting the Maximum Computational Power of Microprocessors using M...
The Pill for Your Migration Hell
The Pill for Your Migration Hell
Transport SDN & OpenDaylight Use Cases in Korea
Transport SDN & OpenDaylight Use Cases in Korea
Online learning for low-latency streaming
Online learning for low-latency streaming
Globecom 2015: Adaptive Raptor Carousel for 802.11
Globecom 2015: Adaptive Raptor Carousel for 802.11
The Role of Inter-Controller Traffic in SDN Controllers Placement
The Role of Inter-Controller Traffic in SDN Controllers Placement
OpenTelemetry For Architects
OpenTelemetry For Architects
Digital_system_design_A (1).ppt
Digital_system_design_A (1).ppt
Basic Tutorial for Robotic Arm
Basic Tutorial for Robotic Arm
Speeding up information extraction programs: a holistic optimizer and a learn...
Speeding up information extraction programs: a holistic optimizer and a learn...
More from Kota Tsuyuzaki
Storlets Project Update for Train
Storlets Project Update for Train
Kota Tsuyuzaki
Project Updates Storlets Denver 2019
Project Updates Storlets Denver 2019
Kota Tsuyuzaki
NVIDIA/deepopsを触ってみた話
NVIDIA/deepopsを触ってみた話
Kota Tsuyuzaki
OpenStack Swift Introduction 2019
OpenStack Swift Introduction 2019
Kota Tsuyuzaki
Case Study: Large Scale Deployment for Machine Learning with Highspeed Storage
Case Study: Large Scale Deployment for Machine Learning with Highspeed Storage
Kota Tsuyuzaki
OpenStack Summit Storlets Project Update Queens
OpenStack Summit Storlets Project Update Queens
Kota Tsuyuzaki
OpenStack Swiftの最新機能とStorlets
OpenStack Swiftの最新機能とStorlets
Kota Tsuyuzaki
Using Storlets/Docker For Large Scale Image Processing
Using Storlets/Docker For Large Scale Image Processing
Kota Tsuyuzaki
OpenStack Swift紹介
OpenStack Swift紹介
Kota Tsuyuzaki
OpenStack Summit Vancouver Swift 報告
OpenStack Summit Vancouver Swift 報告
Kota Tsuyuzaki
Container Listing Update (Liberty Swift Design Summit)
Container Listing Update (Liberty Swift Design Summit)
Kota Tsuyuzaki
日本OpenStackユーザ会 Atlantaサミット報告会 Swift関連報告
日本OpenStackユーザ会 Atlantaサミット報告会 Swift関連報告
Kota Tsuyuzaki
More from Kota Tsuyuzaki
(12)
Storlets Project Update for Train
Storlets Project Update for Train
Project Updates Storlets Denver 2019
Project Updates Storlets Denver 2019
NVIDIA/deepopsを触ってみた話
NVIDIA/deepopsを触ってみた話
OpenStack Swift Introduction 2019
OpenStack Swift Introduction 2019
Case Study: Large Scale Deployment for Machine Learning with Highspeed Storage
Case Study: Large Scale Deployment for Machine Learning with Highspeed Storage
OpenStack Summit Storlets Project Update Queens
OpenStack Summit Storlets Project Update Queens
OpenStack Swiftの最新機能とStorlets
OpenStack Swiftの最新機能とStorlets
Using Storlets/Docker For Large Scale Image Processing
Using Storlets/Docker For Large Scale Image Processing
OpenStack Swift紹介
OpenStack Swift紹介
OpenStack Summit Vancouver Swift 報告
OpenStack Summit Vancouver Swift 報告
Container Listing Update (Liberty Swift Design Summit)
Container Listing Update (Liberty Swift Design Summit)
日本OpenStackユーザ会 Atlantaサミット報告会 Swift関連報告
日本OpenStackユーザ会 Atlantaサミット報告会 Swift関連報告
Recently uploaded
Microsoft AI Transformation Partner Playbook.pdf
Microsoft AI Transformation Partner Playbook.pdf
Willy Marroquin (WillyDevNET)
call girls in Vaishali (Ghaziabad) 🔝 >༒8448380779 🔝 genuine Escort Service 🔝✔️✔️
call girls in Vaishali (Ghaziabad) 🔝 >༒8448380779 🔝 genuine Escort Service 🔝✔️✔️
Delhi Call girls
+971565801893>>SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHAB...
+971565801893>>SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHAB...
Health
Steps To Getting Up And Running Quickly With MyTimeClock Employee Scheduling ...
Steps To Getting Up And Running Quickly With MyTimeClock Employee Scheduling ...
MyIntelliSource, Inc.
How To Troubleshoot Collaboration Apps for the Modern Connected Worker
How To Troubleshoot Collaboration Apps for the Modern Connected Worker
ThousandEyes
Software Quality Assurance Interview Questions
Software Quality Assurance Interview Questions
Arshad QA
Optimizing AI for immediate response in Smart CCTV
Optimizing AI for immediate response in Smart CCTV
shikhaohhpro
Unveiling the Tech Salsa of LAMs with Janus in Real-Time Applications
Unveiling the Tech Salsa of LAMs with Janus in Real-Time Applications
Alberto González Trastoy
5 Signs You Need a Fashion PLM Software.pdf
5 Signs You Need a Fashion PLM Software.pdf
Wave PLM
Right Money Management App For Your Financial Goals
Right Money Management App For Your Financial Goals
Jhone kinadey
Hand gesture recognition PROJECT PPT.pptx
Hand gesture recognition PROJECT PPT.pptx
bodapatigopi8531
SyndBuddy AI 2k Review 2024: Revolutionizing Content Syndication with AI
SyndBuddy AI 2k Review 2024: Revolutionizing Content Syndication with AI
ABDERRAOUF MEHENNI
Tech Tuesday-Harness the Power of Effective Resource Planning with OnePlan’s ...
Tech Tuesday-Harness the Power of Effective Resource Planning with OnePlan’s ...
OnePlan Solutions
Vip Call Girls Noida ➡️ Delhi ➡️ 9999965857 No Advance 24HRS Live
Vip Call Girls Noida ➡️ Delhi ➡️ 9999965857 No Advance 24HRS Live
Call Girls In Delhi Whatsup 9873940964 Enjoy Unlimited Pleasure
CALL ON ➥8923113531 🔝Call Girls Kakori Lucknow best sexual service Online ☂️
CALL ON ➥8923113531 🔝Call Girls Kakori Lucknow best sexual service Online ☂️
anilsa9823
Unlocking the Future of AI Agents with Large Language Models
Unlocking the Future of AI Agents with Large Language Models
aagamshah0812
Short Story: Unveiling the Reasoning Abilities of Large Language Models by Ke...
Short Story: Unveiling the Reasoning Abilities of Large Language Models by Ke...
kellynguyen01
HR Software Buyers Guide in 2024 - HRSoftware.com
HR Software Buyers Guide in 2024 - HRSoftware.com
Fatema Valibhai
How To Use Server-Side Rendering with Nuxt.js
How To Use Server-Side Rendering with Nuxt.js
Andolasoft Inc
The Real-World Challenges of Medical Device Cybersecurity- Mitigating Vulnera...
The Real-World Challenges of Medical Device Cybersecurity- Mitigating Vulnera...
ICS
Recently uploaded
(20)
Microsoft AI Transformation Partner Playbook.pdf
Microsoft AI Transformation Partner Playbook.pdf
call girls in Vaishali (Ghaziabad) 🔝 >༒8448380779 🔝 genuine Escort Service 🔝✔️✔️
call girls in Vaishali (Ghaziabad) 🔝 >༒8448380779 🔝 genuine Escort Service 🔝✔️✔️
+971565801893>>SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHAB...
+971565801893>>SAFE AND ORIGINAL ABORTION PILLS FOR SALE IN DUBAI AND ABUDHAB...
Steps To Getting Up And Running Quickly With MyTimeClock Employee Scheduling ...
Steps To Getting Up And Running Quickly With MyTimeClock Employee Scheduling ...
How To Troubleshoot Collaboration Apps for the Modern Connected Worker
How To Troubleshoot Collaboration Apps for the Modern Connected Worker
Software Quality Assurance Interview Questions
Software Quality Assurance Interview Questions
Optimizing AI for immediate response in Smart CCTV
Optimizing AI for immediate response in Smart CCTV
Unveiling the Tech Salsa of LAMs with Janus in Real-Time Applications
Unveiling the Tech Salsa of LAMs with Janus in Real-Time Applications
5 Signs You Need a Fashion PLM Software.pdf
5 Signs You Need a Fashion PLM Software.pdf
Right Money Management App For Your Financial Goals
Right Money Management App For Your Financial Goals
Hand gesture recognition PROJECT PPT.pptx
Hand gesture recognition PROJECT PPT.pptx
SyndBuddy AI 2k Review 2024: Revolutionizing Content Syndication with AI
SyndBuddy AI 2k Review 2024: Revolutionizing Content Syndication with AI
Tech Tuesday-Harness the Power of Effective Resource Planning with OnePlan’s ...
Tech Tuesday-Harness the Power of Effective Resource Planning with OnePlan’s ...
Vip Call Girls Noida ➡️ Delhi ➡️ 9999965857 No Advance 24HRS Live
Vip Call Girls Noida ➡️ Delhi ➡️ 9999965857 No Advance 24HRS Live
CALL ON ➥8923113531 🔝Call Girls Kakori Lucknow best sexual service Online ☂️
CALL ON ➥8923113531 🔝Call Girls Kakori Lucknow best sexual service Online ☂️
Unlocking the Future of AI Agents with Large Language Models
Unlocking the Future of AI Agents with Large Language Models
Short Story: Unveiling the Reasoning Abilities of Large Language Models by Ke...
Short Story: Unveiling the Reasoning Abilities of Large Language Models by Ke...
HR Software Buyers Guide in 2024 - HRSoftware.com
HR Software Buyers Guide in 2024 - HRSoftware.com
How To Use Server-Side Rendering with Nuxt.js
How To Use Server-Side Rendering with Nuxt.js
The Real-World Challenges of Medical Device Cybersecurity- Mitigating Vulnera...
The Real-World Challenges of Medical Device Cybersecurity- Mitigating Vulnera...
Durability Simulator Design for OpenStack Swift
1.
Copyright©2014 NTT Corp.
All Rights Reserved. Durability Simulator Design for OpenStack Swift (Interactive Durability Calculation Tools) Kota Tsuyuzaki [IRC: kota_] tsuyuzaki.kota@lab.ntt.co.jp NTT Software Innovation Center Copyright(c)2009-2014 NTT CORPORATION. All Rights Reserved.
2.
2 Copyright©2014 NTT
Corp. All Rights Reserved. NTT Confidential •Goal & Benefits •How to calculate? •Demo Outline Etherpad: https://etherpad.openstack.org/p/kilo-swift-durability-simulator
3.
3 Copyright©2014 NTT
Corp. All Rights Reserved. NTT Confidential Issue User I wanna build a durable object storage system by using OpenStack Swift. I wanna know also the durability to confirm it will be enough for our SLA.
4.
4 Copyright©2014 NTT
Corp. All Rights Reserved. NTT Confidential Issue User Provider A Provider B Provider C Hey, guys. Could you tell me the Swift system architecture and its storage durability you support. OpenStack Providers
5.
5 Copyright©2014 NTT
Corp. All Rights Reserved. NTT Confidential Issue User Provider A Provider B Provider C A: 7-9s durability with 3 copies B: 9-9s durability with 3 copies C: 11-9s durability with 3 copies WHAT’S HAPPEN!? WHICH IS CORRECT? OpenStack Providers
6.
6 Copyright©2014 NTT
Corp. All Rights Reserved. NTT Confidential •Goal •Building durability calculation tools supported (or recommended) by Swift community •Enabling to get the calculation result easily from both specs of system component HWs and swift configures. (e.g. # of disks, size of each disk, # of partitions) •Benefits •Swift Administrators (almost beginners) can find their own system durability easily •Enable to standardize the calculation definition among Swift providers •Swift Users can choose the policy for their use case (Replica? EC? Which # of parities are best for you?) Goal & Benefits
7.
7 Copyright©2014 NTT
Corp. All Rights Reserved. NTT Confidential How to calculate the durability?
8.
8 Copyright©2014 NTT
Corp. All Rights Reserved. NTT Confidential For Replica Case
9.
9 Copyright©2014 NTT
Corp. All Rights Reserved. NTT Confidential •Calculation Using Markov Model (Markov Process) •2 Replica -> k = 1, m = 1 •i.e. Data Lost with 2 Fragments •3 Replica -> k = 1, m = 2 •i.e. Data Lost with 3 Fragments •Reference: •[1]: "Reliability Mechanisms for Very Large Storage Systems" •http://www.ssrc.ucsc.edu/Papers/xin-mss03.pdf How to Calculate EC Durability? [1]
10.
10 Copyright©2014 NTT
Corp. All Rights Reserved. NTT Confidential •Redundancy Set[1]: •Definition •A block group composed of data blocks or object and their associated replicas or parity blocks. A single redundancy set will typically contain 1MB to 1TB, though we expect that redundancy sets will be at least 1GB to minimize bookkeeping overhead and reduce the likelihood that two redundancy sets will be stored on the same set of object storage system. •Assuming a Reduandancy Set as a Partition Consideration for Swift’s Partition Ring MD5*(URL) = index partitions idx Copy 1 Copy 2 Copy 3 0 1 5 7 … … … … 8 3 2 6 Partition table from part to device id. From [1]
11.
11 Copyright©2014 NTT
Corp. All Rights Reserved. NTT Confidential •Definition: •Absorbing State: The end state in the state transition model. •P: Transition Probability Matrix Markov Process (1) Absorbing State Temporary State P=푄푈 푂퐼 ퟏ−ퟐ흁ퟐ흁ퟎ 풗ퟏ−(흁+풗)흁 ퟎퟎퟏ Q: Transition Probability Matrix among Temporary State U: Probability Matrix from Temporary State into Absorbing State O: Zero Matrix、I: Identity Matrix State0 State1 State2
12.
12 Copyright©2014 NTT
Corp. All Rights Reserved. NTT Confidential •Time (t) Limitation of State Transition Matrix (P) shows average # of state transition (M) from initial state to absorbing state •MTTDL (Time to be absorbing state) calculated from sum of each rows in MN Markov Process (2) 퐥퐢퐦 풕→∞ 푷풕=ퟎ푴푼 ퟎ푰 M = (I-Q)-1 MTTDLrs = M ퟏ ⋮ ퟏ P= ퟏ−ퟐ흁ퟐ흁ퟎ 풗ퟏ−(흁+풗)흁 ퟎퟎퟏ ퟏ ퟐ흁 흁+풗 흁 ퟐ 풗 흁 ퟐ State Transition Matrix for 2 replica M MTTDLrs ퟏ ퟐ흁ퟐ ퟑ흁+풗 ퟐ흁+풗 Durability = 1 – N/ MTTDLrs Probability for Data Lost Durability 1 - 2푵흁ퟐ ퟏ ퟑ흁+풗 ퟏ ퟐ흁+풗
13.
13 Copyright©2014 NTT
Corp. All Rights Reserved. NTT Confidential For EC Case
14.
14 Copyright©2014 NTT
Corp. All Rights Reserved. NTT Confidential •Object Size(bytes): n •# of Sliced Raw Objects: k •# of Parities: m •Total # of Fragments: k + m •Fragment Size(bytes): n / k (+ checksum) •Total Stored Size (bytes): Fragment Size * (k + m) Erasure Code Definition object Data fragment Data fragment parity fragment parity fragment … … k m encode decode Terminology Reference: http://specs.openstack.org/openstack/ swift-specs/specs/swift/erasure_coding.html
15.
15 Copyright©2014 NTT
Corp. All Rights Reserved. NTT Confidential •Basic Idea •Expansion of Durability Calculation for Replica Model •Calculation Using Markov Model (Markov Process) •Replica Model based on Markov Process: •2 Replica -> k = 1, m = 1 •i.e. Data Lost with 2 Fragments •3 Replica -> k = 1, m = 2 •i.e. Data Lost with 3 Fragments How to Calculate EC Durability? [1] ※ Markov Process works to calculate the durability with matrix calculation. [3]
16.
16 Copyright©2014 NTT
Corp. All Rights Reserved. NTT Confidential •Algorithms •State: Status (exists or lost) for All fragments •Each state is transferred by constant probability •μ = Disk Failure Rate, v = Fragments Repair Rate •Each Rate related to # of Fragments •E.g. RAID related to # of Devices •Extract States to m + 1 (i.e. data lost) Durability Calculation Algorithms 0 1 m-1 m … m+1 state transitions for “m” parities EC D = # of Devices (RAID5) N = k + m (N fragments located in the system) -Nμ v Nμ -(N-1)μ-v (N-m)μ mv (N-(m-1))μ -(N-(m-1))μ-mv
17.
17 Copyright©2014 NTT
Corp. All Rights Reserved. NTT Confidential Demo
18.
18 Copyright©2014 NTT
Corp. All Rights Reserved. NTT Confidential Demo
19.
19 Copyright©2014 NTT
Corp. All Rights Reserved. NTT Confidential Demo
20.
20 Copyright©2014 NTT
Corp. All Rights Reserved. NTT Confidential Kota Tsuyuzaki [IRC: kota_] tsuyuzaki.kota@lab.ntt.co.jp NTT Software Innovation Center Questions? Etherpad: https://etherpad.openstack.org/p/kilo-swift-durability-simulator
Download now