SlideShare a Scribd company logo

Hot and cold data storage

•
•
Rohit Arora
Rohit Arora

In laymen's term, this is a file system that realizes hot and cold data identification, moving cold data to secondary storage (dropbox here), retrieving cold data from secondary storage as an essential activity. In this project we implemented this file system and handled all the general and specific cases to allow seamless transfer of data from hot to cold and cold to hot.

Engineering
1 of 39
Download to read offline
HOT COLD Unified Virtual File System For Hot & Cold Data Storage Aditya Ambre Madhura S. Raghavan Rohit Arora ENTERPRISE STORAGE ARCHITECTURE GROUP 2
HOT COLD CSC 568 Enterprise Storage Architecture (NC State University) AGENDA ➔ Problem Statement ➔ Project Goals and Features ➔ Architecture and Workflow ➔ Verification Cases ➔ Summary
Least Frequently Accessed Data HOT COLD CSC 568 Enterprise Storage Architecture (NC State University) PROBLEM STATEMENT ➔ Lifecycle of Data. ◆ Access frequency. ◆ Storage capacity and hardware characteristics. ➔ User intervention - Running jobs/scripts. ➔ Acknowledging Data temperature ➔ Tight coupling needed between storage components Frequently Accessed Data
HOT COLD CSC 568 Enterprise Storage Architecture (NC State University) WHAT IS A HOT FILE? Data File that ➔ Very frequently accessed. ➔ Mostly contains business critical information. ➔ Needs to be accessed quickly.
HOT COLD CSC 568 Enterprise Storage Architecture (NC State University) WHAT IS A COLD FILE? Data File that ➔ Is infrequently accessed. ➔ Contains less important information. ➔ Need not be quickly accessed.
HOT COLD CSC 568 Enterprise Storage Architecture (NC State University) GOAL: WHAT OUR PROJECT IS? ➔ From decoupled storage components - To - tightly coupled two- tiered storage system ➔ Manage hot & cold data between primary and secondary storage. ➔ Manage primary storage space utilization. ➔ File transfer do not interrupt FS operations. ➔ User agnostic about file transfer and storage. ➔ Optimal storage of cold data.
HOT COLD CSC 568 Enterprise Storage Architecture (NC State University) WHAT OUR PROJECT IS?
HOT COLD CSC 568 Enterprise Storage Architecture (NC State University) FEATURES ➔ Infinite Storage illusion ➔ Automatic cold data identification and transfer ➔ Consistent CRUD operations for both hot and cold files ➔ Block level storage ➔ On the fly deduplication ➔ Uninterrupted file access ➔ File level Consistency ➔ Optimal storage space utilization
HOT COLD CSC 568 Enterprise Storage Architecture (NC State University) OUR ARCHITECTURE Cold File Tracking Hot File Tracking File Tracking Layer Data Block Processing Layer Write block to cold Get block from cold De-duplication COLD STORAGE APPLICATION Write Read FUSE OPERATIONS Read, Write, Delete, Rename, etc. 2f0f3ff2c7439635e7faa85… 3f35ec5fe4ae0b963779c8… 4a8f9ec938243beac4b2d… Hot File Cold File
HOT COLD CSC 568 Enterprise Storage Architecture (NC State University) HOT-TO-COLD WORKFLOW COLD STORAGE APPLICATION Write FUSE {WRITE} OPERATIONS File Tracking Layer Data Block Processing Layer
HOT COLD CSC 568 Enterprise Storage Architecture (NC State University) HOT-TO-COLD WORKFLOW COLD STORAGE APPLICATION Write FUSE {WRITE} OPERATIONS Check: Storage > 70% File Tracking Layer Data Block Processing Layer
HOT COLD CSC 568 Enterprise Storage Architecture (NC State University) HOT-TO-COLD WORKFLOW COLD STORAGE APPLICATION Write FUSE {WRITE} OPERATIONS Check: Storage > 70% Cold File Tracking File Tracking Layer Data Block Processing Layer
HOT COLD CSC 568 Enterprise Storage Architecture (NC State University) HOT-TO-COLD WORKFLOW File Tracking Layer 1. List all the files 2. Sort files by access time - oldest to newest 3. Select files to be transferred - (till <=50%) 4. Sort above files by size - large to small 5. Send the largest & least accessed files to Data Processing layer Cold File tracking
HOT COLD CSC 568 Enterprise Storage Architecture (NC State University) HOT-TO-COLD WORKFLOW File Tracking Layer File 1 1:30 PM 100 KB File 2 4:30 PM 500 KB File 3 3:30 PM 250 KB File 4 2:30 PM 350 KB File 1 1:30 PM 100 KB File 2 4:30 PM 500 KB File 3 3:30 PM 250 KB File 4 2:30 PM 350 KB File 1 1:30 PM 100 KB File 4 2:30 PM 350 KB File 3 3:30 PM 250 KB
HOT COLD CSC 568 Enterprise Storage Architecture (NC State University) HOT-TO-COLD WORKFLOW COLD STORAGE APPLICATION Write FUSE {WRITE} OPERATIONS Check: Storage > 70% File Tracking Layer Cold File Tracking Data Block Processing Layer Write block to cold Cold File
2f0f3ff2… 7439635… e7faa85… 3f35ec5f… e4ae0b9... HOT COLD CSC 568 Enterprise Storage Architecture (NC State University) HOT-TO-COLD WORKFLOW Data Block Processing Layer 1. Request Hashtable 2. Get Hashtable Write Block to Cold COLD STORAGE 1. Request Hashtable 2. Gets Hashtable
HOT COLD CSC 568 Enterprise Storage Architecture (NC State University) HOT-TO-COLD WORKFLOW Data Block Processing Layer 1. Request Hashtable 2. Get Hashtable 3. Calculate block level hash 4. Check for de-duplication Write Block to Cold COLD STORAGE 4. Duplicate? 2f0f3ff2… 7439635… e7faa85… 3f35ec5f… e4ae0b9... Block 1 Block 2 Block 3
HOT COLD CSC 568 Enterprise Storage Architecture (NC State University) HOT-TO-COLD WORKFLOW Data Block Processing Layer 1. Request Hashtable 2. Get Hashtable 3. Calculate block level hash 4. Check for de-duplication 5. Transfer if not duplicate 6. Free block’s memory Write Block to Cold COLD STORAGE 5. Transfer Block 2f0f3ff2… 7439635… e7faa85… 3f35ec5f… e4ae0b9... 5. Update Hashtable Block 1 Block 2 Block 3
HOT COLD CSC 568 Enterprise Storage Architecture (NC State University) HOT-TO-COLD WORKFLOW Data Block Processing Layer 1. Request Hashtable 2. Get Hashtable 3. Calculate block level hash 4. Check for de-duplication 5. Transfer if not duplicate 6. Free block’s memory 7. Send updated hashtable to cold storage Write Block to Cold COLD STORAGE 7. Send Updated Hashtable 2f0f3ff2… 7439635… e7faa85… 3f35ec5f… e4ae0b9... 6.
HOT COLD CSC 568 Enterprise Storage Architecture (NC State University) HOT-TO-COLD WORKFLOW COLD STORAGE APPLICATION Write FUSE {WRITE} OPERATIONS Check: Storage <= 50% File Tracking Layer Cold File Tracking Data Block Processing Layer Write block to cold Cold File De-duplication 2f0f3ff2c7439635e7faa85… 3f35ec5fe4ae0b963779c8… 4a8f9ec938243beac4b2d…
HOT COLD CSC 568 Enterprise Storage Architecture (NC State University) COLD-TO-HOT WORKFLOW COLD STORAGE APPLICATION FUSE {READ} OPERATIONS File Tracking Layer Data Block Processing Layer Read Request 2f0f3ff2c7439635e7faa85… 3f35ec5fe4ae0b963779c8… 4a8f9ec938243beac4b2d…
HOT COLD CSC 568 Enterprise Storage Architecture (NC State University) COLD-TO-HOT WORKFLOW COLD STORAGE APPLICATION FUSE {READ} OPERATIONS File Tracking Layer Data Block Processing Layer Read Request Check: Is File on Hot Storage? 2f0f3ff2c7439635e7faa85… 3f35ec5fe4ae0b963779c8… 4a8f9ec938243beac4b2d…
HOT COLD CSC 568 Enterprise Storage Architecture (NC State University) COLD-TO-HOT WORKFLOW COLD STORAGE APPLICATION FUSE {READ} OPERATIONS File Tracking Layer Data Block Processing Layer Read Request Check: Is File on Hot Storage? Get block from cold No 2f0f3ff2c7439635e7faa85… 3f35ec5fe4ae0b963779c8… 4a8f9ec938243beac4b2d…
HOT COLD CSC 568 Enterprise Storage Architecture (NC State University) COLD-TO-HOT WORKFLOW Data Block Processing Layer 1. Request copy of Hashtable 2. Get Hashtable Get Block from Cold COLD STORAGE 1. Request Hashtable 2. Gets Hashtable
2f0f3ff2… 7439635… e7faa85… 3f35ec5f… e4ae0b9... HOT COLD CSC 568 Enterprise Storage Architecture (NC State University) COLD-TO-HOT WORKFLOW Data Block Processing Layer 1. Request copy of Hashtable 2. Get Hashtable 3. Read block presence on cold Get Block from Cold COLD STORAGE 3. Is block present?
HOT COLD CSC 568 Enterprise Storage Architecture (NC State University) COLD-TO-HOT WORKFLOW Data Block Processing Layer 1. Request copy of Hashtable 2. Get Hashtable 3. Read block presence on cold 4. Request/Get block from cold Get Block from Cold COLD STORAGE 4 Request Block 4. Gets Block 2f0f3ff2… 7439635… e7faa85… 3f35ec5f… e4ae0b9... 2f0f3ff2… 7439635… e7faa85… 3f35ec5f… e4ae0b9... Block 1 Block 2 Block 3
2f0f3ff2… 7439635… e7faa85… 3f35ec5f… e4ae0b9... HOT COLD CSC 568 Enterprise Storage Architecture (NC State University) COLD-TO-HOT WORKFLOW Data Block Processing Layer 1. Request copy of Hashtable 2. Get Hashtable 3. Read block presence on cold 4. Request/Get block from cold 5. Write transferred’ block content to memory block 6. Construct complete file Get Block from Cold COLD STORAGE Block 1 Block 2 Block 3 6.
2f0f3ff2… 7439635… e7faa85… 3f35ec5f… e4ae0b9... HOT COLD CSC 568 Enterprise Storage Architecture (NC State University) COLD-TO-HOT WORKFLOW Data Block Processing Layer 1. Request copy of Hashtable 2. Get Hashtable 3. Read block presence on cold 4. Request/Get block from cold 5. Write transferred’ block content to memory block 6. Construct complete file 7. Delete copy of Hashtable Get Block from Cold COLD STORAGE Block 1 Block 2 Block 3 7. Delete Hashtable
HOT COLD CSC 568 Enterprise Storage Architecture (NC State University) COLD-TO-HOT WORKFLOW COLD STORAGE APPLICATION FUSE {READ} OPERATIONS File Tracking Layer Data Block Processing Layer ReadRead Request Get block from cold Block Read Request No 2f0f3ff2c7439635e7faa85… 3f35ec5fe4ae0b963779c8… 4a8f9ec938243beac4b2d…
HOT COLD CSC 568 Enterprise Storage Architecture (NC State University) MINIMAL THRESHOLD WORKFLOW COLD STORAGE APPLICATION FUSE {READ} OPERATIONS File Tracking Layer Data Block Processing Layer Some Operation Get block from cold Block Read Request Yes 2f0f3ff2c7439635e7faa85… 3f35ec5fe4ae0b963779c8… 4a8f9ec938243beac4b2d… Check: Storage <= 30% Get Cold FileHot File Tracking
HOT COLD CSC 568 Enterprise Storage Architecture (NC State University) READ OPERATION WORKFLOW COLD STORAGE APPLICATION FUSE {READ} OPERATIONS File Tracking Layer Data Block Processing Layer Some Operation Get block from cold Block Read Request Yes 2f0f3ff2c7439635e7faa85… 3f35ec5fe4ae0b963779c8… 4a8f9ec938243beac4b2d… Check: Storage >30% & < 70% Get Cold FileHot File Tracking
HOT COLD CSC 568 Enterprise Storage Architecture (NC State University) QUICK DEMO
HOT COLD CSC 568 Enterprise Storage Architecture (NC State University) SCENARIOS / VERIFICATION CASES I. GENERAL ➔ File System 70% full -> Transfer to cold storage. ➔ File System drops less than 30% -> Transfer from cold storage. ➔ File transfers -> Do not interrupt general FS operations. ➔ Redundant/Duplicate blocks ->Not transferred.
HOT COLD CSC 568 Enterprise Storage Architecture (NC State University) SCENARIOS / VERIFICATION CASES II. SPECIFIC ➔ Files transferred –> Based on access and size. ➔ File removed on hot storage –> After last block is transferred. ➔ File in transition accessed –> Abort transfer, access granted! ➔ File space reclamation and File access –> Synchronized. ➔ Only one background process running at specific time. ➔ Delayed delete (rm) -> Transparent to user.
HOT COLD CSC 568 Enterprise Storage Architecture (NC State University) ASSUMPTIONS ➔ Network is always available. ➔ Hot-Cold classification at file level ➔ Cold Storage is infinite. ➔ Files are not very small or very large. ➔ Delay is accepted for rarely accessed files. ➔ File access granularity – in seconds.
HOT COLD CSC 568 Enterprise Storage Architecture (NC State University) SUMMARY ➔ Acknowledged data temperatures - hot and cold ➔ Project Features ◆ Auto file identification. ◆ File transfer ◆ Deduplication ➔ Architecture and workflows in action. ➔ Design and implementation of file tracking layer ➔ Design and implementation of Block Data Process Layer ➔ Design decisions for specific verification scenarios.
HOT COLD CSC 568 Enterprise Storage Architecture (NC State University) FUTURE SCOPE ➔ Variable block size and Block size specifications. ➔ Garbage collection on secondary/cold storage. ➔ Cold file identification parameters and profiles. ➔ Distributed cold storage.
HOT COLD CSC 568 Enterprise Storage Architecture (NC State University) REFERENCES 1. S. Quinlan and S. Dorward, “Venti: A new approach to archival storage,” in Proceedings of the First USENIX Conference on File and Storage Technologies (FAST), 2002. http://plan9.bell-labs.com/sys/doc/venti/venti. pdf 2. Chuanyi Liu, Dapeng Ju, et al, “Semantic data de-duplication for archival storage systems,” in Proceedings of the 13th IEEE Asia-Pacific Computer Systems Architecture Conference (ACSAC 2008), Hsinchu, Taiwan, August, 2008. 3. Sean Quinlan, Jim McKie Russ Cox, “Fossil, an Archival File Server”, Lucent Technologies Bell Labs, Unpublished memorandum (September 2003). 4. http://www.storiant.com/resources/Cold-Storage-Is-Hot-Again.pdf 5. “What is Unified Storage system ” http://searchstorage.techtarget. com/definition/unified-storage 6. File System in User Space - http://fuse.sourceforge.net/
HOT COLD CSC 568 Enterprise Storage Architecture (NC State University) QUESTIONS ?

Recommended

How to create a non managed standby database
How to create a non managed standby databaseHow to create a non managed standby database
How to create a non managed standby databaseJorge Batista
 
Bluestore
BluestoreBluestore
Bluestore
Patrick McGarry
 
dNFS for DBA's
dNFS for DBA'sdNFS for DBA's
dNFS for DBA's
Marcin Przepiórowski
 
Docker: Containers for Data Science
Docker: Containers for Data ScienceDocker: Containers for Data Science
Docker: Containers for Data Science
Alessandro Adamo
 
The Proto-Burst Buffer: Experience with the flash-based file system on SDSC's...
The Proto-Burst Buffer: Experience with the flash-based file system on SDSC's...The Proto-Burst Buffer: Experience with the flash-based file system on SDSC's...
The Proto-Burst Buffer: Experience with the flash-based file system on SDSC's...
Glenn K. Lockwood
 
Ceph Day KL - Bluestore
Ceph Day KL - Bluestore Ceph Day KL - Bluestore
Ceph Day KL - Bluestore
Ceph Community
 
Hopsfs 10x HDFS performance
Hopsfs 10x HDFS performanceHopsfs 10x HDFS performance
Hopsfs 10x HDFS performance
Jim Dowling
 
[若渴]Study on Side Channel Attacks and Countermeasures
[若渴]Study on Side Channel Attacks and Countermeasures [若渴]Study on Side Channel Attacks and Countermeasures
[若渴]Study on Side Channel Attacks and Countermeasures
Aj MaChInE
 

Recommended

How to create a non managed standby database
How to create a non managed standby databaseHow to create a non managed standby database
How to create a non managed standby databaseJorge Batista
 
Bluestore
BluestoreBluestore
Bluestore
Patrick McGarry
 
dNFS for DBA's
dNFS for DBA'sdNFS for DBA's
dNFS for DBA's
Marcin Przepiórowski
 
Docker: Containers for Data Science
Docker: Containers for Data ScienceDocker: Containers for Data Science
Docker: Containers for Data Science
Alessandro Adamo
 
The Proto-Burst Buffer: Experience with the flash-based file system on SDSC's...
The Proto-Burst Buffer: Experience with the flash-based file system on SDSC's...The Proto-Burst Buffer: Experience with the flash-based file system on SDSC's...
The Proto-Burst Buffer: Experience with the flash-based file system on SDSC's...
Glenn K. Lockwood
 
Ceph Day KL - Bluestore
Ceph Day KL - Bluestore Ceph Day KL - Bluestore
Ceph Day KL - Bluestore
Ceph Community
 
Hopsfs 10x HDFS performance
Hopsfs 10x HDFS performanceHopsfs 10x HDFS performance
Hopsfs 10x HDFS performance
Jim Dowling
 
[若渴]Study on Side Channel Attacks and Countermeasures
[若渴]Study on Side Channel Attacks and Countermeasures [若渴]Study on Side Channel Attacks and Countermeasures
[若渴]Study on Side Channel Attacks and Countermeasures
Aj MaChInE
 
Redis 101
Redis 101Redis 101
Redis 101Geoff Hoffman
 
Open Source Backup Conference 2014: Workshop bareos introduction, by Philipp ...
Open Source Backup Conference 2014: Workshop bareos introduction, by Philipp ...Open Source Backup Conference 2014: Workshop bareos introduction, by Philipp ...
Open Source Backup Conference 2014: Workshop bareos introduction, by Philipp ...
NETWAYS
 
Ceph - High Performance Without High Costs
Ceph - High Performance Without High CostsCeph - High Performance Without High Costs
Ceph - High Performance Without High Costs
Jonathan Long
 
Linux performance tuning & stabilization tips (mysqlconf2010)
Linux performance tuning & stabilization tips (mysqlconf2010)Linux performance tuning & stabilization tips (mysqlconf2010)
Linux performance tuning & stabilization tips (mysqlconf2010)
Yoshinori Matsunobu
 
AFF4: The new standard in forensic imaging and why you should care
AFF4: The new standard in forensic imaging and why you should careAFF4: The new standard in forensic imaging and why you should care
AFF4: The new standard in forensic imaging and why you should care
Bradley Schatz
 
Redis, another step on the road
Redis, another step on the roadRedis, another step on the road
Redis, another step on the road
Yi-Feng Tzeng
 
Accelerating forensic and incident response workflow: the case for a new stan...
Accelerating forensic and incident response workflow: the case for a new stan...Accelerating forensic and incident response workflow: the case for a new stan...
Accelerating forensic and incident response workflow: the case for a new stan...
Bradley Schatz
 
Accelerating forensic and incident response workflow: the case for a new stan...
Accelerating forensic and incident response workflow: the case for a new stan...Accelerating forensic and incident response workflow: the case for a new stan...
Accelerating forensic and incident response workflow: the case for a new stan...
Bradley Schatz
 
Docker and friends at Linux Days 2014 in Prague
Docker and friends at Linux Days 2014 in PragueDocker and friends at Linux Days 2014 in Prague
Docker and friends at Linux Days 2014 in Prague
tomasbart
 
Introduction to redis
Introduction to redisIntroduction to redis
Introduction to redis
Tanu Siwag
 
Introduction to redis
Introduction to redisIntroduction to redis
Introduction to redis
NexThoughts Technologies
 
ZFS for Databases
ZFS for DatabasesZFS for Databases
ZFS for Databasesahl0003
 
OpenZFS data-driven performance
OpenZFS data-driven performanceOpenZFS data-driven performance
OpenZFS data-driven performance
ahl0003
 
Introduction to Git/Github - A beginner's guide
Introduction to Git/Github - A beginner's guideIntroduction to Git/Github - A beginner's guide
Introduction to Git/Github - A beginner's guide
Rohit Arora
 
Splunk-EMC
Splunk-EMCSplunk-EMC
Splunk-EMCBoni Bruno
 
Introduction to Ansible (Pycon7 2016)
Introduction to Ansible (Pycon7 2016)Introduction to Ansible (Pycon7 2016)
Introduction to Ansible (Pycon7 2016)
Ivan Rossi
 
Ansible 2.0
Ansible 2.0Ansible 2.0
Ansible 2.0
Dennis Rowe
 
Elasticsearch Data Analyses
Elasticsearch Data AnalysesElasticsearch Data Analyses
Elasticsearch Data AnalysesAlaa Elhadba
 
Ansible for beginners ...?
Ansible for beginners ...?Ansible for beginners ...?
Ansible for beginners ...?
shirou wakayama
 
Ansible - Introduction
Ansible - IntroductionAnsible - Introduction
Ansible - Introduction
Stephane Manciot
 
Testing Ansible with Jenkins and Docker
Testing Ansible with Jenkins and DockerTesting Ansible with Jenkins and Docker
Testing Ansible with Jenkins and Docker
Dennis Rowe
 
Network Automation: Ansible 102
Network Automation: Ansible 102Network Automation: Ansible 102
Network Automation: Ansible 102
APNIC
 

More Related Content

What's hot

Open Source Backup Conference 2014: Workshop bareos introduction, by Philipp ...
Open Source Backup Conference 2014: Workshop bareos introduction, by Philipp ...Open Source Backup Conference 2014: Workshop bareos introduction, by Philipp ...
Open Source Backup Conference 2014: Workshop bareos introduction, by Philipp ...
NETWAYS
 
Ceph - High Performance Without High Costs
Ceph - High Performance Without High CostsCeph - High Performance Without High Costs
Ceph - High Performance Without High Costs
Jonathan Long
 
Linux performance tuning & stabilization tips (mysqlconf2010)
Linux performance tuning & stabilization tips (mysqlconf2010)Linux performance tuning & stabilization tips (mysqlconf2010)
Linux performance tuning & stabilization tips (mysqlconf2010)
Yoshinori Matsunobu
 
AFF4: The new standard in forensic imaging and why you should care
AFF4: The new standard in forensic imaging and why you should careAFF4: The new standard in forensic imaging and why you should care
AFF4: The new standard in forensic imaging and why you should care
Bradley Schatz
 
Redis, another step on the road
Redis, another step on the roadRedis, another step on the road
Redis, another step on the road
Yi-Feng Tzeng
 
Accelerating forensic and incident response workflow: the case for a new stan...
Accelerating forensic and incident response workflow: the case for a new stan...Accelerating forensic and incident response workflow: the case for a new stan...
Accelerating forensic and incident response workflow: the case for a new stan...
Bradley Schatz
 
Accelerating forensic and incident response workflow: the case for a new stan...
Accelerating forensic and incident response workflow: the case for a new stan...Accelerating forensic and incident response workflow: the case for a new stan...
Accelerating forensic and incident response workflow: the case for a new stan...
Bradley Schatz
 
Docker and friends at Linux Days 2014 in Prague
Docker and friends at Linux Days 2014 in PragueDocker and friends at Linux Days 2014 in Prague
Docker and friends at Linux Days 2014 in Prague
tomasbart
 
Introduction to redis
Introduction to redisIntroduction to redis
Introduction to redis
Tanu Siwag
 
Introduction to redis
Introduction to redisIntroduction to redis
Introduction to redis
NexThoughts Technologies
 
ZFS for Databases
ZFS for DatabasesZFS for Databases
ZFS for Databasesahl0003
 
OpenZFS data-driven performance
OpenZFS data-driven performanceOpenZFS data-driven performance
OpenZFS data-driven performance
ahl0003
 

What's hot (13)

Redis 101
Redis 101Redis 101
Redis 101
 
Open Source Backup Conference 2014: Workshop bareos introduction, by Philipp ...
Open Source Backup Conference 2014: Workshop bareos introduction, by Philipp ...Open Source Backup Conference 2014: Workshop bareos introduction, by Philipp ...
Open Source Backup Conference 2014: Workshop bareos introduction, by Philipp ...
 
Ceph - High Performance Without High Costs
Ceph - High Performance Without High CostsCeph - High Performance Without High Costs
Ceph - High Performance Without High Costs
 
Linux performance tuning & stabilization tips (mysqlconf2010)
Linux performance tuning & stabilization tips (mysqlconf2010)Linux performance tuning & stabilization tips (mysqlconf2010)
Linux performance tuning & stabilization tips (mysqlconf2010)
 
AFF4: The new standard in forensic imaging and why you should care
AFF4: The new standard in forensic imaging and why you should careAFF4: The new standard in forensic imaging and why you should care
AFF4: The new standard in forensic imaging and why you should care
 
Redis, another step on the road
Redis, another step on the roadRedis, another step on the road
Redis, another step on the road
 
Accelerating forensic and incident response workflow: the case for a new stan...
Accelerating forensic and incident response workflow: the case for a new stan...Accelerating forensic and incident response workflow: the case for a new stan...
Accelerating forensic and incident response workflow: the case for a new stan...
 
Accelerating forensic and incident response workflow: the case for a new stan...
Accelerating forensic and incident response workflow: the case for a new stan...Accelerating forensic and incident response workflow: the case for a new stan...
Accelerating forensic and incident response workflow: the case for a new stan...
 
Docker and friends at Linux Days 2014 in Prague
Docker and friends at Linux Days 2014 in PragueDocker and friends at Linux Days 2014 in Prague
Docker and friends at Linux Days 2014 in Prague
 
Introduction to redis
Introduction to redisIntroduction to redis
Introduction to redis
 
Introduction to redis
Introduction to redisIntroduction to redis
Introduction to redis
 
ZFS for Databases
ZFS for DatabasesZFS for Databases
ZFS for Databases
 
OpenZFS data-driven performance
OpenZFS data-driven performanceOpenZFS data-driven performance
OpenZFS data-driven performance
 

Viewers also liked

Introduction to Git/Github - A beginner's guide
Introduction to Git/Github - A beginner's guideIntroduction to Git/Github - A beginner's guide
Introduction to Git/Github - A beginner's guide
Rohit Arora
 
Introduction to Ansible (Pycon7 2016)
Introduction to Ansible (Pycon7 2016)Introduction to Ansible (Pycon7 2016)
Introduction to Ansible (Pycon7 2016)
Ivan Rossi
 
Ansible 2.0
Ansible 2.0Ansible 2.0
Ansible 2.0
Dennis Rowe
 
Elasticsearch Data Analyses
Elasticsearch Data AnalysesElasticsearch Data Analyses
Elasticsearch Data AnalysesAlaa Elhadba
 
Ansible for beginners ...?
Ansible for beginners ...?Ansible for beginners ...?
Ansible for beginners ...?
shirou wakayama
 
Ansible - Introduction
Ansible - IntroductionAnsible - Introduction
Ansible - Introduction
Stephane Manciot
 
Testing Ansible with Jenkins and Docker
Testing Ansible with Jenkins and DockerTesting Ansible with Jenkins and Docker
Testing Ansible with Jenkins and Docker
Dennis Rowe
 
Network Automation: Ansible 102
Network Automation: Ansible 102Network Automation: Ansible 102
Network Automation: Ansible 102
APNIC
 
IT Automation with Ansible
IT Automation with AnsibleIT Automation with Ansible
IT Automation with Ansible
Rayed Alrashed
 
Ansible presentation
Ansible presentationAnsible presentation
Ansible presentation
John Lynch
 
Ansible: How to Get More Sleep and Require Less Coffee
Ansible: How to Get More Sleep and Require Less CoffeeAnsible: How to Get More Sleep and Require Less Coffee
Ansible: How to Get More Sleep and Require Less Coffee
Sarah Z
 
Git 101: Git and GitHub for Beginners
Git 101: Git and GitHub for Beginners Git 101: Git and GitHub for Beginners
Git 101: Git and GitHub for Beginners
HubSpot
 
Ansible tips & tricks
Ansible tips & tricksAnsible tips & tricks
Ansible tips & tricks
bcoca
 
Ansible Introduction
Ansible Introduction Ansible Introduction
Ansible Introduction
Robert Reiz
 

Viewers also liked (16)

Introduction to Git/Github - A beginner's guide
Introduction to Git/Github - A beginner's guideIntroduction to Git/Github - A beginner's guide
Introduction to Git/Github - A beginner's guide
 
Splunk-EMC
Splunk-EMCSplunk-EMC
Splunk-EMC
 
Introduction to Ansible (Pycon7 2016)
Introduction to Ansible (Pycon7 2016)Introduction to Ansible (Pycon7 2016)
Introduction to Ansible (Pycon7 2016)
 
Ansible 2.0
Ansible 2.0Ansible 2.0
Ansible 2.0
 
Elasticsearch Data Analyses
Elasticsearch Data AnalysesElasticsearch Data Analyses
Elasticsearch Data Analyses
 
Ansible for beginners ...?
Ansible for beginners ...?Ansible for beginners ...?
Ansible for beginners ...?
 
Ansible - Introduction
Ansible - IntroductionAnsible - Introduction
Ansible - Introduction
 
Testing Ansible with Jenkins and Docker
Testing Ansible with Jenkins and DockerTesting Ansible with Jenkins and Docker
Testing Ansible with Jenkins and Docker
 
Network Automation: Ansible 102
Network Automation: Ansible 102Network Automation: Ansible 102
Network Automation: Ansible 102
 
IT Automation with Ansible
IT Automation with AnsibleIT Automation with Ansible
IT Automation with Ansible
 
Ansible presentation
Ansible presentationAnsible presentation
Ansible presentation
 
Ansible: How to Get More Sleep and Require Less Coffee
Ansible: How to Get More Sleep and Require Less CoffeeAnsible: How to Get More Sleep and Require Less Coffee
Ansible: How to Get More Sleep and Require Less Coffee
 
Git 101: Git and GitHub for Beginners
Git 101: Git and GitHub for Beginners Git 101: Git and GitHub for Beginners
Git 101: Git and GitHub for Beginners
 
Ansible tips & tricks
Ansible tips & tricksAnsible tips & tricks
Ansible tips & tricks
 
Ansible Introduction
Ansible Introduction Ansible Introduction
Ansible Introduction
 
Slideshare ppt
Slideshare pptSlideshare ppt
Slideshare ppt
 

Similar to Hot and cold data storage

Kyle Hailey Oracle Performance IO Waits
Kyle Hailey Oracle Performance IO WaitsKyle Hailey Oracle Performance IO Waits
Kyle Hailey Oracle Performance IO Waits
cookie1969
 
Maa wp-10g-racprimaryracphysicalsta-131940
Maa wp-10g-racprimaryracphysicalsta-131940Maa wp-10g-racprimaryracphysicalsta-131940
Maa wp-10g-racprimaryracphysicalsta-131940
gopalchsamanta
 
Spark Summit EU talk by Jiri Simsa
Spark Summit EU talk by Jiri SimsaSpark Summit EU talk by Jiri Simsa
Spark Summit EU talk by Jiri Simsa
Alluxio, Inc.
 
Spark Summit EU talk by Jiri Simsa
Spark Summit EU talk by Jiri SimsaSpark Summit EU talk by Jiri Simsa
Spark Summit EU talk by Jiri Simsa
Spark Summit
 
Openstack Swift - Lots of small files
Openstack Swift - Lots of small filesOpenstack Swift - Lots of small files
Openstack Swift - Lots of small files
Alexandre Lecuyer
 
Improving Memory Utilization of Spark Jobs Using Alluxio
Improving Memory Utilization of Spark Jobs Using AlluxioImproving Memory Utilization of Spark Jobs Using Alluxio
Improving Memory Utilization of Spark Jobs Using Alluxio
Alluxio, Inc.
 
Collaborate instant cloning_kyle
Collaborate instant cloning_kyleCollaborate instant cloning_kyle
Collaborate instant cloning_kyleKyle Hailey
 
AEM Meetup Sydney - Content Migration with CRX2Oak
AEM Meetup Sydney - Content Migration with CRX2OakAEM Meetup Sydney - Content Migration with CRX2Oak
AEM Meetup Sydney - Content Migration with CRX2Oak
Michael Henderson
 
Alluxio Presentation at Strata San Jose 2016
Alluxio Presentation at Strata San Jose 2016Alluxio Presentation at Strata San Jose 2016
Alluxio Presentation at Strata San Jose 2016
Jiří Šimša
 
Truly non-intrusive OpenStack Cinder backup for mission critical systems
Truly non-intrusive OpenStack Cinder backup for mission critical systemsTruly non-intrusive OpenStack Cinder backup for mission critical systems
Truly non-intrusive OpenStack Cinder backup for mission critical systems
Dipak Kumar Singh
 
Alluxio Use Cases at Strata+Hadoop World Beijing 2016
Alluxio Use Cases at Strata+Hadoop World Beijing 2016Alluxio Use Cases at Strata+Hadoop World Beijing 2016
Alluxio Use Cases at Strata+Hadoop World Beijing 2016
Alluxio, Inc.
 
Managing ADLS gen2 using Apache Spark
Managing ADLS gen2 using Apache SparkManaging ADLS gen2 using Apache Spark
Managing ADLS gen2 using Apache Spark
Databricks
 
Tales from the four-comma club: Managing Kafka as a service at Salesforce | L...
Tales from the four-comma club: Managing Kafka as a service at Salesforce | L...Tales from the four-comma club: Managing Kafka as a service at Salesforce | L...
Tales from the four-comma club: Managing Kafka as a service at Salesforce | L...
HostedbyConfluent
 
W1.1 i os in database
W1.1 i os in databaseW1.1 i os in database
W1.1 i os in database
gafurov_x
 
Super hybrid2016 tdc
Super hybrid2016 tdcSuper hybrid2016 tdc
Super hybrid2016 tdc
Murilo Miranda
 
Google File System
Google File SystemGoogle File System
Google File System
Junyoung Jung
 
Getting Started with Apache Spark and Alluxio for Blazingly Fast Analytics
Getting Started with Apache Spark and Alluxio for Blazingly Fast AnalyticsGetting Started with Apache Spark and Alluxio for Blazingly Fast Analytics
Getting Started with Apache Spark and Alluxio for Blazingly Fast Analytics
Alluxio, Inc.
 
Next generation storage: eliminating the guesswork and avoiding forklift upgrade
Next generation storage: eliminating the guesswork and avoiding forklift upgradeNext generation storage: eliminating the guesswork and avoiding forklift upgrade
Next generation storage: eliminating the guesswork and avoiding forklift upgrade
Jisc
 
Tachyon-2014-11-21-amp-camp5
Tachyon-2014-11-21-amp-camp5Tachyon-2014-11-21-amp-camp5
Tachyon-2014-11-21-amp-camp5
Haoyuan Li
 

Similar to Hot and cold data storage (20)

Kyle Hailey Oracle Performance IO Waits
Kyle Hailey Oracle Performance IO WaitsKyle Hailey Oracle Performance IO Waits
Kyle Hailey Oracle Performance IO Waits
 
Maa wp-10g-racprimaryracphysicalsta-131940
Maa wp-10g-racprimaryracphysicalsta-131940Maa wp-10g-racprimaryracphysicalsta-131940
Maa wp-10g-racprimaryracphysicalsta-131940
 
Spark Summit EU talk by Jiri Simsa
Spark Summit EU talk by Jiri SimsaSpark Summit EU talk by Jiri Simsa
Spark Summit EU talk by Jiri Simsa
 
Spark Summit EU talk by Jiri Simsa
Spark Summit EU talk by Jiri SimsaSpark Summit EU talk by Jiri Simsa
Spark Summit EU talk by Jiri Simsa
 
Openstack Swift - Lots of small files
Openstack Swift - Lots of small filesOpenstack Swift - Lots of small files
Openstack Swift - Lots of small files
 
B35 Inside rac by Julian Dyke
B35 Inside rac by Julian DykeB35 Inside rac by Julian Dyke
B35 Inside rac by Julian Dyke
 
Improving Memory Utilization of Spark Jobs Using Alluxio
Improving Memory Utilization of Spark Jobs Using AlluxioImproving Memory Utilization of Spark Jobs Using Alluxio
Improving Memory Utilization of Spark Jobs Using Alluxio
 
Collaborate instant cloning_kyle
Collaborate instant cloning_kyleCollaborate instant cloning_kyle
Collaborate instant cloning_kyle
 
AEM Meetup Sydney - Content Migration with CRX2Oak
AEM Meetup Sydney - Content Migration with CRX2OakAEM Meetup Sydney - Content Migration with CRX2Oak
AEM Meetup Sydney - Content Migration with CRX2Oak
 
Alluxio Presentation at Strata San Jose 2016
Alluxio Presentation at Strata San Jose 2016Alluxio Presentation at Strata San Jose 2016
Alluxio Presentation at Strata San Jose 2016
 
Truly non-intrusive OpenStack Cinder backup for mission critical systems
Truly non-intrusive OpenStack Cinder backup for mission critical systemsTruly non-intrusive OpenStack Cinder backup for mission critical systems
Truly non-intrusive OpenStack Cinder backup for mission critical systems
 
Alluxio Use Cases at Strata+Hadoop World Beijing 2016
Alluxio Use Cases at Strata+Hadoop World Beijing 2016Alluxio Use Cases at Strata+Hadoop World Beijing 2016
Alluxio Use Cases at Strata+Hadoop World Beijing 2016
 
Managing ADLS gen2 using Apache Spark
Managing ADLS gen2 using Apache SparkManaging ADLS gen2 using Apache Spark
Managing ADLS gen2 using Apache Spark
 
Tales from the four-comma club: Managing Kafka as a service at Salesforce | L...
Tales from the four-comma club: Managing Kafka as a service at Salesforce | L...Tales from the four-comma club: Managing Kafka as a service at Salesforce | L...
Tales from the four-comma club: Managing Kafka as a service at Salesforce | L...
 
W1.1 i os in database
W1.1 i os in databaseW1.1 i os in database
W1.1 i os in database
 
Super hybrid2016 tdc
Super hybrid2016 tdcSuper hybrid2016 tdc
Super hybrid2016 tdc
 
Google File System
Google File SystemGoogle File System
Google File System
 
Getting Started with Apache Spark and Alluxio for Blazingly Fast Analytics
Getting Started with Apache Spark and Alluxio for Blazingly Fast AnalyticsGetting Started with Apache Spark and Alluxio for Blazingly Fast Analytics
Getting Started with Apache Spark and Alluxio for Blazingly Fast Analytics
 
Next generation storage: eliminating the guesswork and avoiding forklift upgrade
Next generation storage: eliminating the guesswork and avoiding forklift upgradeNext generation storage: eliminating the guesswork and avoiding forklift upgrade
Next generation storage: eliminating the guesswork and avoiding forklift upgrade
 
Tachyon-2014-11-21-amp-camp5
Tachyon-2014-11-21-amp-camp5Tachyon-2014-11-21-amp-camp5
Tachyon-2014-11-21-amp-camp5
 

More from Rohit Arora

Paradigm Wars: Object Oriented Vs Functional Programming in creating MarkParser
Paradigm Wars: Object Oriented Vs Functional Programming in creating MarkParserParadigm Wars: Object Oriented Vs Functional Programming in creating MarkParser
Paradigm Wars: Object Oriented Vs Functional Programming in creating MarkParser
Rohit Arora
 
Forest Cover Type Prediction
Forest Cover Type PredictionForest Cover Type Prediction
Forest Cover Type Prediction
Rohit Arora
 
Facility Booking System
Facility Booking SystemFacility Booking System
Facility Booking System
Rohit Arora
 
Advanced Traveler's Information System
Advanced Traveler's Information SystemAdvanced Traveler's Information System
Advanced Traveler's Information System
Rohit Arora
 
Sign Language Recognition using Wavelet Transform and Neural Networks
Sign Language Recognition using Wavelet Transform and Neural NetworksSign Language Recognition using Wavelet Transform and Neural Networks
Sign Language Recognition using Wavelet Transform and Neural Networks
Rohit Arora
 
Sign Language Recognition using Hough Transform and Neural Networks
Sign Language Recognition using Hough Transform and Neural NetworksSign Language Recognition using Hough Transform and Neural Networks
Sign Language Recognition using Hough Transform and Neural Networks
Rohit Arora
 
Ambient intelligence & Ubiquitous Computing
Ambient intelligence & Ubiquitous ComputingAmbient intelligence & Ubiquitous Computing
Ambient intelligence & Ubiquitous Computing
Rohit Arora
 

More from Rohit Arora (7)

Paradigm Wars: Object Oriented Vs Functional Programming in creating MarkParser
Paradigm Wars: Object Oriented Vs Functional Programming in creating MarkParserParadigm Wars: Object Oriented Vs Functional Programming in creating MarkParser
Paradigm Wars: Object Oriented Vs Functional Programming in creating MarkParser
 
Forest Cover Type Prediction
Forest Cover Type PredictionForest Cover Type Prediction
Forest Cover Type Prediction
 
Facility Booking System
Facility Booking SystemFacility Booking System
Facility Booking System
 
Advanced Traveler's Information System
Advanced Traveler's Information SystemAdvanced Traveler's Information System
Advanced Traveler's Information System
 
Sign Language Recognition using Wavelet Transform and Neural Networks
Sign Language Recognition using Wavelet Transform and Neural NetworksSign Language Recognition using Wavelet Transform and Neural Networks
Sign Language Recognition using Wavelet Transform and Neural Networks
 
Sign Language Recognition using Hough Transform and Neural Networks
Sign Language Recognition using Hough Transform and Neural NetworksSign Language Recognition using Hough Transform and Neural Networks
Sign Language Recognition using Hough Transform and Neural Networks
 
Ambient intelligence & Ubiquitous Computing
Ambient intelligence & Ubiquitous ComputingAmbient intelligence & Ubiquitous Computing
Ambient intelligence & Ubiquitous Computing
 

Recently uploaded

Pile Foundation by Venkatesh Taduvai (Sub Geotechnical Engineering II)-conver...
Pile Foundation by Venkatesh Taduvai (Sub Geotechnical Engineering II)-conver...Pile Foundation by Venkatesh Taduvai (Sub Geotechnical Engineering II)-conver...
Pile Foundation by Venkatesh Taduvai (Sub Geotechnical Engineering II)-conver...
AJAYKUMARPUND1
 
Industrial Training at Shahjalal Fertilizer Company Limited (SFCL)
Industrial Training at Shahjalal Fertilizer Company Limited (SFCL)Industrial Training at Shahjalal Fertilizer Company Limited (SFCL)
Industrial Training at Shahjalal Fertilizer Company Limited (SFCL)
MdTanvirMahtab2
 
RAT: Retrieval Augmented Thoughts Elicit Context-Aware Reasoning in Long-Hori...
RAT: Retrieval Augmented Thoughts Elicit Context-Aware Reasoning in Long-Hori...RAT: Retrieval Augmented Thoughts Elicit Context-Aware Reasoning in Long-Hori...
RAT: Retrieval Augmented Thoughts Elicit Context-Aware Reasoning in Long-Hori...
thanhdowork
 
Planning Of Procurement o different goods and services
Planning Of Procurement o different goods and servicesPlanning Of Procurement o different goods and services
Planning Of Procurement o different goods and services
JoytuBarua2
 
DfMAy 2024 - key insights and contributions
DfMAy 2024 - key insights and contributionsDfMAy 2024 - key insights and contributions
DfMAy 2024 - key insights and contributions
gestioneergodomus
 
Online aptitude test management system project report.pdf
Online aptitude test management system project report.pdfOnline aptitude test management system project report.pdf
Online aptitude test management system project report.pdf
Kamal Acharya
 
Swimming pool mechanical components design.pptx
Swimming pool mechanical components design.pptxSwimming pool mechanical components design.pptx
Swimming pool mechanical components design.pptx
yokeleetan1
 
Water billing management system project report.pdf
Water billing management system project report.pdfWater billing management system project report.pdf
Water billing management system project report.pdf
Kamal Acharya
 
Design and Analysis of Algorithms-DP,Backtracking,Graphs,B&B
Design and Analysis of Algorithms-DP,Backtracking,Graphs,B&BDesign and Analysis of Algorithms-DP,Backtracking,Graphs,B&B
Design and Analysis of Algorithms-DP,Backtracking,Graphs,B&B
Sreedhar Chowdam
 
Harnessing WebAssembly for Real-time Stateless Streaming Pipelines
Harnessing WebAssembly for Real-time Stateless Streaming PipelinesHarnessing WebAssembly for Real-time Stateless Streaming Pipelines
Harnessing WebAssembly for Real-time Stateless Streaming Pipelines
Christina Lin
 
Literature Review Basics and Understanding Reference Management.pptx
Literature Review Basics and Understanding Reference Management.pptxLiterature Review Basics and Understanding Reference Management.pptx
Literature Review Basics and Understanding Reference Management.pptx
Dr Ramhari Poudyal
 
Modelagem de um CSTR com reação endotermica.pdf
Modelagem de um CSTR com reação endotermica.pdfModelagem de um CSTR com reação endotermica.pdf
Modelagem de um CSTR com reação endotermica.pdf
camseq
 
一比一原版(Otago毕业证)奥塔哥大学毕业证成绩单如何办理
一比一原版(Otago毕业证)奥塔哥大学毕业证成绩单如何办理一比一原版(Otago毕业证)奥塔哥大学毕业证成绩单如何办理
一比一原版(Otago毕业证)奥塔哥大学毕业证成绩单如何办理
dxobcob
 
Hybrid optimization of pumped hydro system and solar- Engr. Abdul-Azeez.pdf
Hybrid optimization of pumped hydro system and solar- Engr. Abdul-Azeez.pdfHybrid optimization of pumped hydro system and solar- Engr. Abdul-Azeez.pdf
Hybrid optimization of pumped hydro system and solar- Engr. Abdul-Azeez.pdf
fxintegritypublishin
 
Student information management system project report ii.pdf
Student information management system project report ii.pdfStudent information management system project report ii.pdf
Student information management system project report ii.pdf
Kamal Acharya
 
Unbalanced Three Phase Systems and circuits.pptx
Unbalanced Three Phase Systems and circuits.pptxUnbalanced Three Phase Systems and circuits.pptx
Unbalanced Three Phase Systems and circuits.pptx
ChristineTorrepenida1
 
Final project report on grocery store management system..pdf
Final project report on grocery store management system..pdfFinal project report on grocery store management system..pdf
Final project report on grocery store management system..pdf
Kamal Acharya
 
一比一原版(UMich毕业证)密歇根大学|安娜堡分校毕业证成绩单专业办理
一比一原版(UMich毕业证)密歇根大学|安娜堡分校毕业证成绩单专业办理一比一原版(UMich毕业证)密歇根大学|安娜堡分校毕业证成绩单专业办理
一比一原版(UMich毕业证)密歇根大学|安娜堡分校毕业证成绩单专业办理
zwunae
 
An Approach to Detecting Writing Styles Based on Clustering Techniques
An Approach to Detecting Writing Styles Based on Clustering TechniquesAn Approach to Detecting Writing Styles Based on Clustering Techniques
An Approach to Detecting Writing Styles Based on Clustering Techniques
ambekarshweta25
 
NUMERICAL SIMULATIONS OF HEAT AND MASS TRANSFER IN CONDENSING HEAT EXCHANGERS...
NUMERICAL SIMULATIONS OF HEAT AND MASS TRANSFER IN CONDENSING HEAT EXCHANGERS...NUMERICAL SIMULATIONS OF HEAT AND MASS TRANSFER IN CONDENSING HEAT EXCHANGERS...
NUMERICAL SIMULATIONS OF HEAT AND MASS TRANSFER IN CONDENSING HEAT EXCHANGERS...
ssuser7dcef0
 

Recently uploaded (20)

Pile Foundation by Venkatesh Taduvai (Sub Geotechnical Engineering II)-conver...
Pile Foundation by Venkatesh Taduvai (Sub Geotechnical Engineering II)-conver...Pile Foundation by Venkatesh Taduvai (Sub Geotechnical Engineering II)-conver...
Pile Foundation by Venkatesh Taduvai (Sub Geotechnical Engineering II)-conver...
 
Industrial Training at Shahjalal Fertilizer Company Limited (SFCL)
Industrial Training at Shahjalal Fertilizer Company Limited (SFCL)Industrial Training at Shahjalal Fertilizer Company Limited (SFCL)
Industrial Training at Shahjalal Fertilizer Company Limited (SFCL)
 
RAT: Retrieval Augmented Thoughts Elicit Context-Aware Reasoning in Long-Hori...
RAT: Retrieval Augmented Thoughts Elicit Context-Aware Reasoning in Long-Hori...RAT: Retrieval Augmented Thoughts Elicit Context-Aware Reasoning in Long-Hori...
RAT: Retrieval Augmented Thoughts Elicit Context-Aware Reasoning in Long-Hori...
 
Planning Of Procurement o different goods and services
Planning Of Procurement o different goods and servicesPlanning Of Procurement o different goods and services
Planning Of Procurement o different goods and services
 
DfMAy 2024 - key insights and contributions
DfMAy 2024 - key insights and contributionsDfMAy 2024 - key insights and contributions
DfMAy 2024 - key insights and contributions
 
Online aptitude test management system project report.pdf
Online aptitude test management system project report.pdfOnline aptitude test management system project report.pdf
Online aptitude test management system project report.pdf
 
Swimming pool mechanical components design.pptx
Swimming pool mechanical components design.pptxSwimming pool mechanical components design.pptx
Swimming pool mechanical components design.pptx
 
Water billing management system project report.pdf
Water billing management system project report.pdfWater billing management system project report.pdf
Water billing management system project report.pdf
 
Design and Analysis of Algorithms-DP,Backtracking,Graphs,B&B
Design and Analysis of Algorithms-DP,Backtracking,Graphs,B&BDesign and Analysis of Algorithms-DP,Backtracking,Graphs,B&B
Design and Analysis of Algorithms-DP,Backtracking,Graphs,B&B
 
Harnessing WebAssembly for Real-time Stateless Streaming Pipelines
Harnessing WebAssembly for Real-time Stateless Streaming PipelinesHarnessing WebAssembly for Real-time Stateless Streaming Pipelines
Harnessing WebAssembly for Real-time Stateless Streaming Pipelines
 
Literature Review Basics and Understanding Reference Management.pptx
Literature Review Basics and Understanding Reference Management.pptxLiterature Review Basics and Understanding Reference Management.pptx
Literature Review Basics and Understanding Reference Management.pptx
 
Modelagem de um CSTR com reação endotermica.pdf
Modelagem de um CSTR com reação endotermica.pdfModelagem de um CSTR com reação endotermica.pdf
Modelagem de um CSTR com reação endotermica.pdf
 
一比一原版(Otago毕业证)奥塔哥大学毕业证成绩单如何办理
一比一原版(Otago毕业证)奥塔哥大学毕业证成绩单如何办理一比一原版(Otago毕业证)奥塔哥大学毕业证成绩单如何办理
一比一原版(Otago毕业证)奥塔哥大学毕业证成绩单如何办理
 
Hybrid optimization of pumped hydro system and solar- Engr. Abdul-Azeez.pdf
Hybrid optimization of pumped hydro system and solar- Engr. Abdul-Azeez.pdfHybrid optimization of pumped hydro system and solar- Engr. Abdul-Azeez.pdf
Hybrid optimization of pumped hydro system and solar- Engr. Abdul-Azeez.pdf
 
Student information management system project report ii.pdf
Student information management system project report ii.pdfStudent information management system project report ii.pdf
Student information management system project report ii.pdf
 
Unbalanced Three Phase Systems and circuits.pptx
Unbalanced Three Phase Systems and circuits.pptxUnbalanced Three Phase Systems and circuits.pptx
Unbalanced Three Phase Systems and circuits.pptx
 
Final project report on grocery store management system..pdf
Final project report on grocery store management system..pdfFinal project report on grocery store management system..pdf
Final project report on grocery store management system..pdf
 
一比一原版(UMich毕业证)密歇根大学|安娜堡分校毕业证成绩单专业办理
一比一原版(UMich毕业证)密歇根大学|安娜堡分校毕业证成绩单专业办理一比一原版(UMich毕业证)密歇根大学|安娜堡分校毕业证成绩单专业办理
一比一原版(UMich毕业证)密歇根大学|安娜堡分校毕业证成绩单专业办理
 
An Approach to Detecting Writing Styles Based on Clustering Techniques
An Approach to Detecting Writing Styles Based on Clustering TechniquesAn Approach to Detecting Writing Styles Based on Clustering Techniques
An Approach to Detecting Writing Styles Based on Clustering Techniques
 
NUMERICAL SIMULATIONS OF HEAT AND MASS TRANSFER IN CONDENSING HEAT EXCHANGERS...
NUMERICAL SIMULATIONS OF HEAT AND MASS TRANSFER IN CONDENSING HEAT EXCHANGERS...NUMERICAL SIMULATIONS OF HEAT AND MASS TRANSFER IN CONDENSING HEAT EXCHANGERS...
NUMERICAL SIMULATIONS OF HEAT AND MASS TRANSFER IN CONDENSING HEAT EXCHANGERS...
 

Hot and cold data storage

  • 1. HOT COLD Unified Virtual File System For Hot & Cold Data Storage Aditya Ambre Madhura S. Raghavan Rohit Arora ENTERPRISE STORAGE ARCHITECTURE GROUP 2
  • 2. HOT COLD CSC 568 Enterprise Storage Architecture (NC State University) AGENDA ➔ Problem Statement ➔ Project Goals and Features ➔ Architecture and Workflow ➔ Verification Cases ➔ Summary
  • 3. Least Frequently Accessed Data HOT COLD CSC 568 Enterprise Storage Architecture (NC State University) PROBLEM STATEMENT ➔ Lifecycle of Data. ◆ Access frequency. ◆ Storage capacity and hardware characteristics. ➔ User intervention - Running jobs/scripts. ➔ Acknowledging Data temperature ➔ Tight coupling needed between storage components Frequently Accessed Data
  • 4. HOT COLD CSC 568 Enterprise Storage Architecture (NC State University) WHAT IS A HOT FILE? Data File that ➔ Very frequently accessed. ➔ Mostly contains business critical information. ➔ Needs to be accessed quickly.
  • 5. HOT COLD CSC 568 Enterprise Storage Architecture (NC State University) WHAT IS A COLD FILE? Data File that ➔ Is infrequently accessed. ➔ Contains less important information. ➔ Need not be quickly accessed.
  • 6. HOT COLD CSC 568 Enterprise Storage Architecture (NC State University) GOAL: WHAT OUR PROJECT IS? ➔ From decoupled storage components - To - tightly coupled two- tiered storage system ➔ Manage hot & cold data between primary and secondary storage. ➔ Manage primary storage space utilization. ➔ File transfer do not interrupt FS operations. ➔ User agnostic about file transfer and storage. ➔ Optimal storage of cold data.
  • 7. HOT COLD CSC 568 Enterprise Storage Architecture (NC State University) WHAT OUR PROJECT IS?
  • 8. HOT COLD CSC 568 Enterprise Storage Architecture (NC State University) FEATURES ➔ Infinite Storage illusion ➔ Automatic cold data identification and transfer ➔ Consistent CRUD operations for both hot and cold files ➔ Block level storage ➔ On the fly deduplication ➔ Uninterrupted file access ➔ File level Consistency ➔ Optimal storage space utilization
  • 9. HOT COLD CSC 568 Enterprise Storage Architecture (NC State University) OUR ARCHITECTURE Cold File Tracking Hot File Tracking File Tracking Layer Data Block Processing Layer Write block to cold Get block from cold De-duplication COLD STORAGE APPLICATION Write Read FUSE OPERATIONS Read, Write, Delete, Rename, etc. 2f0f3ff2c7439635e7faa85… 3f35ec5fe4ae0b963779c8… 4a8f9ec938243beac4b2d… Hot File Cold File
  • 10. HOT COLD CSC 568 Enterprise Storage Architecture (NC State University) HOT-TO-COLD WORKFLOW COLD STORAGE APPLICATION Write FUSE {WRITE} OPERATIONS File Tracking Layer Data Block Processing Layer
  • 11. HOT COLD CSC 568 Enterprise Storage Architecture (NC State University) HOT-TO-COLD WORKFLOW COLD STORAGE APPLICATION Write FUSE {WRITE} OPERATIONS Check: Storage > 70% File Tracking Layer Data Block Processing Layer
  • 12. HOT COLD CSC 568 Enterprise Storage Architecture (NC State University) HOT-TO-COLD WORKFLOW COLD STORAGE APPLICATION Write FUSE {WRITE} OPERATIONS Check: Storage > 70% Cold File Tracking File Tracking Layer Data Block Processing Layer
  • 13. HOT COLD CSC 568 Enterprise Storage Architecture (NC State University) HOT-TO-COLD WORKFLOW File Tracking Layer 1. List all the files 2. Sort files by access time - oldest to newest 3. Select files to be transferred - (till <=50%) 4. Sort above files by size - large to small 5. Send the largest & least accessed files to Data Processing layer Cold File tracking
  • 14. HOT COLD CSC 568 Enterprise Storage Architecture (NC State University) HOT-TO-COLD WORKFLOW File Tracking Layer File 1 1:30 PM 100 KB File 2 4:30 PM 500 KB File 3 3:30 PM 250 KB File 4 2:30 PM 350 KB File 1 1:30 PM 100 KB File 2 4:30 PM 500 KB File 3 3:30 PM 250 KB File 4 2:30 PM 350 KB File 1 1:30 PM 100 KB File 4 2:30 PM 350 KB File 3 3:30 PM 250 KB
  • 15. HOT COLD CSC 568 Enterprise Storage Architecture (NC State University) HOT-TO-COLD WORKFLOW COLD STORAGE APPLICATION Write FUSE {WRITE} OPERATIONS Check: Storage > 70% File Tracking Layer Cold File Tracking Data Block Processing Layer Write block to cold Cold File
  • 16. 2f0f3ff2… 7439635… e7faa85… 3f35ec5f… e4ae0b9... HOT COLD CSC 568 Enterprise Storage Architecture (NC State University) HOT-TO-COLD WORKFLOW Data Block Processing Layer 1. Request Hashtable 2. Get Hashtable Write Block to Cold COLD STORAGE 1. Request Hashtable 2. Gets Hashtable
  • 17. HOT COLD CSC 568 Enterprise Storage Architecture (NC State University) HOT-TO-COLD WORKFLOW Data Block Processing Layer 1. Request Hashtable 2. Get Hashtable 3. Calculate block level hash 4. Check for de-duplication Write Block to Cold COLD STORAGE 4. Duplicate? 2f0f3ff2… 7439635… e7faa85… 3f35ec5f… e4ae0b9... Block 1 Block 2 Block 3
  • 18. HOT COLD CSC 568 Enterprise Storage Architecture (NC State University) HOT-TO-COLD WORKFLOW Data Block Processing Layer 1. Request Hashtable 2. Get Hashtable 3. Calculate block level hash 4. Check for de-duplication 5. Transfer if not duplicate 6. Free block’s memory Write Block to Cold COLD STORAGE 5. Transfer Block 2f0f3ff2… 7439635… e7faa85… 3f35ec5f… e4ae0b9... 5. Update Hashtable Block 1 Block 2 Block 3
  • 19. HOT COLD CSC 568 Enterprise Storage Architecture (NC State University) HOT-TO-COLD WORKFLOW Data Block Processing Layer 1. Request Hashtable 2. Get Hashtable 3. Calculate block level hash 4. Check for de-duplication 5. Transfer if not duplicate 6. Free block’s memory 7. Send updated hashtable to cold storage Write Block to Cold COLD STORAGE 7. Send Updated Hashtable 2f0f3ff2… 7439635… e7faa85… 3f35ec5f… e4ae0b9... 6.
  • 20. HOT COLD CSC 568 Enterprise Storage Architecture (NC State University) HOT-TO-COLD WORKFLOW COLD STORAGE APPLICATION Write FUSE {WRITE} OPERATIONS Check: Storage <= 50% File Tracking Layer Cold File Tracking Data Block Processing Layer Write block to cold Cold File De-duplication 2f0f3ff2c7439635e7faa85… 3f35ec5fe4ae0b963779c8… 4a8f9ec938243beac4b2d…
  • 21. HOT COLD CSC 568 Enterprise Storage Architecture (NC State University) COLD-TO-HOT WORKFLOW COLD STORAGE APPLICATION FUSE {READ} OPERATIONS File Tracking Layer Data Block Processing Layer Read Request 2f0f3ff2c7439635e7faa85… 3f35ec5fe4ae0b963779c8… 4a8f9ec938243beac4b2d…
  • 22. HOT COLD CSC 568 Enterprise Storage Architecture (NC State University) COLD-TO-HOT WORKFLOW COLD STORAGE APPLICATION FUSE {READ} OPERATIONS File Tracking Layer Data Block Processing Layer Read Request Check: Is File on Hot Storage? 2f0f3ff2c7439635e7faa85… 3f35ec5fe4ae0b963779c8… 4a8f9ec938243beac4b2d…
  • 23. HOT COLD CSC 568 Enterprise Storage Architecture (NC State University) COLD-TO-HOT WORKFLOW COLD STORAGE APPLICATION FUSE {READ} OPERATIONS File Tracking Layer Data Block Processing Layer Read Request Check: Is File on Hot Storage? Get block from cold No 2f0f3ff2c7439635e7faa85… 3f35ec5fe4ae0b963779c8… 4a8f9ec938243beac4b2d…
  • 24. HOT COLD CSC 568 Enterprise Storage Architecture (NC State University) COLD-TO-HOT WORKFLOW Data Block Processing Layer 1. Request copy of Hashtable 2. Get Hashtable Get Block from Cold COLD STORAGE 1. Request Hashtable 2. Gets Hashtable
  • 25. 2f0f3ff2… 7439635… e7faa85… 3f35ec5f… e4ae0b9... HOT COLD CSC 568 Enterprise Storage Architecture (NC State University) COLD-TO-HOT WORKFLOW Data Block Processing Layer 1. Request copy of Hashtable 2. Get Hashtable 3. Read block presence on cold Get Block from Cold COLD STORAGE 3. Is block present?
  • 26. HOT COLD CSC 568 Enterprise Storage Architecture (NC State University) COLD-TO-HOT WORKFLOW Data Block Processing Layer 1. Request copy of Hashtable 2. Get Hashtable 3. Read block presence on cold 4. Request/Get block from cold Get Block from Cold COLD STORAGE 4 Request Block 4. Gets Block 2f0f3ff2… 7439635… e7faa85… 3f35ec5f… e4ae0b9... 2f0f3ff2… 7439635… e7faa85… 3f35ec5f… e4ae0b9... Block 1 Block 2 Block 3
  • 27. 2f0f3ff2… 7439635… e7faa85… 3f35ec5f… e4ae0b9... HOT COLD CSC 568 Enterprise Storage Architecture (NC State University) COLD-TO-HOT WORKFLOW Data Block Processing Layer 1. Request copy of Hashtable 2. Get Hashtable 3. Read block presence on cold 4. Request/Get block from cold 5. Write transferred’ block content to memory block 6. Construct complete file Get Block from Cold COLD STORAGE Block 1 Block 2 Block 3 6.
  • 28. 2f0f3ff2… 7439635… e7faa85… 3f35ec5f… e4ae0b9... HOT COLD CSC 568 Enterprise Storage Architecture (NC State University) COLD-TO-HOT WORKFLOW Data Block Processing Layer 1. Request copy of Hashtable 2. Get Hashtable 3. Read block presence on cold 4. Request/Get block from cold 5. Write transferred’ block content to memory block 6. Construct complete file 7. Delete copy of Hashtable Get Block from Cold COLD STORAGE Block 1 Block 2 Block 3 7. Delete Hashtable
  • 29. HOT COLD CSC 568 Enterprise Storage Architecture (NC State University) COLD-TO-HOT WORKFLOW COLD STORAGE APPLICATION FUSE {READ} OPERATIONS File Tracking Layer Data Block Processing Layer ReadRead Request Get block from cold Block Read Request No 2f0f3ff2c7439635e7faa85… 3f35ec5fe4ae0b963779c8… 4a8f9ec938243beac4b2d…
  • 30. HOT COLD CSC 568 Enterprise Storage Architecture (NC State University) MINIMAL THRESHOLD WORKFLOW COLD STORAGE APPLICATION FUSE {READ} OPERATIONS File Tracking Layer Data Block Processing Layer Some Operation Get block from cold Block Read Request Yes 2f0f3ff2c7439635e7faa85… 3f35ec5fe4ae0b963779c8… 4a8f9ec938243beac4b2d… Check: Storage <= 30% Get Cold FileHot File Tracking
  • 31. HOT COLD CSC 568 Enterprise Storage Architecture (NC State University) READ OPERATION WORKFLOW COLD STORAGE APPLICATION FUSE {READ} OPERATIONS File Tracking Layer Data Block Processing Layer Some Operation Get block from cold Block Read Request Yes 2f0f3ff2c7439635e7faa85… 3f35ec5fe4ae0b963779c8… 4a8f9ec938243beac4b2d… Check: Storage >30% & < 70% Get Cold FileHot File Tracking
  • 32. HOT COLD CSC 568 Enterprise Storage Architecture (NC State University) QUICK DEMO
  • 33. HOT COLD CSC 568 Enterprise Storage Architecture (NC State University) SCENARIOS / VERIFICATION CASES I. GENERAL ➔ File System 70% full -> Transfer to cold storage. ➔ File System drops less than 30% -> Transfer from cold storage. ➔ File transfers -> Do not interrupt general FS operations. ➔ Redundant/Duplicate blocks ->Not transferred.
  • 34. HOT COLD CSC 568 Enterprise Storage Architecture (NC State University) SCENARIOS / VERIFICATION CASES II. SPECIFIC ➔ Files transferred –> Based on access and size. ➔ File removed on hot storage –> After last block is transferred. ➔ File in transition accessed –> Abort transfer, access granted! ➔ File space reclamation and File access –> Synchronized. ➔ Only one background process running at specific time. ➔ Delayed delete (rm) -> Transparent to user.
  • 35. HOT COLD CSC 568 Enterprise Storage Architecture (NC State University) ASSUMPTIONS ➔ Network is always available. ➔ Hot-Cold classification at file level ➔ Cold Storage is infinite. ➔ Files are not very small or very large. ➔ Delay is accepted for rarely accessed files. ➔ File access granularity – in seconds.
  • 36. HOT COLD CSC 568 Enterprise Storage Architecture (NC State University) SUMMARY ➔ Acknowledged data temperatures - hot and cold ➔ Project Features ◆ Auto file identification. ◆ File transfer ◆ Deduplication ➔ Architecture and workflows in action. ➔ Design and implementation of file tracking layer ➔ Design and implementation of Block Data Process Layer ➔ Design decisions for specific verification scenarios.
  • 37. HOT COLD CSC 568 Enterprise Storage Architecture (NC State University) FUTURE SCOPE ➔ Variable block size and Block size specifications. ➔ Garbage collection on secondary/cold storage. ➔ Cold file identification parameters and profiles. ➔ Distributed cold storage.
  • 38. HOT COLD CSC 568 Enterprise Storage Architecture (NC State University) REFERENCES 1. S. Quinlan and S. Dorward, “Venti: A new approach to archival storage,” in Proceedings of the First USENIX Conference on File and Storage Technologies (FAST), 2002. http://plan9.bell-labs.com/sys/doc/venti/venti. pdf 2. Chuanyi Liu, Dapeng Ju, et al, “Semantic data de-duplication for archival storage systems,” in Proceedings of the 13th IEEE Asia-Pacific Computer Systems Architecture Conference (ACSAC 2008), Hsinchu, Taiwan, August, 2008. 3. Sean Quinlan, Jim McKie Russ Cox, “Fossil, an Archival File Server”, Lucent Technologies Bell Labs, Unpublished memorandum (September 2003). 4. http://www.storiant.com/resources/Cold-Storage-Is-Hot-Again.pdf 5. “What is Unified Storage system ” http://searchstorage.techtarget. com/definition/unified-storage 6. File System in User Space - http://fuse.sourceforge.net/
  • 39. HOT COLD CSC 568 Enterprise Storage Architecture (NC State University) QUESTIONS ?