SlideShare a Scribd company logo
Decentralised Storage
Anurag Dashputre
Agenda
What is Decentralisation + Storage?
What is Decentralised Storage?
Why Decentralised Storage?
Decentralised Storage - Current Projects
Swarm
IPFS - Deep Dive
Basic Commands
Advanced Commands
Demo
Appendix
2
What is Decentralisation + Storage?
Decentralisation is understood as the transfer of authority from a central entity to
a more localised and ‘liberal’ system.
Storage is defined as the retention of retrievable data on a computer or other
electronic system.
3
What is Decentralised Storage?
Decentralised storage is a system of being able to store your files without having
to rely on large, centralized silos of data that don’t undermine important values
such as privacy and freedom of your information.
It is Content-Addressable, rather than Location-Addressable. Every file has a
unique hash of its content.
4
Why Decentralised Storage?
● Availability
○ Censorship Resistant
○ Data geographically spread
○ No "404 Page Not found" error
● Security & Privacy
○ No centralised server storage hence very difficult to hack and breach data
○ Files are not stored directly but as chunks of data spread across multiple nodes
● Cost reduction due to more efficiency
5
Decentralised Storage - Current Projects
● Swarm
● IPFS (Inter Planetary File System)
● Sia
● Storj
6
Swarm
7
Swarm is a distributed storage platform and content distribution service, a native
base layer service of the ethereum web3 stack.
The primary objective of Swarm is to provide a sufficiently decentralized and
redundant store of Ethereum’s public record, in particular to store and distribute
dapp code and data as well as blockchain data.
Refer Swarm Documentation for further details
IPFS - Deep Dive
8
What is IPFS?
9
IPFS is a distributed peer-to-peer (p2p) file sharing system for storing and
accessing files, websites, applications, and data.
IPFS aims to replace HTTP and build a better web for all of us.
HTTP v/s IPFS
​Today, the Internet is based on HyperText
Transfer Protocol (HTTP). HTTP relies on
location addressing which uses IP addresses to
identify the specific server that is hosting the
requested information. This means that the
information has to be fetched from the origin
server.
IPFS is meant to be a replacement for HTTP.
Most notably, IPFS never has a single point of
failure. It’s a peer-to-peer distributed file system
that would decentralize the Internet and make it
much more difficult for a service provider or
hosting network to pull the plug and make
published information suddenly disappear. 10
HTTP vs. IPFS [Source: https://www.maxcdn.com/one/visual-
glossary/interplanetary-file-system/]
How IPFS works?
11
IPFS works by connecting all computing devices with the same system
of files via a system of nodes. It uses a “distributed hash table, an
incentivized block exchange, and a self-certifying namespace.”
In simpler terms, it acts similarly to a torrent system, except that instead of
sharing and exchanging media, IPFS exchanges git objects. This means that the
whole system is based around a simple key-value data store. Any type of content
can be inserted, and it will give back a key that can be used to retrieve the content
again at any time. This is what allows for content addressing instead of location
addressing: The key is completely independent of the origin of the information and
can be hosted anywhere.
How IPFS stores data?
When you add any content on IPFS network, the data is split into chunks of 256Kb.
Each chunk is identified with it’s own hash. These chunks are then distributed to
various nodes on network which have there hash closest to peerId.
12
How IPFS stores data? ...Continued
1. Let us assume that there are 4 nodes with peerId 6789,
789a, 89ab, 9abc respectively
2. We try to add a file name(size= 1Mb) something.mp4. Your
node first calculates that hash of the file, say 7abc.
Additionally the file is broken into 4 chunks of 256 Kb each.
Your node then calculates the hash of the each chunk, say
(7aaa, 8abc, 9a23, 5bcd)
3. Now node broadcasts the each chunk to node with has the
closest peerId numerically. In our mentioned example
chunk with hash 7aaa it closest to hash 789a. Hence this
chunk is send to node with peerId 789a.
4. Similarly, all chunks are send and there address in updated
in DHT.
5. Lastly, the object root hash i.e 7abc is stored, (Root hash
can be stored anywhere, it is assumed that in current
example it is stored in our system) and hashes that it links
to i.e 7abc → [7aaa, 8abc, 9a23, 5bcd]
[Source: https://medium.com/@akshay_111meher/how-ipfs-works-
545e1c890437] 13
How data is divided and stored. Root hash is assumed to be stored on your node.
It is however stored in same way chunks are stored. It could be on anyone,
including yours.
IPFS File Storage - Live Demo
14
How data is retrieved?
On IPFS network, the file is identified solely by it HASH (root hash), in our case
7abc. Once the user requests a file, the request traverses to nodes where hash is
existing using the DHT. If the data points to other chunks (like in our case), even
they are searched same way. Once all chunks are obtained, all of them are simply
concatenated to obtain the main object.
15
Basic
Commands
16
● Install IPFS
● Get IPFS version
● Initialize the IPFS repository
● Get IPFS node id
● Start IPFS node
● Check Peer Nodes
Get IPFS version
17
● Install IPFS
● Get IPFS version
● Initialize the IPFS repository
● Get IPFS node id
● Start IPFS node
● Check Peer nodes
ipfs version
ipfs version 0.4.18
Initialize the IPFS repository
18
● Install IPFS
● Get IPFS version
● Initialize the IPFS repository
● Get IPFS node id
● Start IPFS node
● Check Peer nodes
ipfs init
initializing ipfs node at
/Users/jbenet/.go-ipfs
generating 2048-bit RSA
keypair...done
peer identity:
Qmcpo2iLBikrdf1d6QU6vXuNb6P7hwrbNPW9k
LAH8eG67z
to get started, enter:
ipfs cat
/ipfs/QmYwAPJzv5CZsnA625s3Xf2nemtYgPp
HdWEz79ojWnPbdG/readme
Get IPFS node id
19
● Install IPFS
● Get IPFS version
● Initialize the IPFS repository
● Get IPFS node id
● Start IPFS node
● Check Peer nodes
ipfs id
"ID":
"QmP7JssmhNTpayGoK5ZhBt78hRRBi3VBYQ
yqwMsBsSZBSW"
Start IPFS node
20
● Install IPFS
● Get IPFS version
● Initialize the IPFS repository
● Get IPFS node id
● Start IPFS node
● Check Peer nodes
ipfs daemon
Initializing daemon...
go-ipfs version: 0.4.18-
Repo version: 7
System version: amd64/darwin
Golang version: go1.11.1
Successfully raised file descriptor limit to 2048.
Swarm listening on /ip4/127.0.0.1/tcp/4001
.
Swarm announcing /ip4/127.0.0.1/tcp/4001
.
Daemon is ready
Check peer nodes
21
● Install IPFS
● Get IPFS version
● Initialize the IPFS repository
● Get IPFS node id
● Start IPFS node
● Check Peer nodes
ipfs swarm peers
Advanced
Commands
22
● Check IPFS repository
statistics
● Add a file to IPFS
● Pin objects to local storage
● Remove pinned objects from
local storage
● Download IPFS objects
● Show IPFS object data
● List objects pinned to local
storage
Check IPFS repository statistics
23
● Check IPFS repository statistics
● Add a file to IPFS
● Pin objects to local storage
● Remove pinned objects from
local storage
● Download IPFS objects
● Show IPFS object data
● List objects pinned to local
storage
ipfs stats repo
NumObjects: 4817
RepoSize: 127963949
StorageMax: 10000000000
RepoPath: /Users/anuragd/.ipfs
Version: fs-repo@7
Add a file to IPFS
24
● Check IPFS repository statistics
● Add a file to IPFS
● Pin objects to local storage
● Remove pinned objects from
local storage
● Download IPFS objects
● Show IPFS object data
● List objects pinned to local
storage
ipfs add temp
added
QmWATWQ7fVPP2EFGu71UkfnqhYXDYH566qy
47CnJDgvs8u temp
12 B / 12 B
[==================================
===================================
===================================
===============================]
100.00%
Pin objects to local storage
25
● Check IPFS repository statistics
● Add a file to IPFS
● Pin objects to local storage
● Remove pinned objects from
local storage
● Download IPFS objects
● Show IPFS object data
● List objects pinned to local
storage
ipfs pin add
QmWATWQ7fVPP2EFGu71UkfnqhYXDYH566qy
47CnJDgvs8u
pinned
QmWATWQ7fVPP2EFGu71UkfnqhYXDYH566qy
47CnJDgvs8u recursively
Remove pinned objects from local storage
26
● Check IPFS repository statistics
● Add a file to IPFS
● Pin objects to local storage
● Remove pinned objects from
local storage
● Download IPFS objects
● Show IPFS object data
● List objects pinned to local
storage
ipfs pin rm
QmWATWQ7fVPP2EFGu71UkfnqhYXDYH566qy
47CnJDgvs8u
unpinned
QmWATWQ7fVPP2EFGu71UkfnqhYXDYH566qy
47CnJDgvs8u
Download IPFS objects
27
● Check IPFS repository statistics
● Add a file to IPFS
● Pin objects to local storage
● Remove pinned objects from
local storage
● Download IPFS objects
● Show IPFS object data
● List objects pinned to local
storage
ipfs get
QmWATWQ7fVPP2EFGu71UkfnqhYXDYH566qy
47CnJDgvs8u
Saving file(s) to
QmWATWQ7fVPP2EFGu71UkfnqhYXDYH566qy
47CnJDgvs8u
20 B / 20 B
[==================================
===================================
===================================
============================]
100.00% 0s
Show IPFS object data
28
● Check IPFS repository statistics
● Add a file to IPFS
● Pin objects to local storage
● Remove pinned objects from
local storage
● Download IPFS objects
● Show IPFS object data
● List objects pinned to local
storage
ipfs cat
QmWATWQ7fVPP2EFGu71UkfnqhYXDYH566qy
47CnJDgvs8u
Hello World
ipfs cat
QmWATWQ7fVPP2EFGu71UkfnqhYXDYH566qy
47CnJDgvs8u > temp1
cat temp1
Hello World
List objects pinned to local storage
29
● Check IPFS repository statistics
● Add a file to IPFS
● Pin objects to local storage
● Remove pinned objects from
local storage
● Download IPFS objects
● Show IPFS object data
● List objects pinned to local
storage
ipfs pin ls
Thank You
32

More Related Content

What's hot

Introduction to Software Defined Networking (SDN)
Introduction to Software Defined Networking (SDN)Introduction to Software Defined Networking (SDN)
Introduction to Software Defined Networking (SDN)
Bangladesh Network Operators Group
 
Distributed Computing
Distributed ComputingDistributed Computing
Distributed Computing
Sudarsun Santhiappan
 
VPN - Virtual Private Network
VPN - Virtual Private NetworkVPN - Virtual Private Network
VPN - Virtual Private Network
Peter R. Egli
 
eBPF Workshop
eBPF WorkshopeBPF Workshop
eBPF Workshop
Michael Kehoe
 
Rapid miner
Rapid minerRapid miner
Rapid miner
Manish Champaneri
 
Sdn ppt
Sdn pptSdn ppt
Introduction to Apache ZooKeeper
Introduction to Apache ZooKeeperIntroduction to Apache ZooKeeper
Introduction to Apache ZooKeeper
Saurav Haloi
 
Distributed computing
Distributed computingDistributed computing
Distributed computing
Alokeparna Choudhury
 
Introduction to IoT Architectures and Protocols
Introduction to IoT Architectures and ProtocolsIntroduction to IoT Architectures and Protocols
Introduction to IoT Architectures and Protocols
Abdullah Alfadhly
 
Google cloud-platform-official-icons-and-sample-diagrams
Google cloud-platform-official-icons-and-sample-diagramsGoogle cloud-platform-official-icons-and-sample-diagrams
Google cloud-platform-official-icons-and-sample-diagrams
Dhamotharan Paramasivam
 
The Basic Introduction of Open vSwitch
The Basic Introduction of Open vSwitchThe Basic Introduction of Open vSwitch
The Basic Introduction of Open vSwitch
Te-Yen Liu
 
Virtual private networks (vpn)
Virtual private networks (vpn)Virtual private networks (vpn)
Virtual private networks (vpn)
Avinash Nath
 
Virtual Private Network(VPN)
Virtual Private Network(VPN)Virtual Private Network(VPN)
Virtual Private Network(VPN)
Abrish06
 
Istio
Istio Istio
Istio
Arun prasath
 
NAT (network address translation) & PAT (port address translation)
NAT (network address translation) & PAT (port address translation)NAT (network address translation) & PAT (port address translation)
NAT (network address translation) & PAT (port address translation)
Netwax Lab
 
Data Structures in and on IPFS
Data Structures in and on IPFSData Structures in and on IPFS
Data Structures in and on IPFS
C4Media
 
Task scheduling Survey in Cloud Computing
Task scheduling Survey in Cloud ComputingTask scheduling Survey in Cloud Computing
Task scheduling Survey in Cloud Computing
Ramandeep Kaur
 
Load balancing in cloud computing.pptx
Load balancing in cloud computing.pptxLoad balancing in cloud computing.pptx
Load balancing in cloud computing.pptx
Hitesh Mohapatra
 
Pixeom
PixeomPixeom
Exploring Universal API Management And Flex Gateway
Exploring Universal API Management And Flex GatewayExploring Universal API Management And Flex Gateway
Exploring Universal API Management And Flex Gateway
shyamraj55
 

What's hot (20)

Introduction to Software Defined Networking (SDN)
Introduction to Software Defined Networking (SDN)Introduction to Software Defined Networking (SDN)
Introduction to Software Defined Networking (SDN)
 
Distributed Computing
Distributed ComputingDistributed Computing
Distributed Computing
 
VPN - Virtual Private Network
VPN - Virtual Private NetworkVPN - Virtual Private Network
VPN - Virtual Private Network
 
eBPF Workshop
eBPF WorkshopeBPF Workshop
eBPF Workshop
 
Rapid miner
Rapid minerRapid miner
Rapid miner
 
Sdn ppt
Sdn pptSdn ppt
Sdn ppt
 
Introduction to Apache ZooKeeper
Introduction to Apache ZooKeeperIntroduction to Apache ZooKeeper
Introduction to Apache ZooKeeper
 
Distributed computing
Distributed computingDistributed computing
Distributed computing
 
Introduction to IoT Architectures and Protocols
Introduction to IoT Architectures and ProtocolsIntroduction to IoT Architectures and Protocols
Introduction to IoT Architectures and Protocols
 
Google cloud-platform-official-icons-and-sample-diagrams
Google cloud-platform-official-icons-and-sample-diagramsGoogle cloud-platform-official-icons-and-sample-diagrams
Google cloud-platform-official-icons-and-sample-diagrams
 
The Basic Introduction of Open vSwitch
The Basic Introduction of Open vSwitchThe Basic Introduction of Open vSwitch
The Basic Introduction of Open vSwitch
 
Virtual private networks (vpn)
Virtual private networks (vpn)Virtual private networks (vpn)
Virtual private networks (vpn)
 
Virtual Private Network(VPN)
Virtual Private Network(VPN)Virtual Private Network(VPN)
Virtual Private Network(VPN)
 
Istio
Istio Istio
Istio
 
NAT (network address translation) & PAT (port address translation)
NAT (network address translation) & PAT (port address translation)NAT (network address translation) & PAT (port address translation)
NAT (network address translation) & PAT (port address translation)
 
Data Structures in and on IPFS
Data Structures in and on IPFSData Structures in and on IPFS
Data Structures in and on IPFS
 
Task scheduling Survey in Cloud Computing
Task scheduling Survey in Cloud ComputingTask scheduling Survey in Cloud Computing
Task scheduling Survey in Cloud Computing
 
Load balancing in cloud computing.pptx
Load balancing in cloud computing.pptxLoad balancing in cloud computing.pptx
Load balancing in cloud computing.pptx
 
Pixeom
PixeomPixeom
Pixeom
 
Exploring Universal API Management And Flex Gateway
Exploring Universal API Management And Flex GatewayExploring Universal API Management And Flex Gateway
Exploring Universal API Management And Flex Gateway
 

Similar to Decentralized storage

Distributed Storage with IPFS and Python!
Distributed Storage with IPFS and Python!Distributed Storage with IPFS and Python!
Distributed Storage with IPFS and Python!
Abhinav Srivastava
 
ipfs-171229085327.pdf
ipfs-171229085327.pdfipfs-171229085327.pdf
ipfs-171229085327.pdf
MrNobody695544
 
Introduction to Filecoin
Introduction to Filecoin   Introduction to Filecoin
Introduction to Filecoin
Vanessa Lošić
 
Ceph Day Tokyo - Bring Ceph to Enterprise
Ceph Day Tokyo - Bring Ceph to Enterprise Ceph Day Tokyo - Bring Ceph to Enterprise
Ceph Day Tokyo - Bring Ceph to Enterprise
Ceph Community
 
OSSV [Open System SnapVault]
OSSV [Open System SnapVault]OSSV [Open System SnapVault]
OSSV [Open System SnapVault]
Ashwin Pawar
 
Spectrum Scale Unified File and Object with WAN Caching
Spectrum Scale Unified File and Object with WAN CachingSpectrum Scale Unified File and Object with WAN Caching
Spectrum Scale Unified File and Object with WAN Caching
Sandeep Patil
 
Software Defined Analytics with File and Object Access Plus Geographically Di...
Software Defined Analytics with File and Object Access Plus Geographically Di...Software Defined Analytics with File and Object Access Plus Geographically Di...
Software Defined Analytics with File and Object Access Plus Geographically Di...
Trishali Nayar
 
2012 09-08-josug-jeff
2012 09-08-josug-jeff2012 09-08-josug-jeff
2012 09-08-josug-jeff
Zheng (Jeff) Xu
 
Ipfs installation and demo
Ipfs installation and demoIpfs installation and demo
Ipfs installation and demo
Naveen Davis
 
Freezer - Vietnam OpenStack Technical Meetup #12
Freezer - Vietnam OpenStack Technical Meetup #12Freezer - Vietnam OpenStack Technical Meetup #12
Freezer - Vietnam OpenStack Technical Meetup #12
Vietnam Open Infrastructure User Group
 
Introduction to IPFS & Filecoin - longer version
Introduction to IPFS & Filecoin - longer versionIntroduction to IPFS & Filecoin - longer version
Introduction to IPFS & Filecoin - longer version
TinaBregovi
 
NameNode Analytics - Querying HDFS Namespace in Real Time
NameNode Analytics - Querying HDFS Namespace in Real TimeNameNode Analytics - Querying HDFS Namespace in Real Time
NameNode Analytics - Querying HDFS Namespace in Real Time
Plamen Jeliazkov
 
Apache Spark and Object Stores —for London Spark User Group
Apache Spark and Object Stores —for London Spark User GroupApache Spark and Object Stores —for London Spark User Group
Apache Spark and Object Stores —for London Spark User Group
Steve Loughran
 
Ceph Day Bring Ceph To Enterprise
Ceph Day Bring Ceph To EnterpriseCeph Day Bring Ceph To Enterprise
Ceph Day Bring Ceph To Enterprise
Alex Lau
 
Peter Tiernan - Ceph at the Digital Repository of Ireland
Peter Tiernan - Ceph at the Digital Repository of IrelandPeter Tiernan - Ceph at the Digital Repository of Ireland
Peter Tiernan - Ceph at the Digital Repository of Ireland
dri_ireland
 
Ceph at the Digital Repository of Ireland - Ceph Day Frankfurt
Ceph at the Digital Repository of Ireland - Ceph Day Frankfurt Ceph at the Digital Repository of Ireland - Ceph Day Frankfurt
Ceph at the Digital Repository of Ireland - Ceph Day Frankfurt
Ceph Community
 
AIDevWorldApacheNiFi101
AIDevWorldApacheNiFi101AIDevWorldApacheNiFi101
AIDevWorldApacheNiFi101
Timothy Spann
 
Introduction to Stacki - World's fastest Linux server provisioning Tool
Introduction to Stacki - World's fastest Linux server provisioning ToolIntroduction to Stacki - World's fastest Linux server provisioning Tool
Introduction to Stacki - World's fastest Linux server provisioning Tool
Suresh Paulraj
 
Ceph Day Beijing: Big Data Analytics on Ceph Object Store
Ceph Day Beijing: Big Data Analytics on Ceph Object Store Ceph Day Beijing: Big Data Analytics on Ceph Object Store
Ceph Day Beijing: Big Data Analytics on Ceph Object Store
Ceph Community
 
Security myth of i pv6 and dns64
Security myth of i pv6 and dns64Security myth of i pv6 and dns64
Security myth of i pv6 and dns64
A. S. M. Shamim Reza
 

Similar to Decentralized storage (20)

Distributed Storage with IPFS and Python!
Distributed Storage with IPFS and Python!Distributed Storage with IPFS and Python!
Distributed Storage with IPFS and Python!
 
ipfs-171229085327.pdf
ipfs-171229085327.pdfipfs-171229085327.pdf
ipfs-171229085327.pdf
 
Introduction to Filecoin
Introduction to Filecoin   Introduction to Filecoin
Introduction to Filecoin
 
Ceph Day Tokyo - Bring Ceph to Enterprise
Ceph Day Tokyo - Bring Ceph to Enterprise Ceph Day Tokyo - Bring Ceph to Enterprise
Ceph Day Tokyo - Bring Ceph to Enterprise
 
OSSV [Open System SnapVault]
OSSV [Open System SnapVault]OSSV [Open System SnapVault]
OSSV [Open System SnapVault]
 
Spectrum Scale Unified File and Object with WAN Caching
Spectrum Scale Unified File and Object with WAN CachingSpectrum Scale Unified File and Object with WAN Caching
Spectrum Scale Unified File and Object with WAN Caching
 
Software Defined Analytics with File and Object Access Plus Geographically Di...
Software Defined Analytics with File and Object Access Plus Geographically Di...Software Defined Analytics with File and Object Access Plus Geographically Di...
Software Defined Analytics with File and Object Access Plus Geographically Di...
 
2012 09-08-josug-jeff
2012 09-08-josug-jeff2012 09-08-josug-jeff
2012 09-08-josug-jeff
 
Ipfs installation and demo
Ipfs installation and demoIpfs installation and demo
Ipfs installation and demo
 
Freezer - Vietnam OpenStack Technical Meetup #12
Freezer - Vietnam OpenStack Technical Meetup #12Freezer - Vietnam OpenStack Technical Meetup #12
Freezer - Vietnam OpenStack Technical Meetup #12
 
Introduction to IPFS & Filecoin - longer version
Introduction to IPFS & Filecoin - longer versionIntroduction to IPFS & Filecoin - longer version
Introduction to IPFS & Filecoin - longer version
 
NameNode Analytics - Querying HDFS Namespace in Real Time
NameNode Analytics - Querying HDFS Namespace in Real TimeNameNode Analytics - Querying HDFS Namespace in Real Time
NameNode Analytics - Querying HDFS Namespace in Real Time
 
Apache Spark and Object Stores —for London Spark User Group
Apache Spark and Object Stores —for London Spark User GroupApache Spark and Object Stores —for London Spark User Group
Apache Spark and Object Stores —for London Spark User Group
 
Ceph Day Bring Ceph To Enterprise
Ceph Day Bring Ceph To EnterpriseCeph Day Bring Ceph To Enterprise
Ceph Day Bring Ceph To Enterprise
 
Peter Tiernan - Ceph at the Digital Repository of Ireland
Peter Tiernan - Ceph at the Digital Repository of IrelandPeter Tiernan - Ceph at the Digital Repository of Ireland
Peter Tiernan - Ceph at the Digital Repository of Ireland
 
Ceph at the Digital Repository of Ireland - Ceph Day Frankfurt
Ceph at the Digital Repository of Ireland - Ceph Day Frankfurt Ceph at the Digital Repository of Ireland - Ceph Day Frankfurt
Ceph at the Digital Repository of Ireland - Ceph Day Frankfurt
 
AIDevWorldApacheNiFi101
AIDevWorldApacheNiFi101AIDevWorldApacheNiFi101
AIDevWorldApacheNiFi101
 
Introduction to Stacki - World's fastest Linux server provisioning Tool
Introduction to Stacki - World's fastest Linux server provisioning ToolIntroduction to Stacki - World's fastest Linux server provisioning Tool
Introduction to Stacki - World's fastest Linux server provisioning Tool
 
Ceph Day Beijing: Big Data Analytics on Ceph Object Store
Ceph Day Beijing: Big Data Analytics on Ceph Object Store Ceph Day Beijing: Big Data Analytics on Ceph Object Store
Ceph Day Beijing: Big Data Analytics on Ceph Object Store
 
Security myth of i pv6 and dns64
Security myth of i pv6 and dns64Security myth of i pv6 and dns64
Security myth of i pv6 and dns64
 

Recently uploaded

Deep Dive: AI-Powered Marketing to Get More Leads and Customers with HyperGro...
Deep Dive: AI-Powered Marketing to Get More Leads and Customers with HyperGro...Deep Dive: AI-Powered Marketing to Get More Leads and Customers with HyperGro...
Deep Dive: AI-Powered Marketing to Get More Leads and Customers with HyperGro...
saastr
 
Artificial Intelligence for XMLDevelopment
Artificial Intelligence for XMLDevelopmentArtificial Intelligence for XMLDevelopment
Artificial Intelligence for XMLDevelopment
Octavian Nadolu
 
WeTestAthens: Postman's AI & Automation Techniques
WeTestAthens: Postman's AI & Automation TechniquesWeTestAthens: Postman's AI & Automation Techniques
WeTestAthens: Postman's AI & Automation Techniques
Postman
 
20240609 QFM020 Irresponsible AI Reading List May 2024
20240609 QFM020 Irresponsible AI Reading List May 202420240609 QFM020 Irresponsible AI Reading List May 2024
20240609 QFM020 Irresponsible AI Reading List May 2024
Matthew Sinclair
 
Fueling AI with Great Data with Airbyte Webinar
Fueling AI with Great Data with Airbyte WebinarFueling AI with Great Data with Airbyte Webinar
Fueling AI with Great Data with Airbyte Webinar
Zilliz
 
みなさんこんにちはこれ何文字まで入るの?40文字以下不可とか本当に意味わからないけどこれ限界文字数書いてないからマジでやばい文字数いけるんじゃないの?えこ...
みなさんこんにちはこれ何文字まで入るの?40文字以下不可とか本当に意味わからないけどこれ限界文字数書いてないからマジでやばい文字数いけるんじゃないの?えこ...みなさんこんにちはこれ何文字まで入るの?40文字以下不可とか本当に意味わからないけどこれ限界文字数書いてないからマジでやばい文字数いけるんじゃないの?えこ...
みなさんこんにちはこれ何文字まで入るの?40文字以下不可とか本当に意味わからないけどこれ限界文字数書いてないからマジでやばい文字数いけるんじゃないの?えこ...
名前 です男
 
AI 101: An Introduction to the Basics and Impact of Artificial Intelligence
AI 101: An Introduction to the Basics and Impact of Artificial IntelligenceAI 101: An Introduction to the Basics and Impact of Artificial Intelligence
AI 101: An Introduction to the Basics and Impact of Artificial Intelligence
IndexBug
 
Programming Foundation Models with DSPy - Meetup Slides
Programming Foundation Models with DSPy - Meetup SlidesProgramming Foundation Models with DSPy - Meetup Slides
Programming Foundation Models with DSPy - Meetup Slides
Zilliz
 
How to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdf
How to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdfHow to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdf
How to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdf
Chart Kalyan
 
Cosa hanno in comune un mattoncino Lego e la backdoor XZ?
Cosa hanno in comune un mattoncino Lego e la backdoor XZ?Cosa hanno in comune un mattoncino Lego e la backdoor XZ?
Cosa hanno in comune un mattoncino Lego e la backdoor XZ?
Speck&Tech
 
5th LF Energy Power Grid Model Meet-up Slides
5th LF Energy Power Grid Model Meet-up Slides5th LF Energy Power Grid Model Meet-up Slides
5th LF Energy Power Grid Model Meet-up Slides
DanBrown980551
 
OpenID AuthZEN Interop Read Out - Authorization
OpenID AuthZEN Interop Read Out - AuthorizationOpenID AuthZEN Interop Read Out - Authorization
OpenID AuthZEN Interop Read Out - Authorization
David Brossard
 
Columbus Data & Analytics Wednesdays - June 2024
Columbus Data & Analytics Wednesdays - June 2024Columbus Data & Analytics Wednesdays - June 2024
Columbus Data & Analytics Wednesdays - June 2024
Jason Packer
 
National Security Agency - NSA mobile device best practices
National Security Agency - NSA mobile device best practicesNational Security Agency - NSA mobile device best practices
National Security Agency - NSA mobile device best practices
Quotidiano Piemontese
 
GraphRAG for Life Science to increase LLM accuracy
GraphRAG for Life Science to increase LLM accuracyGraphRAG for Life Science to increase LLM accuracy
GraphRAG for Life Science to increase LLM accuracy
Tomaz Bratanic
 
“Building and Scaling AI Applications with the Nx AI Manager,” a Presentation...
“Building and Scaling AI Applications with the Nx AI Manager,” a Presentation...“Building and Scaling AI Applications with the Nx AI Manager,” a Presentation...
“Building and Scaling AI Applications with the Nx AI Manager,” a Presentation...
Edge AI and Vision Alliance
 
Project Management Semester Long Project - Acuity
Project Management Semester Long Project - AcuityProject Management Semester Long Project - Acuity
Project Management Semester Long Project - Acuity
jpupo2018
 
How to Get CNIC Information System with Paksim Ga.pptx
How to Get CNIC Information System with Paksim Ga.pptxHow to Get CNIC Information System with Paksim Ga.pptx
How to Get CNIC Information System with Paksim Ga.pptx
danishmna97
 
Nordic Marketo Engage User Group_June 13_ 2024.pptx
Nordic Marketo Engage User Group_June 13_ 2024.pptxNordic Marketo Engage User Group_June 13_ 2024.pptx
Nordic Marketo Engage User Group_June 13_ 2024.pptx
MichaelKnudsen27
 
UI5 Controls simplified - UI5con2024 presentation
UI5 Controls simplified - UI5con2024 presentationUI5 Controls simplified - UI5con2024 presentation
UI5 Controls simplified - UI5con2024 presentation
Wouter Lemaire
 

Recently uploaded (20)

Deep Dive: AI-Powered Marketing to Get More Leads and Customers with HyperGro...
Deep Dive: AI-Powered Marketing to Get More Leads and Customers with HyperGro...Deep Dive: AI-Powered Marketing to Get More Leads and Customers with HyperGro...
Deep Dive: AI-Powered Marketing to Get More Leads and Customers with HyperGro...
 
Artificial Intelligence for XMLDevelopment
Artificial Intelligence for XMLDevelopmentArtificial Intelligence for XMLDevelopment
Artificial Intelligence for XMLDevelopment
 
WeTestAthens: Postman's AI & Automation Techniques
WeTestAthens: Postman's AI & Automation TechniquesWeTestAthens: Postman's AI & Automation Techniques
WeTestAthens: Postman's AI & Automation Techniques
 
20240609 QFM020 Irresponsible AI Reading List May 2024
20240609 QFM020 Irresponsible AI Reading List May 202420240609 QFM020 Irresponsible AI Reading List May 2024
20240609 QFM020 Irresponsible AI Reading List May 2024
 
Fueling AI with Great Data with Airbyte Webinar
Fueling AI with Great Data with Airbyte WebinarFueling AI with Great Data with Airbyte Webinar
Fueling AI with Great Data with Airbyte Webinar
 
みなさんこんにちはこれ何文字まで入るの?40文字以下不可とか本当に意味わからないけどこれ限界文字数書いてないからマジでやばい文字数いけるんじゃないの?えこ...
みなさんこんにちはこれ何文字まで入るの?40文字以下不可とか本当に意味わからないけどこれ限界文字数書いてないからマジでやばい文字数いけるんじゃないの?えこ...みなさんこんにちはこれ何文字まで入るの?40文字以下不可とか本当に意味わからないけどこれ限界文字数書いてないからマジでやばい文字数いけるんじゃないの?えこ...
みなさんこんにちはこれ何文字まで入るの?40文字以下不可とか本当に意味わからないけどこれ限界文字数書いてないからマジでやばい文字数いけるんじゃないの?えこ...
 
AI 101: An Introduction to the Basics and Impact of Artificial Intelligence
AI 101: An Introduction to the Basics and Impact of Artificial IntelligenceAI 101: An Introduction to the Basics and Impact of Artificial Intelligence
AI 101: An Introduction to the Basics and Impact of Artificial Intelligence
 
Programming Foundation Models with DSPy - Meetup Slides
Programming Foundation Models with DSPy - Meetup SlidesProgramming Foundation Models with DSPy - Meetup Slides
Programming Foundation Models with DSPy - Meetup Slides
 
How to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdf
How to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdfHow to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdf
How to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdf
 
Cosa hanno in comune un mattoncino Lego e la backdoor XZ?
Cosa hanno in comune un mattoncino Lego e la backdoor XZ?Cosa hanno in comune un mattoncino Lego e la backdoor XZ?
Cosa hanno in comune un mattoncino Lego e la backdoor XZ?
 
5th LF Energy Power Grid Model Meet-up Slides
5th LF Energy Power Grid Model Meet-up Slides5th LF Energy Power Grid Model Meet-up Slides
5th LF Energy Power Grid Model Meet-up Slides
 
OpenID AuthZEN Interop Read Out - Authorization
OpenID AuthZEN Interop Read Out - AuthorizationOpenID AuthZEN Interop Read Out - Authorization
OpenID AuthZEN Interop Read Out - Authorization
 
Columbus Data & Analytics Wednesdays - June 2024
Columbus Data & Analytics Wednesdays - June 2024Columbus Data & Analytics Wednesdays - June 2024
Columbus Data & Analytics Wednesdays - June 2024
 
National Security Agency - NSA mobile device best practices
National Security Agency - NSA mobile device best practicesNational Security Agency - NSA mobile device best practices
National Security Agency - NSA mobile device best practices
 
GraphRAG for Life Science to increase LLM accuracy
GraphRAG for Life Science to increase LLM accuracyGraphRAG for Life Science to increase LLM accuracy
GraphRAG for Life Science to increase LLM accuracy
 
“Building and Scaling AI Applications with the Nx AI Manager,” a Presentation...
“Building and Scaling AI Applications with the Nx AI Manager,” a Presentation...“Building and Scaling AI Applications with the Nx AI Manager,” a Presentation...
“Building and Scaling AI Applications with the Nx AI Manager,” a Presentation...
 
Project Management Semester Long Project - Acuity
Project Management Semester Long Project - AcuityProject Management Semester Long Project - Acuity
Project Management Semester Long Project - Acuity
 
How to Get CNIC Information System with Paksim Ga.pptx
How to Get CNIC Information System with Paksim Ga.pptxHow to Get CNIC Information System with Paksim Ga.pptx
How to Get CNIC Information System with Paksim Ga.pptx
 
Nordic Marketo Engage User Group_June 13_ 2024.pptx
Nordic Marketo Engage User Group_June 13_ 2024.pptxNordic Marketo Engage User Group_June 13_ 2024.pptx
Nordic Marketo Engage User Group_June 13_ 2024.pptx
 
UI5 Controls simplified - UI5con2024 presentation
UI5 Controls simplified - UI5con2024 presentationUI5 Controls simplified - UI5con2024 presentation
UI5 Controls simplified - UI5con2024 presentation
 

Decentralized storage

  • 2. Agenda What is Decentralisation + Storage? What is Decentralised Storage? Why Decentralised Storage? Decentralised Storage - Current Projects Swarm IPFS - Deep Dive Basic Commands Advanced Commands Demo Appendix 2
  • 3. What is Decentralisation + Storage? Decentralisation is understood as the transfer of authority from a central entity to a more localised and ‘liberal’ system. Storage is defined as the retention of retrievable data on a computer or other electronic system. 3
  • 4. What is Decentralised Storage? Decentralised storage is a system of being able to store your files without having to rely on large, centralized silos of data that don’t undermine important values such as privacy and freedom of your information. It is Content-Addressable, rather than Location-Addressable. Every file has a unique hash of its content. 4
  • 5. Why Decentralised Storage? ● Availability ○ Censorship Resistant ○ Data geographically spread ○ No "404 Page Not found" error ● Security & Privacy ○ No centralised server storage hence very difficult to hack and breach data ○ Files are not stored directly but as chunks of data spread across multiple nodes ● Cost reduction due to more efficiency 5
  • 6. Decentralised Storage - Current Projects ● Swarm ● IPFS (Inter Planetary File System) ● Sia ● Storj 6
  • 7. Swarm 7 Swarm is a distributed storage platform and content distribution service, a native base layer service of the ethereum web3 stack. The primary objective of Swarm is to provide a sufficiently decentralized and redundant store of Ethereum’s public record, in particular to store and distribute dapp code and data as well as blockchain data. Refer Swarm Documentation for further details
  • 8. IPFS - Deep Dive 8
  • 9. What is IPFS? 9 IPFS is a distributed peer-to-peer (p2p) file sharing system for storing and accessing files, websites, applications, and data. IPFS aims to replace HTTP and build a better web for all of us.
  • 10. HTTP v/s IPFS ​Today, the Internet is based on HyperText Transfer Protocol (HTTP). HTTP relies on location addressing which uses IP addresses to identify the specific server that is hosting the requested information. This means that the information has to be fetched from the origin server. IPFS is meant to be a replacement for HTTP. Most notably, IPFS never has a single point of failure. It’s a peer-to-peer distributed file system that would decentralize the Internet and make it much more difficult for a service provider or hosting network to pull the plug and make published information suddenly disappear. 10 HTTP vs. IPFS [Source: https://www.maxcdn.com/one/visual- glossary/interplanetary-file-system/]
  • 11. How IPFS works? 11 IPFS works by connecting all computing devices with the same system of files via a system of nodes. It uses a “distributed hash table, an incentivized block exchange, and a self-certifying namespace.” In simpler terms, it acts similarly to a torrent system, except that instead of sharing and exchanging media, IPFS exchanges git objects. This means that the whole system is based around a simple key-value data store. Any type of content can be inserted, and it will give back a key that can be used to retrieve the content again at any time. This is what allows for content addressing instead of location addressing: The key is completely independent of the origin of the information and can be hosted anywhere.
  • 12. How IPFS stores data? When you add any content on IPFS network, the data is split into chunks of 256Kb. Each chunk is identified with it’s own hash. These chunks are then distributed to various nodes on network which have there hash closest to peerId. 12
  • 13. How IPFS stores data? ...Continued 1. Let us assume that there are 4 nodes with peerId 6789, 789a, 89ab, 9abc respectively 2. We try to add a file name(size= 1Mb) something.mp4. Your node first calculates that hash of the file, say 7abc. Additionally the file is broken into 4 chunks of 256 Kb each. Your node then calculates the hash of the each chunk, say (7aaa, 8abc, 9a23, 5bcd) 3. Now node broadcasts the each chunk to node with has the closest peerId numerically. In our mentioned example chunk with hash 7aaa it closest to hash 789a. Hence this chunk is send to node with peerId 789a. 4. Similarly, all chunks are send and there address in updated in DHT. 5. Lastly, the object root hash i.e 7abc is stored, (Root hash can be stored anywhere, it is assumed that in current example it is stored in our system) and hashes that it links to i.e 7abc → [7aaa, 8abc, 9a23, 5bcd] [Source: https://medium.com/@akshay_111meher/how-ipfs-works- 545e1c890437] 13 How data is divided and stored. Root hash is assumed to be stored on your node. It is however stored in same way chunks are stored. It could be on anyone, including yours.
  • 14. IPFS File Storage - Live Demo 14
  • 15. How data is retrieved? On IPFS network, the file is identified solely by it HASH (root hash), in our case 7abc. Once the user requests a file, the request traverses to nodes where hash is existing using the DHT. If the data points to other chunks (like in our case), even they are searched same way. Once all chunks are obtained, all of them are simply concatenated to obtain the main object. 15
  • 16. Basic Commands 16 ● Install IPFS ● Get IPFS version ● Initialize the IPFS repository ● Get IPFS node id ● Start IPFS node ● Check Peer Nodes
  • 17. Get IPFS version 17 ● Install IPFS ● Get IPFS version ● Initialize the IPFS repository ● Get IPFS node id ● Start IPFS node ● Check Peer nodes ipfs version ipfs version 0.4.18
  • 18. Initialize the IPFS repository 18 ● Install IPFS ● Get IPFS version ● Initialize the IPFS repository ● Get IPFS node id ● Start IPFS node ● Check Peer nodes ipfs init initializing ipfs node at /Users/jbenet/.go-ipfs generating 2048-bit RSA keypair...done peer identity: Qmcpo2iLBikrdf1d6QU6vXuNb6P7hwrbNPW9k LAH8eG67z to get started, enter: ipfs cat /ipfs/QmYwAPJzv5CZsnA625s3Xf2nemtYgPp HdWEz79ojWnPbdG/readme
  • 19. Get IPFS node id 19 ● Install IPFS ● Get IPFS version ● Initialize the IPFS repository ● Get IPFS node id ● Start IPFS node ● Check Peer nodes ipfs id "ID": "QmP7JssmhNTpayGoK5ZhBt78hRRBi3VBYQ yqwMsBsSZBSW"
  • 20. Start IPFS node 20 ● Install IPFS ● Get IPFS version ● Initialize the IPFS repository ● Get IPFS node id ● Start IPFS node ● Check Peer nodes ipfs daemon Initializing daemon... go-ipfs version: 0.4.18- Repo version: 7 System version: amd64/darwin Golang version: go1.11.1 Successfully raised file descriptor limit to 2048. Swarm listening on /ip4/127.0.0.1/tcp/4001 . Swarm announcing /ip4/127.0.0.1/tcp/4001 . Daemon is ready
  • 21. Check peer nodes 21 ● Install IPFS ● Get IPFS version ● Initialize the IPFS repository ● Get IPFS node id ● Start IPFS node ● Check Peer nodes ipfs swarm peers
  • 22. Advanced Commands 22 ● Check IPFS repository statistics ● Add a file to IPFS ● Pin objects to local storage ● Remove pinned objects from local storage ● Download IPFS objects ● Show IPFS object data ● List objects pinned to local storage
  • 23. Check IPFS repository statistics 23 ● Check IPFS repository statistics ● Add a file to IPFS ● Pin objects to local storage ● Remove pinned objects from local storage ● Download IPFS objects ● Show IPFS object data ● List objects pinned to local storage ipfs stats repo NumObjects: 4817 RepoSize: 127963949 StorageMax: 10000000000 RepoPath: /Users/anuragd/.ipfs Version: fs-repo@7
  • 24. Add a file to IPFS 24 ● Check IPFS repository statistics ● Add a file to IPFS ● Pin objects to local storage ● Remove pinned objects from local storage ● Download IPFS objects ● Show IPFS object data ● List objects pinned to local storage ipfs add temp added QmWATWQ7fVPP2EFGu71UkfnqhYXDYH566qy 47CnJDgvs8u temp 12 B / 12 B [================================== =================================== =================================== ===============================] 100.00%
  • 25. Pin objects to local storage 25 ● Check IPFS repository statistics ● Add a file to IPFS ● Pin objects to local storage ● Remove pinned objects from local storage ● Download IPFS objects ● Show IPFS object data ● List objects pinned to local storage ipfs pin add QmWATWQ7fVPP2EFGu71UkfnqhYXDYH566qy 47CnJDgvs8u pinned QmWATWQ7fVPP2EFGu71UkfnqhYXDYH566qy 47CnJDgvs8u recursively
  • 26. Remove pinned objects from local storage 26 ● Check IPFS repository statistics ● Add a file to IPFS ● Pin objects to local storage ● Remove pinned objects from local storage ● Download IPFS objects ● Show IPFS object data ● List objects pinned to local storage ipfs pin rm QmWATWQ7fVPP2EFGu71UkfnqhYXDYH566qy 47CnJDgvs8u unpinned QmWATWQ7fVPP2EFGu71UkfnqhYXDYH566qy 47CnJDgvs8u
  • 27. Download IPFS objects 27 ● Check IPFS repository statistics ● Add a file to IPFS ● Pin objects to local storage ● Remove pinned objects from local storage ● Download IPFS objects ● Show IPFS object data ● List objects pinned to local storage ipfs get QmWATWQ7fVPP2EFGu71UkfnqhYXDYH566qy 47CnJDgvs8u Saving file(s) to QmWATWQ7fVPP2EFGu71UkfnqhYXDYH566qy 47CnJDgvs8u 20 B / 20 B [================================== =================================== =================================== ============================] 100.00% 0s
  • 28. Show IPFS object data 28 ● Check IPFS repository statistics ● Add a file to IPFS ● Pin objects to local storage ● Remove pinned objects from local storage ● Download IPFS objects ● Show IPFS object data ● List objects pinned to local storage ipfs cat QmWATWQ7fVPP2EFGu71UkfnqhYXDYH566qy 47CnJDgvs8u Hello World ipfs cat QmWATWQ7fVPP2EFGu71UkfnqhYXDYH566qy 47CnJDgvs8u > temp1 cat temp1 Hello World
  • 29. List objects pinned to local storage 29 ● Check IPFS repository statistics ● Add a file to IPFS ● Pin objects to local storage ● Remove pinned objects from local storage ● Download IPFS objects ● Show IPFS object data ● List objects pinned to local storage ipfs pin ls

Editor's Notes

  1. https://swarm-guide.readthedocs.io/en/latest/ https://swarm-gateways.net/bzz:/example.photoalbum.eth/#1
  2. https://hackernoon.com/a-beginners-guide-to-ipfs-20673fedd3f https://medium.com/coinmonks/a-hands-on-introduction-to-ipfs-ee65b594937
  3. https://www.maxcdn.com/one/visual-glossary/interplanetary-file-system/
  4. https://medium.com/@akshay_111meher/how-ipfs-works-545e1c890437
  5. https://medium.com/@akshay_111meher/how-ipfs-works-545e1c890437
  6. Show http://localhost:5001/webui http://localhost:8080/ipfs/<<hash of content>>
  7. Ipfs add, automatically pins the files
  8. Ipfs add, automatically pins the files
  9. Check Ipfs add, automatically pins the files Check without ipfs daemon - you can check files Check with ipfs daemon - you can see files, which files are uploaded when you are online ipfs.io/ipfs/QmZcR3AvZHxjh8hDxLxmECDzjiSDXLgJnfVogsKw2kJiUY or use webui
  10. Check Ipfs add, automatically pins the files Check without ipfs daemon - you can check files Check with ipfs daemon - you can see files, which files are uploaded when you are online ipfs.io/ipfs/QmZcR3AvZHxjh8hDxLxmECDzjiSDXLgJnfVogsKw2kJiUY or use webui
  11. Check Ipfs add, automatically pins the files Check without ipfs daemon - you can check files Check with ipfs daemon - you can see files, which files are uploaded when you are online ipfs.io/ipfs/QmZcR3AvZHxjh8hDxLxmECDzjiSDXLgJnfVogsKw2kJiUY or use webui
  12. Check Ipfs add, automatically pins the files Check without ipfs daemon - you can check files Check with ipfs daemon - you can see files, which files are uploaded when you are online ipfs.io/ipfs/QmZcR3AvZHxjh8hDxLxmECDzjiSDXLgJnfVogsKw2kJiUY or use webui
  13. Check Ipfs add, automatically pins the files Check without ipfs daemon - you can check files Check with ipfs daemon - you can see files, which files are uploaded when you are online ipfs.io/ipfs/QmZcR3AvZHxjh8hDxLxmECDzjiSDXLgJnfVogsKw2kJiUY or use webui
  14. Check Ipfs add, automatically pins the files Check without ipfs daemon - you can check files Check with ipfs daemon - you can see files, which files are uploaded when you are online ipfs.io/ipfs/QmZcR3AvZHxjh8hDxLxmECDzjiSDXLgJnfVogsKw2kJiUY or use webui
  15. https://itnext.io/build-a-simple-ethereum-interplanetary-file-system-ipfs-react-js-dapp-23ff4914ce4e Execute this commands Ipfs add Ipfs get Ipfs cat