RDM#2
LISBON
Research & Development Meetup
LISBON
David Dias,
P2P SE&R @ Protocol Labs
João Ascensão,
Data Scientist @ Uniplaces
tw: @daviddias gh: @diasdavid
tw: @j...
LISBON
What is RDM?
LISBON
Space to invest time in learning
things that are hard to grasp
LISBON
world of possibilities
LISBON
what actually
gets researched
LISBON
what gets
developed
LISBON
what gets
deployed
LISBON
what most people are doing
PROTOCOL LABS
LISBON
FORMAT
- New topics / Research
- Hacking + Digesting + Publishing
LISBON
TODAY
- The Distributed Web
- Machine Learning / AI
The Distributed Web
Serve data to everywhere
• Motivations
• IPFS, the Protocol
• MerkleDAG (Directed Acyclic Graph)
• DHT (Distributed Hash Table)
• Motivations
• IPFS, the Protocol,
MerkleDAG, IPNS,
Network
• IPFS in action
• IPFS, the project
• filecoin
• libp2p
• sta...
Location Addressing
http://10.20.30.40/foo/bar/baz.png
location path
http://example.com/foo/bar/baz.png
http://10.20.30.40/foo/bar/baz.png
you
http://10.20.30.40/foo/bar/baz.png
10.20.30.40
you
http://10.20.30.40/foo/bar/baz.png
10.20.30.40
you
200 MB
200 MB x 30 x 8 = 48 GB
2,544,837,694
2,387,990,609
508.968 PB
Bandwidth
https://www.backblaze.com/blog/price-gap-storage-vs-bandwidth/
Latency
Amazon Google
DataCenters
Disconnected / Offline
low bw interference congestion travelisp outagedatacenter
Permanence Security
IoT
Control
Offline
Bandwidth
Disconnected
InterPlanetary FileSystem
a protocol to upgrade the web
InterPlanetary FileSystem
InterPlanetary FileSystem
DISTRIBUTED
PERMANENT
MERKLE
THE WEB
PermanentDistributed Safer
SmarterOffline
Faster
SFS
web
DHT+
web
DHT
SFS
routing
network
exchange
merkledag
naming
applications
The Stack
routing
network
exchange
merkledag
naming
applicationsThe Stack
Defining the Data
Moving the Data
Using the Data
IPLD
libp2...
8AyaTQMhJklMo
Ga3WaDgwcAEPc
KQeipgDu64qaj
kr3fNcVt1LVMD
KQsXSobGtbNhF
sv3qA44E7iXAy
gcuSfVhW8RV3z
R9F3MFvbVaqf9
8AyaTQMhJklMo
Ga3WaDgwcAEPc
KQeipgDu64qaj
kr3fNcVt1LVMD
hash( ) -> KQsXSobGtbNhF
KQsXSobGtbNhF
sv3qA44E7iXAy
gcuSfVhW8RV3z...
8AyaTQMhJklMo
Ga3WaDgwcAEPc
KQeipgDu64qaj
kr3fNcVt1LVMD
hash( ) -> KQsXSobGtbNhF
KQsXSobGtbNhF
sv3qA44E7iXAy
gcuSfVhW8RV3z...
cvs/svn
cvs/svn
cvs/svn
merkledag
in IPFS
data forms a dag
it’s called the merkle dag
because the links are hashes
any data structures are
represented as dags
unix files
and dirs
any data structures are
represented as dags
blob, tree, commit
unix files
and dirs
any data structures are
represented as dags
blob, tree, commit
unix files
and dirs
any data structures are
represented as dags
blocks, txns, wallets
blob, tree, commit
unix files
and dirs
any data structures are
represented as dags
blocks, txns, wallets
kv-stores
blob, tree, commit
unix files
and dirs
any data structures are
represented as dags
blocks, txns, wallets
kv-stores
unixfs as a dag
files are dag nodes
unixfs as a dag
files are dag nodes
BIG files may be
split into many
unixfs as a dag
directories are
also dag nodes
files are dag nodes
BIG files may be
split into many
unixfs as a dag
which link
to others
directories are
also dag nodes
files are dag nodes
BIG files may be
split into many
unixfs as a dag
http://10.20.30.40/foo/bar/baz.png
location path
http://example.com/foo/bar/baz.png
http://10.20.30.40/foo/bar/baz.png
location path
http://example.com/foo/bar/baz.png
/ipns/example.com/foo/bar/baz.png
/ipf...
10.20.30.40
you
HTTP
http://10.20.30.40/foo/bar/baz.png
/ipfs/QmW98pJrc6FZ6/foo/bar/baz.png
http://10.20.30.40/foo/bar/baz.png
10.20.30.40
you
/ipfs/QmW98pJrc6FZ6/foo/bar/baz.png
IPFS
/ipfs/QmW98pJrc6FZ6/foo/bar/baz.png
content path
/ipfs/QmW98pJrc6FZ6/foo/bar/baz.png
content path
/ipfs/QmW98pJrc6FZ6/foo/bar/baz.png
content path
foo
bar
baz.png
QmW98pJrc6FZ6
/ipfs/QmW98pJrc6FZ6/foo/bar/baz.png
content path
foo:HB
HB
HC
HD
bar:HC
baz.png:HDQmW98pJrc6FZ6
foo
bar
baz.png
DHT
DHT are the Holy Grail* of P2P Networks
*from a survey paper
1
5
6
10
50
55
1 10 12 20 30 50 54 58 70 88 81 83 88 90 92 95 97 99 101105 300411 500
1 10 12 20 30 50 54 58 70 88 81 83 88 90 92 95 97 99 101105 300411 500
1 10 12 20 30 50 54 58 70 88 81 83 88 90 92 95 97 99 101105 300411 500
hash( )
1 10 12 20 30 50 54 58 70 88 81 83 88 90 92 95 97 99 101105 300411 500
hash( ) = 91
1 10 12 20 30 50 54 58 70 88 81 83 88 90 92 95 97 99 101105 300411 500
hash( ) = 91
1 10 12 20 30 50 54 58 70 88 81 83 88 90 92 95 97 99 101105 300411 500
Who has fim , which hash( ) = 91?
The Distributed Web
• Applications with no-origin
• Leverage network cache
• Resilient to network splits
Serve data to everywhere
Serve data from everywhere
David Dias - http://daviddias.me IPFS http://ipfs.io Protocol Labs http://ipn.io
Thank you!
David Dias - http://daviddias.me IPFS http://ipfs.io Protocol Labs http://ipn.io
RDM#2- The Distributed Web
RDM#2- The Distributed Web
RDM#2- The Distributed Web
RDM#2- The Distributed Web
RDM#2- The Distributed Web
RDM#2- The Distributed Web
RDM#2- The Distributed Web
RDM#2- The Distributed Web
RDM#2- The Distributed Web
RDM#2- The Distributed Web
RDM#2- The Distributed Web
RDM#2- The Distributed Web
RDM#2- The Distributed Web
RDM#2- The Distributed Web
RDM#2- The Distributed Web
RDM#2- The Distributed Web
RDM#2- The Distributed Web
RDM#2- The Distributed Web
RDM#2- The Distributed Web
RDM#2- The Distributed Web
RDM#2- The Distributed Web
RDM#2- The Distributed Web
RDM#2- The Distributed Web
RDM#2- The Distributed Web
RDM#2- The Distributed Web
RDM#2- The Distributed Web
RDM#2- The Distributed Web
RDM#2- The Distributed Web
RDM#2- The Distributed Web
RDM#2- The Distributed Web
RDM#2- The Distributed Web
RDM#2- The Distributed Web
RDM#2- The Distributed Web
RDM#2- The Distributed Web
RDM#2- The Distributed Web
Upcoming SlideShare
Loading in …5
×

RDM#2- The Distributed Web

341 views

Published on

"The Distributed Web" A talk about building distributed web apps for the second Research & Development Meetup

Published in: Engineering
0 Comments
3 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
341
On SlideShare
0
From Embeds
0
Number of Embeds
9
Actions
Shares
0
Downloads
10
Comments
0
Likes
3
Embeds 0
No embeds

No notes for slide

RDM#2- The Distributed Web

  1. 1. RDM#2 LISBON Research & Development Meetup
  2. 2. LISBON David Dias, P2P SE&R @ Protocol Labs João Ascensão, Data Scientist @ Uniplaces tw: @daviddias gh: @diasdavid tw: @jtascensao gh: @jtascensao
  3. 3. LISBON What is RDM?
  4. 4. LISBON Space to invest time in learning things that are hard to grasp
  5. 5. LISBON world of possibilities
  6. 6. LISBON what actually gets researched
  7. 7. LISBON what gets developed
  8. 8. LISBON what gets deployed
  9. 9. LISBON what most people are doing
  10. 10. PROTOCOL LABS
  11. 11. LISBON FORMAT - New topics / Research - Hacking + Digesting + Publishing
  12. 12. LISBON TODAY - The Distributed Web - Machine Learning / AI
  13. 13. The Distributed Web
  14. 14. Serve data to everywhere
  15. 15. • Motivations • IPFS, the Protocol • MerkleDAG (Directed Acyclic Graph) • DHT (Distributed Hash Table)
  16. 16. • Motivations • IPFS, the Protocol, MerkleDAG, IPNS, Network • IPFS in action • IPFS, the project • filecoin • libp2p • starship • gx • full IPFS JavaScript implementation • mutable file system and the files API • Other DHT • Structure P2P Networks • Unstructured P2P Networks • NAT Traversal • Relay/Turn • registry-mirror • smart chunking • Transports • Multiplexers • Self Describing Data Types • Search • Station • and more
  17. 17. Location Addressing
  18. 18. http://10.20.30.40/foo/bar/baz.png location path http://example.com/foo/bar/baz.png
  19. 19. http://10.20.30.40/foo/bar/baz.png you
  20. 20. http://10.20.30.40/foo/bar/baz.png 10.20.30.40 you
  21. 21. http://10.20.30.40/foo/bar/baz.png 10.20.30.40 you
  22. 22. 200 MB
  23. 23. 200 MB x 30 x 8 = 48 GB
  24. 24. 2,544,837,694
  25. 25. 2,387,990,609 508.968 PB
  26. 26. Bandwidth
  27. 27. https://www.backblaze.com/blog/price-gap-storage-vs-bandwidth/
  28. 28. Latency
  29. 29. Amazon Google DataCenters
  30. 30. Disconnected / Offline
  31. 31. low bw interference congestion travelisp outagedatacenter
  32. 32. Permanence Security IoT Control Offline Bandwidth Disconnected
  33. 33. InterPlanetary FileSystem
  34. 34. a protocol to upgrade the web
  35. 35. InterPlanetary FileSystem
  36. 36. InterPlanetary FileSystem
  37. 37. DISTRIBUTED PERMANENT MERKLE THE WEB
  38. 38. PermanentDistributed Safer SmarterOffline Faster
  39. 39. SFS web DHT+
  40. 40. web DHT SFS routing network exchange merkledag naming applications The Stack
  41. 41. routing network exchange merkledag naming applicationsThe Stack Defining the Data Moving the Data Using the Data IPLD libp2p IPNS
  42. 42. 8AyaTQMhJklMo Ga3WaDgwcAEPc KQeipgDu64qaj kr3fNcVt1LVMD KQsXSobGtbNhF sv3qA44E7iXAy gcuSfVhW8RV3z R9F3MFvbVaqf9
  43. 43. 8AyaTQMhJklMo Ga3WaDgwcAEPc KQeipgDu64qaj kr3fNcVt1LVMD hash( ) -> KQsXSobGtbNhF KQsXSobGtbNhF sv3qA44E7iXAy gcuSfVhW8RV3z R9F3MFvbVaqf9
  44. 44. 8AyaTQMhJklMo Ga3WaDgwcAEPc KQeipgDu64qaj kr3fNcVt1LVMD hash( ) -> KQsXSobGtbNhF KQsXSobGtbNhF sv3qA44E7iXAy gcuSfVhW8RV3z R9F3MFvbVaqf9 MERKLE LINK
  45. 45. cvs/svn
  46. 46. cvs/svn
  47. 47. cvs/svn
  48. 48. merkledag
  49. 49. in IPFS data forms a dag it’s called the merkle dag because the links are hashes
  50. 50. any data structures are represented as dags
  51. 51. unix files and dirs any data structures are represented as dags
  52. 52. blob, tree, commit unix files and dirs any data structures are represented as dags
  53. 53. blob, tree, commit unix files and dirs any data structures are represented as dags blocks, txns, wallets
  54. 54. blob, tree, commit unix files and dirs any data structures are represented as dags blocks, txns, wallets kv-stores
  55. 55. blob, tree, commit unix files and dirs any data structures are represented as dags blocks, txns, wallets kv-stores
  56. 56. unixfs as a dag
  57. 57. files are dag nodes unixfs as a dag
  58. 58. files are dag nodes BIG files may be split into many unixfs as a dag
  59. 59. directories are also dag nodes files are dag nodes BIG files may be split into many unixfs as a dag
  60. 60. which link to others directories are also dag nodes files are dag nodes BIG files may be split into many unixfs as a dag
  61. 61. http://10.20.30.40/foo/bar/baz.png location path http://example.com/foo/bar/baz.png
  62. 62. http://10.20.30.40/foo/bar/baz.png location path http://example.com/foo/bar/baz.png /ipns/example.com/foo/bar/baz.png /ipfs/QmW98pJrc6FZ6/foo/bar/baz.png content path
  63. 63. 10.20.30.40 you HTTP http://10.20.30.40/foo/bar/baz.png /ipfs/QmW98pJrc6FZ6/foo/bar/baz.png
  64. 64. http://10.20.30.40/foo/bar/baz.png 10.20.30.40 you /ipfs/QmW98pJrc6FZ6/foo/bar/baz.png IPFS
  65. 65. /ipfs/QmW98pJrc6FZ6/foo/bar/baz.png content path
  66. 66. /ipfs/QmW98pJrc6FZ6/foo/bar/baz.png content path
  67. 67. /ipfs/QmW98pJrc6FZ6/foo/bar/baz.png content path foo bar baz.png QmW98pJrc6FZ6
  68. 68. /ipfs/QmW98pJrc6FZ6/foo/bar/baz.png content path foo:HB HB HC HD bar:HC baz.png:HDQmW98pJrc6FZ6 foo bar baz.png
  69. 69. DHT
  70. 70. DHT are the Holy Grail* of P2P Networks *from a survey paper
  71. 71. 1 5 6 10 50 55
  72. 72. 1 10 12 20 30 50 54 58 70 88 81 83 88 90 92 95 97 99 101105 300411 500
  73. 73. 1 10 12 20 30 50 54 58 70 88 81 83 88 90 92 95 97 99 101105 300411 500
  74. 74. 1 10 12 20 30 50 54 58 70 88 81 83 88 90 92 95 97 99 101105 300411 500 hash( )
  75. 75. 1 10 12 20 30 50 54 58 70 88 81 83 88 90 92 95 97 99 101105 300411 500 hash( ) = 91
  76. 76. 1 10 12 20 30 50 54 58 70 88 81 83 88 90 92 95 97 99 101105 300411 500 hash( ) = 91
  77. 77. 1 10 12 20 30 50 54 58 70 88 81 83 88 90 92 95 97 99 101105 300411 500 Who has fim , which hash( ) = 91?
  78. 78. The Distributed Web
  79. 79. • Applications with no-origin • Leverage network cache • Resilient to network splits
  80. 80. Serve data to everywhere
  81. 81. Serve data from everywhere
  82. 82. David Dias - http://daviddias.me IPFS http://ipfs.io Protocol Labs http://ipn.io
  83. 83. Thank you! David Dias - http://daviddias.me IPFS http://ipfs.io Protocol Labs http://ipn.io

×