Building data infrastructure that accommodates networks of trust. Presentation given by flyingzumwalt at Code4Lib 2016 in Philadelphia.

1. Why Should You
Trust My Data?
building data infrastructure
that accommodates networks of trust
Matt Zumwalt
datjawn.com | databindery.com
@flyingzumwalt
code{4}lib 2016

2. I’m interested in trust.

3. I’m interested in trust.
particularly trust & trustworthiness
when people exchange data

4. there’s a rhythm
to the computing world
centralization decentralization
client-server peer-to-peer

5. mainframes
personal computers
server farms
[“internet of everything”]
“the cloud”
“the PC revolution”
“computers”
“the diamond age”

6. remember
mainframes?

7. image credit wikipedia

11. the www

12. host data
reference each other

15. but data

25. image credit Torkild Retvedt

26. $
$
$
$
$
$
$

27. By 2019 the data created by IoE
devices alone will be 49 times higher
than all the traffic that moved through
datacenters in 2014.
it won’t scale.
Reference:
Cisco Global Cloud Index

28. Worldwide Storage Capacity in 2012:
2.5 zettabytes
Total Data Center Traffic in 2016:
10.4 zettabytes per year
Anticipated data created by Internet of
Everything (IoE) devices in 2019:
507.5 zettabytes per year
References:
NetApp
Cisco Global Cloud Index
gigaom
Washington Post

29. distributed data web
“You can’t propose that something
be a universal space and at the
same time keep control of it.”
- Tim Berners Lee

39. this relies on trust

40. elements of
trustworthiness
authority & reputation
integrity & provenance
synergy or compatibility
consistency
etc…

41. we’ve got this
Organisms have been solving
these problems for eons
Humans for millennia
Librarians for centuries
Software developers for decades

42. ‘git for (tabular) data’
transparency & reproducibility
http://datjawn.com
builds from the work of
http://dat-data.com
Tabular: rows & columns (ie. Spreadsheets, CSV, SQL DBs)

43. history has
branches

44. initial commit
a set of changes
commit those changes
and describe them
Who made the changes?
Why did they make them?
When did they commit them?

45. more changes
commit those changes

46. different changes
committed to a
different branch

47. other changes
on another branch

48. merge
two branches

49. get a specific version
prove it’s identical
know who made it

50. Files are data.
They have histories.
Metadata are data.
They have histories too.
Whatever the data,
The same patterns apply.

51. How does this get
replicated?

52. client-server approach

53. peer to peer approach

58. the tide has
already shifted

59. Stop building server-side applications.
Assume that data are anywhere and/or everywhere.
Assume that your software will be run in many places.
Erase your distinctions between server and client.
Let data grow branches - build trees (ie. Merkle DAGs)
Stop thinking of data as singular.
Stop thinking of datasets as monolithic.
Embrace redundancy & replication.
Understand that trustworthiness and authority are dynamic.
Broaden your sense of “now”.
Appreciate provenance.
there are no servers
there is only the web

60. Meet the
dat jawn team on
Wednesday…
Matt Zumwalt
datjawn.com | databindery.com
@flyingzumwalt
code{4}lib 2016