Dat is a non-profit that provides tools for versioning and sharing data. It allows researchers to import, version, publish, and replicate data through a simple workflow and command line interface. Dat uses decentralized peer-to-peer technology so data can be cloned, pulled, and pushed between researchers' computers. This facilitates improved collaboration and reproducibility in research.

1. Dat: version and share
your data
Karissa McKelvey
Software Developer and Project Manager
and Science Evangelist and Designer (I wear a lot of hats)
U.S. Open Data
@karissamck

2. karissa $ ~

3. dat is a non profit

5. Reproducible Research

6. “A rule of thumb … is that half of published
research cannot be replicated”

7. How do we replicate research today?

8. How do we replicate research today?
collaborate on

9. How do we replicate research today?
collaborate on
data analysis

10. How do we collaborate today?

11. How do we collaborate today?

12. How do we collaborate today?

13. How do we collaborate today?

14. ????????

22. How do we replicate research today?

24. me@home $ dat push
me@campus $ dat pull
you@work $ dat clone

25. dat workflow
• import
• version
• publish
• replicate

26. .csv.csvdata
you

27. .csv.csvdata
you

28. .csv.csvdata
you

29. .csv.csvdata
import
you

30. $ dat init
$ dat add dataset cities
$ dat add rows cities cities.csv
$ dat add files cities city_model.gz
import

31. .csv.csvdata
import
http://my-data.bids.edu
you

32. $ dat listen

33. .csv.csvdata
import
http://my-data.bids.edu
publish
you

34. .csv.csvdata
import
http://my-data.bids.edu
publish
you

35. $ dat clone

36. .csv.csvdata
import
http://my-data.bids.edu
publish
.csv.csvdata
you

37. Versioning
$ dat add files cities us_cities_viz.png
This will override us_cities_viz.png at
c2342. OK?
$ dat cities add rows updated_data.csv
This will update 3,434,245 rows. OK?
$ dat push

38. .csv.csvdata
import
http://my-data.bids.edu
publish
.csv.csvdata
you

39. http://my-data.bids.edu
publish
.csv.csvdata
http://my-data.indiana.edu

40. .csv
.png.png.png
.csv.csv
.csv.csv.R
.csv.csv.py
INTEROPERABILITY in Python and R

41. .csv
.png.png.png
.csv.csv
.csv.csv.R
.csv.csv.py
ECOSYSTEM

42. • Goal: manipulate datasets with scripting
• Supported keywords: run, pipe, map, reduce,
fork, keyword
• Bash-like
• Platform-independent
• Uses node.js streams (fast!)
Datscript

43. Top: Datscript “pipe” command
Bottom: Equivalent command in bash
Datscript: pipeline example

44. Datscript: example commands
background
- executes command, but doesn’t wait for it to finish
map
- pipes first argument into rest of arguments
run
- a serial command (executes and finishes command)

47. Karissa McKelvey - @karissamck
Melanie Cebula - @melaniecebula
http://dat-data.com

48. .csv
.png.png.png
.csv.csv
.csv.csv.R
.csv.csv.py
INTEROPERABILITY in Python and R

49. .csv
.png.png.png
.csv.csv
.csv.csv.R
.csv.csv.py
ECOSYSTEM