Camlistore is a personal datastore project initiated by Brad Fitzpatrick in Go, designed to consolidate and securely store personal content. It features content-addressable storage, a multi-layer architecture for managing blobs and indexing, and extensibility for various clients and platforms. The system aims to provide easy deployment on cloud platforms and includes a web UI along with support for versioning and data sharing.

1. Camlistore.org
Your, Personal Datastore for Life
A side project in GoLang by Brad Fitzpatrick. Maintainer Mathieu Lonjaret.
Reprised by Clive Boulton for Google Developer Groups NYC. Jan 9, 2017.
https://www.meetup.com/gdgnyc/events/234963670/

2. Entrepreneurs: personal memories are priceless.

4. Demo

5. How Brad got Camlistore going
Brad Fitzpatrick @bradfitz is a member of the Go team at Google, working
primarily on the standard library.
● Brad wanted to create a system to store all of the content he’s created,
which is currently scattered across multiple sites and disks. He started
hacking in this directory:
$ Content Addressable Multi Level Indexed Storage = Camlistore

6. Camlistore Goals
● Content-addressable
● Searchable
● Web UI
● Extensibility: FUSE, etc.
Majority of Camlistore is pure Go. (Ed. note: It’s a great repo to read and learn
from.)

7. Content-addressable advantages
● Sync any which way, no conflicts
● (merge resolution at higher layers)
● corruption verification/repair
● de-duping
Content Addressable: What things are named depends on their content. Two
identical things have the same name. For example, the "name" or "key" for the
data is the SHA-1 for the data, ala git.

8. Multi-level
● blob storage
● indexing
● search
● web UI, FUSE, CU, apps are just clients of search + blob storage
Multi-Layer: The whole storage stack is built out of several layers. The blob
store sits on the bottom, and only knows about bytes, and access is via the
SHA-1 of those bytes. Things that you might store (Files, directories, sets,
collections of tweets, social graphs, etc) build on top of the blob store by
additional blobs that hold pointers to data blobs. Again, it's sort of like git. A
front-end might sit on top of that abstraction.

9. Indexed
● ask the indexer
● tags
● search = find
Indexed: blobs of JSON that have a few special attributes are recognized and
indexed. So, you might have a bunch of blobs with these special attributes (ie,
"tag") and be able to ask the indexer "Give me all blobs with tag equal to foo",
rather than having to search through the blobs directly.

10. Overview
Permanode
random #
Set-attribute
camliContent
Directory
JSON
File
Bytes
File
Bytes
File
Bytes
File
Bytes
Directory
JSON
File
JSON
File
JSON
unassigned blobs
object
Set-attribute
camliContent

11. Logically

12. Features of @camlistore
● Personal Datastore for life: Like Git for Social Backup.
● Security is private by default (share if you wish, more sharing in v.10)
● FUSE file system
● Data versioning, time travel
● Snappy UI (react.js like Instagram moving to gopherjs)
● Clients in JavaScript, Java, Obj-C,
● Deploy Web, Phone, Home, S3, GCE (Docker)
● Targets Desktop Chrome, Firefox, Safari, Edge (last two stable versions).
● Targets Safari and Chrome on Android and iOS tablets (last two stable ver)

13. Why Brad chose Go for Camlistore
● He considered a few other candidate languages:
● Perl and Python: too slow and single-threaded
● C++ and Java: tedious and not fun (slow builds)
● Go? Yeah.
As Brad hacked on Camlistore, he realized he needs to implement support for
various formats and protocols (like mime/multipart) and improve stdlib
packages (like os/exec nee exec, database/sql, etc.).
Camlistore ⇔ 20 per cent time (now a side project)

14. East deployment to GCP (now)
Makes firing up your own Camlistore server on GCE+MySQL+Cloud Storage a few clicks from a web app, using
CoreOS, Docker, and the various Google Cloud Platform APIs..
Visit https://camlistore.org/launch/ which lets you create your own camlistore instance on GCE. The Camlistore website
will say:
Step 1) go to https://console.developers.google.com/project
Step 2) click "Create Project". remember its name.
Step 3) Enable billing. (tell Google your credit card info) [Ed note. Dev’s sign up for GCP get $x00 credit for 60 days]
Step 4) Enter your project name: [ ]
[ Create ]
You click Create, and then we send you on an OAuth2 voyage to get access to create VMs & DBs & Buckets on your
behalf, and then we create it all, including creating a minimal CoreOS VM that boots right up into a tiny Docker
container just running camlistore. [Ed note, the API forms are also Camlistore open source]

15. Web UI
● React → gopherjs
● Infinite scroll
● Lightbox
● HTML history API
● Search
…
Camlistore Android app: demo time!
https://play.google.com/store/apps/details?id=org.camlistore

16. Command line tools $
● camget (blobs, trees, shares)
● camput (blobs, files, dirs, shares, permanodes, claims)
● camtool {init, sync, init, describe, claims…)
● cammount

17. Camlistore is pure Go: (no C)
● HTTP client/server, TLS, AES, …
● JSON, XML
● WebSockets
● SMTP server (coming)
● Rolling checksums
● GIF/JPG/PNG en/dec, resizing, thumbnails
● FUSE
● Notification bus
● database/sql, MySQL, Postgres, MongoDB
● OpenPGP signing & verification
Ed note: NYC is adoption center for Blockchain technologies, also written in GoLang

18. More
@camlistore
https://camlistore.org
Reprised by clive boulton twitter @iC