Prophet
a path out of the cloud
http://syncwith.us
jesse@bestpractical.com
You may know me from...
RT (Request Tracker)
Jifty
SVK
Hiveminder
Perl 6
T-shirts
I’ve been hacking on an
open source database
called “Prophet”
It has an API like
Amazon SimpleDB or
Google App Engine’s...
It’s designed for
“team-scale” apps
It’s built for P2P
replication and
disconnected use
But first, a brief
digression...
...about cloud
computing
☁
☔
Living in the cloud
=
sharecropping(佃农)
(That’s bad)
☹
The bad old days:
Pic of sharecroppers
You farmed land you
didn’t own...
...with tools you
couldn’t really afford
You paid for it with
part of your harvest...
It sounded like a
pretty sweet deal...
...until things got bad
(Things always got bad)
(Internet)
(Cloud)
(Internet)
In a bad year, you got
further in debt to
the land owner
So, what does this have
to do with software?
The (more recent)
bad old days:
pic of mainframes
You ran code you didn’t
own on hardware you
didn’t own
Things started to get
better in the 1980s
Pic of PCs
Users started to be
able to make choices
about computing...
They weren’t all rosy
Pic of BSOD
Sometimes new
versions of software
broke things...
...leaving you locked in
to old versions
pic of win 31?
Things got ‘better’
rms
che
Now, things are getting
worse again...
What happens when
your favorite service
goes down?
pic of twitter being
down
...or stops accepting
new signups?
...or starts making
arbitrary choices about
what’s ‘safe’ content?
...or breaks
You don’t own the
services you use
You probably don’t
even have a contract
When a service
provider cuts you off,
you lose
Not so secret shame:
I still need the cloud
My calendar lives at
google.com
I make a Web 2.0 todo
list service called
Hiveminder.com
pic of hiveminder
☣ Using hosted apps
is going to hurt you! ☣
What about Google
Gears,Adobe Air, etc?
Great. now you can use
your word processer
while you’re offline!
Pic of wordperfect
Real offline apps
should not need servers
Real offline apps
should sync like you do
Back to that
database thing...
JesseVincent
Chia-liang Kao
We work together
CL lives in Taipei
Jesse lives in Boston
Sometimes we need
to work face to face
TPE - BOS:
TPE - HNL:
BOS - HNL:
9410 mi
5,095 mi
5,069 mi
Step 1: Go to Hawaii for “work”
Step 2: ???
Step 3: Prophet!
Our Plan
The Plan Backfired
We were there for 8 days
We wrote 8000 lines of Perl
We figured out step 2
Step 2:
Build a Disconnected
Syncable Database
Fallacies of Distributed Computing
1.The network is reliable.
2. Latency is zero.
3. Bandwidth is infinite.
4.The network i...
A grounded, peer to peer
replicated, disconnected,
versioned, property
database with self-healing
conflict resolution
Proph...
What do all those
buzzwords mean?
grounded
Runs here
grounded
Not here
grounded
Runs at the edge
Doesn’t need to run in the cloud
Syncs with services you already use
(Adaptors talk to “Foreign ...
Update any replica
Pull from any replica
Push to any replica
Publish a replica
Changes will propagate
peer-to-peer replica...
Real-time replication is hard to scale
It only “works” with constant connectivity
I don’t have constant connectivity
Neith...
Every update is recorded as a change set
Change sets don’t lose any data
(so you can use them to go backwards)
All history...
Atomic operations
CREATE, READ, UPDATE, DELETE, SEARCH
Record types can have optional validation
and canonicalization
Reco...
Remembers all conflict resolutions
Syncs all resolutions with your peers
Detects identical conflicts
Uses your peers’ resolu...
Working with Prophet
RESTy API
GET /records.json
GET /records/Cars.json
GET /records/Cars/716499-5F9-4AC4-827.json
GET /records/Cars/716499-5F9...
RESTy API
Yes, we should be using PUT and DELETE
(or just provide an OpenRESTY adaptor)
Yes, you can have a commit bit and...
Native API
(Yes, the core is Perl.)
my $cli = Prophet::CLI->new();
my $cxn = $cli->app_handle->handle;
my $record = Prophe...
What could you build
with Prophet?
A bug tracker: “simple defects”
•id. Status, Summary
•(Arbitrary other properties too)
•History
•Comments
•Attachments
sd
Initialize
./bin/sd init
./bin/sd ticket create --
summary="Can't sync sd with Google Code"
status=new
Created ticket 5 (93BF979E-08C1-11DD-94C3-D4...
./bin/sd ticket search --regex publish
29 } new the online help doesn't describe publish
34 } new publish a static html vi...
./bin/sd ticket update
--uuid 93BF979E-08C1-11DD-94C3-D4B1FCEE7EC4
-- status=resolved
Updates
Bugs on my laptop
aren’t interesting.
Jesse
sd publish --to fsck.com:public_html/sd/
CL
sd clone --from http://my.com/~jesse/sd
Sync!
Jesse
sd server
CL
sd pull --local
Hackathon mode!
My project has a bug tracker
Actually, mine use two:
• RT
• hiveminder.com
My project has a bug tracker
Foreign Replicas
Prophet makes Foreign Replicas easy
SD gets them "for free"
(Using only the public REST API)
It took an afternoon
Mirror an RT instance into SD
Share it with your peers using prophet...
(Using only the public REST API)
...and one for Hiveminder
I can sync my bugs with
RT or Hiveminder
Actually, it’s better
I can sync between RT
and Hiveminder
I can sync between two
different RTs, too
• Trac
• Launchpad
• Google Code
• SourceForge
• Bugzilla
• Jira
• GForge
• debbugs
• GNATS
• todo.txt
• Lighthouse
• Redm...
What else can you use
Prophet for?
•CRM
•Bug tracking
•Sales orders
•Phone book
•Blog
•Trading Card
Database
•Ideas?
All the databases you
want while offline.
How about a P2P BBS?
Prophet doesn’t need a server.
You can sync over sneakernet.
“Private” Social Networks
A look inside Prophet
Anatomy of a
Prophet Replica
The bits and pieces
Database UUID
Replica UUID
Record Store
Changeset Store
Resolution Database
Configuration metadata
The Record Store
Stores individual records by type
Not guaranteed to have all old versions
The Changeset Store
Stores every change to a set of records
Guaranteed to have all old changesets
Replaying all changesets...
Replica Backends
Filesystem
Readable
Flat files
Compact
Fast
(Not yet fully atomic)
HTTP
Designed to let you “publish” databases
Flat-files, Currently read-only.
Same format as the filesystem replica type.
Backends are pluggable!
The filesystem is cheap and easy
The filesystem is portable
Help us write new backends:
CouchDB, Pos...
Prophet is designed to sync with “other”
databases and systems
They don’t need to support all of Prophet’s
features - Prop...
Synchronization
Publish
Serialize and export all of a replica's
resolutions and changesets
Pull
Integrate unseen resolutions and then
unseen changesets from a replica
Push
Integrate new resolutions and changesets
into a replica
Conflicts
Figures out the best resolution
“Nullifies” the conflict so the changeset can
be cleanly integrated
Integrates the conflictin...
Prophet has clever ways to figure out the best
resolution.
If there are previous resolutions for the same
conflict and a maj...
Scaling
Scaling to giant clusters is boring
(Can I play the “They’re not Green” card here?)
Scales to many weakly connected peers
...
We have a political agenda.
Cloud computing is not Open.
APIs for “export” are not good enough.
You should always have ful...
Do you have 10 billion
bugs, customer contacts
or sales orders?
Can I come
work for you?
We would love a scalable,
high performance
Prophet backend
Getting Involved
Project Status
Simple, well-defined Perl API
RESTy web API (with microserver)
Fast, lightweight backend
Small, active dev c...
Better ergonomics
Improved search and indexing
(Including full-text indexing)
Client libraries for other languages
Proper ...
Prophet
8225 lines of code and doc
2120 lines of tests
sd
2751 lines of code and doc
1121 lines of tests
Codebase
Prophet is very young
Prophet designed in April
Prophet core implemented in April
SD designed in April
SD built in June an...
We need your help!
Kick-ass functional and text indexing
Backend data store improvements
Slick GUIs for syncing
More Forei...
Prophet
http://syncwith.us/prophet/download
SD
http://syncwith.us/sd/download
Getting Prophet
http://syncwith.us
prophet-subscribe@lists.bestpractical.com
#prophet on freenode IRC
谢 谢!
Prophet - Beijing Perl Workshop
Prophet - Beijing Perl Workshop
Prophet - Beijing Perl Workshop
Prophet - Beijing Perl Workshop
Prophet - Beijing Perl Workshop
Prophet - Beijing Perl Workshop
Prophet - Beijing Perl Workshop
Prophet - Beijing Perl Workshop
Upcoming SlideShare
Loading in …5
×

Prophet - Beijing Perl Workshop

1,591 views
1,471 views

Published on

Published in: Technology, Spiritual
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
1,591
On SlideShare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
15
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Prophet - Beijing Perl Workshop

  1. 1. Prophet a path out of the cloud http://syncwith.us jesse@bestpractical.com
  2. 2. You may know me from... RT (Request Tracker) Jifty SVK Hiveminder Perl 6 T-shirts
  3. 3. I’ve been hacking on an open source database called “Prophet”
  4. 4. It has an API like Amazon SimpleDB or Google App Engine’s...
  5. 5. It’s designed for “team-scale” apps
  6. 6. It’s built for P2P replication and disconnected use
  7. 7. But first, a brief digression...
  8. 8. ...about cloud computing ☁ ☔
  9. 9. Living in the cloud = sharecropping(佃农)
  10. 10. (That’s bad) ☹
  11. 11. The bad old days:
  12. 12. Pic of sharecroppers
  13. 13. You farmed land you didn’t own...
  14. 14. ...with tools you couldn’t really afford
  15. 15. You paid for it with part of your harvest...
  16. 16. It sounded like a pretty sweet deal...
  17. 17. ...until things got bad
  18. 18. (Things always got bad)
  19. 19. (Internet)
  20. 20. (Cloud) (Internet)
  21. 21. In a bad year, you got further in debt to the land owner
  22. 22. So, what does this have to do with software?
  23. 23. The (more recent) bad old days:
  24. 24. pic of mainframes
  25. 25. You ran code you didn’t own on hardware you didn’t own
  26. 26. Things started to get better in the 1980s
  27. 27. Pic of PCs
  28. 28. Users started to be able to make choices about computing...
  29. 29. They weren’t all rosy
  30. 30. Pic of BSOD
  31. 31. Sometimes new versions of software broke things...
  32. 32. ...leaving you locked in to old versions
  33. 33. pic of win 31?
  34. 34. Things got ‘better’
  35. 35. rms che
  36. 36. Now, things are getting worse again...
  37. 37. What happens when your favorite service goes down?
  38. 38. pic of twitter being down
  39. 39. ...or stops accepting new signups?
  40. 40. ...or starts making arbitrary choices about what’s ‘safe’ content?
  41. 41. ...or breaks
  42. 42. You don’t own the services you use
  43. 43. You probably don’t even have a contract
  44. 44. When a service provider cuts you off, you lose
  45. 45. Not so secret shame: I still need the cloud
  46. 46. My calendar lives at google.com
  47. 47. I make a Web 2.0 todo list service called Hiveminder.com
  48. 48. pic of hiveminder
  49. 49. ☣ Using hosted apps is going to hurt you! ☣
  50. 50. What about Google Gears,Adobe Air, etc?
  51. 51. Great. now you can use your word processer while you’re offline!
  52. 52. Pic of wordperfect
  53. 53. Real offline apps should not need servers
  54. 54. Real offline apps should sync like you do
  55. 55. Back to that database thing...
  56. 56. JesseVincent
  57. 57. Chia-liang Kao
  58. 58. We work together
  59. 59. CL lives in Taipei Jesse lives in Boston
  60. 60. Sometimes we need to work face to face
  61. 61. TPE - BOS: TPE - HNL: BOS - HNL: 9410 mi 5,095 mi 5,069 mi
  62. 62. Step 1: Go to Hawaii for “work” Step 2: ??? Step 3: Prophet! Our Plan
  63. 63. The Plan Backfired We were there for 8 days We wrote 8000 lines of Perl We figured out step 2
  64. 64. Step 2: Build a Disconnected Syncable Database
  65. 65. Fallacies of Distributed Computing 1.The network is reliable. 2. Latency is zero. 3. Bandwidth is infinite. 4.The network is secure. 5.Topology doesn't change. 6.There is one administrator. 7.Transport cost is zero. 8.The network is homogeneous.
  66. 66. A grounded, peer to peer replicated, disconnected, versioned, property database with self-healing conflict resolution Prophet
  67. 67. What do all those buzzwords mean?
  68. 68. grounded Runs here
  69. 69. grounded Not here
  70. 70. grounded Runs at the edge Doesn’t need to run in the cloud Syncs with services you already use (Adaptors talk to “Foreign Replicas”)
  71. 71. Update any replica Pull from any replica Push to any replica Publish a replica Changes will propagate peer-to-peer replicated
  72. 72. Real-time replication is hard to scale It only “works” with constant connectivity I don’t have constant connectivity Neither do you Prophet sync can happen whenever disconnected
  73. 73. Every update is recorded as a change set Change sets don’t lose any data (so you can use them to go backwards) All history is introspectable Replication just replays changesets versioned
  74. 74. Atomic operations CREATE, READ, UPDATE, DELETE, SEARCH Record types can have optional validation and canonicalization Records of the same type do not need to have the same properties Add and remove properties at will property database
  75. 75. Remembers all conflict resolutions Syncs all resolutions with your peers Detects identical conflicts Uses your peers’ resolutions to “vote” for the winner of a conflict self-healing conflict resolution
  76. 76. Working with Prophet
  77. 77. RESTy API GET /records.json GET /records/Cars.json GET /records/Cars/716499-5F9-4AC4-827.json GET /records/Cars/716499-5F9-4AC4-827/wheels.json POST /records/Cars.json POST /records/Cars/716499-5F9-4AC4-827.json POST /records/Cars/716499-5F9-4AC4-827/wheels.json
  78. 78. RESTy API Yes, we should be using PUT and DELETE (or just provide an OpenRESTY adaptor) Yes, you can have a commit bit and help us fix it :)
  79. 79. Native API (Yes, the core is Perl.) my $cli = Prophet::CLI->new(); my $cxn = $cli->app_handle->handle; my $record = Prophet::Record->new( handle => $cxn, type => 'Person' ); my $uuid = $record->create( props => { name => 'Jesse', age => 31 } ); $record->set_prop( name => 'age', value => 32 ); my $people = Prophet::Collection->new( handle => $cxn, type => 'Person' ); $people->matching( sub { shift->prop('species') ne 'cat' } );
  80. 80. What could you build with Prophet?
  81. 81. A bug tracker: “simple defects” •id. Status, Summary •(Arbitrary other properties too) •History •Comments •Attachments sd
  82. 82. Initialize ./bin/sd init
  83. 83. ./bin/sd ticket create -- summary="Can't sync sd with Google Code" status=new Created ticket 5 (93BF979E-08C1-11DD-94C3-D4B1FCEE7EC4) Create
  84. 84. ./bin/sd ticket search --regex publish 29 } new the online help doesn't describe publish 34 } new publish a static html view of records 35 } new publish should create a static rss file List and Search
  85. 85. ./bin/sd ticket update --uuid 93BF979E-08C1-11DD-94C3-D4B1FCEE7EC4 -- status=resolved Updates
  86. 86. Bugs on my laptop aren’t interesting.
  87. 87. Jesse sd publish --to fsck.com:public_html/sd/ CL sd clone --from http://my.com/~jesse/sd Sync!
  88. 88. Jesse sd server CL sd pull --local Hackathon mode!
  89. 89. My project has a bug tracker
  90. 90. Actually, mine use two: • RT • hiveminder.com My project has a bug tracker
  91. 91. Foreign Replicas Prophet makes Foreign Replicas easy SD gets them "for free"
  92. 92. (Using only the public REST API) It took an afternoon Mirror an RT instance into SD Share it with your peers using prophet Sync changes back from your peers to RT Supports Comments and Attachments Wrote an RT Replica for SD
  93. 93. (Using only the public REST API) ...and one for Hiveminder
  94. 94. I can sync my bugs with RT or Hiveminder
  95. 95. Actually, it’s better
  96. 96. I can sync between RT and Hiveminder
  97. 97. I can sync between two different RTs, too
  98. 98. • Trac • Launchpad • Google Code • SourceForge • Bugzilla • Jira • GForge • debbugs • GNATS • todo.txt • Lighthouse • Redmine • FogBugz • What else? We need more replica definitions:
  99. 99. What else can you use Prophet for?
  100. 100. •CRM •Bug tracking •Sales orders •Phone book •Blog •Trading Card Database •Ideas? All the databases you want while offline.
  101. 101. How about a P2P BBS? Prophet doesn’t need a server. You can sync over sneakernet. “Private” Social Networks
  102. 102. A look inside Prophet
  103. 103. Anatomy of a Prophet Replica
  104. 104. The bits and pieces Database UUID Replica UUID Record Store Changeset Store Resolution Database Configuration metadata
  105. 105. The Record Store Stores individual records by type Not guaranteed to have all old versions
  106. 106. The Changeset Store Stores every change to a set of records Guaranteed to have all old changesets Replaying all changesets will create an exact clone of the replica
  107. 107. Replica Backends
  108. 108. Filesystem Readable Flat files Compact Fast (Not yet fully atomic)
  109. 109. HTTP Designed to let you “publish” databases Flat-files, Currently read-only. Same format as the filesystem replica type.
  110. 110. Backends are pluggable! The filesystem is cheap and easy The filesystem is portable Help us write new backends: CouchDB, Postgres, SQLite, MySQL, S3, AppEngine, $YOUR_FAVORITE_DB
  111. 111. Prophet is designed to sync with “other” databases and systems They don’t need to support all of Prophet’s features - Prophet knows how to interpret mumbo-jumbo from the Cloud Foreign Replicas will usually be app specific All current examples are for SD Foreign Replicas
  112. 112. Synchronization
  113. 113. Publish Serialize and export all of a replica's resolutions and changesets
  114. 114. Pull Integrate unseen resolutions and then unseen changesets from a replica
  115. 115. Push Integrate new resolutions and changesets into a replica
  116. 116. Conflicts
  117. 117. Figures out the best resolution “Nullifies” the conflict so the changeset can be cleanly integrated Integrates the conflicting changeset Records the resolution as a new changeset Records the resolution decision in the resolution database Resolving Conflicts
  118. 118. Prophet has clever ways to figure out the best resolution. If there are previous resolutions for the same conflict and a majority agree, use that If the merger has specified a “prefer this side” choice, use that Prompt the user to make a decision, giving them info about previous decisions for this conflict “The Best Resolution”
  119. 119. Scaling
  120. 120. Scaling to giant clusters is boring (Can I play the “They’re not Green” card here?) Scales to many weakly connected peers You are not Google Does anyone here work for Google? Current target is databases of O(50k) records How does it scale?
  121. 121. We have a political agenda. Cloud computing is not Open. APIs for “export” are not good enough. You should always have full control. You probably don’t need to store 10 billion records in one database. Why not, then?
  122. 122. Do you have 10 billion bugs, customer contacts or sales orders?
  123. 123. Can I come work for you?
  124. 124. We would love a scalable, high performance Prophet backend
  125. 125. Getting Involved
  126. 126. Project Status Simple, well-defined Perl API RESTy web API (with microserver) Fast, lightweight backend Small, active dev community Great test coverage ...less than great documentation coverage
  127. 127. Better ergonomics Improved search and indexing (Including full-text indexing) Client libraries for other languages Proper security model More apps Our Plans
  128. 128. Prophet 8225 lines of code and doc 2120 lines of tests sd 2751 lines of code and doc 1121 lines of tests Codebase
  129. 129. Prophet is very young Prophet designed in April Prophet core implemented in April SD designed in April SD built in June and July
  130. 130. We need your help! Kick-ass functional and text indexing Backend data store improvements Slick GUIs for syncing More Foreign Replicas for SD Documentation improvements A clever logo New applications
  131. 131. Prophet http://syncwith.us/prophet/download SD http://syncwith.us/sd/download Getting Prophet
  132. 132. http://syncwith.us prophet-subscribe@lists.bestpractical.com #prophet on freenode IRC 谢 谢!

×