Your SlideShare is downloading. ×
0
Step 3: Prophet
                             jesse@bestpractical.com
                             clkao@bestpractical.com
...
Cloud computing is
                         Sharecropping.

                           (       )
Friday, May 16, 2008     ...
Chia-liang Kao



Friday, May 16, 2008                    3
Jesse Vincent



Friday, May 16, 2008                   4
We work together



Friday, May 16, 2008                      5
CL lives in Taipei
                       Jesse lives in Boston


Friday, May 16, 2008                           6
Sometimes we need
                       to work face to face


Friday, May 16, 2008                          7
TPE~BOS: 9410 mi
                       TPE-HNL: 5,095 mi
                       BOS-HNL: 5,069 mi


Friday, May 16, 2008 ...
Our Plan
                       Step 1: Go to Hawaii for “work”
                       Step 2: ???
                       ...
The Plan Backfired
                       We were there for 8 days
                       We wrote 8000 lines of Perl
     ...
Step 2:

                       Build a Disconnected
                        Syncable Database


Friday, May 16, 2008     ...
Prophet



Friday, May 16, 2008             12
Prophet
                       A semirelational,
                       peer to peer replicated,
                       di...
Bribery!
                       Two tshirts.


                       If you write a (real) patch for Prophet
            ...
Getting Prophet

                       Prophet
                       http://code.bestpractical.com/bps-public/Prophet/

...
What do all those
                       buzzwords mean?


Friday, May 16, 2008                       16
semirelational
                       Joins are expensive
                       No backend Join support (yet)




Friday,...
peer-to-peer replicated
                       Update any replica
                       Pull from any replica
           ...
disconnected
                       Real-time replication is hard to scale
                       It only works in the clo...
versioned
                       Compare a record to any point in the past
                       All changes fully logged...
property database
                       Atomic operations
                         CREATE, READ, UPDATE, DELETE, SEARCH
 ...
self-healing conflict
                            resolution
                       Remembers all conflict resolutions
     ...
What could you build
                         with Prophet?


Friday, May 16, 2008                          23
sd
                       A bug tracker: “simple defects”
                         • id. Status, Summary
                 ...
./bin/sd ticket create
                          --summary quot;Can't sync sd with Google Codequot;
                      ...
./bin/sd ticket search --regex .


                       93BF979E-08C1-11DD-94C3-D4B1FCEE7EC4 Can't sync sd
             ...
./bin/sd ticket update
                         --uuid 93BF979E-08C1-11DD-94C3-D4B1FCEE7EC4
                         --sta...
Bugs on my laptop
                       aren’t interesting.


Friday, May 16, 2008                         28
Syncable!
                       Jesse
                       ./bin/sd publish /tmp/mybugs
                       scp -rvp...
My project has a bug tracker




Friday, May 16, 2008              30
My project has a bug tracker
                       Actually, mine use two:
                        •   RT
               ...
Foreign Replicas
                       Prophet makes Foriegn Replicas easy
                       SD gets them quot;for f...
Wrote an RT Replica for SD
                       (Using only the public REST API)
                       It took an after...
...and one for Hiveminder
                       (Using only the public REST API)




Friday, May 16, 2008                ...
I can sync my bugs with
                     RT or Hiveminder


Friday, May 16, 2008                        34
Actually, it’s better



Friday, May 16, 2008                           35
I can sync between RT
                           and Hiveminder


Friday, May 16, 2008                           36
I can sync between two
                      different RTs, too


Friday, May 16, 2008                       37
We need more replica
                           definitions:
                       •                 •
                   ...
What can you use
                        Prophet for?


Friday, May 16, 2008                      39
All your “little”
                          databases


Friday, May 16, 2008                       40
All the databases you
                        want while offline.
                       •              •
                 ...
“Private” Social Networks
                       How about a P2P BBS?


                       Prophet doesn’t need a serv...
A look inside Prophet



Friday, May 16, 2008                           43
Terms and Concepts
                       •                       •
                           Database                Pro...
Database Backends



Friday, May 16, 2008                       45
The Record Store
                       Stores indivdual records by type




Friday, May 16, 2008                         ...
The Changeset Store
                       Stores each atomic change to a set of
                       records
          ...
Native Replica Types



Friday, May 16, 2008                          48
Subversion
                       Slow
                       Steady
                       Robust
                       ...
Filesystem
                       Readable
                       Flat files
                       Compact
               ...
HTTP
                       Designed to let you “publish” databases
                       Flat-files, Currently read-only....
“Foreign” replicas
                       Will usually be app specific
                       All current examples are in S...
Synchronization



Friday, May 16, 2008                     53
Publish
                       Serialize and export all of a replica's
                       resolutions and changesets

...
Pull
                       Integrate unseen resolutions and then
                       unseen changesets from a replica
...
Push
                       Integrate new resolutions and changesets
                       into a replica




Friday, May...
Resolving Conflicts
                       Figures out the best resolution
                       “Nullifies” the conflict so...
“The Best Resolution”

                       Prophet has clever ways to figure out the best
                       resolut...
We don't have a proof
                       for the algorithm yet
                        We do have dozens of runs of ra...
How does it scale?
                       Vertical scale is boring
                       Designed to scale to many peers
...
How does it scale?
                       Vertical scale is boring
                       Designed to scale to many peers
...
Why not, then?

                       We just have a political agenda.
                       Web 2.0 is not Open Source....
(Do you have 10 billion
                  bugs, customer contacts
                     or sales orders?)

Friday, May 16, ...
That said, we'd love to
                        see a scalable, high
                       performance prophet
          ...
Project Status
                       Simple, well-defined Perl API
                       RESTy web API (with microserver)...
Our Plans
                       Improved search and indexing
                         (Including full-text indexing)
    ...
Prophet is very young
                       Many, many hours of design
                       About 10 days of two hacker...
Codebase
                       Prophet
                        5479 lines of code and doc
                        1693 li...
Getting Prophet

                       Prophet
                       http://code.bestpractical.com/bps-public/Prophet/

...
Getting Involved
                       prophet-subscribe@lists.bestpractical.com
                       #prophet on freen...
Thanks!



Friday, May 16, 2008             70
Upcoming SlideShare
Loading in...5
×

Prophet - A peer to peer replicated disconnected database

11,888

Published on

A semirelational,
peer to peer replicated,
disconnected, versioned,
property database with
self-healing conflict resolution, as presented at YAPC::Asia 2008

Published in: Technology, News & Politics
1 Comment
8 Likes
Statistics
Notes
  • Great work.
    Thanks for sharing.

    http://www.e-rachat-credits.fr/
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
No Downloads
Views
Total Views
11,888
On Slideshare
0
From Embeds
0
Number of Embeds
4
Actions
Shares
0
Downloads
75
Comments
1
Likes
8
Embeds 0
No embeds

No notes for slide

Transcript of "Prophet - A peer to peer replicated disconnected database"

  1. 1. Step 3: Prophet jesse@bestpractical.com clkao@bestpractical.com Friday, May 16, 2008 1
  2. 2. Cloud computing is Sharecropping. ( ) Friday, May 16, 2008 2
  3. 3. Chia-liang Kao Friday, May 16, 2008 3
  4. 4. Jesse Vincent Friday, May 16, 2008 4
  5. 5. We work together Friday, May 16, 2008 5
  6. 6. CL lives in Taipei Jesse lives in Boston Friday, May 16, 2008 6
  7. 7. Sometimes we need to work face to face Friday, May 16, 2008 7
  8. 8. TPE~BOS: 9410 mi TPE-HNL: 5,095 mi BOS-HNL: 5,069 mi Friday, May 16, 2008 8
  9. 9. Our Plan Step 1: Go to Hawaii for “work” Step 2: ??? Step 3: Prophet! Friday, May 16, 2008 9
  10. 10. The Plan Backfired We were there for 8 days We wrote 8000 lines of Perl We figured out step 2 Friday, May 16, 2008 10
  11. 11. Step 2: Build a Disconnected Syncable Database Friday, May 16, 2008 11
  12. 12. Prophet Friday, May 16, 2008 12
  13. 13. Prophet A semirelational, peer to peer replicated, disconnected, versioned, property database with self-healing conflict resolution Friday, May 16, 2008 13
  14. 14. Bribery! Two tshirts. If you write a (real) patch for Prophet during this talk, you get a tshirt. Send patches to jesse@bestpractical.com Friday, May 16, 2008 14
  15. 15. Getting Prophet Prophet http://code.bestpractical.com/bps-public/Prophet/ SD http://code.bestpractical.com/bps-public/sd/ Friday, May 16, 2008 15
  16. 16. What do all those buzzwords mean? Friday, May 16, 2008 16
  17. 17. semirelational Joins are expensive No backend Join support (yet) Friday, May 16, 2008 17
  18. 18. peer-to-peer replicated Update any replica Pull from any replica Push to any replica Publish a replica Changes will propagate Friday, May 16, 2008 18
  19. 19. disconnected Real-time replication is hard to scale It only works in the cloud I don't live in the cloud I want my data when I'm offline Prophet sync can happen whenever Friday, May 16, 2008 19
  20. 20. versioned Compare a record to any point in the past All changes fully logged Undo changes Use history to be smart Friday, May 16, 2008 20
  21. 21. property database Atomic operations CREATE, READ, UPDATE, DELETE, SEARCH Record types can have optional validation and canonicalization Records of the same type do not need to have the same properties Add and remove properties at will Friday, May 16, 2008 21
  22. 22. self-healing conflict resolution Remembers all conflict resolutions Syncs all resolutions with your peers Detects identical conflicts Uses your peers’ resolutions to “vote” for the winner of a conflict Friday, May 16, 2008 22
  23. 23. What could you build with Prophet? Friday, May 16, 2008 23
  24. 24. sd A bug tracker: “simple defects” • id. Status, Summary • History • Comments • Attachments Friday, May 16, 2008 24
  25. 25. ./bin/sd ticket create --summary quot;Can't sync sd with Google Codequot; --status new Created ticket 93BF979E-08C1-11DD-94C3-D4B1FCEE7EC4 Friday, May 16, 2008 25
  26. 26. ./bin/sd ticket search --regex . 93BF979E-08C1-11DD-94C3-D4B1FCEE7EC4 Can't sync sd with Google Code new Friday, May 16, 2008 26
  27. 27. ./bin/sd ticket update --uuid 93BF979E-08C1-11DD-94C3-D4B1FCEE7EC4 --status resolved ./bin/sd ticket search --regex . 93BF979E-08C1-11DD-94C3-D4B1FCEE7EC4 Can't sync sd with Google Code resolved Friday, May 16, 2008 27
  28. 28. Bugs on my laptop aren’t interesting. Friday, May 16, 2008 28
  29. 29. Syncable! Jesse ./bin/sd publish /tmp/mybugs scp -rvp /tmp/mybugs my.com:public_html/sd CL ./bin/sd pull prophet:http://my.com/~jesse/sd Friday, May 16, 2008 29
  30. 30. My project has a bug tracker Friday, May 16, 2008 30
  31. 31. My project has a bug tracker Actually, mine use two: • RT • hiveminder.com Friday, May 16, 2008 30
  32. 32. Foreign Replicas Prophet makes Foriegn Replicas easy SD gets them quot;for freequot; Friday, May 16, 2008 31
  33. 33. Wrote an RT Replica for SD (Using only the public REST API) It took an afternoon Mirror an RT instance into SD Share it with your peers using prophet Sync changes back from your peers to RT Supports Comments and Attachments Friday, May 16, 2008 32
  34. 34. ...and one for Hiveminder (Using only the public REST API) Friday, May 16, 2008 33
  35. 35. I can sync my bugs with RT or Hiveminder Friday, May 16, 2008 34
  36. 36. Actually, it’s better Friday, May 16, 2008 35
  37. 37. I can sync between RT and Hiveminder Friday, May 16, 2008 36
  38. 38. I can sync between two different RTs, too Friday, May 16, 2008 37
  39. 39. We need more replica definitions: • • Trac GNATS • • Google Code What else? • SourceForge • Bugzilla • Jira • GForge • debbugs Friday, May 16, 2008 38
  40. 40. What can you use Prophet for? Friday, May 16, 2008 39
  41. 41. All your “little” databases Friday, May 16, 2008 40
  42. 42. All the databases you want while offline. • • CRM Blog • Bug tracking • Trading Card Database • Sales orders • Ideas? • Phone book Friday, May 16, 2008 41
  43. 43. “Private” Social Networks How about a P2P BBS? Prophet doesn’t need a server. You can sync over sneakernet. Friday, May 16, 2008 42
  44. 44. A look inside Prophet Friday, May 16, 2008 43
  45. 45. Terms and Concepts • • Database Property Change • • Database Replica Conflict • • “Foreign” Replica Resolution • • Record Nullifcation Change Set • • Record Type Resolution Change Set • • Change Set Merge • • Change Merge Ticket Friday, May 16, 2008 44
  46. 46. Database Backends Friday, May 16, 2008 45
  47. 47. The Record Store Stores indivdual records by type Friday, May 16, 2008 46
  48. 48. The Changeset Store Stores each atomic change to a set of records Replaying all changesets will create an exact clone of the replica Friday, May 16, 2008 47
  49. 49. Native Replica Types Friday, May 16, 2008 48
  50. 50. Subversion Slow Steady Robust Supports remote sync Requires Subversion Perl Bindings Friday, May 16, 2008 49
  51. 51. Filesystem Readable Flat files Compact Fast (Not yet fully atomic) Friday, May 16, 2008 50
  52. 52. HTTP Designed to let you “publish” databases Flat-files, Currently read-only. Same format as the filesystem replica type. Friday, May 16, 2008 51
  53. 53. “Foreign” replicas Will usually be app specific All current examples are in SD Friday, May 16, 2008 52
  54. 54. Synchronization Friday, May 16, 2008 53
  55. 55. Publish Serialize and export all of a replica's resolutions and changesets Friday, May 16, 2008 54
  56. 56. Pull Integrate unseen resolutions and then unseen changesets from a replica Friday, May 16, 2008 55
  57. 57. Push Integrate new resolutions and changesets into a replica Friday, May 16, 2008 56
  58. 58. Resolving Conflicts Figures out the best resolution “Nullifies” the conflict so the changeset can be cleanly integrated Integrates the conflicting changeset Records the resolution as a new changeset Records the resolution decision in the resolution database Friday, May 16, 2008 57
  59. 59. “The Best Resolution” Prophet has clever ways to figure out the best resolution. If there are previous resolutions for the same conflict and a majority agree, use that If the merger has specified a “prefer this side” choice, use that Prompt the user to make a decision, giving them info about previous decisions for this conflict Friday, May 16, 2008 58
  60. 60. We don't have a proof for the algorithm yet We do have dozens of runs of randomized testing. So far, it always stabilizes sanely. Friday, May 16, 2008 59
  61. 61. How does it scale? Vertical scale is boring Designed to scale to many peers You are not Google Current target is databases of O(50k) records Friday, May 16, 2008 60
  62. 62. How does it scale? Vertical scale is boring Designed to scale to many peers You are not Google Does anyone here work for Google? Current target is databases of O(50k) records Friday, May 16, 2008 60
  63. 63. Why not, then? We just have a political agenda. Web 2.0 is not Open Source. Your data shouldn’t be ‘exportable’ from the cloud. You should always have full control. So we dont need to store 10 billion records in one database. Friday, May 16, 2008 61
  64. 64. (Do you have 10 billion bugs, customer contacts or sales orders?) Friday, May 16, 2008 62
  65. 65. That said, we'd love to see a scalable, high performance prophet replica store Friday, May 16, 2008 63
  66. 66. Project Status Simple, well-defined Perl API RESTy web API (with microserver) Fast, lightweight backend Small, active dev community Great test coverage Horrible POD coverage Friday, May 16, 2008 64
  67. 67. Our Plans Improved search and indexing (Including full-text indexing) Query language Proper security model Jifty, Catalyst, Rails models? Friday, May 16, 2008 65
  68. 68. Prophet is very young Many, many hours of design About 10 days of two hackers hacking Friday, May 16, 2008 66
  69. 69. Codebase Prophet 5479 lines of code and doc 1693 lines of tests sd 1695 lines of code and doc 876 lines of tests Friday, May 16, 2008 67
  70. 70. Getting Prophet Prophet http://code.bestpractical.com/bps-public/Prophet/ SD http://code.bestpractical.com/bps-public/sd/ Friday, May 16, 2008 68
  71. 71. Getting Involved prophet-subscribe@lists.bestpractical.com #prophet on freenode IRC Friday, May 16, 2008 69
  72. 72. Thanks! Friday, May 16, 2008 70
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×