Search

Prophet - A peer to peer replicated disconnected database

Loading...

Flash Player 9 (or above) is needed to view presentations.
We have detected that you do not have it on your computer. To install it, go here.

0 comments

Post a comment

    Post a comment
    Embed Video
    Edit your comment Cancel

    Notes on slide 1

    5 Favorites, 1 Group & 1 Event

    Prophet - A peer to peer replicated disconnected database - Presentation Transcript

    1. Step 3: Prophet jesse@bestpractical.com clkao@bestpractical.com Friday, May 16, 2008 1
    2. Cloud computing is Sharecropping. ( ) Friday, May 16, 2008 2
    3. Chia-liang Kao Friday, May 16, 2008 3
    4. Jesse Vincent Friday, May 16, 2008 4
    5. We work together Friday, May 16, 2008 5
    6. CL lives in Taipei Jesse lives in Boston Friday, May 16, 2008 6
    7. Sometimes we need to work face to face Friday, May 16, 2008 7
    8. TPE~BOS: 9410 mi TPE-HNL: 5,095 mi BOS-HNL: 5,069 mi Friday, May 16, 2008 8
    9. Our Plan Step 1: Go to Hawaii for “work” Step 2: ??? Step 3: Prophet! Friday, May 16, 2008 9
    10. The Plan Backfired We were there for 8 days We wrote 8000 lines of Perl We figured out step 2 Friday, May 16, 2008 10
    11. Step 2: Build a Disconnected Syncable Database Friday, May 16, 2008 11
    12. Prophet Friday, May 16, 2008 12
    13. Prophet A semirelational, peer to peer replicated, disconnected, versioned, property database with self-healing conflict resolution Friday, May 16, 2008 13
    14. Bribery! Two tshirts. If you write a (real) patch for Prophet during this talk, you get a tshirt. Send patches to jesse@bestpractical.com Friday, May 16, 2008 14
    15. Getting Prophet Prophet http://code.bestpractical.com/bps-public/Prophet/ SD http://code.bestpractical.com/bps-public/sd/ Friday, May 16, 2008 15
    16. What do all those buzzwords mean? Friday, May 16, 2008 16
    17. semirelational Joins are expensive No backend Join support (yet) Friday, May 16, 2008 17
    18. peer-to-peer replicated Update any replica Pull from any replica Push to any replica Publish a replica Changes will propagate Friday, May 16, 2008 18
    19. disconnected Real-time replication is hard to scale It only works in the cloud I don't live in the cloud I want my data when I'm offline Prophet sync can happen whenever Friday, May 16, 2008 19
    20. versioned Compare a record to any point in the past All changes fully logged Undo changes Use history to be smart Friday, May 16, 2008 20
    21. property database Atomic operations CREATE, READ, UPDATE, DELETE, SEARCH Record types can have optional validation and canonicalization Records of the same type do not need to have the same properties Add and remove properties at will Friday, May 16, 2008 21
    22. self-healing conflict resolution Remembers all conflict resolutions Syncs all resolutions with your peers Detects identical conflicts Uses your peers’ resolutions to “vote” for the winner of a conflict Friday, May 16, 2008 22
    23. What could you build with Prophet? Friday, May 16, 2008 23
    24. sd A bug tracker: “simple defects” • id. Status, Summary • History • Comments • Attachments Friday, May 16, 2008 24
    25. ./bin/sd ticket create --summary \"Can't sync sd with Google Code\" --status new Created ticket 93BF979E-08C1-11DD-94C3-D4B1FCEE7EC4 Friday, May 16, 2008 25
    26. ./bin/sd ticket search --regex . 93BF979E-08C1-11DD-94C3-D4B1FCEE7EC4 Can't sync sd with Google Code new Friday, May 16, 2008 26
    27. ./bin/sd ticket update --uuid 93BF979E-08C1-11DD-94C3-D4B1FCEE7EC4 --status resolved ./bin/sd ticket search --regex . 93BF979E-08C1-11DD-94C3-D4B1FCEE7EC4 Can't sync sd with Google Code resolved Friday, May 16, 2008 27
    28. Bugs on my laptop aren’t interesting. Friday, May 16, 2008 28
    29. Syncable! Jesse ./bin/sd publish /tmp/mybugs scp -rvp /tmp/mybugs my.com:public_html/sd CL ./bin/sd pull prophet:http://my.com/~jesse/sd Friday, May 16, 2008 29
    30. My project has a bug tracker Friday, May 16, 2008 30
    31. My project has a bug tracker Actually, mine use two: • RT • hiveminder.com Friday, May 16, 2008 30
    32. Foreign Replicas Prophet makes Foriegn Replicas easy SD gets them \"for free\" Friday, May 16, 2008 31
    33. Wrote an RT Replica for SD (Using only the public REST API) It took an afternoon Mirror an RT instance into SD Share it with your peers using prophet Sync changes back from your peers to RT Supports Comments and Attachments Friday, May 16, 2008 32
    34. ...and one for Hiveminder (Using only the public REST API) Friday, May 16, 2008 33
    35. I can sync my bugs with RT or Hiveminder Friday, May 16, 2008 34
    36. Actually, it’s better Friday, May 16, 2008 35
    37. I can sync between RT and Hiveminder Friday, May 16, 2008 36
    38. I can sync between two different RTs, too Friday, May 16, 2008 37
    39. We need more replica definitions: • • Trac GNATS • • Google Code What else? • SourceForge • Bugzilla • Jira • GForge • debbugs Friday, May 16, 2008 38
    40. What can you use Prophet for? Friday, May 16, 2008 39
    41. All your “little” databases Friday, May 16, 2008 40
    42. All the databases you want while offline. • • CRM Blog • Bug tracking • Trading Card Database • Sales orders • Ideas? • Phone book Friday, May 16, 2008 41
    43. “Private” Social Networks How about a P2P BBS? Prophet doesn’t need a server. You can sync over sneakernet. Friday, May 16, 2008 42
    44. A look inside Prophet Friday, May 16, 2008 43
    45. Terms and Concepts • • Database Property Change • • Database Replica Conflict • • “Foreign” Replica Resolution • • Record Nullifcation Change Set • • Record Type Resolution Change Set • • Change Set Merge • • Change Merge Ticket Friday, May 16, 2008 44
    46. Database Backends Friday, May 16, 2008 45
    47. The Record Store Stores indivdual records by type Friday, May 16, 2008 46
    48. The Changeset Store Stores each atomic change to a set of records Replaying all changesets will create an exact clone of the replica Friday, May 16, 2008 47
    49. Native Replica Types Friday, May 16, 2008 48
    50. Subversion Slow Steady Robust Supports remote sync Requires Subversion Perl Bindings Friday, May 16, 2008 49
    51. Filesystem Readable Flat files Compact Fast (Not yet fully atomic) Friday, May 16, 2008 50
    52. HTTP Designed to let you “publish” databases Flat-files, Currently read-only. Same format as the filesystem replica type. Friday, May 16, 2008 51
    53. “Foreign” replicas Will usually be app specific All current examples are in SD Friday, May 16, 2008 52
    54. Synchronization Friday, May 16, 2008 53
    55. Publish Serialize and export all of a replica's resolutions and changesets Friday, May 16, 2008 54
    56. Pull Integrate unseen resolutions and then unseen changesets from a replica Friday, May 16, 2008 55
    57. Push Integrate new resolutions and changesets into a replica Friday, May 16, 2008 56
    58. Resolving Conflicts Figures out the best resolution “Nullifies” the conflict so the changeset can be cleanly integrated Integrates the conflicting changeset Records the resolution as a new changeset Records the resolution decision in the resolution database Friday, May 16, 2008 57
    59. “The Best Resolution” Prophet has clever ways to figure out the best resolution. If there are previous resolutions for the same conflict and a majority agree, use that If the merger has specified a “prefer this side” choice, use that Prompt the user to make a decision, giving them info about previous decisions for this conflict Friday, May 16, 2008 58
    60. We don't have a proof for the algorithm yet We do have dozens of runs of randomized testing. So far, it always stabilizes sanely. Friday, May 16, 2008 59
    61. How does it scale? Vertical scale is boring Designed to scale to many peers You are not Google Current target is databases of O(50k) records Friday, May 16, 2008 60
    62. How does it scale? Vertical scale is boring Designed to scale to many peers You are not Google Does anyone here work for Google? Current target is databases of O(50k) records Friday, May 16, 2008 60
    63. Why not, then? We just have a political agenda. Web 2.0 is not Open Source. Your data shouldn’t be ‘exportable’ from the cloud. You should always have full control. So we dont need to store 10 billion records in one database. Friday, May 16, 2008 61
    64. (Do you have 10 billion bugs, customer contacts or sales orders?) Friday, May 16, 2008 62
    65. That said, we'd love to see a scalable, high performance prophet replica store Friday, May 16, 2008 63
    66. Project Status Simple, well-defined Perl API RESTy web API (with microserver) Fast, lightweight backend Small, active dev community Great test coverage Horrible POD coverage Friday, May 16, 2008 64
    67. Our Plans Improved search and indexing (Including full-text indexing) Query language Proper security model Jifty, Catalyst, Rails models? Friday, May 16, 2008 65
    68. Prophet is very young Many, many hours of design About 10 days of two hackers hacking Friday, May 16, 2008 66
    69. Codebase Prophet 5479 lines of code and doc 1693 lines of tests sd 1695 lines of code and doc 876 lines of tests Friday, May 16, 2008 67
    70. Getting Prophet Prophet http://code.bestpractical.com/bps-public/Prophet/ SD http://code.bestpractical.com/bps-public/sd/ Friday, May 16, 2008 68
    71. Getting Involved prophet-subscribe@lists.bestpractical.com #prophet on freenode IRC Friday, May 16, 2008 69
    72. Thanks! Friday, May 16, 2008 70

    + obrajesseobrajesse, 2 years ago

    custom

    5096 views, 5 favs, 2 embeds more stats

    A semirelational,
    peer to peer replicated,
    discon more

    More info about this document

    © All Rights Reserved

    Go to text version

    • Total Views 5096
      • 4470 on SlideShare
      • 626 from embeds
    • Comments 0
    • Favorites 5
    • Downloads 50
    Most viewed embeds
    • 620 views on http://syncwith.us
    • 6 views on http://www.syncwith.us

    more

    All embeds
    • 620 views on http://syncwith.us
    • 6 views on http://www.syncwith.us

    less

    Flagged as inappropriate Flag as inappropriate
    Flag as inappropriate

    Select your reason for flagging this presentation as inappropriate. If needed, use the feedback form to let us know more details.

    Cancel
    File a copyright complaint
    Having problems? Go to our helpdesk?

    Categories

    Tags

    Groups / Events

    -->
    Search
    RSS Feed
    What's new?

    © 2009 SlideShare Inc. All Rights Reserved