Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Prophet - Beijing Perl Workshop


Published on

Published in: Technology, Spiritual
  • Be the first to comment

  • Be the first to like this

Prophet - Beijing Perl Workshop

  1. 1. Prophet a path out of the cloud
  2. 2. You may know me from... RT (Request Tracker) Jifty SVK Hiveminder Perl 6 T-shirts
  3. 3. I’ve been hacking on an open source database called “Prophet”
  4. 4. It has an API like Amazon SimpleDB or Google App Engine’s...
  5. 5. It’s designed for “team-scale” apps
  6. 6. It’s built for P2P replication and disconnected use
  7. 7. But first, a brief digression...
  8. 8. ...about cloud computing ☁ ☔
  9. 9. Living in the cloud = sharecropping(佃农)
  10. 10. (That’s bad) ☹
  11. 11. The bad old days:
  12. 12. Pic of sharecroppers
  13. 13. You farmed land you didn’t own...
  14. 14. ...with tools you couldn’t really afford
  15. 15. You paid for it with part of your harvest...
  16. 16. It sounded like a pretty sweet deal...
  17. 17. ...until things got bad
  18. 18. (Things always got bad)
  19. 19. (Internet)
  20. 20. (Cloud) (Internet)
  21. 21. In a bad year, you got further in debt to the land owner
  22. 22. So, what does this have to do with software?
  23. 23. The (more recent) bad old days:
  24. 24. pic of mainframes
  25. 25. You ran code you didn’t own on hardware you didn’t own
  26. 26. Things started to get better in the 1980s
  27. 27. Pic of PCs
  28. 28. Users started to be able to make choices about computing...
  29. 29. They weren’t all rosy
  30. 30. Pic of BSOD
  31. 31. Sometimes new versions of software broke things...
  32. 32. ...leaving you locked in to old versions
  33. 33. pic of win 31?
  34. 34. Things got ‘better’
  35. 35. rms che
  36. 36. Now, things are getting worse again...
  37. 37. What happens when your favorite service goes down?
  38. 38. pic of twitter being down
  39. 39. ...or stops accepting new signups?
  40. 40. ...or starts making arbitrary choices about what’s ‘safe’ content?
  41. 41. ...or breaks
  42. 42. You don’t own the services you use
  43. 43. You probably don’t even have a contract
  44. 44. When a service provider cuts you off, you lose
  45. 45. Not so secret shame: I still need the cloud
  46. 46. My calendar lives at
  47. 47. I make a Web 2.0 todo list service called
  48. 48. pic of hiveminder
  49. 49. ☣ Using hosted apps is going to hurt you! ☣
  50. 50. What about Google Gears,Adobe Air, etc?
  51. 51. Great. now you can use your word processer while you’re offline!
  52. 52. Pic of wordperfect
  53. 53. Real offline apps should not need servers
  54. 54. Real offline apps should sync like you do
  55. 55. Back to that database thing...
  56. 56. JesseVincent
  57. 57. Chia-liang Kao
  58. 58. We work together
  59. 59. CL lives in Taipei Jesse lives in Boston
  60. 60. Sometimes we need to work face to face
  61. 61. TPE - BOS: TPE - HNL: BOS - HNL: 9410 mi 5,095 mi 5,069 mi
  62. 62. Step 1: Go to Hawaii for “work” Step 2: ??? Step 3: Prophet! Our Plan
  63. 63. The Plan Backfired We were there for 8 days We wrote 8000 lines of Perl We figured out step 2
  64. 64. Step 2: Build a Disconnected Syncable Database
  65. 65. Fallacies of Distributed Computing 1.The network is reliable. 2. Latency is zero. 3. Bandwidth is infinite. 4.The network is secure. 5.Topology doesn't change. 6.There is one administrator. 7.Transport cost is zero. 8.The network is homogeneous.
  66. 66. A grounded, peer to peer replicated, disconnected, versioned, property database with self-healing conflict resolution Prophet
  67. 67. What do all those buzzwords mean?
  68. 68. grounded Runs here
  69. 69. grounded Not here
  70. 70. grounded Runs at the edge Doesn’t need to run in the cloud Syncs with services you already use (Adaptors talk to “Foreign Replicas”)
  71. 71. Update any replica Pull from any replica Push to any replica Publish a replica Changes will propagate peer-to-peer replicated
  72. 72. Real-time replication is hard to scale It only “works” with constant connectivity I don’t have constant connectivity Neither do you Prophet sync can happen whenever disconnected
  73. 73. Every update is recorded as a change set Change sets don’t lose any data (so you can use them to go backwards) All history is introspectable Replication just replays changesets versioned
  74. 74. Atomic operations CREATE, READ, UPDATE, DELETE, SEARCH Record types can have optional validation and canonicalization Records of the same type do not need to have the same properties Add and remove properties at will property database
  75. 75. Remembers all conflict resolutions Syncs all resolutions with your peers Detects identical conflicts Uses your peers’ resolutions to “vote” for the winner of a conflict self-healing conflict resolution
  76. 76. Working with Prophet
  77. 77. RESTy API GET /records.json GET /records/Cars.json GET /records/Cars/716499-5F9-4AC4-827.json GET /records/Cars/716499-5F9-4AC4-827/wheels.json POST /records/Cars.json POST /records/Cars/716499-5F9-4AC4-827.json POST /records/Cars/716499-5F9-4AC4-827/wheels.json
  78. 78. RESTy API Yes, we should be using PUT and DELETE (or just provide an OpenRESTY adaptor) Yes, you can have a commit bit and help us fix it :)
  79. 79. Native API (Yes, the core is Perl.) my $cli = Prophet::CLI->new(); my $cxn = $cli->app_handle->handle; my $record = Prophet::Record->new( handle => $cxn, type => 'Person' ); my $uuid = $record->create( props => { name => 'Jesse', age => 31 } ); $record->set_prop( name => 'age', value => 32 ); my $people = Prophet::Collection->new( handle => $cxn, type => 'Person' ); $people->matching( sub { shift->prop('species') ne 'cat' } );
  80. 80. What could you build with Prophet?
  81. 81. A bug tracker: “simple defects” •id. Status, Summary •(Arbitrary other properties too) •History •Comments •Attachments sd
  82. 82. Initialize ./bin/sd init
  83. 83. ./bin/sd ticket create -- summary="Can't sync sd with Google Code" status=new Created ticket 5 (93BF979E-08C1-11DD-94C3-D4B1FCEE7EC4) Create
  84. 84. ./bin/sd ticket search --regex publish 29 } new the online help doesn't describe publish 34 } new publish a static html view of records 35 } new publish should create a static rss file List and Search
  85. 85. ./bin/sd ticket update --uuid 93BF979E-08C1-11DD-94C3-D4B1FCEE7EC4 -- status=resolved Updates
  86. 86. Bugs on my laptop aren’t interesting.
  87. 87. Jesse sd publish --to CL sd clone --from Sync!
  88. 88. Jesse sd server CL sd pull --local Hackathon mode!
  89. 89. My project has a bug tracker
  90. 90. Actually, mine use two: • RT • My project has a bug tracker
  91. 91. Foreign Replicas Prophet makes Foreign Replicas easy SD gets them "for free"
  92. 92. (Using only the public REST API) It took an afternoon Mirror an RT instance into SD Share it with your peers using prophet Sync changes back from your peers to RT Supports Comments and Attachments Wrote an RT Replica for SD
  93. 93. (Using only the public REST API) ...and one for Hiveminder
  94. 94. I can sync my bugs with RT or Hiveminder
  95. 95. Actually, it’s better
  96. 96. I can sync between RT and Hiveminder
  97. 97. I can sync between two different RTs, too
  98. 98. • Trac • Launchpad • Google Code • SourceForge • Bugzilla • Jira • GForge • debbugs • GNATS • todo.txt • Lighthouse • Redmine • FogBugz • What else? We need more replica definitions:
  99. 99. What else can you use Prophet for?
  100. 100. •CRM •Bug tracking •Sales orders •Phone book •Blog •Trading Card Database •Ideas? All the databases you want while offline.
  101. 101. How about a P2P BBS? Prophet doesn’t need a server. You can sync over sneakernet. “Private” Social Networks
  102. 102. A look inside Prophet
  103. 103. Anatomy of a Prophet Replica
  104. 104. The bits and pieces Database UUID Replica UUID Record Store Changeset Store Resolution Database Configuration metadata
  105. 105. The Record Store Stores individual records by type Not guaranteed to have all old versions
  106. 106. The Changeset Store Stores every change to a set of records Guaranteed to have all old changesets Replaying all changesets will create an exact clone of the replica
  107. 107. Replica Backends
  108. 108. Filesystem Readable Flat files Compact Fast (Not yet fully atomic)
  109. 109. HTTP Designed to let you “publish” databases Flat-files, Currently read-only. Same format as the filesystem replica type.
  110. 110. Backends are pluggable! The filesystem is cheap and easy The filesystem is portable Help us write new backends: CouchDB, Postgres, SQLite, MySQL, S3, AppEngine, $YOUR_FAVORITE_DB
  111. 111. Prophet is designed to sync with “other” databases and systems They don’t need to support all of Prophet’s features - Prophet knows how to interpret mumbo-jumbo from the Cloud Foreign Replicas will usually be app specific All current examples are for SD Foreign Replicas
  112. 112. Synchronization
  113. 113. Publish Serialize and export all of a replica's resolutions and changesets
  114. 114. Pull Integrate unseen resolutions and then unseen changesets from a replica
  115. 115. Push Integrate new resolutions and changesets into a replica
  116. 116. Conflicts
  117. 117. Figures out the best resolution “Nullifies” the conflict so the changeset can be cleanly integrated Integrates the conflicting changeset Records the resolution as a new changeset Records the resolution decision in the resolution database Resolving Conflicts
  118. 118. Prophet has clever ways to figure out the best resolution. If there are previous resolutions for the same conflict and a majority agree, use that If the merger has specified a “prefer this side” choice, use that Prompt the user to make a decision, giving them info about previous decisions for this conflict “The Best Resolution”
  119. 119. Scaling
  120. 120. Scaling to giant clusters is boring (Can I play the “They’re not Green” card here?) Scales to many weakly connected peers You are not Google Does anyone here work for Google? Current target is databases of O(50k) records How does it scale?
  121. 121. We have a political agenda. Cloud computing is not Open. APIs for “export” are not good enough. You should always have full control. You probably don’t need to store 10 billion records in one database. Why not, then?
  122. 122. Do you have 10 billion bugs, customer contacts or sales orders?
  123. 123. Can I come work for you?
  124. 124. We would love a scalable, high performance Prophet backend
  125. 125. Getting Involved
  126. 126. Project Status Simple, well-defined Perl API RESTy web API (with microserver) Fast, lightweight backend Small, active dev community Great test coverage ...less than great documentation coverage
  127. 127. Better ergonomics Improved search and indexing (Including full-text indexing) Client libraries for other languages Proper security model More apps Our Plans
  128. 128. Prophet 8225 lines of code and doc 2120 lines of tests sd 2751 lines of code and doc 1121 lines of tests Codebase
  129. 129. Prophet is very young Prophet designed in April Prophet core implemented in April SD designed in April SD built in June and July
  130. 130. We need your help! Kick-ass functional and text indexing Backend data store improvements Slick GUIs for syncing More Foreign Replicas for SD Documentation improvements A clever logo New applications
  131. 131. Prophet SD Getting Prophet
  132. 132. #prophet on freenode IRC 谢 谢!