SlideShare a Scribd company logo
1 of 72
Download to read offline
Step 3: Prophet
                             jesse@bestpractical.com
                             clkao@bestpractical.com




Friday, May 16, 2008                                   1
Cloud computing is
                         Sharecropping.

                           (       )
Friday, May 16, 2008                        2
Chia-liang Kao



Friday, May 16, 2008                    3
Jesse Vincent



Friday, May 16, 2008                   4
We work together



Friday, May 16, 2008                      5
CL lives in Taipei
                       Jesse lives in Boston


Friday, May 16, 2008                           6
Sometimes we need
                       to work face to face


Friday, May 16, 2008                          7
TPE~BOS: 9410 mi
                       TPE-HNL: 5,095 mi
                       BOS-HNL: 5,069 mi


Friday, May 16, 2008                       8
Our Plan
                       Step 1: Go to Hawaii for “work”
                       Step 2: ???
                       Step 3: Prophet!




Friday, May 16, 2008                                     9
The Plan Backfired
                       We were there for 8 days
                       We wrote 8000 lines of Perl
                       We figured out step 2




Friday, May 16, 2008                                 10
Step 2:

                       Build a Disconnected
                        Syncable Database


Friday, May 16, 2008                          11
Prophet



Friday, May 16, 2008             12
Prophet
                       A semirelational,
                       peer to peer replicated,
                       disconnected, versioned,
                       property database with
                       self-healing conflict resolution




Friday, May 16, 2008                                     13
Bribery!
                       Two tshirts.


                       If you write a (real) patch for Prophet
                       during this talk, you get a tshirt.


                       Send patches to jesse@bestpractical.com



Friday, May 16, 2008                                             14
Getting Prophet

                       Prophet
                       http://code.bestpractical.com/bps-public/Prophet/


                       SD
                       http://code.bestpractical.com/bps-public/sd/



Friday, May 16, 2008                                                       15
What do all those
                       buzzwords mean?


Friday, May 16, 2008                       16
semirelational
                       Joins are expensive
                       No backend Join support (yet)




Friday, May 16, 2008                                   17
peer-to-peer replicated
                       Update any replica
                       Pull from any replica
                       Push to any replica
                       Publish a replica
                       Changes will propagate




Friday, May 16, 2008                            18
disconnected
                       Real-time replication is hard to scale
                       It only works in the cloud
                       I don't live in the cloud
                       I want my data when I'm offline
                       Prophet sync can happen whenever




Friday, May 16, 2008                                            19
versioned
                       Compare a record to any point in the past
                       All changes fully logged
                       Undo changes
                       Use history to be smart




Friday, May 16, 2008                                               20
property database
                       Atomic operations
                         CREATE, READ, UPDATE, DELETE, SEARCH
                       Record types can have optional validation
                       and canonicalization
                        Records of the same type do not need to
                       have the same properties
                       Add and remove properties at will


Friday, May 16, 2008                                               21
self-healing conflict
                            resolution
                       Remembers all conflict resolutions
                       Syncs all resolutions with your peers
                       Detects identical conflicts
                       Uses your peers’ resolutions to “vote” for
                       the winner of a conflict




Friday, May 16, 2008                                                22
What could you build
                         with Prophet?


Friday, May 16, 2008                          23
sd
                       A bug tracker: “simple defects”
                         • id. Status, Summary
                         • History
                         • Comments
                         • Attachments



Friday, May 16, 2008                                     24
./bin/sd ticket create
                          --summary quot;Can't sync sd with Google Codequot;
                          --status new


                       Created ticket 93BF979E-08C1-11DD-94C3-D4B1FCEE7EC4




Friday, May 16, 2008                                                         25
./bin/sd ticket search --regex .


                       93BF979E-08C1-11DD-94C3-D4B1FCEE7EC4 Can't sync sd
                       with Google Code new




Friday, May 16, 2008                                                        26
./bin/sd ticket update
                         --uuid 93BF979E-08C1-11DD-94C3-D4B1FCEE7EC4
                         --status resolved


                       ./bin/sd ticket search --regex .

                       93BF979E-08C1-11DD-94C3-D4B1FCEE7EC4 Can't sync sd
                       with Google Code resolved




Friday, May 16, 2008                                                        27
Bugs on my laptop
                       aren’t interesting.


Friday, May 16, 2008                         28
Syncable!
                       Jesse
                       ./bin/sd publish /tmp/mybugs
                       scp -rvp /tmp/mybugs my.com:public_html/sd


                       CL
                       ./bin/sd pull prophet:http://my.com/~jesse/sd




Friday, May 16, 2008                                                   29
My project has a bug tracker




Friday, May 16, 2008              30
My project has a bug tracker
                       Actually, mine use two:
                        •   RT
                        •   hiveminder.com




Friday, May 16, 2008                             30
Foreign Replicas
                       Prophet makes Foriegn Replicas easy
                       SD gets them quot;for freequot;




Friday, May 16, 2008                                         31
Wrote an RT Replica for SD
                       (Using only the public REST API)
                       It took an afternoon
                       Mirror an RT instance into SD
                       Share it with your peers using prophet
                       Sync changes back from your peers to RT
                       Supports Comments and Attachments



Friday, May 16, 2008                                             32
...and one for Hiveminder
                       (Using only the public REST API)




Friday, May 16, 2008                                      33
I can sync my bugs with
                     RT or Hiveminder


Friday, May 16, 2008                        34
Actually, it’s better



Friday, May 16, 2008                           35
I can sync between RT
                           and Hiveminder


Friday, May 16, 2008                           36
I can sync between two
                      different RTs, too


Friday, May 16, 2008                       37
We need more replica
                           definitions:
                       •                 •
                           Trac              GNATS

                       •                 •
                           Google Code       What else?

                       •   SourceForge

                       •   Bugzilla

                       •   Jira

                       •   GForge

                       •   debbugs



Friday, May 16, 2008                                      38
What can you use
                        Prophet for?


Friday, May 16, 2008                      39
All your “little”
                          databases


Friday, May 16, 2008                       40
All the databases you
                        want while offline.
                       •              •
                        CRM             Blog
                       • Bug tracking • Trading Card
                                        Database
                       • Sales orders
                                      • Ideas?
                       • Phone book


Friday, May 16, 2008                                   41
“Private” Social Networks
                       How about a P2P BBS?


                       Prophet doesn’t need a server.
                       You can sync over sneakernet.




Friday, May 16, 2008                                    42
A look inside Prophet



Friday, May 16, 2008                           43
Terms and Concepts
                       •                       •
                           Database                Property Change

                       •                       •
                           Database Replica        Conflict

                       •                       •
                           “Foreign” Replica       Resolution

                       •                       •
                           Record                  Nullifcation Change Set

                       •                       •
                           Record Type             Resolution Change Set

                       •                       •
                           Change Set              Merge

                       •                       •
                           Change                  Merge Ticket



Friday, May 16, 2008                                                         44
Database Backends



Friday, May 16, 2008                       45
The Record Store
                       Stores indivdual records by type




Friday, May 16, 2008                                      46
The Changeset Store
                       Stores each atomic change to a set of
                       records
                       Replaying all changesets will create an exact
                       clone of the replica




Friday, May 16, 2008                                                   47
Native Replica Types



Friday, May 16, 2008                          48
Subversion
                       Slow
                       Steady
                       Robust
                       Supports remote sync
                       Requires Subversion Perl Bindings




Friday, May 16, 2008                                       49
Filesystem
                       Readable
                       Flat files
                       Compact
                       Fast
                       (Not yet fully atomic)




Friday, May 16, 2008                            50
HTTP
                       Designed to let you “publish” databases
                       Flat-files, Currently read-only.
                       Same format as the filesystem replica type.




Friday, May 16, 2008                                                51
“Foreign” replicas
                       Will usually be app specific
                       All current examples are in SD




Friday, May 16, 2008                                    52
Synchronization



Friday, May 16, 2008                     53
Publish
                       Serialize and export all of a replica's
                       resolutions and changesets




Friday, May 16, 2008                                             54
Pull
                       Integrate unseen resolutions and then
                       unseen changesets from a replica




Friday, May 16, 2008                                           55
Push
                       Integrate new resolutions and changesets
                       into a replica




Friday, May 16, 2008                                              56
Resolving Conflicts
                       Figures out the best resolution
                       “Nullifies” the conflict so the changeset can
                       be cleanly integrated
                       Integrates the conflicting changeset
                       Records the resolution as a new changeset
                       Records the resolution decision in the
                       resolution database


Friday, May 16, 2008                                                 57
“The Best Resolution”

                       Prophet has clever ways to figure out the best
                       resolution.
                       If there are previous resolutions for the same
                       conflict and a majority agree, use that
                       If the merger has specified a “prefer this side”
                       choice, use that
                       Prompt the user to make a decision, giving
                       them info about previous decisions for this
                       conflict
Friday, May 16, 2008                                                     58
We don't have a proof
                       for the algorithm yet
                        We do have dozens of runs of randomized
                        testing.
                        So far, it always stabilizes sanely.




Friday, May 16, 2008                                              59
How does it scale?
                       Vertical scale is boring
                       Designed to scale to many peers
                       You are not Google


                       Current target is databases of O(50k) records




Friday, May 16, 2008                                                   60
How does it scale?
                       Vertical scale is boring
                       Designed to scale to many peers
                       You are not Google
                       Does anyone here work for Google?
                       Current target is databases of O(50k) records




Friday, May 16, 2008                                                   60
Why not, then?

                       We just have a political agenda.
                       Web 2.0 is not Open Source.
                       Your data shouldn’t be ‘exportable’ from
                       the cloud.
                       You should always have full control.
                        So we dont need to store 10 billion
                       records in one database.

Friday, May 16, 2008                                              61
(Do you have 10 billion
                  bugs, customer contacts
                     or sales orders?)

Friday, May 16, 2008                         62
That said, we'd love to
                        see a scalable, high
                       performance prophet
                           replica store

Friday, May 16, 2008                             63
Project Status
                       Simple, well-defined Perl API
                       RESTy web API (with microserver)
                       Fast, lightweight backend
                       Small, active dev community
                       Great test coverage
                       Horrible POD coverage



Friday, May 16, 2008                                      64
Our Plans
                       Improved search and indexing
                         (Including full-text indexing)
                         Query language
                       Proper security model
                       Jifty, Catalyst, Rails models?




Friday, May 16, 2008                                      65
Prophet is very young
                       Many, many hours of design
                       About 10 days of two hackers hacking




Friday, May 16, 2008                                          66
Codebase
                       Prophet
                        5479 lines of code and doc
                        1693 lines of tests
                       sd
                        1695 lines of code and doc
                        876 lines of tests



Friday, May 16, 2008                                 67
Getting Prophet

                       Prophet
                       http://code.bestpractical.com/bps-public/Prophet/


                       SD
                       http://code.bestpractical.com/bps-public/sd/



Friday, May 16, 2008                                                       68
Getting Involved
                       prophet-subscribe@lists.bestpractical.com
                       #prophet on freenode IRC




Friday, May 16, 2008                                               69
Thanks!



Friday, May 16, 2008             70

More Related Content

More from Jesse Vincent

Building a keyboard from scratch
Building a keyboard from scratchBuilding a keyboard from scratch
Building a keyboard from scratchJesse Vincent
 
So, I made a keyboard
So, I made a keyboardSo, I made a keyboard
So, I made a keyboardJesse Vincent
 
Dancing App Stores - Android Open 2011
Dancing App Stores - Android Open 2011Dancing App Stores - Android Open 2011
Dancing App Stores - Android Open 2011Jesse Vincent
 
Perl 5.16 and Beyond - YAPC::Asia 2011
Perl 5.16 and Beyond - YAPC::Asia 2011Perl 5.16 and Beyond - YAPC::Asia 2011
Perl 5.16 and Beyond - YAPC::Asia 2011Jesse Vincent
 
Perl 5.16 and beyond
Perl 5.16 and beyondPerl 5.16 and beyond
Perl 5.16 and beyondJesse Vincent
 
OSCON 2011 - Perl 5.16 and beyond
OSCON 2011 - Perl 5.16 and beyondOSCON 2011 - Perl 5.16 and beyond
OSCON 2011 - Perl 5.16 and beyondJesse Vincent
 
RT4 - The whole sordid story
RT4 - The whole sordid storyRT4 - The whole sordid story
RT4 - The whole sordid storyJesse Vincent
 
K-9 Mail for Android
K-9 Mail for AndroidK-9 Mail for Android
K-9 Mail for AndroidJesse Vincent
 
P2P Bug Tracking with SD
P2P Bug Tracking with SDP2P Bug Tracking with SD
P2P Bug Tracking with SDJesse Vincent
 
SD - A peer to peer issue tracking system
SD - A peer to peer issue tracking systemSD - A peer to peer issue tracking system
SD - A peer to peer issue tracking systemJesse Vincent
 
Hacking your Kindle (OSCON Lightning Talk)
Hacking your Kindle (OSCON Lightning Talk)Hacking your Kindle (OSCON Lightning Talk)
Hacking your Kindle (OSCON Lightning Talk)Jesse Vincent
 
Beginning Kindle Hackery
Beginning Kindle HackeryBeginning Kindle Hackery
Beginning Kindle HackeryJesse Vincent
 
SD, a P2P bug tracking system
SD, a P2P bug tracking systemSD, a P2P bug tracking system
SD, a P2P bug tracking systemJesse Vincent
 
Prophet - Beijing Perl Workshop
Prophet - Beijing Perl WorkshopProphet - Beijing Perl Workshop
Prophet - Beijing Perl WorkshopJesse Vincent
 
Beijing Perl Workshop 2008 Hiveminder Secret Sauce
Beijing Perl Workshop 2008 Hiveminder Secret SauceBeijing Perl Workshop 2008 Hiveminder Secret Sauce
Beijing Perl Workshop 2008 Hiveminder Secret SauceJesse Vincent
 
A brief introduction to RTIR
A brief introduction to RTIRA brief introduction to RTIR
A brief introduction to RTIRJesse Vincent
 
An introduction to RTIR
An introduction to RTIRAn introduction to RTIR
An introduction to RTIRJesse Vincent
 
Prophet: a path out of the Cloud
Prophet: a path out of the CloudProphet: a path out of the Cloud
Prophet: a path out of the CloudJesse Vincent
 
Hiveminder - Everything but the Secret Sauce
Hiveminder - Everything but the Secret SauceHiveminder - Everything but the Secret Sauce
Hiveminder - Everything but the Secret SauceJesse Vincent
 

More from Jesse Vincent (20)

Building a keyboard from scratch
Building a keyboard from scratchBuilding a keyboard from scratch
Building a keyboard from scratch
 
So, I made a keyboard
So, I made a keyboardSo, I made a keyboard
So, I made a keyboard
 
Dancing App Stores - Android Open 2011
Dancing App Stores - Android Open 2011Dancing App Stores - Android Open 2011
Dancing App Stores - Android Open 2011
 
Perl 5.16 and Beyond - YAPC::Asia 2011
Perl 5.16 and Beyond - YAPC::Asia 2011Perl 5.16 and Beyond - YAPC::Asia 2011
Perl 5.16 and Beyond - YAPC::Asia 2011
 
Perl 5.16 and beyond
Perl 5.16 and beyondPerl 5.16 and beyond
Perl 5.16 and beyond
 
OSCON 2011 - Perl 5.16 and beyond
OSCON 2011 - Perl 5.16 and beyondOSCON 2011 - Perl 5.16 and beyond
OSCON 2011 - Perl 5.16 and beyond
 
RT4 - The whole sordid story
RT4 - The whole sordid storyRT4 - The whole sordid story
RT4 - The whole sordid story
 
K-9 Mail for Android
K-9 Mail for AndroidK-9 Mail for Android
K-9 Mail for Android
 
Perl 5.12.0
Perl 5.12.0Perl 5.12.0
Perl 5.12.0
 
P2P Bug Tracking with SD
P2P Bug Tracking with SDP2P Bug Tracking with SD
P2P Bug Tracking with SD
 
SD - A peer to peer issue tracking system
SD - A peer to peer issue tracking systemSD - A peer to peer issue tracking system
SD - A peer to peer issue tracking system
 
Hacking your Kindle (OSCON Lightning Talk)
Hacking your Kindle (OSCON Lightning Talk)Hacking your Kindle (OSCON Lightning Talk)
Hacking your Kindle (OSCON Lightning Talk)
 
Beginning Kindle Hackery
Beginning Kindle HackeryBeginning Kindle Hackery
Beginning Kindle Hackery
 
SD, a P2P bug tracking system
SD, a P2P bug tracking systemSD, a P2P bug tracking system
SD, a P2P bug tracking system
 
Prophet - Beijing Perl Workshop
Prophet - Beijing Perl WorkshopProphet - Beijing Perl Workshop
Prophet - Beijing Perl Workshop
 
Beijing Perl Workshop 2008 Hiveminder Secret Sauce
Beijing Perl Workshop 2008 Hiveminder Secret SauceBeijing Perl Workshop 2008 Hiveminder Secret Sauce
Beijing Perl Workshop 2008 Hiveminder Secret Sauce
 
A brief introduction to RTIR
A brief introduction to RTIRA brief introduction to RTIR
A brief introduction to RTIR
 
An introduction to RTIR
An introduction to RTIRAn introduction to RTIR
An introduction to RTIR
 
Prophet: a path out of the Cloud
Prophet: a path out of the CloudProphet: a path out of the Cloud
Prophet: a path out of the Cloud
 
Hiveminder - Everything but the Secret Sauce
Hiveminder - Everything but the Secret SauceHiveminder - Everything but the Secret Sauce
Hiveminder - Everything but the Secret Sauce
 

Recently uploaded

The Importance of Indoor Air Quality (English)
The Importance of Indoor Air Quality (English)The Importance of Indoor Air Quality (English)
The Importance of Indoor Air Quality (English)IES VE
 
CyberSecurity - Computers In Libraries 2024
CyberSecurity - Computers In Libraries 2024CyberSecurity - Computers In Libraries 2024
CyberSecurity - Computers In Libraries 2024Brian Pichman
 
Introduction - IPLOOK NETWORKS CO., LTD.
Introduction - IPLOOK NETWORKS CO., LTD.Introduction - IPLOOK NETWORKS CO., LTD.
Introduction - IPLOOK NETWORKS CO., LTD.IPLOOK Networks
 
LF Energy Webinar - Unveiling OpenEEMeter 4.0
LF Energy Webinar - Unveiling OpenEEMeter 4.0LF Energy Webinar - Unveiling OpenEEMeter 4.0
LF Energy Webinar - Unveiling OpenEEMeter 4.0DanBrown980551
 
UiPath Studio Web workshop Series - Day 3
UiPath Studio Web workshop Series - Day 3UiPath Studio Web workshop Series - Day 3
UiPath Studio Web workshop Series - Day 3DianaGray10
 
Emil Eifrem at GraphSummit Copenhagen 2024 - The Art of the Possible.pptx
Emil Eifrem at GraphSummit Copenhagen 2024 - The Art of the Possible.pptxEmil Eifrem at GraphSummit Copenhagen 2024 - The Art of the Possible.pptx
Emil Eifrem at GraphSummit Copenhagen 2024 - The Art of the Possible.pptxNeo4j
 
Scenario Library et REX Discover industry- and role- based scenarios
Scenario Library et REX Discover industry- and role- based scenariosScenario Library et REX Discover industry- and role- based scenarios
Scenario Library et REX Discover industry- and role- based scenariosErol GIRAUDY
 
Trailblazer Community - Flows Workshop (Session 2)
Trailblazer Community - Flows Workshop (Session 2)Trailblazer Community - Flows Workshop (Session 2)
Trailblazer Community - Flows Workshop (Session 2)Muhammad Tiham Siddiqui
 
Key Trends Shaping the Future of Infrastructure.pdf
Key Trends Shaping the Future of Infrastructure.pdfKey Trends Shaping the Future of Infrastructure.pdf
Key Trends Shaping the Future of Infrastructure.pdfCheryl Hung
 
2024.03.12 Cost drivers of cultivated meat production.pdf
2024.03.12 Cost drivers of cultivated meat production.pdf2024.03.12 Cost drivers of cultivated meat production.pdf
2024.03.12 Cost drivers of cultivated meat production.pdfThe Good Food Institute
 
Keep Your Finger on the Pulse of Your Building's Performance with IES Live
Keep Your Finger on the Pulse of Your Building's Performance with IES LiveKeep Your Finger on the Pulse of Your Building's Performance with IES Live
Keep Your Finger on the Pulse of Your Building's Performance with IES LiveIES VE
 
UiPath Studio Web workshop series - Day 2
UiPath Studio Web workshop series - Day 2UiPath Studio Web workshop series - Day 2
UiPath Studio Web workshop series - Day 2DianaGray10
 
Q4 2023 Quarterly Investor Presentation - FINAL - v1.pdf
Q4 2023 Quarterly Investor Presentation - FINAL - v1.pdfQ4 2023 Quarterly Investor Presentation - FINAL - v1.pdf
Q4 2023 Quarterly Investor Presentation - FINAL - v1.pdfTejal81
 
GraphSummit Copenhagen 2024 - Neo4j Vision and Roadmap.pptx
GraphSummit Copenhagen 2024 - Neo4j Vision and Roadmap.pptxGraphSummit Copenhagen 2024 - Neo4j Vision and Roadmap.pptx
GraphSummit Copenhagen 2024 - Neo4j Vision and Roadmap.pptxNeo4j
 
TrustArc Webinar - How to Live in a Post Third-Party Cookie World
TrustArc Webinar - How to Live in a Post Third-Party Cookie WorldTrustArc Webinar - How to Live in a Post Third-Party Cookie World
TrustArc Webinar - How to Live in a Post Third-Party Cookie WorldTrustArc
 
Webinar: The Art of Prioritizing Your Product Roadmap by AWS Sr PM - Tech
Webinar: The Art of Prioritizing Your Product Roadmap by AWS Sr PM - TechWebinar: The Art of Prioritizing Your Product Roadmap by AWS Sr PM - Tech
Webinar: The Art of Prioritizing Your Product Roadmap by AWS Sr PM - TechProduct School
 
The Zero-ETL Approach: Enhancing Data Agility and Insight
The Zero-ETL Approach: Enhancing Data Agility and InsightThe Zero-ETL Approach: Enhancing Data Agility and Insight
The Zero-ETL Approach: Enhancing Data Agility and InsightSafe Software
 
Technical SEO for Improved Accessibility WTS FEST
Technical SEO for Improved Accessibility  WTS FESTTechnical SEO for Improved Accessibility  WTS FEST
Technical SEO for Improved Accessibility WTS FESTBillieHyde
 
Flow Control | Block Size | ST Min | First Frame
Flow Control | Block Size | ST Min | First FrameFlow Control | Block Size | ST Min | First Frame
Flow Control | Block Size | ST Min | First FrameKapil Thakar
 

Recently uploaded (20)

SheDev 2024
SheDev 2024SheDev 2024
SheDev 2024
 
The Importance of Indoor Air Quality (English)
The Importance of Indoor Air Quality (English)The Importance of Indoor Air Quality (English)
The Importance of Indoor Air Quality (English)
 
CyberSecurity - Computers In Libraries 2024
CyberSecurity - Computers In Libraries 2024CyberSecurity - Computers In Libraries 2024
CyberSecurity - Computers In Libraries 2024
 
Introduction - IPLOOK NETWORKS CO., LTD.
Introduction - IPLOOK NETWORKS CO., LTD.Introduction - IPLOOK NETWORKS CO., LTD.
Introduction - IPLOOK NETWORKS CO., LTD.
 
LF Energy Webinar - Unveiling OpenEEMeter 4.0
LF Energy Webinar - Unveiling OpenEEMeter 4.0LF Energy Webinar - Unveiling OpenEEMeter 4.0
LF Energy Webinar - Unveiling OpenEEMeter 4.0
 
UiPath Studio Web workshop Series - Day 3
UiPath Studio Web workshop Series - Day 3UiPath Studio Web workshop Series - Day 3
UiPath Studio Web workshop Series - Day 3
 
Emil Eifrem at GraphSummit Copenhagen 2024 - The Art of the Possible.pptx
Emil Eifrem at GraphSummit Copenhagen 2024 - The Art of the Possible.pptxEmil Eifrem at GraphSummit Copenhagen 2024 - The Art of the Possible.pptx
Emil Eifrem at GraphSummit Copenhagen 2024 - The Art of the Possible.pptx
 
Scenario Library et REX Discover industry- and role- based scenarios
Scenario Library et REX Discover industry- and role- based scenariosScenario Library et REX Discover industry- and role- based scenarios
Scenario Library et REX Discover industry- and role- based scenarios
 
Trailblazer Community - Flows Workshop (Session 2)
Trailblazer Community - Flows Workshop (Session 2)Trailblazer Community - Flows Workshop (Session 2)
Trailblazer Community - Flows Workshop (Session 2)
 
Key Trends Shaping the Future of Infrastructure.pdf
Key Trends Shaping the Future of Infrastructure.pdfKey Trends Shaping the Future of Infrastructure.pdf
Key Trends Shaping the Future of Infrastructure.pdf
 
2024.03.12 Cost drivers of cultivated meat production.pdf
2024.03.12 Cost drivers of cultivated meat production.pdf2024.03.12 Cost drivers of cultivated meat production.pdf
2024.03.12 Cost drivers of cultivated meat production.pdf
 
Keep Your Finger on the Pulse of Your Building's Performance with IES Live
Keep Your Finger on the Pulse of Your Building's Performance with IES LiveKeep Your Finger on the Pulse of Your Building's Performance with IES Live
Keep Your Finger on the Pulse of Your Building's Performance with IES Live
 
UiPath Studio Web workshop series - Day 2
UiPath Studio Web workshop series - Day 2UiPath Studio Web workshop series - Day 2
UiPath Studio Web workshop series - Day 2
 
Q4 2023 Quarterly Investor Presentation - FINAL - v1.pdf
Q4 2023 Quarterly Investor Presentation - FINAL - v1.pdfQ4 2023 Quarterly Investor Presentation - FINAL - v1.pdf
Q4 2023 Quarterly Investor Presentation - FINAL - v1.pdf
 
GraphSummit Copenhagen 2024 - Neo4j Vision and Roadmap.pptx
GraphSummit Copenhagen 2024 - Neo4j Vision and Roadmap.pptxGraphSummit Copenhagen 2024 - Neo4j Vision and Roadmap.pptx
GraphSummit Copenhagen 2024 - Neo4j Vision and Roadmap.pptx
 
TrustArc Webinar - How to Live in a Post Third-Party Cookie World
TrustArc Webinar - How to Live in a Post Third-Party Cookie WorldTrustArc Webinar - How to Live in a Post Third-Party Cookie World
TrustArc Webinar - How to Live in a Post Third-Party Cookie World
 
Webinar: The Art of Prioritizing Your Product Roadmap by AWS Sr PM - Tech
Webinar: The Art of Prioritizing Your Product Roadmap by AWS Sr PM - TechWebinar: The Art of Prioritizing Your Product Roadmap by AWS Sr PM - Tech
Webinar: The Art of Prioritizing Your Product Roadmap by AWS Sr PM - Tech
 
The Zero-ETL Approach: Enhancing Data Agility and Insight
The Zero-ETL Approach: Enhancing Data Agility and InsightThe Zero-ETL Approach: Enhancing Data Agility and Insight
The Zero-ETL Approach: Enhancing Data Agility and Insight
 
Technical SEO for Improved Accessibility WTS FEST
Technical SEO for Improved Accessibility  WTS FESTTechnical SEO for Improved Accessibility  WTS FEST
Technical SEO for Improved Accessibility WTS FEST
 
Flow Control | Block Size | ST Min | First Frame
Flow Control | Block Size | ST Min | First FrameFlow Control | Block Size | ST Min | First Frame
Flow Control | Block Size | ST Min | First Frame
 

Prophet - A peer to peer replicated disconnected database

  • 1. Step 3: Prophet jesse@bestpractical.com clkao@bestpractical.com Friday, May 16, 2008 1
  • 2. Cloud computing is Sharecropping. ( ) Friday, May 16, 2008 2
  • 5. We work together Friday, May 16, 2008 5
  • 6. CL lives in Taipei Jesse lives in Boston Friday, May 16, 2008 6
  • 7. Sometimes we need to work face to face Friday, May 16, 2008 7
  • 8. TPE~BOS: 9410 mi TPE-HNL: 5,095 mi BOS-HNL: 5,069 mi Friday, May 16, 2008 8
  • 9. Our Plan Step 1: Go to Hawaii for “work” Step 2: ??? Step 3: Prophet! Friday, May 16, 2008 9
  • 10. The Plan Backfired We were there for 8 days We wrote 8000 lines of Perl We figured out step 2 Friday, May 16, 2008 10
  • 11. Step 2: Build a Disconnected Syncable Database Friday, May 16, 2008 11
  • 13. Prophet A semirelational, peer to peer replicated, disconnected, versioned, property database with self-healing conflict resolution Friday, May 16, 2008 13
  • 14. Bribery! Two tshirts. If you write a (real) patch for Prophet during this talk, you get a tshirt. Send patches to jesse@bestpractical.com Friday, May 16, 2008 14
  • 15. Getting Prophet Prophet http://code.bestpractical.com/bps-public/Prophet/ SD http://code.bestpractical.com/bps-public/sd/ Friday, May 16, 2008 15
  • 16. What do all those buzzwords mean? Friday, May 16, 2008 16
  • 17. semirelational Joins are expensive No backend Join support (yet) Friday, May 16, 2008 17
  • 18. peer-to-peer replicated Update any replica Pull from any replica Push to any replica Publish a replica Changes will propagate Friday, May 16, 2008 18
  • 19. disconnected Real-time replication is hard to scale It only works in the cloud I don't live in the cloud I want my data when I'm offline Prophet sync can happen whenever Friday, May 16, 2008 19
  • 20. versioned Compare a record to any point in the past All changes fully logged Undo changes Use history to be smart Friday, May 16, 2008 20
  • 21. property database Atomic operations CREATE, READ, UPDATE, DELETE, SEARCH Record types can have optional validation and canonicalization Records of the same type do not need to have the same properties Add and remove properties at will Friday, May 16, 2008 21
  • 22. self-healing conflict resolution Remembers all conflict resolutions Syncs all resolutions with your peers Detects identical conflicts Uses your peers’ resolutions to “vote” for the winner of a conflict Friday, May 16, 2008 22
  • 23. What could you build with Prophet? Friday, May 16, 2008 23
  • 24. sd A bug tracker: “simple defects” • id. Status, Summary • History • Comments • Attachments Friday, May 16, 2008 24
  • 25. ./bin/sd ticket create --summary quot;Can't sync sd with Google Codequot; --status new Created ticket 93BF979E-08C1-11DD-94C3-D4B1FCEE7EC4 Friday, May 16, 2008 25
  • 26. ./bin/sd ticket search --regex . 93BF979E-08C1-11DD-94C3-D4B1FCEE7EC4 Can't sync sd with Google Code new Friday, May 16, 2008 26
  • 27. ./bin/sd ticket update --uuid 93BF979E-08C1-11DD-94C3-D4B1FCEE7EC4 --status resolved ./bin/sd ticket search --regex . 93BF979E-08C1-11DD-94C3-D4B1FCEE7EC4 Can't sync sd with Google Code resolved Friday, May 16, 2008 27
  • 28. Bugs on my laptop aren’t interesting. Friday, May 16, 2008 28
  • 29. Syncable! Jesse ./bin/sd publish /tmp/mybugs scp -rvp /tmp/mybugs my.com:public_html/sd CL ./bin/sd pull prophet:http://my.com/~jesse/sd Friday, May 16, 2008 29
  • 30. My project has a bug tracker Friday, May 16, 2008 30
  • 31. My project has a bug tracker Actually, mine use two: • RT • hiveminder.com Friday, May 16, 2008 30
  • 32. Foreign Replicas Prophet makes Foriegn Replicas easy SD gets them quot;for freequot; Friday, May 16, 2008 31
  • 33. Wrote an RT Replica for SD (Using only the public REST API) It took an afternoon Mirror an RT instance into SD Share it with your peers using prophet Sync changes back from your peers to RT Supports Comments and Attachments Friday, May 16, 2008 32
  • 34. ...and one for Hiveminder (Using only the public REST API) Friday, May 16, 2008 33
  • 35. I can sync my bugs with RT or Hiveminder Friday, May 16, 2008 34
  • 37. I can sync between RT and Hiveminder Friday, May 16, 2008 36
  • 38. I can sync between two different RTs, too Friday, May 16, 2008 37
  • 39. We need more replica definitions: • • Trac GNATS • • Google Code What else? • SourceForge • Bugzilla • Jira • GForge • debbugs Friday, May 16, 2008 38
  • 40. What can you use Prophet for? Friday, May 16, 2008 39
  • 41. All your “little” databases Friday, May 16, 2008 40
  • 42. All the databases you want while offline. • • CRM Blog • Bug tracking • Trading Card Database • Sales orders • Ideas? • Phone book Friday, May 16, 2008 41
  • 43. “Private” Social Networks How about a P2P BBS? Prophet doesn’t need a server. You can sync over sneakernet. Friday, May 16, 2008 42
  • 44. A look inside Prophet Friday, May 16, 2008 43
  • 45. Terms and Concepts • • Database Property Change • • Database Replica Conflict • • “Foreign” Replica Resolution • • Record Nullifcation Change Set • • Record Type Resolution Change Set • • Change Set Merge • • Change Merge Ticket Friday, May 16, 2008 44
  • 47. The Record Store Stores indivdual records by type Friday, May 16, 2008 46
  • 48. The Changeset Store Stores each atomic change to a set of records Replaying all changesets will create an exact clone of the replica Friday, May 16, 2008 47
  • 50. Subversion Slow Steady Robust Supports remote sync Requires Subversion Perl Bindings Friday, May 16, 2008 49
  • 51. Filesystem Readable Flat files Compact Fast (Not yet fully atomic) Friday, May 16, 2008 50
  • 52. HTTP Designed to let you “publish” databases Flat-files, Currently read-only. Same format as the filesystem replica type. Friday, May 16, 2008 51
  • 53. “Foreign” replicas Will usually be app specific All current examples are in SD Friday, May 16, 2008 52
  • 55. Publish Serialize and export all of a replica's resolutions and changesets Friday, May 16, 2008 54
  • 56. Pull Integrate unseen resolutions and then unseen changesets from a replica Friday, May 16, 2008 55
  • 57. Push Integrate new resolutions and changesets into a replica Friday, May 16, 2008 56
  • 58. Resolving Conflicts Figures out the best resolution “Nullifies” the conflict so the changeset can be cleanly integrated Integrates the conflicting changeset Records the resolution as a new changeset Records the resolution decision in the resolution database Friday, May 16, 2008 57
  • 59. “The Best Resolution” Prophet has clever ways to figure out the best resolution. If there are previous resolutions for the same conflict and a majority agree, use that If the merger has specified a “prefer this side” choice, use that Prompt the user to make a decision, giving them info about previous decisions for this conflict Friday, May 16, 2008 58
  • 60. We don't have a proof for the algorithm yet We do have dozens of runs of randomized testing. So far, it always stabilizes sanely. Friday, May 16, 2008 59
  • 61. How does it scale? Vertical scale is boring Designed to scale to many peers You are not Google Current target is databases of O(50k) records Friday, May 16, 2008 60
  • 62. How does it scale? Vertical scale is boring Designed to scale to many peers You are not Google Does anyone here work for Google? Current target is databases of O(50k) records Friday, May 16, 2008 60
  • 63. Why not, then? We just have a political agenda. Web 2.0 is not Open Source. Your data shouldn’t be ‘exportable’ from the cloud. You should always have full control. So we dont need to store 10 billion records in one database. Friday, May 16, 2008 61
  • 64. (Do you have 10 billion bugs, customer contacts or sales orders?) Friday, May 16, 2008 62
  • 65. That said, we'd love to see a scalable, high performance prophet replica store Friday, May 16, 2008 63
  • 66. Project Status Simple, well-defined Perl API RESTy web API (with microserver) Fast, lightweight backend Small, active dev community Great test coverage Horrible POD coverage Friday, May 16, 2008 64
  • 67. Our Plans Improved search and indexing (Including full-text indexing) Query language Proper security model Jifty, Catalyst, Rails models? Friday, May 16, 2008 65
  • 68. Prophet is very young Many, many hours of design About 10 days of two hackers hacking Friday, May 16, 2008 66
  • 69. Codebase Prophet 5479 lines of code and doc 1693 lines of tests sd 1695 lines of code and doc 876 lines of tests Friday, May 16, 2008 67
  • 70. Getting Prophet Prophet http://code.bestpractical.com/bps-public/Prophet/ SD http://code.bestpractical.com/bps-public/sd/ Friday, May 16, 2008 68
  • 71. Getting Involved prophet-subscribe@lists.bestpractical.com #prophet on freenode IRC Friday, May 16, 2008 69