SlideShare a Scribd company logo
Step 3: Prophet
                             jesse@bestpractical.com
                             clkao@bestpractical.com




Friday, May 16, 2008                                   1
Cloud computing is
                         Sharecropping.

                           (       )
Friday, May 16, 2008                        2
Chia-liang Kao



Friday, May 16, 2008                    3
Jesse Vincent



Friday, May 16, 2008                   4
We work together



Friday, May 16, 2008                      5
CL lives in Taipei
                       Jesse lives in Boston


Friday, May 16, 2008                           6
Sometimes we need
                       to work face to face


Friday, May 16, 2008                          7
TPE~BOS: 9410 mi
                       TPE-HNL: 5,095 mi
                       BOS-HNL: 5,069 mi


Friday, May 16, 2008                       8
Our Plan
                       Step 1: Go to Hawaii for “work”
                       Step 2: ???
                       Step 3: Prophet!




Friday, May 16, 2008                                     9
The Plan Backfired
                       We were there for 8 days
                       We wrote 8000 lines of Perl
                       We figured out step 2




Friday, May 16, 2008                                 10
Step 2:

                       Build a Disconnected
                        Syncable Database


Friday, May 16, 2008                          11
Prophet



Friday, May 16, 2008             12
Prophet
                       A semirelational,
                       peer to peer replicated,
                       disconnected, versioned,
                       property database with
                       self-healing conflict resolution




Friday, May 16, 2008                                     13
Bribery!
                       Two tshirts.


                       If you write a (real) patch for Prophet
                       during this talk, you get a tshirt.


                       Send patches to jesse@bestpractical.com



Friday, May 16, 2008                                             14
Getting Prophet

                       Prophet
                       http://code.bestpractical.com/bps-public/Prophet/


                       SD
                       http://code.bestpractical.com/bps-public/sd/



Friday, May 16, 2008                                                       15
What do all those
                       buzzwords mean?


Friday, May 16, 2008                       16
semirelational
                       Joins are expensive
                       No backend Join support (yet)




Friday, May 16, 2008                                   17
peer-to-peer replicated
                       Update any replica
                       Pull from any replica
                       Push to any replica
                       Publish a replica
                       Changes will propagate




Friday, May 16, 2008                            18
disconnected
                       Real-time replication is hard to scale
                       It only works in the cloud
                       I don't live in the cloud
                       I want my data when I'm offline
                       Prophet sync can happen whenever




Friday, May 16, 2008                                            19
versioned
                       Compare a record to any point in the past
                       All changes fully logged
                       Undo changes
                       Use history to be smart




Friday, May 16, 2008                                               20
property database
                       Atomic operations
                         CREATE, READ, UPDATE, DELETE, SEARCH
                       Record types can have optional validation
                       and canonicalization
                        Records of the same type do not need to
                       have the same properties
                       Add and remove properties at will


Friday, May 16, 2008                                               21
self-healing conflict
                            resolution
                       Remembers all conflict resolutions
                       Syncs all resolutions with your peers
                       Detects identical conflicts
                       Uses your peers’ resolutions to “vote” for
                       the winner of a conflict




Friday, May 16, 2008                                                22
What could you build
                         with Prophet?


Friday, May 16, 2008                          23
sd
                       A bug tracker: “simple defects”
                         • id. Status, Summary
                         • History
                         • Comments
                         • Attachments



Friday, May 16, 2008                                     24
./bin/sd ticket create
                          --summary quot;Can't sync sd with Google Codequot;
                          --status new


                       Created ticket 93BF979E-08C1-11DD-94C3-D4B1FCEE7EC4




Friday, May 16, 2008                                                         25
./bin/sd ticket search --regex .


                       93BF979E-08C1-11DD-94C3-D4B1FCEE7EC4 Can't sync sd
                       with Google Code new




Friday, May 16, 2008                                                        26
./bin/sd ticket update
                         --uuid 93BF979E-08C1-11DD-94C3-D4B1FCEE7EC4
                         --status resolved


                       ./bin/sd ticket search --regex .

                       93BF979E-08C1-11DD-94C3-D4B1FCEE7EC4 Can't sync sd
                       with Google Code resolved




Friday, May 16, 2008                                                        27
Bugs on my laptop
                       aren’t interesting.


Friday, May 16, 2008                         28
Syncable!
                       Jesse
                       ./bin/sd publish /tmp/mybugs
                       scp -rvp /tmp/mybugs my.com:public_html/sd


                       CL
                       ./bin/sd pull prophet:http://my.com/~jesse/sd




Friday, May 16, 2008                                                   29
My project has a bug tracker




Friday, May 16, 2008              30
My project has a bug tracker
                       Actually, mine use two:
                        •   RT
                        •   hiveminder.com




Friday, May 16, 2008                             30
Foreign Replicas
                       Prophet makes Foriegn Replicas easy
                       SD gets them quot;for freequot;




Friday, May 16, 2008                                         31
Wrote an RT Replica for SD
                       (Using only the public REST API)
                       It took an afternoon
                       Mirror an RT instance into SD
                       Share it with your peers using prophet
                       Sync changes back from your peers to RT
                       Supports Comments and Attachments



Friday, May 16, 2008                                             32
...and one for Hiveminder
                       (Using only the public REST API)




Friday, May 16, 2008                                      33
I can sync my bugs with
                     RT or Hiveminder


Friday, May 16, 2008                        34
Actually, it’s better



Friday, May 16, 2008                           35
I can sync between RT
                           and Hiveminder


Friday, May 16, 2008                           36
I can sync between two
                      different RTs, too


Friday, May 16, 2008                       37
We need more replica
                           definitions:
                       •                 •
                           Trac              GNATS

                       •                 •
                           Google Code       What else?

                       •   SourceForge

                       •   Bugzilla

                       •   Jira

                       •   GForge

                       •   debbugs



Friday, May 16, 2008                                      38
What can you use
                        Prophet for?


Friday, May 16, 2008                      39
All your “little”
                          databases


Friday, May 16, 2008                       40
All the databases you
                        want while offline.
                       •              •
                        CRM             Blog
                       • Bug tracking • Trading Card
                                        Database
                       • Sales orders
                                      • Ideas?
                       • Phone book


Friday, May 16, 2008                                   41
“Private” Social Networks
                       How about a P2P BBS?


                       Prophet doesn’t need a server.
                       You can sync over sneakernet.




Friday, May 16, 2008                                    42
A look inside Prophet



Friday, May 16, 2008                           43
Terms and Concepts
                       •                       •
                           Database                Property Change

                       •                       •
                           Database Replica        Conflict

                       •                       •
                           “Foreign” Replica       Resolution

                       •                       •
                           Record                  Nullifcation Change Set

                       •                       •
                           Record Type             Resolution Change Set

                       •                       •
                           Change Set              Merge

                       •                       •
                           Change                  Merge Ticket



Friday, May 16, 2008                                                         44
Database Backends



Friday, May 16, 2008                       45
The Record Store
                       Stores indivdual records by type




Friday, May 16, 2008                                      46
The Changeset Store
                       Stores each atomic change to a set of
                       records
                       Replaying all changesets will create an exact
                       clone of the replica




Friday, May 16, 2008                                                   47
Native Replica Types



Friday, May 16, 2008                          48
Subversion
                       Slow
                       Steady
                       Robust
                       Supports remote sync
                       Requires Subversion Perl Bindings




Friday, May 16, 2008                                       49
Filesystem
                       Readable
                       Flat files
                       Compact
                       Fast
                       (Not yet fully atomic)




Friday, May 16, 2008                            50
HTTP
                       Designed to let you “publish” databases
                       Flat-files, Currently read-only.
                       Same format as the filesystem replica type.




Friday, May 16, 2008                                                51
“Foreign” replicas
                       Will usually be app specific
                       All current examples are in SD




Friday, May 16, 2008                                    52
Synchronization



Friday, May 16, 2008                     53
Publish
                       Serialize and export all of a replica's
                       resolutions and changesets




Friday, May 16, 2008                                             54
Pull
                       Integrate unseen resolutions and then
                       unseen changesets from a replica




Friday, May 16, 2008                                           55
Push
                       Integrate new resolutions and changesets
                       into a replica




Friday, May 16, 2008                                              56
Resolving Conflicts
                       Figures out the best resolution
                       “Nullifies” the conflict so the changeset can
                       be cleanly integrated
                       Integrates the conflicting changeset
                       Records the resolution as a new changeset
                       Records the resolution decision in the
                       resolution database


Friday, May 16, 2008                                                 57
“The Best Resolution”

                       Prophet has clever ways to figure out the best
                       resolution.
                       If there are previous resolutions for the same
                       conflict and a majority agree, use that
                       If the merger has specified a “prefer this side”
                       choice, use that
                       Prompt the user to make a decision, giving
                       them info about previous decisions for this
                       conflict
Friday, May 16, 2008                                                     58
We don't have a proof
                       for the algorithm yet
                        We do have dozens of runs of randomized
                        testing.
                        So far, it always stabilizes sanely.




Friday, May 16, 2008                                              59
How does it scale?
                       Vertical scale is boring
                       Designed to scale to many peers
                       You are not Google


                       Current target is databases of O(50k) records




Friday, May 16, 2008                                                   60
How does it scale?
                       Vertical scale is boring
                       Designed to scale to many peers
                       You are not Google
                       Does anyone here work for Google?
                       Current target is databases of O(50k) records




Friday, May 16, 2008                                                   60
Why not, then?

                       We just have a political agenda.
                       Web 2.0 is not Open Source.
                       Your data shouldn’t be ‘exportable’ from
                       the cloud.
                       You should always have full control.
                        So we dont need to store 10 billion
                       records in one database.

Friday, May 16, 2008                                              61
(Do you have 10 billion
                  bugs, customer contacts
                     or sales orders?)

Friday, May 16, 2008                         62
That said, we'd love to
                        see a scalable, high
                       performance prophet
                           replica store

Friday, May 16, 2008                             63
Project Status
                       Simple, well-defined Perl API
                       RESTy web API (with microserver)
                       Fast, lightweight backend
                       Small, active dev community
                       Great test coverage
                       Horrible POD coverage



Friday, May 16, 2008                                      64
Our Plans
                       Improved search and indexing
                         (Including full-text indexing)
                         Query language
                       Proper security model
                       Jifty, Catalyst, Rails models?




Friday, May 16, 2008                                      65
Prophet is very young
                       Many, many hours of design
                       About 10 days of two hackers hacking




Friday, May 16, 2008                                          66
Codebase
                       Prophet
                        5479 lines of code and doc
                        1693 lines of tests
                       sd
                        1695 lines of code and doc
                        876 lines of tests



Friday, May 16, 2008                                 67
Getting Prophet

                       Prophet
                       http://code.bestpractical.com/bps-public/Prophet/


                       SD
                       http://code.bestpractical.com/bps-public/sd/



Friday, May 16, 2008                                                       68
Getting Involved
                       prophet-subscribe@lists.bestpractical.com
                       #prophet on freenode IRC




Friday, May 16, 2008                                               69
Thanks!



Friday, May 16, 2008             70

More Related Content

More from Jesse Vincent

Building a keyboard from scratch
Building a keyboard from scratchBuilding a keyboard from scratch
Building a keyboard from scratch
Jesse Vincent
 
So, I made a keyboard
So, I made a keyboardSo, I made a keyboard
So, I made a keyboard
Jesse Vincent
 
Dancing App Stores - Android Open 2011
Dancing App Stores - Android Open 2011Dancing App Stores - Android Open 2011
Dancing App Stores - Android Open 2011
Jesse Vincent
 
Perl 5.16 and Beyond - YAPC::Asia 2011
Perl 5.16 and Beyond - YAPC::Asia 2011Perl 5.16 and Beyond - YAPC::Asia 2011
Perl 5.16 and Beyond - YAPC::Asia 2011
Jesse Vincent
 
Perl 5.16 and beyond
Perl 5.16 and beyondPerl 5.16 and beyond
Perl 5.16 and beyond
Jesse Vincent
 
OSCON 2011 - Perl 5.16 and beyond
OSCON 2011 - Perl 5.16 and beyondOSCON 2011 - Perl 5.16 and beyond
OSCON 2011 - Perl 5.16 and beyond
Jesse Vincent
 
RT4 - The whole sordid story
RT4 - The whole sordid storyRT4 - The whole sordid story
RT4 - The whole sordid story
Jesse Vincent
 
K-9 Mail for Android
K-9 Mail for AndroidK-9 Mail for Android
K-9 Mail for Android
Jesse Vincent
 
Perl 5.12.0
Perl 5.12.0Perl 5.12.0
Perl 5.12.0
Jesse Vincent
 
P2P Bug Tracking with SD
P2P Bug Tracking with SDP2P Bug Tracking with SD
P2P Bug Tracking with SD
Jesse Vincent
 
SD - A peer to peer issue tracking system
SD - A peer to peer issue tracking systemSD - A peer to peer issue tracking system
SD - A peer to peer issue tracking system
Jesse Vincent
 
Hacking your Kindle (OSCON Lightning Talk)
Hacking your Kindle (OSCON Lightning Talk)Hacking your Kindle (OSCON Lightning Talk)
Hacking your Kindle (OSCON Lightning Talk)
Jesse Vincent
 
Beginning Kindle Hackery
Beginning Kindle HackeryBeginning Kindle Hackery
Beginning Kindle Hackery
Jesse Vincent
 
SD, a P2P bug tracking system
SD, a P2P bug tracking systemSD, a P2P bug tracking system
SD, a P2P bug tracking system
Jesse Vincent
 
Prophet - Beijing Perl Workshop
Prophet - Beijing Perl WorkshopProphet - Beijing Perl Workshop
Prophet - Beijing Perl Workshop
Jesse Vincent
 
Beijing Perl Workshop 2008 Hiveminder Secret Sauce
Beijing Perl Workshop 2008 Hiveminder Secret SauceBeijing Perl Workshop 2008 Hiveminder Secret Sauce
Beijing Perl Workshop 2008 Hiveminder Secret Sauce
Jesse Vincent
 
A brief introduction to RTIR
A brief introduction to RTIRA brief introduction to RTIR
A brief introduction to RTIR
Jesse Vincent
 
An introduction to RTIR
An introduction to RTIRAn introduction to RTIR
An introduction to RTIR
Jesse Vincent
 
Prophet: a path out of the Cloud
Prophet: a path out of the CloudProphet: a path out of the Cloud
Prophet: a path out of the Cloud
Jesse Vincent
 
Hiveminder - Everything but the Secret Sauce
Hiveminder - Everything but the Secret SauceHiveminder - Everything but the Secret Sauce
Hiveminder - Everything but the Secret Sauce
Jesse Vincent
 

More from Jesse Vincent (20)

Building a keyboard from scratch
Building a keyboard from scratchBuilding a keyboard from scratch
Building a keyboard from scratch
 
So, I made a keyboard
So, I made a keyboardSo, I made a keyboard
So, I made a keyboard
 
Dancing App Stores - Android Open 2011
Dancing App Stores - Android Open 2011Dancing App Stores - Android Open 2011
Dancing App Stores - Android Open 2011
 
Perl 5.16 and Beyond - YAPC::Asia 2011
Perl 5.16 and Beyond - YAPC::Asia 2011Perl 5.16 and Beyond - YAPC::Asia 2011
Perl 5.16 and Beyond - YAPC::Asia 2011
 
Perl 5.16 and beyond
Perl 5.16 and beyondPerl 5.16 and beyond
Perl 5.16 and beyond
 
OSCON 2011 - Perl 5.16 and beyond
OSCON 2011 - Perl 5.16 and beyondOSCON 2011 - Perl 5.16 and beyond
OSCON 2011 - Perl 5.16 and beyond
 
RT4 - The whole sordid story
RT4 - The whole sordid storyRT4 - The whole sordid story
RT4 - The whole sordid story
 
K-9 Mail for Android
K-9 Mail for AndroidK-9 Mail for Android
K-9 Mail for Android
 
Perl 5.12.0
Perl 5.12.0Perl 5.12.0
Perl 5.12.0
 
P2P Bug Tracking with SD
P2P Bug Tracking with SDP2P Bug Tracking with SD
P2P Bug Tracking with SD
 
SD - A peer to peer issue tracking system
SD - A peer to peer issue tracking systemSD - A peer to peer issue tracking system
SD - A peer to peer issue tracking system
 
Hacking your Kindle (OSCON Lightning Talk)
Hacking your Kindle (OSCON Lightning Talk)Hacking your Kindle (OSCON Lightning Talk)
Hacking your Kindle (OSCON Lightning Talk)
 
Beginning Kindle Hackery
Beginning Kindle HackeryBeginning Kindle Hackery
Beginning Kindle Hackery
 
SD, a P2P bug tracking system
SD, a P2P bug tracking systemSD, a P2P bug tracking system
SD, a P2P bug tracking system
 
Prophet - Beijing Perl Workshop
Prophet - Beijing Perl WorkshopProphet - Beijing Perl Workshop
Prophet - Beijing Perl Workshop
 
Beijing Perl Workshop 2008 Hiveminder Secret Sauce
Beijing Perl Workshop 2008 Hiveminder Secret SauceBeijing Perl Workshop 2008 Hiveminder Secret Sauce
Beijing Perl Workshop 2008 Hiveminder Secret Sauce
 
A brief introduction to RTIR
A brief introduction to RTIRA brief introduction to RTIR
A brief introduction to RTIR
 
An introduction to RTIR
An introduction to RTIRAn introduction to RTIR
An introduction to RTIR
 
Prophet: a path out of the Cloud
Prophet: a path out of the CloudProphet: a path out of the Cloud
Prophet: a path out of the Cloud
 
Hiveminder - Everything but the Secret Sauce
Hiveminder - Everything but the Secret SauceHiveminder - Everything but the Secret Sauce
Hiveminder - Everything but the Secret Sauce
 

Recently uploaded

Monitoring and Managing Anomaly Detection on OpenShift.pdf
Monitoring and Managing Anomaly Detection on OpenShift.pdfMonitoring and Managing Anomaly Detection on OpenShift.pdf
Monitoring and Managing Anomaly Detection on OpenShift.pdf
Tosin Akinosho
 
How to use Firebase Data Connect For Flutter
How to use Firebase Data Connect For FlutterHow to use Firebase Data Connect For Flutter
How to use Firebase Data Connect For Flutter
Daiki Mogmet Ito
 
AI 101: An Introduction to the Basics and Impact of Artificial Intelligence
AI 101: An Introduction to the Basics and Impact of Artificial IntelligenceAI 101: An Introduction to the Basics and Impact of Artificial Intelligence
AI 101: An Introduction to the Basics and Impact of Artificial Intelligence
IndexBug
 
June Patch Tuesday
June Patch TuesdayJune Patch Tuesday
June Patch Tuesday
Ivanti
 
Artificial Intelligence for XMLDevelopment
Artificial Intelligence for XMLDevelopmentArtificial Intelligence for XMLDevelopment
Artificial Intelligence for XMLDevelopment
Octavian Nadolu
 
Your One-Stop Shop for Python Success: Top 10 US Python Development Providers
Your One-Stop Shop for Python Success: Top 10 US Python Development ProvidersYour One-Stop Shop for Python Success: Top 10 US Python Development Providers
Your One-Stop Shop for Python Success: Top 10 US Python Development Providers
akankshawande
 
みなさんこんにちはこれ何文字まで入るの?40文字以下不可とか本当に意味わからないけどこれ限界文字数書いてないからマジでやばい文字数いけるんじゃないの?えこ...
みなさんこんにちはこれ何文字まで入るの?40文字以下不可とか本当に意味わからないけどこれ限界文字数書いてないからマジでやばい文字数いけるんじゃないの?えこ...みなさんこんにちはこれ何文字まで入るの?40文字以下不可とか本当に意味わからないけどこれ限界文字数書いてないからマジでやばい文字数いけるんじゃないの?えこ...
みなさんこんにちはこれ何文字まで入るの?40文字以下不可とか本当に意味わからないけどこれ限界文字数書いてないからマジでやばい文字数いけるんじゃないの?えこ...
名前 です男
 
Skybuffer SAM4U tool for SAP license adoption
Skybuffer SAM4U tool for SAP license adoptionSkybuffer SAM4U tool for SAP license adoption
Skybuffer SAM4U tool for SAP license adoption
Tatiana Kojar
 
Energy Efficient Video Encoding for Cloud and Edge Computing Instances
Energy Efficient Video Encoding for Cloud and Edge Computing InstancesEnergy Efficient Video Encoding for Cloud and Edge Computing Instances
Energy Efficient Video Encoding for Cloud and Edge Computing Instances
Alpen-Adria-Universität
 
Deep Dive: AI-Powered Marketing to Get More Leads and Customers with HyperGro...
Deep Dive: AI-Powered Marketing to Get More Leads and Customers with HyperGro...Deep Dive: AI-Powered Marketing to Get More Leads and Customers with HyperGro...
Deep Dive: AI-Powered Marketing to Get More Leads and Customers with HyperGro...
saastr
 
Salesforce Integration for Bonterra Impact Management (fka Social Solutions A...
Salesforce Integration for Bonterra Impact Management (fka Social Solutions A...Salesforce Integration for Bonterra Impact Management (fka Social Solutions A...
Salesforce Integration for Bonterra Impact Management (fka Social Solutions A...
Jeffrey Haguewood
 
HCL Notes und Domino Lizenzkostenreduzierung in der Welt von DLAU
HCL Notes und Domino Lizenzkostenreduzierung in der Welt von DLAUHCL Notes und Domino Lizenzkostenreduzierung in der Welt von DLAU
HCL Notes und Domino Lizenzkostenreduzierung in der Welt von DLAU
panagenda
 
How to Get CNIC Information System with Paksim Ga.pptx
How to Get CNIC Information System with Paksim Ga.pptxHow to Get CNIC Information System with Paksim Ga.pptx
How to Get CNIC Information System with Paksim Ga.pptx
danishmna97
 
Choosing The Best AWS Service For Your Website + API.pptx
Choosing The Best AWS Service For Your Website + API.pptxChoosing The Best AWS Service For Your Website + API.pptx
Choosing The Best AWS Service For Your Website + API.pptx
Brandon Minnick, MBA
 
TrustArc Webinar - 2024 Global Privacy Survey
TrustArc Webinar - 2024 Global Privacy SurveyTrustArc Webinar - 2024 Global Privacy Survey
TrustArc Webinar - 2024 Global Privacy Survey
TrustArc
 
Taking AI to the Next Level in Manufacturing.pdf
Taking AI to the Next Level in Manufacturing.pdfTaking AI to the Next Level in Manufacturing.pdf
Taking AI to the Next Level in Manufacturing.pdf
ssuserfac0301
 
How to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdf
How to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdfHow to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdf
How to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdf
Chart Kalyan
 
Project Management Semester Long Project - Acuity
Project Management Semester Long Project - AcuityProject Management Semester Long Project - Acuity
Project Management Semester Long Project - Acuity
jpupo2018
 
Driving Business Innovation: Latest Generative AI Advancements & Success Story
Driving Business Innovation: Latest Generative AI Advancements & Success StoryDriving Business Innovation: Latest Generative AI Advancements & Success Story
Driving Business Innovation: Latest Generative AI Advancements & Success Story
Safe Software
 
Unlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdf
Unlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdfUnlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdf
Unlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdf
Malak Abu Hammad
 

Recently uploaded (20)

Monitoring and Managing Anomaly Detection on OpenShift.pdf
Monitoring and Managing Anomaly Detection on OpenShift.pdfMonitoring and Managing Anomaly Detection on OpenShift.pdf
Monitoring and Managing Anomaly Detection on OpenShift.pdf
 
How to use Firebase Data Connect For Flutter
How to use Firebase Data Connect For FlutterHow to use Firebase Data Connect For Flutter
How to use Firebase Data Connect For Flutter
 
AI 101: An Introduction to the Basics and Impact of Artificial Intelligence
AI 101: An Introduction to the Basics and Impact of Artificial IntelligenceAI 101: An Introduction to the Basics and Impact of Artificial Intelligence
AI 101: An Introduction to the Basics and Impact of Artificial Intelligence
 
June Patch Tuesday
June Patch TuesdayJune Patch Tuesday
June Patch Tuesday
 
Artificial Intelligence for XMLDevelopment
Artificial Intelligence for XMLDevelopmentArtificial Intelligence for XMLDevelopment
Artificial Intelligence for XMLDevelopment
 
Your One-Stop Shop for Python Success: Top 10 US Python Development Providers
Your One-Stop Shop for Python Success: Top 10 US Python Development ProvidersYour One-Stop Shop for Python Success: Top 10 US Python Development Providers
Your One-Stop Shop for Python Success: Top 10 US Python Development Providers
 
みなさんこんにちはこれ何文字まで入るの?40文字以下不可とか本当に意味わからないけどこれ限界文字数書いてないからマジでやばい文字数いけるんじゃないの?えこ...
みなさんこんにちはこれ何文字まで入るの?40文字以下不可とか本当に意味わからないけどこれ限界文字数書いてないからマジでやばい文字数いけるんじゃないの?えこ...みなさんこんにちはこれ何文字まで入るの?40文字以下不可とか本当に意味わからないけどこれ限界文字数書いてないからマジでやばい文字数いけるんじゃないの?えこ...
みなさんこんにちはこれ何文字まで入るの?40文字以下不可とか本当に意味わからないけどこれ限界文字数書いてないからマジでやばい文字数いけるんじゃないの?えこ...
 
Skybuffer SAM4U tool for SAP license adoption
Skybuffer SAM4U tool for SAP license adoptionSkybuffer SAM4U tool for SAP license adoption
Skybuffer SAM4U tool for SAP license adoption
 
Energy Efficient Video Encoding for Cloud and Edge Computing Instances
Energy Efficient Video Encoding for Cloud and Edge Computing InstancesEnergy Efficient Video Encoding for Cloud and Edge Computing Instances
Energy Efficient Video Encoding for Cloud and Edge Computing Instances
 
Deep Dive: AI-Powered Marketing to Get More Leads and Customers with HyperGro...
Deep Dive: AI-Powered Marketing to Get More Leads and Customers with HyperGro...Deep Dive: AI-Powered Marketing to Get More Leads and Customers with HyperGro...
Deep Dive: AI-Powered Marketing to Get More Leads and Customers with HyperGro...
 
Salesforce Integration for Bonterra Impact Management (fka Social Solutions A...
Salesforce Integration for Bonterra Impact Management (fka Social Solutions A...Salesforce Integration for Bonterra Impact Management (fka Social Solutions A...
Salesforce Integration for Bonterra Impact Management (fka Social Solutions A...
 
HCL Notes und Domino Lizenzkostenreduzierung in der Welt von DLAU
HCL Notes und Domino Lizenzkostenreduzierung in der Welt von DLAUHCL Notes und Domino Lizenzkostenreduzierung in der Welt von DLAU
HCL Notes und Domino Lizenzkostenreduzierung in der Welt von DLAU
 
How to Get CNIC Information System with Paksim Ga.pptx
How to Get CNIC Information System with Paksim Ga.pptxHow to Get CNIC Information System with Paksim Ga.pptx
How to Get CNIC Information System with Paksim Ga.pptx
 
Choosing The Best AWS Service For Your Website + API.pptx
Choosing The Best AWS Service For Your Website + API.pptxChoosing The Best AWS Service For Your Website + API.pptx
Choosing The Best AWS Service For Your Website + API.pptx
 
TrustArc Webinar - 2024 Global Privacy Survey
TrustArc Webinar - 2024 Global Privacy SurveyTrustArc Webinar - 2024 Global Privacy Survey
TrustArc Webinar - 2024 Global Privacy Survey
 
Taking AI to the Next Level in Manufacturing.pdf
Taking AI to the Next Level in Manufacturing.pdfTaking AI to the Next Level in Manufacturing.pdf
Taking AI to the Next Level in Manufacturing.pdf
 
How to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdf
How to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdfHow to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdf
How to Interpret Trends in the Kalyan Rajdhani Mix Chart.pdf
 
Project Management Semester Long Project - Acuity
Project Management Semester Long Project - AcuityProject Management Semester Long Project - Acuity
Project Management Semester Long Project - Acuity
 
Driving Business Innovation: Latest Generative AI Advancements & Success Story
Driving Business Innovation: Latest Generative AI Advancements & Success StoryDriving Business Innovation: Latest Generative AI Advancements & Success Story
Driving Business Innovation: Latest Generative AI Advancements & Success Story
 
Unlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdf
Unlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdfUnlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdf
Unlock the Future of Search with MongoDB Atlas_ Vector Search Unleashed.pdf
 

Prophet - A peer to peer replicated disconnected database

  • 1. Step 3: Prophet jesse@bestpractical.com clkao@bestpractical.com Friday, May 16, 2008 1
  • 2. Cloud computing is Sharecropping. ( ) Friday, May 16, 2008 2
  • 5. We work together Friday, May 16, 2008 5
  • 6. CL lives in Taipei Jesse lives in Boston Friday, May 16, 2008 6
  • 7. Sometimes we need to work face to face Friday, May 16, 2008 7
  • 8. TPE~BOS: 9410 mi TPE-HNL: 5,095 mi BOS-HNL: 5,069 mi Friday, May 16, 2008 8
  • 9. Our Plan Step 1: Go to Hawaii for “work” Step 2: ??? Step 3: Prophet! Friday, May 16, 2008 9
  • 10. The Plan Backfired We were there for 8 days We wrote 8000 lines of Perl We figured out step 2 Friday, May 16, 2008 10
  • 11. Step 2: Build a Disconnected Syncable Database Friday, May 16, 2008 11
  • 13. Prophet A semirelational, peer to peer replicated, disconnected, versioned, property database with self-healing conflict resolution Friday, May 16, 2008 13
  • 14. Bribery! Two tshirts. If you write a (real) patch for Prophet during this talk, you get a tshirt. Send patches to jesse@bestpractical.com Friday, May 16, 2008 14
  • 15. Getting Prophet Prophet http://code.bestpractical.com/bps-public/Prophet/ SD http://code.bestpractical.com/bps-public/sd/ Friday, May 16, 2008 15
  • 16. What do all those buzzwords mean? Friday, May 16, 2008 16
  • 17. semirelational Joins are expensive No backend Join support (yet) Friday, May 16, 2008 17
  • 18. peer-to-peer replicated Update any replica Pull from any replica Push to any replica Publish a replica Changes will propagate Friday, May 16, 2008 18
  • 19. disconnected Real-time replication is hard to scale It only works in the cloud I don't live in the cloud I want my data when I'm offline Prophet sync can happen whenever Friday, May 16, 2008 19
  • 20. versioned Compare a record to any point in the past All changes fully logged Undo changes Use history to be smart Friday, May 16, 2008 20
  • 21. property database Atomic operations CREATE, READ, UPDATE, DELETE, SEARCH Record types can have optional validation and canonicalization Records of the same type do not need to have the same properties Add and remove properties at will Friday, May 16, 2008 21
  • 22. self-healing conflict resolution Remembers all conflict resolutions Syncs all resolutions with your peers Detects identical conflicts Uses your peers’ resolutions to “vote” for the winner of a conflict Friday, May 16, 2008 22
  • 23. What could you build with Prophet? Friday, May 16, 2008 23
  • 24. sd A bug tracker: “simple defects” • id. Status, Summary • History • Comments • Attachments Friday, May 16, 2008 24
  • 25. ./bin/sd ticket create --summary quot;Can't sync sd with Google Codequot; --status new Created ticket 93BF979E-08C1-11DD-94C3-D4B1FCEE7EC4 Friday, May 16, 2008 25
  • 26. ./bin/sd ticket search --regex . 93BF979E-08C1-11DD-94C3-D4B1FCEE7EC4 Can't sync sd with Google Code new Friday, May 16, 2008 26
  • 27. ./bin/sd ticket update --uuid 93BF979E-08C1-11DD-94C3-D4B1FCEE7EC4 --status resolved ./bin/sd ticket search --regex . 93BF979E-08C1-11DD-94C3-D4B1FCEE7EC4 Can't sync sd with Google Code resolved Friday, May 16, 2008 27
  • 28. Bugs on my laptop aren’t interesting. Friday, May 16, 2008 28
  • 29. Syncable! Jesse ./bin/sd publish /tmp/mybugs scp -rvp /tmp/mybugs my.com:public_html/sd CL ./bin/sd pull prophet:http://my.com/~jesse/sd Friday, May 16, 2008 29
  • 30. My project has a bug tracker Friday, May 16, 2008 30
  • 31. My project has a bug tracker Actually, mine use two: • RT • hiveminder.com Friday, May 16, 2008 30
  • 32. Foreign Replicas Prophet makes Foriegn Replicas easy SD gets them quot;for freequot; Friday, May 16, 2008 31
  • 33. Wrote an RT Replica for SD (Using only the public REST API) It took an afternoon Mirror an RT instance into SD Share it with your peers using prophet Sync changes back from your peers to RT Supports Comments and Attachments Friday, May 16, 2008 32
  • 34. ...and one for Hiveminder (Using only the public REST API) Friday, May 16, 2008 33
  • 35. I can sync my bugs with RT or Hiveminder Friday, May 16, 2008 34
  • 37. I can sync between RT and Hiveminder Friday, May 16, 2008 36
  • 38. I can sync between two different RTs, too Friday, May 16, 2008 37
  • 39. We need more replica definitions: • • Trac GNATS • • Google Code What else? • SourceForge • Bugzilla • Jira • GForge • debbugs Friday, May 16, 2008 38
  • 40. What can you use Prophet for? Friday, May 16, 2008 39
  • 41. All your “little” databases Friday, May 16, 2008 40
  • 42. All the databases you want while offline. • • CRM Blog • Bug tracking • Trading Card Database • Sales orders • Ideas? • Phone book Friday, May 16, 2008 41
  • 43. “Private” Social Networks How about a P2P BBS? Prophet doesn’t need a server. You can sync over sneakernet. Friday, May 16, 2008 42
  • 44. A look inside Prophet Friday, May 16, 2008 43
  • 45. Terms and Concepts • • Database Property Change • • Database Replica Conflict • • “Foreign” Replica Resolution • • Record Nullifcation Change Set • • Record Type Resolution Change Set • • Change Set Merge • • Change Merge Ticket Friday, May 16, 2008 44
  • 47. The Record Store Stores indivdual records by type Friday, May 16, 2008 46
  • 48. The Changeset Store Stores each atomic change to a set of records Replaying all changesets will create an exact clone of the replica Friday, May 16, 2008 47
  • 50. Subversion Slow Steady Robust Supports remote sync Requires Subversion Perl Bindings Friday, May 16, 2008 49
  • 51. Filesystem Readable Flat files Compact Fast (Not yet fully atomic) Friday, May 16, 2008 50
  • 52. HTTP Designed to let you “publish” databases Flat-files, Currently read-only. Same format as the filesystem replica type. Friday, May 16, 2008 51
  • 53. “Foreign” replicas Will usually be app specific All current examples are in SD Friday, May 16, 2008 52
  • 55. Publish Serialize and export all of a replica's resolutions and changesets Friday, May 16, 2008 54
  • 56. Pull Integrate unseen resolutions and then unseen changesets from a replica Friday, May 16, 2008 55
  • 57. Push Integrate new resolutions and changesets into a replica Friday, May 16, 2008 56
  • 58. Resolving Conflicts Figures out the best resolution “Nullifies” the conflict so the changeset can be cleanly integrated Integrates the conflicting changeset Records the resolution as a new changeset Records the resolution decision in the resolution database Friday, May 16, 2008 57
  • 59. “The Best Resolution” Prophet has clever ways to figure out the best resolution. If there are previous resolutions for the same conflict and a majority agree, use that If the merger has specified a “prefer this side” choice, use that Prompt the user to make a decision, giving them info about previous decisions for this conflict Friday, May 16, 2008 58
  • 60. We don't have a proof for the algorithm yet We do have dozens of runs of randomized testing. So far, it always stabilizes sanely. Friday, May 16, 2008 59
  • 61. How does it scale? Vertical scale is boring Designed to scale to many peers You are not Google Current target is databases of O(50k) records Friday, May 16, 2008 60
  • 62. How does it scale? Vertical scale is boring Designed to scale to many peers You are not Google Does anyone here work for Google? Current target is databases of O(50k) records Friday, May 16, 2008 60
  • 63. Why not, then? We just have a political agenda. Web 2.0 is not Open Source. Your data shouldn’t be ‘exportable’ from the cloud. You should always have full control. So we dont need to store 10 billion records in one database. Friday, May 16, 2008 61
  • 64. (Do you have 10 billion bugs, customer contacts or sales orders?) Friday, May 16, 2008 62
  • 65. That said, we'd love to see a scalable, high performance prophet replica store Friday, May 16, 2008 63
  • 66. Project Status Simple, well-defined Perl API RESTy web API (with microserver) Fast, lightweight backend Small, active dev community Great test coverage Horrible POD coverage Friday, May 16, 2008 64
  • 67. Our Plans Improved search and indexing (Including full-text indexing) Query language Proper security model Jifty, Catalyst, Rails models? Friday, May 16, 2008 65
  • 68. Prophet is very young Many, many hours of design About 10 days of two hackers hacking Friday, May 16, 2008 66
  • 69. Codebase Prophet 5479 lines of code and doc 1693 lines of tests sd 1695 lines of code and doc 876 lines of tests Friday, May 16, 2008 67
  • 70. Getting Prophet Prophet http://code.bestpractical.com/bps-public/Prophet/ SD http://code.bestpractical.com/bps-public/sd/ Friday, May 16, 2008 68
  • 71. Getting Involved prophet-subscribe@lists.bestpractical.com #prophet on freenode IRC Friday, May 16, 2008 69