SlideShare a Scribd company logo
1 of 47
Download to read offline
unba.se
Distributed Database for Human Interaction
Daniel Norman
CTO, güdTECH
unba.se contributor
Twitter: @DreamingInCode
AMW43Asilomar Microcomputer Workshop 2017
unba.se
crates.io/crates/unbase
gitter.im/unbase
Notes
● Much science remains to be done
● Sweet demo - 404
✅ SHIP IT!
Synopticon
What I’ll cover during this talk
● What is unbase?
● Yegads, Why?
● Motivating suppositions
● How does it work?
● Will it have the desired properties?
What?
What is Unbase?
● Brand new open source project
● Peer-to-peer “database”
● Distributed immutable Merkle DAG
● Focus on data locality
● All nodes are equal
● Strong causal consistency model
● Written in Rust!
(intended for FFI, wasm, javascript transpilation)
What?
What isn’t Unbase?
● No Servers Just bigger/smaller nodes
● Not blockchain at least not how we tend to think of blockchain
● No DHT Graph traversal > logical addressing
● No a priori resource planning Neither human nor hash bucket
● No stored state* *on demand projection of stored operations
● No wallclock FFS it’s 2017
● Not remotely production-ready Super pre-alpha
What?
Unbase Goals
● Represent human intent by storing operations, not resultant state
● Unified consistency model across heterogeneous environments
● Unified framework for building fancy data types (CRDT, etc)
● Reactive / Observable everything
● High throughput / low latency between local peers
● Increase availability / reliability
● Less “Weird” behavior
● Reduced cognitive load on the developer
What?
The Roadmap
● Integral content-filtered pubsub Your database is a message queue
● User definable type system User defined CRDT / OT types
● Stored procedures and triggers All your business logic are belong to us
● Policy-authority managed schemata Crypto-signed DDL
● Alpha Release For experimental applications
● Proper Science Does it do what it says on the tin?
● ??? Your guess is as good as mine
● Profit Truly serverless infrastructure smashes the plutocracy
What?
Who else is in this space?
● Google Spanner
● Interplanetary Filesystem https://ipfs.io
● Etherium https://www.ethereum.org/
● AntidoteDB http://syncfree.github.io/antidote/
● Scuttlebutt https://www.scuttlebutt.nz
● Akka http://akka.io/
And many many more...
...but none of these do quite what I’m looking for.
Yegads, Why?!
(TLDR; physics, masochism)
Yegads, Why?
Origin Story
Ye Olde Web Stack:
● Mysql / Mongo
● RabbitMQ
● ElasticSearch / OLAP
● Microservices / Nginx / HAproxy
Consistency models in your application
Yegads, Why?
Origin Story
When we scale Ye Olde web stack?
● Sharding of DB, MQ, Search, Analytics
● Geo-replication / CAP
● Communication data races
● Developers create ad-hoc overlay consistency models
● Lots of time spent cobbling
● High cognitive load
Yegads, Why?
Some Examples
● Total internet outage Officemates can’t collaborate?
● LieFi Slack + elevator =
● Application state gets “weird” Mental system-model is wrong
● Replication failure Rube Goldberg state-machine
Yegads, Why?
“Observe locally, act globally”
Fig III. State machine for plugging leaky abstractions
Yegads, Why?
We’ve tried lots of things
● On-prem systems
● Multi-mastering
● Geo-sharding
● Eventual consistency
● Simulated Simultaneity ( global wallclock )
Yegads, Why?
But most require travel
Distance between you and
the stuff you probably care
about:
Much much less
Distance between you and the
arbiter of linearization:
Usually thousands of Km
(Stuff you care about)
Collaborating with Bob &
Alice down the hall
You
Messages/docs you authored
Messages/docs you read
Your wristwatch
Your IOT devices
Next door neighbors
The linearization you must visit
Backhoes
Congestion
Light travel time
Alligators
Net non-neutrality
BGP Screwups
Power outages
State Hacking
Spanning tree errors
Tripped-over cables
Cat on server
Submarine cable break
Settlement-free
peering disputes
State Surveillance
Packet corruption
Lie-FiUnder Provisioned hardware
Cheap electrolytic capacitors
Thousands of Km of:
DNS errors
Saturated cell-tower backhaul
RF reflections
Interference from microwave oven
Cosmic Rays
Disgruntled employees
DDOS Attacks
Rat-chewed cables
Corrosion Late internet payments
Gravitational time-dilation
Doppler effect
F*#KING BOINGO
Failed B-side Power
Core router problem
F*#KING TMOBILE
F*#KING NETGEAR
Cabinet switch failure
Misconfigured health check
NAT Misconfiguration
Load balancer failure
NTP failures
Yegads, Why?
Mythology
● Networks can be made reliable
● A single arbiter of state makes a consistency-model “strong”
● Objective state exists
● Objective simultaneity exists
It would seem that our foundational abstractions are leaky.
Yegads, Why?
Avoidance Behaviors
● It’s working for me. Plausible deniability is essential
● That’s unsupported ”You’re holding it wrong”
● Open more reqs for the ops team ”Cobble harder minions!”
● It’s Amazon’s problem! The blame game
● Akka solves it! Event-sourced locality is magical locality
● NuoDB/Spanner solves it! Proprietary magic, now with unicorns!
Loading Slide...
(waiting is incompatible with human causal expectations)
Yegads, Why?
Expectations
Yegads, Why?
What do humans expect?
I set my glass on the table ≻ It’s there when I pick it up.
Edge computing IRL.
Local, coordination-free consistency.
No need to visit Ashburn.
Yegads, Why?
This problem isn’t going away
● Collaborative systems are exploding OT or GTFO
● System complexity becoming unmanageable How much must twitter spend?
● Bandwidth reqs will exponentiate, backbone will not A hololens on every face
● Outages worsen as safety/mission critical uses proliferate Fedex global API is down again?
● Single threaded execution won’t last Sorry node.js
● Stagnant network service offerings Dark fiber revolution my ass
● State is an illusion, sort of...
● Computation and storage must occur nearer the edge
● A priori resource planning is undesirable
● Durability is non-boolean
● Distributed causal consistency is possible, efficient, desirable
● Leaky abstractions are expensive to cobble together
Synopticon
Motivating Suppositions
Motivating suppositions
State is an illusion, sort of
● A projection of observed events Essentially no such thing as shared state
● “State” is arguably an abstraction Not unlike “solid matter” vs coulomb force
● Some abstractions we need not question ”Why don’t I fall through the floor?”
● Collaborative systems* warrant it ALL multi-user systems are collaborative
* All multi user systems are collaborative systems
Philosophy Corner:
How does reality
differ from causality?
Motivating suppositions
Durability is not a boolean
● All data has a nonzero probability of loss
● We care more about some data than other data
● Single-copy durability is not high-durability
Motivating suppositions
Location, Location, Location
● Humans expect physical artifacts Proximity the only way to meet this expectation
● Travel to arbiter unacceptable default assumption Reality has no centralized arbiters, nor should we
● Data should be localized by origin and utility Geo-sharding and hash-sharding a poor solution
● A priori shard planning is expensive It’s expensive, and you’ll mess it up*
*Automated a priori planning only marginally better than humans
When you squeeze a balloon…
Motivating suppositions
Behaviors > Planning
How does it work?
Synopticon
How does it work?
● Distributed Merkle-style DAG The easy part
● Sparse vector clocks Similar to Interval Tree Clocks. Baquero et al
● Selective-hearing gossip Essential for traffic reduction
● Commutative index merging Permits eventual convergence
● Infectious knowledge Notice when insufficiently converged
● “Gravity” and “pressure” Bring important data near, push away the rest
How does it work?
Review
● Immutable Merkle-style DAG The easy part
● Sparse vector clocks Vector clock compression scheme
● Selective-hearing gossip Essential for traffic reduction
● Commutative index merging Permits eventual convergence
● Infectious knowledge Notice when insufficiently converged
● Gravity and pressure Bring important data near, push away the rest
Will it have the desired properties?
Will it:
● Be comparable with RDBMS performance?
● Meet its consistency-model invariants?
● Scale as envisioned?
● Provide an ergonomic interface for developers?
● Make users happy?
NFI - Only experimental data will bear this out!
With unbase, we hope to:
● Align system performance with user’s causal expectations
● Increase availability (for data I care about)
● Reduce system complexity
● Save time and money on system operations
● Build in collaboration-awareness from the start
Synopticon
Conclusion
Thank You!
Daniel Norman
CTO, güdTECH
unba.se contributor
Twitter: @DreamingInCode
AMW43Asilomar Microcomputer Workshop 2017
unba.se
crates.io/crates/unbase
gitter.im/unbase
#unbase at irc.mozilla.orgunba.se/design/bibliography.html
Do you like rust and believe this crazy thing is possible?
I need your help! Join me!
Bonus slides
● What is consistency? Serializability usually just a hack for causality
● What is Availability? Declared based on an observational standard
● What is a Partition? Only exists according to an observational standard
(With sufficiently tortured definitions, one can prove a ham sandwich.)
See Martin Kleppmann’s excellent paper: A critique of the CAP theorem – arxiv.org/abs/1509.05393
Obligatory CAP Slide
Problems with CAP Theorem

More Related Content

Similar to AMW43 - Unba.se, Distributed database for human interaction

Your Testing Is Flawed: Introducing A New Open Source Tool For Accurate Kuber...
Your Testing Is Flawed: Introducing A New Open Source Tool For Accurate Kuber...Your Testing Is Flawed: Introducing A New Open Source Tool For Accurate Kuber...
Your Testing Is Flawed: Introducing A New Open Source Tool For Accurate Kuber...StormForge .io
 
Why Node, Express and Postgres - presented 23 Feb 15, Talkjs, Microsoft Audit...
Why Node, Express and Postgres - presented 23 Feb 15, Talkjs, Microsoft Audit...Why Node, Express and Postgres - presented 23 Feb 15, Talkjs, Microsoft Audit...
Why Node, Express and Postgres - presented 23 Feb 15, Talkjs, Microsoft Audit...Calvin Tan
 
Voxxed Athens 2018 - Methods and Practices for Guaranteed Failure in Big Data
Voxxed Athens 2018 - Methods and Practices for Guaranteed Failure in Big DataVoxxed Athens 2018 - Methods and Practices for Guaranteed Failure in Big Data
Voxxed Athens 2018 - Methods and Practices for Guaranteed Failure in Big DataVoxxed Athens
 
Quick dive into the big data pool without drowning - Demi Ben-Ari @ Panorays
Quick dive into the big data pool without drowning - Demi Ben-Ari @ PanoraysQuick dive into the big data pool without drowning - Demi Ben-Ari @ Panorays
Quick dive into the big data pool without drowning - Demi Ben-Ari @ PanoraysDemi Ben-Ari
 
JavaScript for Enterprise Applications
JavaScript for Enterprise ApplicationsJavaScript for Enterprise Applications
JavaScript for Enterprise ApplicationsPiyush Katariya
 
kranonit S06E01 Игорь Цинько: High load
kranonit S06E01 Игорь Цинько: High loadkranonit S06E01 Игорь Цинько: High load
kranonit S06E01 Игорь Цинько: High loadKrivoy Rog IT Community
 
The Professional Programmer
The Professional ProgrammerThe Professional Programmer
The Professional ProgrammerDave Cross
 
Advanced Administration, Monitoring and Backup
Advanced Administration, Monitoring and BackupAdvanced Administration, Monitoring and Backup
Advanced Administration, Monitoring and BackupMongoDB
 
#OSSPARIS19 - How to improve database observability - CHARLES JUDITH, Criteo
#OSSPARIS19 - How to improve database observability - CHARLES JUDITH, Criteo#OSSPARIS19 - How to improve database observability - CHARLES JUDITH, Criteo
#OSSPARIS19 - How to improve database observability - CHARLES JUDITH, CriteoParis Open Source Summit
 
MongoDB World 2019: Packing Up Your Data and Moving to MongoDB Atlas
MongoDB World 2019: Packing Up Your Data and Moving to MongoDB AtlasMongoDB World 2019: Packing Up Your Data and Moving to MongoDB Atlas
MongoDB World 2019: Packing Up Your Data and Moving to MongoDB AtlasMongoDB
 
Transversal Delivery Pipeline by Mike Nescot and Nick Grace
Transversal Delivery Pipeline by Mike Nescot and Nick GraceTransversal Delivery Pipeline by Mike Nescot and Nick Grace
Transversal Delivery Pipeline by Mike Nescot and Nick GraceDevOpsDays Baltimore
 
Transversal Delivery Pipeline by Mike Nescot and Nick Grace
Transversal Delivery Pipeline by Mike Nescot and Nick GraceTransversal Delivery Pipeline by Mike Nescot and Nick Grace
Transversal Delivery Pipeline by Mike Nescot and Nick GraceDevOpsDays Baltimore
 
Big Data in 200 km/h | AWS Big Data Demystified #1.3
Big Data in 200 km/h | AWS Big Data Demystified #1.3  Big Data in 200 km/h | AWS Big Data Demystified #1.3
Big Data in 200 km/h | AWS Big Data Demystified #1.3 Omid Vahdaty
 
AWS Big Data Demystified #1.2 | Big Data architecture lessons learned
AWS Big Data Demystified #1.2 | Big Data architecture lessons learned AWS Big Data Demystified #1.2 | Big Data architecture lessons learned
AWS Big Data Demystified #1.2 | Big Data architecture lessons learned Omid Vahdaty
 
BISSA: Empowering Web gadget Communication with Tuple Spaces
BISSA: Empowering Web gadget Communication with Tuple SpacesBISSA: Empowering Web gadget Communication with Tuple Spaces
BISSA: Empowering Web gadget Communication with Tuple SpacesSrinath Perera
 
Software architecture, Patterns for Scale
Software architecture, Patterns for ScaleSoftware architecture, Patterns for Scale
Software architecture, Patterns for ScaleiGbanam
 
How to get started with Site Reliability Engineering
How to get started with Site Reliability EngineeringHow to get started with Site Reliability Engineering
How to get started with Site Reliability EngineeringAndrew Kirkpatrick
 
Enterprise Frameworks: Java & .NET
Enterprise Frameworks: Java & .NETEnterprise Frameworks: Java & .NET
Enterprise Frameworks: Java & .NETAnant Corporation
 

Similar to AMW43 - Unba.se, Distributed database for human interaction (20)

Your Testing Is Flawed: Introducing A New Open Source Tool For Accurate Kuber...
Your Testing Is Flawed: Introducing A New Open Source Tool For Accurate Kuber...Your Testing Is Flawed: Introducing A New Open Source Tool For Accurate Kuber...
Your Testing Is Flawed: Introducing A New Open Source Tool For Accurate Kuber...
 
Why Node, Express and Postgres - presented 23 Feb 15, Talkjs, Microsoft Audit...
Why Node, Express and Postgres - presented 23 Feb 15, Talkjs, Microsoft Audit...Why Node, Express and Postgres - presented 23 Feb 15, Talkjs, Microsoft Audit...
Why Node, Express and Postgres - presented 23 Feb 15, Talkjs, Microsoft Audit...
 
Voxxed Athens 2018 - Methods and Practices for Guaranteed Failure in Big Data
Voxxed Athens 2018 - Methods and Practices for Guaranteed Failure in Big DataVoxxed Athens 2018 - Methods and Practices for Guaranteed Failure in Big Data
Voxxed Athens 2018 - Methods and Practices for Guaranteed Failure in Big Data
 
Quick dive into the big data pool without drowning - Demi Ben-Ari @ Panorays
Quick dive into the big data pool without drowning - Demi Ben-Ari @ PanoraysQuick dive into the big data pool without drowning - Demi Ben-Ari @ Panorays
Quick dive into the big data pool without drowning - Demi Ben-Ari @ Panorays
 
JavaScript for Enterprise Applications
JavaScript for Enterprise ApplicationsJavaScript for Enterprise Applications
JavaScript for Enterprise Applications
 
kranonit S06E01 Игорь Цинько: High load
kranonit S06E01 Игорь Цинько: High loadkranonit S06E01 Игорь Цинько: High load
kranonit S06E01 Игорь Цинько: High load
 
The Professional Programmer
The Professional ProgrammerThe Professional Programmer
The Professional Programmer
 
Advanced Administration, Monitoring and Backup
Advanced Administration, Monitoring and BackupAdvanced Administration, Monitoring and Backup
Advanced Administration, Monitoring and Backup
 
#OSSPARIS19 - How to improve database observability - CHARLES JUDITH, Criteo
#OSSPARIS19 - How to improve database observability - CHARLES JUDITH, Criteo#OSSPARIS19 - How to improve database observability - CHARLES JUDITH, Criteo
#OSSPARIS19 - How to improve database observability - CHARLES JUDITH, Criteo
 
MongoDB World 2019: Packing Up Your Data and Moving to MongoDB Atlas
MongoDB World 2019: Packing Up Your Data and Moving to MongoDB AtlasMongoDB World 2019: Packing Up Your Data and Moving to MongoDB Atlas
MongoDB World 2019: Packing Up Your Data and Moving to MongoDB Atlas
 
DEVIEW 2013
DEVIEW 2013DEVIEW 2013
DEVIEW 2013
 
Transversal Delivery Pipeline by Mike Nescot and Nick Grace
Transversal Delivery Pipeline by Mike Nescot and Nick GraceTransversal Delivery Pipeline by Mike Nescot and Nick Grace
Transversal Delivery Pipeline by Mike Nescot and Nick Grace
 
Transversal Delivery Pipeline by Mike Nescot and Nick Grace
Transversal Delivery Pipeline by Mike Nescot and Nick GraceTransversal Delivery Pipeline by Mike Nescot and Nick Grace
Transversal Delivery Pipeline by Mike Nescot and Nick Grace
 
Big Data in 200 km/h | AWS Big Data Demystified #1.3
Big Data in 200 km/h | AWS Big Data Demystified #1.3  Big Data in 200 km/h | AWS Big Data Demystified #1.3
Big Data in 200 km/h | AWS Big Data Demystified #1.3
 
AWS Big Data Demystified #1.2 | Big Data architecture lessons learned
AWS Big Data Demystified #1.2 | Big Data architecture lessons learned AWS Big Data Demystified #1.2 | Big Data architecture lessons learned
AWS Big Data Demystified #1.2 | Big Data architecture lessons learned
 
BISSA: Empowering Web gadget Communication with Tuple Spaces
BISSA: Empowering Web gadget Communication with Tuple SpacesBISSA: Empowering Web gadget Communication with Tuple Spaces
BISSA: Empowering Web gadget Communication with Tuple Spaces
 
Software architecture, Patterns for Scale
Software architecture, Patterns for ScaleSoftware architecture, Patterns for Scale
Software architecture, Patterns for Scale
 
How to get started with Site Reliability Engineering
How to get started with Site Reliability EngineeringHow to get started with Site Reliability Engineering
How to get started with Site Reliability Engineering
 
Enterprise Frameworks: Java & .NET
Enterprise Frameworks: Java & .NETEnterprise Frameworks: Java & .NET
Enterprise Frameworks: Java & .NET
 
Data Science as Scale
Data Science as ScaleData Science as Scale
Data Science as Scale
 

Recently uploaded

Katarzyna Lipka-Sidor - BIM School Course
Katarzyna Lipka-Sidor - BIM School CourseKatarzyna Lipka-Sidor - BIM School Course
Katarzyna Lipka-Sidor - BIM School Coursebim.edu.pl
 
Comparative study of High-rise Building Using ETABS,SAP200 and SAFE., SAFE an...
Comparative study of High-rise Building Using ETABS,SAP200 and SAFE., SAFE an...Comparative study of High-rise Building Using ETABS,SAP200 and SAFE., SAFE an...
Comparative study of High-rise Building Using ETABS,SAP200 and SAFE., SAFE an...Erbil Polytechnic University
 
National Level Hackathon Participation Certificate.pdf
National Level Hackathon Participation Certificate.pdfNational Level Hackathon Participation Certificate.pdf
National Level Hackathon Participation Certificate.pdfRajuKanojiya4
 
Malware Detection By Machine Learning Presentation.pptx
Malware Detection By Machine Learning  Presentation.pptxMalware Detection By Machine Learning  Presentation.pptx
Malware Detection By Machine Learning Presentation.pptxalishapatidar2021
 
DEVICE DRIVERS AND INTERRUPTS SERVICE MECHANISM.pdf
DEVICE DRIVERS AND INTERRUPTS  SERVICE MECHANISM.pdfDEVICE DRIVERS AND INTERRUPTS  SERVICE MECHANISM.pdf
DEVICE DRIVERS AND INTERRUPTS SERVICE MECHANISM.pdfAkritiPradhan2
 
List of Accredited Concrete Batching Plant.pdf
List of Accredited Concrete Batching Plant.pdfList of Accredited Concrete Batching Plant.pdf
List of Accredited Concrete Batching Plant.pdfisabel213075
 
High Voltage Engineering- OVER VOLTAGES IN ELECTRICAL POWER SYSTEMS
High Voltage Engineering- OVER VOLTAGES IN ELECTRICAL POWER SYSTEMSHigh Voltage Engineering- OVER VOLTAGES IN ELECTRICAL POWER SYSTEMS
High Voltage Engineering- OVER VOLTAGES IN ELECTRICAL POWER SYSTEMSsandhya757531
 
Correctly Loading Incremental Data at Scale
Correctly Loading Incremental Data at ScaleCorrectly Loading Incremental Data at Scale
Correctly Loading Incremental Data at ScaleAlluxio, Inc.
 
"Exploring the Essential Functions and Design Considerations of Spillways in ...
"Exploring the Essential Functions and Design Considerations of Spillways in ..."Exploring the Essential Functions and Design Considerations of Spillways in ...
"Exploring the Essential Functions and Design Considerations of Spillways in ...Erbil Polytechnic University
 
Risk Management in Engineering Construction Project
Risk Management in Engineering Construction ProjectRisk Management in Engineering Construction Project
Risk Management in Engineering Construction ProjectErbil Polytechnic University
 
Input Output Management in Operating System
Input Output Management in Operating SystemInput Output Management in Operating System
Input Output Management in Operating SystemRashmi Bhat
 
chpater16.pptxMMMMMMMMMMMMMMMMMMMMMMMMMMM
chpater16.pptxMMMMMMMMMMMMMMMMMMMMMMMMMMMchpater16.pptxMMMMMMMMMMMMMMMMMMMMMMMMMMM
chpater16.pptxMMMMMMMMMMMMMMMMMMMMMMMMMMMNanaAgyeman13
 
Mine Environment II Lab_MI10448MI__________.pptx
Mine Environment II Lab_MI10448MI__________.pptxMine Environment II Lab_MI10448MI__________.pptx
Mine Environment II Lab_MI10448MI__________.pptxRomil Mishra
 
Artificial Intelligence in Power System overview
Artificial Intelligence in Power System overviewArtificial Intelligence in Power System overview
Artificial Intelligence in Power System overviewsandhya757531
 
Turn leadership mistakes into a better future.pptx
Turn leadership mistakes into a better future.pptxTurn leadership mistakes into a better future.pptx
Turn leadership mistakes into a better future.pptxStephen Sitton
 
Internship PPT ukai thermal power station .pptx
Internship PPT ukai thermal power station .pptxInternship PPT ukai thermal power station .pptx
Internship PPT ukai thermal power station .pptxmalikavita731
 
Energy Awareness training ppt for manufacturing process.pptx
Energy Awareness training ppt for manufacturing process.pptxEnergy Awareness training ppt for manufacturing process.pptx
Energy Awareness training ppt for manufacturing process.pptxsiddharthjain2303
 
Module-1-(Building Acoustics) Noise Control (Unit-3). pdf
Module-1-(Building Acoustics) Noise Control (Unit-3). pdfModule-1-(Building Acoustics) Noise Control (Unit-3). pdf
Module-1-(Building Acoustics) Noise Control (Unit-3). pdfManish Kumar
 
11. Properties of Liquid Fuels in Energy Engineering.pdf
11. Properties of Liquid Fuels in Energy Engineering.pdf11. Properties of Liquid Fuels in Energy Engineering.pdf
11. Properties of Liquid Fuels in Energy Engineering.pdfHafizMudaserAhmad
 

Recently uploaded (20)

Katarzyna Lipka-Sidor - BIM School Course
Katarzyna Lipka-Sidor - BIM School CourseKatarzyna Lipka-Sidor - BIM School Course
Katarzyna Lipka-Sidor - BIM School Course
 
Comparative study of High-rise Building Using ETABS,SAP200 and SAFE., SAFE an...
Comparative study of High-rise Building Using ETABS,SAP200 and SAFE., SAFE an...Comparative study of High-rise Building Using ETABS,SAP200 and SAFE., SAFE an...
Comparative study of High-rise Building Using ETABS,SAP200 and SAFE., SAFE an...
 
National Level Hackathon Participation Certificate.pdf
National Level Hackathon Participation Certificate.pdfNational Level Hackathon Participation Certificate.pdf
National Level Hackathon Participation Certificate.pdf
 
Malware Detection By Machine Learning Presentation.pptx
Malware Detection By Machine Learning  Presentation.pptxMalware Detection By Machine Learning  Presentation.pptx
Malware Detection By Machine Learning Presentation.pptx
 
DEVICE DRIVERS AND INTERRUPTS SERVICE MECHANISM.pdf
DEVICE DRIVERS AND INTERRUPTS  SERVICE MECHANISM.pdfDEVICE DRIVERS AND INTERRUPTS  SERVICE MECHANISM.pdf
DEVICE DRIVERS AND INTERRUPTS SERVICE MECHANISM.pdf
 
List of Accredited Concrete Batching Plant.pdf
List of Accredited Concrete Batching Plant.pdfList of Accredited Concrete Batching Plant.pdf
List of Accredited Concrete Batching Plant.pdf
 
High Voltage Engineering- OVER VOLTAGES IN ELECTRICAL POWER SYSTEMS
High Voltage Engineering- OVER VOLTAGES IN ELECTRICAL POWER SYSTEMSHigh Voltage Engineering- OVER VOLTAGES IN ELECTRICAL POWER SYSTEMS
High Voltage Engineering- OVER VOLTAGES IN ELECTRICAL POWER SYSTEMS
 
Correctly Loading Incremental Data at Scale
Correctly Loading Incremental Data at ScaleCorrectly Loading Incremental Data at Scale
Correctly Loading Incremental Data at Scale
 
"Exploring the Essential Functions and Design Considerations of Spillways in ...
"Exploring the Essential Functions and Design Considerations of Spillways in ..."Exploring the Essential Functions and Design Considerations of Spillways in ...
"Exploring the Essential Functions and Design Considerations of Spillways in ...
 
Risk Management in Engineering Construction Project
Risk Management in Engineering Construction ProjectRisk Management in Engineering Construction Project
Risk Management in Engineering Construction Project
 
Input Output Management in Operating System
Input Output Management in Operating SystemInput Output Management in Operating System
Input Output Management in Operating System
 
chpater16.pptxMMMMMMMMMMMMMMMMMMMMMMMMMMM
chpater16.pptxMMMMMMMMMMMMMMMMMMMMMMMMMMMchpater16.pptxMMMMMMMMMMMMMMMMMMMMMMMMMMM
chpater16.pptxMMMMMMMMMMMMMMMMMMMMMMMMMMM
 
Mine Environment II Lab_MI10448MI__________.pptx
Mine Environment II Lab_MI10448MI__________.pptxMine Environment II Lab_MI10448MI__________.pptx
Mine Environment II Lab_MI10448MI__________.pptx
 
Artificial Intelligence in Power System overview
Artificial Intelligence in Power System overviewArtificial Intelligence in Power System overview
Artificial Intelligence in Power System overview
 
Turn leadership mistakes into a better future.pptx
Turn leadership mistakes into a better future.pptxTurn leadership mistakes into a better future.pptx
Turn leadership mistakes into a better future.pptx
 
Internship PPT ukai thermal power station .pptx
Internship PPT ukai thermal power station .pptxInternship PPT ukai thermal power station .pptx
Internship PPT ukai thermal power station .pptx
 
Energy Awareness training ppt for manufacturing process.pptx
Energy Awareness training ppt for manufacturing process.pptxEnergy Awareness training ppt for manufacturing process.pptx
Energy Awareness training ppt for manufacturing process.pptx
 
Module-1-(Building Acoustics) Noise Control (Unit-3). pdf
Module-1-(Building Acoustics) Noise Control (Unit-3). pdfModule-1-(Building Acoustics) Noise Control (Unit-3). pdf
Module-1-(Building Acoustics) Noise Control (Unit-3). pdf
 
POWER SYSTEMS-1 Complete notes examples
POWER SYSTEMS-1 Complete notes  examplesPOWER SYSTEMS-1 Complete notes  examples
POWER SYSTEMS-1 Complete notes examples
 
11. Properties of Liquid Fuels in Energy Engineering.pdf
11. Properties of Liquid Fuels in Energy Engineering.pdf11. Properties of Liquid Fuels in Energy Engineering.pdf
11. Properties of Liquid Fuels in Energy Engineering.pdf
 

AMW43 - Unba.se, Distributed database for human interaction

  • 1. unba.se Distributed Database for Human Interaction Daniel Norman CTO, güdTECH unba.se contributor Twitter: @DreamingInCode AMW43Asilomar Microcomputer Workshop 2017 unba.se crates.io/crates/unbase gitter.im/unbase
  • 2. Notes ● Much science remains to be done ● Sweet demo - 404
  • 4. Synopticon What I’ll cover during this talk ● What is unbase? ● Yegads, Why? ● Motivating suppositions ● How does it work? ● Will it have the desired properties?
  • 5. What? What is Unbase? ● Brand new open source project ● Peer-to-peer “database” ● Distributed immutable Merkle DAG ● Focus on data locality ● All nodes are equal ● Strong causal consistency model ● Written in Rust! (intended for FFI, wasm, javascript transpilation)
  • 6. What? What isn’t Unbase? ● No Servers Just bigger/smaller nodes ● Not blockchain at least not how we tend to think of blockchain ● No DHT Graph traversal > logical addressing ● No a priori resource planning Neither human nor hash bucket ● No stored state* *on demand projection of stored operations ● No wallclock FFS it’s 2017 ● Not remotely production-ready Super pre-alpha
  • 7. What? Unbase Goals ● Represent human intent by storing operations, not resultant state ● Unified consistency model across heterogeneous environments ● Unified framework for building fancy data types (CRDT, etc) ● Reactive / Observable everything ● High throughput / low latency between local peers ● Increase availability / reliability ● Less “Weird” behavior ● Reduced cognitive load on the developer
  • 8. What? The Roadmap ● Integral content-filtered pubsub Your database is a message queue ● User definable type system User defined CRDT / OT types ● Stored procedures and triggers All your business logic are belong to us ● Policy-authority managed schemata Crypto-signed DDL ● Alpha Release For experimental applications ● Proper Science Does it do what it says on the tin? ● ??? Your guess is as good as mine ● Profit Truly serverless infrastructure smashes the plutocracy
  • 9. What? Who else is in this space? ● Google Spanner ● Interplanetary Filesystem https://ipfs.io ● Etherium https://www.ethereum.org/ ● AntidoteDB http://syncfree.github.io/antidote/ ● Scuttlebutt https://www.scuttlebutt.nz ● Akka http://akka.io/ And many many more...
  • 10. ...but none of these do quite what I’m looking for.
  • 12. Yegads, Why? Origin Story Ye Olde Web Stack: ● Mysql / Mongo ● RabbitMQ ● ElasticSearch / OLAP ● Microservices / Nginx / HAproxy
  • 13. Consistency models in your application
  • 14. Yegads, Why? Origin Story When we scale Ye Olde web stack? ● Sharding of DB, MQ, Search, Analytics ● Geo-replication / CAP ● Communication data races ● Developers create ad-hoc overlay consistency models ● Lots of time spent cobbling ● High cognitive load
  • 15. Yegads, Why? Some Examples ● Total internet outage Officemates can’t collaborate? ● LieFi Slack + elevator = ● Application state gets “weird” Mental system-model is wrong ● Replication failure Rube Goldberg state-machine
  • 16. Yegads, Why? “Observe locally, act globally” Fig III. State machine for plugging leaky abstractions
  • 17. Yegads, Why? We’ve tried lots of things ● On-prem systems ● Multi-mastering ● Geo-sharding ● Eventual consistency ● Simulated Simultaneity ( global wallclock )
  • 18. Yegads, Why? But most require travel Distance between you and the stuff you probably care about: Much much less Distance between you and the arbiter of linearization: Usually thousands of Km (Stuff you care about) Collaborating with Bob & Alice down the hall You Messages/docs you authored Messages/docs you read Your wristwatch Your IOT devices Next door neighbors The linearization you must visit Backhoes Congestion Light travel time Alligators Net non-neutrality BGP Screwups Power outages State Hacking Spanning tree errors Tripped-over cables Cat on server Submarine cable break Settlement-free peering disputes State Surveillance Packet corruption Lie-FiUnder Provisioned hardware Cheap electrolytic capacitors Thousands of Km of: DNS errors Saturated cell-tower backhaul RF reflections Interference from microwave oven Cosmic Rays Disgruntled employees DDOS Attacks Rat-chewed cables Corrosion Late internet payments Gravitational time-dilation Doppler effect F*#KING BOINGO Failed B-side Power Core router problem F*#KING TMOBILE F*#KING NETGEAR Cabinet switch failure Misconfigured health check NAT Misconfiguration Load balancer failure NTP failures
  • 19. Yegads, Why? Mythology ● Networks can be made reliable ● A single arbiter of state makes a consistency-model “strong” ● Objective state exists ● Objective simultaneity exists It would seem that our foundational abstractions are leaky.
  • 20. Yegads, Why? Avoidance Behaviors ● It’s working for me. Plausible deniability is essential ● That’s unsupported ”You’re holding it wrong” ● Open more reqs for the ops team ”Cobble harder minions!” ● It’s Amazon’s problem! The blame game ● Akka solves it! Event-sourced locality is magical locality ● NuoDB/Spanner solves it! Proprietary magic, now with unicorns!
  • 21. Loading Slide... (waiting is incompatible with human causal expectations) Yegads, Why? Expectations
  • 22. Yegads, Why? What do humans expect? I set my glass on the table ≻ It’s there when I pick it up. Edge computing IRL. Local, coordination-free consistency. No need to visit Ashburn.
  • 23. Yegads, Why? This problem isn’t going away ● Collaborative systems are exploding OT or GTFO ● System complexity becoming unmanageable How much must twitter spend? ● Bandwidth reqs will exponentiate, backbone will not A hololens on every face ● Outages worsen as safety/mission critical uses proliferate Fedex global API is down again? ● Single threaded execution won’t last Sorry node.js ● Stagnant network service offerings Dark fiber revolution my ass
  • 24. ● State is an illusion, sort of... ● Computation and storage must occur nearer the edge ● A priori resource planning is undesirable ● Durability is non-boolean ● Distributed causal consistency is possible, efficient, desirable ● Leaky abstractions are expensive to cobble together Synopticon Motivating Suppositions
  • 25. Motivating suppositions State is an illusion, sort of ● A projection of observed events Essentially no such thing as shared state ● “State” is arguably an abstraction Not unlike “solid matter” vs coulomb force ● Some abstractions we need not question ”Why don’t I fall through the floor?” ● Collaborative systems* warrant it ALL multi-user systems are collaborative * All multi user systems are collaborative systems Philosophy Corner: How does reality differ from causality?
  • 26. Motivating suppositions Durability is not a boolean ● All data has a nonzero probability of loss ● We care more about some data than other data ● Single-copy durability is not high-durability
  • 27. Motivating suppositions Location, Location, Location ● Humans expect physical artifacts Proximity the only way to meet this expectation ● Travel to arbiter unacceptable default assumption Reality has no centralized arbiters, nor should we ● Data should be localized by origin and utility Geo-sharding and hash-sharding a poor solution ● A priori shard planning is expensive It’s expensive, and you’ll mess it up* *Automated a priori planning only marginally better than humans
  • 28. When you squeeze a balloon… Motivating suppositions Behaviors > Planning
  • 29.
  • 30. How does it work?
  • 31. Synopticon How does it work? ● Distributed Merkle-style DAG The easy part ● Sparse vector clocks Similar to Interval Tree Clocks. Baquero et al ● Selective-hearing gossip Essential for traffic reduction ● Commutative index merging Permits eventual convergence ● Infectious knowledge Notice when insufficiently converged ● “Gravity” and “pressure” Bring important data near, push away the rest
  • 32.
  • 33.
  • 34.
  • 35.
  • 36.
  • 37.
  • 38.
  • 39.
  • 40.
  • 41.
  • 42. How does it work? Review ● Immutable Merkle-style DAG The easy part ● Sparse vector clocks Vector clock compression scheme ● Selective-hearing gossip Essential for traffic reduction ● Commutative index merging Permits eventual convergence ● Infectious knowledge Notice when insufficiently converged ● Gravity and pressure Bring important data near, push away the rest
  • 43. Will it have the desired properties? Will it: ● Be comparable with RDBMS performance? ● Meet its consistency-model invariants? ● Scale as envisioned? ● Provide an ergonomic interface for developers? ● Make users happy? NFI - Only experimental data will bear this out!
  • 44. With unbase, we hope to: ● Align system performance with user’s causal expectations ● Increase availability (for data I care about) ● Reduce system complexity ● Save time and money on system operations ● Build in collaboration-awareness from the start Synopticon Conclusion
  • 45. Thank You! Daniel Norman CTO, güdTECH unba.se contributor Twitter: @DreamingInCode AMW43Asilomar Microcomputer Workshop 2017 unba.se crates.io/crates/unbase gitter.im/unbase #unbase at irc.mozilla.orgunba.se/design/bibliography.html Do you like rust and believe this crazy thing is possible? I need your help! Join me!
  • 47. ● What is consistency? Serializability usually just a hack for causality ● What is Availability? Declared based on an observational standard ● What is a Partition? Only exists according to an observational standard (With sufficiently tortured definitions, one can prove a ham sandwich.) See Martin Kleppmann’s excellent paper: A critique of the CAP theorem – arxiv.org/abs/1509.05393 Obligatory CAP Slide Problems with CAP Theorem