CS519 - Cloud Types for Eventual Consistency


Published on

Overview of the paper "Cloud Types for Eventual Consistency" by Burckhardt et al. presented at Oregon State University for "Software Evolution for Mobility" class on Oct 10th 2013. Presentation time: 20 min

Published in: Technology, Business
  • Oleg,

    thanks for your comment!

    Slide #12 - that's the initial proposed solution of 'Cloud types for Eventual Consistency' published by Microsoft researchers: http://research.microsoft.com/pubs/163842/final-with-color.pdf
    it has 'geeky' academic terminology.
    Slide #13 - it is one of the ways of addressing well-known CAP problem (consistency, availability, partition tolerance)
    Slide #18 - that's how people do it now. presented solution is not fully implemented yet because it is still in the research (not development) phase. It is partly implemented in a scope of TouchDevelop platform though.
    #22 - gray - cloud, blue - device. I should have copied text too.. :)
    #23 - yield returns control immediately while flush takes some time. tennis ball represents control.
    #25 - it is assumed that all other types could be constructed using basic building blocks: CInt, CString and CSet. I would not say it is similar to divide-and-conquer.
    Are you sure you want to  Yes  No
    Your message goes here
  • Marvellous!

    Slide #6 - how about biz and industrial apps, apart from collaboration?

    Slide #7 - doubt that. for example, i do not feel any need for communication but require a couple of trustworthy data sources which can provide me with qualified info that meets my needs and that i can consume while drinking a cup of team, not longer

    Slide #8 - 'stored there somewhere beneath the Capitol Building'... sorry, memories :) by the way, where's sharing gone?

    Slide #12 - did not get it. kind of data containers such as maps, vectors etc.? (ok, looks likes final slides give a hint)

    Slide #13 - does it have anything to do with distributed file systems, distributed/multitenant databases or any other gears?

    Slide #18 - 3-tier beast. can old programming techniques be reused for the cloud beast? i mean legacy code.

    Slide #22 - where are trx boundaries?

    Slide #23 - did not get it. `yield` is kind of pushing into a queue? or it refers to write buffer?

    Slide #25 - what about data dependencies? is it kind of divide-and-conquer?
    Are you sure you want to  Yes  No
    Your message goes here
No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide
  • a transaction schedule is serializable if its outcome (e.g., the resulting database state) is equal to the outcome of its transactions executed serially, i.e., sequentially without overlapping in time
  • CS519 - Cloud Types for Eventual Consistency

    1. 1. CLOUD TYPES FOR EVENTUAL CONSISTENCY Class: CS 519 – Software Evolution for Mobility Presenter: Sergii Shmarkatiuk Date: 10/09/2013
    2. 2. MOTIVATION 2 Today there are more mobile applications available for download and usage than desktop applications.
    3. 3. MOTIVATION 3 And number of mobile applications does not go down
    4. 4. MOTIVATION 4 It goes up!
    5. 5. MOTIVATION 5 Mobile applications need to communicate with online services and with each other
    6. 6. 6 MOTIVATION: WHY MOBILE APPS COMMUNICATE? Personal Publishing Games Data Collection Collaboration Sync and Backup Transactions Blog Facebook Wall Website Music Video SkyDrive Surveys High Scores Shared Lists Shared Calendar Shared Spreadsheet Real-time Turn-based Store Auction Matchmaking Remote Control Home Control Robotics Media Player
    7. 7. MOTIVATION 7 Increased need of communication means more data
    8. 8. MOTIVATION 8 One of the most effective ways of storing data for mobile devices is a cloud
    9. 9. MOTIVATION 9 Being effective way of storage, cloud data manipulation poses some challenges for developers
    10. 10. MOTIVATION 10 Developers usually have responsibility of managing different aspects of cloud data manipulation Cloud data Storage Synchroni zation Caching Manipulati on Conflicts resolution
    11. 11. MOTIVATION 11 Cloud data manipulation should be simplified for more convenient use
    12. 12. PROPOSED SOLUTION 12 Implementation of eventually consistent storage at the programming language level using cloud data types.
    13. 13. WHAT IS EVENTUAL CONSISTENCY?  Weak consistency model of storing shared data in a distributed system that allows clients to perform updates against any replica at any time.  One of the approaches to the CAP (consistency, availability, partition tolerance) problem  Guarantee that all updates are eventually delivered to all replicas, and that they are applied in a consistent order  Transactional model providing basic requirements of atomicity and isolation without possibility of transactions serialization 13
    14. 14. 14 TRANSACTIONS: PICK ANY TWO Atomicity Isolation Serializability
    15. 15. 15 TRANSACTIONS: CHOICE OF EVENTUAL CONSISTENCY MODEL Atomicity Isolation Serializability Revisions
    16. 16. REVISION DIAGRAMS BURCKHARDT’2011: SEMANTICS OF CONCURRENT REVISIONS 16Data propagate only along edges
    17. 17. 18 Node Storage Compute Node Storage Compute Node Storage Compute Basic (peer-to-peer) Using Cloud Infrastructure Storage Storage Compute Compute Compute Storage Storage Client Client Client Client Client Client DISTRIBUTED SYSTEMS
    18. 18. HOW TO PROGRAM USING CLOUD CONCEPTS? 19 Layer Storage Storage Compute Compute Compute Storage Storage Client Client Client Client Client Client Client Not physically secure Unreliable Cannot detect failures Potentially many Cloud Compute Physically secure, not so many Not reliable: no persistent state Can detect failures somewhat Relatively Expensive Cloud Storage Secure Reliable Can be very cheap
    19. 19. MOTIVATING EXAMPLE: GROCERY LIST 20 milk bread eggs cilantro sardines guava
    20. 20. DATA MANAGEMENT MODEL 21 device 1 device 2cloud• Client code:  Declare data types  read/update data  yield (=polite sync)  flush (=forced sync) • Under the hood:  Revision diagram rules
    21. 21. 22 device 1 device 2cloud IMPLICIT TRANSACTIONS • At yield Runtime has permission to send or receive updates. Call this frequently, e.g. automatically “on idle”. • In between yields Runtime is not allowed to send or receive updates • Implies: all client code executes in a (eventually consistent) transaction … … … … … … … yield yield yield yieldyield yield yield yield
    22. 22. STRONG CONSISTENCY ON- DEMAND  flush primitive blocks until local state has reached main revision and result has come back to device  Sufficient to implement strong consistency  Flush blocks –times out if server connection is not available. flush (blocks) (continue)
    23. 23. YIELD/FLUSH EXAMPLE 24 yield (=polite sync) flush(=forced sync)
    24. 24. FORK-JOIN AUTOMATON (FJA) 25  Data set is copied on forking  Data is manipulated in isolation after fork  When data is joined, changes are merged.  The merge is fully defined by the data type declarations.  some types may include custom merge functions  there is no failure, rollback, or retry B D CA fork fork fork join join
    25. 25. PAPER CONTRIBUTIONS  Cloud types definition (CInt, CString, CSet)  Formal description of language constructs (big-step notation)  Fork-join automaton operations formalization (create, delete, propagate, fork, join, …)  Formalization of distribution operations (yield-pull, yield-flush, sync-pull, sync-flush, …)  Formalization of language constructs used by developers in client applications (new, delete, entries, all, yield, flush) 26
    26. 26. PAPER CONTRIBUTIONS 27 No evaluation Reference/specification for developers implementing eventual consistency with cloud types
    27. 27. PITFALLS 28 Last writer wins! Subtle difference between cloud type methods set and add
    28. 28. RELATED WORK  CRDT (commutative replicated data types)  Cloud types allow non-commutative operations  No integration with fork-join automaton  Concurrent revisions approach  Necessity of explicit merge (rfork, rjoin, …)  Persistent data types  Do not take into account transactions or distribution  Operational transformations  Very similar to eventual consistency with cloud types, but difficult to implement  More focused on correctness checks  OLAP/OLTP  Not distributed  Google's Drive's Realtime API, Dropbox Sync API, Firebase  Complicated, too many things to care of 29
    29. 29. QUESTIONS TO DISCUSS  Sergii: Is model of eventual consistency with cloud types really suitable for users data management?  Sergii: How does eventual consistency with cloud types ensure that there are no clones entities?  Sergii: How could developers use yield/flush operations effectively in their code? What is the typical case/example of flush or yield statement usage?  Sergii: How does data actually become eventually consistent? What mechanism does ensure that data distribution model is correct? 30
    30. 30. QUESTIONS TO DISCUSS  Michael: Since the development of cloud types make the functionality of cloud synchronization available to the user as a type, how will that affect tools such as debugging? Will the debugging tools show the users the revision diagrams and expose the underling cloud functionality to the users, or will it hide it from them? Will this cloud functionality confuse users as to how these types are behaving?  Michael: How scalable is the solution? At what point will it start to break down because there are too many users hitting it? Will it start to break down with 10 users accessing the same data? 1000? 1M? 31
    31. 31. FOLLOW-UP RESEARCH SUGGESTIONS  Sergii: Research on how existing types could be mapped to cloud types  Sergii: Study model limitations (silent conflict resolution)  Michael: While auto-synchronizing primitive cloud types are great for certain applications, it would be interesting to research ways to give users the power of eventual constancy while still providing a way for them to have finer grained control over the data. One use case would be to allow users to choose which other user's changes they would like to become constant with, or someone who's changes they chose to ignore 32