Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

ERS downscale2016


Published on

The Entity Registry System as extended by Mihai Gramada and Christophe Gueret. Presented at Downscale2016 ( by Victor de Boer.

Published in: Education
  • Be the first to comment

  • Be the first to like this

ERS downscale2016

  1. 1. Entity Registry System From concept to deployment Mihai Gramada, Christophe Gueret (Victor de Boer)
  2. 2. Linked data - In computing, linked data (often capitalized as Linked Data) describes a method of publishing structured data so that it can be interlinked and become more useful through semantic queries (Wikipedia) - Tim Berners Lee - DBpedia alone 3.4 million concepts described by 1 billion triples
  3. 3. Example solutions - centralised - rdf4j - neo4j - gremlin (apache tinkerpop) - virtuoso - many others
  4. 4. Entity Registry System - decentralised - linked data in document store (s, p1, o1), (s, p2, o2) -> {“id”: s, “p1”: o1, “p2”: o2, ..} - designed for poor network connectivity
  5. 5. Example: OLPCs and RPi
  6. 6. Architecture - contributors, bridges, aggregator - avahi(zeroconf) for peer discovery - couchdb for heavy lifting
  7. 7. Contributor - read and write content - private, public and cache - private never shared - public can be searched by others and distributed
  8. 8. Bridge - can connect isolated parts of the system - similar to a cache (if network goes down, data can still be read from the bridge) - reduce O(n^2) links down to O(n)
  9. 9. Aggregator - optional component - high-performance node/cluster - read-only entry retrieval of data - contributors and bridges push public data to it
  10. 10. Deployment example
  11. 11. Replication strategy
  12. 12. Similar solutions Nintendo streetpass and spotpass(3DS only, no aggregator) Transparent Inter-process Communication - nodes, zones, cluster (only logical grouping) - ip layer, bridges reduce links Sugar network (OLPC) - in sugar network, clients can communicate to nodes(equivalent of bridges) or Master node(equivalent of Aggregator) - ers has more flexibility in the data format.
  13. 13. Initial status - not working, - very limited testing - a bit frustrating to install - not much investigation (real world tests) Thus far, the highest number of concurrent users of ERS has been 4 XO laptops, with one bridge, all in the same geographical location
  14. 14. Research Questions - Can the Entity Registration system reliably perform in a real-world scenario ? (i.e. provide the required functionality in a robust manner)? - Does the ERS scale to a large number of users? - How does the ERS cope with poor network connectivity?
  15. 15. “local” tests - unit tests for storage, daemon and api communication - basic operations performance tests - entity creation 5/s - property edits 20/s - value edits 20/s
  16. 16. Docker - much larger number of nodes - faster to start - docker hub image -> ers can be ran within seconds on any x86 device (atm no docker image for arm)
  17. 17. “Real world” simulation - simulations so far have a more or less ideal network - reality is a bit different
  18. 18. Simian army - - december 2012 amazon employee launched a maintenance process against the running production system which deleted the state information needed by load balancers - problems on Christmas Eve at 1:45 p.m. - lasted until 9:41 a.m. on Christmas Day, an outage of about 20 hours - no Netflix on christmas - unhappy customers
  19. 19. Simian army - cont - clunky to integrate - only works on aws
  20. 20. Monkeybusiness - imitate simian army - works on vagrant and docker - work in progress :)
  21. 21. Latency - insert network issues - latency - loss - reordering - duplication - corruption
  22. 22. Real world case - Conference deployment - Simulate conference social network - Think LinkedIn without central server - Profile, skills - Endorsements - Fixed bridge, mobile contributors
  23. 23. Conference results Tested with up to 30 nodes and a bridge. - Can tolerate disconnects/poor network - It just works™
  24. 24. Remote merchants - vendors in different remote villages - no network connectivity - box on a truck that visits every farmer, provides up to date information on the prices of the other villages, picks up any new information from current one - Fixed contributors, mobile bridge
  25. 25. Behavior - As long as network isn’t too bad(will be detailed) if the truck stops for a couple of seconds we achieve synchronization
  26. 26. Network tolerance Fixed 5 seconds wait time, contributor writing as fast as it can - 100 ms each way - 15% loss/corruption each way - duplication is fine - mostly binary progress
  27. 27. Research Questions Revisited Can the Entity Registration system reliably perform in a real-world scenario ? (i.e. provide the required functionality in a robust manner)? Suite deployed, tests indicate yes - Does the ERS scale to a large number of users? 40 nodes - How does the ERS cope with poor network connectivity? - If connection for .5 sec or longer: yes
  28. 28. Discussion