Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

RedisDay London 2018 - Atom: Redis for Robotics

62 views

Published on

RedisDay London 2018

Published in: Technology
  • Be the first to comment

  • Be the first to like this

RedisDay London 2018 - Atom: Redis for Robotics

  1. 1. Atom: Redis for Robotics Dan Pipe-Mazo, Elementary Robotics
  2. 2. ELEMENTARY ROBOTICS NOV 15 Elementary Robotics: Who We Are We believe that there are two main items holding back the robotics industry: Cost & Ease of programming. We’re building low-cost, data-rich robots from the ground up with an emphasis on performance and ease of use.
  3. 3. ELEMENTARY ROBOTICS NOV 15 Robotics Stack: Basics HARDWARE DRIVERS SOFTWARE Joints Motors Sensors Talk to all hardware 
 in low-level Programming languages/ kernel modules Proprietary communications protocols Implements high-level logic and algorithms for the system. 
 Calls driver APIs as necessary to get things done.
  4. 4. ELEMENTARY ROBOTICS NOV 15 ABSTRACTION SCALABILITY LANGUAGE SUPPORT Standardize complex hardware and algorithms into simple paradigms Easy horizontal scaling for compute-heavy tasks Certain tasks are better suited to certain languages Software Stack: Objectives DEPENDENCIES Eliminate dependency issues INTROSPECTION Easily audit the system for debug/monitoring
  5. 5. ELEMENTARY ROBOTICS NOV 15 Software Stack: Solution + API Create a specification, API, and set of libraries called atom prebuilt with all dependencies and shipped in a docker container. MICRO-SERVICES Create micro services called elements that use the atom libraries, in their preferred language, to implement higher-level APIs TECHNOLOGY Use redis streams for performant, flexible distributed data publishing and command/response paradigms.
  6. 6. ELEMENTARY ROBOTICS NOV 15 Redis Streams: overview XADD s1 MAXLEN ~ X k1 v1 k2 v2 … ... stream: s1 0 - k1 v1 k2 v2 1 - k1 v1 k2 v2 2 - k1 v1 k2 v2 X - k1 v1 k2 v2
  7. 7. ELEMENTARY ROBOTICS NOV 15 Redis Streams: overview XADD s2 MAXLEN ~ X kA vA kB vB … XADD s1 MAXLEN ~ X k1 v1 k2 v2 … stream: s2 ... 0 - kA vA kB vB 1 - kA vA kB vB 2 - kA vA kB vB X - kA vA kB vB ... stream: s1 0 - k1 v1 k2 v2 1 - k1 v1 k2 v2 2 - k1 v1 k2 v2 X - k1 v1 k2 v2
  8. 8. ELEMENTARY ROBOTICS NOV 15 Redis Streams: overview XADD s2 MAXLEN ~ X kA vA kB vB … XADD s1 MAXLEN ~ X k1 v1 k2 v2 … stream: s2 ... stream: s1 0 - k1 v1 k2 v2 1 - k1 v1 k2 v2 2 - k1 v1 k2 v2 X - k1 v1 k2 v2 XREAD BLOCK N STREAMS s1 $ XREAD BLOCK N STREAMS s1 ID Subscribe to all entries from one stream ... 0 - kA vA kB vB 1 - kA vA kB vB 2 - kA vA kB vB X - kA vA kB vB
  9. 9. ELEMENTARY ROBOTICS NOV 15 Redis Streams: overview XADD s2 MAXLEN ~ X kA vA kB vB … XADD s1 MAXLEN ~ X k1 v1 k2 v2 … stream: s2 ... stream: s1 0 - k1 v1 k2 v2 1 - k1 v1 k2 v2 2 - k1 v1 k2 v2 X - k1 v1 k2 v2 XREAD BLOCK N STREAMS s1 $ XREAD BLOCK N STREAMS s1 ID Subscribe to all entries from one stream XREAD BLOCK N STREAMS s1 s2 ID1 ID2 Subscribe to all entries from multiple streams ... 0 - kA vA kB vB 1 - kA vA kB vB 2 - kA vA kB vB X - kA vA kB vB
  10. 10. ELEMENTARY ROBOTICS NOV 15 Redis Streams: overview XADD s2 MAXLEN ~ X kA vA kB vB … XADD s1 MAXLEN ~ X k1 v1 k2 v2 … stream: s2 ... stream: s1 0 - k1 v1 k2 v2 1 - k1 v1 k2 v2 2 - k1 v1 k2 v2 X - k1 v1 k2 v2 XREAD BLOCK N STREAMS s1 $ XREAD BLOCK N STREAMS s1 ID Subscribe to all entries from one stream XREAD BLOCK N STREAMS s1 s2 ID1 ID2 Subscribe to all entries from multiple streams XREVRANGE s1 + - COUNT N Get the latest N entries from a stream ... 0 - kA vA kB vB 1 - kA vA kB vB 2 - kA vA kB vB X - kA vA kB vB
  11. 11. ELEMENTARY ROBOTICS NOV 15 Redis Streams: overview XADD s2 MAXLEN ~ X kA vA kB vB … XADD s1 MAXLEN ~ X k1 v1 k2 v2 … stream: s2 ... stream: s1 0 - k1 v1 k2 v2 1 - k1 v1 k2 v2 2 - k1 v1 k2 v2 X - k1 v1 k2 v2 XREAD BLOCK N STREAMS s1 $ XREAD BLOCK N STREAMS s1 ID Subscribe to all entries from one stream XREAD BLOCK N STREAMS s1 s2 ID1 ID2 Subscribe to all entries from multiple streams XREVRANGE s1 + - COUNT N Get the latest N entries from a stream XRANGE s1 - + COUNT N Get the oldest N entries from a stream ... 0 - kA vA kB vB 1 - kA vA kB vB 2 - kA vA kB vB X - kA vA kB vB
  12. 12. ELEMENTARY ROBOTICS NOV 15 Atom Specification and Examples
  13. 13. ELEMENTARY ROBOTICS NOV 15 Atom: Data Publishing PUBLISHER: element “foo”, streams s1 and s2 SUBSCRIBER: Options to read from the stream XADD data:foo:s1 MAXLEN ~ X k1 v1 k2 v2 … XADD data:foo:s2 MAXLEN ~ X k3 v3 k4 v4 … XREAD BLOCK T STREAMS data:foo:s1 data:foo:s2 idS1 idS2 XREVRANGE data:foo:s1 + - COUNT N XRANGE data:foo:s1 - + COUNT N
  14. 14. ELEMENTARY ROBOTICS NOV 15 Atom: Data Publishing Example CAMERA USB camera element MAIN COMPUTE XADD frames elem:cam:frames REDIS
  15. 15. ELEMENTARY ROBOTICS NOV 15 Atom: Data Publishing Example CAMERA USB camera element MAIN COMPUTE XADD frames elem:cam:frames REDIS video packaging element XREAD BLOCK N
  16. 16. ELEMENTARY ROBOTICS NOV 15 Atom: Data Publishing Example CAMERA USB camera element MAIN COMPUTE XADD frames elem:cam:frames REDIS video packaging element ML classification element XREAD BLOCK N XREVRANGE + -
  17. 17. ELEMENTARY ROBOTICS NOV 15 Atom: Data Publishing Example CAMERA USB camera element MAIN COMPUTE XADD frames elem:cam:frames REDIS video packaging element ML classification element ML retraining element XREAD BLOCK N XREVRANGE + - XRANGE last +
  18. 18. ELEMENTARY ROBOTICS NOV 15 Atom: Sync or Async Req/Rep Requestor: “foo” Responder: “bar”REDIS elem:bar:req XREAD BLOCK 0 STREAMS elem:bar:req lastEntryID elem:foo:rep
  19. 19. ELEMENTARY ROBOTICS NOV 15 Atom: Sync or Async Req/Rep Requestor: “foo” Responder: “bar”REDIS elem:bar:req XREAD BLOCK 0 STREAMS elem:bar:req lastEntryIDXADD elem:bar:req MAXLEN ~ X elem foo cmd Y data Z elem:foo:rep
  20. 20. ELEMENTARY ROBOTICS NOV 15 Atom: Sync or Async Req/Rep Requestor: “foo” Responder: “bar”REDIS elem:bar:req (entryID, {“elem”: foo, “cmd”: Y, “data”: Z}) XREAD BLOCK 0 STREAMS elem:bar:req lastEntryIDXADD elem:bar:req MAXLEN ~ X elem foo cmd Y data Z elem:foo:rep
  21. 21. ELEMENTARY ROBOTICS NOV 15 Atom: Sync or Async Req/Rep Requestor: “foo” Responder: “bar”REDIS elem:bar:req (entryID, {“elem”: foo, “cmd”: Y, “data”: Z}) XREAD BLOCK 0 STREAMS elem:bar:req lastEntryIDXADD elem:bar:req MAXLEN ~ X elem foo cmd Y data Z XREAD BLOCK 1000 STREAMS elem:foo:rep lastEntryID elem:foo:rep
  22. 22. ELEMENTARY ROBOTICS NOV 15 Atom: Sync or Async Req/Rep Requestor: “foo” Responder: “bar”REDIS elem:bar:req (entryID, {“elem”: foo, “cmd”: Y, “data”: Z}) XREAD BLOCK 0 STREAMS elem:bar:req lastEntryID XADD elem:foo:rep MAXLEN ~ X elem bar id ID time T XADD elem:bar:req MAXLEN ~ X elem foo cmd Y data Z XREAD BLOCK 1000 STREAMS elem:foo:rep lastEntryID elem:foo:rep
  23. 23. ELEMENTARY ROBOTICS NOV 15 Atom: Sync or Async Req/Rep Requestor: “foo” Responder: “bar”REDIS elem:bar:req (entryID, {“elem”: foo, “cmd”: Y, “data”: Z}) XREAD BLOCK 0 STREAMS elem:bar:req lastEntryID XADD elem:foo:rep MAXLEN ~ X elem bar id ID time T XADD elem:bar:req MAXLEN ~ X elem foo cmd Y data Z (entryID, {“elem”: bar, “id”: ID, “time”: T}) XREAD BLOCK 1000 STREAMS elem:foo:rep lastEntryID elem:foo:rep
  24. 24. ELEMENTARY ROBOTICS NOV 15 Atom: Sync or Async Req/Rep Requestor: “foo” Responder: “bar”REDIS elem:bar:req (entryID, {“elem”: foo, “cmd”: Y, “data”: Z}) XREAD BLOCK 0 STREAMS elem:bar:req lastEntryID XADD elem:foo:rep MAXLEN ~ X elem bar id ID time T XADD elem:bar:req MAXLEN ~ X elem foo cmd Y data Z (entryID, {“elem”: bar, “id”: ID, “time”: T}) XREAD BLOCK 1000 STREAMS elem:foo:rep lastEntryID elem:foo:rep XREAD BLOCK T STREAMS elem:foo:rep lastEntryID
  25. 25. ELEMENTARY ROBOTICS NOV 15 Atom: Sync or Async Req/Rep Requestor: “foo” Responder: “bar”REDIS elem:bar:req (entryID, {“elem”: foo, “cmd”: Y, “data”: Z}) XREAD BLOCK 0 STREAMS elem:bar:req lastEntryID XADD elem:foo:rep MAXLEN ~ X elem bar id ID time T XADD elem:foo:rep MAXLEN ~ X elem bar id ID resp R XADD elem:bar:req MAXLEN ~ X elem foo cmd Y data Z (entryID, {“elem”: bar, “id”: ID, “time”: T}) XREAD BLOCK 1000 STREAMS elem:foo:rep lastEntryID elem:foo:rep XREAD BLOCK T STREAMS elem:foo:rep lastEntryID
  26. 26. ELEMENTARY ROBOTICS NOV 15 Atom: Sync or Async Req/Rep Requestor: “foo” Responder: “bar”REDIS elem:bar:req (entryID, {“elem”: foo, “cmd”: Y, “data”: Z}) XREAD BLOCK 0 STREAMS elem:bar:req lastEntryID XADD elem:foo:rep MAXLEN ~ X elem bar id ID time T XADD elem:foo:rep MAXLEN ~ X elem bar id ID resp R XADD elem:bar:req MAXLEN ~ X elem foo cmd Y data Z (entryID, {“elem”: bar, “id”: ID, “time”: T}) XREAD BLOCK 1000 STREAMS elem:foo:rep lastEntryID elem:foo:rep (entryID, {“elem”: bar, “id”: ID, “resp”: R}) XREAD BLOCK T STREAMS elem:foo:rep lastEntryID
  27. 27. ELEMENTARY ROBOTICS NOV 15 Atom: Req/Rep Example MAIN COMPUTE REDIS user element robot element elem:robot:req elem:user:rep Go to (x, y, z)
  28. 28. ELEMENTARY ROBOTICS NOV 15 Atom: Req/Rep Example MAIN COMPUTE REDIS user element robot element elem:robot:req elem:user:rep Go to (x, y, z) ACK
  29. 29. ELEMENTARY ROBOTICS NOV 15 Atom: Req/Rep Example COMMS MAIN COMPUTE REDIS MOTOR 1 MOTOR 2 MOTOR N ... user element robot element elem:robot:req elem:user:rep Go to (x, y, z) ACK
  30. 30. ELEMENTARY ROBOTICS NOV 15 Atom: Req/Rep Example COMMS MAIN COMPUTE REDIS MOTOR 1 MOTOR 2 MOTOR N ... user element robot element elem:robot:req elem:user:rep Go to (x, y, z) ACK path planning element camera element
  31. 31. ELEMENTARY ROBOTICS NOV 15 Atom: Req/Rep Example COMMS MAIN COMPUTE REDIS MOTOR 1 MOTOR 2 MOTOR N ... user element robot element elem:robot:req elem:user:rep Go to (x, y, z) ACK Done path planning element camera element
  32. 32. ELEMENTARY ROBOTICS NOV 15 Atom: Scaling & Updates REDIS STREAM CONSUMER GROUPS XGROUP CREATE elem:bar:req consumers $ UPDATES (ASSUMING SAME API) Spin up the new instance, monitor performance, spin down the old instance. SCALING Just spin up another instance of the element on the same redis master. XREADGROUP consumers me BLOCK 0 COUNT 1 STREAMS elem:bar:req < XACK elem:bar:req consumers entryID
  33. 33. ELEMENTARY ROBOTICS NOV 15 Atom: Scaling/Update Example MAIN COMPUTE REDISML classification element (copy 1) elem:class:req elem:user:rep user element Classify A, B, C, D A
  34. 34. ELEMENTARY ROBOTICS NOV 15 Atom: Scaling/Update Example MAIN COMPUTE REDISML classification element (copy 1) ML classification element (copy 2) elem:class:req elem:user:rep user element Classify A, B, C, D A B
  35. 35. ELEMENTARY ROBOTICS NOV 15 Atom: Scaling/Update Example MAIN COMPUTE REDISML classification element (copy 1) ML classification element (copy 2) elem:class:req elem:user:rep user element Classify A, B, C, D Result A Results Result B
  36. 36. ELEMENTARY ROBOTICS NOV 15 Atom: Scaling/Update Example MAIN COMPUTE REDISML classification element (copy 1) ML classification element (copy 2) elem:class:req elem:user:rep user element Classify A, B, C, D C D
  37. 37. ELEMENTARY ROBOTICS NOV 15 Atom: Scaling/Update Example MAIN COMPUTE REDISML classification element (copy 1) ML classification element (copy 2) elem:class:req elem:user:rep user element Classify A, B, C, D Result C Results Result D
  38. 38. ELEMENTARY ROBOTICS NOV 15 Atom: Basic Discovery ELEMENT DISCOVERY DATA STREAM DISCOVERY SCAN cursor MATCH elem: SCAN cursor MATCH data: SCAN cursor MATCH data:foo COMMAND DISCOVERY In the works!
  39. 39. ELEMENTARY ROBOTICS NOV 15 Atom: Serialization OVERALL COMMAND/ RESPONSE DATA JPEG/TIFF best for camera/PointCloud data, Protobuf pretty good for most other data.
 No single serialization protocol was best/optimal for all payloads/use cases. No Enforced Serialization Protobuf performed well and RPC service specification worked well for strongly typed command/response types.
  40. 40. ELEMENTARY ROBOTICS NOV 15 Atom: API class Element(): __init__(self, name) __del__(self) # Commands
 command_add( self, name, handler, timeout=DEFAULT_TIMEOUT) command_loop( self) command_send( self, element, command, data, block=True) # Data entry_write( self, name, {“k1”: v1, “k2”:v2, … }, maxlen=DEFAULT_MAXLEN) entry_read_n( self, element, stream, n) entry_read_loop( self, {“s1”: h1, “s2”: h2, …}, n_loops=None) # Discovery get_all_elements( self) get_all_streams( self, element_name=None)
  41. 41. ELEMENTARY ROBOTICS NOV 15 Atom: Docker NUCLEUS DOCKER CONTAINER ATOM DOCKER CONTAINER ELEMENT DOCKER CONTAINERS User-developed elements FROM elementaryrobotics/atom FROM redis/alpine Contains custom redis conf + atom CLI Latest tested atom libraries All dependencies, protobuf, etc. MOLECULE DOCKER COMPOSE FILE Launch nucleus and all elements Set up shared tmpfs between containers & unix socket for redis

  42. 42. ELEMENTARY ROBOTICS NOV 15 Atom: User Development Flow 1 Copy template from atom github repo. 2 Implement element in language of choosing 3 Build element with 4 Launch element, nucleus and all dependencies with docker-compose up -d docker-compose build 5 Done!
  43. 43. ELEMENTARY ROBOTICS NOV 15 Summary ATOM SET OF TOOLS OUR WEBSITE elementaryrobotics.com. Please visit and sign up for more info on atom. Allow for powerful yet simple new ways of programming robotic systems. WE’RE HIRING! REDIS STREAMS We're working on atom to bring the power of redis streams to robotics. We build robots in LA! Send me an email directly at dan@elementaryrobotics.com

  44. 44. Thank you! Please check out our website at elementaryrobotics.com and sign up for more info about atom!

×