Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

SSH I/O Streaming via Redis-based Persistent Message Queue -Mani Tadayon

604 views

Published on

Redis is a rock-solid platform for a
variety of real-world use cases, in particular as a poor man’s message queue. At Apple Maps, we built a service to show live
I/O from thousands of concurrent SSH sessions in real-time using Redis, Lua scripts, node.js and HTML5 Server-Sent Events.
Although our architecture isn’t ideal, and we would do things differently today, our system has performed very well in the
real-world over the past couple of years. In particular, after some initial failures, it has scaled very well as usage has grown
much faster than we had ever anticipated. I’ll talk about the initial design, implementation, and the evolution of specific
features to address real-world memory usage and performance challenges

Published in: Technology
  • Be the first to comment

SSH I/O Streaming via Redis-based Persistent Message Queue -Mani Tadayon

  1. 1. START job "A" I/O stdin, stdout, … SSH session "X" SEND queue "A:X" LISTEN queue "X:A" AUTHORIZE HTTP SSE Architecture for SSH I/O streaming SSH I/O streaming via Redis-based persistent message queue Mani Tadayon RedisConf 2016
  2. 2. About me • @bwsr_sr • Working in software since ’01 • Using Redis since ’13 <shameless-plug> • Just finished a book on Ruby testing: RSpec Essentials • http://amzn.com/1784395900 </shameless-plug>
  3. 3. How to build a message queue with Redis • Live listener > subscribe mychannel • Publisher > publish mychannel mymessage And that’s my talk, thanks for listening! DoDo’s mad!
  4. 4. – http://antirez.com/news/88 “Redis apparently is at the same time the best and the worst system to use like that.”
  5. 5. Demo
  6. 6. START job "A" I/O stdin, stdout, … SSH host "X" SEND queue "A:X" LISTEN queue “A:X” AUTHORIZE HTTP SSE Architecture for SSH I/O streaming
  7. 7. How to build a persistent message queue with Redis • Retrieve persisted messages > lrange mykey 0 -1 • Publisher > rpush mykey mymessage What about a live listener?
  8. 8. The best of both worlds • Live listener > subscribe mychannel • Retrieve persisted messages > lrange mykey 0 -1 • Publisher > rpush mykey mymessage > publish mychannel mymessage That’s pretty much it. But the SSH I/O feature needs a few more features.
  9. 9. • Lookup by session ID (“A:X”) • Use a simple list of key names > rpush byjob mykey > expire byjob 604800 • Lookup by hostname (“A:*”) • Use a zset with timestamp as score > zadd byhost 1463012431 mykey > zremrangebyscore byhost 0 1462407631
  10. 10. • Wrap each message in a transaction • Protect against excessive memory usage (150GB in 2 hours on the 1st day…) • Limit number of persisted messages per job • Expire persisted messages (more aggressively for verbose jobs) • Stop sending messages above a threshold (handled outside Redis) > multi > rpush mykey mymessage > expire mykey 604800 > ltrim mykey -100 -1 > publish mychannel mymessage > exec
  11. 11. • Set up indexes for lookup (only once per job) > rpush byjob mykey > expire byjob 604800 > zadd byhost 1463012431 mykey > zremrangebyscore byhost 0 1462407631
  12. 12. Key and channel names ssh-io:session:event:persisted:$ID:$FQDN ssh-io:session:event:live:$ID:$FQDN ssh-io:session:event_lookup:by_job:$ID ssh-io:session:event_lookup:by_hostname:$FQDN
  13. 13. > multi > rpush ssh-io:session:event:persisted:b8042948:fakehost.example.com {"session_started":"About to run 4 commands on fakehost.example.com for CustomScript- TestStreamingSimple:b8042948","io_type":"session_started","timestamp":1462407631} > expire ssh-io:session:event:persisted:b8042948:fakehost.example.com 604800 > ltrim ssh-io:session:event:persisted:b8042948:fakehost.example.com -100 -1 > publish ssh-io:session:event:live:b8042948:fakehost.example.com {"session_started":"About to run 4 commands on fakehost.example.com for CustomScript- TestStreamingSimple:b8042948","io_type":"session_started","timestamp":1462407631} > exec > rpush ssh-io:session:event_lookup:by_job:b8042948 ssh- io:session:event:persisted:b8042948:fakehost.example.com > expire ssh-io:session:event_lookup:by_job:b8042948 604800 > zadd ssh-io:session:event_lookup:by_hostname:fakehost.example.com 1463012431 ssh- io:session:event:persisted:b8042948:fakehost.example.com > zremrangebyscore ssh-io:session:event_lookup:by_hostname:fakehost.example.com 0 1462407631
  14. 14. > multi > rpush ssh-io:session:event:persisted:b8042948:fakehost.example.com {"username":"deploy","stdin":"echo "ping number 1"","hostname":"fakehost.example.com","io_type":"stdin","timestamp":1462407631} > expire ssh-io:session:event:persisted:b8042948:fakehost.example.com 604800 > ltrim ssh-io:session:event:persisted:b8042948:fakehost.example.com -100 -1 > publish ssh-io:session:event:live:b8042948:fakehost.example.com {"username":"deploy","stdin":"echo "ping number 1"","hostname":"fakehost.example.com","io_type":"stdin","timestamp":1462407631} > exec # … multi,rpush,expire,ltrim,publish,exec … > multi > rpush ssh-io:session:event:persisted:b8042948:fakehost.example.com {"session_finished":"Ran 4 of 4 commands on fakehost.example.com for CustomScript- TestStreamingSimple:b8042948","success":false,"io_type":"session_finished","timestamp": 1462407631} > expire ssh-io:session:event:persisted:b8042948:fakehost.example.com 604800 > ltrim ssh-io:session:event:persisted:b8042948:fakehost.example.com -100 -1 > publish ssh-io:session:event:live:b8042948:fakehost.example.com {"session_finished":"Ran 4 of 4 commands on fakehost.example.com for CustomScript- TestStreamingSimple:b8042948","success":false,"io_type":"session_finished","timestamp": 1462407631}
  15. 15. Lookup by hostname • Lua script that uses the zset index • Returns all events, in order, per hostname
  16. 16. $ redis-cli zrange ssh-io:session:event_lookup:by_hostname:fakehost.example.com 0 -1
 1) "ssh-io:session:event:persisted:0bd6005b-1999-42ab-9b54-2585e0383bcb:fakehost.example.com"
 2) "ssh-io:session:event:persisted:84ae935f-55a2-4210-a356-87a0da7c3b58:fakehost.example.com"
 3) "ssh-io:session:event:persisted:35696242-a59d-4a1f-b36f-4ee988b558c5:fakehost.example.com"
 4) "ssh-io:session:event:persisted:b72488ec-816c-48b9-a48d-ab31cbc41802:fakehost.example.com"
 5) "ssh-io:session:event:persisted:5b13b4bd-2800-47fe-a821-a548c90054b6:fakehost.example.com"
 6) "ssh-io:session:event:persisted:26078a92-e349-43a7-9375-d0ce511b1dbd:fakehost.example.com"
 7) "ssh-io:session:event:persisted:4c3f31f0-dd2d-411f-bfbe-d6591698bf3a:fakehost.example.com"
 8) "ssh-io:session:event:persisted:60983166-f580-44d6-9058-3b8bb04c1441:fakehost.example.com"
 9) "ssh-io:session:event:persisted:572de4a5-3d15-4160-ab42-006407662ca8:fakehost.example.com"
 10) "ssh-io:session:event:persisted:3b405df9-3618-4cc2-b245-4dac4b7a203b:fakehost.example.com"
 11) "ssh-io:session:event:persisted:5968f887-9228-4963-9515-b92eda944063:fakehost.example.com"
 12) "ssh-io:session:event:persisted:ebcd6068-4a22-48f1-a333-36a312bb78f0:fakehost.example.com"
 13) "ssh-io:session:event:persisted:d22ae42d-4343-46b8-a477-8fa057bc34b5:fakehost.example.com"
 14) "ssh-io:session:event:persisted:49c994db-9d3d-4453-ac83-8dbf038a655d:fakehost.example.com"
 15) "ssh-io:session:event:persisted:2fc5ea10-c8f8-42e7-93d4-073b68826ffa:fakehost.example.com"
 16) "ssh-io:session:event:persisted:b8042948-63f1-4c6f-afc8-8d2384b3b155:fakehost.example.com"
 17) "ssh-io:session:event:persisted:5fee9662-af66-4027-9963-ff59716dc7e0:fakehost.example.com"
 18) "ssh-io:session:event:persisted:c919f801-ab8b-4f31-92f8-3f51d6b7b7dc:fakehost.example.com"
 19) "ssh-io:session:event:persisted:22f480d9-31cb-4054-9861-51d0275c9468:fakehost.example.com"
 20) "ssh-io:session:event:persisted:7710e61f-921c-4d6d-bbf4-e9c35d932f1a:fakehost.example.com"
  17. 17. -- finds the keys for all sessions for... 
 -- ...a given hostname using lookup lists,
 -- then retrieve all events in all keys
 -- use nested numeric lua tables (i.e. arrays)...
 -- ...since redis will wipe out string keys
 -- see: http://redis.io/commands/eval 
 -- (Conversion between Lua and Redis data types) 
 local keys = redis.call("zrange", KEYS[1], 0, -1)
 local returner = {}
 for i, key in ipairs(keys) do
 returner[i] = {
 key,
 redis.call("lrange", key, 0, -1)
 }
 end
 return returner

  18. 18. $ redis-cli eval "$(cat values-from-lookup-set.lua)" 
 1 ssh-io:session:event_lookup:by_hostname:fakehost.example.com
 
 1) 1) "ssh-io:session:event:persisted: 0bd6005b-1999-42ab-9b54-2585e0383bcb:fakehost.example.com"
 2) 1) "{"session_started":"About to run 4 commands on fakehost.example.com for :0bd6005b-1999-42ab-9b54-2585e0383bcb","io_type": "session_started","timestamp":1462405267}"
 
 # … 18 more sessions …
 
 20) 1) "ssh-io:session:event:persisted:7710e61f-921c-4d6d-bbf4- e9c35d932f1a:fakehost.example.com"
 2) 1) "{"session_started":"About to run 6 commands on fakehost.example.com for CustomScript-TestStreamingSimple:7710e61f-921c-4d6d- bbf4-e9c35d932f1a","io_type":"session_started","timestamp":1462524784}"
  19. 19. Authentication • From browser directly to node.js web service • Web app creates an auth token • Token written to Redis… • …and stored in browser session • 1 day expiry • Simple Lua script to create or retrieve token
  20. 20. local token = ''
 
 if redis.call('exists', KEYS[1]) == 1 then
 token = redis.call('get', KEYS[1])
 else
 redis.call('setex', KEYS[1], ARGV[1], ARGV[2])
 token = ARGV[2]
 end
 
 return token
  21. 21. $ redis-cli eval "$(cat stream_auth_token.lua)" 
 1 ssh-io:session:stream:auth:myuser 86400 "7292ed99-da44-42c3-897a-745b6e43ac33"
 # => "7292ed99-da44-42c3-897a-745b6e43ac33"
 
 $ redis-cli eval "$(cat stream_auth_token.lua)" 
 1 ssh-io:session:stream:auth:myuser 86400 "some-new-random-token"
 # => "7292ed99-da44-42c3-897a-745b6e43ac33"
  22. 22. START job "A" I/O stdin, stdout, … SSH host "X" SEND queue "A:X" LISTEN queue “A:X” AUTHORIZE HTTP SSE Architecture for SSH I/O streaming
  23. 23. Thanks for listening No Shiba Inus were harmed in the making of this presentation

×