RedisConf19

1. PRESENTED BY
Reinforcement learning on
hundreds of thousands of cores
Henrique Pondé de Oliveira Pinto
@henriqueponde
OpenAI, Dota Team

2. Agenda
Dota + OpenAI
Training state of the art neural network
Why Redis? 3 problems and solutions
Future work ahead

3. Dota + OpenAI

4. Dota

5. OpenAI
Research Organization focused on
Machine Learning.
Mission: Ensure that Artificial General
Intelligence benefits all of humanity.

6. Chess, GO, Dota
Length of games (# of moves):
• Chess: 40
• GO: 150
• Dota: 20,000
Moves/Turn (to consider):
• Chess: 35
• GO: 250
• Dota: 1000
VA LV E
R

7. Chess, GO, Dota
Unique challenges in Dota:
• Requires coordination
• Imperfect information
VA LV E
R

8. Dota Championship 2018

9. Dota Championship 2018

10. How do you learn to play Dota? - Reinforcement Learning
ACT
OBSERVE
OPTIMIZE

11. How do you learn to play Dota? - Reinforcement Learning
• Agent: OpenAI Five
• Environment: Dota
• 180 years of games/day
ACT
OBSERVE
OPTIMIZE

12. Reinforcement Learning - Humanoid

13. Reinforcement Learning - OpenAI Five

14. Training state of the art neural network

15. Training Setup
ROLLOUT

16. Training Setup
ROLLOUT ROLLOUTROLLOUT
…

17. Training Setup
ROLLOUTS OPTIMIZERS

18. Training Setup
ROLLOUTS OPTIMIZERS
100,000x
CPU cores
100x
GPUs

19. Training Setup
ROLLOUTS OPTIMIZERS
?
100,000x 100x

20. Training Setup
ROLLOUTS OPTIMIZERS
?
CONTROLLER

21. Training Setup
ROLLOUTS OPTIMIZERS
CONTROLLER
Config

22. Training Setup
ROLLOUTS OPTIMIZERS
CONTROLLER
Observations

23. Training Setup
ROLLOUTS OPTIMIZERS
CONTROLLER
Parameters

24. Why Redis?
Storage of state

25. Redis - Storage of state
ROLLOUTS OPTIMIZERS
CONTROLLER
Config

26. Redis - Storage of state
ROLLOUTS OPTIMIZERS
CONTROLLER
Parameters

27. Redis - Storage of state
• Holds all the “state” of the training
• Config
• Parameters
• Single source of truth
• Redis: Easy to create disk backup

28. Redis - Storage of state
Redis RDB files act as a commit history of multiple
training runs
big-run
idea-1
idea-2

29. Why Redis?
Streams of Data

30. Redis - Streams of Data
ROLLOUTS OPTIMIZERS
CONTROLLER
Parameters
100,000x 100x
Parameters
~500 MB

31. Redis - Streams of Data
ROLLOUTS
OPTIMIZER
…
CONTROLLER
SUBCONTROLLER
ROLLOUTS
OPTIMIZER
SUBCONTROLLER

32. Redis - Streams of Data
…
CONTROLLER

33. Redis - Streams of Data
…
CONTROLLER
Parameters

34. Why Redis?
Distributed Coordination

35. Redis - Distributed Coordination
Redis is single threaded. That can be used to solve
coordination problems across machines.

36. Redis - Distributed Coordination
Redis is single threaded. That can be used to solve
coordination problems across machines.
Can we build a distributed lock with Redis?

37. Redis - Distributed Coordination
Redis is single threaded. That can be used to solve
coordination problems across machines.
Can we build a distributed lock with Redis?
Possible solution:
• Single key: ‘lock’
• 0: unlocked
• 1: locked

38. Redis - Distributed Coordination
class Lock(object):
"""
A shared, distributed Lock. Using Redis for locking allows the Lock
to be shared across processes and/or machines.
It's left to the user to resolve deadlock issues and make sure
multiple clients play nicely together.
"""

39. A deeper look into Lock
class Lock(object):
…
LUA_RELEASE_SCRIPT = """
local token = redis.call('get', KEYS[1])
if not token or token ~= ARGV[1] then
return 0
end
redis.call('del', KEYS[1])
return 1
"""
…
LUA_REACQUIRE_SCRIPT = """
…

40. Transactions
GET treats, SET treats+1 GET treats, SET treats+1

41. Transactions
0
0
0
GET
GET

42. Transactions
0
0
0
GET
GET
SET 1
SET 1
1
OK
OK

43. Redis Transactions
Commands:
• WATCH
• MULTI
• EXEC
Downsides:
• Transactions can fail
• Hard to reason about

44. Redis Transactions
0 WATCH
SET 1
GET
MULTI
EXEC

45. Using Lua Scripts
0

46. Using Lua Scripts
0
1
2

47. Lua Scripts >> Transactions
“…everything you can do with a Redis transaction,
you can also do with a script, and usually the script
will be both simpler and faster.”
- https://redis.io/topics/transactions

48. Increment Lua Script
—- Increments number of treats
local num_treats = redis.call('get', 'treats')
num_treats = num_treats + 1
return redis.call('set', 'treats', num_treats)

49. How Lua Scripts solve Distributed Problems
…
CONTROLLER
Parameters

50. Lua Scripts - Leader Election
• Problem: Multiple identical nodes. Who is the leader?
• Requirement: All nodes must agree on one leader

51. Lua Scripts - Leader Election
• Problem: Multiple identical nodes. Who is the leader?
• Requirement: All nodes must agree on one leader
• Possible solution: “leader” and “members” keys.

52. Lua Scripts - Leader Election
local node_id = …
local leader_exists = redis.call('exists', 'leader')
if leader_exists then
redis.call('sadd', 'members', node_id)
return 0
else
redis.call('set', 'leader', node_id)
return 1
end

53. How Lua Scripts solve Distributed Problems
…
CONTROLLER

54. Lua Scripts - Load Balancer
• Problem: How do we assign subcontrollers to rollouts?
• Requirement: The load should be evenly split

55. Lua Scripts - Load Balancer
• Problem: How do we assign subcontrollers to rollouts?
• Requirement: The load should be evenly split
• Possible solution: Sorted set for number of connected rollouts

56. Lua Scripts - Load Balancer
-- subcontrollers sorted set key
local zkey = …
-- zrange returns a sorted array from lowest score to highest
local subcontroller = redis.call('zrange', zkey, 0, 0)[1]
-- Increment the count of the subcontroller
redis.call('zincrby', zkey, 1, subcontroller)
return subcontroller

57. Why Lua Scripts are amazing
• Better version of Transactions
• Execute arbitrary logic in the Redis itself
• Multiple inputs
• Multiple outputs

58. Future work ahead
Can we scale 10x?

59. Back to the Experiment Setup
…
CONTROLLER

60. Simulations: How slow does it get?

61. Simulations: Why is the setup time so high?
All configs are in Controller Redis
Any new instance must pull its job from it.
Crash loop nightmare:
• Instance wakes up, asks for config
• Timeout
• Crash
• Instance restarts, asks for config

62. Simulations: Backing-off

63. Redis-Proxy
…
CONTROLLER

64. Redis-Proxy
…
PROXY PROXY
CONTROLLER

65. Redis-Proxy
PROXY
CONTROLLER
• Reduces number of connections
• Caches responses
• Allows request prioritization

66. Redis-Proxy

67. Rapid: Distributed Training Framework

68. Rapid: Distributed Training Framework
Dota

69. Rapid: Distributed Training Framework
RoboticsDota

70. Wrapping up
• How to train a neural network at scale
• Redis
• Storage (Checkpointing)
• Streams
• Lua Scripts (Distributed Coordination)
• Future work ahead
• https://openai.com/blog/openai-five-finals/
• https://www.twitch.tv/openai

71. Thank you!

72. Questions and Answers
More information
• https://openai.com/blog/openai-five-finals/
• https://www.twitch.tv/openai
@OPENAI
@HENRIQUEPONDE