Slides of the first talk of the "Paris AI in production" meetup, held in Paris on 12 October 2018.

2. How to deploy a new model in
production without overhead
Laura Calem - AI in Production meetup - 12.10.18

3. Train Prod

4. Train Prod
Dataset
Versionning
Reproducibility
Actual training

5. Train Prod
Deployment
Scalability
Visibility
Resources access

6. Train Prod

7. Train Prod

8. Train Prod

10. Use Docker
1

15. New models
2

16. 2.1
Have an external
module with a
Batcher class

17. 2.1
Have an external
module with a
Batcher class
Mother class

18. 2.1
Have an external
module with a
Batcher class
Mother class
ZMQ server

19. 2.1
Have an external
module with a
Batcher class
Mother class
ZMQ server
Will handle receiving data

20. 2.1
Have an external
module with a
Batcher class
Mother class
ZMQ server
Will handle receiving data
Call a process_batch class

21. 2.1
Have an external
module with a
Batcher class
Mother class
ZMQ server
Will handle receiving data
Call a process_batch class
Handle sending response

22. 2.1 Batcher mother class

23. 2.1 Batcher mother class
class Batcher(object):
def __init__(self):
…

24. 2.1 Batcher mother class
class Batcher(object):
def __init__(self):
…
def get_parser(self):
…

25. 2.1 Batcher mother class
class Batcher(object):
def __init__(self):
…
def get_parser(self):
…
def run_server(self):
…

26. 2.1 Batcher mother class
class Batcher(object):
def __init__(self):

27. 2.1 Batcher mother class
class Batcher(object):
def __init__(self):
parser = self.get_parser()
self.args = parser.parse_args()
# augment args if needed

28. 2.1 Batcher mother class
class Batcher(object):
def __init__(self):
parser = self.get_parser()
self.args = parser.parse_args()
# augment args if needed
self.batcher_init() # will be redefined
self.host = self.args.host
self.port = self.args.port

29. 2.1 Batcher mother class
class Batcher(object):
def run_server(self):

30. 2.1 Batcher mother class
class Batcher(object):
def run_server(self):
self.context = zmq.Context()
self.server = self.context.socket(zmq.REP)

31. 2.1 Batcher mother class
class Batcher(object):
def run_server(self):
self.context = zmq.Context()
self.server = self.context.socket(zmq.REP)
zmq_bind_addr = "%s:%s" % (self.host, self.port)
self.server.bind(zmq_bind_addr)

32. 2.1 Batcher mother class
class Batcher(object):
def run_server(self):
self.context = zmq.Context()
self.server = self.context.socket(zmq.REP)
zmq_bind_addr = "%s:%s" % (self.host, self.port)
self.server.bind(zmq_bind_addr)
while True:

33. 2.1 Batcher mother class
class Batcher(object):
def run_server(self):
self.context = zmq.Context()
self.server = self.context.socket(zmq.REP)
zmq_bind_addr = "%s:%s" % (self.host, self.port)
self.server.bind(zmq_bind_addr)
while True:
message = self.server.recv()

34. 2.1 Batcher mother class
class Batcher(object):
def run_server(self):
self.context = zmq.Context()
self.server = self.context.socket(zmq.REP)
zmq_bind_addr = "%s:%s" % (self.host, self.port)
self.server.bind(zmq_bind_addr)
while True:
message = self.server.recv()
response = self.handle_request(message)

35. 2.1 Batcher mother class
class Batcher(object):
def run_server(self):
self.context = zmq.Context()
self.server = self.context.socket(zmq.REP)
zmq_bind_addr = "%s:%s" % (self.host, self.port)
self.server.bind(zmq_bind_addr)
while True:
message = self.server.recv()
response = self.handle_request(message)
self.server.send(response)

36. 2.1 Batcher mother class
class Batcher(object):
def handle_request(self, message):

37. 2.1 Batcher mother class
class Batcher(object):
def handle_request(self, message):
# handle message as images / filter / etc
batch = self.create_batch(message)

38. 2.1 Batcher mother class
class Batcher(object):
def handle_request(self, message):
# handle message as images / filter / etc
batch = self.create_batch(message)
result = self.process_batch(batch) # will be redefined

39. 2.1 Batcher mother class
class Batcher(object):
def handle_request(self, message):
# handle message as images / filter / etc
batch = self.create_batch(message)
result = self.process_batch(batch) # will be redefined
# format result into zmq response
return self.format_result(result)

40. 2.1 Batcher mother class
class Batcher(object):
def __init__(self):
…
def get_parser(self):
…
def run_server(self):
…

41. 2.2
Derive Batcher
class in your new
module

42. 2.2
Derive Batcher
class in your new
module
batcher_init

43. 2.2
Derive Batcher
class in your new
module
batcher_init
process_batch

44. 2.2
Derive Batcher
class in your new
module
batcher_init
process_batch
entrypoint

45. 2.2 Batcher child class

46. 2.2 Batcher child class
class MyBatcher(Batcher):
def batcher_init(self):

47. 2.2 Batcher child class
class MyBatcher(Batcher):
def batcher_init(self):
# extract model config from args
config = self.make_config(self.args)

48. 2.2 Batcher child class
class MyBatcher(Batcher):
def batcher_init(self):
# extract model config from args
config = self.make_config(self.args)
self.model = MyModel(config)

49. 2.2 Batcher child class
class MyBatcher(Batcher):
def process_batch(self, batch):

50. 2.2 Batcher child class
class MyBatcher(Batcher):
def process_batch(self, batch):
pred = self.model.predict(batch)

51. 2.2 Batcher child class
class MyBatcher(Batcher):
def process_batch(self, batch):
tic = time.time()
pred = self.model.predict(batch)
toc = time.time()
pred_time = toc - tic

52. 2.2 Batcher child class
class MyBatcher(Batcher):
def process_batch(self, batch):
tic = time.time()
pred = self.model.predict(batch)
toc = time.time()
pred_time = toc - tic
return self.format_result(pred, pred_time)

53. 2.2 Batcher child class - Entrypoint
if __name__ == "__main__":

54. 2.2 Batcher child class - Entrypoint
if __name__ == "__main__":
batcher = MyBatcher()
batcher.run_server()

55. Production pipeline
3

60. One api to rule
them all

61. One api to rule
them all
docker compose

62. One api to rule
them all
docker compose
api waits for every container

63. One api to rule
them all
docker compose
api waits for every container
api gets inference requests

64. One api to rule
them all
docker compose
api waits for every container
api gets inference requests
api talks to containers

66. Final words

68. lvl 1

69. lvl 1
Manual training

70. lvl 1
Manual training
Script for inference

71. lvl 1
Manual training
Script for inference
No real prod

72. lvl 1
Manual training
Script for inference
No real prod
lvl 2

73. lvl 1
Manual training
Script for inference
No real prod
lvl 2

74. lvl 1
Manual training
Script for inference
No real prod
lvl 2
Manual training

75. lvl 1
Manual training
Script for inference
No real prod
lvl 2
Manual training
Prod pipeline

76. lvl 1
Manual training
Script for inference
No real prod
lvl 2
Manual training
Prod pipeline
Fixed

77. lvl 1
Manual training
Script for inference
No real prod
lvl 2
Manual training
Prod pipeline
Fixed
lvl 3

78. lvl 1
Manual training
Script for inference
No real prod
lvl 2
Manual training
Prod pipeline
Fixed
lvl 3
Training pipeline

79. lvl 1
Manual training
Script for inference
No real prod
lvl 2
Manual training
Prod pipeline
Fixed
lvl 3
Training pipeline
Prod pipeline

80. lvl 1
Manual training
Script for inference
No real prod
lvl 2
Manual training
Prod pipeline
Fixed
lvl 3
Training pipeline
Prod pipeline
Distributed

81. lvl 1
Manual training
Script for inference
No real prod
lvl 2
Manual training
Prod pipeline
Fixed
lvl 3
Training pipeline
Prod pipeline
Distributed
Celery
MRQ
Python RQ

82. Thank you!