Outfittery's mission is to provide relevant fashion to men. In the past it was our stylists that put together the best outfits for our customers. But since about a year ago we started to rely more on intelligent algorithms to augment our human experts. This transition to become a data driven company has left its marks on our IT landscape: In the beginning we just did simple A/B tests. Then we wanted to use more complex logic so we added a generic data enrichment layer. Later we also provided easy configurability to steer processes. And this in turn enabled us to orchestrate our machine learning algorithms as self contained Docker containers within a Kubernetes cluster. All in all it's a nice setup that we are pretty happy with. It then really took us some time to realise that we actually had built a delivery platform to deliver just any pure function that our data scientists come up with - directly into our microservices landscape. We just now started to use it that way; we just put their R&D experiments directly into production... :-) This talk will guide you through this journey, explain how this platform is built, and what we do with it.

1. Decision Making Based on Machine Learning
at .
Jesper Richter-Reichhelm (@jrirei)

2. In the beginning there was void and every customer was
seeing the same version of the website.

3. But then came along a Product manager and spoke:

4. “What is this?
I need to be able to test variants of our products.
I will introduce something!”

5. And she took an off-the-shelf solution and now every
customer could see a different version of the website.
And after it was done she was happy and
life was good.

6. But then came along a Data Scientist and spoke:

7. “What is this?
I need to be able to add complex algorithms
to steer business processes.
I will take this and make it better!”

8. And he hacked together wrote a complex decision system
that he would call Smart Gateway because it could steer
processes throughout all the company.
And after it was done both were happy and
life was good.

9. But then came along a Software Engineer and spoke:

10. “What is this?
If there would come just one strong wind,
everything would fall down and crash.
I will take this and make it better!”

11. And he would take the Smart Gateway and rewrite it from
scratch refactor it a bit to make it production ready.
And after it was done all three were happy and
life was good.

12. But then all three looked at the thing that they have
created together and spoke:

13. “What is this?
How did this become something so much more powerful
than what we set out to do in the first place?”

14. “Actually we have truly build a …”

15. Relevant fashion for men

17. Day 1

18. Day 1 Day 3

19. Day 1 Day 3 Day 4

20. Day 1 Day 3 Day 4 Day 5

21. Customer journey is quite “long”
Day 1 Day 3 Day 4 Day 5 Day 19
2-3 weeks to know what we’ve sold

22. Order economics demand complex tradeoffs
Cart Cost
Goods kept
Returned

23. Order economics demand complex tradeoffs
Cart Cost
operational profit
Goods kept
Buying
Logistics
Service
Other

24. Science
=
Data & Tech
Art
=
Stylist team
King
=
Customer

25. Science
=
Data & Tech
Art
=
Stylist team
Cart Cost
Data Driven Vision

26. AB Tests
Chapter 1 driven by Product

27. AB Tests with an external service

28. AB Tests with an external service

29. Canary testing with nginx

30. Canary testing with nginx

32. Problem: Parallel experiments with canary tests
A B

33. Problem: Parallel experiments with canary tests
AB BAAA BB

34. Problem: External tool can only track conversion

35. Problem: External tool can only track conversion
Day 1 Day 3 Day 4 Day 5 Day 19
2-3 weeks to know what we’ve sold

36. Problem: External tool does not support decision points
Cart Cost
operational profit
Goods kept
Buying
Logistics
Service
Other

37. Problem: External tool does not support decision points

38. Smart Gateway
Chapter 2 driven by Data Science

39. Goals
• Integration with IT platform
• Decision Points yield persistent results
• Complex decision making logic to allow process steering
• Configurable for flexibility

40. Smart Gateway is integrated in platform
Service
Service Smart Gateway
Service
REST
Black Box

41. Smart Gateways have persistence
Service
Service Smart Gateway
Service
Black Box
SQL

42. Smart Gateways are “smart”
• Data Scientists create an algorithm, called Decider
• Algorithm can have DWH access
• Algorithm is packed in a Docker container and listens to http calls

43. Smart Gateways are “smart”
Service
Service Smart Gateway
Service
Black Box
SQL

44. Smart Gateways are “smart”
Service
Service Smart Gateway
Service
Decider
Decider
Decider
DWH
http
SQL

45. Smart Gateways are configurable
• Sampling: routing of traffic to experiments
• Experiments can opt in independently
• Experiments define groups (random distribution)
• Groups map to a Decider

46. Smart Gateways are configurable
Decider
Decider
Decider
Decider
Decider
Decider
Sampling
Experiment
Experiment
25%
50% of “DE”
default
“A”
“B”
Age < 30 ? “A” : “B”
Age < 35 ? “A” : “B”
<something smart>
<some default>
50%
50%
50%
30%
20%

48. Problem: Data is not real time and can’t be consistent
Service
Service Smart Gateway
Service
Decider
Decider
Decider
DWH
data import
SQL

49. Problem: One single point of failure
Service
Service Smart Gateway
Service
Decider
Decider
Decider
DWH
SQL
all on one host
default
decision

50. Smart Gateway v2
Chapter 3 driven by IT

51. Goals
• Separation of concerns
• Data access is real time
• Production ready setup

52. Created a new team to shift responsibilities
Stylist Support
Logistics Support
Web
Data / ML
BI

53. Created a new team to shift responsibilities
Stylist Support
Logistics Support
Web
Data Delivery Data / ML
BI

54. Created a new team to shift responsibilities
Service
Service Smart Gateway
Service
Decider
Decider
Decider
DWH
SQL

55. Configuration is now DB based
• It simply got too big
• Decision Point
• n Experiments per Decision Point
• n Deciders per Experiment
• REST API to change config on the fly

56. Smart Gateways do “data enrichment”
• Deciders become pure functions
• Before sampling of traffic, call is enriched with data
• Enrichment is configurable
• Enriched data can be used to decide if experiments “opt in”

57. Smart Gateways now do “data enrichment”
Enrichment Sampling Experiment
Decider
Decider
DeciderRetrieve data:
• REST call
• constant value
• LocalStore (K/V)
• JS Script

58. Smart Gateways become production ready
• Professional Docker setup
• Orchestrating Docker containers via Kubernetes

59. Decider
Decider
Decider
Smart Gateway
Smart Gateways become production ready
Service
Service Smart Gateway
Service
Decider
Decider
Decider
SQL
Service
Service
SQL Redis
Data Job

60. Example use cases
• AB testing
• Decision Points
• User discrimination to prepayment
• … and many more
• Also very nice: shadow mode

62. Smart Gateways
Service
Service Smart GatewaySmart Gateway
Service
DeciderDecider
DeciderDecider
DeciderDecider
Service
Service
RedisSQL
Data Job

63. Smart Gateways are a runtime environment for pure functions
Service
Service Smart GatewaySmart Gateway
Service
Decider𝛌
Decider𝛌
Decider𝛌
Service
Service
RedisSQL
Data Job

64. Delivery platform for pure functions
Chapter 4 driven by Product, Data Science, and IT

65. Our own delivery platform for algorithms

66. Example 1: Supporting our stylists in decision making
Filters…
Customer
Info
Order /
Basket
Available Stock
• For every order
• For every article in stock
• Calculate a score how likely
customer is to keep this article
• Score is used to encourage or
discourage stylist to pick an article

67. Example 2: Scanning return slips
• External partner would take too
long
• Implemented algorithm on our own
• Every scan is handled via Smart
Gateway
• Information on follow on orders
is now available

68. Filters…
Customer
Info
Order /
Basket
Available Stock
Smart Gateway
120 sec execution time3 MB data (input / output)

69. Smart Gateway became asynchronous (if necessary)
Service
Service Smart GatewaySmart Gateway
Service
Decider𝛌
Decider𝛌
Decider𝛌
Service
Service
RedisSQL
Data Job

70. AWESOME

71. Conclusion
By me

72. But then all three looked at the thing that they have
created together and spoke:

73. “What is this?
How did this become something so much more powerful
than what we set out to do in the first place?”

74. “Actually we have truly build a system that can run
just any pure function.
AB tests and process steering are just the beginning.
Maybe speech recognition is next. Or maybe…”

75. With that they started to brainstorm all the cool things
they could do in the future.

76. Time and the next Hackathon will tell
what they made out of it.
And maybe I can come back next year
to tell you all about it.

77. The End

78. Questions?
Jesper Richter-Reichhelm (@jrirei)