Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

How we integrate Machine Learning Algorithms into our IT Platform at Outfittery

891 views

Published on

Outfittery's mission is to provide relevant fashion to men. In the past it was our stylists that put together the best outfits for our customers. But since about a year ago we started to rely more on intelligent algorithms to augment our human experts.

This transition to become a data driven company has left its marks on our IT landscape:
In the beginning we just did simple A/B tests. Then we wanted to use more complex logic so we added a generic data enrichment layer. Later we also provided easy configurability to steer processes. And this in turn enabled us to orchestrate our machine learning algorithms as self contained Docker containers within a Kubernetes cluster. All in all it's a nice setup that we are pretty happy with.
It then really took us some time to realise that we actually had built a delivery platform to deliver just any pure function that our data scientists come up with - directly into our microservice landscape. We just now started to use it that way; we just put their R&D experiments directly into production... :-)

This talk will guide you through this journey, explain how this platform is built, and what we do with it.

Published in: Technology
  • Be the first to comment

How we integrate Machine Learning Algorithms into our IT Platform at Outfittery

  1. 1. Jesper Richter-Reichhelm (@jrirei) How we integrate Machine Learning Algorithms
 into our IT Platform BI & Marketing - Architecture … with annotations! … with annotations! … with annotations! … with annotations!
  2. 2. In the beginning there was void and every customer was seeing the same version of the website. A quick trailer
 for the talk The whole content in just 2 minutes!
  3. 3. But then came along a Product manager and spoke:
  4. 4. “What is this? I need to be able to test variants of our products. I will introduce something!”
  5. 5. And she took an off-the-shelf solution and now every customer could see a different version of the website. And after it was done she was happy and
 life was good.
  6. 6. But then came along a Data Scientist and spoke:
  7. 7. “What is this? I need to be able to add complex algorithms
 to steer business processes. I will take this and make it better!”
  8. 8. And he hacked wrote a complex decision system that he would call Smart Gateway because it could steer processes throughout all the company. And after it was done both were happy and
 life was good.
  9. 9. But then came along a Software Engineer and spoke:
  10. 10. “What is this? If there would come just one strong wind,
 everything would fall down and crash. I will take this and make it better!”
  11. 11. And he would take the Smart Gateway and rewrite it from scratch refactor it a bit to make it production ready. And after it was done all three were happy and
 life was good.
  12. 12. But then all three looked at the thing that they have created together and spoke:
  13. 13. “What is this? How did this become something so much more powerful
 than what we set out to do in the first place?”
  14. 14. “Actually we have truly build a …” … read the whole talk to find out! :-) Hey, that was just a trailer…
  15. 15. Relevant fashion for men Short commercial after the trailer…
  16. 16. Outfittery’s business model: Customer - a man - orders a box. He may or may not schedule a talk to a stylist.
  17. 17. Day 1 Stylists pack an individual box
  18. 18. Day 1 Day 3 Box is sent to the customer
  19. 19. Day 1 Day 3 Day 4 Customer is happy to unpack the box
  20. 20. Day 1 Day 3 Day 4 Day 5 … but is too busy to send his return back quickly
  21. 21. Customer journey is quite “long” Day 1 Day 3 Day 4 Day 5 Day 19 2-3 weeks to know what we’ve sold
  22. 22. Order economics demand complex tradeoffs Cart Cost Goods kept Returned
  23. 23. Order economics demand complex tradeoffs Cart Cost operational profit Goods kept Buying Logistics Service Other
  24. 24. Art = Stylist team King = Customer
  25. 25. Science = Data & Tech Art = Stylist team King = Customer
  26. 26. Science = Data & Tech Art = Stylist team Cart Cost Data Driven Vision
  27. 27. AB Tests Chapter 1 driven by Product
  28. 28. AB Tests with an external service It’s a good service, easy to setup and use.
  29. 29. AB Tests with an external service But it’s focused on conversion.
  30. 30. Canary testing with nginx For more complex ab tests …
  31. 31. Canary testing with nginx … we deploy multiple version of the website.
  32. 32. It works, but…
  33. 33. Problem: Parallel experiments with canary tests A B This is nice for a single test…
  34. 34. Problem: Parallel experiments with canary tests AB BAAA BB … but it’s hell to run multiple tests in parallel!
  35. 35. Problem: External tool can only track conversion This works well for conversion tests…
  36. 36. Problem: External tool can only track conversion Day 1 Day 3 Day 4 Day 5 Day 19 2-3 weeks to know what we’ve sold … but it fails when faced with our long lead times.
  37. 37. Problem: External tool does not support decision points Cart Cost operational profit Goods kept Buying Logistics Service Other … but it fails when handling different KPIs.
  38. 38. Problem: External tool does not support decision points Also: We not only do AB tests, … we want decision points to steer processes.
  39. 39. Smart Gateway Chapter 2 driven by Data Science
  40. 40. Goals • Integration with IT platform • Decision Points yield persistent results • Complex decision making logic to allow process steering • Configurable for flexibility
  41. 41. Smart Gateway is integrated in platform Service Service Smart Gateway Service REST Black Box Implemen- tation handled as black box On the left: responsibility
 of IT (CTO) On the right: responsibility of Data (CDO)
  42. 42. Smart Gateways have persistence Service Service Smart Gateway Service Black Box SQL
  43. 43. Smart Gateways are “smart” • Data Scientists create an algorithm, called Decider • Algorithm can have DWH access • Algorithm is packed in a Docker container and listens to http calls
  44. 44. Smart Gateways are “smart” Service Service Smart Gateway Service Black Box SQL
  45. 45. Smart Gateways are “smart” Service Service Smart Gateway Service Decider Decider Decider DWH http SQL Inside the
 “black box”
  46. 46. Smart Gateways are configurable • Sampling: routing of traffic to experiments • Experiments can opt in independently • Experiments define groups (random distribution) • Groups map to a Decider … all in one gigantic
 YAML file. :-)
  47. 47. Smart Gateways are configurable Phase 1: Sample traffic to experiments Phase 2:
 Treatment by a Decider
  48. 48. Worked nice for about 1 year,
 but …
  49. 49. Problem: Data is not real time and can’t be consistent Service Service Smart Gateway Service Decider Decider Decider DWH data import SQL Break consistency and not applicable for real time Data imports are periodic batch jobs.
  50. 50. Problem: One single point of failure Service Service Smart Gateway Service Decider Decider Decider DWH SQL all on one host default
 decision … which essentially breaks the “black box”. This forces clients to provide default values
  51. 51. Smart Gateway v2 Chapter 3 driven by IT
  52. 52. Goals • Separation of concerns • Data access is real time • Production ready setup
  53. 53. Created a new team to shift responsibilities Stylist Support Logistics Support Web Data / ML BI
  54. 54. Created a new team to shift responsibilities Stylist Support Logistics Support Web Data Delivery Data / ML BI A new IT team… … sitting “within” the Data department.
  55. 55. Created a new team to shift responsibilities Service Service Smart Gateway Service Decider Decider Decider DWH SQL New team took over Smart Gateway
 (dev + ops)
  56. 56. Configuration is now DB based • It simply got too big • Decision Point • n Experiments per Decision Point • n Deciders per Experiment • REST API to change config on the fly Data dep. can update config / deciders “on the fly”. Bye bye gigantic YAML file!
  57. 57. Smart Gateways do “data enrichment” • Deciders become pure functions • Before sampling of traffic, call is enriched with data • Enrichment is configurable • Enriched data can be used to decide if experiments “opt in” Data consistency and real time can be guaranteed. IT is now responsible for providing data:
  58. 58. Smart Gateways now do “data enrichment” Enrichment Sampling Treatment Decider Decider DeciderRetrieve data: • REST call • constant value • LocalStore (K/V) • JS Script Routing to experiment: • filtering possible • “once only” possible Group assignment: • defined group size • random selection Added a new “data enrichment” phase
  59. 59. Smart Gateways become production ready • Professional Docker setup • Orchestrating Docker containers via Kubernetes
  60. 60. Decider Decider Decider Smart Gateway Smart Gateways become production ready Service Service Smart Gateway Service Decider Decider Decider SQLRedis Service Service SQL
  61. 61. Example use cases • AB testing • Decision Points • User discrimination to prepayment • … and many more
  62. 62. Current status: We are happy!
  63. 63. Smart Gateways Service Service Smart GatewaySmart Gateway Service DeciderDecider DeciderDecider DeciderDecider Service Service RedisSQL Data Job But we can do more:
  64. 64. Smart Gateways are a runtime environment for pure functions Service Service Smart GatewaySmart Gateway Service Decider𝛌 Decider𝛌 Decider𝛌 Service Service RedisSQL Data Job Pure functions can do more than just “decide”!
  65. 65. Delivery platform for pure functions Chapter 4 driven by Product, Data Science, and IT
  66. 66. Our own delivery platform for algorithms Ok: AWS Lambda is more powerful but our solution is deeply integrated.
  67. 67. Example 1: Supporting our stylists in decision making AI helping stylists to make better decisions Sorry, no real screenshots in the public. :-)
  68. 68. Example 1: Supporting our stylists in decision making • For every order • For every article in stock • Calculate a score how likely customer is to keep this article • Score is used to encourage or discourage stylist to pick that article Output is a scoring of articles for that specific box
  69. 69. Example 1: Supporting our stylists in decision making
  70. 70. Example 2: Scanning return slips Closing a process gap - fast!
  71. 71. Example 2: Return slip scans • External partner would take too long • Implemented algorithm on our own • Every scan is handled via Smart Gateway • Information on follow on orders is now available Output: Which checkboxes were checked
  72. 72. Smart Gateway became asynchronous (if necessary) Service Service Smart GatewaySmart Gateway Service Decider𝛌 Decider𝛌 Decider𝛌 Service Service RedisSQL Data Job These “jobs” needed async handling: We use RabbitMQ and callbacks.
  73. 73. Smart Gateway became asynchronous (if necessary) 60 sec execution time only 5 sec execution time but 10 MB data (input / output) Jobs with long execution time or quite big data reqs can be handled now.
  74. 74. Conclusion By me
  75. 75. But then all three looked at the thing that they have created together and spoke: The talk was held at a cinema so let’s stick with the theme.
  76. 76. “What is this? How did this become something so much more powerful
 than what we set out to do in the first place?”
  77. 77. “Actually we have truly build a system that can run
 just any pure function. AB tests and process steering are just the beginning. Maybe speech recognition is next. Or maybe…”
  78. 78. With that they started to brainstorm all the cool things they could do in the future.
  79. 79. Time and the next Hackathon will tell
 what they made out of it. And maybe I can come back next year
 to tell you all about it.
  80. 80. The End
  81. 81. Questions? Jesper Richter-Reichhelm (@jrirei) Feel free to contact me if you have questions!

×