ML Workshop 1: A New Architecture for Machine Learning Logistics

2. © 2017 MapR Technologies 2 Contact Information Ted Dunning, PhD Chief Application Architect, MapR Technologies Committer, PMC member, board member, ASF O’Reilly author Email tdunning@mapr.com tdunning@apache.org Twitter @Ted_Dunning

11. © 2017 MapR Technologies 11 Some Key Points • Note that all models see identical inputs • All models run in production setting • All models send scores to same stream • The rendezvous server decides which scores to ignore • Roll forward, roll back, correlated comparison are all now trivial

14. © 2017 MapR Technologies 14 Quality & Reproducibility of Input Data is Important! • Recording raw-ish data is really a big deal – Data as seen by a model is worth gold – Data reconstructed later often has time-machine leaks – Databases were made for updates, streams are safer • Raw data is useful for non-ML cases as well (think flexibility) • Decoy model records training data as seen by models under development & evaluation

16. © 2017 MapR Technologies 16 What Does the Canary Do? • The canary is a real model, but is very rarely updated • The canary results are almost never used for decisioning • The virtue of the canary is stability • Comparing to the canary results gives insight into new models

17. © 2017 MapR Technologies 17 Isolated Development With Stream Replication Model 1 Model 2 Model 3 request Raw Add external data Input Internal 1 Internal 2 Internal 3 The world Model 4 Raw New external data Input Internal 4 Production Development

21. © 2017 MapR Technologies 21 Some Details • Inside the rendezvous server – Message contents … highlight return address – Rendezvous mailbox – Schedule ideas • Inside a model container – Identical inputs makes scaling easy – Nearly stateless models – Streaming shims, latency rig

22. © 2017 MapR Technologies 22 Message Content • Input request contains request data plus administrivia { timestamp: 1501020498314, messageId: "2a5f2b61fdd848d7954a51b49c2a9e2c", return: "proxy-217" provenance: { ... }, diagnostics: { ... }, ... application specific data here .. }

23. © 2017 MapR Technologies 23 Rendezvous Schedules • Simple part – Up to deadline, accept preferred models – Up to next deadline, accept more models – Near final deadline, accept default answer • But also some probabilistic choice • And also consider external experimental control – Inject as external state – Use in rendezvous to select model result – Open question how much power to expose

25. © 2017 MapR Technologies 25 Model Life Cycle • Developer / modeler produces container spec – And uses this to build their development article • QA inspects container spec – And uses this to build a test article • Security inspects container spec – And uses this to build final artifact • Important to use tools like Grafeas to inspect supply chain http://bit.ly/grafeas • Important that each step be inspectable

27. © 2017 MapR Technologies 27 Scaling Up • Note about streams – At millions of updates per server, the streams aren’t part of the streaming question • Scaling up state injection – Partition raw input, replicate state injector – Beware external throughput limits – State injection does avoid duplicate queries • Scaling up models – Stateless models allow trivial scaling – Sequence state typically also trivial to scale • Scaling up the rendezvous – Match partition on raw and scores – Replicate trivially

40. © 2017 MapR Technologies 40 Additional Resources O’Reilly report by Ted Dunning & Ellen Friedman © March 2017 Read free courtesy of MapR: https://mapr.com/geo-distribution-big-data-and-analytics/ O’Reilly book by Ted Dunning & Ellen Friedman © March 2016 Read free courtesy of MapR: https://mapr.com/streaming-architecture-using- apache-kafka-mapr-streams/

41. © 2017 MapR Technologies 41 Additional Resources O’Reilly book by Ted Dunning & Ellen Friedman © June 2014 Read free courtesy of MapR: https://mapr.com/practical-machine-learning- new-look-anomaly-detection/ O’Reilly book by Ellen Friedman & Ted Dunning © February 2014 Read free courtesy of MapR: https://mapr.com/practical-machine-learning/

42. © 2017 MapR Technologies 42 Additional Resources by Ellen Friedman 8 Aug 2017 on MapR blog: https://mapr.com/blog/tensorflow-mxnet-caffe-h2o-which-ml-best/ by Ted Dunning 13 Sept 2017 in InfoWorld: https://www.infoworld.com/article/3223 688/machine-learning/machine- learning-skills-for-software- engineers.html

43. © 2017 MapR Technologies 43 New book: Machine Learning Logistics Model Management in the Real World O’Reilly book by Ellen Friedman & Ted Dunning © Sept 2017 Download free from MapR http://info.mapr.com/2017_Content_Machine-Learning- Logistics_eBook_Prereg_RegistrationPage.html Going to Strata Data NYC? Book will be released 26 Sept 2017: Visit MapR booth for free book signings or to talk about logistics