Recommended
Recommended
More Related Content
What's hot
What's hot (19)
Similar to devopsdays Amsterdam 2018 - Chaos while deploying AI and making sure it doesn’t hurt your app
Similar to devopsdays Amsterdam 2018 - Chaos while deploying AI and making sure it doesn’t hurt your app (20)
Recently uploaded
Recently uploaded (20)
devopsdays Amsterdam 2018 - Chaos while deploying AI and making sure it doesn’t hurt your app
- 1. #dodams @thiagoavadore Thiago de Faria DataOps & AI Lead @LINKIT Chaos while deploying AI and making sure it doesn’t hurt your app
- 6. #dodams @thiagoavadore GOOGLE AI MAKES PHONE CALLS??!! CAN YOU DO SOMETHING LIKE THAT USING THAT TENSORFLOW STUFF? IT WOULD REALLY HELP US GET MORE FUNDING. - PERSON A, VP OF PRODUCT, LAST WEEK
- 7. #dodams @thiagoavadore WAIT… CAN YOU ALSO DO THAT IN A CONTAINER USING BLOCKCHAIN? - PERSON B, ONE DAY LATER * PERSON B WAS APPOINTED VP OF PRODUCT
- 8. #dodams @thiagoavadore SO OBVIOUS! WE WILL CREATE AN AI DEVOPS CLOUD NATIVE APPLICATION ON KUBERNETES USING BLOCKCHAIN THAT THE USER WILL EXPERIENCE WITH A VR HEADSET AND THIS WILL BE OUR DIGITAL TRANSFORMATION DISRUPTIVE STRATEGY! - PERSON C, YESTERDAY * PERSON C IS THE NEW CEO
- 9. @thiagoavadore#dodams AI MAKE COMPUTERS CAPABLE OF DOING THINGS THAT WHEN DONE BY A HUMAN, WOULD BE THOUGHT TO REQUIRE INTELLIGENCE
- 10. #dodams @thiagoavadore AI MACHIN E LEARNIN G MAKE MACHINES FIND PATTERNS WITHOUT EXPLICITLY PROGRAMMING THEM TO DO SO
- 13. #dodams @thiagoavadore WHILE YOU ROCK & THROUGHPUT INCREASES, A DATA SCIENTIST BUILDS A MODEL…
- 15. #dodams @thiagoavadore CSV's ON A LAPTOP..
- 17. #dodams @thiagoavadore SHOW TO THE CEO!
- 18. @thiagoavadore#dodams WE ARE AN AI COMPANY!
- 20. @thiagoavadore#dodams THE TIMELESS CONFLICT PART II - THE RETURN OF AI IT Ops Dev ML Dev
- 21. #dodams @thiagoavadore PROBLEMS I HAVE SEEN (some I may have even caused…)
- 22. @thiagoavadore#dodams DOING WHAT YOU ARE NOT GOOD & HIRED FOR…
- 30. #dodams @thiagoavadore DEMO GODS, be kind…
- 31. @thiagoavadore#dodams The amazing pictures are from the great JD Hancock! http://photos.jdhancock.com/
Editor's Notes
- All jokes aside like this twitter we all know that Devops is not an end state and never claimed to be… that is why we have thousands of definitions and they are all right In the end, Devops is combination of practices that want to improve your flow, feedback and continuous learning… and to avoid the inherent confront… —>
- In almost every IT organization, there is an inherent conflict between Development and IT Operations which creates a downward spiral, resulting in ever-slower time to market for new products and features, reduced quality, increased outages, and, worst of all, an ever-increasing amount of technical debt.
- Peace in the galaxy!
- But don’t ge me wrong we are surrounded by hype, not just on AI and surrounded by posing that we are something that we are not…
- And also this… with that person being the one that decides the products you build. But joke aside, you are probably working with something called AI, using ML. If not, there is a big chance that something with it will be in your PROD in the upcoming months… So first, let’s try to clarify some concepts and understand what is AI, ML, DL and cases that you should use it!
- But that is not everything It is not just dev and ops… we have been adding practices to involved the business, QA, infused and all the comes with it And now we have the Data Scientists, ML engineers and more people trying to bring their models to production And why we must talk about this? Well… because there is a hype. The same way that people declare Devops’cy, people are declaring they are an AI company now!
- But jokes aside, you are here because you trust the movement and you want to share knowledge. On your daily work you try to apply the practices and adjust them to your situation … you may have a good well structure delivery pipeline, engineers interested in the business, being on call is not a huge drama anymore You have a nice team that works together and does blameless post-mortems, open communication and all works well… We took into account dev, ops, qa, infosec and others and devops started. You now know about empathy, optimising the flow, reducing the number of handoffs You treat infrastructure as code You deploy on Kubernetes You have Continuous Integration, Build and Release
- And how usually data scientist build models??
- When building a model you need data… and that is the most fundamental part. You arrive at a new problem and you need a data dump, discussing with the business and fetching information. You quickly find out that your data resides in different systems and there is no common ground between them… yeah, sure! There is a data warehouse, but one per department… and your project needs lots of things! You spend weeks fetching that data from different systems, stitching them together and getting dumps from different users. You end up with a csv, json, orc or parquet file to build your model. You save the steps and people you reached out and you think to yourself… I am going to sort this out when I finish the model… first I need to check if there is something here, right? You end up with this file on your local machine or saved on a fileserver..
- In a parallel process you start to evaluate your model… Classifying, regression, clustering… try out some simple GLM, go to decision trees, give some intermediate feedback and move on… You usually don’t know anyone from the Ops department, you don’t know how to deploy this in a Kubernetes cluster so you just stick with training it on your Python virtual env while talking to the business. You do this for months… you even say you use Scrum because you have a Jira board, but we all know that you just have tickets like “Improve Model” … like, what is DoD of that no one knows! Anyhow, you make it work and you end with a model that predicts or classifies well your dataset! Tears of joy! —>
- You make a presentation, release and show off your findings to the CEO! You explain that your model is stable, that you have a training set, a validation set and a test set… you used cross validation. You show stuff around like AUC, accuracy, confusion matrix and everyone is happy!
- The CEO calls a PR, announces that we are a bastion of light for the future and blah blah blah… we are an AI company now!
- And here we are again… another wall appeared and something comes up with a model that you have to find a way to deploy! Sounds familiar right? Now developers have to figure it out how to work on that models and turn into a serving system… Operations people need to figure it out where and how to run it… how to monitor and operate this. And you have to deploy a model that has been under development for weeks or months … the data scientist doesn’t know how to server it, should we do RPC or API… how many requests. You are again with one of the huge problems that needed to be addressed in the Phoenix Project and DevOps Handbook - Large Batches going through your pipeline and many others!! Sounds familiar?
- We already addressed this problem before right? The 3 ways, improving the flow, feedback and X. But not for ML usually. How to address that? First, for that flow that we saw the data scientist going from his laptop to the ppt presentation… First, we need to treat Analytics and ML models building as we do with application code… source control, small batches, being able to toggle features on production, decrease the handoffs and many other good practices that are being the new Standar! Why? Because I have seen problems… lots of problems! —>
- The amount of resources and time spent building models that never see the light of production or do not actually solve a business issue is amazing… for example…
- The mist ouch between business, marketing and ML building is very normal… there are innumerous cases where the Data Scientist does not have a support and he only thinks about the model quality part. He creates an amazing model with large accuracy but when he doesn’t know how/when to deploy it… the Operations team has no clue about the code and how that works. It ends up with the Data Scientist having to do tasks that he doesn’t know and the Ops vice-versa… this results in long-time to market and poor quality deployments. I have been in companies where ops teams were trying to deploy models that were built 6 months ago… 6 months!
- You get data from different systems because you never knew if this was actually going to be valuable and/or go to production… You have to join data from different systems, make assumptions, do some feature engineering to build the accurate model you showed to the CEO. But know you want to retrain/calibrate your model every day or week or month and building this data pipeline is quite hard and unstable… well, you can’t ask a data scientist to be a data engineer all of a sudden… when they are able to achieve this, you may end up with a feature engineering pipeline that won’t scale well and hard to operate…
- When the model was built, the idea is to run in batches… Now we have this amazing Kafka cluster and we just want to plug in the model to a consumer right? That should work huh?? The architect just so easily showed to the CEO that they just need to draw a new line… This involves a lot of things! Yeah, you can train and batches and serve to a streaming… but your data pipeline to be able to call the ML model server needs to be rebuild… you may have to join several data streams, change some things and retrain the model…
- As well as servers that have configuration drift or processes that change within time ML models also drift from the initial part. You may notice that the accuracy of your model decreases within time… this may be because your model is impacting the way that people interact with the application, so now the features you had are not as relevant anymore… And you may also have models that are fragile, especially when you depend on external data… let’s say that one of the important features of your model is GDP per capita. But now Trump decides that the calculation of GDP will change… how bad your model will be able to cope with it? Model drift and fragility mean that you need to calibrate your model constantly, add new features and data inputs or even completely rebuild your model and network… the work is never 100% finished!
- You hire new ML engineers or Data Scientists… or they switch project. The previous team working on this model only has 2 commits for 3 months… so what did they do? Which models they tried? This is very common… Data Scientists tend to work 1 or 2 sprints in a model trying to improve it without pushing anything to source control. Some of them are used to only showing models that are 95% accuracy or more… They may have tried 20 different algorithms with more than 20 combinations of hyper parameters each… so they ran over 400 models, but you only have access to 2-3 per month. This is terrible because new people that join the team will try out the same things and have the same mistakes before, wasting time and effort.
- The accuracy and interaction of end-users with ML models is hard to evaluate… I told someone that this picture has a cat… was that useful? I will never automatically know… I need to have lots of telemetry and evaluate proxy usages: Since the model started, do our acquisition pipeline changed? Are people using more/less the related features to it? Has the time on app changed? Do I see spikes on requests? What may have changed? You need to have lots of telemetry, otherwise you will only know after the fact… this is for everything, I know… but here you will never know if your models is right or not, unless someone complains!
- To tackle these problems we must first address the culture change and all things related to Devops… batch sizes, communication and feedback, telemetry and so on!
- We need to automate better the processes involved in ML building and help data scientists to build integration tests, regression tests and a continuous delivery pipeline This means that every models should be production ready for day one… so define first how you will serve it and build that pipeline before hand. When having a satisfactory dataset, a data scientist can build something that is 70% accurate for lots of problems in a question of days…