Recommended
Recommended
More Related Content
Similar to Angular & TensorflowJs
Similar to Angular & TensorflowJs (6)
Recently uploaded
Recently uploaded (20)
Angular & TensorflowJs
- 1. 0.64 0.92 0.94 dining bottle bottle 0.76 chair 0.78 @mauriziovitale_ github.com/mauriziovitale/angulartensorflowjs maurizio.dev & Tech Lead Maurizio Vitale
- 2. Agenda • Searching images by content • TensorflowJs • Demo • Code &@mauriziovitale_
- 7. &@mauriziovitale_ • Useless name • Missing ML metadata • Manual metadata Problem
- 8. Cloud store Machine Learning Metadata Search by content &@mauriziovitale_ Store Image Add metadata Save metadata Search image Batch of images InputOutput Google Firestorage TensorflowJs Model Google Firestore Angular
- 10. &@mauriziovitale_ Project setup • Create a project • Install • Angularfire2 • @tensorflow-model/ coco-ssd • Create a module
- 11. Import Modules • Angularfire2 • Angular Firestore • Angular Storage &@mauriziovitale_
- 12. &@mauriziovitale_ Html Ts Store Image • Generate Component • Store file in Firestorage
- 13. &@mauriziovitale_ Save info file • Save on Firestore
- 14. Preview Images • Get reference of the collection • Recognize the classes &@mauriziovitale_
- 15. Recognition Service • Load the model • Use it &@mauriziovitale_
- 16. Search by content • Add the input • Filter the class array &@mauriziovitale_
- 17. &@mauriziovitale_ Conclusion Why should I use TensorflowJs ? • Build a custom model • No network latency • Content Privacy • Offline
Editor's Notes
- Good afternoon, today I will start by giving you a little introduction about the issue of searching for images by content. Then I will show you how TensorflowJs can help us to solve it. And finally, I will show you a demo of how the final application works and how to code it.
- Searching is the most common activity that users do on the web. And these are the current hot trend keywords coming from Google Trends right now. As you can see, people are searching for football teams, celebrities and recent events. When it comes to searching for images, it is the content within the images that people are interested in. For example, which objects do you recognize in this image? We can see a chair, a dinner table, two bottles of sauce and a sandwich. Each of these items can be represented in JSON format. This is how we store the info of an object, the position, the name and a score. The score is a value between ZERO and ONE which indicates how likely the object represents one particular class. For example, in this case, the model is 0.76% sure that this object is a sandwich. If we take one step back, we can notice that thanks to our brain and our ability to recognize objects, we are able to add this metadata manually. This is a really difficult (and time consuming) task and you will need hundreds or thousands of lines of code to make a computer recognize it with the traditional approach. So what if there was a different way to teach a computer how to recognise what it sees. What if you could have a computer learn in the same way that a human does. This is the core of machine learning and this is where Tensorflowjs comes in. Thanks to a pre-trained model, coco-ssd, we will be able to detect all this metadata automatically. Once we have this info, it's easy to fetch all images that contain, for example, a sandwich.
- Searching is the most common activity that users do on the web. And these are the current hot trend keywords coming from Google Trends right now. As you can see, people are searching for football teams, celebrities and recent events. When it comes to searching for images, it is the content within the images that people are interested in. For example, which objects do you recognize in this image? We can see a chair, a dinner table, two bottles of sauce and a sandwich. Each of these items can be represented in JSON format. This is how we store the info of an object, the position, the name and a score. The score is a value between ZERO and ONE which indicates how likely the object represents one particular class. For example, in this case, the model is 0.76% sure that this object is a sandwich. If we take one step back, we can notice that thanks to our brain and our ability to recognize objects, we are able to add this metadata manually. This is a really difficult (and time consuming) task and you will need hundreds or thousands of lines of code to make a computer recognize it with the traditional approach. So what if there was a different way to teach a computer how to recognise what it sees. What if you could have a computer learn in the same way that a human does. This is the core of machine learning and this is where Tensorflowjs comes in. Thanks to a pre-trained model, coco-ssd, we will be able to detect all this metadata automatically. Once we have this info, it's easy to fetch all images that contain, for example, a sandwich.
- I have already mentioned one TensorFlowJs model but there are actually other models available out there too. Let me show you a couple of examples. We have coco-ssd which is able to detect and recognize 90 different classes of objects. Mobilenet, which is optimized for the mobile. Another one is body pix which is able to recognize the pixels of your body so that you can remove the background or do segmentation. The last one that I'd like to show is posenet which shows the joints of the body. As I said, these models are already trained. You don't need to know how ML works or what a loss function or the bias is. You just need to load the model and use it.
- Of course, with TensorflowJs you can create and train your own model. If you are interested in such a thing, I can recommend some videos by these guys (Lauren and Hussain).
- Now that you have an idea of what we can do with TensorflowJs, let's focus on the problem we are trying to solve in this talk. Imagine having a repository with thousands of images with useless names. As a user, you know the content you are looking for but you don't know the file name. So you write 'cat' in the search and you see that it doesn't give you any results. This is because there is no metadata that contains such info. To solve the problem what we can do is add this info in the description field and if we try again, we will see that the file is now there. Because we don't like to do manual stuff, we will build our own application, which uses ML to automate this process.
- These are the steps and libraries that our angular app will use. Just to give you a quick overview, we have: - Google Firestorage, which is used to store files in the cloud. - The Tensorflowjs model which detects the objects. - Firestore which is a Real time database where we store our data... ...and Angular, which is used to create our app. Let's see the flow in action: We have the images that we want to upload, so what do we do? We select one and with Firestorage, we upload the image to the cloud. This means that the next time the user visits the website, the image will still be there. Then we will pass the image to the model. We can imagine that a model is like a function. It needs an input and it will return an output. In this case the input is the image and the output the metadata. After that, we will save this metadata in Firestore and finally, we will be able to search our images by content. But let's see how it works in practice...
- This is what our final application looks like. On the left hand side, we have our application and on the right hand side, we have the firestore/storage console. Here, I am uploading a couple of images and Every time we upload an image, we store the image in the cloud and we create an instance on the firestore that contains the basic info such as name, location of the file. For now, it doesn't contain the detected objects. Then, when we click on the recognize button we pass the image to the tensorflow model that will detect the objects and will return the classes that can be displayed. With this new info, we can enrich and update the image metadata on firestore. Once we have all the metadata classes for each image, we can easily search the images based on the content. As a final step, we can delete the image from both storage and firestore Now let's go ahead and build this thing.
- We start with a brand new angular app. we use the angular cli new to generate a new project. You can select your preference and in a couple of seconds you have a basic angular app. Then we install angularfire2 dependency that allows us to use firestore and firestorage And finally we install TensorFlow model coco-ssd.
- The first step is to import all the modules in our recognition module and initialize our angular fire app. You can get the firebase settings from the google console. Inside the RecognitionModule we need to initialise our angular fire app with the project config that you can pick from the firebase console.
- Once we have imported all the modules, we can generate a component that will contain our upload widget. Starting from the HTML we have a simple input button that is listening to the changeEvent. When this event is fired, the method upload will be triggered. We are only allowing the png and jpg images files to be uploaded. To improve the UI a little bit, we are hiding the default input file element and we use angular material button which has an icon and then choose a file sentence. Here we are binding the two elements and the result is that every time we click on the material button, the upload method will be called. On the component side we have the implementation of the upload method which has as input the list of files. We can get the first file from the list by calling target dot files 0. To be able to upload a file in firestorage cloud bucket we need a unique id that represents the location of the file in the bucket. So we generate a random id with math random. At this point, we have every thing we need to save the file in the cloud. We inject the angular fire storage in the constructor and we use it to make a reference to the actual location in the bucket than calling the put will start the actual upload of the file. Let's see what the app looks like at this point.
- This is the same code that I was showing you before. The only difference is that we are now taking a reference of the AngularFireUploadTask which does most of the magic for the file upload itself. While the file is uploading, it provides a couple of observables that we can listen to in order to monitor the progress. Here we use the snapshot changes which is an observable that emits an object every few hundred milliseconds that contains the data about the file upload. Because our goal is to get the DownloadUrl we need to be sure that the upload is completed. We can use the finalize operator of rxJs that runs after the observable is completed. Because the getDonwnloadUrl returns a promise, we can create an async function and resolve the promise with the await. At this point we have all the info so it can be stored in firestore the real time database. We create an object that contains the info that we want to save on the database, specifically the name, class (that for now is empty but will contain the detected objects within the image) and the location of the file. We inject an angularfirestore and we make a reference to the image collection and then we add the document to the database.
- The next step is to render all the images saved on the database on the page. So we generate a new component still using the cli., I am skipping this step here Then we inject the AngularFirestore into the constructor and we get a reference of the same collection as before. Here we could subscribe to that collection to get all the items but the problem is that in this way we won't have the id of the single item. So what we do is we subscribe to the snapshotChanges observable and basically we merge the data of the document and the id of the document. On the HTML side, I am using a material card just to have a nice UI. And with the directive we are subscribing to the collection with async pipe and with NgFor we display the info like the name of the image and the content of the image. Then we use the mat chip to show the different classes detected and finally we have a recognize button. Which calls the recognize method using the item and the reference of the image as input.
- if you have a specific problem with TesnsorflowJs you to create your own neural network. There is no network latency because we are not calling any external API such as Amazon ML or Azure Because we are not calling any API your content is safe because it remains on the device This means that you can do recognition even without a connection