This document provides instructions for building and training an object detection model in PowerApps to identify three varieties of tea from images. It then describes how to create a canvas app that uses the trained model to detect tea items in an uploaded image and allow adjusting the on-hand inventory counts. Key steps include uploading and tagging training images, training the model, testing the model accuracy, publishing the model, and building an app with controls linked to the model and a Common Data Service entity to display and update inventory data.

1. Object detection
Objectdetectionletsyoucount,locate,andidentifyselectedobjectswithinanyimage. Youcan use this
model inPowerAppstoextractinformationfrompicturesyoutake withthe camera,orloadintoan app.
In thislab,we will buildandtraina detectionmodelandbuildanappthat usesthe detectionmodel to
identifyobjectsfromavailable images.
Note: If you are buildingthe firstmodel inanenvironment,clickonExplore Templatestogetstarted.
Exercise 1
In thisexercise we will buildandtrainthe ObjectDetectionmodel forthree varietiesof tea.
1. In PowerAppsmaker,expandAIBuilderandselectBuild. SelectObjectDetection.
2. Name yourmodel GreenTeaProduct Detection andbecause youare workingina shared
environmentalsomake sure toinclude yourname aspart of the model name. Thiswill make it
easiertofindlater. Clickcreate.

2. 3. Your screenshouldnowlooklike the image here.
4. Notice the progressindicatoronthe left. Those are the stepswe will follow now tobuildand
trainour model.

3. 5. We are now goingto define the objectswe are tracking. Clickonthe Selectobjectnames.
6. From the entitylist,selectObjectDetection Product.

4. 7. Selectthe Name field andclickSelectfield.
8. Selectthe teaitems andclickNext.
9. Notice the progressindicatorhasmovedforwardtothe Add imagesstep.

5. 10. Clickadd images.
11. Selectimagesfromthe setprovided. Youwill needenoughimagestoprovide 15samplesfor
each type of tea we are tracking.

6. 12. Approve the uploadof images.ClickUploadimages.Afterthe uploadcompletes,clickClose.
13. Clicknexttobegintaggingthe images.
14. Selectthe firstimage tobegintagging.

7. 15. Hoveroverthe image,nearanitemyouwishto tag. A dotted-linedbox shouldappeararound
the item. It has beendetectedasasingle itemthatcan be tagged.
16. Clickon the itemandselectthe matchingobjectname.

8. 17. If the pre-definedselectorisnotaccurate,as inthe below example,youcandrag the container
to draw itto accuratelytag the item.
18. Do thisfor eachiteminthe image andfor eachimage inyour set. Whenyouhave taggedall of
the imagesyouuploadedclickDone Tagginginthe topright of the screen.

9. 19. Once youhave completedtagging,youwillgetasummaryof the tags. If you haven’ttagged
enoughforanalysis,youwill needtoloadandtag more examples.
20. Once youhave definedenoughtagsfortrainingthe model,youwill be allowedtoinitiate the
training. ClickNext.

10. 21. ClickTrain.
22. The trainingtakesa fewmoments.

11. 23. Navigate tothe savedmodel view andconfirmyourmodel hascompletedtraining.
24. Selectthe model youjustmade.
25. SelectQuicktest.

12. 26. Upload or drag and dropone of your test imagestobe analyzed.
27. You will see the analysisandlevel of confidence forthe match.

13. 28. Upload an image youknowwill notmatch. You will see the analysisandlevel of confidence for
the match.
29. Clickclose.

14. 30. Publishyourmodel.
Exercise 2
We will nowcreate acanvas app youcan use fordetectingthe itemsthathave beentrainedinour
model. The productwill be detectedfromthe image andyouwill be able toadjuston-handinventory
for the item.
1. Navigate toApps,andselectCreate anapp, thenselectCanvas. If asked,grantpermissionforthe
app to use youractive CDS credentials.
2. SelectBlankappwithPhone layout.

15. 3. On the makercanvas,selectthe Inserttab inthe ribbonand expandAIBuilder. SelectObject
detectortoplace thiscontrol onyour app.
4. Selectthe AImodel youbuilt.
5. Resize the control tobetteruse the space.

16. 6. Make sure to leave roomformore itemswe will be placingsoon.
7. Playyourapp.

17. 8. Clickon Detect.
9. Choose one of yourtest imagesandclickOpen.

18. 10. The image will nowbe analyzed.
11. Our model hasdetected eachteainthe image.

19. 12. Exitthe app player.
Bonus exercise- build out the data in your canvas app
1. We will nowselectourdatasource. SelectView fromthe ribbonand selectDataSources.
2. Click+ Add Data Source.

20. 3. Addthe CommonData Service datasource. Do not use CommonData Service (current
environment).
4. Selectthe ObjectDetection Products entityandclickConnect.
5. Close the Data pane.

21. 6. WithScreen1selectedinthe Tree view,navigatetothe Insertribbontab,expandGalleryandselect
Blankvertical gallery.
7. Rename the GalleryproductGallery. Youare re-namingthe gallerysoyoucanreference itfromyour
formulas.
8. Resize andmove the gallerycontrol tofitthe available space onthe screen,leavingsome space at
the bottomfor usinglater.

22. 9. Selectthe editiconfromthe gallery.
10. Adda label tothe gallery.

23. 11. Clickeditagainand add a Textinputbox tothe gallery. Resize andplace ittoline upwiththe label
we’ve alreadyplaced. We will be updatinginventorycountsinthistextbox.
12. Rename the TextInputinventoryInput.Youare renamingthiscontrol soyoucan reference itfrom
your formulas.

24. 13. Withfocus onthe Screen1inthe Tree view,clickinthe ribbononInsertandselectButton.
14. Drag and move the buttonto the bottomof the screen,double clickonitto editthe text.Rename it
to Update.

25. 15. We will nowaddthe usermessage togive the userconfirmationtheirsubmissionwasaccepted;we
will define thislogiclater. WithfocusonScreen1,inserta label,dragitto the bottomof the screen.

26. 16. We will nowaddlogicto the controlswe’ve placedonthe screen. Selectthe Galleryandreplace
the Itemsformulawiththe following.
'Object Detection Products'
17. Selectthe label inyourgallery. Replace the Textformulawiththe following:
ThisItem.Name
18. SelectinventoryInputandreplace the formulaforDefaultwiththe following:
LookUp('Object Detection Products',Name = ThisItem.Name).'Inventory Total'
19. Selectthe otherlabel (the one thatshowsatthe bottom of the screen) andreplace itstextwiththe
following:
usermessage

27. 20. You’ll notice thatarea nowlooksblank. We will configure thatmessage inournextstep.
21. Selectthe buttoncontrol andreplace the OnSelectwiththe following:
ForAll(productGallery.AllItems,Patch('Object Detection Products',LookUp('Object
Detection Products',Name=DisplayName),{'Inventory

28. Total':Value(inventoryInput.Text)}));Set(usermessage,"Updated " &
CountRows(productGallery.AllItems) & " items")
22. Playthe app again.
23. ClickDetect.
24. Selectanimage to evaluate.
25. Update the quantityforthe correct productand clickUpdate.

29. 26. The bottom shouldshowamessage now.