Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Unleash some AI into the Wild - IT.A.K.E

45 views

Published on

We’ll dive into a real-world case on how A.I. can assist medical workers in the field. From eliminating time-consuming documentation to gaining valuable insights into their work. By letting AI take care of common tasks, medical workers can focus on delivering essential medical care.

In this session, I’ll take you along on a AI-First technical journey, from patient identification handled by Cognitive Services, efficiency improvement with natural language processing and handling data in the cloud. Besides that we take a look at the collected data and see how we can make a predictive model.

We took the app to some remote places in Uganda and beta tested it. We will zoom into some challenges we faced running it in the field, discuss the tools and components used and give you a peek into future steps.

Published in: Technology
  • Be the first to comment

  • Be the first to like this

Unleash some AI into the Wild - IT.A.K.E

  1. 1. @hboelman | www.henkboelman.com
  2. 2. How�can�AI�improve�the�efficiency�of� medical�field�workers�in�remote� Ugandan�villages?
  3. 3. I'mI'm Henk BoelmanHenk Boelman
  4. 4. I'mI'm Henk BoelmanHenk Boelman
  5. 5. I'mI'm Henk BoelmanHenk Boelman
  6. 6. @hboelman https://nl.linkedin.com/in/henkboelman https://github.com/hnky https://www.henkboelman.com
  7. 7. Like to make softwareLike to make softwareLike to make softwareLike to make softwareLike to make softwareLike to make software
  8. 8. VolunteerVolunteer for Suubi Reachout
  9. 9. TendoCare.AITendoCare.AI TendoCare.ai is a platform that enables medical workers to be more effective in the eld. TendoCare.ai is a platform that enables medical workers to be more effective in the eld. TendoCare.AITendoCare.AI
  10. 10. Make work easierMake work easier
  11. 11. Make work easierMake work easier Better insightsBetter insights
  12. 12. Make work easierMake work easier Better insightsBetter insights And fun to make..And fun to make..
  13. 13. AI rstAI rst How can AI make this app easier to use
  14. 14. Develop on the MacDevelop on the Mac Just something to try out...
  15. 15. Xamarin Visual�Studio� App�Center .NET�core� Backend Visual�Studio� Team�Services GitHub
  16. 16. Put intelligence APIs to workPut intelligence APIs to work Cognitive ServicesCognitive Services Microsoft Azure
  17. 17. VisionVision Image-processing algorithms to smartly identify, caption and moderate your pictures. � SpeechSpeech Convert spoken audio into text, use voice for veri cation, or add speaker recognition to your app. � LanguageLanguage KnowledgeKnowledge Map complex information and data in order to solve tasks such as intelligent recommendations and semantic search.� Allow your apps to process natural language with pre-built scripts, evaluate sentiment and learn how to recognize what users want.
  18. 18. Patient identi cationPatient identi cation
  19. 19. Patient identi cationPatient identi cation Search by typing a nameSearch by typing a name
  20. 20. Patient identi cationPatient identi cation Search by typing a nameSearch by typing a name Kiyemba�Buyondo
  21. 21. Patient identi cationPatient identi cation Search by typing a nameSearch by typing a name Kiyemba�Buyondo Ngenyi�Nyanjura
  22. 22. Patient identi cationPatient identi cation Search by typing a nameSearch by typing a name Kiyemba�Buyondo Ngenyi�Nyanjura
  23. 23. Take a picture Tendocare API Patient File
  24. 24. Take a picture Tendocare API Patient File
  25. 25. Face ApiFace Api faceAttributes":�{� ���"smile":�0.003,� ���"gender":�"male",� ���"age":�56.3,� ���"glasses":�"NoGlasses",� } Face detection & description Person identi cation �
  26. 26. Emotions ApiEmotions Api "anger":�����0.6506504,� "contempt":��0.0159038156,� "disgust":���0.2675257,� "fear":������0.000070601156,� "happiness":�0.0190101564,� "neutral":���0.0373256877,� "sadness":���0.009163826,� "surprise":��0.0003498507
  27. 27. Register a personRegister a person
  28. 28. Register a personRegister a person Create�a�PersonGroup� PersongroupName
  29. 29. Register a personRegister a person Create�a�PersonGroup� Create�a�Person PersongroupID PersonName PersongroupName
  30. 30. Register a personRegister a person Create�a�PersonGroup� Create�a�Person Add�a�Face�to�the� PersongroupID PersonName PersongroupName PersonID Image (stream / url) PersongroupName
  31. 31. Register a personRegister a person Create�a�PersonGroup� Create�a�Person Add�a�Face�to�the� Train�the�persongroup PersongroupID PersonName PersongroupName PersonID Image (stream / url) PersongroupName PersongroupName
  32. 32. Identify a personIdentify a person
  33. 33. Identify a personIdentify a person Detect Image (stream / url)
  34. 34. Identify a personIdentify a person Detect Identify PersongroupID FaceID Image (stream / url)
  35. 35. Identify a personIdentify a person Detect Identify PersongroupID FaceID Image (stream / url) PersonId
  36. 36. Take a picture api.tendocare.ai Patient File
  37. 37. Detect�Face/LoadPatientFileByFace Identify�the�face
  38. 38. Detect�Face/LoadPatientFileByFace Identify�the�face 1. Register with FaceAPI 2. Create a new PatientFile
  39. 39. Detect�Face/LoadPatientFileByFace 1. Load PatientFile by personId Identify�the�face 1. Register with FaceAPI 2. Create a new PatientFile
  40. 40. Detect�Face/LoadPatientFileByFace 1. Load PatientFile by personId Identify�the�face Return� PatientFile 1. Register with FaceAPI 2. Create a new PatientFile
  41. 41. DemoDemo Tendocare�Face�demo
  42. 42. DemoDemo Tendocare�Face�demo
  43. 43. PatientFile by objectPatientFile by object
  44. 44. PatientFile by objectPatientFile by object
  45. 45. PatientFile by objectPatientFile by object
  46. 46. An easy-to-use tool for creating your own custom image classi er � No background in computer vision or deep learning required. Custom Vision ServiceCustom Vision Service
  47. 47. Model Custom Vision Service Classi cation
  48. 48. Model Custom Vision Service Classi cation Object detection
  49. 49. Model "Sticker B" Custom Vision Classi cationClassi cation
  50. 50. Custom Vision Object detectionObject detection
  51. 51. APIExport to: Custom Vision
  52. 52. DemoDemo Creating a custom image classi er
  53. 53. PatientFile by objectPatientFile by object Object detection
  54. 54. Computer Vision ApiComputer Vision Api "a�man�flying�through�the�air� while�riding�a�skateboard"
  55. 55. { "name": "water", "con dence": 0.997288346 }, { "name": "outdoor", "con dence": 0.9940638 }, { "name": "sky", "con dence": 0.9912407 }, { "name": " shing", "con dence": 0.970712841 }, "a�man�water�skiing�behind�a�boat"
  56. 56. {� ��"name":�"JOHN�TRAVOLTA",� ��"faceRectangle":�{� �������"left":�516,�"top":�152,� ����������"width":�206,�"height":�206� ��},� ��"confidence":�0.9996151� } {� ��"name":�"SAMUEL�L.�JACKSON",����� ��"faceRectangle":�{� �����"left":�1107,�"top":�172,�� �������"width":�203,�"height":�203�� ���},� ���"confidence":�0.988874555� }
  57. 57. Computer Vision ApiComputer Vision Api Read text in imagesRead text in images
  58. 58. Computer Vision ApiComputer Vision Api Read handwritten textRead handwritten text in imagesin images
  59. 59. DemoDemo Detect handwriting in an image
  60. 60. PatientFile by objectPatientFile by object Object detection Read handwriting
  61. 61. Custom Vision + Computer VisionCustom Vision + Computer Vision 1. Train a model with Custom Vision (CoreML) 2. Send the detected image to the Computer Vision API
  62. 62. Custom Vision + Computer VisionCustom Vision + Computer Vision 1. Train a model with Custom Vision (CoreML) 2. Send the detected image to the Computer Vision API Scan�� object
  63. 63. Custom Vision + Computer VisionCustom Vision + Computer Vision 1. Train a model with Custom Vision (CoreML) 2. Send the detected image to the Computer Vision API Scan�� object frame Classify
  64. 64. Custom Vision + Computer VisionCustom Vision + Computer Vision 1. Train a model with Custom Vision (CoreML) 2. Send the detected image to the Computer Vision API Scan�� object 90%�� confidence�frame Classify
  65. 65. Custom Vision + Computer VisionCustom Vision + Computer Vision 1. Train a model with Custom Vision (CoreML) 2. Send the detected image to the Computer Vision API Scan�� object 90%�� confidence�frame Classify Send�to� Vision�API
  66. 66. Custom Vision + Computer VisionCustom Vision + Computer Vision 1. Train a model with Custom Vision (CoreML) 2. Send the detected image to the Computer Vision API Scan�� object 90%�� confidence� Read�� handwritingframe Classify Send�to� Vision�API
  67. 67. DemoDemo Xamarin�IOS�vision�prototype
  68. 68. DemoDemo Xamarin�IOS�vision�prototype
  69. 69. Making notesMaking notes
  70. 70. Bing Speech APIBing Speech API Convert speech to text and back again to understand user intent
  71. 71. Speech to textSpeech to text Convert speech to text. REST API or WebSocket 3 Modes (Interactive, Conversation, Dictation) 10 Languages
  72. 72. Text to speechText to speech Convert text to speech. REST API call OutputFormat Gender Name Language
  73. 73. Record�the� audio�file.�
  74. 74. Record�the� audio�file.� Post�to�api
  75. 75. Record�the� audio�file.� Post�to�api Put�it�on� blob�storage
  76. 76. Record�the� audio�file.� Post�to�api Put�it�on� blob�storage Azure�function� (blob�trigger)
  77. 77. Record�the� audio�file.� Post�to�api Put�it�on� blob�storage Azure�function� (blob�trigger) Call�Bing� Speech�to�text� API
  78. 78. Record�the� audio�file.� Post�to�api Put�it�on� blob�storage Azure�function� (blob�trigger) Call�Bing� Speech�to�text� API Update�the� patientfile
  79. 79. DemoDemo Using an Azure Function for Speech to Text
  80. 80. SecuritySecurity
  81. 81. Speaker Recognition APISpeaker Recognition API�� Use speech to identify and authenticate individual speakers
  82. 82. Create a speaker pro leCreate a speaker pro le "my voice is my passport verify me"
  83. 83. Create a speaker pro leCreate a speaker pro le Load� phrases "my voice is my passport verify me"
  84. 84. Create a speaker pro leCreate a speaker pro le Load� phrases Create� verification� profile "my voice is my passport verify me"
  85. 85. Create a speaker pro leCreate a speaker pro le Load� phrases Create� verification� profile Enrole� phrase "my voice is my passport verify me"
  86. 86. Create a speaker pro leCreate a speaker pro le Load� phrases Create� verification� profile Enrole� phrase 3x "my voice is my passport verify me"
  87. 87. Create a speaker pro leCreate a speaker pro le Load� phrases Create� verification� profile Enrole� phrase 3x created � Profile "my voice is my passport verify me"
  88. 88. Verify speakerVerify speaker Container WAV Encoding PCM Rate 16K Sample Format 16 bit Channels Mono Record the phrase Call the API (Veri cationPro leId & Enrollment sentence as bytes) Handle response (result: Accept | con dence: High)
  89. 89. DemoDemo Xamarin�voice�prototype
  90. 90. DemoDemo Xamarin�voice�prototype
  91. 91. Cognitive Services are a great way to kickstart your intelligent project.
  92. 92. Future StepsFuture Steps
  93. 93. Build our own wound classi erBuild our own wound classi er
  94. 94. Project PragueProject Prague Incorporate gesture-based controls into your apps.
  95. 95. https://github.com/hnky/Project-Prague-Demo/https://github.com/hnky/Project-Prague-Demo/
  96. 96. Face Identi cation Patient identi cation Custom + Computer Vision Alternative for identi cation Speech Recognition Making notes using voice Speaker Recognition For authentication RecapRecap
  97. 97. RememberRemember
  98. 98. It is just an API requestIt is just an API request RememberRemember
  99. 99. https://twitter.com/hboelman https://nl.linkedin.com/in/henkboelman https://github.com/hnky https://www.henkboelman.com Thank youThank you tendocare.aitendocare.ai more information on:

×