Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Azure Day 2019: Bot meets AI (Martin Schreiber & Mattia Baldinger)

32 views

Published on

In unserer digitalen Welt kommt der persönliche Austausch oft zu kurz. Dieses kostbare Gut zu schützen, nehmen ein Bot und und eine intelligente Kaffemaschine nun selber in die Hand: Anstatt einfach in die Kaffepause zu gehen, lasse ich dies den Bot wissen. Der Bot meldet sich bei mir, wenn er per Zufallsprinzip eine geeignete Person für eine gemeinsame Kaffeepause gefunden hat und schickt uns beide zum Kaffeeaustausch. Ähnlich macht es die intelligente Kaffeemaschine: sie kann mich identifizieren und ebenfalls per Bot eine Person nach Zufall suchen lassen, die zu mir zum Kaffee kommt. Das ganze spielt im Microsoft Teams und Microsoft Azure Cognitive Services Ecosystem. Wir zeigen Euch, wie es dazu kam und wie das System aufgebaut ist.

Published in: Technology
  • Be the first to comment

  • Be the first to like this

Azure Day 2019: Bot meets AI (Martin Schreiber & Mattia Baldinger)

  1. 1. news.trivadis.com/blog@trivadis Together for a coffee - Bot meets AI mattia.baldinger@trivadis.com martin.schreiber@trivadis.com
  2. 2. Mattia Martin Prototyper and lateral thinker for the digital world of today and tomorrow. @gruemscheli schreibermartin.wordpress.com If there is no solution to your problem, ask Mattia ☺
  3. 3. The Story
  4. 4. Trivadis was asked by their partner to find a solution to the problem of the lack of interaction between employees. The partner had the idea to integrate the solution into their collaboration platform Microsoft Teams. During the discussion, it quickly became clear that a chat bot was needed to get employees to drink coffee together and create/sharing ideas. At the same time, the dream of an "intelligent" coffee machine was born... To start with, we had to talk about and understand “conversations”...
  5. 5. Conversation Flow
  6. 6. How does the conversation flow looks like? Do you want to drink drink a coffee with Sven? I want to drink a coffee with someone... Yes, why not... Ok. Sven is on the way to the coffee maschine. Have a nice time. Ok thanks By the way, if you have a good idea in your coffee break, let the innovation team know about it. SvenjaChatbot
  7. 7. How does the conversation flow looks like? I want to drink a coffee with someone... Yes, why not... Ok. Sven is on the way to the coffee maschine. Have a nice time. Ok thanks By the way, if you have a good idea in your coffee break, let the innvoation team know about it. Do you want to drink drink a coffee with Svenja? I want to drink a coffee with someone... Yes, why not... Ok. Svena is on the way to the coffee maschine. Have a nice time. Ok thanks By the way, if you have a good idea in your coffee break, let the innvoation team know about it. 12 Do you want to drink drink a coffee Sven? 3 4 5 6 7 8 9 Sven Svenja Chatbot
  8. 8. What are we missing? I didn't find somebody yet, but I put you in the queue for the next 15 minutes. I will get back to you if somebody joins the queue. I want to drink a coffee with someone... Sorry I didn’t found somebody for you. Let me know when you want to drink a coffee again.. Ok thanks 1 2 3 Sven Chatbot 4
  9. 9. What are we missing? I want to drink a coffee with someone... Yes, why not... Sven isn’t available anymore, I will look for somebody else.. Do you want to drink drink a coffee with Svenja? I want to drink a coffee with someone... You have been removed from the queue due inactivity, let me know if you want to drink a coffee again. 12 Do you want to drink drink a coffee Sven? 3 4 5 Sven Svenja Chatbot 6
  10. 10. Azure & Bot Framework
  11. 11. What is Microsoft Bot Framework? Bot Framework SDK • Connectors • Dialogs • State Management Supported Languages • Typescript • C# • Java • Python Development Tools • Emulator • Templates • Connectors • Solutions Services • QNaMaker • Speech Services • Luis
  12. 12. Whats does Azure provide for our solution? Azure Bot Service Provides Channels Sven Svenja {"id":"Sven", "message":" I want to drink a coffee"} <client>Sven</client> <content> I want to drink a coffee<content>
  13. 13. Whats does Azure provide for our solution? Azure App Service Chatbot WebserviceAzure Bot Service Provides Channels{"user":"Sven", "message":" I want to drink a coffee"} Sven Svenja {"user":"Svenja", "message":" I want to drink a coffee"}
  14. 14. Whats does Azure provide for our solution? Azure App Service Chatbot Webservice LUIS Language Understanding Azure Bot Service Provides Channels Azure Blob Storage Stores Chatbot State "I want to drink a coffee" Intent: coffee Probabiliy: 1 State: Bot State: User State: Conversation Sven Svenja
  15. 15. Whats does Azure provide for our solution? Azure SQL Database Stores app data Azure App Service Chatbot Webservice Azure App Service Backend REST Service LUIS Language Understanding Azure Bot Service Provides Channels Azure Blob Storage Stores Chatbot State Azure Application Insights Monitors Web App Web App Mobile App Insert into queue (...) Insert into match (...) Post: queue/sven Post: match/sven Sven Svenja
  16. 16. Wish: While waiting for coffee, it should be possible to invite others to coffee without using your personal telephone or computer (which you might not have with you). One Solution: A shared device (coffee machine or computer) interacts with the bot.
  17. 17. Bot Conversation on a Shared Device Sven (with beard)
  18. 18. Bot Conversation on a Shared Device Sven (with beard) Alex (with beard)
  19. 19. Bot Conversation on a Shared Device Sven (with beard) Alex (with beard) Mattia
  20. 20. Quick Authentication on a Shared Device Me (relaxed) Alex (in a hurry)
  21. 21. Alternatives?
  22. 22. Face Identification for Authentication Facial Landmarks Accuracy 1:105 (MSFT) dataturks.com/blog/face-verification-api-comparison.php Accuracy 1:106 (AAPL) TrueDept Image •≥ 99% Confidence •≥ 99.3 True Positive •0…0.13% False Positive •≥ 92.6% Confidence •≥ 87.3…97.8% True Positive •0…7.6% False Positive •≥ 93.4% Confidence •≥72…88.3% True Positive •0…12.6% False Positive
  23. 23. Let’s use Azure Cognitive Services ❷ Image 0+ Face IDs (Landmarks) ❸ recognise face(s) ❹ 1 Face (Landm.) 0 | 1 Person ID ❺ identify person ❻ Application Image [ add ] + train *-User ID = Person ID
  24. 24. What makes a Person Identifiable? <image> -> candidates: [ { faceId: <guid>, faceRectangle: { width, height, left, top }, faceLandmarks: { pupilLeft: {x,y}, pupilRight: { … }, … }, faceAttributes: { age:45, gender:male, smile:0.88, facialHair: { moustache, … }, glasses: "sunglasses", … } } ] faceId -> faces: [ { personId: <guid>, confidence: 0.92 }, { personId: <guid>, confidence: 0.89 } ] personId -> person: { persistedFaceIds: [ <guid>, <guid>, <guid> ], name: <user-*>, userData: <*-user id> } ❸ recognise face ❺ identify face ❻ get user data for application
  25. 25. Partial face ❶ Clear face Low res. Low quality 2+ Faces Image 0+ Faces 1 Face 0 | 1 Person ID Local Recognition reduces time and cost
  26. 26. - • tracking.js rich set for object detection/tracking -• face-api.js compter vision tasks; uses TensorFlow (slow loading) -• clmtrackr face landmark tracking (lag) -• pico.js face tracking (fast, smoth)- -• jeelizWeboji for mimics detection -• headtrackr tracking (fast, unsteady rect) -• facedetector simple to use (no tracking) ( tinyurl.com/y3y8yba3 ) -• OpenCV.js the classic - compter vision tasks; subset of OpenCV (by Intel Russia in 1999!)- Partial face ❶ Clear face Low res. Low quality 2+ Faces Local Recognition reduces time and cost
  27. 27. Discrimination by skin colour and gender? (uhm...) theverge.com <1% on lighter skinned males 7% on lighter skinned females 12% on darker skinned males 35% on darker skinned females
  28. 28. face identification = applied AI, piece of cake! biometric data: device or cloud? gender and ethnic equality? user acceptance - prototype 1st! have camera + optical factors under control application integration piece of cake!

×