En esta charla veremos como utilizar servicios cognitivos en la nube desde python de dos de los grandes actores: IBM Watson y Microsoft Azure Cognitive Services. Presentaremos una introducción a dichos servicios y finalmente veremos una aplicación sobre la construcción de un robot asistente para personas mayores. El ponente será Javier Palanca, investigador en IA en la UPV.

1. Servicios
Cognitivos
para construir un
robot en Python
Javi Palanca

2. LA INTELIGENCIA
ARTIFICIAL
LO ESTÁ
PETANDO

3. Desarrollo
ad-hoc

4. •Multitud de algoritmos
•Bajo nivel
•Personalización
•Software Libre
•Aprovechan la GPU
•Conocimiento avanzado
•Sin modelos pre-entrenados
•From scratch
•Necesitas una GPU

5. Servicios
Cognitivos

6. •Puesta en marcha muy
rápida
•Acceso a grandes
recursos hardware
•APIs de acceso variadas
(http, python, …)
•Interfaces gráficas
•Permiten bajar de nivel
•Servicios de pago
•Dependencia de una empresa
•Pocos algoritmos pre-
determinados
•Velocidad de acceso
(Auth tokens, …)
•No es Software Libre

7. Servicios Ofrecidos
Computer
Vision
Natural
Language
Understanding
Face
Recognition
Speech
Recognition
Chatbot
Conversation
Neural
Networks
Language
Translator
Sentiment
Analysis
Text to
Speech
Speech to
Text
Machine
Learning
Regression
Virtual Servers
Cloud Infrastructure
Storage DevOps
Image
Recognition
Text
Recognition
Networks
Clustering

9. Step #1
portal.azure.com

10. Step #2
Create Azure
Machine Learning
Workspace
Step #1
portal.azure.com

11. Step #2
Create Azure
Machine Learning
Workspace
Step #1
portal.azure.com
Step #3
Launch Machine
Learning Studio

12. Step #2
Create Azure
Machine Learning
Workspace
Step #1
portal.azure.com
Step #3
Launch Machine
Learning Studio
Step #4
Profit!

14. from azureml import Workspace
ws = Workspace(workspace_id='4c29e1adeba2e5a7cbeb0e4f4adfb4df',
authorization_token='f4f3ade2c6aefdb1afb043cd8bcf3daf',
endpoint='https://europewest.studio.azureml.net/')
$ pip install azureml
name = 'my existing dataset'
dataset = ws.datasets[name]
dataset.update_from_dataframe(
dataframe=frame,
data_type_id=DataTypeIds.GenericCSV,
name='my new name',
description='my new description'
)
Get workspace
Get and update dataset with pandas DataFrame

15. from azureml import services
@services.publish(workspace, workspace_token)
@services.types(a = float, b = float)
@services.returns(float)
def func(a, b):
return a / b
Publish service

16. import cognitive_face as CF
KEY = '<Subscription Key>'
CF.Key.set(KEY)
BASE_URL = 'https://westus.api.cognitive.microsoft.com/face/v1.0/'
CF.BaseUrl.set(BASE_URL)
# You can use this example JPG or replace the URL below with your own URL to a JPEG image.
img_url = 'https://raw.githubusercontent.com/Microsoft/Cognitive-Face-Windows/master/Data/
detection1.jpg'
faces = CF.face.detect(img_url)
print(faces)
$ pip install cognitive_face
Face Detection
[ {
'faceId': '68a0f8cf-9dba-4a25-afb3-f9cdf57cca51',
'faceRectangle': {
'width': 89, 'top': 66,
'height': 89, 'left': 446
}
} ]

17. Face Detection
[{
"faceId": "e387e58f-eb27-4294-b8ac-7f3fd504de89",
"faceRectangle": {"top": 128, "left": 459, "width": 224, "height": 224},
"faceAttributes": {
"hair": {
"bald": 0.0,
"hairColor": [
{"color": “brown", "confidence": 1.0},
{"color": “blond", "confidence": 0.69},
...
]
},
"smile": 0.939,
"gender": "female",
"age": 23.4,
"facialHair": {"moustache": 0.0, "beard": 0.0,"sideburns": 0.0},
"glasses": "ReadingGlasses",
"makeup": { "eyeMakeup": true,"lipMakeup": true},
"emotion": {
"anger": 0.037, "contempt": 0.001,
"disgust": 0.015, "fear": 0.001,
"happiness": 0.939, "neutral": 0.001,
"sadness": 0.0, "surprise": 0.007
},
"accessories": [{ "type": “glasses", "confidence": 0.99}],
},
"faceLandmarks": {
"pupilLeft": {"x": 504.8,"y": 206.8},
"pupilRight": {"x": 602.5, "y": 178.4},
"noseTip": {"x": 593.5,"y": 247.3},
...
"underLipBottom": {"x": 600.3,"y": 324.8}
}
}]
Similar face search
Face grouping

18. from azure.cognitiveservices.language.textanalytics import TextAnalyticsAPI
textanalytics = TextAnalyticsAPI(azure_region, credentials)
documents = {'documents' : [
{'id': '1', 'language': 'en',
'text': 'I had a wonderful experience! The rooms were wonderful and the staff was helpful.'},
{'id': '2', 'language': 'en',
'text': 'I had a terrible time at the hotel. The staff was rude and the food was awful.'},
{'id': '3', 'language': 'es',
‘text': 'Los caminos que llevan hasta Monte Rainier son espectaculares y hermosos.'}
]}
response = textanalytics.sentiment(documents)
$ pip install azure-cognitiveservices-language-textanalytics
Sentiment Analysis
{
"documents": [
{ “id": “1", "score": 0.7673527002334595 },
{ "id": “2", "score": 0.18574094772338867 },
{ "id": "3","score": 0.5 }
],
"errors": []
}

20. console.bluemix.net

21. Visual Recognition

22. Visual Recognition
with open('./beagle.zip', 'rb') as beagle,
open('./golden-retriever.zip', 'rb') as goldenretriever,
open('./husky.zip', 'rb') as husky,
open('./cats.zip', 'rb') as cats:
model = visual_recognition.create_classifier(
'dogs',
beagle_positive_examples=beagle,
goldenretriever_positive_examples=goldenretriever,
husky_positive_examples=husky,
negative_examples=cats)
Create Image Classifier
with open(‘./dog.1234.jpg’, 'rb') as images_file:
classes = visual_recognition.classify(
images_file,
threshold='0.6',
classifier_ids='dogsx2018x03x17_1725181949,Connectors_424118776')
Classify Image
import json
from watson_developer_cloud import VisualRecognitionV3
visual_recognition = VisualRecognitionV3(
'2016-05-20',
api_key='{api_key}')

23. Intents - Entities - Dialogs
Watson Assistant - Conversations

24. Watson Assistant - Conversations - Intents

25. Watson Assistant - Conversations - Entities

26. Watson Assistant - Conversations - Dialogs

27. import json
import watson_developer_cloud
assistant = watson_developer_cloud.AssistantV1(
username='{username}',
password='{password}',
version='2018-02-16'
)
response = assistant.message(
workspace_id='9978a49e-ea89-4493-b33d-82298d3db20d',
input={
'text': ‘Play some jazz music'
}
)
Get response to user input
Watson Assistant - Conversations
response = assistant.update_intent(
workspace_id = '9978a49e-ea89-4493-b33d-82298d3db20d',
intent = 'hello',
new_examples = [
{'text': 'Good afternoon'}
],
new_description = 'Updated intent'
)
Update intent

28. import json
import watson_developer_cloud
text_to_speech = TextToSpeechV1(
username='{username}',
password='{password}',
url=‘https://stream.watsonplatform.net/text-to-speech/api'
)
response = text_to_speech.synthesize(transcription,
accept='audio/wav',
voice="es-ES_EnriqueVoice")
play(response.content)
Text to Speech
Text to Speech - Speech toText
import wave
import pyaudio
import simpleaudio as sa
def play(input_array):
p = pyaudio.PyAudio()
wf = wave.open(io.BytesIO(input_array))
wave_obj = sa.WaveObject.from_wave_read(wf)
play_obj = wave_obj.play()
play_obj.wait_done()
p.terminate()
Play audio

29. Watson Studio - Machine Learning

30. Watson Studio - Machine Learning

31. Rasselrobot assistant for the elderly
Maite Giménez · Jaume Jordán · Javier Palanca · Jaime Rincón
IBM Challenge Cogs For Good

32. Detección del nicho
DEFINICIÓN DE LA NECESIDAD:
Incremento de población de tercera edad y de-
pendientes.
Necesidad de acompañamiento y asistencia.
Aumentar la autosuficiencia de la persona
asistida.
Garantizar la seguridad de la persona con de-
pendencia en el hogar.

33. Propuesta de valor
Asistente conversacional.
Recomendaciones para una vida activa y satisfactoria.
Monitorización de constantes vitales.
Detección de caidas y de situaciones de riesgo.
Sistema de alertas a familiares.
RASSEL TE CUIDA COMO ELLA TE CUIDÓ A TI.

34. esquema general
del prototipo
esquema general
del prototipo

35. esquema general
pulsera
ROBOT
dependiente
FAMILIA
robot

36. características de la pulsera
pulsera
SENSORES:
Acelerómetro y giroscopio IMU.
Sensor de fotopletismografía.
Electrocardiografía: requiere interacción
de la persona.
GCR: resistencia de piel

37. características del robot
robot
SENSORES:
Cámara.
Micrófono.
Altavoz.
Expresividad facial con motores.
Pantalla táctil.

39. funcionalidad de rassel
CONVERSACIÓN: Watson assistant, text2speech, speech2text.
RECOMENDADOR ACTIVIDADES: Collaborative filtering y Reinforce-
ment Learning.
RECORDATORIOS MEDICACIÓN.
MONITORIZACIÓN DE CONSTANTES VITALES: IBM IoT.
DETECCIÓN DE CAÍDAS: Acelerómetro y Visual Recognition IBM.
ALERTAS Y AVISOS.

40. Rasselrobot assistant for the elderly
Te cuida como ella cuidó de ti
esquema general
del prototipo