Using SharePoint with cognitive services for automatic image tagging - Presentation from SPSVie 2017 (SharePoint Saturday Vienna, 11.11.2017)

1. Using SharePoint with
Cognitive Services for
automatic image tagging
Markus Hintner

5. Microsoft
Cognitive
Services
Give your apps
a human side

6. Computer Vision API
Distill actionable
information from images
Video API
Analyze, edit, and
process videos within
your app
Face API
Detect, identify, analyze,
organize, and tag faces in
photos
Content Moderator
API
Machine-assisted
moderation of text,
images, and videos,
augmented with human
review tools
Vision
Emotion API
Personalize experiences
with emotion recognition

10. Visual Feature Description
Categories categorizes image content according to a taxonomy defined in
documentation
Tags tags the image with a detailed list of words related to the image
content
Description describes the image content with a complete English sentence
Faces detects if faces are present. If present, generate coordinates, gender
and age
ImageType detects if image is clipart or a line drawing
Color determines the accent color, dominant color, and whether an image is
black&white
Adult detects if the image is pornographic in nature (depicts nudity or a sex
act). Sexually suggestive content is also detected

11. Celebrities - identifies celebrities if detected in the image
Landmarks - identifies landmarks if detected in the image
Supported Languages (as of today): EN and ZH (Simplified Chinese)

12. Subscription key which provides access to this API.
Found in your Cognitive Services accounts

14. raw image binary or
image URL
Supported image formats: JPEG, PNG, GIF, BMP.
Image file size must be less than 4MB.
Image dimensions must be at least 50 x 50

18. WebHooks

20. HTTP/1.1 200 OK
Content-Type: text/plain
{randomString}
SharePoint
Service
POST /_api/web/lists('list-id')/subscriptions
Yourapplication
POST https://{your host}/your/webhook/service
?validationToken={randomString}
YourWebHook
notificationservice
endpoint
HTTP/1.1 201 Created
Yourapplication
Content-Type: application/json
{
"resource": "https://contoso.sharepoint.com/_api/web/lists({id})",
"notificationUrl": "https://{your host}/your/webhook/service ",
"expirationDateTime": "2018-04-11T10:45:57+00:00"
}

21. HTTP/1.1 200 OK
SharePoint
Service
YourWebHook
notificationservice
endpoint
POST https://{your host}/your/webhook/service
{
"value":[
{
"subscriptionId":"91779246-afe9-4525-b122-6c199ae89211",
"clientState":"00000000-0000-0000-0000-000000000000",
"expirationDateTime":"2016-04-30T17:27:00.0000000Z",
"resource":"b9f6f714-9df8-470b-b22e-653855e1c181",
"tenantId":"00000000-0000-0000-0000-000000000000",
"siteUrl":"/",
"webId":"dbc5a806-e4d4-46e5-951c-6344d70b62fa"
}
]
}

22. YourWebHook
notificationservice
endpoint
SharePoint
Service
POST https://{your host}/your/webhook/service
Storage
Queue
WebJob
HTTP/1.1 200 OK

23. SharePoint
Service
WebHook creation: POST /_api/web/lists('list-id')/subscriptions
Yourapplication
Grab “CurrentChangeToken” from list
YourWebHook
notificationservice
endpoint POST https://{your host}/your/webhook/service
Storage
Queue
WebJob
Grab change
token from DB
Persis last used token
per subscription
SQL
Azure
DBPersist token per
subscription

24. The WebHook payload is very small
Notifications are batched because processing depends on the CSOM GetChanges() call

25. PATCH /_api/web/lists('list-
id')/subscriptions(‘subscriptionID’)
Have a background job that regularly renews the needed subscriptions (recommended model)
Renew at notification time (will drop WebHook if there’s no event within the defined expiration window)

30. Visual Content
Intelligence
By Diego Grez - Radiografía_pulmones_Francisca_Lorca.jpg, CC BY-SA 3.0,
https://commons.wikimedia.org/w/index.php?curid=10302947

33. Questions?

34. markus@hintner.cc
@MarkusHintner