Azure Cognitive Services allow developers to build powerful AI-based solutions, enabling different capabilities in our software: vision. speech, search, text analytics, language understanding, and much more. Basically, the model is already built by Microsoft, you just need to do an API call to the Azure cloud and the service retrieves a result. For instance, you send a message and the Text Analytics API returns its sentiment score. However, there might be cases in which our customers need a local, non-cloud AI solution (either because of limited Internet access or data compliance). This is now possible thanks to the latest update of Azure Cognitive Services, which offers containerization support. Using containers, we can still deliver ML-driven solutions while keeping the data in-house. In this talk, we'll explore what it takes to configure and use containers in Azure Cognitive Services. Demos will be showcased as well for local Face and Text Cognitive Services.

1. Bringing AI to the Edge:
On-premise Azure Cognitive
Services using containers
Luis Beltrán

3. Luis Beltrán
• Microsoft MVP in Developer Technologies
• Researcher & PhD student at Tomas Bata University in Zlín
• Lecturer at Tecnológico Nacional de México in Celaya
@darkicebeam
luis@luisbeltran.mx

4. Agenda

5. Agenda
• Cognitive Services
• Docker Containers
• Container Support in Azure Cognitive Services
• Requirements
• Demo

6. Cognitive Services

7. We learn and
use technology.
Technology learns us
and discovers ways to be helpful.
Answering
Coordinating
Listening
Observing
Anticipating

8. What are Azure Cognitive Services?
“Enable your apps to see, hear, interpret, and interact in more human
ways.”
“An API for accessing the real world”
“Really hard problems Microsoft has already solved so you don’t
have to”

13. Text Analytics API

15. Computer Vision API

18. A variety of real-world applications
Vision Speech Language
Natural Language Processing
Intent: PlayCall
Knowledge
Here are the top results:
The purpose of Customer Life-cycle Management (CLM)
is to maximize both customer retention and .... Predictive
trend analysis provides business visibility.
Oct 28, 2015 – Here are FIVE key trends in 2014 that
would help marketers in rolling ... Of late, marketers are
looking at customer lifecycle management (CLM)
Jan 5, 2016 – The top 10 customer service trends for
2016 that .... North American Consumer
Search
Here is what I found:
It also investigates the top three expected
Fraud Detection and Prevention programs, in
terms of demand in key markets…
First, let’s point out that there is not one
absolute answer—there are “pros” and “cons” to
each. Those who favor in-house…
Michael heads fraud prevention tool. Online
and mobile shopping are expected to continue
growing apace…

19. Docker Containers

21. Containers

24. Virtual Machines
Virtual Machine Scale
Sets
Containers
Azure App
Services
Azure
Functions
Where do containers fit?

25. Docker integration
https://hub.docker.com

26. PowerShell for Docker
https://github.com/Microsoft/Docker-PowerShell/
Get-Container | Stop-Container

27. Docker for Windows

28. Containerization is an approach to software
distribution in which an application or service,
including its dependencies & configuration, is
packaged together as a container image.
With little or no modification, a container image can
be deployed on a container host.

29. Containers are isolated from each other and the
underlying operating system, with a smaller footprint
than a virtual machine.
Containers can be instantiated from container images
for short-term tasks, and removed when no longer
needed.

30. Container Support in
Azure Cognitive Services

31. • What if you can't upload an image or text to the cloud?
• Sending data outside your network might be subject to
regulatory or privacy policies.
• If you could analyze the images or text locally, your
application could benefit from reduced latency and
bandwidth.
Now, several of the Azure Cognitive Services APIs are
available as Docker containers.

32. • You can download a container that provides the exact same APIs
as the cloud-based services, and run it on a local Linux-based
server or edge device.
• Images and text are processed directly in the container and never
sent to the cloud.
• A connection to Azure is required only for billing, which is at the
same rate as the cloud-based services (including a free tier).
Container support in Azure Cognitive Services allows developers
to use the same rich APIs that are available in Azure, and enables
flexibility in where to deploy and host the services that come with
Docker containers.

33. Features and benefits
• Run AI Cognitive Services on-premises
• Process lots of data (High throughput)
• No constraints in network connection (low latency)
• Portable architecture

34. Features and benefits
Control over data: Allow customers to choose where
these Cognitive Services process their data.
This is essential for customers that cannot send data to
the cloud but need access to Cognitive Services
technology.
Support consistency in hybrid environments – across data,
management, identity, and security.

35. What’s available?
Currently available in preview for parts of:
• Computer Vision
• Face
• Text Analytics
• and Language Understanding (LUIS).

38. Requirements

39. Prerequisites
• Docker Engine: Docker provides packages that configure
the Docker environment on macOS, Linux, and Windows.
On Windows, Docker must be configured to support Linux
containers. Docker containers can also be deployed directly to
Azure Kubernetes Service or Azure Container Instances.
• Cognitive Service resource created on Azure: Docker
must be configured to allow the containers to connect
with and send billing data to Azure.

40. Prerequisites
Container requirements: Individual containers can
have their own requirements, as well, including server
and memory allocation requirements.

41. https://docs.docker.com/toolbox/toolbox_install_windows/
docker-machine stop
docker-machine rm default
docker-machine create -d virtualbox --virtualbox-cpu-count=2 --virtualbox-memory=2048 default

42. Demo
https://github.com/icebeam7/KrakowSentimentAnalysis

53. Resources
❑ Container support in Azure Cognitive Services
https://docs.microsoft.com/en-us/azure/cognitive-services/cognitive-services-
container-support
❑ Containers support of Cognitive Services (video)
https://azure.microsoft.com/en-us/resources/videos/containers-support-of-
cognitive-services/
❑ Cognitive Services
http://microsoft.com/cognitive

54. Resources
❑ Docker Images (Azure Cognitive Services)
https://hub.docker.com/_/microsoft-azure-cognitive-services
❑ Containers samples for Cognitive Services
https://github.com/Azure-Samples/cognitive-services-containers-samples
❑ Cognitive Services
https://azure.microsoft.com/en-us/blog/running-cognitive-service-containers/

55. Resources
❑ Install and Run LUIS docker containers
https://docs.microsoft.com/en-us/azure/cognitive-services/luis/luis-container-
howto
❑ Install and Run Recognize Text containers
https://docs.microsoft.com/en-us/azure/cognitive-services/computer-
vision/computer-vision-how-to-install-containers
❑ Running Cognitive Service containers
https://docs.microsoft.com/es-es/azure/cognitive-services/face/face-how-to-
install-containers

56. Resources
❑ Install and Run Text Analytics containers
https://docs.microsoft.com/es-es/azure/cognitive-services/text-analytics/how-
tos/text-analytics-how-to-install-containers

57. Luis Beltrán
THANK YOU!
Questions?
@darkicebeam
luis@luisbeltran.mx