This document discusses Microsoft's Azure OpenAI Service and how it can be used to build applications using large language models. Some key points: - Azure OpenAI Service provides access to models from OpenAI like GPT-3 through Microsoft's Azure cloud platform while ensuring security, privacy and responsible AI. - It allows generating complex documents, steering models with nuanced instructions, and customizing models for any language or dialect. - Example capabilities include content generation, summarization, code generation, and semantic search. These can be applied to use cases like call center analytics, software documentation, and marketing content creation. - Tools are discussed for developing applications using prompt engineering, grounding models with domain-specific

1. ChatGPT and not only:
how can you use the
power of Generative AI
at scale
Maxim Salnikov
DeveloperProductivity Business LeadatMicrosoft

2. • Building on web platform since 90s
• Organizing developer communities and technical conferences
• Speaking, training, blogging: Webdev, Cloud, OpenAI
Helping developers to succeed with the Cloud & AI technology
I’m Maxim Salnikov

3. 87%
of organizations
believe AI will give
them a competitive
edge
50%
of organizations have
adopted AI in at least
one business area
Sources: MIT Sloan Management Review, The state of AI in 2022--and a half decade in review | McKinsey
Why AI?

4. Powered by Azure AI
Applications Partner Solutions
Application Platform
AI Builder
Power BI Power Apps Power Automate Power Virtual Agents
Scenario-Based
Services
Bot Service Cognitive Search Document Intelligence Video Indexer Metrics Advisor Immersive Reader
Customizable AI
Models
Vision Speech Language Decision
Azure OpenAI
Service
ML Platform Azure Machine Learning

5. Microsoft and OpenAI partnership
Ensure that artificial
general intelligence
(AGI) benefits humanity
Empower every person
and organization on the
planet to achieve more
Azure OpenAI Service
GPT-4 &
GPT-3.5-Turbo
Text
Chat Completions
Conversation
DALL·E 3
Image
Whisper
Transcription &
Translation

6. Generate complex documents
Steer with nuanced instructions
Instruct and annotate in any language,
slang, dialect
GPT-4

7. Azure
OpenAI Service
Deployed in your Azure subscription,
secured by you, and tied to your datasets
and applications
Large, pretrained AI models to unlock
new scenarios
Custom AI models fine-tuned with your
data and hyperparameters
Built-in responsible AI to detect and
mitigate harmful use
Enterprise-grade security with role-based
access control (RBAC) and private networks

8. Capabilities and use cases

9. Main capabilities
Content generation Summarization Code generation Semantic search
Call center analytics:
automatically generate
responses to
customer inquiries
Generate personalized
UI for your website
Call center analytics:
summary of customer
support conversation lo
gs
Subject matter expert
document: summarization
(e.g. Financial reporting,
analyst articles)
Social media trends
summarization
Convert natural
language to SQL (or vice
versa) for
telemetry data
Convert natural
language to query
proprietary data models
Code documentation
Search reviews for a
specific product/service
Information discovery
and knowledge mining

10. Solving business problems
Business
Problem
Productivity is lagging
Need for process
Automation
Degraded Customer
Experience
Creating Content is Time
Consuming
Business
Needs
Increase Productivity Automate Processes
Improve Customer
Experience
Build Creative Content
Solutions
• Conversational
Search/Knowledge
Insights
• Code Generation and
Documentation
• Trend Forecasting
• Report Summarization
& Generation
• Document Processing
• Workflow Management
• Fraud Detection
• Supply Chain
Optimization
• Intelligent Contact
Center
• Agent/Employee
Assistance
• Virtual Assistance
• Call Analytics
• Call Summarization
• Marketing/Sales
Content Generation
• Personalized Content
Generation
• Product Design &
Development
• Digital Art
What can
Generative
AI Do?
Generate New Revenue Streams
Deliver Differentiated Customer Experiences
Modernize Internal Processes

11. Content
creation
& design
Accelerated
automation
Personalized
marketing
Chatbots and
virtual assistants
Product &
service
innovation
Language
translation &
language
processing
Fraud detection
Predictive
analytics &
forecasting
Creative writing
& content
generation
Medical
research &
diagnosis
Top 10 usecases

12. How to start building LLM app?

13. Build on solid foundation
Apps
Plugin extensibility
Copilots
Prompt Flow & Model Evaluation
Metaprompt
Data grounding
Plugin execution
Foundation models
AI infrastructure
Your applications
Azure OpenAI
Service
Built-in safety
system and
responsible AI tools

14. Grounding
is the process of using large language models (LLMs) with
information that is use-case specific, relevant, and not available
as part of the LLM's trained knowledge.

15. Prompt Engineering
Fine-tuning
LLMOps
Responsible AI
Vast majority
of use cases
Grounding options
Training

16. Prompt Engineering
Is the process of designing, refining, and optimizing input
prompts to guide a model toward producing more accurate
outputs while keeping cost efficiency

17. Prompt
Text input that provides
some framing as to how
the engine should behave
You are an intelligent assistant helping Contoso Inc
employees with their healthcare plan questions and
employee handbook questions. Answer the following
question using only the data provided in the
sources below.
Question: Does my health plan cover annual
eye exams?
Sources:
1. Northwind Health Plus offers coverage for vision
exams, glasses, and contact lenses, as well as dental
exams, cleanings, and fillings.
2. Northwind Standard only offers coverage for
vision exams and glasses.
3. Both plans offer coverage for vision and
dental services.
User provided question
that needs to be answered
Sources used to
answer the question
Response
Based on the provided information,
it can be determined that both
health plans offered by Northwind
Health Plus and Northwind Standard
provide coverage for vision exams.
Therefore, your health plan should
cover annual eye exams.
Bringing data into the prompt

18. Will my sleeping
bag work for my
trip to Patagonia
next month?
User input
Historical weather
lookup
Intent mapping
Personalization Product info
Recommendations
engine
???
Prompt engineering LLM
Yes, your Elite Eco
sleeping bag is
rated to 21.6F,
which is below the
average low
temperature in
Patagonia in
September
Output
More context

19. Tools

20. Semantic Kernel
https://github.com/microsoft/semantic-kernel
https://github.com/langchain-ai/langchain
LangChain

21. 1. Prototyping
2. Experimenting
3. Iterating
4. Deploying
5. Optimizing
LLM applications

22. Operationalize
LLM app
development
• Private data access and controls
• Prompt engineering
• CI/CD
• Iterative experimentation
• Versioning and reproducibility
• Deployment and optimization
• Safe and Responsible AI
Design and development
Develop flow based on prompt
to extend the capability
Debug, run, and evaluate
flow with small data
Modify flow (prompts and
tools etc.)
No If satisfied
Yes
Evaluation and refinement
Evaluate flow against large
dataset with different metrics
(quality, relevance, safety, etc.)
If satisfied
Yes
Optimization and production
Optimize flow
Deploy and
monitor flow
Get end user
feedback

23. Prompt flow for LLMOps
• Extensive evaluation capabilities for prompt
engineering workflows
• Prompt flow definitions as first-class entities
(YAML)
• Managed API connections for CI/CD across dev,
test, prod
• Multiple authoring interfaces including code-first,
CLI and UI
• Inter-op with Python libs like Guidance, Semantic
Kernel, and LangChain
Capabilities
https://github.com/microsoft/promptflow

25. Prompt flow ecosystem
VS code extension + Prompt flow package
 Install Prompt flow extension in VS Code desktop
 Flexibly switch envs with promptflow package installed
 Enjoy the similar authoring UI with AzureML
workspace
Prompt flow package in your IDE
 Pip install promptflow package
 Develop flow in any IDE
 Execution: flow init, test, run, visualize, etc.

26. Authoring
Develop your LLM flow from
scratch
• Construct a flow using pre-built
tools
• Support custom code
• Clone flows from samples
• Track run history

27. Prompt variants
• Create dynamic prompts using
external data and few shot
samples
• Edit your complex prompts in full
screen
• Quickly tune prompt and LLM
configuration with variants

28. Evaluation
• Evaluate flow performance with your
own data
• Use pre-built evaluation flows
• Build your own custom evaluation
flows
• Compare multiple variants or runs to
pick best flow
• Add new evaluations to a finished run
• Ensure accuracy by scaling the size of
data in evaluation

29. Deploy
• Seamless transition from development
to production with Azure ML’s
managed online endpoints

30. Prompt flow in Azure ML benefits
• Integrates into existing CI/CD processes to manage prompts
• Shorter time to higher quality prompts through experimentation
• Historical tracking of prompt authoring, metric validation and
certification
• Enterprise security for API connectivity, data access and deployment
https://ml.azure.com

31. App or
Copilot agent
API & SDK
Azure OpenAI
Service on your data
Data Sources
(search, files, databases, storage etc.)
Additional 3P Data Sources
(files, databases, storage data etc.)
Azure OpenAI Service on your data
https://learn.microsoft.com/en-us/azure/ai-services/openai/concepts/use-your-data

32. Ingest / Connect
● Connect your data source
whatever it is & wherever
it is
Ground, Chunk, Tune
& Tone
● Unlock the full
protentional of your data
Share & Use
● Share with your
customers & organization
Index, semantic search, vector
search, authenticate, personalize,
company policies and more
Documents, files,
Cognitive Search, blob, local file
upload ….
Easy to integrate within your
organization or with your
customers simple APIs, SDK,
Customized Web App
End-to-end RAG scaffolds

33. Projects from Norway

34. The solution to the size and fit problem is knowing the personal preferences of
your customers on each item-model-level

35. • Skjold.ai protects all traffic
• Check suspicious messages
• Cybercrime is growing
• Criminals adopt new technology
• Instant feedback if the
messages is considered safe
or could be fraud
• Explain red flags and
educate the user
• Integrated in the iPhone
SMS app
• Available for iPhone and
Android
AI-powered scan of suspicious
messages
Read more
www.skjold.ai/demo
Mobile first protection against cyber crime built with Azure OpenAI

37. Kahoot! launches new AI features for more powerful
learning
https://kahoot.com/kahoot-news/kahoot-microsoft-collaboration-ai-features/

38. Introducing Cognite AI, the Generative AI Accelerator for
Industrial Data and Value Realization
https://www.cognite.com/en/press-release/introducing-cognite-ai

39. Top questions answered

40. How is my data used in Azure OpenAI Service?
When you use Azure OpenAI Service, your prompts (inputs) and completions (outputs), your
embeddings, and your training data
Are NOT available to other customers.
Are NOT available to OpenAI.
Are NOT used to improve OpenAI models.
Are NOT used to improve any Microsoft or 3rd party products or services.
Are NOT used for automatically improving Azure OpenAI models for your use
in your resource (The models are stateless, unless you explicitly fine-tune
models with your training data).
Your fine-tuned Azure OpenAI models are available exclusively for your use.
The Azure OpenAI
Service is fully
controlled by
Microsoft; Microsoft
hosts the OpenAI
models in Microsoft’s
Azure environment and
the Service does NOT
interact with any
services operated by
OpenAI (e.g., ChatGPT,
or the OpenAI API).
https://learn.microsoft.com/en-us/legal/cognitive-services/openai/data-privacy

41. How does Responsible AI work in Azure OpenAI?
Customer Application
Prompt
Azure OpenAI
Endpoint
RAI
Filtered
Response
Responsible AI Model Ensemble
Hate
Sexual
Text
Images
Abuse Concern?
Azure AI Content Safety – Configurable Filters

42. Maxim.Salnikov@microsoft.com
Let’s connect and chat!
Thank you!