ai SUMMIT Speaker

1. Classified as Microsoft Confidential
Intro
Maisa Penha
Principal TPM
Azure OpenAI
www.linkedin.com/in/maisapenha

2. Art of the Possible: Visual Extraction and Reasoning
Input
Prompt
Completion
Can you explain this meme?
This meme is a joke that combines two unrelated things: pictures of the earth from space and chicken nuggets.
The text of the meme suggests that the image below is a beautiful picture of the earth from space. However, the image is
actually of chicken nuggets arranged to vaguely resemble a map of the world.
The humor in this meme comes from the unexpected juxtaposition of the text and the image. The text sets up an expectation of
a majestic image of the earth, but the image is actually something mundane and silly.

3. Art of the Possible: Web Page Design from a Visual
Input
Prompt
Completion

4. Art of the Possible: Visual Extraction and Reasoning
Input
Prompt
Completion
What are a couple meals I can make with this?
Based on the items visible in this fridge, here are a couple of meal ideas:
1) Yogurt parfait: Layer yogurt, strawberries and blueberries in a bowl or jar. Top with honey or nuts for added texture and
flavor.
2) Carrot and hummus wrap: Spread hummus on a tortilla or wrap. Top with shredded carrots and any desired veggies.
Roll up and enjoy!

5. Classified as Microsoft Confidential
Microsoft is powered by Azure AI
Applications Partner Solutions
Application Platform
AI Builder
Power BI Power Apps Power Automate Power Virtual Agents
Scenario-Based
Services
Bot Service Cognitive Search Document Intelligence Video Indexer Metrics Advisor Immersive Reader
Customizable AI
Models
Vision Speech Language Decision
Azure OpenAI
Service
ML Platform Azure Machine Learning

6. Classified as Microsoft Confidential
Microsoft and OpenAI partnership
Ensure that artificial
general intelligence
(AGI) benefits humanity
Empower every person
and organization on the
planet to achieve more
Azure OpenAI Service
GPT-4 &
GPT-3.5-Turbo
Text
Chat Completions
Conversation
DALL·E 2
Image
Whisper
Transcription &
Translation

7. Artificial Intelligence
1956 Artificial Intelligence
the field of computer science that seeks to create
intelligent machines that can replicate or exceed
human intelligence
Machine Learning
1997 Machine Learning
subset of AI that enables machines to learn from
existing data and improve upon that data to make
decisions or predictions
Deep Learning 2017 Deep Learning
a machine learning technique in which layers of
neural networks are used to process data and make
decisions
Generative AI
2021 Generative AI
Create new written, visual, and auditory content
given prompts or existing data.

8. Foundation Models
Training Data
Text
Images
Speech
Structured Data
3d Signals
Tasks
Question and
Answering
Sentiment Analysis
Information
Extraction
Image Captioning
Object Recognition
Training Adaptation
Foundation
Model
Instruction Follow
Transformer Model
Code Generation
Code
Understanding

9. DALL·E
GPT-3.5-Turbo
GPT-4
Generative Text Models, with varying capabilities and uses
Generative
Image Model
Azure OpenAI Service
Provisioned
throughput model
Functions and
Plugins
Deploy on your
own data

10. What is Prompt Engineering?
 Prompt engineering is a concept in Natural Language Processing
(NLP) that involves embedding descriptions of tasks in input to
prompt the model to output the desired results.
 Prompt typically includes problem descriptions, instructions on how
to solve the problem, and examples of correct problem and solution
pairs.

11. Azure OpenAI | GPT Ideate, Experiment and Fine-Tuning
Iterate on ideas with
a general-purpose
text-in/text-out interface
Prompt
Summarize game commentary
into highlights:
Shey Peddy is applying ball pressure at
the top against Sabrina Ionescu. At
7:48 remaining in the quarter; Peddy
What are the main highlights of the
game so far?
Sample response
The game has been close with Phoenix
leading New York 7-5. Shey Peddy has
been key for Phoenix.
Prompt
Turn game commentary into highlights:
Commentary: What a pickup she has
Main highlights: New York has domina
###
Commentary:
1. Turner is so important defensively to
2. Griner pulled way out, Hartley with
3. At 1:54 remaining in the quarter, Pho
Examples
Inputs
Sample response
Main highlights:
1. New York has had a strong run in the
2. Phoenix leading by 1 point, 24-23
3. New York Liberty's comeback has be
Results
Prompt and completion examples
Custom-tuning
"hyperparams": {
"batch_size": 4,
"learning_rate_multiplier": 0.1,
"n_epochs": 4,
"prompt_loss_weight": 0.1,
"use_packing": true
}
Refine with examples
(‘few shot learning’) with
a simple UX
Optimize accuracy and
latency to validate proof
of concept fast
Azure OpenAI
Service

12. What is Fine-Tuning?
Fine-tuning is a way of utilizing transfer learning. Specifically, fine-tuning
is a process that takes a model that has already been trained and tune it
using a labeled dataset for a specific task.
Fine-tuning results in a new model being generated with
updated weights and biases. This contrasts with few-shot learning in
which model weights and biases are not updated.
To fine-tune a model, you'll need a set of training examples that
each consist of a single input ("prompt") and its associated output
("completion").

13. Reduce Hallucination using Prompt Engineering
Include
Include
instructions of
requesting the
model not to
make up stuff
but stay with
facts.
Restrict
Restrict the
output (e.g.,
choose from a
confined list
instead of
generating free
form strings)
Add
Add Chain of
Thought style
of instruction,
"Solve the
problem step
by step."
Repeat
Repeat most
important
instructions in
the prompt a
couple of
times.
Position
Position most
important
instructions in
the last.

14. Prompt Engineering Guide
1. Give clearer instructions
2. Split complex tasks into simpler subtasks
3. Structure the instruction to keep the model on task
4. Prompt the model to explain before answering
5. Ask for justifications of many possible answers, and then synthesize
6. Generate many outputs, and then use the model to pick the best one
7. Custom-tune custom models to maximize performance

15. Classified as Microsoft Confidential
Retrieval Augmented Generation
Grounding for intelligent applications
App UX Orchestrator
Azure OpenAI
Service
Azure Cognitive Search Data Sources
(files, databases, etc.)
Query →
Knowledge
Prompt + Knowledge →
Response
Transform into
Embeddings
-2, -1 , 0, 1
2, 3, 4, 5
6, 7, 8, 9

16. Classified as Microsoft Confidential
Retrieval Augmented Generation Example: Bring your data to the
prompt
Prompt
You are an intelligent assistant helping
Contoso, Inc. employees with questions
about their healthcare plan as well as the
employee handbook. Answer the following
question using only the data provided in the
sources below.
Question: Does my health plan cover annual
eye exams?
Sources:
1. Northwind Health Plus offers coverage for
vision exams, glasses, and contact lenses, as
well as dental exams, cleanings, and fillings.
2. Northwind Standard only offers coverage for
vision exams and glasses.
3. Both plans offer coverage for vision and
dental services.
Response
Based on the provided information,
it can be determined that both
health plans offered by Northwind
Health Plus and Northwind Standard
provide coverage for vision exams.
Therefore, your health plan should
cover annual eye exams.
Text input that provides some
framing as to how the engine
should behave
User provided question that
needs to be answered
Sources used to
answer the question

17. Classified as Microsoft Confidential
Introduction to top use cases
Business
Problem
Productivity is lagging
Need for process
Automation
Degraded Customer
Experience
Creating Content is Time
Consuming
Business
Needs
Increase Productivity Automate Processes
Improve Customer
Experience
Build Creative Content
Solutions
• Conversational
Search/Knowledge
Insights
• Code Generation and
Documentation
• Trend Forecasting
• Report Summarization &
Generation
• Document Processing
• Workflow Management
• Fraud Detection
• Supply Chain
Optimization
• Intelligent Contact
Center
• Agent/Employee
Assistance
• Virtual Assistance
• Call Analytics
• Call Summarization
• Marketing/Sales Content
Generation
• Personalized Content
Generation
• Product Design &
Development
• Digital Art
What can
Generative
AI Do?
Generate New Revenue Streams
Deliver Differentiated Customer Experiences
Modernize Internal Processes

18. Classified as Microsoft Confidential
• Private Data Access​
• Prompt Engineering​
• Performance and Evaluation​
• Versioning and Reproducibility​
• AI Safety
Production
Demo
• Prompt Engineering process need for a streamlined approach
• Prompt engineering is iterative and challenging, involving several steps
Data
preparation
Crafting
tailored
prompts
Executing
prompts using
the LLM API
Evaluate the
generated
content
Optimization in
iteration
Production
design, finetune, evaluation,
refinement, and versioning
Challenges from Demo to
Production

19. Classified as Microsoft Confidential
AzureML's Prompt Flow
Capabilities Overview
• Develop workflows
• Develop flows that connect to
various language models,
external data sources, tools,
and customer codes​…
• Test and Evaluate
• Test with large dataset in parallel.
• Evaluate the AI quality of
the workflows with metrics:
• performance, groundedness
, accuracy…
• Prompt Tuning
• Easy prompt tuning​ with variants
and versions.
• Compare and Deploy
• Visually compare cross experiments.
• One click deploy a workflow to
endpoint and test apps​.

20. Classified as Microsoft Confidential
Features and Benefits
✓ Experimentation Platform​
✓ Optimized for tuning and
iterating LLM empowered
workflow​
Evaluate
Bulk Test
Flow
Tune
Deploy
Dataset