Build enterprise-
grade AI agents with
Azure AI Agent
Service
SATO Naoki (Neo)
Principal Software Engineer
Microsoft Corporation
Every app
will be reinvented
with AI
New apps
will be built that
weren’t possible before
Copilot & AI stack
Copilot devices
Copilot
Visual
Studio
+
Copilot
Studio
GitHub
+
Azure
Cloud to Edge
Azure AI Foundry
Data
Infrastructure
Trustworthy AI
Copilot & AI stack
Agentic AI
A new frontier:
Agentic AI
use AI to automate and execute business processes,
nearly autonomously
AI agents can plan, adapt to new information, and
execute tasks independently, making them capable
of handling complex, dynamic environments,
decision-making and autonomous action. They can
learn from feedback, adjust their strategies, and
operate with minimal human oversight.
Reason over a provided business process
Retrieve context to complete the process
Perform an action for the end-user
K
Inefficiency
and High
Operational
Costs
Organizations face significant challenges
with complex, manual, and time-consuming
workflows and minimal automation
Lack of
Visibility &
Control
Human Error
Scalability
Issues
Promising even more efficiency, value, and advantage
VALUE
Yesterday
Today
User
Chatbot
Able to answer questions
“What is the dress code
in the office?”
User
Single AI Agent
Able to take action
“Order a laptop for a new
employee”
Human Supervisor
User
HR Agent
Able to collaboratively solve
complex tasks
“Onboard 5 employees by
Monday”
Human Supervisor
IT Agent
Training Agent
User
RPA
Able to complete repetitive
tasks
“Inputs new hire information
into HR System”
Sends welcome
email
HR Manager
IT Manager
Training Manager
New Hire
Manually sets up
user accounts and
provisions access
Gets new hire
forms
Has questions
Answers
questions
Completes
paperwork
Confirm computer
preference
Confirmed
Initiates IT
process
Missing
applications
submits IT
ticket.
Contacts
procureme
nt to obtain
license.
Initiates
training
process
Schedule
conflict
Reviews
training
schedule and
reschedules
Sends feedback
survey
Waits for more
data before
analyzing and
optimizing the
entire process.
Completes the
training still has
questions
TODAY
Existing solutions can only automate very specific tasks that have clear inputs and outputs
Initial Interaction Document Verification
Access &
Technology Provisioning
Training
Onboarding Feedback
Loop
Survey is
completed
RPA automates the verification of
completion and accuracy
with predefined rules.
RPA automates assigning predefined training
modules but cannot adjust based on user
feedback
• Provides a personalized
onboarding journey based
on interaction
IT Agent
Training Agent
New Hire
TOMORROW
With AI Agents, these steps can be automated for the first time, while keeping human in the loop.
HR Agent
• Assesses documents and
learns from interactions
• Analyzes role, experience, and learning
preferences to recommend training
• Gathers real-time feedback
• Identifies patterns makes informed
decisions
Provides adaptive
training
• Sets up user accounts, adapts to
troubleshoot unexpected issues, and
learns from errors.
Fills out forms and has no
questions
The new hire has
questions during the
training.
Orchestrates the additional
processes
Human in the Loop, Supervisor and Approver
Initial Interaction Document Verification
Access &
Technology Provisioning
Training
Onboarding Feedback
Loop
Key Applications of
AI Agents
This cutting-edge shift promises to ignite a fresh
wave of productivity and efficiency
Intelligent Decision making
Increased Productivity
Enhanced Efficiency
Cost Reduction
Automation is undergoing a thrilling
transformation with autonomous
agents now being designed to
proactively plan actions and perform
tasks for users.
AI Agent workflows can enhance efficiency, accuracy, and customer
satisfaction
Key Use Cases Across Industries
• Assists employees in
booking business trips
• Integrates with Tripadvisor,
Outlook, and SharePoint
• Books via Teams chat or
email
• Uses OCR to gather receipts
• Automates expense report
submission and tracking
Travel Booking &
Expense Management
• Personalized onboarding
assistant for new hires
• Uses LLMs grounded in HR
data from SharePoint
• Provide relevant training
materials
• Schedule orientations and
set up software accounts
• Monitor task completion
and ensure efficient
onboarding
Employee Onboarding
• Diagnoses issues by
referencing history and
product manuals
• Provides tailored solutions
or escalates through
automated workflows
• Creates tickets and
schedules follow-ups
• Updates CRM records,
enhancing future support
Personalized Customer
Support
• Analytics data from data
lake and data warehouse
• Responds to user requests
in natural language
• Generates insights,
visualization, and sends via
Teams or email
• Automates data handling
for real-time, effortless
decision-making
Data Analytics and
Reporting
Challenges in Development
with AI Agents
4 primary considerations
K
Providing agents
with the right
context
Knowledge Evaluation
Ensuring they
complete their
tasks correctly
Actions
Giving them the
tools to
complete their
tasks
Security
Ensuring they
only have access
to what they
should
Successfully developing AI Agents requires
So far, building agents from scratch been difficult to do
Tool Integration
Creating a cohesive system through complex integration of various
tools and APIs that have different interfaces, data formats, and
requirements.
Interoperability
Achieving interoperability between different tools and platforms to
ensure that data can be shared and understood across different
systems.
Scalability
Handling increased data volumes, more complex computations, and
higher user loads without degrading performance.
Real-time Processing
Ensuring tools can handle real-time requirements without significant
latency.
Maintenance
Making labor–intensive updates to integrated tools for compatibility
with new versions and prevention of obsolescence and security
vulnerabilities.
Flexibility
Modifying or customizing existing tools or developing new ones to
meet unique requirements.
Error Handling
Ensuring errors are handled gracefully and continue functioning
despite tool failures or unexpected inputs is critical for reliability.
Security
Implementing robust encryption, access controls, and compliance
with privacy regulations to protect sensitive data.
Organizations need platforms that enable rapid
development of performant, secure AI Agents
Current Frameworks
Security and data privacy risks
What’s Needed
Secure, responsible AI that protects
sensitive information and behaves
compliantly
Lack of integrated tools, insecure data
grounding, challenging orchestration
Ineffective deployment of AI across websites,
applications, and production environments
Restrictive pre-defined models that are
challenging to customize
Flexible models that enable processing and
integration of information from multiple
modalities or types of data
Connected complex workflow automation
grounded by seamless connection to
enterprise data
Tools and APIs that seamlessly integrate
across enterprise applications
Visual Studio
Copilot Studio
GitHub
Azure AI
Foundry SDK
Azure AI Foundry
Model Catalog
Open-source models
Foundational models Task models Industry models
Azure AI
Content Safety
Azure
AI Search
Azure AI
Agent Service
Azure OpenAI
Service
Observability
Customization
Evaluations Governance Monitoring
Announcing
Azure AI Agent Service
Public Preview
Empower developers to securely build, deploy, and scale AI agents with ease
Flexible Model
Selection
Extensive Data
Connections
Enterprise-grade
Security
Rapid Development
and Automation
AI.Azure.com
The full package
Built-in enterprise
readiness
BYO-file storage
(coming soon)
BYO-search index
Azure AI Foundry SDK – Agent Service
OBO Authorization Support
Enhanced Observability
Extensive Ecosystem of Tools
Knowledge
Microsoft Fabric*
SharePoint*
Grounding with Bing Search
Azure AI Search
Your own licensed data*
Files (local or Azure Blob)
File Search
Code Interpreter
Actions
Azure Logic Apps*
OpenAPI 3.0 Specified Tools
Azure Functions
Model
Catalog
Azure OpenAI Service
(GPT-4o, GPT-4o mini)
Models-as-a-Service
Llama 3.1-405B-Instruct
Mistral Large
Cohere-Command-R-Plus
*Indicates feature is coming soon
Introducing
Azure AI Foundry SDK – Agent Service
Azure OpenAI
Assistants API
+ +
Extensive Ecosystem of Tools
Model Catalog
Actions
Azure AI Search
Microsoft Fabric (coming soon)
SharePoint (coming soon)
Grounding with Bing Search
Azure Logic Apps (coming soon)
OpenAPI 3.0 Specified Tools
Azure Functions
Your own licensed data (coming soon)
Files (local or Azure Blob)
File Search
Code Interpreter
Azure OpenAI Service
(GPT-4o, GPT-4o mini)
Models-as-a-Service
Llama 3.1-405B-Instruct
Mistral Large
Cohere-Command-R-Plus
Built-In Enterprise Readiness
BYO-file storage
(coming soon)
BYO-search index BYO-thread storage BYO-virtual network
(coming soon)
OBO Authorization Support Enhanced Observability
Knowledge
How does your agent work?
User
Travel Booking
Agent
“Help me book a trip to New York
for a client meeting? I need to fly out
next Monday and return on Friday.” Knowledge Sources
(search, files, databases, storage etc.)
Models
(Azure OpenAI Service, Models-as-a-
Service)
Actions
(Pre-built or custom tools to
automate processes)
“I’ve booked your trip to New York
as requested. Here are details:…”
How does your agent work?
Step 1:
Create an Agent
Step 2:
Create a Thread
Step 3:
Run the Agent
Step 5:
Check the Run status
Step 6:
Display the Agent’s
Response
Agent
Travel Planning Agent
Instructions
You are a travel booking and
expense management assistant
designed to help employees plan,
book, and manage business
travel.
Run 2
Model
Tools (optional)
File Search
Code Interpreter
Function Calling
Bing Search
Microsoft SharePoint (coming soon)
Microsoft Fabric (coming soon)
Azure Logic Apps (coming soon)
Azure Functions
OpenAPI 3.0 specified tools
User’s message
I need to book a hotel
in New York for 2 stays.
Agent’s message
Here are some
suggestions:
Run 1
1
Use Tripadvisor API
to search the
nearest hotel
Create message
2
Your data (optional)
User’s message
What’s the daily meal
allowance for the
business trip?
Agent’s message
The daily allowance for
your business trip is
$75, as per company
policy.
Use Microsoft
SharePoint to
query the
company travel
policy
Create message
2
1
Thread
Travel Planning
Azure AI Search
Files (local or Azure Blob)
Chat Completions API vs Assistants API vs Agents
Assistants API
Chat Completions API
• Lightweight and powerful
• Inherently stateless
• Assistants API features, plus
• Build with a model of your choice
(OpenAI, Llama, Mistral, Cohere…)
• Real time web-grounding with Bing
• Secure grounding on enterprise
data in SharePoint and Fabric
• Bring Your Own Licensed data
(Tripadvisor)
• Connect to 1400+ data sources and
services with Azure Logic Apps
• Long running, event driven actions
with Azure Functions
• Standardized OpenAPI 3.0 tools
• Bring Your Own Storage
• Bring Your Own Private Network
• Bring Your Own AI Search Resource
• Limitless scaling with PTUs
• Open Telemetry based tracing
vs
Azure AI Agent Service
• Build with OpenAI models
• Stateful (inbuilt conversation state
management)
• Access persistent threads
• Automatic management of the
model’s context window
• Access files in several formats
• Utilized Microsoft-managed
storage
• File Search (API handles chunking,
embeddings storage and
creation, and implementing vector
search)
• Code Interpreter
• Function Calling
vs
Customers are innovating with Azure AI Agent Service
Demo
https://youtu.be/ltt7JNd30Ag?si=Y2Cbdx8YZOPFWJzB&t=1506
Rapid
Development
and Automation
Rapid Development and Automation
Out-of-the-box
connectors
Connection with
3P APIs
Custom code Code
Interpreter
Client-side
functions
Automation
1,400 built-in
connectors through
Azure Logic Apps
(coming soon)
• Create and run automated workflows
with Azure Logic Apps with little to no
code using the visual designer and
selecting from prebuilt operations.
• Simply define the business logic for
your workflow to connect your agent to
external systems, tools, and APIs.
• Microsoft connectors include Microsoft
products such as Azure App Service,
Dynamics365 Customer Voice,
Microsoft Teams, M365 Excel
• Non-Microsoft connectors
include enterprise services such as
MongoDB, Dropbox, Jira, Gmail, Twilio,
SAP, Stripe, ServiceNow and many
more.
Playground experience in Azure AI Foundry portal is coming soon in January
Automation
Build standardized
custom tools with
OpenAPI 3.0
specifications
• Connect your AI agent to an external
API using an OpenAPI 3.0 specified tool,
allowing for scalable interoperability
with various applications
• Build an organization-side tools library
specified with the OpenAPI 3.0 standard
• Enable your custom tools to
authenticate access and connections
with managed identities (Microsoft
Entra ID) for added security, making it
ideal for integrating with existing
infrastructure or web services
Screenshot
Playground experience in Azure AI Foundry portal is coming soon in January
Automation
Implement stateless
or stateful code-
based actions with
Azure Functions
• Enable your agent to perform external
interactions with the world, such as
calling APIs or asynchronously sending
and waiting for events.
• Azure Functions and Azure Durable
Actions enable you to execute
serverless code for synchronous,
asynchronous, long-running, and
event-driven actions such as
approving invoices with human-in-the-
loop, monitor an end-to-end product
supply chain over long periods of time,
and many more.
Screenshot
Playground experience in Azure AI Foundry portal is coming soon in January
Automation
Perform advanced
data analysis with
Code Interpreter
• Code Interpreter allows your agent to
write and run Python code in a
sandboxed execution environment,
handling diverse data formats and
generate files with data and visuals.
• With Code Interpreter enabled, your
agent can run code iteratively to solve
more challenging code, math, and data
analysis problems.
• Unlike the Assistants API, you can
integrate data in your storage to use
with this tool.
Screenshot
Playground experience in Azure AI Foundry portal is coming soon in January
Automation
Extend Llama Stack
agents with cloud-
based tools
• Supports the agent protocol for
developers that are using Llama Stack
SDKs
• Natively offer a scalable, cloud-hosted,
enterprise grade tools
• Wireline compatible with Llama Stack
• Protocol adapter will be a standalone
distributable, discoverable from the
llama-stack OSS repo directly
Screenshot
Playground experience in Azure AI Foundry portal is coming soon in January
Extensive
Data
Connections
Extensive Knowledge Connections
Public web data Corporate data Structured data Unstructured
data
Licensed data
Data Connections
Real-time, web data
grounding with Bing
Search
• Create more reliable and trustworthy
applications with real-time public
information from Bing
• Grounding with Bing Search allows your
agents to integrate real-time public
web data, ensuring their response is
accurate and up to date.
• By including supporting URLs and
search query links, Grounding with Bing
Search enhances trust and
transparency, empowering the users to
verify responses with the original
sources.
Playground experience in Azure AI Foundry portal is coming soon in January
Data Connections
Grounding with
private data in
Microsoft SharePoint
(coming soon)
• Grounding with SharePoint makes your
SharePoint content more accessible to
your end users.
• Enterprise-grade security features, such
as On-behalf-of (OBO) authentication
for SharePoint, ensure secure and
controlled access for end users.
Playground experience in Azure AI Foundry portal is coming soon in January
Data Connections
Chat with your
structured data in
Microsoft Fabric
(coming soon)
• Integrate your agents with Fabric AI
Skill to unlock powerful data analysis
capabilities.
• Fabric AI Skills can transform
enterprise data into conversational
Q&A systems, allowing users to
interact with data through chat and
uncover data-driven and actionable
insights effortlessly.
• On-behalf-of (OBO) authentication
simplifies access to enterprise data in
Fabric while maintaining robust
security, ensuring proper access
control and enterprise-grade
protection.
Playground experience in Azure AI Foundry portal is coming soon in January
Data Connections
Incorporate private
data in Azure AI
Search, Blob, and
your local files
• Bring your existing Azure AI Search
index or create a new one using
the improved File Search tool.
• This tool leverages a built-in
ingestion pipeline to process files
from your local system or Azure
Blob Storage.
• Maintain complete control over
your data as your files remain in
your own storage, and your Azure
AI Search resource ingests them
Playground experience in Azure AI Foundry portal is coming soon in January
Data Connections
Gain a competitive
advantage with
licensed data
(private preview)
• Empower your agents to deliver
nuanced, informed answers tailored to
specific use cases
• Integrate licensed data from
specialized data providers, such as
Tripadvisor, to enrich agent responses
• Tripadvisor enhances the quality of
your agent’s responses to travel
related queries with high-quality, fresh
data, such as travel guidance and
reviews (coming soon)
Playground experience in Azure AI Foundry portal is coming soon in January
Flexible
Model
Selection
Flexible Model Selection
Azure OpenAI
Service models
Models as a
Service
(Serverless API)
Azure AI
Services models
Multi-modal
support
Fine-tuned
models
Flexible Model Selection
Leverage Models as
a Service (MaaS)
• Support a variety of models from Azure
AI Foundry model catalog to power
your agent’s reasoning. The supported
models include:
• GPT-4o
• GPT-4o mini
• Llama 3.1
• Mistral Large
• Cohere Command R+
• Leverage Model Inference API to easily
swap models and compare
performance to find the best model for
your specific needs.
Flexible Model Selection
Use fine-tuned
models for your
agents
• Customize AI models to address your
unique business needs with fine-tuning,
enabling you to build task-specific
agents while optimizing token costs.
• Through collaborations with partners
like Scale AI, you can efficiently label
training data and create fine-tuned
models that integrate seamlessly with
Azure AI Agent Service.
• This process enhances the performance
of AI agents, streamlining development
and reducing time to production.
Azure AI Agent Service supports fine-tuned gpt-35-turbo (0125) as of Jan 2025
Flexible Model Selection
Multi-modal support
across text, image,
and audio
• Unlock new scenarios with multi-modal
support, enabling AI agents to process
and respond to diverse data formats
beyond text, expanding the potential
use cases.
• Support for GPT-4o’s image and audio
modalities so that you can analyze and
combine data from various formats to
deliver comprehensive insights, make
decisions, and provide relevant outputs
tailored to specific user needs
Playground experience in Azure AI Foundry portal is coming soon in January
Enterprise-grade
Security
Enterprise-grade Security
Bring your own
storage
Keyless setup
and
authentication
Private Virtual
Network support
Tracing/
monitoring
Content filters
Security
Bring your own
storage and search
• Bring your own storage account
(coming soon) and Azure AI Search
resource for custom data handling.
• All files you upload will be stored in
your storage account, ensuring you
maintain complete control over your
data.
• All indexes get created using your
connected Azure AI Search resource.
Playground experience in Azure AI Foundry portal is coming soon in January
Security
Bring your own
virtual network
(coming soon)
• No public egress foundational
infrastructure ensures the right
authentication and security for your
agents and tools, without you having to
do trusted service bypass
• Container injection allows the platform
network to host APIs and inject a
subnet into your network, enabling
local communication of your Azure
resources within the same virtual
network (VNet).
• If your resources are marked as private
and non-discoverable from the Internet,
the platform network can still access
them, provided the necessary
credentials and authorization are in
place
Security
Monitor agent
performance with
OpenTelemetry-based
training (coming soon)
• Gain insights into your AI agent's
performance and reliability
• OpenTelemetry-compatible metrics for
offline and online evaluation of agent
outputs through the Azure AI Foundry
SDK
• Add local variables and intermediate
results to trace decorator for detailed
tracing capabilities for user defined
functions
• Options to prevent sensitive or large
data logging as per OpenTelemetry
standards
Security
Build responsibly
with content filters
and XPIA mitigation
(coming soon)
• Supports prebuilt and custom content
filters that detect harmful content at
varying severity levels.
• Prompt shields protect agents against
cross-prompt injection attacks from
malicious actors.
• As with Azure OpenAI Service, prompts
and completions processed by the
Azure AI Agent Service are not used to
train, retrain, or improve Microsoft or
3rd party products or services without
your permission. Customers can delete
their stored data when they see fit.
Basic agent setup
• Overview: All agents created in this project use multi-
tenant search and storage resources fully managed by
Microsoft. You don't have visibility or control over these
underlying Azure resources.
• Required customer resources:
• AI hub
• AI project
• AI Services/AOAI
• How to use: Deploy the basic setup template
• Tool Implications:
• File search and code interpreter
• Uploaded files get stored in Microsoft managed
storage
• Vector stores get created using a Microsoft
managed search resource
• Azure AI Search tool is not supported
• Azure Blob Storage with file search is not supported
Standard agent setup
• Overview: Agents use customer-owned, single-tenant
search and storage resources. With this setup, you
have full control and visibility over these resources,
but you incur costs based on your usage.
• Required customer resources:
• AI hub
• AI project
• AI Services/AOAI
• Key Vault
• Storage account
• Azure AI Search
• How to use: Deploy the standard setup template
• Tool Implications:
• File search and code interpreter
• Uploaded files get stored in your Azure
Blob Storage account
• Vector stores get created using your Azure
AI Search resource
Start building agents with Copilot Studio…
Build, deploy, realize value and prototype faster for simple scenarios
Build agents for business
processes without setting up
& managing your own
infrastructure…
Copilot Studio Azure AI Foundry
...and scale with Azure AI
Bring your vectorized indices
from Azure AI Search
Access 1800+ models from
Azure AI model catalog
...or you can start with code first solutions
Control and customize with options to manage your own infrastructure
Copilot Studio
Build highly customized
apps, Agents, and APIs
using Azure AI Foundry...
...and integrate Copilot
Studio components with
Microsoft 365 Agent
SDK
including the ability to
customize and deploy
app UI to 15+ channels
Visual Studio/ GitHub
Learn more about the better-together with Copilot Studio and Azure AI Foundry
Better together:CopilotStudioandAzure AI
Deploy agents with
Azure AI Foundry
Orchestrate them together with
AutoGen and Semantic Kernel
Single-agent Multi-agent
1 2
State-of-the-art
research SDK
Production-ready
and stable SDK
Managed agent
micro-services
Ideation Production
Agent Sample #1
Web Search and Extraction
Web Search
Query
Decomposition
Retrieve Web
Sites
Search
Query
Site Analyzer
Extract HTML
Body
Screenshot
Analysis
Verifier
Memory Thoughts
Error Handling
Page Transition
Convert to
Markdown
Page 1
Page 2
Text
Text
Screenshot
Generator
Web Result 1
* Azure AI
Content
Understanding
Azure
AI
Search
Doc Search
* We can use the Azure AI Content Understanding service to extract images and videos, and transform them from unstructured to structured data
Azure AI
Agent
Service
Azure AI
Agent
Service
Azure AI
Agent
Service
Agent Sample #2
Call Center Agent Assist
Query Parsing
C u s t o m e r Q u e r y
My SmartHome
thermostat isn’t
connecting to Wi-Fi,
and I keep seeing an
error code E-22. How
can I fix this?
LLM
{ Wi-Fi }
{ Error: E-22 }
{ Thermostat X10 }
Planner Agent
Azure AI
Agent
Service
Video Agent Doc Agent Web Agent
1. Doc Agent → Search internal support
documentation for error code "E-22" and Wi-Fi
troubleshooting.
2. Video Agent → Locate and timestamp
troubleshooting video content related to Wi-Fi
connection and error code E-22.
3. Web Search Agent → Search external sources
for recent discussions, tips, or product-specific
issues regarding Wi-Fi and the error code.
D y n a m i c P l a n
Gathers internal
troubleshooting steps.
Summarizes video clips
related to Wi-Fi and
error code E-22.
Finds the latest external
resources related to the
issue
* Abstraction
Recent Online
Findings
from Web
Internal Steps
from Documents
Video Link
and Timestamps
from Videos
* We can use the Azure AI Content Understanding service to extract images and videos, and transform them from unstructured to structured data
Azure AI
Agent
Service
Azure AI
Agent
Service
Azure AI
Agent
Service
Agent Sample #3
AI Teammates powered by multi-agents
Marketing Director
Role: Social Media Manager
Qualification: Expert
Skills: Content creation
Coding...
AI Teammate
Marketing Director
AI Teammate
Publish to
Linked In
Create an AI Teammate
Collaborate with AI Teammate
Invite AI Teammate to
Teams meeting
Conversation AI Coding Agent Content Gen
Azure AI
Agent
Service
Azure AI
Agent
Service
Azure AI
Agent
Service
Create AI Teammate
Get started with Agent Service on
Microsoft Learn Docs
https://aka.ms/MicrosoftLearn_Agents
Get started with Azure AI Agents
through Azure AI Foundry SDK
Watch the Microsoft Ignite sessions on-
demand
 Building intelligent multi-agent systems
with Azure AI
Read the Techcommunity announcement
Thank you

Build enterprise-grade AI agents with Azure AI Agent Service

  • 1.
    Build enterprise- grade AIagents with Azure AI Agent Service SATO Naoki (Neo) Principal Software Engineer Microsoft Corporation
  • 2.
    Every app will bereinvented with AI New apps will be built that weren’t possible before
  • 3.
    Copilot & AIstack Copilot devices Copilot
  • 4.
    Visual Studio + Copilot Studio GitHub + Azure Cloud to Edge AzureAI Foundry Data Infrastructure Trustworthy AI Copilot & AI stack
  • 5.
  • 6.
    Agentic AI use AIto automate and execute business processes, nearly autonomously AI agents can plan, adapt to new information, and execute tasks independently, making them capable of handling complex, dynamic environments, decision-making and autonomous action. They can learn from feedback, adjust their strategies, and operate with minimal human oversight. Reason over a provided business process Retrieve context to complete the process Perform an action for the end-user
  • 7.
    K Inefficiency and High Operational Costs Organizations facesignificant challenges with complex, manual, and time-consuming workflows and minimal automation Lack of Visibility & Control Human Error Scalability Issues
  • 8.
    Promising even moreefficiency, value, and advantage VALUE Yesterday Today User Chatbot Able to answer questions “What is the dress code in the office?” User Single AI Agent Able to take action “Order a laptop for a new employee” Human Supervisor User HR Agent Able to collaboratively solve complex tasks “Onboard 5 employees by Monday” Human Supervisor IT Agent Training Agent User RPA Able to complete repetitive tasks “Inputs new hire information into HR System”
  • 9.
    Sends welcome email HR Manager ITManager Training Manager New Hire Manually sets up user accounts and provisions access Gets new hire forms Has questions Answers questions Completes paperwork Confirm computer preference Confirmed Initiates IT process Missing applications submits IT ticket. Contacts procureme nt to obtain license. Initiates training process Schedule conflict Reviews training schedule and reschedules Sends feedback survey Waits for more data before analyzing and optimizing the entire process. Completes the training still has questions TODAY Existing solutions can only automate very specific tasks that have clear inputs and outputs Initial Interaction Document Verification Access & Technology Provisioning Training Onboarding Feedback Loop Survey is completed RPA automates the verification of completion and accuracy with predefined rules. RPA automates assigning predefined training modules but cannot adjust based on user feedback
  • 10.
    • Provides apersonalized onboarding journey based on interaction IT Agent Training Agent New Hire TOMORROW With AI Agents, these steps can be automated for the first time, while keeping human in the loop. HR Agent • Assesses documents and learns from interactions • Analyzes role, experience, and learning preferences to recommend training • Gathers real-time feedback • Identifies patterns makes informed decisions Provides adaptive training • Sets up user accounts, adapts to troubleshoot unexpected issues, and learns from errors. Fills out forms and has no questions The new hire has questions during the training. Orchestrates the additional processes Human in the Loop, Supervisor and Approver Initial Interaction Document Verification Access & Technology Provisioning Training Onboarding Feedback Loop
  • 11.
  • 12.
    This cutting-edge shiftpromises to ignite a fresh wave of productivity and efficiency Intelligent Decision making Increased Productivity Enhanced Efficiency Cost Reduction Automation is undergoing a thrilling transformation with autonomous agents now being designed to proactively plan actions and perform tasks for users.
  • 13.
    AI Agent workflowscan enhance efficiency, accuracy, and customer satisfaction Key Use Cases Across Industries • Assists employees in booking business trips • Integrates with Tripadvisor, Outlook, and SharePoint • Books via Teams chat or email • Uses OCR to gather receipts • Automates expense report submission and tracking Travel Booking & Expense Management • Personalized onboarding assistant for new hires • Uses LLMs grounded in HR data from SharePoint • Provide relevant training materials • Schedule orientations and set up software accounts • Monitor task completion and ensure efficient onboarding Employee Onboarding • Diagnoses issues by referencing history and product manuals • Provides tailored solutions or escalates through automated workflows • Creates tickets and schedules follow-ups • Updates CRM records, enhancing future support Personalized Customer Support • Analytics data from data lake and data warehouse • Responds to user requests in natural language • Generates insights, visualization, and sends via Teams or email • Automates data handling for real-time, effortless decision-making Data Analytics and Reporting
  • 14.
  • 15.
    4 primary considerations K Providingagents with the right context Knowledge Evaluation Ensuring they complete their tasks correctly Actions Giving them the tools to complete their tasks Security Ensuring they only have access to what they should Successfully developing AI Agents requires
  • 16.
    So far, buildingagents from scratch been difficult to do Tool Integration Creating a cohesive system through complex integration of various tools and APIs that have different interfaces, data formats, and requirements. Interoperability Achieving interoperability between different tools and platforms to ensure that data can be shared and understood across different systems. Scalability Handling increased data volumes, more complex computations, and higher user loads without degrading performance. Real-time Processing Ensuring tools can handle real-time requirements without significant latency. Maintenance Making labor–intensive updates to integrated tools for compatibility with new versions and prevention of obsolescence and security vulnerabilities. Flexibility Modifying or customizing existing tools or developing new ones to meet unique requirements. Error Handling Ensuring errors are handled gracefully and continue functioning despite tool failures or unexpected inputs is critical for reliability. Security Implementing robust encryption, access controls, and compliance with privacy regulations to protect sensitive data.
  • 17.
    Organizations need platformsthat enable rapid development of performant, secure AI Agents Current Frameworks Security and data privacy risks What’s Needed Secure, responsible AI that protects sensitive information and behaves compliantly Lack of integrated tools, insecure data grounding, challenging orchestration Ineffective deployment of AI across websites, applications, and production environments Restrictive pre-defined models that are challenging to customize Flexible models that enable processing and integration of information from multiple modalities or types of data Connected complex workflow automation grounded by seamless connection to enterprise data Tools and APIs that seamlessly integrate across enterprise applications
  • 18.
    Visual Studio Copilot Studio GitHub AzureAI Foundry SDK Azure AI Foundry Model Catalog Open-source models Foundational models Task models Industry models Azure AI Content Safety Azure AI Search Azure AI Agent Service Azure OpenAI Service Observability Customization Evaluations Governance Monitoring Announcing
  • 19.
    Azure AI AgentService Public Preview Empower developers to securely build, deploy, and scale AI agents with ease Flexible Model Selection Extensive Data Connections Enterprise-grade Security Rapid Development and Automation AI.Azure.com
  • 20.
    The full package Built-inenterprise readiness BYO-file storage (coming soon) BYO-search index Azure AI Foundry SDK – Agent Service OBO Authorization Support Enhanced Observability Extensive Ecosystem of Tools Knowledge Microsoft Fabric* SharePoint* Grounding with Bing Search Azure AI Search Your own licensed data* Files (local or Azure Blob) File Search Code Interpreter Actions Azure Logic Apps* OpenAPI 3.0 Specified Tools Azure Functions Model Catalog Azure OpenAI Service (GPT-4o, GPT-4o mini) Models-as-a-Service Llama 3.1-405B-Instruct Mistral Large Cohere-Command-R-Plus *Indicates feature is coming soon
  • 21.
    Introducing Azure AI FoundrySDK – Agent Service Azure OpenAI Assistants API + + Extensive Ecosystem of Tools Model Catalog Actions Azure AI Search Microsoft Fabric (coming soon) SharePoint (coming soon) Grounding with Bing Search Azure Logic Apps (coming soon) OpenAPI 3.0 Specified Tools Azure Functions Your own licensed data (coming soon) Files (local or Azure Blob) File Search Code Interpreter Azure OpenAI Service (GPT-4o, GPT-4o mini) Models-as-a-Service Llama 3.1-405B-Instruct Mistral Large Cohere-Command-R-Plus Built-In Enterprise Readiness BYO-file storage (coming soon) BYO-search index BYO-thread storage BYO-virtual network (coming soon) OBO Authorization Support Enhanced Observability Knowledge
  • 22.
    How does youragent work? User Travel Booking Agent “Help me book a trip to New York for a client meeting? I need to fly out next Monday and return on Friday.” Knowledge Sources (search, files, databases, storage etc.) Models (Azure OpenAI Service, Models-as-a- Service) Actions (Pre-built or custom tools to automate processes) “I’ve booked your trip to New York as requested. Here are details:…”
  • 23.
    How does youragent work? Step 1: Create an Agent Step 2: Create a Thread Step 3: Run the Agent Step 5: Check the Run status Step 6: Display the Agent’s Response Agent Travel Planning Agent Instructions You are a travel booking and expense management assistant designed to help employees plan, book, and manage business travel. Run 2 Model Tools (optional) File Search Code Interpreter Function Calling Bing Search Microsoft SharePoint (coming soon) Microsoft Fabric (coming soon) Azure Logic Apps (coming soon) Azure Functions OpenAPI 3.0 specified tools User’s message I need to book a hotel in New York for 2 stays. Agent’s message Here are some suggestions: Run 1 1 Use Tripadvisor API to search the nearest hotel Create message 2 Your data (optional) User’s message What’s the daily meal allowance for the business trip? Agent’s message The daily allowance for your business trip is $75, as per company policy. Use Microsoft SharePoint to query the company travel policy Create message 2 1 Thread Travel Planning Azure AI Search Files (local or Azure Blob)
  • 24.
    Chat Completions APIvs Assistants API vs Agents Assistants API Chat Completions API • Lightweight and powerful • Inherently stateless • Assistants API features, plus • Build with a model of your choice (OpenAI, Llama, Mistral, Cohere…) • Real time web-grounding with Bing • Secure grounding on enterprise data in SharePoint and Fabric • Bring Your Own Licensed data (Tripadvisor) • Connect to 1400+ data sources and services with Azure Logic Apps • Long running, event driven actions with Azure Functions • Standardized OpenAPI 3.0 tools • Bring Your Own Storage • Bring Your Own Private Network • Bring Your Own AI Search Resource • Limitless scaling with PTUs • Open Telemetry based tracing vs Azure AI Agent Service • Build with OpenAI models • Stateful (inbuilt conversation state management) • Access persistent threads • Automatic management of the model’s context window • Access files in several formats • Utilized Microsoft-managed storage • File Search (API handles chunking, embeddings storage and creation, and implementing vector search) • Code Interpreter • Function Calling vs
  • 25.
    Customers are innovatingwith Azure AI Agent Service
  • 26.
  • 28.
  • 29.
    Rapid Development andAutomation Out-of-the-box connectors Connection with 3P APIs Custom code Code Interpreter Client-side functions
  • 30.
    Automation 1,400 built-in connectors through AzureLogic Apps (coming soon) • Create and run automated workflows with Azure Logic Apps with little to no code using the visual designer and selecting from prebuilt operations. • Simply define the business logic for your workflow to connect your agent to external systems, tools, and APIs. • Microsoft connectors include Microsoft products such as Azure App Service, Dynamics365 Customer Voice, Microsoft Teams, M365 Excel • Non-Microsoft connectors include enterprise services such as MongoDB, Dropbox, Jira, Gmail, Twilio, SAP, Stripe, ServiceNow and many more. Playground experience in Azure AI Foundry portal is coming soon in January
  • 31.
    Automation Build standardized custom toolswith OpenAPI 3.0 specifications • Connect your AI agent to an external API using an OpenAPI 3.0 specified tool, allowing for scalable interoperability with various applications • Build an organization-side tools library specified with the OpenAPI 3.0 standard • Enable your custom tools to authenticate access and connections with managed identities (Microsoft Entra ID) for added security, making it ideal for integrating with existing infrastructure or web services Screenshot Playground experience in Azure AI Foundry portal is coming soon in January
  • 32.
    Automation Implement stateless or statefulcode- based actions with Azure Functions • Enable your agent to perform external interactions with the world, such as calling APIs or asynchronously sending and waiting for events. • Azure Functions and Azure Durable Actions enable you to execute serverless code for synchronous, asynchronous, long-running, and event-driven actions such as approving invoices with human-in-the- loop, monitor an end-to-end product supply chain over long periods of time, and many more. Screenshot Playground experience in Azure AI Foundry portal is coming soon in January
  • 33.
    Automation Perform advanced data analysiswith Code Interpreter • Code Interpreter allows your agent to write and run Python code in a sandboxed execution environment, handling diverse data formats and generate files with data and visuals. • With Code Interpreter enabled, your agent can run code iteratively to solve more challenging code, math, and data analysis problems. • Unlike the Assistants API, you can integrate data in your storage to use with this tool. Screenshot Playground experience in Azure AI Foundry portal is coming soon in January
  • 34.
    Automation Extend Llama Stack agentswith cloud- based tools • Supports the agent protocol for developers that are using Llama Stack SDKs • Natively offer a scalable, cloud-hosted, enterprise grade tools • Wireline compatible with Llama Stack • Protocol adapter will be a standalone distributable, discoverable from the llama-stack OSS repo directly Screenshot Playground experience in Azure AI Foundry portal is coming soon in January
  • 35.
  • 36.
    Extensive Knowledge Connections Publicweb data Corporate data Structured data Unstructured data Licensed data
  • 37.
    Data Connections Real-time, webdata grounding with Bing Search • Create more reliable and trustworthy applications with real-time public information from Bing • Grounding with Bing Search allows your agents to integrate real-time public web data, ensuring their response is accurate and up to date. • By including supporting URLs and search query links, Grounding with Bing Search enhances trust and transparency, empowering the users to verify responses with the original sources. Playground experience in Azure AI Foundry portal is coming soon in January
  • 38.
    Data Connections Grounding with privatedata in Microsoft SharePoint (coming soon) • Grounding with SharePoint makes your SharePoint content more accessible to your end users. • Enterprise-grade security features, such as On-behalf-of (OBO) authentication for SharePoint, ensure secure and controlled access for end users. Playground experience in Azure AI Foundry portal is coming soon in January
  • 39.
    Data Connections Chat withyour structured data in Microsoft Fabric (coming soon) • Integrate your agents with Fabric AI Skill to unlock powerful data analysis capabilities. • Fabric AI Skills can transform enterprise data into conversational Q&A systems, allowing users to interact with data through chat and uncover data-driven and actionable insights effortlessly. • On-behalf-of (OBO) authentication simplifies access to enterprise data in Fabric while maintaining robust security, ensuring proper access control and enterprise-grade protection. Playground experience in Azure AI Foundry portal is coming soon in January
  • 40.
    Data Connections Incorporate private datain Azure AI Search, Blob, and your local files • Bring your existing Azure AI Search index or create a new one using the improved File Search tool. • This tool leverages a built-in ingestion pipeline to process files from your local system or Azure Blob Storage. • Maintain complete control over your data as your files remain in your own storage, and your Azure AI Search resource ingests them Playground experience in Azure AI Foundry portal is coming soon in January
  • 41.
    Data Connections Gain acompetitive advantage with licensed data (private preview) • Empower your agents to deliver nuanced, informed answers tailored to specific use cases • Integrate licensed data from specialized data providers, such as Tripadvisor, to enrich agent responses • Tripadvisor enhances the quality of your agent’s responses to travel related queries with high-quality, fresh data, such as travel guidance and reviews (coming soon) Playground experience in Azure AI Foundry portal is coming soon in January
  • 42.
  • 43.
    Flexible Model Selection AzureOpenAI Service models Models as a Service (Serverless API) Azure AI Services models Multi-modal support Fine-tuned models
  • 44.
    Flexible Model Selection LeverageModels as a Service (MaaS) • Support a variety of models from Azure AI Foundry model catalog to power your agent’s reasoning. The supported models include: • GPT-4o • GPT-4o mini • Llama 3.1 • Mistral Large • Cohere Command R+ • Leverage Model Inference API to easily swap models and compare performance to find the best model for your specific needs.
  • 45.
    Flexible Model Selection Usefine-tuned models for your agents • Customize AI models to address your unique business needs with fine-tuning, enabling you to build task-specific agents while optimizing token costs. • Through collaborations with partners like Scale AI, you can efficiently label training data and create fine-tuned models that integrate seamlessly with Azure AI Agent Service. • This process enhances the performance of AI agents, streamlining development and reducing time to production. Azure AI Agent Service supports fine-tuned gpt-35-turbo (0125) as of Jan 2025
  • 46.
    Flexible Model Selection Multi-modalsupport across text, image, and audio • Unlock new scenarios with multi-modal support, enabling AI agents to process and respond to diverse data formats beyond text, expanding the potential use cases. • Support for GPT-4o’s image and audio modalities so that you can analyze and combine data from various formats to deliver comprehensive insights, make decisions, and provide relevant outputs tailored to specific user needs Playground experience in Azure AI Foundry portal is coming soon in January
  • 47.
  • 48.
    Enterprise-grade Security Bring yourown storage Keyless setup and authentication Private Virtual Network support Tracing/ monitoring Content filters
  • 49.
    Security Bring your own storageand search • Bring your own storage account (coming soon) and Azure AI Search resource for custom data handling. • All files you upload will be stored in your storage account, ensuring you maintain complete control over your data. • All indexes get created using your connected Azure AI Search resource. Playground experience in Azure AI Foundry portal is coming soon in January
  • 50.
    Security Bring your own virtualnetwork (coming soon) • No public egress foundational infrastructure ensures the right authentication and security for your agents and tools, without you having to do trusted service bypass • Container injection allows the platform network to host APIs and inject a subnet into your network, enabling local communication of your Azure resources within the same virtual network (VNet). • If your resources are marked as private and non-discoverable from the Internet, the platform network can still access them, provided the necessary credentials and authorization are in place
  • 51.
    Security Monitor agent performance with OpenTelemetry-based training(coming soon) • Gain insights into your AI agent's performance and reliability • OpenTelemetry-compatible metrics for offline and online evaluation of agent outputs through the Azure AI Foundry SDK • Add local variables and intermediate results to trace decorator for detailed tracing capabilities for user defined functions • Options to prevent sensitive or large data logging as per OpenTelemetry standards
  • 52.
    Security Build responsibly with contentfilters and XPIA mitigation (coming soon) • Supports prebuilt and custom content filters that detect harmful content at varying severity levels. • Prompt shields protect agents against cross-prompt injection attacks from malicious actors. • As with Azure OpenAI Service, prompts and completions processed by the Azure AI Agent Service are not used to train, retrain, or improve Microsoft or 3rd party products or services without your permission. Customers can delete their stored data when they see fit.
  • 53.
    Basic agent setup •Overview: All agents created in this project use multi- tenant search and storage resources fully managed by Microsoft. You don't have visibility or control over these underlying Azure resources. • Required customer resources: • AI hub • AI project • AI Services/AOAI • How to use: Deploy the basic setup template • Tool Implications: • File search and code interpreter • Uploaded files get stored in Microsoft managed storage • Vector stores get created using a Microsoft managed search resource • Azure AI Search tool is not supported • Azure Blob Storage with file search is not supported
  • 54.
    Standard agent setup •Overview: Agents use customer-owned, single-tenant search and storage resources. With this setup, you have full control and visibility over these resources, but you incur costs based on your usage. • Required customer resources: • AI hub • AI project • AI Services/AOAI • Key Vault • Storage account • Azure AI Search • How to use: Deploy the standard setup template • Tool Implications: • File search and code interpreter • Uploaded files get stored in your Azure Blob Storage account • Vector stores get created using your Azure AI Search resource
  • 55.
    Start building agentswith Copilot Studio… Build, deploy, realize value and prototype faster for simple scenarios Build agents for business processes without setting up & managing your own infrastructure… Copilot Studio Azure AI Foundry ...and scale with Azure AI Bring your vectorized indices from Azure AI Search Access 1800+ models from Azure AI model catalog
  • 56.
    ...or you canstart with code first solutions Control and customize with options to manage your own infrastructure Copilot Studio Build highly customized apps, Agents, and APIs using Azure AI Foundry... ...and integrate Copilot Studio components with Microsoft 365 Agent SDK including the ability to customize and deploy app UI to 15+ channels Visual Studio/ GitHub Learn more about the better-together with Copilot Studio and Azure AI Foundry Better together:CopilotStudioandAzure AI
  • 57.
    Deploy agents with AzureAI Foundry Orchestrate them together with AutoGen and Semantic Kernel Single-agent Multi-agent 1 2 State-of-the-art research SDK Production-ready and stable SDK Managed agent micro-services Ideation Production
  • 58.
    Agent Sample #1 WebSearch and Extraction Web Search Query Decomposition Retrieve Web Sites Search Query Site Analyzer Extract HTML Body Screenshot Analysis Verifier Memory Thoughts Error Handling Page Transition Convert to Markdown Page 1 Page 2 Text Text Screenshot Generator Web Result 1 * Azure AI Content Understanding Azure AI Search Doc Search * We can use the Azure AI Content Understanding service to extract images and videos, and transform them from unstructured to structured data Azure AI Agent Service Azure AI Agent Service Azure AI Agent Service
  • 59.
    Agent Sample #2 CallCenter Agent Assist Query Parsing C u s t o m e r Q u e r y My SmartHome thermostat isn’t connecting to Wi-Fi, and I keep seeing an error code E-22. How can I fix this? LLM { Wi-Fi } { Error: E-22 } { Thermostat X10 } Planner Agent Azure AI Agent Service Video Agent Doc Agent Web Agent 1. Doc Agent → Search internal support documentation for error code "E-22" and Wi-Fi troubleshooting. 2. Video Agent → Locate and timestamp troubleshooting video content related to Wi-Fi connection and error code E-22. 3. Web Search Agent → Search external sources for recent discussions, tips, or product-specific issues regarding Wi-Fi and the error code. D y n a m i c P l a n Gathers internal troubleshooting steps. Summarizes video clips related to Wi-Fi and error code E-22. Finds the latest external resources related to the issue * Abstraction Recent Online Findings from Web Internal Steps from Documents Video Link and Timestamps from Videos * We can use the Azure AI Content Understanding service to extract images and videos, and transform them from unstructured to structured data Azure AI Agent Service Azure AI Agent Service Azure AI Agent Service
  • 60.
    Agent Sample #3 AITeammates powered by multi-agents Marketing Director Role: Social Media Manager Qualification: Expert Skills: Content creation Coding... AI Teammate Marketing Director AI Teammate Publish to Linked In Create an AI Teammate Collaborate with AI Teammate Invite AI Teammate to Teams meeting Conversation AI Coding Agent Content Gen Azure AI Agent Service Azure AI Agent Service Azure AI Agent Service Create AI Teammate
  • 61.
    Get started withAgent Service on Microsoft Learn Docs https://aka.ms/MicrosoftLearn_Agents Get started with Azure AI Agents through Azure AI Foundry SDK Watch the Microsoft Ignite sessions on- demand  Building intelligent multi-agent systems with Azure AI Read the Techcommunity announcement
  • 62.