Generative AI has been rapidly evolving, enabling different and more sophisticated interactions with Large Language Models (LLMs) like those available in IBM watsonx.ai or Meta Llama2. In this session, we will take a use case based approach to look at how we can leverage LLMs together with existing automation technologies like Workflow, Content Management, and Decisions to enable new solutions.

1. The professional home for the engineering and technology community worldwide
ieee.org
The professional home for the engineering and technology community worldwide
Approaches in using Generative AI in Automation Solutions
IEEE International Conference on Autonomic Computing and Self-Organizing Systems (ACSOS) 2023
Allen Chan, IBM Distinguished Engineer and CTO, IBM Business Automation
1

2. The professional home for the engineering and technology community worldwide
2 ieee.org
The professional home for the engineering and technology community worldwide
Allen Chan
2
Co-Author of 2 books on Automation and AI
Recent collaboration with University of Toronto (Prof. Eric Yu) and
York University (Prof. Alex Sendervich) on AI Research
1.Transformer Models for Activity Mining in Knowledge-Intensive Processes. Feb 2023. Book: Business Process
Management Workshops
2.Managing and Simplifying Cognitive Business Operations Using Process Architecture Models. May 2019.
Book: Advanced Information Systems Engineering
3.Solution Patterns for Machine Learning. May 2019. Book: Advanced Information System Engineering.
4.Modeling and Analyzing Process Architecture for Context-Driven Adaptation: Designing Cognitively-Enhanced
Business Processes for Enterprises. Oct 2018. 2018 IEEE 22nd International Enterprise Distributed Object
Computing Conference (EDOC)
5.Designing Process Architectures for User Engagement with Enterprise Cognitive Systems. Oct 2017. IFIP
Working Conference on The Practice of Enterprise Modeling

3. The professional home for the engineering and technology community worldwide
Agenda
3
▸Generative AI is not Enough
▸Content Use Cases
▸Decisions Use Cases
▸Workflow Use Cases
▸Summary
▸Q&A

4. The professional home for the engineering and technology community worldwide
1 Generate
▸ Generate content
based on request,
e.g., write an email,
draft a proposal
2 Classify
▸ Read and classify
input based on few
examples. Can be
used to categorize
customer issues or
group inputs.
3 Summarize
▸ Transform dense text
into your
personalized
overview, capturing
key points.
4 Extract
▸ Pull the information
you want from large
documents. Identify
named entities,
parse terms and
conditions, etc.
4
Generative AI Core Capabilities

5. The professional home for the engineering and technology community worldwide
GenerativeAI alone is not enough
5
Contrary to popular belief, Generative AI does not:
1. Think
2. Reason
3. Check for Facts
4. Coordinate Tasks
5. Perform Real Work
=> To do that, we need to combine Generative AI with Automations

6. The professional home for the engineering and technology community worldwide
Cost
Enterprise
challenges
with using
Gen AI
▸ Training on LLM is expensive and requires
hundreds or thousands of GPU hours. Simple
approaches can easily lead to token explosion
and exponential cost.
▸ LLM is designed to process “language” data, not
Excel, not PDF, not JSON, not Powerpoint, not
photos… so all information will need to be
“normalized” before it can be stored in LLM.
▸ Many documents in an enterprise content
repository have access control and contain
sensitive and private data. LLMs have no access
control or data governance.
Text Only
6
Access
Control

7. The professional home for the engineering and technology community worldwide
Sensitive
Data
Enterprise
challenges
with using
Gen AI
▸ Since LLMs do not forget, guardrails must be
provided to ensure sensitive and private data
are not stored intentionally or
unintentionally in the model.
▸ We want the model to answer questions based
on facts and not making it up (“hallucinating”).
In addition, we must be able to trace the
answers back to the original documents.
▸ Different LLMs have different performance and
quality. They use different algorithms, different
training data, different weighting on data, and
different pre- and post-processing. The quality of
the answers will rely on picking the correct LLM for
the problem domain.
Trust &
Traceability
7
Prediction
Quality

8. The professional home for the engineering and technology community worldwide
Use Cases
Content
▸ Provide answers to questions
with LLMs
Decisions
▸ Add reasoning and decisions to
LLMs
Workflow
▸ Automate work with LLMs
8

9. The professional home for the engineering and technology community worldwide
1
▸ Feed documents
directly into fine-
tuning
2
▸ Leverage ECM Built-
in Search
3
▸ Refine inputs with
knowledge pre-
processing
4
▸ Adopt a VectorDB
for rich semantic
search
9
Content –
Provide answers to
questions with LLMs

10. The professional home for the engineering and technology community worldwide
10
Pros
• The approach is straightforward and easy to
implement
• Allows users to get answers directly from LLM
Cons
• Takes a lot of time and money to fine-tune the model with
millions of documents
• Not all documents are textual and can be consumed as-is… so
any answer could be incomplete.
• No role-based access control means all data used in fine-
tuning will be available to anyone that is asking the question
• Mixing enterprise data with public-domain foundational data
can easily lead to hallucination
Content – Feed documents
directly into fine-tuning
1. Take documents in Document Store use them directly in Fine-
Tuning.
2. Build a Q&A system to just the question through the prompt.

11. The professional home for the engineering and technology community worldwide
11
Pros
• The approach is straightforward and easy to
implement
• Get answers directly from the information source –
chances of hallucination is greatly limited.
• The token count is limited only to information
passed to the model.
• Role-based access control can be applied
Cons
• We can only find documents with matching search
keywords
• Additional processing would be required to limit the
number of matching results as there might be too many
tokens.
• Given the result is the entire document, it would be
difficult to extract the precise relevant sections.
Content – Leverage Document
Store Built-in Search
1. Perform a search first and retrieve the documents
matching the search criteria.
2. Feed the document as part of the prompt and ask
the LLM to limit the answers to only information in
the prompt.

12. The professional home for the engineering and technology community worldwide
12
Pros
• In addition to having the same pros as the previous
approach, the additional knowledge pre-processor
allows more document types to be passed to the
LLMs
• By removing sensitive data ahead of the prompt, we
reduce the risk of sensitive data being stored in the
LLMs.
Cons
• We can only find documents with matching search
keywords
• Additional processing would be required to limit the
number of matching results as there might be too many
tokens.
• Given the result is the entire document, it would be
difficult to extract the precise relevant sections.
Content – Leverage
Knowledge Pre-processor
1. Leverage existing FileNet Search
2. Inject a Knowledge Pre-processor before we
generate the prompt to perform OCR and other
information filtering to remove sensitive and
personal data.

13. The professional home for the engineering and technology community worldwide
13
Pros
• Use of VectorDB allows use of document fragments
in search results, thus reducing the token counts
and increases the result relevance.
• Knowledge aggregator/formatter can combine
information from multiple data sources to create a
single answer.
Cons
• Having to pre-process all the documents is a time-
consuming process.
Content – Adopt Retrieval
Augmentation Generation (RAG)
approach for semantic search
1. Leverage a VectorDB to shard, index and perform
semantic search on documents
2. Include a knowledge aggregator/formatter to combine
results from LLMs and original data to created complete
answers.

14. The professional home for the engineering and technology community worldwide
1
▸ NLU + Reasoning
2
▸ Reasoning followed
by NLG
3
▸ Generate business
rules from business
policies
14
Decisions –
Add reasoning and
decisions to LLMs

15. The professional home for the engineering and technology community worldwide
15
Pros
• Straightforward implementation requiring minimal
or no fine tuning.
• Allow rules to be used in non-traditional API-based
implementation e.g., in a chatbot.
Cons
• Limited to identified text extraction capabilities.
• In cases where the extraction process does not find the
expected data from the input text, it’s crucial to set
appropriate guardrails to handle such scenarios during
the reasoning phase.
Decisions – NLU + Reasoning
1. Leverage NLU capabilities of LLMs to extract
business entities (e.g., city name, job role) and
pass those as inputs to Decisions.

16. The professional home for the engineering and technology community worldwide
16
Pros
• Straightforward implementation requiring minimal
or no fine tuning.
• Allow rules to be used in non-traditional API-based
implementation e.g., in a chatbot.
Cons
• Prompt will need to be prepared carefully to ensure
generated outputs reflect the intent of the decision
outcomes.
Decisions – Reasoning
followed by NLG
1. Leverage NLU capabilities of LLMs to extract
business entities (e.g., city name, job role) and
pass those as inputs to Decisions.

17. The professional home for the engineering and technology community worldwide
17
Pros
• Leveraging the LLM as a knowledge extraction tool
enables the extraction of rules and an underlying
data ontology. The extracted rules, once reviewed
by a human, can automate decisions with
traceability and determinism.
Cons
• Depending on the business domains, further fine-
turning will be required.
• Require a good amount of prompt engineering to
ensure the generated rules model is syntactically and
semantically correct.
• Generate models will require require by SME and still
need to be tested/validated.
Decisions – Generate business
rules from business policies
1. Extract automation assets, including business rules, data
models, and signatures, from plain text business policies.
2. Use the generated rules models as starting point to create
business decisions.

18. The professional home for the engineering and technology community worldwide
1
▸ Classify inputs and
perform requests
2
▸ Generate responses
as part of a business
processes
3
▸ Automate LLM
finetuning and
exception handling
4
▸ Generate workflow
from description
18
Workflow –
Automate work with LLMs

19. The professional home for the engineering and technology community worldwide
19
Pros
• Straightforward implementation requiring minimal
or no fine-tuning.
• Allow workflow to work directly with natural
language inputs without pre-processing
Cons
• Limited to identified text extraction capabilities.
• In cases where the extraction process does not find the
expected data from the input text, it’s crucial to set
appropriate guardrails to handle such scenarios during
the reasoning phase.
Workflow – Classify/Extract
inputs and perform work
Leverage LLM to classify and/or extract input data and use
those information to perform work
If LLM fails to classify the information, we can delegate to a
human user to decide and feed the decision back to the
system.

20. The professional home for the engineering and technology community worldwide
20
Pros
• Support a more intuitive engagement experience
with users.
• Support other non-API based interaction, e.g., with
smart assistants like Alexa, voice-based interaction
with text-to-speech
Cons
• Prompt will need to be prepared carefully to ensure
generated outputs reflect the intent of the work
outcomes.
Workflow – Generate responses
as part of a business processes
Use LLM to generate responses (e.g., email
messages, work summary, task notification) directly
from business processes data.

21. The professional home for the engineering and technology community worldwide
21
Pros
• This is a must-have for any fine-tuning or indexing
exercise that uses enterprise data.
Cons
• Incur additional review and approval cost.
Workflow – Automate LLM finetuning
and exception handling
Use workflow to provide Human-in-the-loop review
and approval before knowledge data will be fed to
LLM for fine tuning.

22. The professional home for the engineering and technology community worldwide
22
Pros
▸ Provide a natural language approach to generate
business processes.
▸ The quality of the generated process will rely on the
quality of the LLMs.
▸ It is also possible to ask the LLM to generate the
process from just a simple business goal, but the result
will rely on the foundational data that was used to do
the initial model training.
Cons
▸ Do not expect to get a fully working BPMN model, work such as
defining the data types, decision logic, user interfaces and
validating the process will be required.
Workflow – Generate Workflow
from Description
From a high level description of a business process, generate a
BPMN process that can be used as a starting point for an
automation workflow solution.
Due to different generation quality of the LLM models, it is better
to use LLM to summarize or extract procedures from the business
process document and use a post-processor to take the output
and turn that into a formal BPMN document.

23. The professional home for the engineering and technology community worldwide
1
▸ Gen AI, when used
appropriately, can
speed up certain
data processing.
2
▸ Combine Gen AI
and Automations
can accelerate your
digital
transformation
journey
3
▸ Careful planning
and consideration
for private and
sensitive data is a
must for enterprise
usage.
4
▸ LLM models are evolving
very fast and each day
there are new models.
Understanding which
models to use for the
purpose at hand is the
key to yield good results.
23
Summary