Micropropagation of Madagascar periwinkle (Catharanthus roseus)
Let's Chat to Find the APIs: Connecting Human, LLM and Knowledge Graph through AI Chain——KPL.pptx
1. Let's Chat to Find the APIs: Connecting Human, LLM
and Knowledge Graph through AI Chain
Qing Huang Zhenyu Wan Zhenchang Xing Changjing Wang Xiwei Xu Qinghua Lu
Jieshan Chen
3. Significance
2) If there is no optimal clarification
path, more clarification rounds are
needed to clarify user intent.
1) If the clarification knowledge base is
limited, it is difficult to accurately clarify
user intent.
4. Existing Work and Challenges
Existing Work: To recommend APIs that match human intent, existing
work[1][2] recommends APIs by means of question clarification based on
limited knowledge bases.
Existing Challenges: Out of vocabulary (OOV)
[1] Eberhart, Zachary, and Collin McMillan. "Generating clarifying questions for query refinement in source code search." 2022 IEEE International Conference
on Software Analysis, Evolution and Reengineering (SANER). IEEE, 2022.
[2] Huang, Qing, et al. "Answering Uncertain, Under-Specified API Queries Assisted by Knowledge-Aware Human-AI Dialogue." arXiv preprint
arXiv:2304.14163 (2023).
Inherent Challenges: LLM output randomly
Solution: Large Language Model (LLM)
8. Evaluation
RQ1: What is the effectiveness of each AI unit?
RQ2: How well does KPL perform in extending query statements across two scenarios, namely, query
statement covered in API KG and those not covered in API KG?
RQ3: How effective is the knowledge-guided pathfinding strategy employed in our approach?
9. Evaluation
Dataset
1) KG Query Statement (KGQS): For fairness, this dataset contains 60 query statements randomly selected from
KAHAID's API KG, each paired with its corresponding ground-truth API[1]. We use 6,000 API descriptions from
the API KG to obtain an example of the best path for guiding the LLM. To satisfy the scenario of “query
statement covered in KG”, we randomly select 60 API descriptions from the 6,000 API descriptions as query
statements and use their related APIs in the API KG as the ground-truth APIs.
2) non-KG Query Statement (non-KGQS): Also for the sake of fairness, this dataset is the evaluation dataset of
KAHAID, a state-of-the-art approach, which contains 60 queries each paired with multiple ground-truth APIs.
We reuse this high quality dataset of KAHAID, which are manually collected from the Stack Overflow using
strict criteria, which could be found in KAHAID's paper. It's worth noting that none of the query statements in
this dataset are covered in the API KG.
[1] Huang, Qing, et al. "Answering Uncertain, Under-Specified API Queries Assisted by Knowledge-Aware Human-AI Dialogue." arXiv
preprint arXiv:2304.14163 (2023).
10. Evaluation
RQ1: What is the effectiveness of each AI unit?
[3] Blackwell, Alan F., and Thomas RG Green. "A Cognitive Dimensions questionnaire optimised for users." PPIG. Vol. 13. 2000.
TEST CASE
questionnaires
Details in the questionnaire refer to Blackwell et
al[3].
11. Evaluation
RQ2: How well does KPL perform in extending query statements across two scenarios, namely, query
statement covered in API KG and those not covered in API KG?
12. Evaluation
RQ3: How effective is the knowledge-guided pathfinding strategy employed in our approach?
KPLw/oKPS: method with KG but without extracting the optimal path strategy.
KPLw/oK: method without KG guidance.
13. Conclusion
Significance
Issue
Existing work
and challenges
Our Approach
We propose a novel knowledge-guided query clarification approach for API recommendation that leverages KG-
guided LLM to address OOV failures and improve the fluency of conversations.
We leverage the structured API knowledge and entity relationships in KG to filter out noise and transfer the optimal
clarification path from KG to the LLM, increasing the efficiency of the clarification process.
Our approach is broken down into an AI-chain that consists of five steps, each handled by a separate LLM call,
resulting in improved accuracy, efficiency, and fluency for query clarification in API recommendation.
Our approach bridges the gap between KG and LLM, combining the strengths and compensating for the
weaknesses of both.
14. Let's Chat to Find the APIs: Connecting Human, LLM
and Knowledge Graph through AI Chain
Thank you!
wanzy@jxnu.edu.cn
Editor's Notes
Hi everyone.
My name is Zhenyu Wan, a graduate student from the Jiangxi Normal University in China. Today, the paper I am going to present is called "Let's Chat to Find the APIs: Connecting Human, LLM and Knowledge Graph through AI Chain". OK, First, I will introduce the ISSUE in the paper.
API recommendation tools recommend APIs by obtaining user query, which can help developers quickly find APIs that meet their query.
However, these API recommendation tools cannot accurately recommend due to short user query, so tools to extend user query have been developed.
There are two ways to extend tools: direct extension and clarification extension.
Direct extension method (b) cannot accurately extend user query and does not understand user intent.
Clarification extension methods (c) and (d), while capable of capturing user intent during interactions with the user, may have knowledge base limitations or no optimal clarification paths.
When one gets user intent using clarification extension method,
If the clarification knowledge base is limited, it is difficult to accurately clarify user intent.
If there is no optimal clarification path, more clarification rounds are needed to clarify user intent.
In order to accurately recommend APIs, existing works [1][2] ask clarification questions to obtain user intent.
These methods clarify user query based on a limited knowledge base, which can lead to OOV issues.
On the other hand, in order to solve the OOV problem, we can use LLM to ask clarification questions, but LLM may ask random questions about user query.
Therefore, we use the optimal clarification paths in external API knowledge base to guide LLM in asking clarification questions.
Our approach is based on large language models, such as gpt-3.5, which have rich API knowledge/can solve OOV problems.
We also utilize KG as an external API knowledge base to guide LLM to ask clarification questions with optimal clarification paths.
The whole method revolves around the idea of ai-chain and builds a complete set of problem clarification chains.
The red part in the figure is the clarification question and option generation part. The user gives query, and KG matches the optimal path according to the query, and guides LLM to raise clarification questions as examples.
LLM then gives possible options through query and problems for users to choose.
As shown in the black part, the user makes a choice and gives an answer.
In the blue part of the figure, LLM combines the query and clarification records to extend the query in a targeted manner, and finally recommends APIs that meet the user intent based on the extended query.
This table shows the best clarification path in the form of domain knowledge we obtain from API KG.
For example, the best questioning aspect in round 1 is "event", ask the question accordingly, and answer it. At this time, the best clarification aspect in the next round is "purpose".
Round2 asks questions and answers with the aspect of "purpose". At this time, the best clarification aspect in the next round is "status".
Round3 asks questions with the "status" aspect, gives answers, and finally recommends APIs.
The above is an example of an optimal path to clarification.
Here are the prompts used by 5 AI units in our method.
(a) The best question aspect generation unit, which gives the task query and the best question aspect examples matched from KG, so that LLM can generate the best question aspect according to user query.
(b) The clarification question generation unit provides background settings and limits the question aspects, allowing LLM to raise clarification questions based on known information.
(c) The alternatice option generation unit provides possible answer options based on user needs and clarification questions raised.
(d) query extension unit, which extends query in a targeted manner based on query and clarification records.
(e) API recommendation unit, which recommends APIs according to the extended query.
We set up three RQs to demonstrate the effectiveness of our approach.
RQ1: What is the effectiveness of each AI unit?
RQ2: How well does KPL perform in extending query statements across two scenarios, namely, query statement covered in API KG and those not covered in API KG?
RQ3: How effective is the knowledge-guided pathfinding strategy employed in our approach?
In the experiment, our data comes from the Stack Overflow data set of Huang et al. 60 in their dataset were processed at a more fine-grained level.
In addition, in order to exclude the influence of KG, we followed the method of Huang et al. and collected 60 pieces of data from Stack Overflow that were not in the API KG.
In this experiment, we demonstrated the effectiveness of each AI unit we set up through user experience.
We invited three master students with more than three years of programming experience who were not involved in our project to experience our approach and fill out a questionnaire to score each AI unit.
In the five-point scoring system, each AI unit received a high score of about 4.5, and the kappa coefficient was 0.6213, ensuring the consistency of the scoring.
This shows that the 5 AI units set by our method are reasonable and effective.
In this experiment we not only test the effectiveness of our method in KG data, but also the effectiveness of our method for query that are not in KG.
We compared the two methods ZaCQ and KAHAID from the four metrics of MRR, MAP, Precision and Recall respectively.
Overall, our method performs better than the other two methods at every round of clarification, whether using data from KG or not.
Our method shows good accuracy in both data, which proves that the knowledge in KG does not directly affect LLM decision-making, but guides LLM with "experience".
In this experiment, we compare our method with method without KG guidance and method with KG but without extracting the optimal path strategy.
the results demonstrate the effectiveness of KG guidance and optimal path extraction strategy.
Among the four evaluation metrics, our method performs 15%-22% better than the other two methods in each round of clarification.
In addition, methods without KG are sometimes better than methods with KG but without extracting the optimal path strategy. This is because the optimal path is not extracted, causing knowledge to incorrectly guide LLM.
In general, we propose a new knowledge-guided clarification method for API recommendation using KG to guide LLM, which not only solves the oov problem but also prevents LLM from asking random questions.
We leverage the structured API knowledge and entity relations in KG to filter noise and transfer the optimal clarification path from KG to LLM, thus improving the efficiency of the clarification process.
Our method is decomposed into an AI chain consisting of five steps, each of which is handled by a separate LLM call, thereby improving the accuracy, efficiency, and fluency of query clarification in API recommendation.
Our approach bridges the gap between KG and LLM, combining the strengths of both and compensating for the weaknesses of both.
