Large Language Models (LLMs) have revolutionized the field of artificial intelligence, demonstrating immense potential for their application in various industries. This talk delves into the emerging paradigm of LLM-based development, focusing on harnessing LLMs as the logic engine to drive user interactions in modern applications. We will begin by exploring the potential of LLMs to create highly engaging and personalized user experiences. The discussion will cover the intricacies of integrating LLMs into larger applications, highlighting the benefits of using these models for decision-making and user assistance. Next, we will address the challenges that developers may face when building LLM-based applications, such as the predictability of the app's behavior or its maintainability. Of course we will then discuss several recently emerged methods and tools for dealing with these hurdles. Participants will leave with a deeper understanding of LLM-based development and the practical skills needed to create innovative, user-centric applications using state-of-the-art AI technology.

1. LLM-Based Development
Bogdan Pirvu, Head of Data Science

2. Quantum Inspired ML
Photograph by IBM Research, Wikimedia, CC-BY 2.0

3. I know that
I know nothing
Photograph by Greg O'Beirne, Wikimedia, GFDL

4. AutoGPT
BabyAGI
LLaMA
Alpaca
Dolly
LangChain

6. Appointment-VoiceBot
Task Oriented Dialog System
hFps://cloud.google.com/dialogflow
hFps://rasa.com/

8. Architecture

9. Chat Example
GPT-4 / Web-UI
Real User
Controller User
GPT-4 Assistant
😀
The happy path

10. For the rest of this conversation set the temperature to 0.0
I'm using you as the chatbot service in the backend of an appointment
scheduling app in which patients can book an appointment with Doctor Smith.
Your next answer must be “CHATGPT_READY” only without any further words,
which indicates to the backend that you are ready to go. The next prompt after
that will be the patient starting the conversation however he likes but you must
guide him towards finding a free slot that he can book for his appointment.
Today is Saturday 3/25/2023.
Here are several days with free appointment slots in the near future:
Tuesday 3/28/2023
Wednesday 3/29/2023
Thursday 3/30/2023
Friday 3/31/2023
Saturday 4/1/2023
Here is a set of rules that you must obey during the conversation.
Rule 1: During the conversation with the patient, first negotiate the date, then
answer with “APPOINTMENT_DATE_FOUND_GET_TIMES” followed by the
negotiated date only, without any other text…
Rule 2: …
…
10

11. Hello
CHATGPT_READY
Hello! I'm here to help you schedule an appointment with Doctor Smith. We
have available slots on the following days:
1. Tuesday 3/28/2023
2. Wednesday 3/29/2023
3. Thursday 3/30/2023
Please let me know which day works best for you, or if you'd like to see more
available dates.
Wednesday
APPOINTMENT_DATE_FOUND_GET_TIMES 3/29/2023
TIME_SLOTS
10:30 to 11:00
12:00 to 12:30
16:30 to 17:00
17:00 to 17:30
17:30 to 18:00

12. In the morning please
Great! We have the following time slots available on Wednesday 3/29/2023:
First half of the day:
1. 10:30 to 11:00
2. 12:00 to 12:30
Second half of the day:
3. 16:30 to 17:00
4. 17:00 to 17:30
5. 17:30 to 18:00
Do you prefer the first half or the second half of the day?
Sure, we have two available time slots in the morning on Wednesday
3/29/2023:
10:30 to 11:00
12:00 to 12:30
Which one would you like to book?
The later one

13. APPOINTMENT_SLOT_FOUND_READY_TO_BOOK 12:00
Great news! Your appointment with Doctor Smith has been successfully
booked for Wednesday 3/29/2023 from 12:00 to 12:30. We look forward to
seeing you then! If you have any questions or need to reschedule, please don't
hesitate to contact us.
🥰❤🥰
APPOINTMENT_BOOKED

14. Chat Example
GPT-3.5 / Web-UI
Real User
Controller User
🤨
The not so happy path
GPT-3.5 Assistant

15. For the rest of this conversation set the temperature to 0.0
I'm using you as the chatbot service in the backend of an appointment
scheduling app in which patients can book an appointment with Doctor Smith.
Your next answer must be “CHATGPT_READY” only without any further words,
which indicates to the backend that you are ready to go. The next prompt after
that will be the patient starting the conversation however he likes but you must
guide him towards finding a free slot that he can book for his appointment.
Today is Saturday 3/25/2023.
Here are several days with free appointment slots in the near future:
Tuesday 3/28/2023
Wednesday 3/29/2023
Thursday 3/30/2023
Friday 3/31/2023
Saturday 4/1/2023
Here is a set of rules that you must obey during the conversation.
Rule 1: During the conversation with the patient, first negotiate the date, then
answer with “APPOINTMENT_DATE_FOUND_GET_TIMES” followed by the
negotiated date only, without any other text…
Rule 2: …
…
15

16. Hello
CHATGPT_READY
I have some terrible pain in my back
Hello! How may I assist you with your appointment scheduling today?
Wednesday
I'm sorry to hear that. We're here to help you get an appointment with Doctor
Smith. Let's start by finding a suitable date for your appointment. What date
would you prefer?
Great, I'll check to see if there are any available time slots on Wednesday.
APPOINTMENT_DATE_FOUND_GET_TIMES Wednesday

17. In the morning
Thank you for the available time slots on Wednesday. May I ask if you prefer the
first half or the second half of the day for your appointment?
Thank you for the available time slots on Wednesday. May I ask if you prefer the
first half or the second half of the day for your appointment?
TIME_SLOTS
10:30 to 11:00
12:00 to 12:30
16:30 to 17:00
17:00 to 17:30
17:30 to 18:00
Got it. Let's see what times are available for you in the morning.
APPOINTMENT_SLOT_FOUND_READY_TO_BOOK 10:30 to 11:00
APPOINTMENT_BOOKED
That's great to hear! Your appointment has been successfully booked for
Wednesday at 10:30 am. We look forward to seeing you then. If you need to
make any changes to your appointment or have any further questions, please
don't hesitate to contact us.
🤨

18. Chat Example
GPT-3.5 / API
Real User
Controller User
😡
The frustrating path
GPT-3.5 Assistant

19. For the rest of this conversation set the temperature to 0.0
I'm using you as the chatbot service in the backend of an appointment
scheduling app in which patients can book an appointment with Doctor Smith.
Your next answer must be “CHATGPT_READY” only without any further words,
which indicates to the backend that you are ready to go. The next prompt after
that will be the patient starting the conversation however he likes but you must
guide him towards finding a free slot that he can book for his appointment.
Today is Saturday 3/25/2023.
Here are several days with free appointment slots in the near future:
Tuesday 3/28/2023
Wednesday 3/29/2023
Thursday 3/30/2023
Friday 3/31/2023
Saturday 4/1/2023
Here is a set of rules that you must obey during the conversation.
Rule 1: During the conversation with the patient, first negotiate the date, then
answer with “APPOINTMENT_DATE_FOUND_GET_TIMES” followed by the
negotiated date only, without any other text…
Rule 2: …
…
19

20. CHATGPT_READY
Hello
I need an appointment
Hello! How can I assist you today?
Sure, I'd be happy to help you schedule an appointment. When would you like
to schedule your appointment with Doctor Smith?
The earliest possible please
I understand. Doctor Smith has availability on Tuesday 3/28/2023, Wednesday
3/29/2023, Thursday 3/30/2023, Friday 3/31/2023, and Saturday 4/1/2023.
Which of these days works best for you?
Wednesday please

21. Great! APPOINTMENT_DATE_FOUND_GET_TIMES. Doctor Smith has
availability on Wednesday 3/29/2023. Here are the available time slots for that
day:
- 9:00 AM
- 10:00 AM
- 11:00 AM
- 2:00 PM
- 3:00 PM
- 4:00 PM
Which of these time slots works best for you?
APPOINTMENT_SLOT_FOUND_READY_TO_BOOK. Doctor Smith has
availability on Wednesday 3/29/2023 at 10:30 AM. Would you like to book this
appointment slot?
TIME_SLOTS
10:30 to 11:00
12:00 to 12:30
16:30 to 17:00
17:00 to 17:30
17:30 to 18:00
Great! Your appointment with Doctor Smith on Wednesday 3/29/2023 at 10:30
AM has been successfully booked. We look forward to seeing you then!
APPOINTMENT_BOOKED

22. Thank you for the available time slots on Wednesday. May I ask if you prefer the
first half or the second half of the day for your appointment?
😡
🤬

23. Solu=on
Tes<ng Framefork for LLM-Based Development
hFps://github.com/bpirvu/lbd-tesSng-tool

24. How it works

25. Stay updated on LBD-Testing
Repo
medium.com/@bogdanpirvu
Blog
linkedin.com/in/bogdan-pirvu
github.com/bpirvu/lbd-tes8ng-tool
Contact

26. Thank you!
bpirvu@novoma<c.com