1. Loading your life into a
Vector Database

2. 2
2
@bnchrch
Ben Church
Who am I?
● Live 3 flights away on the northern part of
Vancouver Island
● Engineer at Airbyte
● Like to Tinker
● Love to Teach
● A masochist to hit the potholes first so you
don’t have to

3. 3
3

4. Stubbing your toe on
Token Limits

5. RAG Against the Machine
(Retrieval Augmented Generation)

6. Stealing my friends Strava
Data so I don’t have to
exercise as much as I told my
Wife I would.

7. 7
7
New Systems,
New Possibilities,
New Boundaries,
New Constraints,
New Considerations.

8. 8
8
Agenda
● Constraints of a modern LLM based
system
● Considerations that result and how to
work with this new technology
● The Second Order Effects and the
Future
● Through the lens of a simple
application
● Vectors and Vector Databases
● (simple) LLM Application Architecture
● Context Stuffing
● Token Limits
● GraphQL Types
● GraphQL Introspection

9. 9
9

10. 10
10

11. 11
11
Step 0: Share Context

12. 12
12
What does it mean to be a “simple” LLM application?

13. 13
13
��
Credit: Nicole Choi and Github https://github.blog/2023-10-30-the-architecture-of-todays-llm-applications/

14. 14
14
Retrieval Augmented Generation
source: https://www.ml6.eu/blogpost/leveraging-llms-on-your-domain-specific-knowledge-base

15. 15
15
Retrieval Augmented Generation
(with GQL)

16. 16
16
Constraint: Unstructured to Structured Data

17. 17
17
Vectors
● Embeddings == Vectors used in ML/AI
● An Array of Numbers (Weights)
● Representation of similarity
● Used for similarity searches
● Not Unique to LLMs or AI
● Vector DBs make use of Vector Index’s
Credit https://stackoverflow.blog/2023/10/09/from-prototype-to-production-vector-databases-in-generative-ai-applications/

18. 18
18
Vector Database
Credit https://dejanualex.medium.com/llm-and-vector-databases-d2530f03f6be
● Vector DBs make use of Vector Index’s
● Optimized for similarity searches
● Often built with the LLM use case in mind
● Specifically in Context Retrieval System

19. 19
19
Consider: Building on top of a Vector DB

20. 20
20
Consider: A Unified Datastore and API

21. 21
21
Step 1: Load your Data

22. 22
22
Calling All Sources

23. 23
23
Setting up Weaviate Cloud

24. 24
24
Setting up a Destination

25. 25
25
Setting up a Source

26. 26
26
Sync it!

27. 27
27
Thank you Melissa! 󰣘

28. 28
28
Step 2: Query for Context

29. 29
29

30. 30
30
Weaviate? GraphQL? Why?

31. 31
31
Introspection, Descriptions, Types

32. 32
32
Complex Filters and Aggregation

33. 34
34
It Solves Two Major Constraints

34. 36
36
Constraint: Token Limits
● There is a Limit on the amount of Context
you can Stuff into a Query
● That limit is set by the model
● It can be Low
● It can be High
○ Latest Chat GPT
○ Anthropic
● Tension between Token Limits and
meaningful Vectors
○ Makes Time Series Questions Difficult

35. 37
37
Constraint: Too Much Context
● As Context Size increases
● Hallucinations increase
● Answer Quality decreases
● The balancing act of relevancy

36. 38
38
Consider: A Flexible, Accessible, Schema & Query Language

37. 39
39
Step 2: Build

38. 41
41
Not that complicated…
1. Get a question
2. Ask the LLM to Transform it to a Vector
3. Get the GraphQL Schema
4. Ask the LLM to generate a Retrieval Query for our
Contextual Data
5. Use the Retrieval Query to get our Contextual
Data
6. Ask the LLM to generate the final answer

39. 42
42
Question

40. 43
43
Vector

41. 44
44
Introspection

42. 45
45
Example Record

43. 46
46
Retrieval Query

44. 47
47
Answer

45. 48
48
Other Interesting Queries

46. 49
49
Constraint: Bad at Math

47. 50
50
Constraint: Non-Deterministic

48. 52
52
For the next project

49. 53
53
For the next project
● LLMs are
○ Non-deterministic
○ Bad at Math
○ Limited by their tokens
● Context and meaningful vectors are at odds with your
models Token limit
● You need a strategy for pulling in the most amount of
context in the least amount of text
● Your Data Models, API’s and Error Messages matter
more in this paradigm

50. 54
54
Thoughts for the Future

51. Will Tom Cruise
Live Forever?

52. 58
58
Thank You