1 Table = 1000 Words? Foundation Models for Tabular Data

1 | © Copyright 2024 Zilliz
1
1 Table = 1000 Words?
Foundation Models for Tabular Data
Stefan Webb, Developer Advocate @ Zilliz

2
2 | © Copyright 9/25/23 Zilliz
Speaker
Stefan Webb
Developer Advocate
stefan.webb@zilliz.com
stefan-webb

3
Milvus is an Open-Source Vector Database to
store, index, manage, and use the massive
number of embedding vectors generated by
deep neural networks and LLMs.
contributors
400
stars
30K
docker pulls
66M
forks
2.7K
+
Milvus: High-performance, scalable vector database

4
Milvus Users

5
Deployment Options
Milvus Lite
● Locally hosted
● Suitable for prototyping
and demos
Milvus Standalone
● Single remote/local server
● “Medium” scale
● Simplified setup,
maintenance, etc.
compared to cluster
Milvus Cluster
● Distributed system
● Many different types of
nodes
● Scales to 100s of billions
of vectors

6

7
Why Not Traditional Databases?
Suboptimal
Indexing / Search
Scaling Inadequate Query
& Analytics Support

8
BM25
● Simple implementation
● Good Retrieval performance
○ Queries Per Second
○ Accuracy
● Used by others in the industry
○ ElasticSearch
○ Apache Lucene
○ MongoDB Atlas Search
○ …

9
Full Text Search

10
Benchmarks
Shows 3-20x faster comparing with open
source Milvus
At least 6x faster than other vector databases
https://github.com/zilliztech/VectorDBBench

11
Structured vs Unstructured Data?
“Structuredˮ “Unstructuredˮ
Text, Image, Audio, Video, Geo, 3D, Graphs, Networks, Molecules

12
Example Application
Table
Query
Response
Gen AI
“Black Magicˮ

13
Other Applications
● Summarization
● Table understanding
● Fact verification
● Table-to-text generation
● Text-to-table generation
● Etc.

14
CONTENTS
01 Foundation Models for Tabular Data
02 Training the Model
Data, Data, Data
03
Agents
04
Code Demo
05

15
1. Foundation Models for
Tabular Data

16
Modeling Paradigm
Table
Encoder
Translator
/ Adaptor
Combined
Prompt
Tokenizer
text
table
LLM response
(fine-tuned)
Qwen2.5
“QFormerˮ
Specialized
Transformer

17
TableGPT2
Su et al., 2024 Zhejiang University)]

18
Table Encoder

19
QFormer

20
2. Training the Model

21
Training
LLM Table Encoder LLM
DPO

22
3. Data Collection /
Processing

23
Data Collection
● Focus on coding applications (important to BI)
● Data:
○ Coding data
■ StackOverflow
■ GitHub
○ General
■ Textbook
● Finance
● Maths
● Biology
● Etc.
■ Kaggle
● Data analysis QAs
○ Table-related QA
● Synthetically generated table QAs
○ Vetted by humans
○ Synthethize then refine

24
Tabular Data
● Database tables
● Web pages
● Excel
● Academic task
● Research datasets
● Misc.
Data Curation
● Filtering
● Cleaning
● Augmentation

25
4. Agents for Tabular
Data Analysis

26
Agentic Workflow
normalize tables, perform vector
database lookup with Milvus

27
System Prompt
[https://github.com/tablegpt/tablegpt-agent/blob/main/src/tablegpt/agent/data_analyzer.py]

28
5. Code and Demo
• TableGPT2  FineTuned Qwen2.5
• Documentation explaining agentic workflow

29
Summary

30
Summary
● We can learn self-supervised / unsupervised representations of
tables
● We can then learn to combine these representations with
foundation models, e.g. LLMs, to work across modalities
● Many applications in Business Intelligence
○ Construct an Agent with Milvus, Langchain, etc.
● TableGPT2 is only one such approach!

31
Criticisms of TableGPT2
● Code and weights for table encoder not currently available (as
of Jan 13, 2024
● Trouble reconciling TableLLM and TableGPT2 benchmark
tables. GPT3.5 and GPT4 often best
● Potential cherry-picking of benchmarks and baselines

Milvus Office
Hours!
Get 11 hands-on support on your
vector database projects

Women in AI
RAG Hackathon
Connect, learn, and push the
boundaries of AI technology
in a supportive and inspiring
environment!
📍Stanford, Palo Alto
🗓 Jan 25, 2025
🔗 lu.ma/women-in-tech

34
x.com/milvusio
linkedin.com/company/
the-milvus-project
milvus.io/discord
github.com/milvus-io/milvus
zilliz.com/learn/
milvus-notebooks
milvus.io/docs/tutorials-
overview.md

1 Table = 1000 Words? Foundation Models for Tabular Data

More Related Content

Similar to 1 Table = 1000 Words? Foundation Models for Tabular Data

More from Zilliz

Recently uploaded

1 Table = 1000 Words? Foundation Models for Tabular Data