This document discusses Ibis, an analytics library that generates SQL expressions and strings for execution on SQL backends. It introduces Substrait, a standardized relational algebra format that can serialize Ibis expressions. Substrait decouples the expression building in Ibis from the execution engines, allowing Ibis to potentially support hundreds of different backends. The document demonstrates compiling an Ibis expression to the Substrait format and shows how Ibis can consume any SQL dialect by compiling it to Ibis expressions via a Substrait producer.

1. Presented
by:
Ibis and Substrait: Standardized
Analytics
06/23/2022
Hussain Sultan

2. 2
Ibis produces Expressions or Strings for
execution on SQL backends
t.groupby('region')
.aggregate(topk=t.funding_rounds.count())
.sort_by(ibis.desc('topk'))
.head()
Produces
SELECT region
,count(funding_rounds) AS topk
FROM companies
GROUP BY region
ORDER BY topk
SQL Strings generated for
Execution Engine
SQL
Backends
<sqlalchemy.sql.selectable.S
elect object>
SQLAlchemy Expression

3. 3
String Execution Backend
Ibis API Operations Compile SQL String
Backend Specific SQL Flavor and Support
Type Check and Optimize
Rewrite
Build Ibis Expression

4. 4
Expression Generating Backends
API Operation Compile
SA
Expression
Backend Specific Expressions or SQLAlchemy
Type Check and Optimize
Rewrite
Build Ibis Expression

5. 5
Ibis Backends - LOC
SQLAlchemy-based
Backends tend to be lower LOC

6. 6
What if Ibis supported 100’s of backends?
DB Engines lists 398 Databases in their popularity ranking
https://db-engines.com/en/ranking

7. 7
Substrait: Serialized Relational Algebra
substrait
views
ibis
python r scala
babel sql presto sql
dplyr
Producers
Consumers
ibis

8. 8
Substrait: Serialized Relational Algebra
Plan
Relations
Types
i8 i32
fp32 struct
Expressions
Functions
add
rank
join agg
filter read
case
literal
Serialization
protobuf text
● Types
● Expressions
● Functions
● Relations
● Plans
● Serialization formats

9. 9
What’s Next?

10. 10
Ibis-substrait producer is actively being
developed
SQL
String
Python
Expression
Under Development
Ibis API
Compile
Substrait
Producer

11. 11
Demo

12. 12
Takeaways
● Ibis as a substrait producer, decouples the execution
engines and expression building
● Check out Gil Forsyth’s gist that explains the duckdb and
ibis substrait integration:
https://gist.github.com/gforsyth/496d680e1e29f0876df937e
e5091e1b8

13. 13
Thank you!

14. 14
2. What if we want to consume any SQL
DSL into Ibis expressions?
API Ops
DSL Execute

15. 15
Ibis support multiple execution paradigms
SQL
String
Python
Expression
Direct
Execution
Focus of Today’s Talk
1
Ibis API
2
Compile