Ibis + Substrait.pdf
•
0 likes•32 views
Ibis and Substrait are driving standardization in analytics system
Recommended
Recommended
More Related Content
Similar to Ibis + Substrait.pdf
Similar to Ibis + Substrait.pdf (20)
Recently uploaded
Recently uploaded (20)
Ibis + Substrait.pdf
- 1. Presented by: Ibis and Substrait: Standardized Analytics 06/23/2022 Hussain Sultan
- 2. 2 Ibis produces Expressions or Strings for execution on SQL backends t.groupby('region') .aggregate(topk=t.funding_rounds.count()) .sort_by(ibis.desc('topk')) .head() Produces SELECT region ,count(funding_rounds) AS topk FROM companies GROUP BY region ORDER BY topk SQL Strings generated for Execution Engine SQL Backends <sqlalchemy.sql.selectable.S elect object> SQLAlchemy Expression
- 3. 3 String Execution Backend Ibis API Operations Compile SQL String Backend Specific SQL Flavor and Support Type Check and Optimize Rewrite Build Ibis Expression
- 4. 4 Expression Generating Backends API Operation Compile SA Expression Backend Specific Expressions or SQLAlchemy Type Check and Optimize Rewrite Build Ibis Expression
- 5. 5 Ibis Backends - LOC SQLAlchemy-based Backends tend to be lower LOC
- 6. 6 What if Ibis supported 100’s of backends? DB Engines lists 398 Databases in their popularity ranking https://db-engines.com/en/ranking
- 7. 7 Substrait: Serialized Relational Algebra substrait views ibis python r scala babel sql presto sql dplyr Producers Consumers ibis
- 8. 8 Substrait: Serialized Relational Algebra Plan Relations Types i8 i32 fp32 struct Expressions Functions add rank join agg filter read case literal Serialization protobuf text ● Types ● Expressions ● Functions ● Relations ● Plans ● Serialization formats
- 10. 10 Ibis-substrait producer is actively being developed SQL String Python Expression Under Development Ibis API Compile Substrait Producer
- 11. 11 Demo
- 12. 12 Takeaways ● Ibis as a substrait producer, decouples the execution engines and expression building ● Check out Gil Forsyth’s gist that explains the duckdb and ibis substrait integration: https://gist.github.com/gforsyth/496d680e1e29f0876df937e e5091e1b8
- 13. 13 Thank you!
- 14. 14 2. What if we want to consume any SQL DSL into Ibis expressions? API Ops DSL Execute
- 15. 15 Ibis support multiple execution paradigms SQL String Python Expression Direct Execution Focus of Today’s Talk 1 Ibis API 2 Compile