Using Ibis to work with small and large data in Python

1. Presented by:
Elephants, ibises and a more Pythonic
way to work with databases
May 12, 2023
Marlene Mhangami

2. 2
● Harare, Zimbabwe
● Developer Advocate Voltron
Data
● Vice Chair ACM practitioner
board
● Previous Vice Chair and
Director for the Python
Software Foundation
@marlene_zw
marlene.bsky.social
marlenemhangami.com
Marlene Mhangami

3. 3
Art Around
Malaga
post conference exploration
image of Malaga from spain.info

4. European fine arts, decorative arts and architecture from the 3rd to 19th centuries.
https://www.wga.hu/index_database.html

5. Let’s look at some code
and art!

6. 6
Art Around New
York
post conference exploration
image of Time Square from frommers.com

7. 7
The MET
post conference exploration
image of ASAP Rocky and Rihanna from the
New York Times 2023

8. The Metropolitan Museum of Art Collection (over 450000 works of art)
https://www.metmuseum.org/art/the-collection

9. 9
Which tool should we use to
work with the MET dataset?
▶ Dataset on Kaggle is 36GB
▶ A typical laptop has 8GB of RAM
▶ 5-10X as much RAM as the dataset size
Add item
or phrase
we ❤️
pandas,
but what
do we do
when it
breaks?

10. 10
Which tool should we use to
work with the MET dataset?
▶ Database engines are built to handle
data of any size
▶ Many data scientists use Python
▶ Ibis generates SQL and the database
engine (backend) does the heavy lifting
Add item
or phrase
generate
sql and let
the engine
do the
lifting🦾

11. 11
- pandas-like
- compiles to sql
(executed efficiently)
* anything you can
write with a SELECT
statement you can
write in ibis

12. 12
The MET on Google BigQuery
▶ 200,000 art pieces
▶ Data hosted in the cloud publicly
▶ Data is consistently updated
▶ Uses SQL to query

13. Let’s look at some code
and art!

14. 14
With The Right Tools Big
Data Should Feel Small

15. 15
Tools For Big Data in Python
▶ Ibis
▶ Polars
▶ Apache Arrow
▶ DuckDB

16. 16
Try Ibis Out!

17. marlene@voltrondata.com
@marlene_zw
Contact
TRY THE WORKSHOP!
Twitter
SOCIAL MEDIA
GENERAL CONTACT
https://voltrondata-labs.github.io/2023-jonthebeach-ibis/

18. Thanks