Ibis: Seamless Transition Between Pandas and Spark

1. Ibis:
Seamless Transition Between
Pandas and Spark

2. ● Modeling Platform and Tools @ Two Sigma
● Apache Spark, Apache Arrow, Ibis
About Me: Li Jin

3. ● Software engineer at Two Sigma
● Distributed compute, Spark
● Running, hiking, traveling
About Me: Hyonjee Joo

4. Outline
● Introduction: A common data science problem
● Ibis: A high level introduction
● Ibis Language
● Pyspark backend for Ibis
● Demo
● Conclusion

5. df = pd.read_parquet('s3://…')
df['feature'] = (df['v1'] + df['v2']) / 2
df['feature'] = df.groupby('key')['feature']
.rolling(7, min_periods=1).mean()
.sort_index(level=1)
.reset_index(drop=True)
df.groupby('key').agg({'feature': {'max', 'min'}})
A common data science problem...

6. ● Pandas is good for medium size data (<10G)
● You machine doesn’t have enough RAM to hold all the data
● You want to use more CPUs
You are happy until...The code runs too slow

7. ● Use a bigger machine (You can get machines with 1T RAM these
days)
● Pros:
○ Low human cost: no code change
● Cons:
○ Not scalable
○ Same software limits (e.g., single thread)
○ Can be difficult depending on your organization
Try a few things...

8. ● Use a generic way to distribution pandas code:
○ e.g.: sparkContext.parallelize(range(2000, 2020)).map(lambda year:
compute_for_year(year)).collect()
○ dask.delayed
● Pros:
○ Medium human cost: small code change
○ Scalable
● Cons:
○ Works only for embarrassingly parallel problems
Try a few things...

9. Try a few things...
● Use a distributed dataframe library
○ Spark
○ Dask
○ Koalas
○ …
● Pros:
○ Scalable
● Cons:
○ High human cost: learn another API
○ Not obvious which one to use

10. Take a step back...
df = pd.read_parquet('s3://…')
df['feature'] = (df['v1'] + df['v2']) / 2
df['feature2'] = df.groupby('key')['feature']
.rolling(7, min_periods=1).mean()
.sort_index(level=1)
.reset_index(drop=True)
df.groupby('key').agg({'feature2': {'max', 'min'}})

11. Separation of API and execution
● The problem:
○ software and hardware that we use (pandas on single machine) cannot
execute the computation we want to do.
● Can we separate “what we want to do” and “how it is done”

12. Separation of API and execution
● Yes!
● SQL is a way to express the compute that is independent of the
software and hardware
● Can we have something that is like SQL, but for Python?

13. Don’t limit yourself to what you can use today
● Current “DataFrame”:
○ Pandas
○ Spark
○ Dask
● Future “DataFrame”:
○ Arrow Dataset
○ Cudf / dask-cudf
○ Ray / Modin
○ ...

14. Ibis: A high level
introduction

15. Ibis: Python Data Analysis Productivity Framework
● Open source
● Started in 2014 by Wes McKinney
● Worked on by top committers of Pandas:
○ Wes McKinney
○ Phillip Cloud
○ Jeff Reback

16. Ibis components
● ibis “language”
○ The ibis language that is used to express the computation
● ibis “backends”
○ Modules that translate ibis expressions to sth that is executable by
different computation engines
■ ibis.pandas
■ ibis.pyspark
■ Ibis.bigquery
■ ...

17. Ibis language
● Table API
○ Projection
○ Filtering
○ Join
○ Groupby
○ Sort
○ Window
○ Aggregation
○ …
○ AsofJoin
○ UDFs
● Ibis expressions (intermediate representation)

18. Ibis language example
● Table API:
○ table = table.mutate(v3=table[‘v1’] + table[‘v2’])
○ type(table) = ibis.TableExpr
● Ibis expressions:

19. Ibis backends
● Ibis expressions -> Backend specific expressions
● table.mutate(v3=table[‘v1’] + table[‘v2’])
○ Pandas: df = df.assign(v3=df[‘v1’] + df[‘v2’])
○ PySpark: df = df.withColumn(‘v3’, df[‘v1’] + df[‘v2’])

20. Ibis Example

21. Recall our earlier example in Pandas
df['feature'] = (df['v1'] + df['v2']) / 2
df['feature2'] = df.groupby('key')['feature']
.rolling(7, min_periods=1).mean()
.sort_index(level=1)
.reset_index(drop=True)
df.groupby('key').agg({'feature2': {'max', 'min'}})

22. Basic expression and column selection
Pandas expression
df['feature'] = (df['v1'] + df['v2']) / 2

23. Basic expression and column selection
Ibis expression
table = table.mutate(
feature=(table.v1 + table.v2)/2
)
Pandas expression
df['feature'] = (df['v1'] + df['v2']) / 2

24. Group-by and windowed aggregation
Ibis expression
table = table.mutate(
feature=(table.v1 + table.v2)/2
)
Pandas expression
df['feature'] = (df['v1'] + df['v2']) / 2
df['feature2'] = df.groupby('key')['feature']
.rolling(3, min_periods=1).mean()
.sort_index(level=1)
.reset_index(drop=True)

25. Group-by and windowed aggregation
Ibis expression
table = table.mutate(
feature=(table.v1 + table.v2)/2
)
w = ibis.window(group_by='key', order_by='v1',
preceding=2, following=0)
table = table.mutate(
feature2=table.feature.mean().over(w)
)
Pandas expression
df['feature'] = (df['v1'] + df['v2']) / 2
df['feature2'] = df.groupby('key')['feature']
.rolling(3, min_periods=1).mean()
.sort_index(level=1)
.reset_index(drop=True)

26. Composite aggregation
Ibis expression
table = table.mutate(
feature=(table.v1 + table.v2)/2
)
w = ibis.window(group_by='key', order_by='v1',
preceding=2, following=0)
table = table.mutate(
feature2=table.feature.mean().over(w)
)
Pandas expression
df['feature'] = (df['v1'] + df['v2']) / 2
df['feature2'] = df.groupby('key')['feature']
.rolling(3, min_periods=1).mean()
.sort_index(level=1)
.reset_index(drop=True)
df.groupby('key').agg(
{'feature2': {'max', 'min'}}
)

27. Composite aggregation
Ibis expression
table = table.mutate(
feature=(table.v1 + table.v2)/2
)
w = ibis.window(group_by='key', order_by='v1',
preceding=2, following=0)
table = table.mutate(
feature2=table.feature.mean().over(w)
)
metrics = [table.feature2.min(),
table.feature2.max()]
table.group_by('key').aggregate(metrics)
Pandas expression
df['feature'] = (df['v1'] + df['v2']) / 2
df['feature2'] = df.groupby('key')['feature']
.rolling(3, min_periods=1).mean()
.sort_index(level=1)
.reset_index(drop=True)
df.groupby('key').agg(
{'feature2': {'max', 'min'}}
)

28. Final translation
Ibis expression
table = table.mutate(
feature=(table.v1 + table.v2)/2
)
w = ibis.window(group_by='key', order_by='v1',
preceding=2, following=0)
table = table.mutate(
feature2=table.feature.mean().over(w)
)
metrics = [table.feature2.min(),
table.feature2.max()]
table.group_by('key').aggregate(metrics)
Pandas expression
df['feature'] = (df['v1'] + df['v2']) / 2
df['feature2'] = df.groupby('key')['feature']
.rolling(3, min_periods=1).mean()
.sort_index(level=1)
.reset_index(drop=True)
df.groupby('key').agg(
{'feature2': {'max', 'min'}}
)

29. Ibis: PySpark
backend

30. A distributed backend for Ibis with PySpark
Ibis expression tree

31. Initialize Ibis PySpark client
Ibis expression
session = SparkSession.builder.getOrCreate()
client = ibis.pyspark.connect(session)

32. Ibis expression
table = client.table('table')
Access table in Ibis

33. Basic expression and column selection
Ibis expression
table.mutate(
feature=(table.v1 + table.v2)/2
)

34. Basic expression and column selection
Ibis expression
table.mutate(
feature=(table.v1 + table.v2)/2
)

35. Basic expression and column selection
Ibis expression
table.mutate(
feature=(table.v1 + table.v2)/2
)

36. Basic expression and column selection
Ibis expression
table.mutate(
feature=(table.v1 + table.v2)/2
)
Code
@compiles(ops.Divide)
def compile_divide(t, expr, scope, **kwargs):
op = expr.op()
left = t.translate(op.left, scope)
right = t.translate(op.right, scope)
return left / right

37. Basic expression and column selection
Ibis expression
table.mutate(
feature=(table.v1 + table.v2)/2
)
Code
@compiles(ops.Divide)
def compile_divide(t, expr, scope, **kwargs):
op = expr.op()
left = t.translate(op.left, scope)
right = t.translate(op.right, scope)
return left / right
t is a PySparkTranslator

38. Basic expression and column selection
Ibis expression
table.mutate(
feature=(table.v1 + table.v2)/2
)
Code
@compiles(ops.Divide)
def compile_divide(t, expr, scope, **kwargs):
op = expr.op()
left = t.translate(op.left, scope)
right = t.translate(op.right, scope)
return left / right
t is a PySparkTranslator
It has a translate() method that
translates an Ibis expression into a
PySpark object.

39. Basic expression and column selection
Ibis expression
table.mutate(
feature=(table.v1 + table.v2)/2
)
Code
@compiles(ops.Divide)
def compile_divide(t, expr, scope, **kwargs):
op = expr.op()
left = t.translate(op.left, scope)
right = t.translate(op.right, scope)
return left / right
expr is the ibis expression to
translate

40. Basic expression and column selection
Ibis expression
table.mutate(
feature=(table.v1 + table.v2)/2
)
Code
@compiles(ops.Divide)
def compile_divide(t, expr, scope, **kwargs):
op = expr.op()
left = t.translate(op.left, scope)
right = t.translate(op.right, scope)
return left / right
scope is a dict that caches results
of previously translated Ibis
expressions.

41. Basic expression and column selection
Ibis expression
table.mutate(
feature=(table.v1 + table.v2)/2
)
Code
@compiles(ops.Divide)
def compile_divide(t, expr, scope, **kwargs):
op = expr.op()
left = t.translate(op.left, scope)
right = t.translate(op.right, scope)
return left / right
left and right are
PySpark columns

42. Basic expression and column selection
Ibis expression
table.mutate(
feature=(table.v1 + table.v2)/2
)
Code
@compiles(ops.Divide)
def compile_divide(t, expr, scope, **kwargs):
op = expr.op()
left = t.translate(op.left, scope)
right = t.translate(op.right, scope)
return left / right
PySpark column division

43. Basic expression and column selection
Ibis expression
table.mutate(
feature=(table.v1 + table.v2)/2
)
Code
@compiles(ops.Divide)
def compile_divide(t, expr, scope, **kwargs):
op = expr.op()
left = t.translate(op.left, scope)
right = t.translate(op.right, scope)
return left / right

44. Basic expression and column selection
Ibis expression
table.mutate(
feature=(table.v1 + table.v2)/2
)
Code
@compiles(ops.Add)
def compile_add(t, expr, scope, **kwargs):
op = expr.op()
left = t.translate(op.left, scope)
right = t.translate(op.right, scope)
return left + right

45. Basic expression and column selection
Ibis expression
table.mutate(
feature=(table.v1 + table.v2)/2
)
Code
@compiles(ops.Add)
def compile_add(t, expr, scope, **kwargs):
op = expr.op()
left = t.translate(op.left, scope)
right = t.translate(op.right, scope)
return left + right
left and right are
PySpark columns
PySpark column addition

46. Basic expression and column selection
Ibis expression
table.mutate(
feature=(table.v1 + table.v2)/2
)
Code
@compiles(ops.Add)
def compile_add(t, expr, scope, **kwargs):
op = expr.op()
left = t.translate(op.left, scope)
right = t.translate(op.right, scope)
return left + right

47. Basic expression and column selection
Ibis expression
table.mutate(
feature=(table.v1 + table.v2)/2
)
Code
@compiles(ops.TableColumn)
def compile_column(t, expr, scope, **kwargs):
op = expr.op()
column_name = op.name
pyspark_df = t.translate(op.table, scope)
return pyspark_df[column_name]

48. Group-by and windowed aggregation
Ibis expression
w = ibis.window(group_by='key', order_by='v1',
preceding=2, following=0)
table = table.mutate(
feature2=table.feature.mean().over(w)
)

49. Group-by and windowed aggregation
Ibis expression
w = ibis.window(group_by='key', order_by='v1',
preceding=2, following=0)
table = table.mutate(
feature2=table.feature.mean().over(w)
)

50. Group-by and windowed aggregation
Ibis expression
w = ibis.window(group_by='key', order_by='v1',
preceding=2, following=0)
table = table.mutate(
feature2=table.feature.mean().over(w)
)

51. Group-by and windowed aggregation
Ibis expression
w = ibis.window(group_by='key', order_by='v1',
preceding=2, following=0)
table = table.mutate(
feature2=table.feature.mean().over(w)
)

52. Group-by and windowed aggregation
Ibis expression
w = ibis.window(group_by='key', order_by='v1',
preceding=2, following=0)
table = table.mutate(
feature2=table.feature.mean().over(w)
)
Code
@compiles(ops.WindowOp)
def compile_window_op(t, expr, scope, **kwargs):
op = expr.op()
ibis_window = op.window
operand = op.expr
.
# pull out grouping_keys, ordering_keys, start
# and end from ibis_window
pyspark_window = Window.partitionBy(
grouping_keys).orderBy(ordering_keys)
.rowsBetween(start, end)
return t.translate(operand, scope,
window=pyspark_window)

53. Group-by and windowed aggregation
Ibis expression
w = ibis.window(group_by='key', order_by='v1',
preceding=2, following=0)
table = table.mutate(
feature2=table.feature.mean().over(w)
)
Code
@compiles(ops.WindowOp)
def compile_window_op(t, expr, scope, **kwargs):
op = expr.op()
ibis_window = op.window
operand = op.expr
.
# pull out grouping_keys, ordering_keys, start
# and end from ibis_window
pyspark_window = Window.partitionBy(
grouping_keys).orderBy(ordering_keys)
.rowsBetween(start, end)
return t.translate(operand, scope,
window=pyspark_window)

54. Group-by and windowed aggregation
Ibis expression
w = ibis.window(group_by='key', order_by='v1',
preceding=2, following=0)
table = table.mutate(
feature2=table.feature.mean().over(w)
)
Code
@compiles(ops.WindowOp)
def compile_window_op(t, expr, scope, **kwargs):
op = expr.op()
ibis_window = op.window
operand = op.expr
.
# pull out grouping_keys, ordering_keys, start
# and end from ibis_window
pyspark_window = Window.partitionBy(
grouping_keys).orderBy(ordering_keys)
.rowsBetween(start, end)
return t.translate(operand, scope,
window=pyspark_window)

55. Group-by and windowed aggregation
Ibis expression
w = ibis.window(group_by='key', order_by='v1',
preceding=2, following=0)
table = table.mutate(
feature2=table.feature.mean().over(w)
)
Code
@compiles(ops.WindowOp)
def compile_window_op(t, expr, scope, **kwargs):
op = expr.op()
ibis_window = op.window
operand = op.expr
.
# pull out grouping_keys, ordering_keys, start
# and end from ibis_window
pyspark_window = Window.partitionBy(
grouping_keys).orderBy(ordering_keys)
.rowsBetween(start, end)
return t.translate(operand, scope,
window=pyspark_window)

56. Group-by and windowed aggregation
Ibis expression
w = ibis.window(group_by='key', order_by='v1',
preceding=2, following=0)
table = table.mutate(
feature2=table.feature.mean().over(w)
)
Code
@compiles(ops.Mean)
def compile_mean(t, expr, scope, context=None,
**kwargs):
def fn(col):
if 'window' in kwargs:
window = kwargs['window']
return F.mean(col).over(window)
else:
return F.mean(col)
return compile_aggregator(
t, expr, scope, fn, context, **kwargs
)

57. Group-by and windowed aggregation
Ibis expression
w = ibis.window(group_by='key', order_by='v1',
preceding=2, following=0)
table = table.mutate(
feature2=table.feature.mean().over(w)
)
Code
@compiles(ops.Mean)
def compile_mean(t, expr, scope, context=None,
**kwargs):
def fn(col):
if 'window' in kwargs:
window = kwargs['window']
return F.mean(col).over(window)
else:
return F.mean(col)
return compile_aggregator(
t, expr, scope, fn, context, **kwargs
)
where F is pyspark.sql.functions

58. Composite aggregation
Ibis expression
metrics = [
table.feature2.min(),
table.feature2.max()
]
result =
table.group_by('key').aggregate(metrics)

59. Composite aggregation
Ibis expression
metrics = [
table.feature2.min(),
table.feature2.max()
]
result =
table.group_by('key').aggregate(metrics)

60. Composite aggregation
Ibis expression
metrics = [
table.feature2.min(),
table.feature2.max()
]
result =
table.group_by('key').aggregate(metrics)

61. Composite aggregation
Ibis expression
metrics = [
table.feature2.min(),
table.feature2.max()
]
result =
table.group_by('key').aggregate(metrics)

62. Composite aggregation
Ibis expression
metrics = [
table.feature2.min(),
table.feature2.max()
]
result =
table.group_by('key').aggregate(metrics)
Code
@compiles(ops.Aggregation)
def compile_aggregation(t, expr, scope, **kwargs):
op = expr.op()
pyspark_df = t.translate(op.table, scope)
if op.by:
context = AggregationContext.GROUP
aggs = [_compile_agg(...) for m in op.metrics]
bys = [t.translate(b, ...) for b in op.by]
return pyspark_df.groupby(*bys).agg(*aggs)
else:
context = AggregationContext.ENTIRE
aggs = [_compile_agg(...) for m in
op.metrics]

63. Composite aggregation
Ibis expression
metrics = [
table.feature2.min(),
table.feature2.max()
]
result =
table.group_by('key').aggregate(metrics)
Code
@compiles(ops.Aggregation)
def compile_aggregation(t, expr, scope, **kwargs):
op = expr.op()
pyspark_df = t.translate(op.table, scope)
if op.by:
context = AggregationContext.GROUP
aggs = [_compile_agg(...) for m in op.metrics]
bys = [t.translate(b, ...) for b in op.by]
return pyspark_df.groupby(*bys).agg(*aggs)
else:
context = AggregationContext.ENTIRE
aggs = [_compile_agg(...) for m in
op.metrics]
List of
PySpark
columns/
column
expressions

64. Composite aggregation
Ibis expression
metrics = [
table.feature2.min(),
table.feature2.max()
]
result =
table.group_by('key').aggregate(metrics)
Code
@compiles(ops.Aggregation)
def compile_aggregation(t, expr, scope, **kwargs):
op = expr.op()
pyspark_df = t.translate(op.table, scope)
if op.by:
context = AggregationContext.GROUP
aggs = [_compile_agg(...) for m in op.metrics]
bys = [t.translate(b, ...) for b in op.by]
return pyspark_df.groupby(*bys).agg(*aggs)
else:
context = AggregationContext.ENTIRE
aggs = [_compile_agg(...) for m in
op.metrics]
PySpark
groupby
and
aggregate

65. Demo

66. Conclusion

67. Conclusion
● Separate API from execution
● Don’t limit only to what you can use today, think about what you can
use in the future
○ Arrow Dataset
○ Modin
○ cudf / dask-cudf
○ ...

68. Thanks!
ice.xelloss@gmail.com (@icexelloss)
hyonjee.joo@twosigma.com (@hjoo)