The document discusses challenges in migrating imperative deep learning programs to graph execution. It provides examples of TensorFlow imperative code that uses features like Python side effects and variables that do not directly translate to graph execution. Specifically, it shows how a model that uses a counter variable to increment another variable on each call would not work as expected, as the initial counter value is captured during tracing, resulting in the variable being incremented on each call rather than just the first one. This demonstrates common problems that can arise from migrating imperative code to graphs and result in unexpected numerical results or reduced performance.

1. Introduction Motivation Approach Methodology Results Conclusion
Challenges in Migrating Imperative Deep
Learning Programs to Graph Execution: An
Empirical Study
Tatiana Castro Vélez1
Raffi Khatchadourian1,2
Mehdi
Bagherzadeh3
Anita Raja1,2
1
City University of New York (CUNY) Graduate Center, USA
2
City University of New York (CUNY) Hunter College, USA
3
Oakland University, USA
Computer Science, George Mason University, March 24, 2022
To appear at the International Conference on Mining Software
Repositories (MSR ’22) co-located with ICSE ’22.
Tatiana Castro Vélez, Raffi Khatchadourian, Mehdi Bagherzadeh, Anita Raja Challenges in Migrating Imperative DL Programs to Graphs 1 / 32

2. Introduction Motivation Approach Methodology Results Conclusion
Deep Learning Systems & Run-time Performance
Machine Learning (ML), including Deep Learning (DL), systems are
pervasive.
As datasets grow, efficiency becomes essential to support
responsiveness [Zhou et al., 2020].
For efficiency, DL frameworks have traditionally embraced a deferred
execution-style supporting graph-based (DNN) computation.
Scalable, but development is . . .
Error-prone.
Cumbersome.
Produces programs that are difficult to debug.
Because graph computation executes statements in a non-imperative
order, traditional SE tools cannot help troubleshoot bugs [Arpteg
et al., 2018].
Tatiana Castro Vélez, Raffi Khatchadourian, Mehdi Bagherzadeh, Anita Raja Challenges in Migrating Imperative DL Programs to Graphs 2 / 32

3. TensorFlow Deferred Execution-style Code
1 # Build a graph.
2 a = tf.constant(5.0)
3 b = tf.constant(6.0)
4 c = a * b
5
6 # Launch graph in a session.
7 sess = tf.Session()
8
9 # Evaluate the tensor `c`.
10 print(sess.run(c)) # prints 30.0
Lines 2–4 build a computation graph.
Line 4 does not execute until the Session created on line 7 is run on
line 10.
No native support common imperative program constructs, e.g.,
iteration.

4. TensorFlow Deferred Execution-style Code
1 # Build a graph.
2 a = tf.constant(5.0)
3 b = tf.constant(6.0)
4 c = a * b
5
6 # Launch graph in a session.
7 sess = tf.Session()
8
9 # Evaluate the tensor `c`.
10 print(sess.run(c)) # prints 30.0
Lines 2–4 build a computation graph.
Line 4 does not execute until the Session created on line 7 is run on
line 10.
No native support common imperative program constructs, e.g.,
iteration.

5. TensorFlow Deferred Execution-style Code
1 # Build a graph.
2 a = tf.constant(5.0)
3 b = tf.constant(6.0)
4 c = a * b
5
6 # Launch graph in a session.
7 sess = tf.Session()
8
9 # Evaluate the tensor `c`.
10 print(sess.run(c)) # prints 30.0
Lines 2–4 build a computation graph.
Line 4 does not execute until the Session created on line 7 is run on
line 10.
No native support common imperative program constructs, e.g.,
iteration.

6. Introduction Motivation Approach Methodology Results Conclusion
Imperative DL Programming, Eager Execution &
Hybridization
More natural, less error-prone, and easier-to-debug imperative DL
frameworks (e.g., TensorFlow Eager, Keras, PyTorch) encouraging
eager execution have emerged.
Come at the expense of run-time performance.
Hybrid approaches (e.g., Hybridize, TorchScript, AutoGraph) have
emerged.
Integrated into mainstream DL frameworks (e.g.,
TensorFlow, MXNet, PyTorch).
Execute imperative DL programs as static graphs at run-time.
Tatiana Castro Vélez, Raffi Khatchadourian, Mehdi Bagherzadeh, Anita Raja Challenges in Migrating Imperative DL Programs to Graphs 4 / 32

7. TensorFlow Imperative (OO) DL Model Code (Images)
1 class SequentialModel(tf.keras.Model):
2 def __init__(self, **kwargs):
3 super(SequentialModel, self).__init__(...)
4 self.flatten = layers.Flatten(input_shape=(28, 28))
5 num_layers = 100 # Add many small layers.
6 self.layers = [layers.Dense(64, activation = "relu") for n in
range(num_layers)]
,
→
7 self.dropout = tf.keras.layers.Dropout(0.2)
8 self.dense_2 = tf.keras.layers.Dense(10)
9
10 @tf.function(...) # Executes model as graph (optional args).
11 def __call__(self, x):
12 x = self.flatten(x)
13 for layer in self.layers:
14 x = layer(x)
15 x = self.dropout(x)
16 x = self.dense_2(x)
17 return x
On line 10, AutoGraph used to potentially enhance performance.
Decorates model’s call() method with @tf.function, possibly
providing optional yet influential decorator arguments.
At run-time, call()’s execution will be “traced” (∼9.22 speedup).

8. TensorFlow Imperative (OO) DL Model Code (Images)
1 class SequentialModel(tf.keras.Model):
2 def __init__(self, **kwargs):
3 super(SequentialModel, self).__init__(...)
4 self.flatten = layers.Flatten(input_shape=(28, 28))
5 num_layers = 100 # Add many small layers.
6 self.layers = [layers.Dense(64, activation = "relu") for n in
range(num_layers)]
,
→
7 self.dropout = tf.keras.layers.Dropout(0.2)
8 self.dense_2 = tf.keras.layers.Dense(10)
9
10 @tf.function(...) # Executes model as graph (optional args).
11 def __call__(self, x):
12 x = self.flatten(x)
13 for layer in self.layers:
14 x = layer(x)
15 x = self.dropout(x)
16 x = self.dense_2(x)
17 return x
On line 10, AutoGraph used to potentially enhance performance.
Decorates model’s call() method with @tf.function, possibly
providing optional yet influential decorator arguments.
At run-time, call()’s execution will be “traced” (∼9.22 speedup).

9. TensorFlow Imperative (OO) DL Model Code (Images)
1 class SequentialModel(tf.keras.Model):
2 def __init__(self, **kwargs):
3 super(SequentialModel, self).__init__(...)
4 self.flatten = layers.Flatten(input_shape=(28, 28))
5 num_layers = 100 # Add many small layers.
6 self.layers = [layers.Dense(64, activation = "relu") for n in
range(num_layers)]
,
→
7 self.dropout = tf.keras.layers.Dropout(0.2)
8 self.dense_2 = tf.keras.layers.Dense(10)
9
10 @tf.function(...) # Executes model as graph (optional args).
11 def __call__(self, x):
12 x = self.flatten(x)
13 for layer in self.layers:
14 x = layer(x)
15 x = self.dropout(x)
16 x = self.dense_2(x)
17 return x
On line 10, AutoGraph used to potentially enhance performance.
Decorates model’s call() method with @tf.function, possibly
providing optional yet influential decorator arguments.
At run-time, call()’s execution will be “traced” (∼9.22 speedup).

10. Introduction Motivation Approach Methodology Results Conclusion
Hybridization Drawbacks
Necessitate non-trivial, specialized metadata [Jeong et al., 2019].
Exhibit limitations and known issues with native program constructs.
Subtle considerations are required to:
Make code amenable to safe, accurate, and efficient graph execution.
Avoid performance bottlenecks and semantically inequivalent results.
Tatiana Castro Vélez, Raffi Khatchadourian, Mehdi Bagherzadeh, Anita Raja Challenges in Migrating Imperative DL Programs to Graphs 6 / 32

11. Introduction Motivation Approach Methodology Results Conclusion
Imperative TensorFlow Code With Python Side-effects
1 @tf.function
2 def f(x):
3 print("Input: ", x)
4 f(1)
5 f(1)
6 f(2)
Output (expecting 1, 1, 2):
Input: 1
Input: 2
Side-effect producing, native Python statements are problematic for
tf.function-decorated functions (i.e., “tf.functions”).
Example
Printing, list appending, global variable mutation.
Because they are traced, a function’s behavior is “etched” into its
corresponding graph.
Can have unexpectant results, executing side-effects multiple times
or not at all.
Side-effects occur when tf.functions are called the first time.
Subsequent calls with similar arguments execute the graph instead.
Tatiana Castro Vélez, Raffi Khatchadourian, Mehdi Bagherzadeh, Anita Raja Challenges in Migrating Imperative DL Programs to Graphs 7 / 32

12. Introduction Motivation Approach Methodology Results Conclusion
Imperative TensorFlow Code With Python Side-effects
1 @tf.function
2 def f(x):
3 print("Input: ", x)
4 f(1)
5 f(1)
6 f(2)
Output (expecting 1, 1, 2):
Input: 1
Input: 2
Side-effect producing, native Python statements are problematic for
tf.function-decorated functions (i.e., “tf.functions”).
Example
Printing, list appending, global variable mutation.
Because they are traced, a function’s behavior is “etched” into its
corresponding graph.
Can have unexpectant results, executing side-effects multiple times
or not at all.
Side-effects occur when tf.functions are called the first time.
Subsequent calls with similar arguments execute the graph instead.
Tatiana Castro Vélez, Raffi Khatchadourian, Mehdi Bagherzadeh, Anita Raja Challenges in Migrating Imperative DL Programs to Graphs 7 / 32

13. Introduction Motivation Approach Methodology Results Conclusion
Imperative TensorFlow Code With Python Side-effects
1 @tf.function
2 def f(x):
3 print("Input: ", x)
4 f(1)
5 f(1)
6 f(2)
Output (expecting 1, 1, 2):
Input: 1
Input: 2
Side-effect producing, native Python statements are problematic for
tf.function-decorated functions (i.e., “tf.functions”).
Example
Printing, list appending, global variable mutation.
Because they are traced, a function’s behavior is “etched” into its
corresponding graph.
Can have unexpectant results, executing side-effects multiple times
or not at all.
Side-effects occur when tf.functions are called the first time.
Subsequent calls with similar arguments execute the graph instead.
Tatiana Castro Vélez, Raffi Khatchadourian, Mehdi Bagherzadeh, Anita Raja Challenges in Migrating Imperative DL Programs to Graphs 7 / 32

14. Introduction Motivation Approach Methodology Results Conclusion
Imperative TensorFlow Code With Python Side-effects
1 @tf.function
2 def f(x):
3 print("Input: ", x)
4 f(1)
5 f(1)
6 f(2)
Output (expecting 1, 1, 2):
Input: 1
Input: 2
Side-effect producing, native Python statements are problematic for
tf.function-decorated functions (i.e., “tf.functions”).
Example
Printing, list appending, global variable mutation.
Because they are traced, a function’s behavior is “etched” into its
corresponding graph.
Can have unexpectant results, executing side-effects multiple times
or not at all.
Side-effects occur when tf.functions are called the first time.
Subsequent calls with similar arguments execute the graph instead.
Tatiana Castro Vélez, Raffi Khatchadourian, Mehdi Bagherzadeh, Anita Raja Challenges in Migrating Imperative DL Programs to Graphs 7 / 32

15. Introduction Motivation Approach Methodology Results Conclusion
Imperative TensorFlow Code With Python Side-effects
1 @tf.function
2 def f(x):
3 print("Input: ", x)
4 f(1)
5 f(1)
6 f(2)
Output (expecting 1, 1, 2):
Input: 1
Input: 2
Side-effect producing, native Python statements are problematic for
tf.function-decorated functions (i.e., “tf.functions”).
Example
Printing, list appending, global variable mutation.
Because they are traced, a function’s behavior is “etched” into its
corresponding graph.
Can have unexpectant results, executing side-effects multiple times
or not at all.
Side-effects occur when tf.functions are called the first time.
Subsequent calls with similar arguments execute the graph instead.
Tatiana Castro Vélez, Raffi Khatchadourian, Mehdi Bagherzadeh, Anita Raja Challenges in Migrating Imperative DL Programs to Graphs 7 / 32

16. Introduction Motivation Approach Methodology Results Conclusion
Imperative TensorFlow Code Using a Counter
1 class Model(tf.Module):
2 def __init__(self):
3 self.v = tf.Variable(0)
4 self.counter = 0
5
6 @tf.function
7 def __call__(self):
8 if self.counter == 0:
9 self.counter += 1
10 self.v.assign_add(1)
11 return self.v
12 m = Model()
13 for n in range(3):
14 print(m().numpy())
Output (expecting 1, 1, 1):
1
2
3
A model uses a counter to safeguard a variable incrementation.
The initial value of counter (line 4), however, is captured during
tracing upon the first model invocation (line 14).
Variable v is incremented unconditionally (line 10) each time the
model is invoked.
Such problems are common in migrating to graph execution.
Can result in suspicious numerical results or lower performance.
Tatiana Castro Vélez, Raffi Khatchadourian, Mehdi Bagherzadeh, Anita Raja Challenges in Migrating Imperative DL Programs to Graphs 8 / 32

17. Introduction Motivation Approach Methodology Results Conclusion
Imperative TensorFlow Code Using a Counter
1 class Model(tf.Module):
2 def __init__(self):
3 self.v = tf.Variable(0)
4 self.counter = 0
5
6 @tf.function
7 def __call__(self):
8 if self.counter == 0:
9 self.counter += 1
10 self.v.assign_add(1)
11 return self.v
12 m = Model()
13 for n in range(3):
14 print(m().numpy())
Output (expecting 1, 1, 1):
1
2
3
A model uses a counter to safeguard a variable incrementation.
The initial value of counter (line 4), however, is captured during
tracing upon the first model invocation (line 14).
Variable v is incremented unconditionally (line 10) each time the
model is invoked.
Such problems are common in migrating to graph execution.
Can result in suspicious numerical results or lower performance.
Tatiana Castro Vélez, Raffi Khatchadourian, Mehdi Bagherzadeh, Anita Raja Challenges in Migrating Imperative DL Programs to Graphs 8 / 32

18. Introduction Motivation Approach Methodology Results Conclusion
Imperative TensorFlow Code Using a Counter
1 class Model(tf.Module):
2 def __init__(self):
3 self.v = tf.Variable(0)
4 self.counter = 0
5
6 @tf.function
7 def __call__(self):
8 if self.counter == 0:
9 self.counter += 1
10 self.v.assign_add(1)
11 return self.v
12 m = Model()
13 for n in range(3):
14 print(m().numpy())
Output (expecting 1, 1, 1):
1
2
3
A model uses a counter to safeguard a variable incrementation.
The initial value of counter (line 4), however, is captured during
tracing upon the first model invocation (line 14).
Variable v is incremented unconditionally (line 10) each time the
model is invoked.
Such problems are common in migrating to graph execution.
Can result in suspicious numerical results or lower performance.
Tatiana Castro Vélez, Raffi Khatchadourian, Mehdi Bagherzadeh, Anita Raja Challenges in Migrating Imperative DL Programs to Graphs 8 / 32

19. Introduction Motivation Approach Methodology Results Conclusion
Imperative TensorFlow Code Using a Counter
1 class Model(tf.Module):
2 def __init__(self):
3 self.v = tf.Variable(0)
4 self.counter = 0
5
6 @tf.function
7 def __call__(self):
8 if self.counter == 0:
9 self.counter += 1
10 self.v.assign_add(1)
11 return self.v
12 m = Model()
13 for n in range(3):
14 print(m().numpy())
Output (expecting 1, 1, 1):
1
2
3
A model uses a counter to safeguard a variable incrementation.
The initial value of counter (line 4), however, is captured during
tracing upon the first model invocation (line 14).
Variable v is incremented unconditionally (line 10) each time the
model is invoked.
Such problems are common in migrating to graph execution.
Can result in suspicious numerical results or lower performance.
Tatiana Castro Vélez, Raffi Khatchadourian, Mehdi Bagherzadeh, Anita Raja Challenges in Migrating Imperative DL Programs to Graphs 8 / 32

20. Introduction Motivation Approach Methodology Results Conclusion
Imperative TensorFlow Code Using a Counter
1 class Model(tf.Module):
2 def __init__(self):
3 self.v = tf.Variable(0)
4 self.counter = 0
5
6 @tf.function
7 def __call__(self):
8 if self.counter == 0:
9 self.counter += 1
10 self.v.assign_add(1)
11 return self.v
12 m = Model()
13 for n in range(3):
14 print(m().numpy())
Output (expecting 1, 1, 1):
1
2
3
A model uses a counter to safeguard a variable incrementation.
The initial value of counter (line 4), however, is captured during
tracing upon the first model invocation (line 14).
Variable v is incremented unconditionally (line 10) each time the
model is invoked.
Such problems are common in migrating to graph execution.
Can result in suspicious numerical results or lower performance.
Tatiana Castro Vélez, Raffi Khatchadourian, Mehdi Bagherzadeh, Anita Raja Challenges in Migrating Imperative DL Programs to Graphs 8 / 32

21. Introduction Motivation Approach Methodology Results Conclusion
Developer Burden
When migrating imperative DL code to graph execution, developers are
burdened with:
Making their code compatible with the underlying execution model.
Manually specifying which functions that should be converted.
Issues
When and where should @tf.function should be applied?
Calling tf.functions recursively could cause infinite loops.
Even if a recursion seems to work, the tf.function will be traced
multiple times (“retracing”) potentially impacting performance.
Using @tf.function on small computations can be dominated by graph
creation overhead.
Tatiana Castro Vélez, Raffi Khatchadourian, Mehdi Bagherzadeh, Anita Raja Challenges in Migrating Imperative DL Programs to Graphs 9 / 32

22. Introduction Motivation Approach Methodology Results Conclusion
Developer Burden
When migrating imperative DL code to graph execution, developers are
burdened with:
Making their code compatible with the underlying execution model.
Manually specifying which functions that should be converted.
Issues
When and where should @tf.function should be applied?
Calling tf.functions recursively could cause infinite loops.
Even if a recursion seems to work, the tf.function will be traced
multiple times (“retracing”) potentially impacting performance.
Using @tf.function on small computations can be dominated by graph
creation overhead.
Tatiana Castro Vélez, Raffi Khatchadourian, Mehdi Bagherzadeh, Anita Raja Challenges in Migrating Imperative DL Programs to Graphs 9 / 32

23. Introduction Motivation Approach Methodology Results Conclusion
Developer Burden
When migrating imperative DL code to graph execution, developers are
burdened with:
Making their code compatible with the underlying execution model.
Manually specifying which functions that should be converted.
Issues
When and where should @tf.function should be applied?
Calling tf.functions recursively could cause infinite loops.
Even if a recursion seems to work, the tf.function will be traced
multiple times (“retracing”) potentially impacting performance.
Using @tf.function on small computations can be dominated by graph
creation overhead.
Tatiana Castro Vélez, Raffi Khatchadourian, Mehdi Bagherzadeh, Anita Raja Challenges in Migrating Imperative DL Programs to Graphs 9 / 32

24. Introduction Motivation Approach Methodology Results Conclusion
Developer Burden
When migrating imperative DL code to graph execution, developers are
burdened with:
Making their code compatible with the underlying execution model.
Manually specifying which functions that should be converted.
Issues
When and where should @tf.function should be applied?
Calling tf.functions recursively could cause infinite loops.
Even if a recursion seems to work, the tf.function will be traced
multiple times (“retracing”) potentially impacting performance.
Using @tf.function on small computations can be dominated by graph
creation overhead.
Tatiana Castro Vélez, Raffi Khatchadourian, Mehdi Bagherzadeh, Anita Raja Challenges in Migrating Imperative DL Programs to Graphs 9 / 32

25. Introduction Motivation Approach Methodology Results Conclusion
Developer Burden
When migrating imperative DL code to graph execution, developers are
burdened with:
Making their code compatible with the underlying execution model.
Manually specifying which functions that should be converted.
Issues
When and where should @tf.function should be applied?
Calling tf.functions recursively could cause infinite loops.
Even if a recursion seems to work, the tf.function will be traced
multiple times (“retracing”) potentially impacting performance.
Using @tf.function on small computations can be dominated by graph
creation overhead.
Tatiana Castro Vélez, Raffi Khatchadourian, Mehdi Bagherzadeh, Anita Raja Challenges in Migrating Imperative DL Programs to Graphs 9 / 32

26. Introduction Motivation Approach Methodology Results Conclusion
Developer Burden
When migrating imperative DL code to graph execution, developers are
burdened with:
Making their code compatible with the underlying execution model.
Manually specifying which functions that should be converted.
Issues
When and where should @tf.function should be applied?
Calling tf.functions recursively could cause infinite loops.
Even if a recursion seems to work, the tf.function will be traced
multiple times (“retracing”) potentially impacting performance.
Using @tf.function on small computations can be dominated by graph
creation overhead.
Tatiana Castro Vélez, Raffi Khatchadourian, Mehdi Bagherzadeh, Anita Raja Challenges in Migrating Imperative DL Programs to Graphs 9 / 32

27. Introduction Motivation Approach Methodology Results Conclusion
Developer Burden
When migrating imperative DL code to graph execution, developers are
burdened with:
Making their code compatible with the underlying execution model.
Manually specifying which functions that should be converted.
Issues
When and where should @tf.function should be applied?
Calling tf.functions recursively could cause infinite loops.
Even if a recursion seems to work, the tf.function will be traced
multiple times (“retracing”) potentially impacting performance.
Using @tf.function on small computations can be dominated by graph
creation overhead.
Tatiana Castro Vélez, Raffi Khatchadourian, Mehdi Bagherzadeh, Anita Raja Challenges in Migrating Imperative DL Programs to Graphs 9 / 32

28. Introduction Motivation Approach Methodology Results Conclusion
Using Hybridization Parameters
1 model = SequentialModel()
2 res1 = model(tf.constant([1, 2, 3]))
3 res2 = model(tf.constant([1, 2, 3, 4, 5]))
WARNING: 5 out of the last 5
calls triggered
tf.function retracing.
Tracing is expensive.
,
→
,
→
,
→
Listing: DL model client code using varying datasets.
Decorating the correct function but with incorrect decorator
arguments may result in performance degradation.
Retracing helps ensure that the correct graphs are generated for
each set of inputs.
Excessive retracing may cause code to run more slowly had
tf.function not been used.
Invoking a model multiple times using different datasets produces
the warning on the right.
Tatiana Castro Vélez, Raffi Khatchadourian, Mehdi Bagherzadeh, Anita Raja Challenges in Migrating Imperative DL Programs to Graphs 10 / 32

29. Introduction Motivation Approach Methodology Results Conclusion
Using Hybridization Parameters
1 model = SequentialModel()
2 res1 = model(tf.constant([1, 2, 3]))
3 res2 = model(tf.constant([1, 2, 3, 4, 5]))
WARNING: 5 out of the last 5
calls triggered
tf.function retracing.
Tracing is expensive.
,
→
,
→
,
→
Listing: DL model client code using varying datasets.
Decorating the correct function but with incorrect decorator
arguments may result in performance degradation.
Retracing helps ensure that the correct graphs are generated for
each set of inputs.
Excessive retracing may cause code to run more slowly had
tf.function not been used.
Invoking a model multiple times using different datasets produces
the warning on the right.
Tatiana Castro Vélez, Raffi Khatchadourian, Mehdi Bagherzadeh, Anita Raja Challenges in Migrating Imperative DL Programs to Graphs 10 / 32

30. Introduction Motivation Approach Methodology Results Conclusion
Using Hybridization Parameters
1 model = SequentialModel()
2 res1 = model(tf.constant([1, 2, 3]))
3 res2 = model(tf.constant([1, 2, 3, 4, 5]))
WARNING: 5 out of the last 5
calls triggered
tf.function retracing.
Tracing is expensive.
,
→
,
→
,
→
Listing: DL model client code using varying datasets.
Decorating the correct function but with incorrect decorator
arguments may result in performance degradation.
Retracing helps ensure that the correct graphs are generated for
each set of inputs.
Excessive retracing may cause code to run more slowly had
tf.function not been used.
Invoking a model multiple times using different datasets produces
the warning on the right.
Tatiana Castro Vélez, Raffi Khatchadourian, Mehdi Bagherzadeh, Anita Raja Challenges in Migrating Imperative DL Programs to Graphs 10 / 32

31. Introduction Motivation Approach Methodology Results Conclusion
Using Hybridization Parameters
1 model = SequentialModel()
2 res1 = model(tf.constant([1, 2, 3]))
3 res2 = model(tf.constant([1, 2, 3, 4, 5]))
WARNING: 5 out of the last 5
calls triggered
tf.function retracing.
Tracing is expensive.
,
→
,
→
,
→
Listing: DL model client code using varying datasets.
Decorating the correct function but with incorrect decorator
arguments may result in performance degradation.
Retracing helps ensure that the correct graphs are generated for
each set of inputs.
Excessive retracing may cause code to run more slowly had
tf.function not been used.
Invoking a model multiple times using different datasets produces
the warning on the right.
Tatiana Castro Vélez, Raffi Khatchadourian, Mehdi Bagherzadeh, Anita Raja Challenges in Migrating Imperative DL Programs to Graphs 10 / 32

32. Introduction Motivation Approach Methodology Results Conclusion
Limiting Retracing
To fix the problem, specify an input_signature in the model code:
Example
@tf.function(input_signature=(tf.TensorSpec(shape=[None],
dtype=tf.int32),))
,
→
A [None] dimension in the tf.TensorSpec allows for flexibility in
trace (graph) reuse.
Since tensors are matched on their shape, a None wild card allows
tf.functions to reuse traces for variably-sized input.
Can occur when sequences or images are of different lengths or sizes,
respectively.
Since each call no longer produces a trace, the warning disappears
and the performance bottleneck is averted.
Tatiana Castro Vélez, Raffi Khatchadourian, Mehdi Bagherzadeh, Anita Raja Challenges in Migrating Imperative DL Programs to Graphs 11 / 32

33. Introduction Motivation Approach Methodology Results Conclusion
Limiting Retracing
To fix the problem, specify an input_signature in the model code:
Example
@tf.function(input_signature=(tf.TensorSpec(shape=[None],
dtype=tf.int32),))
,
→
A [None] dimension in the tf.TensorSpec allows for flexibility in
trace (graph) reuse.
Since tensors are matched on their shape, a None wild card allows
tf.functions to reuse traces for variably-sized input.
Can occur when sequences or images are of different lengths or sizes,
respectively.
Since each call no longer produces a trace, the warning disappears
and the performance bottleneck is averted.
Tatiana Castro Vélez, Raffi Khatchadourian, Mehdi Bagherzadeh, Anita Raja Challenges in Migrating Imperative DL Programs to Graphs 11 / 32

34. Introduction Motivation Approach Methodology Results Conclusion
Limiting Retracing
To fix the problem, specify an input_signature in the model code:
Example
@tf.function(input_signature=(tf.TensorSpec(shape=[None],
dtype=tf.int32),))
,
→
A [None] dimension in the tf.TensorSpec allows for flexibility in
trace (graph) reuse.
Since tensors are matched on their shape, a None wild card allows
tf.functions to reuse traces for variably-sized input.
Can occur when sequences or images are of different lengths or sizes,
respectively.
Since each call no longer produces a trace, the warning disappears
and the performance bottleneck is averted.
Tatiana Castro Vélez, Raffi Khatchadourian, Mehdi Bagherzadeh, Anita Raja Challenges in Migrating Imperative DL Programs to Graphs 11 / 32

35. Introduction Motivation Approach Methodology Results Conclusion
Limiting Retracing
To fix the problem, specify an input_signature in the model code:
Example
@tf.function(input_signature=(tf.TensorSpec(shape=[None],
dtype=tf.int32),))
,
→
A [None] dimension in the tf.TensorSpec allows for flexibility in
trace (graph) reuse.
Since tensors are matched on their shape, a None wild card allows
tf.functions to reuse traces for variably-sized input.
Can occur when sequences or images are of different lengths or sizes,
respectively.
Since each call no longer produces a trace, the warning disappears
and the performance bottleneck is averted.
Tatiana Castro Vélez, Raffi Khatchadourian, Mehdi Bagherzadeh, Anita Raja Challenges in Migrating Imperative DL Programs to Graphs 11 / 32

36. Introduction Motivation Approach Methodology Results Conclusion
Hybridization Alternatives
Alternatives to hybridization exists, e.g., JANUS [Jeong et al., 2019].
Require custom Python interpreters,
May be impractical for industry.
Support only specific Python constructs.
Must be updated with new language versions for consistency.
Python, for example, has historically underwent major revisions.
Tatiana Castro Vélez, Raffi Khatchadourian, Mehdi Bagherzadeh, Anita Raja Challenges in Migrating Imperative DL Programs to Graphs 12 / 32

37. Introduction Motivation Approach Methodology Results Conclusion
Motivation
Knowledge gap in how hybridization is used in real-world DL
applications.
Challenges in successfully applying hybridization are underexplored.
Without such insight, DL systems may be inefficient, fallible, and
difficult to maintain.
Insight
Advances in DL are likely to be futile if they cannot be effectively used.
Tatiana Castro Vélez, Raffi Khatchadourian, Mehdi Bagherzadeh, Anita Raja Challenges in Migrating Imperative DL Programs to Graphs 13 / 32

38. Introduction Motivation Approach Methodology Results Conclusion
Approach
Empirical study on common development challenges in migrating
imperative DL code to graph execution using hybridization in
open-source DL systems.
Discover bug patterns and corresponding challenges involved in
writing reliable yet performant imperative DL code.
Focus on hybridization in TensorFlow.
Implications
Help drive:
New automated migration techniques.
IDE code completion.
Automated (data science-specific) refactoring mining approaches,
e.g., RefactoringMiner 2.0 [Tsantalis et al., 2020].
Tatiana Castro Vélez, Raffi Khatchadourian, Mehdi Bagherzadeh, Anita Raja Challenges in Migrating Imperative DL Programs to Graphs 14 / 32

39. Introduction Motivation Approach Methodology Results Conclusion
Advance knowledge of this emerging yet pervasive hybrid paradigm.
Provide feedback to language and API designers for future API
versions.
Help tool designers comprehend difficulties with writing performant
imperative DL code.
Propose preliminary recommendations, best practices, and
anti-patterns for practitioners in using hybridization effectively.
Assist educators in teaching hybridization APIs.
Tatiana Castro Vélez, Raffi Khatchadourian, Mehdi Bagherzadeh, Anita Raja Challenges in Migrating Imperative DL Programs to Graphs 15 / 32

40. Introduction Motivation Approach Methodology Results Conclusion
Subject DL Systems
Analyzed occurrences of tf.function in 250 Python DL projects:
19.7 MLOC.
470 manually examined code patches (Git commits).
446 manually examined bug reports (GitHub issues).
Vary widely in domain, application, size, and popularity.
Non-trivial GitHub metrics (stars, forks, collaborators).
Mix of DL libraries, frameworks, and applications.
Used in previous studies or in a DS dataset [Biswas et al., 2019].
Non-trivial portion involving DL.
Include projects from Apache, Apple, Google, and NVIDIA.
Also include lesser-known repositories.
Tatiana Castro Vélez, Raffi Khatchadourian, Mehdi Bagherzadeh, Anita Raja Challenges in Migrating Imperative DL Programs to Graphs 16 / 32

41. Mining for Changesets and GitHub Issues
Mined repositories for:
Git commits whose
changeset contains
tf.function.
GitHub issues mentioning
“tf.function.”
Hybridization relatively new.
tf.function released on
Sep 30, 2019.
Changesets
Used gitcproc [Casalnuovo et al., 2017] to classify Git commit
changesets (patches) representing hybridization bug fixes using NLP.
GitHub Issues
Used standard GitHub search API.
Filtered issues containing irrelevant discussion using a model.

42. Introduction Motivation Approach Methodology Results Conclusion
Manual Examination
Git Commits
Manually examined all commits.
Utilized referenced bug reports
and commit log messages.
GitHub Issues
Randomly selected a subset of
issues to manually examine.
Favored issues containing code
snippets.
Studied changesets and GitHub issues to determine hybridization
bugs and usage challenge categories.
Contacted developers for clarification.
Tatiana Castro Vélez, Raffi Khatchadourian, Mehdi Bagherzadeh, Anita Raja Challenges in Migrating Imperative DL Programs to Graphs 18 / 32

43. Introduction Motivation Approach Methodology Results Conclusion
Table: Studied subjects.
subj KLOC studied periods cmts/iss kws exe
fixes 122 10,879 2015-11-06 to 2021-01-14 199,140 470 470
reports 167 17,378 2012-05-07 to 2021-08-11 237,232 704 446
Total 250*
19,677*
2012-05-07 to 2021-08-11 436,372 1,174 916
*
Represents unique totals due to subject overlap between the study portions.
Tatiana Castro Vélez, Raffi Khatchadourian, Mehdi Bagherzadeh, Anita Raja Challenges in Migrating Imperative DL Programs to Graphs 19 / 32

44. Introduction Motivation Approach Methodology Results Conclusion
Table: Studied subjects.
subj KLOC studied periods cmts/iss kws exe
fixes 122 10,879 2015-11-06 to 2021-01-14 199,140 470 470
reports 167 17,378 2012-05-07 to 2021-08-11 237,232 704 446
Total 250*
19,677*
2012-05-07 to 2021-08-11 436,372 1,174 916
*
Represents unique totals due to subject overlap between the study portions.
Tatiana Castro Vélez, Raffi Khatchadourian, Mehdi Bagherzadeh, Anita Raja Challenges in Migrating Imperative DL Programs to Graphs 19 / 32

45. Introduction Motivation Approach Methodology Results Conclusion
Table: Studied subjects.
subj KLOC studied periods cmts/iss kws exe
fixes 122 10,879 2015-11-06 to 2021-01-14 199,140 470 470
reports 167 17,378 2012-05-07 to 2021-08-11 237,232 704 446
Total 250*
19,677*
2012-05-07 to 2021-08-11 436,372 1,174 916
*
Represents unique totals due to subject overlap between the study portions.
Tatiana Castro Vélez, Raffi Khatchadourian, Mehdi Bagherzadeh, Anita Raja Challenges in Migrating Imperative DL Programs to Graphs 19 / 32

46. Introduction Motivation Approach Methodology Results Conclusion
Quantitative Analysis
Manually examined 916 commits/GitHub issues.
Found 280 hybridization challenges (bug fixes/reported issues).
Devised a challenge category hierarchy—in part—by using
TensorFlow documentation.
Tatiana Castro Vélez, Raffi Khatchadourian, Mehdi Bagherzadeh, Anita Raja Challenges in Migrating Imperative DL Programs to Graphs 20 / 32

47. Figure: Discovered problem categories (hierarchical).

48. Introduction Motivation Approach Methodology Results Conclusion
Table: Discovered top-level problem categories.
problem abbr cmts iss total
Performance PRF 74 37 111
API misuse APM 23 30 53
Incompatibility INC 16 33 49
TensorFlow bug TFB 4 18 22
Other OTH 14 2 16
Unknown UKN 10 0 10
Test TST 8 0 8
Debuggability DBG 4 2 6
Exposed variable state EVS 1 1 2
Compilation error CMP 1 0 1
Numerical errors NME 1 0 1
Segmentation fault SEG 1 0 1
Total 157 123 280
Tatiana Castro Vélez, Raffi Khatchadourian, Mehdi Bagherzadeh, Anita Raja Challenges in Migrating Imperative DL Programs to Graphs 22 / 32

49. Introduction Motivation Approach Methodology Results Conclusion
Findings Summary I
Hybridization
1 Is prone to API misuse.
2 Can result in performance degradation—the opposite of its intention.
3 Has limited application due to execution mode incompatibility.
At 39.64% (111/280), performance problems was the largest category
involving tf.function usage.
Despite its intent to improve code performance, in 7.21% (8/111) of
cases, tf.function caused performance degradation.
Only 54.95% (61/111) of imperative DL code performance problems
were fixed by adding @tf.function.
The remaining 45.05% were due to existing usages of tf.function.
25.23% of performance fixes involve altering developer-supplied
arguments to tf.function(...).
18.92% of performance problems involve incorrect input tensor
shape specifications.
Tatiana Castro Vélez, Raffi Khatchadourian, Mehdi Bagherzadeh, Anita Raja Challenges in Migrating Imperative DL Programs to Graphs 23 / 32

50. Introduction Motivation Approach Methodology Results Conclusion
Findings Summary II
At 18.93%, API misuse—using tf.function inconsistently w.r.t. its
documentation—was the second largest problem category.
37.74% (20/53) of API misuse was caused by developers not
understanding how to use hybridization APIs correctly.
To fix API misuse, tf.function was removed in 28.30% of cases.
In 46.67% of these cases, developers abandoned hybridization due to
API confusion, of which 62.50% resulted in run-time errors.
Execution mode incompatibility—at 17.50% (49/280)—was the third
largest problem category.
Developers struggle to seamlessly use similar constructs between
different modes.
81.63% of incompatibility problems lead to run-time errors or
unexpected results.
Do not surface until after running the code.
TensorFlow bugs made up 7.86% of discovered problems.
Developers were offered a workaround or waited for a new framework
version.
9.09% of these involve deadlocks.
Tatiana Castro Vélez, Raffi Khatchadourian, Mehdi Bagherzadeh, Anita Raja Challenges in Migrating Imperative DL Programs to Graphs 24 / 32

51. Qualitative Analysis I
Performance
1 + @tf.function
2 def pm(linear):
3 state = lpt_init(linear, a0=0.1, order=1)
4 final_state = nbody(state, stages, nc)
5 tfinal_field = cic_paint(tf.zeros_like(linear), final_state[0])
6 return tfinal_field
pm() is decorated with @tf.function (line 1).
Using tf.function “ensure[s] that the graph for [a] function is
compiled once and not every time it is called, thus gaining in speed
and performance” [Modi, 2021].
Best Practice
Favor @tf.function on Python functions containing imperative,
otherwise eagerly-executed, DL code to improve performance.

52. Qualitative Analysis II
https://github.com/tensorflow/addons/pull/2264
Developers struggle to enhance performance using hybridization.
“. . . it does far too much hidden magic . . .” [Roberts, 2020].
Retracing can cause worse performance than not using hybridization.
One problem is related to hybridizing inner functions (above).
The fix involved moving @tf.function to the top-level function,
making it run ∼25%–40% faster.
The root cause was that nested functions are uncachable.
Function nesting is a common Python modularity mechanism!
Anti-Pattern
Hybridizing nested functions may cause performance degradation.
Sacrifice modularity by hybridizing top-level function or refactoring to top.

53. Qualitative Analysis III
Input Signatures
1 - @tf.function
2 + @tf.function(input_signature=[
3 + tf.TensorSpec(shape=(None, self.num_states), dtype=tf.float32),
4 + tf.TensorSpec(shape=(None, self.num_actions), dtype=tf.float32),
5 + tf.TensorSpec(shape=(None, 1), dtype=tf.float32),
6 + tf.TensorSpec(shape=(None, self.num_states), dtype=tf.float32),])
7 def update_weights(s, a, r, sn): # ...
Arguments to tf.function(), particularly involving input tensor
shapes, may also influence performance.
An underspecified input signature—one of the most used
tf.function parameters that we observed.
On lines 2–6, a performance regression was fixed by adding an
input_signature to a weight distribution tf.function to “make
sure it does not recreate graph, which will slow down training
significantly” [Koesnadi, 2021].
The sequence of tf.TensorSpecs specifies the intended tensor shapes
and data types (dtypes) that will be supplied to update_weights().

54. Qualitative Analysis IV
Otherwise, a separate (concrete) function (graph) is instantiated for
each inferred input signature, which—depending on context—may
result in retracing.
Best Practice
If possible, supply an input_signature argument to tf.function with
the intended shape and types of any input tensors to avert retracing—a
practice similar to that of providing type annotations to variables in
dynamic languages to assist with type inferencing.

55. Introduction Motivation Approach Methodology Results Conclusion
More Preliminary Best Practices
When an operation is deemed incompatible with hybridization, check
the documentation to see if additional steps are required to make
the imperative DL code more amenable to graph conversion.
Framework limitations may impede performance enhancements.
Check for potential workarounds of (unresolved) TensorFlow bugs.
Use tf.config.run_functions_eagerly(True) to temporarily
disable tf.function to facilitate debugging.
More details in the paper! http://arxiv.org/abs/2201.09953
Tatiana Castro Vélez, Raffi Khatchadourian, Mehdi Bagherzadeh, Anita Raja Challenges in Migrating Imperative DL Programs to Graphs 29 / 32

56. Introduction Motivation Approach Methodology Results Conclusion
Conclusion
Studied hybridization bugs and challenges in migrating imperative
DL code to graph execution.
1 Despite the purpose of hybridization, care must be taken to avoid
performance degradation.
2 Not all imperative DL code is amenable to graph conversion.
3 Must be cognizant of which code is running in graph mode and
which is not.
Future Work
Analyze alternative developer resources, e.g., Stack Overflow.
Develop automated refactoring approaches.
Integrate our results into automated refactoring detection
techniques.
Tatiana Castro Vélez, Raffi Khatchadourian, Mehdi Bagherzadeh, Anita Raja Challenges in Migrating Imperative DL Programs to Graphs 30 / 32

57. Introduction Motivation Approach Methodology Results Conclusion
For Further Reading I
Abadi, Martı́n et al. (2016). “TensorFlow: A System for Large-Scale Machine Learning”. In: Symposium on Operating Systems
Design and Implementation.
Agrawal, Akshay et al. (2019). TensorFlow Eager: A Multi-Stage, Python-Embedded DSL for Machine Learning. arXiv:
1903.01855 [cs.PL].
Apache (Apr. 8, 2021). Hybridize. Apache MXNet documentation. url:
https://mxnet.apache.org/versions/1.8.0/api/python/docs/tutorials/packages/gluon/blocks/hybridize.html (visited
on 04/08/2021).
Arpteg, A., B. Brinne, L. Crnkovic-Friis, and J. Bosch (2018). “Software Engineering Challenges of Deep Learning”. In: Euromicro
Conference on Software Engineering and Advanced Applications. IEEE, pp. 50–59. doi: 10.1109/SEAA.2018.00018.
Biswas, S., M. J. Islam, Y. Huang, and H. Rajan (2019). “Boa Meets Python: A Boa Dataset of Data Science Software in Python
Language”. In: Mining Software Repositories, pp. 577–581. doi: 10.1109/MSR.2019.00086.
Casalnuovo, Casey, Yagnik Suchak, Baishakhi Ray, and Cindy Rubio-González (2017). “GitcProc: A tool for processing and
classifying GitHub commits”. In: International Symposium on Software Testing and Analysis. ISSTA ’17. ACM, pp. 396–399. doi:
10.1145/3092703.3098230.
Chen, Tianqi, Mu Li, Yutian Li, Min Lin, Naiyan Wang, Minjie Wang, Tianjun Xiao, Bing Xu, Chiyuan Zhang, and Zheng Zhang
(2015). “MXNet: A Flexible and Efficient Machine Learning Library for Heterogeneous Distributed Systems”. In: Workshop on
Machine Learning Systems at NIPS. arXiv: 1512.01274 [cs.DC].
Chollet, François (2020). Deep Learning with Python. 2nd ed. Manning.
Facebook Inc. (2019). PyTorch Documentation. TorchScript. en. url: https://pytorch.org/docs/stable/jit.html (visited on
02/19/2021).
Tatiana Castro Vélez, Raffi Khatchadourian, Mehdi Bagherzadeh, Anita Raja Challenges in Migrating Imperative DL Programs to Graphs 31 / 32

58. Introduction Motivation Approach Methodology Results Conclusion
For Further Reading II
Jeong, Eunji, Sungwoo Cho, Gyeong-In Yu, Joo Seong Jeong, Dong-Jin Shin, Taebum Kim, and Byung-Gon Chun (July 2019).
“Speculative Symbolic Graph Execution of Imperative Deep Learning Programs”. In: SIGOPS Oper. Syst. Rev. 53.1, pp. 26–33.
issn: 0163-5980. doi: 10.1145/3352020.3352025.
Koesnadi, Samuel Matthew (Feb. 26, 2021). Fixed all . . . this should work. samuelmat19/DDPG-tf2. 02a3f29. ML6. url:
https://github.com/samuelmat19/DDPG-tf2/commit/02a3f297#r47584455 (visited on 01/12/2022).
Modi, Chirag (Apr. 16, 2021). bug boxsize=nc. modichirag/galference. af1664e. UC Berkeley. url: https://git.io/J9ciM
(visited on 01/10/2022).
Moldovan, Dan, James M. Decker, Fei Wang, Andrew A. Johnson, Brian K. Lee, Zachary Nado, D. Sculley, Tiark Rompf, and
Alexander B. Wiltschko (2019). AutoGraph: Imperative-style Coding with Graph-based Performance. arXiv: 1810.08061 [cs.PL].
Paszke, Adam et al. (Dec. 3, 2019). PyTorch: An Imperative Style, High-Performance Deep Learning Library. arXiv: 1912.01703
[cs.LG].
Roberts, Chase (May 26, 2020). Added jitted ncon. Pull Request #623. google/TensorNetwork. Xanadu. url:
https://git.io/J9cMx (visited on 01/10/2022).
Tsantalis, Nikolaos, Ameya Ketkar, and Danny Dig (2020). “RefactoringMiner 2.0”. In: IEEE Trans. Softw. Eng. doi:
10.1109/TSE.2020.3007722.
Zhou, Weijie, Yue Zhao, Guoqiang Zhang, and Xipeng Shen (2020). “HARP: Holistic Analysis for Refactoring Python-Based
Analytics Programs”. In: International Conference on Software Engineering, pp. 506–517. doi: 10.1145/3377811.3380434.
Tatiana Castro Vélez, Raffi Khatchadourian, Mehdi Bagherzadeh, Anita Raja Challenges in Migrating Imperative DL Programs to Graphs 32 / 32