A high-level overview of the core idea behind movement pruning.

1. Movement Pruning
Explain Like I’m Five - Victor SANH
@SanhEstPasMoi

2. John is at the bank to open a new account.
To celebrate her new client, the bank officer wants to offer
John a welcome gift. In fact, it is the bank’s policy to offer
every new client a stock of their choice among a selection of
market stocks.

3. This month, there are 3 choices on the table: stock A, stock B,
and stock C.
John is really excited and is extremely pleased to see the bank
live up to its reputation. He now must choose a stock for his
welcome package.
Current market
value: $200
Current market
value: $75
Current market
value: $200

4. If you were John, which stock would you choose?

5. John wants to take stock A or C. They have the highest current
market value.
But John doesn’t need cash right away. He has a preference for
the future and sees this stock as an investment opportunity.
Current market
value: $200
Current market
value: $75
Current market
value: $200

6. He tells the bank officer: “I am missing some data to make an
informed decision: what is the average year-to-year return of
these stocks?”

7. The bank officer replies: “That is an excellent question John!
Here is the information”
Current market
value: $200
Y2Y return: +1%
Current market
value: $75
Y2Y return: +30%
Current market
value: $200
Y2Y return: -5%

8. John feels he has all the information he needs to make an
informed decision.
If you were John, which stock would you choose?

9. Weight pruning is analogous to choosing a stock for a
welcome package: pruning consists in removing weights or
connections in the network by selecting and keeping the most
important ones. Magnitude pruning and movement pruning
give different definitions of importance.
John’s first instinct was to choose stock A or C. It is a
reasonable choice: stock A and C have the current highest
market values.
That is what magnitude pruning does: it selects the weights
with the highest absolute value.

10. However, stocks are moving objects. Everyday, they are
traded on the market and the stock values go up or down.
Stock C seems to be a really bad investment decision: even if it
has a high market value, the Y2Y return is negative and John
would be losing money.
If the stocks continue to perform similarly, stock B would be
double the price of stock A in less than 7 years. Stock B has the
highest positive quantity of movement.
That’s what movement pruning does: it selects the weights
that are moving the most away from 0.

11. In the context of transfer learning, pretraining can be
understood as the history of these stocks. After pretraining,
the stocks reach their current market values, and that is when
John has to decide which stock to select (and “prune” the rest).
Fine-tuning is analogous to John’s period of investment: he
chooses the stock that maximizes its movement away from 0
during fine-tuning.

12. Paper: https://arxiv.org/abs/2005.07683