選択的状態空間モデル（Selective SSM）を用いることで、必要最小限の情報に着目し計算効率の向上を達成するMambaについて紹介しています

1. Mamba: A new era or
ephemeral?
Mamba: Linear-Time Sequence Modeling with Selective State
Spaces

2. Transformer cannot
Drawbacks of transformers are finite window, quadratic scaling with respect to the
window length
O(L^2)
Generating tokens for a sequence of length L needs roughly L² computations which can be costly if the
sequence length increases.

3. RNN: recurrent neural network
Fast inference, linear complexity
Drawbacks
- small memory
- not parallelizable

4. SSM: state space model
Input x
output y
hidden state h
Looks
familiar 🧐

5. Mamba how to solve small memory problem
Matrix A responsible for updating hidden state H, how to update memory along
sequence continues?
hippo as High-order Polynomial Projection:

6. Mamba: A [NxD] matrix definition
Input shape: [2, 64, 16]
A: [32, 16]

7. SSM: parallel training
What hinders RNN is that its nonlinearity tanh() cannot be fastly trained.
From definition, SSM kind of like a RNN without tanh()

8. SSM: parallel training
In a way, it’s like a convolution operation

9. Mamba: selective SSM
Not Content-awareness: independent A, B, C results in problems with
content-awareness.
In comparison, these tasks are relatively easy for Transformers since they
dynamically change their attention based on the input sequence. They can
selectively “look” or “attend” at different parts of the sequence.

10. Mamba: selective SSM
Mamba makes matrics B, C, and step size Δ depend on the input, which is similar
to transformer
This raised another issue -> cannot train parallelly like in S4

11. parallel scan?
As long as algorithms satisfy the associative prosperity, can be parallelized!
t is the parallel thread
feels like
DP

12. hardware-aware algorithm
Skip

13. Results and thoughts
- useful
- Can it replace transformers?