eventnet asynchronous recursive event processing の説明資料 研究室　勉強会資料

1. /18
EventNet: Asynchronous Recursive Event Processing
Study 2020/09/18
2011136 Ryuta Shitomi
1
Yusuke Sekikawa, Kosuke Hara, and Hideo Saito, CVPR2019

2. /18
The event camera
2
 High dynamic range
 Do not suffer from motion blur
 No latency
 Recently, frame-based approach have achieved impressive results in scenarios.

3. /18
The event data from event-based camera
3
 When the new event occurred:
𝑒𝑖 = 𝑥𝑖, 𝑦𝑖, 𝑝𝑖, 𝑡 − 𝑡′
𝑡′
: time of the last event
 Asynchronous
 Sparse
 Many events (1 mega event per sec)
↓
Need a new learning approach to differ from frame-based
approach

4. /18
Frame-based approach for an event streams
4
 Integration 𝛥𝑡 to create a frame and feed a neural
net.
 Disadvantages:
Cannot use of the sparseness of the data
Redundant computation
A. I. Maqueda, A. Loquercio, G. Gallego, N. N. Garcia, and D. Scaramuzza. Event-based vision meets deep
learning on steering prediction for self-driving cars. CoRR, abs/1804.01310, 2018.
𝜟𝒕

5. /18
Purpose and Contribution
5
 Develop a novel neural network:
can process an extremely high-rate and variable length.
for real-time processing of an asynchronous event stream.
to output recursively using temporal information.
𝜏′
𝜏
𝑡
𝑝𝑜𝑙𝑎𝑟𝑖𝑡𝑦

6. /18
Problem Statement
6
 Consider a sequence of events within a 𝜏 ms interval based on the latest time stamp 𝑡𝑗 of
the event:
𝒆𝑗 = 𝑒𝑖 𝑖 = 𝑗 − 𝑛 𝑗 + 1, … , 𝑗}
𝑒𝑖 = 𝑥𝑖, 𝑦𝑖, 𝑝𝑖, Δ𝑡𝑗,𝑖 Δ𝑡𝑗,𝑖 = 𝑡𝑗 − 𝑡𝑖
 Consider a neural network 𝑓 to estimate 𝑦𝑗 given event stream 𝒆𝒋
𝑦𝑗 = 𝑓 𝒆𝑗 𝑦𝑗: Task dependent
𝜏
𝑡
𝒆𝒋
𝜏
𝒆𝒋+𝟏

7. /18
Problem Statement
7
 To realize a trainable neural network 𝑓 that satisfies following conditions:
1. End-to-End Trainable
2. Event-Wise Processing
3. Efficient Recursive Processing
4. Local Permutation Invariance
𝜏
𝑡
𝒆𝒋
𝜏
𝒆𝒋+𝟏

8. /18
Using PointNet architecture
8
 PointNet solves the problem by approximating the function 𝑓:
𝑦𝑗 = 𝑓 𝒆𝑗 = 𝑔 max ℎ 𝑒𝑗−𝑛 𝑗 +1 , … , ℎ 𝑒𝑗
where ℎ: ℝ4
→ ℝ 𝐾
, 𝑔: ℝ 𝐾
→ ℝ
 Because of the max(∙) , the permutation of events does not change the output 𝑦𝑗.
MLP
 Problem:
A huge amount of computation of ℎ ∙ and max ∙ .
Should run on the event rate. (could be more than 1MEPS)
When a new j + 1 -th event arrives, Δtj,i = tj − ti changes to tj+1 − ti .
Many previous event has already been processed by h(∙).

9. /18
EventNet
9
 Introduce the ”Temporal Coding” and “Recursive Processing” to overcome the difficulty.
The Δ𝑡 only changes as the new event is being received.
Split the ℎ(𝑒):
ℎ 𝑒𝑖 = 𝑐(ℎ 𝑒𝑖
−
, Δ𝑡𝑗,𝑖), 𝑒−
: (𝑥, 𝑦, 𝑝)
𝑐 ∙ :temporal coding function, ℎ: ℝ3
→ ℂ 𝐾
𝑦𝑗 become:
𝑓 𝒆𝑗 = 𝑔 max 𝑐 𝑧𝑗−𝑛 𝑗 +1, Δ𝑡𝑗,𝑗−𝑛 𝑗 +1 , … , 𝑐 𝑧𝑗, 0
where 𝑧𝑖 = ℎ 𝑒−
∈ ℂ 𝐾
𝑐 ∙ , max ∙ need to be computed for all events.
Temporal Coding

10. /18
EventNet
10
• Assume the norm of each element of 𝑧𝑖 is less than 1 (by tanh).
• max ∙ , 𝑐 ∙ need to be recursive.
Example)
𝑠𝑗 = max 𝑠𝑗−1, 𝑐 𝑧𝑗, 0
𝑠𝑗+1 = max 𝑠𝑗, 𝑐 𝑧𝑗+1, 0
• Propose temporal coding function to satisfy this condition:
𝑎𝑗,𝑖 = 𝑐 𝑧𝑖, Δ𝑡𝑗,𝑖 = 𝑧𝑖 −
Δ𝑡𝑗,𝑖
𝜏
+
exp −i
2𝜋Δ𝑡𝑗,𝑖
𝜏
Recursive Processing
Encode the elapsed time

11. /18
Temporal Coding Function
11
𝑎𝑗,𝑖 = 𝑐 𝑧𝑖, Δ𝑡𝑗,𝑖 = 𝑧𝑖 −
Δ𝑡𝑗,𝑖
𝜏
+
exp −i
2𝜋Δ𝑡𝑗,𝑖
𝜏
When the new event is received:
𝑎𝑗+1,𝑖 = 𝑐 𝑧𝑖, Δ𝑡𝑗+1,𝑖 = 𝑧𝑖 −
Δ𝑡𝑗,𝑖
𝜏
−
𝑡𝑗+1 − 𝑡𝑗
𝜏
+
exp −i
2𝜋(Δ𝑡𝑗,𝑖 − 𝑡𝑗+1 + 𝑡𝑗
𝜏
Recursive Processing
𝑡
|𝑎 𝑗,𝑖|
𝑒𝑗
New event is received
𝑡
|𝑎 𝑗,𝑖|
𝑒𝑗 𝑒𝑗+1
− 𝑡𝑗+1 − 𝑡𝑗

12. /18
Temporal Coding Function
12
𝑡
|𝑎 𝑗,𝑖|
𝑒𝑗
New event is received
𝑡
|𝑎 𝑗+1,𝑖|
𝑒𝑗 𝑒𝑗+1
− 𝑡𝑗+1 − 𝑡𝑗
𝑎𝑗,𝑗−2 = max 𝑎𝑗,𝑗−5, … , 𝑎𝑗,𝑗
 when you new event is received.
𝑎𝑗+1,𝑗−2 = max 𝑐 𝑎𝑗,𝑗−2, 𝑡𝑗+1 − 𝑡𝑗 , 𝑎𝑗+1,𝑗+1
 Therefore, 𝑦𝑗+1:
𝑦𝑗+1 = 𝑓 𝒆𝒋+𝟏 = 𝑔 max 𝑐 𝑠𝑗, 𝑡𝑗+1 − 𝑡𝑗 , ℎ 𝑒𝑗+1
−
𝑒𝑗−2
𝑒𝑗−5
|𝑎 𝑗,𝑗−2|

13. /18
Overview of EventNet
13
 First event is arrived:
𝒆0 = {𝑒0}
𝑧0 = ℎ(𝑒0
−
)
𝑦0 = 𝑓 𝒆0 = 𝑔 max 𝑐 𝑧0, 0
𝑡
𝑐(𝑧0, 0)
𝑗 = 0
𝑗 = 1
 Next event is arrived:
𝒆1 = {𝑒0, 𝑒1}
𝑠1 = max(𝑐 𝑧0, 𝑡1 − 𝑡0 , ℎ(𝑒1
−
)
𝑦1 = 𝑓 𝒆1 = 𝑔(𝑠1)
 Next event is arrived:
𝒆2 = {𝑒0, 𝑒1, 𝑒2}
𝑠2 = max 𝑐 𝑠1, 𝑡2 − 𝑡1 , ℎ 𝑒2
−
𝑦3 = 𝑔 𝑠2
𝑗 = 2

14. /18
EventNet Architecture
14
 Create Look-up table(LUT) for decreasing inference time.
(There are only 𝑊 × 𝐻 × 2 patterns in inputs 𝑒−
)
ℎ(∙)
𝑔(∙)

15. /18
Experiments
15
 ETHTED+ dataset for target motion estimation and semantic segmentation
 MVSEC dataset for ego-motion estimation

16. /18
Results for target motion estimation
16

17. /18
Results for ego-motion
17
 Estimated ego motions are shown as yellow.
 Estimated at the rate of 1000 Hz.

18. /18
Compare the computation time [μs]
18
 Using newly received event, PointNet need to process the all events again.