Insight Data Engineering Project

1. TwitterStock
Yuhan Niu
Insight Data Engineering Fellow

2. Pipeline

3. A N
M
Z
Y
N
T
W
T
L
P
A
T
R
C
TWTR
NYT
AMZNAAPL
AMC
Trie

4. A N
M
Z
Y
N
T
W
T
L
P
A
T
R
C
TWTR
NYT
AMZNAAPL
AMC
Stockman: $AMC still shows no sign of pulling back!Trie

5. A N
M
Z
Y
N
T
W
T
L
P
A
T
R
C
TWTR
NYT
AMZNAAPL
AMC
Stockman: $AMC still shows no sign of pulling back!Trie

6. N
M
Z
Y
N
T
W
T
L
P
A
T
R
C
TWTR
NYT
AMZNAAPL
AMC
Stockman: $AMC still shows no sign of pulling back!Trie

7. N
Z
Y
N
T
W
T
L
P
A
T
R
C
TWTR
NYT
AMZNAAPL
AMC
Stockman: $AMC still shows no sign of pulling back!Trie

8. N
Z
Y
N
T
W
T
L
P
A
T
R
TWTR
NYT
AMZNAAPL
AMC
Trie Stockman: $AMC still shows no sign of pulling back!

9. N
Z
Y
N
T
W
T
L
P
A
T
R
TWTR
NYT
AMZNAAPL
AMC
Stockman: $AMC still shows no sign of pulling back!Trie

10. Broadcast

11. Broadcast

12. Time Window
Window length = 1 min
Sliding interval = 2 secBatch interval = 2 sec

13. Time Window
Window length = 1 min
Batch interval = 2 sec

14. Time Window
Window length = 1 min
Batch interval = 2 sec

15. Trade-off
Information loss => Accuracy

16. Trade-off
Information loss => Accuracy
Capacity
35,000 tweets per second

17. About Me
Yuhan Niu
• M.S. Mechanical Engineering, UCLA
• Like traveling, basketball and movies

19. Comparison
Hashset
Trie
Input rate: 29021 records/sec
Batch interval: 2 seconds