Demo Video www.larry-lai.com/trp.html Tennis Real Play (TRP) is an interactive tennis game system constructed with models extracted from videos of real matches. The key techniques proposed for TRP include player modeling and video-based player/court rendering. For player model creation, we propose a database normalization process and a behavioral transition model of tennis players, which might be a good alternative for motion capture in the conventional video games. For player/court rendering, we propose a framework for rendering vivid game characters and providing the real-time ability. We can say that image-based rendering leads to a more interactive and realistic rendering. Experiments show that video games with vivid viewing effects and characteristic players can be generated from match videos without much user intervention. Because the player model can adequately record the ability and condition of a player in the real world, it can then be used to roughly predict the results of real tennis matches in the next days. The results of a user study reveal that subjects like the increased interaction, immersive experience, and enjoyment from playing TRP.

1. 1
Tennis Real Play:
an Interactive Tennis Game with Models
from Real Videos
Jui-Hsin Lai (Larry), Chieh-Li Chen, Po-Chen Wu,
Chieh-Chi Kao, and Shao-Yi Chien
Demo Video www.larry-lai.com/trp.html

2. 2
Motivation
Play the Video
try to control the player in the video
play video games after watching a match
a more immersive way to enjoy sports video
a versatile function for multimedia

3. 3
Challenge
The first work to construct game from match videos
Challenge 2: Player Rendering
in response to user’s control
incomplete database
Challenge 3: Game System
integrate background/foreground rendering,
interactive user dialogue
real-time processing
Challenge 1: Player Database
collect the database from video
hitting postures, player movements, statistics

4. 4
Outline
Tennis Real Pla
y
Challenge 1: Player Databas
e
Challenge 2: Player Renderin
g
Challenge 3: Game Syste
m
Experimental Result
s
Conclusion

5. Trimap Generation
5
Original video Reconstructed video
Foreground Segmentation
Background video is generated and used for foreground
segmentation
[Lai 11] J.-H. Lai, et al., “Tennis Video 2.0: A New Presentation of Sports Videos with
Content Separation and Rendering,” JVCI 2011.
Matting Result

6. 6
Segmented Player Clips
The sizes of segmented players are variant due to
player’s position on the court
camera focal length
Dif
fi
cult to render the player with such disarray

7. 7
Database Normalization
Position in video Position on
fiducial coordinate
How to normalize the player’s size
normalize the height of player mask?
difficult by mask analysis from player leaning forward
Propose fiducial coordinate
Z(Px, Py) =
@Gx(Px, Py)
@x
@Gy(Px, Py)
@y
.
Zooming Factor
Py
0
= Gy(Px, Py) =
m3Px + m4Py + m5
m6Px + m7Py + 1
,
Px
0
= Gx(Px, Py) =
m0Px + m1Py + m2
m6Px + m7Py + 1
,
Projection Position

8. 8
Motion
Hit
Serve
Standby
Behavior Model
Plays an important role in the whole system
not only classify the database but also model the player
behavior
Serve => Standby => Motion => Standby => Hit
various hitting postures
motion paths running in different directions
Player clips are classified into four categories

9. 9
Outline
Tennis Real Pla
y
Challenge 1: Player Databas
e
Challenge 2: Player Renderin
g
Challenge 3: Game Syste
m
Experimental Result
s
Conclusion

10. 10
Motion-Clip Selection
Several clips form longer path
shorten the distance to B
running along the path of AB connection
minimize number of cascading clips
B
A
A1
rendered character reflects player’s motion in the match
videos
Moving characteristics depend on database
B
A
A1
A2

11. 11
...
Current clip
Clips in Hit category
Similarity
Computation
Hit1,1
Hitn,1
HD1
HDn
CFt
CFt-1
Selection criteria
shape similarity
texture similarity
Hit-Clip Selection
0 degree
45
-45
Forehand
Strength
Statistics as Hitting Properties
0 degree
45
-45
Backhand
Strength
hitting preference of the rendered player reflects the
player’s performance in match videos
Hitting strength of rendering refers to statistics

12. 12
Smoothing Transition (1/2)
Player Clip 1 Player Clip 2
Not easy to find the perfect connection
Need to smooth the cascade
Smooth the shape -- view morphing
label the matching points
calculate the warping mesh
needs to manually
label the matching
points
[Seitz 96] S. M. Seitz and C. R. Dyer, “View morphing,” SIGGRAPH 1996.

13. 13
Smoothing Transition (2/2)
Improved with labeling automatically
feature detection and matching pairs
modify the cost function with the feature term
W =
X
i
X
j
|I1(i, j) ˜
I2(i, j)|2
+
n
X
k=1
|P1(k) P̃2(k)|2
, (1)
Feature Term
Clip A Clip B
Insert the
transition frames

14. 14
Outline
Tennis Real Pla
y
Challenge 1: Player Databas
e
Challenge 2: Player Renderin
g
Challenge 3: Game Syste
m
Experimental Result
s
Conclusion

15. [Lai 11] J.-H. Lai, et al., “Tennis Video 2.0: A New Presentation of Sports Videos with Content
Separation and Rendering,” JVCI 2011.
15
Construct the 3D scene model from 2D image
Background Rendering
[Horry 97] Y. Horry, et al., “Tour into the picture: Using a spidery mesh interface to make
animation from a single image,” SIGGRAPH 97.

16. 16
(1)
Bottom Audience
(2)
(3)
Left Audience
(4)
(5)
Right Audience
(6)
(7)
Player B
Ball
Net
Ref.
B.B.
Player A
x
y
[ R | t ]
(x0, y0)
f0
Game System
x
y
[ R | t ]
(x0, y0)
f0
Left Eye
x
y
[ R | t ]
(x0, y0)
f0
Right Eye
Integrate with foreground objects, scenes and game model

17. 17
Outline
Tennis Real Pla
y
Challenge 1: Player Databas
e
Challenge 2: Player Renderin
g
Challenge 3: Game Syste
m
Experimental Result
s
Conclusion

18. 18
Rendering Results

19. 19
Computation Analysis
Real-time Computation
Intel i7 2.6 GHz CPU with 4GB RAM
Rendering resolution: 720 x 480
Frame rate: 30 fps
Item Computation Units
Clip Selection 45
Smooth Transition 160 ⇠ 270
Background Rendering 100
Foreground Rendering 20 ⇠ 90
Others 40

20. 20
Prediction of Game Result
Game prediction with player statistics
players controlled by the computer
a random value added to each hitting direction/strength
simulation results: 55% v.s. 51% and 61% v.s. 54%
not perfectly match the results, but predict the winner
correctly
might be used for prediction of match results in the future
Game Video 2009 French Open S.-F. 2009 Wimbledon Open S.-F.
Name of Player A Roger Federer Serena Williams
Name of Player B Juan Martin del Potro Elena Dementieva
Game Points A-B 3-6, 7-6, 2-6, 6-1, 6-4 6-7, 7-5, 8-6
Video(%) 51 : 49 54 : 46
Simulation(%) 55 : 45 61 : 39

21. 21
Subjective Evaluation (1/3)
Evaluators, 20 undergraduates, give the score(1 to 5) of
satisfaction, i.e., 1, Very unsatisfied; 5, Very satisfied.
Q.1 Interaction.
Q.2 Immersive experience.
Q.3 Interesting.
Q.4 Innovative application.
Q.5 Willing to play TRP after watching videos?
Full statement of Q1: Did you
feel you have interacted with
video content when playing
TRP?

22. 22
Q.6 Entertainment levels.
Q.7 Realism of visual effects.
Q.8 Interactiveness.
Q.9 Preference.
Next, evaluators were required to play tennis games in
Wii Sports and Top Spin 3(TP3) on PS3.
Subjective Evaluation (2/3)
Wii is the comparison basis, give the score(1 to 5) for
TRP and TP3, i.e., 1, Much worse; 5, Much better.
Top Spin 3
Wii Sports

23. 23
Subjective Evaluation (3/3)
0
1
2
3
4
5
Q.1 Q.2 Q.3 Q.4 Q.5 Q.6
increase of interaction and
immersive experience
have enjoyment and willing to
play TRP
Subjects identify with
TRP compared to Wii
more vivid visual effects
realistic player properties
TRP compared to TP3
lower in terms of entertainment,
visual effects, and preference
without dozens of individuals to
draw texture and build game system
0
1
2
3
4
5
Q.1 Q.2 Q.3 Q.4 Q.5 Q.6
0
1
2
3
4
5
Q.1 Q.2 Q.3 Q.4 Q.5 Q.6 Q.7 Q.8 Q.9
TRP TP3 Wii

24. 24
Demonstration in Public
2010 Taipei International Invention Show & Technomart (Taiwan)
2010 Taichung Information Technology Month (Taiwan)
2010 Kaohsiung Information Technology Month (Taiwan)
Most users would ask after playing
“How can I get this function for my TV?”

25. 25
Outline
Tennis Real Pla
y
Challenge 1: Player Databas
e
Challenge 2: Player Renderin
g
Challenge 3: Game Syste
m
Experimental Result
s
Conclusion

26. 26
Conclusion (1/3)
The proposed methods
Player Rendering
algorithms of Motion-Clip and Hit-Clip selection
an automatic smoothing transition
Game System
integration of player/scene rendering
real-time operation
Player Database
fiducial coordinate for normalization
4-state transition model

27. 27
Conclusion (2/3)
Contribution
1. the first work to integrate video-based rendering and
interactive sports game running in real-time
2. without motion capture system, construct game from
video without much human assistance
3. game results can reflect the match results in real world
& predict match results in the following days
4. a novel way to enjoy sports video & a more immersive
experience

28. 28
Conclusion (3/3)
Possible Extensions
methods with better smoothing transition but need to be
accelerated
limitation of viewing angle improved by multi-view
cameras
tennis video is an example
Football Real Play
Baseball Real Play

29. 29
Thanks for Your Attention
More Information
http://media.ee.ntu.edu.tw/larry/trp

30. 30
Moving Model of Ball
X
Z
Y
xt = Ftxt 1 + Htut + Ctut + wt,
xt : State of the ball position, velocity, and acceleration at time t
ut : Identity matrix of ball state
Ft : Model of gravity and air friction exerting on the ball in the air
Moving model in 3D space
Ht : Model of ball hitting by players
Ct : Model of court friction exerting on the ball when bouncing
wt : Process noise

31. 31
Interaction dialogue is important in a game
tennis gestures -- forehand stroke, forehand volley, backhand
stroke, backhand volley, serve
XYZ signals => SVM classifier
transfer temporal signals into spatial signals
18 feature vectors
average recognition rate is 95%
Signal Analysis of Wiimote
Z
X
Y
r = 1.2 G
K0
K1
K2
K3
K4
K5
K6
K7
K8
K9
K10
K11
K12
K13
K14
K15
K16
K17
K18
Interaction Dialogue
- analyze and recognize signals
from Wiimote to control the player
X-axis
Y-axis
Z-axis