Milton Chen: Experiments in Desktop Videoconferencing

Experiments in Videoconferencing Milton Chen CTO http://vseelab.com

The VSee Auditorium desktop interface 15’ x 5’ video wall

VSee

2 nd place Stanford-Berkeley Innovator’s award

3 rd place Stanford business plan competition

Intel CEO Paul Otellini keynote

Oracle Executive VP Chuck Rozwat keynote

Chuck Rozwat keynote

“ the breakthrough that collaboration gurus have been hunting for” -

Jack Hirsch VP of Technology Shell

“ the world’s best videoconferencing system” - Cdr. Eric Rasmussen Iraq Humanitarian Operations Center Department of Defense

“ uniquely suited for

planetwalk” -

John Francis Goodwill Ambassador

United Nations

What if there is no network infrastructure ?

Office of Secretary of Defense, State Department, NATO, United Nations … Strong Angel Kona, Hawaii 17-22 July 2004 VSee was selected as the real-time communication system

VSee at Strong Angel Provide global communication from a temporary shelter VIP presentation between Kona and DC

Ad-hoc peer-to-peer WiFi ~ 0.5 mile ~1 - 10 mile

Experiment 1: convoy protection VSee hops from car to car

Experiment 2: air-to-surface Can also airdrop arbitrary data setup screen shot

Experiment 3: ocean search and rescue

Experiment 3: ocean search and rescue The bottom video was from the live underwater camera held by the swimmer. The map with GPS annotation was shared using VSee setup screen shot

Experiment 3: ocean search and rescue

no pre-existing infrastructure

VSee leverages what you have

Internet

Internet2

Satellite

WiMax

Cell phone

VSee ad-hoc peer-to-peer WiFi

Laptop + wireless card is all you need

Afghanistan

Visual fidelity comparable to high-end hardware

Secure (FIPS 140-2 and triple 256 bit AES )

Never crash (59-day challenge)

Trivial to use (less than 60 seconds for 1 st time users)

Kabul Nov 2004

March 2005 From VSee deployment team

VSee for tsunami relief UN headquarters in Jakarta

VSee in Darfur for refugee management CARE International field office Sudan, Africa

but

Why is videoconferencing not ubiquitous

World’s first videoconferencing system

75 years later

Technology limitations

Inadequate visual communication science

April 7, 1927 - Bell Labs 3x2 inch black&white display 1 msec end-to-end latency

VSee Peer-to-peer wireless How well can we judge eye contact “ The heart is stirred more slowly by the ear than by the eye .” – Horace

Eye contact stirs us to action [Sharbat Gula, photographed by McCurry ‘83]

Eye contact fires up our brain [Kampe et al. ’01 Nature ]

Eye contact sensitivity is high

Spatial perception task

As good as Snellen acuity

[Gibson and Pick ’63] 2 m * 6 observers judged 1 looker looker observer 0 8.5 -8.5 0 100 stdev = 2.8 ° Eye contact (%) Angle (deg)

Sensitivity is symmetric

Cline ’67

Kruger and Huckstedt ‘69

Anstis, et al. ’69

Stokes ’69

Ellgring ’70

PicturePhone camera above display Hydra camera below display

Eye contact is difficult Looking into the camera Attempting eye contact

Solutions to eye contact Half-silvered mirror [Rosenthal ’47] MAJIC [Okada, et al. ’94] ClearBoard [Ishii, et al. ’92] GazeMaster [Gemmell, et al. ’00]

Methodology Observers watch videos of looker Large display with camera at the center

Eye contact?

Sensitivity is asymmetric * 16 observers judged recorded videos of 1 looker

An anatomical explanation looking at you looking sideways looking up looking down eye closing Illustrations from The Artist’s Guide to Facial Expression [Faigin ’90]

VSee Eye contact How well can we judge lip sync “ We shape our tools, and there after our tools shape us” - Marshal Mcluhan

Why read lips

Improves comprehension

Background noise [Sumby and Pollack ’54]

Hearing loss [Binnie, Montgomery, Jackson ’86]

[Yarbus ’ 67]

Audio ahead of the video

Videoconferencing

1 msec to encode audio

Up to 250 msec to encode MPEG-4

Detectable skew

130 msec [Dixon and Spitz ’80]

80 msec [Steinmetz ’96]

Conventional lip synchronization encode network decode A a v time Unsynchronized encode network decode sync a , v Audio delay line A delay skew

Attribute delay and skew to remote person

=> person is not believable?

=> person is slow?

[Reeves and Nass ’96]

encode network decode A a v time Unsynchronized encode network decode sync a , v Audio delay line A delay skew

A new lip sync method encode network decode sync synchronized and low perceived latency a v a v encode network decode A a v time Unsynchronized encode network decode sync a , v Audio delay line A Round trip delay

Methodology

Recorded 3 speakers

44.1KHz x 16 bps uncompressed audio

320x240x30fps uncompressed video

Sentences consist of easy to lipread words

Speaker 1 female native speaker Speaker 2 male native speaker Speaker 3 male non-native speaker

Perception of variable AV skew * 16 subjects judging recorded videos

VSee Eye contact Lip sync What frame rate is necessary “ We express ourselves into existence .” – Iris Murdoch

Minimum required frame rate

Full motion 10-30 fps

Tolerable 5 fps

[Tang and Isaac ’93]

Lip synchronization 5 fps

[Watson and Sasse ’96]

Content understanding 5 fps

[Ghinea and Thomas ’98]

Sign language recognition 1 fps

[Johnson and Caird ’96]

Gesture Detection Algorithm input image frame difference after erosion Visualization of algorithm

Gesture sensitive transmission allows dynamic discussion 15 fps ~0.2 fps 0.2 fps * 8 groups of 4 people during a discussion * requires 10% of full motion bandwidth

Other studies [ Conveying Conversational Cues Through Video PhD Dissertation, 2003] When is a smile not a smile Value of f2f for discussion Visualizing the pulse of Classroom

VSee Eye contact Lip sync Gesture Telework “ Laugher is the shortest distance between two people ” – Victor Borge

VSee customers

telework => less money and influence