Lecture 11 maja pantic - automic analysis of facial behaviour

1. Machine Analysis of Facial Behavior Maja Pantic iBUG @ Imperial College London

2. 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 Identity Age Gender Beauty Personality

3. 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 Emotions, Intentions, Attitudes

4. 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013

5. 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 squint lip

6. press ANGER

7. 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 Happiness?

11. Anger?

15. Surprise? Fear?

20. Disgust?

24. Sadness?

26. 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 Facial

27. Action

28. Units

29. (Ekman

31. Friesen,

32. 1978)

33. 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 AU1 + AU4 AU7 AU17 AU22 Facial

34. Action

35. Units

36. (Ekman

38. Friesen,

39. 1978)

50. --

51. code

52. basic

53. emotion

54. displays

55. agnostic AU2 + AU4 AU25 AU11 AU5 + AU7

66. --

67. code

68. non-basic

69. emotion

70. displays

71. AU5 AU26 AU8

79. --

80. code

81. expressions

82. of

83. attitudes

87. AU9 AU2 AU20 AU5 AU19 + AU27 --

88. code

89. any

90. expression

95. 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 Facial

96. Action

97. Units

98. (Ekman

100. Friesen,

101. 1978)

102. 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 AU26? AU29? Expressionless Expressionless AU26 AU29

103. 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 FG

104. reduced KL

106. IJ

107. reduced AU7 AU24 Face

108. Shape

109. Model

110. +

111. Rule-based

112. approach

113. (Pantic

115. Rothkrantz,

116. 1999-2004)

117. 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 AU22 AU14 ✦

118. A

119. pioneering

120. effort

121. towards

122. machine

123. analysis

124. of

125. AUs ✦

126. Introduced

127. a

128. point-based

129. face

130. model

131. suitable

132. for

134. automatic

135. unambiguous

136. discrimination

137. of

138. 32/45

139. AU ✦

140. First

141. effort

142. to

143. use

144. profile

145. images ✦

146. Many

147. groups

148. world-wide

149. adopted

150. the

151. approach

152. (

153. 700

154. citations

155. to

156. the

157. related

158. publications) ✦

159. Not

160. all

161. AUs

162. could

163. be

164. recognised

165. (e.g.

166. AU14,

167. AU22) ✦

168. Facial

169. motion

170. and

171. dynamics

172. could

173. not

174. be

175. handled

176. 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 Expressionless Expressive

177. 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013

178. 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 Gabor

179. Filters

180. +

181. Boosting

182. Point

183. Detection

184. (Vukadinovic

186. Pantic,

187. SMC’05) Face and Facial Point Tracking Facial Expression Classification

188. 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 Face% Detec(on% Ini(al% Sampling% Region% Obtain%new%es(mate% +% New%evidence% Target%distribu(on% Update%Target%Dist.% max th? Early%stop%condi(on% Update%Sampling%Region% 2)%Use%global%info.% (shape)% 1)%Use%local%info.%% (reg.%es(mates)% While%it##itmax# LBP

189. +

190. SVR

191. +

192. MRF

193. (Martinez

195. Valstar

197. Binefa

199. Pantic,

200. CVPR‘10,

201. PAMI

202. 2013) Face and Facial Point Tracking Facial Expression Classification

203. 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 LBP

204. +

205. SVR

206. +

207. MRF

208. (Martinez

210. Valstar

212. Binefa

214. Pantic,

215. CVPR‘10,

216. PAMI

217. 2013) error$(frac)on$IOD)$ frac)on$of$images$BioID$DB:$Compared$to$the$state:of:the:art$ SEMAINE$database$ Mul)PIE$database:$Ligh)ng$condi)ons$ Mul)PIE$database:$Head$rota)on$ Face and Facial Point Tracking Facial Expression Classification

218. 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 Particle

219. Filtering

220. with

221. Factorized

222. Likelihoods

223. (Patras

225. Pantic,

226. FG’04) Face and Facial Point Tracking Facial Expression Classification

227. 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 Particle

228. Filtering

229. with

230. Factorized

231. Likelihoods

232. (Patras

234. Pantic,

235. FG’04) Face and Facial Point Tracking Facial Expression Classification € πk = φ d q sk( ),hj( ),σp( )j ∑ πk: Parzen density estimation sk: particle {hj}: collection of the training data q(.): transformation function for registration d(.): distance function φ(.): Parzen kernel (Gaussian kernel with standard deviation σp)

236. 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 Face and Facial Point Tracking Facial Expression Classification IGO-PCA

237. (Tzimiropoulos

239. Zafeiriou

241. Pantic,

242. PAMI

243. 2012) IGOGradientsIntensityRGB ✤ Computing Image Gradient Orientations ✤ Measuring face similarity using IGO and a cosine kernel = +

244. 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 IGO-PCA

246. face

247. verification

248. (Tzimiropoulos

250. Zafeiriou

252. Pantic,

253. PAMI

254. 2012) Face and Facial Point Tracking Facial Expression Classificationy- th we sis p- of (c) on b’s is the part of the face occluded by the scarf. Fig. 2 (c) plots the distribution of ∆φ in P2. Before proceeding for (a) (b) 0 1 2 3 4 5 6.2832 0 5 10 15 20 25 ∆Φ (radius) (c) Fig. 2. (a)-(b) An image pair used in our experiments. (c) The distribution of ∆φ for the part of the face occluded by the scarf. computing the value of s in P2, we need the following theorem [22] (please refer to Section I of supplementary material for a proof). Theorem I Let u(.) be a random process and u(t) ∼ U[0, 2π) then: • E[ X cos u(t)dt] = 0 for any non-empty interval X ∈ R. (a) (b) 0 0.1 0.2 0 0.1 0.2 0.3 0.4 0.5 0.6 0.7 0.8 0.9 1 Correlation Fig. 3. Cosine-based correlation of im tation for the case of synthetic occlu of images considered for our experim and expected values of s/N(P) as percentage of occlusion N(P2)/N(P)

255. 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 IGO-PCA

257. face

258. tracking

259. (Liwicki

260. et

261. al,

262. TNNLS

263. 2012,

264. IJCV

268. face

269. tracking

270. (Liwicki

271. et

272. al,

273. TNNLS

274. 2012,

275. IJCV

276. 2013)

277. 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 IGO-PCA

279. generic

280. AAM

281. fitting

282. (Tzimiropoulos

283. et

284. al,

285. ACCV

286. 2012,

287. ICCV

288. 2013) Face and Facial Point Tracking Facial Expression Classification ✤ Annotate a set of facial landmarks in training images (set T) ✤ Learn a Shape Model that represents the variations in T ✤ Learn an Appearance Model from image textures deﬁned by T PCA$ PCA$ ✤ AAM Fitting: Placemodel inimage Measure similarity Update model

291. generic

292. AAM

293. fitting

294. (Tzimiropoulos

295. et

296. al,

297. ACCV

298. 2012,

299. ICCV

300. 2013)

301. 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 HoG-PCA

303. Constrained

304. Local

305. Models

306. (Asthana

307. et

308. al,

309. CVPR

310. 2013,

311. PAMI

312. 2014) HoG features convolve with HoG ﬁlters train a weak learner U for each point ﬁt update a set of weak learners U Face and Facial Point Tracking Facial Expression Classification

313. 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 HoG-PCA

315. Constrained

316. Local

317. Models

318. (Asthana

319. et

320. al,

321. CVPR

322. 2013,

323. PAMI

324. 2014) Face and Facial Point Tracking Facial Expression Classification Results for Multi-PIE Results for LFPW ✤ C/CUDA implementation: 30 fps (ibug.doc.ic.ac.uk/resources)

325. 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 Face and Facial Point Tracking Facial Expression Classification

326. 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 Face and Facial Point Tracking Facial Expression Classification

327. 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 1)

328. Frame-based

329. AU

330. classification

331. often

332. leads

333. to

334. “on-off-on”

335. results 2)

336. Encoding

337. temporal

338. evolution/

339. segments

340. of

341. AUs

342. can

343. remedy

344. this

345. Feature-based

346. approach

347. (Valstar

349. Pantic,

350. ICCV-W’07,

351. TSMC-B

353. 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 1)

354. Frame-based

355. AU

356. classification

357. often

358. leads

359. to

361. results 2)

362. Encoding

363. temporal

364. evolution/

365. segments

366. of

367. AUs

368. can

369. remedy

370. this

371. Feature-based

372. approach

373. (Valstar

375. Pantic,

376. ICCV-W’07,

377. TSMC-B

379. 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 1)

380. Frame-based

381. AU

382. classification

383. often

384. leads

385. to

387. results 2)

388. Encoding

389. temporal

390. evolution

391. of

392. AUs

393. is

394. necessary

395. for

396. understanding

397. subtle

398. and

399. complex

400. facial

401. behaviours

402. Face and Facial Point Tracking Facial Expression Classification

403. 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 1)

404. Frame-based

405. AU

406. classification

407. often

408. leads

409. to

411. results 2)

412. Encoding

413. temporal

414. evolution

415. of

416. AUs

417. is

418. necessary

419. for

420. understanding

421. subtle

422. and

423. complex

424. facial

425. behaviours

426. Face and Facial Point Tracking Facial Expression Classification

427. 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 Frame-

429. Feature-based,

430. Rule-based

431. approach

433. (Pantic

435. Patras,

436. TSMC-B

438. 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 Event-

440. Feature-based,

441. SVM

442. +

443. HMM

444. approach

446. (Valstar

448. Pantic,

449. ICCV-W’07) Best

450. Paper

451. Award Face and Facial Point Tracking Facial Expression Classification

452. 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 Event-

454. DT-based,

455. GB

456. +

457. HMM

458. approach

460. (Koelstra

462. Pantic,

463. FG’08,

464. PAMI

465. 2010) Best

466. Paper

468. 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 Event-

470. DT-based,

471. GB

472. +

473. HMM

474. approach

476. (Koelstra

478. Pantic,

479. FG’08,

480. PAMI

481. 2010) Best

482. Paper

484. 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 Face and Facial Point Tracking Facial Expression Classification context

485. (W5+):

486. who?

487. when?

488. where?

489. what?

490. how?

491. why? … Feature-based,

492. context-sensitive

493. CORF

494. approach

495. to

496. AU

497. intensity

498. estimation

499. (Rudovic

500. et

501. al,

502. PAMI

503. 2014)

504. 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 Face and Facial Point Tracking Facial Expression Classification Feature-based,

505. context-sensitive

506. CORF

507. approach

508. to

509. AU

510. intensity

511. estimation

512. (Rudovic

513. et

514. al,

515. PAMI

516. 2014) ( )cs CORF w h− + ( )CORF w h+ ) ( ) h CRF w SR SVM+RVM context

517. (W5+):

518. who?

519. when?

520. where?

521. what?

522. how?

523. why?

524. 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 Disgust Happiness Sadness Anger Fear Surprise Valence positivenegative Arousal highlow expressionl Face and Facial Point Tracking Facial Expression Classification

525. 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 OA-RVM

526. Regression

527. for

528. V/A

529. Prediction

531. (Nicolaou

533. Gunes

535. Pantic,

536. FG’11) Best

537. Paper

539. 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 OA-RVM

540. Regression

541. for

542. V/A

543. Prediction

545. (Nicolaou

547. Gunes

549. Pantic,

550. FG’11) Best

551. Paper

553. 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013

554. 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 (Nicolaou

555. et

556. al,

557. ECCV’12,

558. PAMI

559. 2014)

560. 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 (Zafeiriou

562. Pantic,

563. CVPR-W’11) (Nicolaou

564. et

565. al,

566. ECCV’12,

567. PAMI

568. 2014)

569. 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 (Sandbach

570. et

571. al,

572. IVCJ

573. 2012,

574. BMVC’12) 4D-FAB

575. project

576. (EPSRC) capture motion between frames construct vector projections do quad-tree decompositions ! neutral onset offset apex neutral classify in 6 basic emotion categories using GentleBoost + HMM

577. 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013 (dyadic

578. interactions

579. analysis) www.ibug.doc.ac.uk/resources

580. Lessons Learned 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013

581. 1)

582. Work

583. with

584. naturalistic

585. rather

586. than

587. acted

588. behaviour

589. data! acted behaviour (MMI database) naturalistic behaviour (SEMAINE database) 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013

590. 1)

591. Work

592. with

593. naturalistic

594. rather

595. than

596. acted

597. behaviour

598. data! 2)

599. dynamics

600. of

601. the

602. behaviour

603. are

604. the

605. clue

606. to

607. behaviour

608. understanding! 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013

609. 1)

610. Work

611. with

612. naturalistic

613. rather

614. than

615. acted

616. behaviour

617. data! 2)

618. dynamics

619. of

620. the

621. behaviour

622. are

623. the

624. clue

625. to

626. behaviour

627. understanding! 3)

628. multi-dimensional

629. continuos

630. behaviour

631. interpretation

632. is

633. the

634. solution! 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013

635. 1)

636. Work

637. with

638. naturalistic

639. rather

640. than

641. acted

642. behaviour

643. data! 2)

644. dynamics

645. of

646. the

647. behaviour

648. are

649. the

650. clue

651. to

652. behaviour

653. understanding! 3)

654. multi-dimensional

655. continuos

656. behaviour

657. interpretation

658. is

659. the

660. solution! 4)

661. no

662. excellent

663. people

665. -

667. no

668. excellent

669. work

670. ! 1996 1997 1998 1999 2000 2001 2002 2003 2004 2005 2006 2007 2008 2009 2010 2011 2012 2013

Lecture 11 maja pantic - automic analysis of facial behaviour

Recommended

Recommended

More Related Content

More from mustafa sarac

More from mustafa sarac (20)

Recently uploaded

Recently uploaded (20)

Lecture 11 maja pantic - automic analysis of facial behaviour