This document describes an automatic eyeglasses replacement system for a 3D virtual try-on application. The system uses facial landmark detection and head pose estimation to remove a user's eyeglasses in video sequences and replace them with 3D virtual glasses models. Texture mapping and image blending techniques produce a natural result. The system runs in real-time and effectively replaces glasses for trying on virtual pairs without requiring the user to actually change their prescription glasses.

1. Automatic Eyeglasses Replacement
for a 3D Virtual Try-on System
Takumi Kobayashi, Yuta Sugiura, Hideo Saito
(Keio University)
Yuji Uema
(JINS Inc.)

2. 2
Background
[1] https://instagram-press.com/blog/2017/09/28/a-new-face-filter-to-customize-your-look-and-transport-yourself-to-new-places. [2] https://itunes.apple.com/us/app/ray-
ban-virtual-try-on/id900630471?mt=8. [3] https://itunes.apple.com/us/app/glasses-by-warby-parker/id1107693363?mt=8
[1]
[3]
[2]
AR apps for glasses try-on
It is difficult to clearly observe their face in a screen without wearing prescription
eyeglasses.
Users who have
poor eyesight …

3. 3
Purpose
• Our system replaces real glasses with virtual glasses.
• Propose a glasses try-on system using just a single camera.

4. 4
System Overview
Facial landmarks
detection
by CE-CLM
eyeglasses
removal
Output
Texture mapping
3D virtual
eyeglasses
3D fitting
Input
Natural face image
Video sequences

5. 5
Face detection and tracking
† A. Zadeh, T. Baltrušaitis, and Louis-Philippe Morency., “Convolutional experts constrained local model for facial landmark detection,” in Computer Vision and Pattern
Recognition Workshops, pp.2519 - 2528, 2017.
Convolutional Experts Constrained Local Model (CE-CLM)†
CE-CLM† is capable of:
• Detection of 68 facial landmarks
• 2D and 3D points of each
landmarks
• Head pose estimation

6. 6
• Take a shot of user’s frontal face in advance.
• Clip a skin texture image of the region around eyes.
• Warp and map the textures to match the 3D facial landmarks in video sequences.
Eyeglasses Removal
Natural face image Video sequences
3D warping

7. 7
Image Blending
We smoothly blend skin textures to the face wearing glasses according to
alpha values 𝛼𝛼.
0.4 0.4 0.3 0.2 0.1 0.0
0.5 0.4 0.4 0.3 0.2 0.1
0.6 0.5 0.4 0.4 0.3 0.2
0.8 0.8 0.7 0.4 0.3 0.3
1.0 0.9 0.8 0.5 0.4 0.3
1.0 1.0 0.9 0.7 0.5 0.4
𝐼𝐼 = 𝑆𝑆𝛼𝛼 + 1 − 𝛼𝛼 𝐷𝐷
𝐼𝐼 : Output image pixel
𝛼𝛼 : Alpha value (0 ≤ 𝛼𝛼 ≤ 1)
𝑆𝑆 : Source image pixel
𝐷𝐷 : Destination image pixel
Blurred alpha mask image Image blending result

8. 8
• The temples of the virtual eyeglasses may
actually be hidden by the user’s head.
• We judge which parts of virtual glasses should not be visible by 3D head pose
estimation.
Occlusion Problem
Before After

9. 9
Demo Video

10. 10
Demo Video
Input
Virtual Try-onEyeglasses removal

11. 11
Demo Video
Input
Virtual Try-onEyeglasses removal

12. 12
Effectiveness of our system
No replacement Proposed method

13. 13
Discussions
Our system runs at 18fps. (on a PC with an Intel Core i7 @ 3.60 GHz CPU and
32GB RAM)
• Limitations
• Our method depends on the accuracy of
head pose estimation.
• Impossible to completely remove the parts of
glasses protruding from the face.

14. 14
Conclusions & Future Work
• Conclusions
• We proposed a 3D virtual try-on system, which replace eyeglasses with
virtual eyeglasses in video sequences.
• We utilize 3D head pose estimation method and we remove glasses in a
video by blending skin texture mask.
• We confirmed that eyeglasses removal is effective for the reality of
appearances.
• Future work
• Image segmentation on eyeglasses.
• To adapt to changes in skin color due to lighting and shadows.