Presentation of the paper "Is Face Recognition Safe from Realizable Attacks?" authored by Sanjay Saha and Terence Sim of the Department of Computer Science at the National University of Singapore. The paper was presented at the INTERNATIONAL JOINT CONFERENCE ON BIOMETRICS (IJCB 2020).

1. Is Face Recognition Safe from Realizable Attacks? – Sanjay Saha*, Terence Sim IJCB 2020 | slide #1
Is Face Recognition Safe from
Realizable Attacks?
Sanjay Saha (presenting), Terence Sim
National University of Singapore
{sanjaysaha, tsim}@comp.nus.edu.sg

2. Is Face Recognition Safe from Realizable Attacks? – Sanjay Saha*, Terence Sim IJCB 2020 | slide #2
Motivation
• Face recognition is a highly used biometric.
• A common target to various attacks.
• State-of-the-art FRSs are vulnerable to previous
attacks.
Source: link M. Sharif, L. Bauer, and M. K. Reiter, “Accessorize to a
Crime : Real and Stealthy Attacks on State-of-the-Art
Face Recognition,” pp. 1528–1540, 2016.
Source: link
Our goal is to examine vulnerability of FRSs in
Black-box settings

3. Is Face Recognition Safe from Realizable Attacks? – Sanjay Saha*, Terence Sim IJCB 2020 | slide #3
Realizable Attack on Face Recognition
Systems
• FRS is used to authenticate users of the building.
• FRS is attended by a security guard.
• The Attacker wants to get into the building by fooling the FRS.
• He also must try not raise suspicion of the security guard.
• The attacker does not know (black-box) the internal
structure of the FRS.
Main requirements of a
Realizable Attack on FRS.

4. Is Face Recognition Safe from Realizable Attacks? – Sanjay Saha*, Terence Sim IJCB 2020 | slide #4
Proposed Method

5. Is Face Recognition Safe from Realizable Attacks? – Sanjay Saha*, Terence Sim IJCB 2020 | slide #5
Attack Scheme
Attack Gallery Victim(s)
Break-in
Attacker is NOT
IN the gallery No specific Victim
Impersonation
Attacker is NOT
IN the gallery
One, selected by
Attacker
Evasion
Attacker is IN
the gallery
No specific Victim if
it is not the Attacker

6. Is Face Recognition Safe from Realizable Attacks? – Sanjay Saha*, Terence Sim IJCB 2020 | slide #7
Face Synthesizer
• Main focus is to generate realizable face images
so that it can be replicated in real-world scenarios.
• We use the MMDA Face Synthesizer1
• Capable of generating realistic-looking faces from the given different
variations of training face images.
𝑝 = ℎ1, ℎ2
𝑓𝑎𝑐𝑒 ℎ𝑎𝑖𝑟
𝑚1, 𝑚2, 𝑚3
𝑚𝑎𝑟𝑘
𝑔1, 𝑔2
𝑒𝑦𝑒𝑔𝑙𝑎𝑠𝑠
𝑒1, 𝑒2
𝑒𝑥𝑝𝑟𝑒𝑠𝑠𝑖𝑜𝑛
• MMDA because it is easy to train and it produces good quality realistic
images.
1Terence Sim and Li Zhang. Controllable face privacy. In 2015 11th IEEE International Conference
and Workshops on Automatic Face and Gesture Recognition (FG), volume 4, pages 1–8. IEEE, 2015

7. Is Face Recognition Safe from Realizable Attacks? – Sanjay Saha*, Terence Sim IJCB 2020 | slide #9
Training the Face Synthesizer
Two other attackers
72 combinations of
Attacker 1

8. Is Face Recognition Safe from Realizable Attacks? – Sanjay Saha*, Terence Sim IJCB 2020 | slide #10
Results

9. Is Face Recognition Safe from Realizable Attacks? – Sanjay Saha*, Terence Sim IJCB 2020 | slide #11
Successful Attacks (1)
Successful Break-in attacks
AttackersBroke-inas
Generated faces

10. Is Face Recognition Safe from Realizable Attacks? – Sanjay Saha*, Terence Sim IJCB 2020 | slide #12
Successful Attacks (2)
Successful Impersonation attacks
AttackersTargetedas
Generated faces

11. Is Face Recognition Safe from Realizable Attacks? – Sanjay Saha*, Terence Sim IJCB 2020 | slide #13
Realizing the attacks
Generated face Real Face
Attackers
Victims
Generated faceReal Face

12. Is Face Recognition Safe from Realizable Attacks? – Sanjay Saha*, Terence Sim IJCB 2020 | slide #14
Unsuccessful Attacks
• When the ‘gallery’ is built with faces from different race, gender, etc.
• Evasion attacks
Attacker 1 Attacker 2
Failed Impersonation attack by Attacker 1, and Attacker 2

13. Is Face Recognition Safe from Realizable Attacks? – Sanjay Saha*, Terence Sim IJCB 2020 | slide #15
Findings – Variation of Galleries
Counting successful attacks for different galleries with varying difficulties

14. Is Face Recognition Safe from Realizable Attacks? – Sanjay Saha*, Terence Sim IJCB 2020 | slide #16
Findings – Different Gallery Sizes
Increasing gallery sizes increases chance of successful attacks

15. Is Face Recognition Safe from Realizable Attacks? – Sanjay Saha*, Terence Sim IJCB 2020 | slide #17
Summary
• Approximately 21.8% of the attacks (Break-in and
Impersonation) were successful for all the attackers.
• Real perturbations of the attackers’ face succeed in
break-in and impersonation attacks.
• Impersonation attacks depend on the similarity between the Attacker
and the Victim.
• Good initial parameter vector also plays an important role.
• Our work makes a clarion call for urgent research to address these
vulnerabilities in FRS. We hope other researchers will take up this
challenge.

16. Is Face Recognition Safe from Realizable Attacks? – Sanjay Saha*, Terence Sim IJCB 2020 | slide #18