The document introduces 'labellessface', a framework aimed at improving fairness in face recognition models without relying on attribute labels, addressing issues related to racial bias and dataset construction. It features a fair class margin penalty system that adjusts weights based on individual class favoritism during training, ultimately seeking to enhance individual authentication accuracy. The framework utilizes normalized features and dynamically updates margins in response to the classification accuracy of each individual compared to an average baseline.

1. LabellessFace:
Tetsushi Ohki1,2,*, Yuya Sato1,*, Masakatsu Nishigaki1, Koichi Ito3
1Shizuoka University, 2RIKEN AIP,3Tohoku University
Fair Metric Learning for Face Recognition
without Attribute Labels

2. Motivation
One of the most prominent concerns regarding racial bias is the
racial composition within the dataset. However, Constructing a fair
and fair and large-scale dataset is often have problems.
Human-annotated labels poses
challenges in terms of time, cost, and
potential biases.
Scalability Label dependency
Label dependent fairness improvement
cannot guarantee fairness for unknown
attributes
Can we improve a fairness notion without
assuming the target attribute labels?

3. LabellessFace | Framework
Normalized
feature
Fair class margin penalty
Face recognition
model
Normalized weights
Update class
favoritism level
Softmax
Cross-entropy loss
Each epoch
Weights
- Dynamically updates different
margins for each individuals based
on class favoritism level while
progressing the training through the
fair class margin penalty process
- Fair class margin penalty is de
fi
ned
based on the class favoritism level
- Class favoritism level is determined
based on how much the recognition
accuracy for each individual deviates
from the overall average using the
training samples.

4. Normalized
feature
Fair class margin penalty
Face recognition
model
Normalized weights
Update class
favoritism level
Softmax
Cross-entropy loss
Each epoch
Weights
LabellessFace | Fair Class Margin Penalty
margin
coef
fi
cient
class
favoritism level
A coef
fi
cient dc (margin coef
fi
cient) is added to the
basic ArcFace loss function to minimize the bias in
individual authentication accuracy.

5. LabellessFace | Class Favoritism Level
<latexit sha1_base64="OvMG1a6seQaHfJwt8OM/agCa86M=">AAADhHicjVLNThRBEK5lFHH94e9C4mXDZo0Hs6lFEcLBkHDxuIALJCxuZsbepcP8pWd2ASfwALwAJp4g8WB8DC+8AAcewXDExIsHvu4ZNLpB6MlMV31dX1V90+VEnowT5rPCgHXn7uC9ofvFBw8fPR4eGR1bicOuckXDDb1QrTl2LDwZiEYiE0+sRUrYvuOJVWdrQZ+v9oSKZRi8TXYjseHbnUC2pWsngN41Qxxqblpvyb3WSJmrbFap36jlRpnyVQ9HC0xNek8hudQlnwQFlMD2yKYYzzrViCkCtkEpMAVLmnNBe1QEt4sogQgb6Ba+HXjrORrA1zljw3ZRxcOrwCxRhU/5C1/wCX/l7/zr2lypyaF72cXuZFwRtYYPJpZ/3sjysSe0+YcFRuVajgCulakbtCXUplmjSUJjZBCt1s3q9D4cXizPLVXSp3zM59B5xGf8DUqD3g/386JY+vSfvh30nFUP4G+bv+YbHQHuKQWub6FjkB1EauRKZYhs2ldASnnc/u/IJrRJeBKx8S1raH23q5FF9tcoYiZr/05gv7EyVa29qk4vvizPT+XTOURPaJKeYQJnaJ7eUJ0a6FTRRzqiY2vQem69sKaz0IFCzhmnv5b1+hKhZMW/</latexit>
Pi
average
average
average
<latexit sha1_base64="C2SNkiRsrtwuhWLsFwxdkSIt6p4=">AAADeXicjVK7bhNBFD328gjmkQQaJBqDZRRRWNfhKapINJRJjJNISWTtLmNnlH1pd20IFnwAEi0pqECiQHwGDT9AkU9AlEGigIIzswsIrJDManfuPXPPvffsXC8JdJaL7FWqzrHjJ05OnaqdPnP23PTM7PmVLB6mvur6cRCna56bqUBHqpvrPFBrSarc0AvUqrd9z5yvjlSa6Th6kO8kajN0B5Hua9/NCXX6Pd2baUhL7KpPGu3SaKBci/FsRbCBh4jhY4gQChFy2gFcZHzW0YYgIbaJMbGUlrbnCk9RI3fIKMUIl+g2vwN66yUa0Tc5M8v2WSXgm5JZR1M+yTvZl4/yXj7LjwNzjW0O08sOd6/gqqQ3/fxi59uhrJB7jq0/LDKaB3IUcaMsPURbjj7uWE2aGhOLGLV+UWf0ZHe/c3e5Ob4qb+QLdb6WPflApdHoq/92SS2/+k/fHnsuqkf0H9m/FlodEe9pTNzcwsAijxlpkF8qY2YzfkqkXsY9+x25QW2anmZsdsQaRt/RahSRkzVqnMn2vxM4aazMt9q3WjeXbjQW5svpnMIlXMEcJ/A2FnAfi+iy0wFe4CV2q9+dy86cc60IrVZKzgX8tZzrPwG9McGu</latexit>
fi
<latexit sha1_base64="Cves9VRMJMxHWvJ0Ime/9hk+vjM=">AAADeXicjVK7bhNBFD328gjmkQQaJBqDZRRRWNfhKapINJRJjJNISWTtLmMzZF/aXRuCBR+AREsKKpAoEJ9Bww9Q5BMQZZAooODM7AICKySz2p17z9xz7z0710sCneUiu5Wqc+ToseNTJ2onT50+Mz0ze3Yli4epr7p+HMTpmudmKtCR6uY6D9Rakio39AK16m3dMeerI5VmOo7u5duJ2gzdQaT72ndzQp1+72FvpiEtsas+abRLo4FyLcazFcEG7iOGjyFCKETIaQdwkfFZRxuChNgmxsRSWtqeKzxFjdwhoxQjXKJb/A7orZdoRN/kzCzbZ5WAb0pmHU35JO9kTz7Ke/ksP/bNNbY5TC/b3L2Cq5Le9PPznW8HskLuOR78YZHR3JejiBtl6QHacvRxy2rS1JhYxKj1izqjJzt7ndvLzfFleSNfqPO17MoHKo1GX/23S2r51X/69thzUT2i/8j+tdDqiHhPY+LmFgYWecxIg/xSGTOb8VMi9TLu2e/IDWrT9DRjs0PWMPoOV6OInKxR40y2/53ASWNlvtW+0bq+dK2xMF9O5xQu4BLmOIE3sYC7WESXnQ7wAi+xU/3uXHTmnCtFaLVScs7hr+Vc/QnARMGv</latexit>
fj
Face Recognition
Model
softmax
<latexit sha1_base64="0bR206Aly3H3HxRwtifYC8NJX70=">AAAEGXicjVLLbtNAFL2ugZbwaAobJDYjolSpENEkghYhkCqxYdkHaSt1qmg8GTuj+sV4Ehqs8AH8AIuuisQC8RWITcUWsegnIJZFYsOCa8c8o5KOZc+9Z86ZO8dzndhXiaH0yJqyz5w9Nz1zvnTh4qXLs+W5KxtJ1NNCtkTkR3rL4Yn0VShbRhlfbsVa8sDx5aaz+zBb3+xLnagofGwGsdwJuBcqVwluEGqXDzpt8YD50jUsJcyRngpTrjUfDFPfH5ISczUXaXOYNshNwuReXGMeDwJOmOhEhrhtsTAk86SGAblP6AJhbJKm+6e2kDJPPsnVMuwU9ZlWXtfUS+1yhdZpPsh40CiCChRjJZqzKDDoQAQCehCAhBAMxj5wSPDZhgZQiBHbgRQxjZHK1yUMoYTaHrIkMjiiu/j1MNsu0BDzbM8kVwus4uOrUUmgSj/RN/SYHtK39DP9fuJeab5HdpYBzs5IK+P27Itr698mqgKcDXR/q1BRPVEjEc+c6QneDLhwN/ek0GOcI5lbMarTf/byeP3eWjWdp6/oF/R5QI/oe3Qa9r+K16tybf8/53bwzKPqIeZP878W5D5CvKcU8ewWvBzZQ2aG/HQZ4W5ZrhEhBe/5LyZDbwozhdzklDUyf6erMWKO18h6svFvB44HG816Y7F+Z/V2ZblZdOcMXIcbUMMOXIJleAQr0AJhTVu3rEVryd6339mH9ocRdcoqNFfhr2F//AH0SvgO</latexit>
dc =
(
2
1+exp( ·fc) (fc < 0)
2
1+exp( ·h·fc) (fc 0)
<latexit sha1_base64="U1WeUPg/qUgtdkUwCCpi5WW4+Bs=">AAADjnicjVLLThRBFL1D+4DxwaAJIXEzYTLqalJDzGhMjAQ2LHk4QMKQSXdbDAX9SlXNCLbDB/gDLlhBwoLgX7jxB1zwCYYlJm5ceKq61egEoTrdde+pe+69p+t6SSCUZuy0MORcu37j5vBI8dbtO3dHS2P3llXclT5v+nEQy1XPVTwQEW9qoQO+mkjuhl7AV7ztWXO+0uNSiTh6pXcTvh66nUhsCN/VgNql8ZbmOzptKi7L/a3MeaT67VKF1Zhd5UGjnhsVytd8PFZg1KLXFJNPXQqJU0QadkAuKTxrVCdGCbB1SoFJWMKec+pTEdwuojgiXKDb+HbgreVoBN/kVJbto0qAV4JZpir7wo7ZOfvMTthX9uPCXKnNYXrZxe5lXJ60R99PLH2/lBVi17T5hwVG9UIOB26UyUu0adqgZ1aTgMbEIkatn9Xpvf1wvvR8sZo+ZIfsDDoP2Cn7BKVR75t/tMAX9//Tt4ees+oR/Df2r4VWR4R7SoGbW+hYZAeRBvmlMkY240sg5Txu73dkC9oEPIFYdcUaRt/VamSRgzWKmMn6vxM4aCxP1eqNWmPhSWV6Jp/OYXpAk/QYE/iUpmmO5qmJTt/RIZ3QR6fkNJwXzsssdKiQc+7TX8uZ+wloMMl2</latexit>
User j’s
<latexit sha1_base64="sLyPTHtj9BenwRjH0ailztBTgFo=">AAADjnicjVLNbtNAEJ7U/JQUaAoSQuISNQriFE0QCggJUcGlx/6lrdRUkW22YVV7ba03ocWUB+AFOHCiEgdU3oILL8Chj4A4FokLB75dGxBEpV3L3plv55uZzztBGsnMMB9WJrwzZ8+dn7xQnbp46fJ0bebKapYMdSi6YRIlej3wMxFJJbpGmkisp1r4cRCJtWD7sT1fGwmdyUStmN1UbMb+QMktGfoGUL92rWfEjslXtC+VVIO6xLnI9vq1BrfYrfq40S6NBpVrIZmpMPXoCSUU0pBiEqTIwI7IpwzPBrWJKQW2STkwDUu6c0F7VAV3iCiBCB/oNr4DeBslquDbnJljh6gS4dVg1qnJn/k9H/EnPuAv/OPYXLnLYXvZxR4UXJH2p19dX/5+IivGbujpHxYYzWM5ArhVpk/QZmiL7jlNEhpTh1i1YVFn9Pz10fL9pWZ+k/f5K3S+5UP+CKVq9C18tyiW3vyn7wA9F9UV/Gfur8VOh8I95cDtLQwcsoNIi/xSmSCb9TWQehn38ndkD9okPInY7JQ1rL7T1Sgix2tUMZPtfydw3Fi93Wp3Wp3FO425R+V0TtINmqVbmMC7NEfztEBddPqC9umAPng1r+M98B4WoROVknOV/lre/E9JTMls</latexit>
Training images
<latexit sha1_base64="sLyPTHtj9BenwRjH0ailztBTgFo=">AAADjnicjVLNbtNAEJ7U/JQUaAoSQuISNQriFE0QCggJUcGlx/6lrdRUkW22YVV7ba03ocWUB+AFOHCiEgdU3oILL8Chj4A4FokLB75dGxBEpV3L3plv55uZzztBGsnMMB9WJrwzZ8+dn7xQnbp46fJ0bebKapYMdSi6YRIlej3wMxFJJbpGmkisp1r4cRCJtWD7sT1fGwmdyUStmN1UbMb+QMktGfoGUL92rWfEjslXtC+VVIO6xLnI9vq1BrfYrfq40S6NBpVrIZmpMPXoCSUU0pBiEqTIwI7IpwzPBrWJKQW2STkwDUu6c0F7VAV3iCiBCB/oNr4DeBslquDbnJljh6gS4dVg1qnJn/k9H/EnPuAv/OPYXLnLYXvZxR4UXJH2p19dX/5+IivGbujpHxYYzWM5ArhVpk/QZmiL7jlNEhpTh1i1YVFn9Pz10fL9pWZ+k/f5K3S+5UP+CKVq9C18tyiW3vyn7wA9F9UV/Gfur8VOh8I95cDtLQwcsoNIi/xSmSCb9TWQehn38ndkD9okPInY7JQ1rL7T1Sgix2tUMZPtfydw3Fi93Wp3Wp3FO425R+V0TtINmqVbmMC7NEfztEBddPqC9umAPng1r+M98B4WoROVknOV/lre/E9JTMls</latexit>
Training images
softmax
<latexit sha1_base64="U/RhVGUaagrUBGjZ8Y3gJxySC1s=">AAADhHicjVLLThRBFL1DA+II8nBj4mbCZAwLMrnDQw0LQ+LG5fAYIGFg0t0WY0m/Ut0zgB38AH9AE1eQuDB+Bht/wAWfYFxi4saFp6objEx4VKe77j11z733dF0n8mScMJ8W+qz+gcE7Q3eL94ZH7o+OjU+sxWFHuaLhhl6oNhw7Fp4MRCORiSc2IiVs3/HEurP7Qp+vd4WKZRisJgeR2PLtdiB3pGsngLabIQ41N6233hy2xspcZbNKvUYtN8qUr3o4XmBq0isKyaUO+SQooAS2RzbFeDapRkwRsC1KgSlY0pwLOqQiuB1ECUTYQHfxbcPbzNEAvs4ZG7aLKh5eBWaJKvydv/AZf+Ov/IP/XJkrNTl0LwfYnYwrotbo+4crv29k+dgTev2PBUblSo4ArpWpG7QltEPPjCYJjZFBtFo3q9N9++FsZWG5kj7mY/4JnUd8yidQGnR/uZ+XxPKna/p20HNWPYC/Z/6ab3QEuKcUuL6FtkH2EamRc5UhsmlfASnlce8uIpvQJuFJxMa3rKH13a5GFtlbo4iZrF2ewF5jbaZae1KdX5orL87k0zlEj2iSpjCBT2mRXlKdGuhU0Uc6omNr0Jq2Zq35LLSvkHMe0H/Lev4XpHjFwA==</latexit>
Pj
class
favoritism
level
<latexit sha1_base64="GugXQiHLNm3c3I8oPNShaU3pcoo=">AAADjnicjVJLbhNBEC1n+ATziQMSQmJjYRlYWWWEDEJCRLDJMh+cRIoja2bomFbmp+62SRjMAbgAC1ZEYoHCLdhwARY5AmIZJDYseN0zgMAKSY9muup1vap60xVkkdSGeb8y5Z04eer09Jnq2XPnL8zUZi+u6HSoQtEN0yhVa4GvRSQT0TXSRGItU8KPg0isBluP7PnqSCgt0+Sx2cnERuwPErkpQ98A6tcu94zYNnlXC1Ufy8K5ocf9WoNb7FZ90miXRoPKtZDOVph69IRSCmlIMQlKyMCOyCeNZ53axJQB26AcmIIl3bmgMVXBHSJKIMIHuoXvAN56iSbwbU7t2CGqRHgVmHVq8md+zwf8iff4C/84NFfucthedrAHBVdk/ZlXV5a/H8mKsRt6+ocFRvNQjgBulakjtBnapLtOk4TGzCFWbVjUGT1/fbB8b6mZX+dd/gqdb3mfP0JpMvoWvlsUS2/+03eAnovqCfxn7q/FTkeCe8qB21sYOGQbkRb5pTJFNusrIPUy7uXvyB60SXgSsfqYNay+49UoIidrVDGT7X8ncNJYudVqd1qdxduNuYfldE7TVbpGNzGBd2iO5mmBuuj0Be3SHn3wal7Hu+89KEKnKiXnEv21vPmfZRTJdQ==</latexit>
User i’s
<latexit
sha1_base64="tt/u+dtqDJaPOPsT4zRHSdKGv5M=">AAADoHicjVI9TxRRFL3LqOD6wYKNic2EzRoaN3eNAUJFpKFzF1wgsmQzMz6WF+YrM7OrMFl+AIWthRUkFoSWFhsb/4AFP8FYYmJj4XlvRlE3CG8y8+49955735l37dCVccJ8Whgyrl2/MTxys3jr9p27o6Wx8eU46EaOaDqBG0SrthULV/qimcjEFathJCzPdsWKvTWv4is9EcUy8J8n26FY96yOLzekYyWA2qXJVoCwYqf1fnvefGSeA3+G+u1Smauslzlo1HKjTPmqB2MFpha9pIAc6pJHgnxKYLtkUYxnjWrEFAJbpxRYBEvquKA+FcHtIksgwwK6hW8H3lqO+vBVzVizHXRx8UZgmlThz3zIZ/yJj/gL/7iwVqprqLNsY7czrgjbo3v3l75fyvKwJ7R5zgKjciFHAFfKoku0JbRBM1qThMZQI0qtk/Xp7bw9W5pdrKQP+YC/Quc+n/JHKPV735z3DbH47j/ntnHmrLsP/5X+a57W4eOeUuDqFjoaeY1MhfxSGaCa8iMgZp63+zuzBW0SnkRufMUeSt/VemSZgz2KmMnavxM4aCw/rtamqlONJ+W5p/l0jtADmqBJTOA0zdEC1amJk76hYzqhD8aEsWA8MxpZ6lAh59yjv5bx4icb/tEM</latexit>
P
C
P
<latexit sha1_base64="qXX4CoVSFrrvA5/K5//xUbcnysk=">AAADj3icjVLLTttAFL3BfdD0QUIXrdRN1ChVV9FNVSXAAkV0Q3fhEUAiKLLdIYzwS7aTQq3wAfwAi65o1UXV9iu66Q90wSegLqnEhkXPjE1RhVIYy/a9Z86ZO2fmWoEjo5j5KDdm3Lh56/b4nfzde/cfTBSKkyuR3w9t0bZ9xw/XLDMSjvREO5axI9aCUJiu5YhVa/uVml8diDCSvrcc7wZiwzV7ntyUthkD6hYedXxMK3VyEbWGw26hzNXp6To3pkqXg1qV9ShTNlp+McfUoTfkk019ckmQRzFih0yK8KxTjZgCYBuUAAsRST0vaEh5aPtgCTBMoNv49pCtZ6iHXK0ZabWNKg7eEMoSVfgnf+YT/sFf+JjPRq6V6DXUXnbxt1KtCLoT+4+XTq9UufjHtHWhgqIyUiOAK2fhFd5i2qQp7UnCY6AR5dZO6wzeHZwszSxWkmf8gX/B5yEf8Xc49Qa/7U8LYvH9f/ZtYc9pdQ/5W31qrvbh4Z4S4OoWehrZAVMh5y59rKbyEEgp4+39ZXbgTSKT4EbXrKH8Xa9GyrxcI4+ePG+80uhg5UW1Vq/WF16Wm3NZd47TE3pKz9GBDWrSPLWorU/4I32lb0bRaBizRjOljuUyzUP6Zxiv/wBqxsrA</latexit>
P
<latexit
sha1_base64="415X51qSNvuF78/R0QNbK7MeRKI=">AAADeXicjVI9bxNBEH328RHMRxJokGgMllFEYY0BBUQVQUOZxDiJlETW3bF2Vrkv3Z0NwYIfgERLCiqQKBA/g4Y/QJGfgCiDRAEFb/cOEFgh2dPdzrydNzPvdrwk0FkuslepOseOnzg5dap2+szZc9Mzs+dXsniY+qrrx0GcrnlupgIdqW6u80CtJalyQy9Qq972PXO+OlJppuPoQb6TqM3QHUS6r303J9Tp9/zeTENaYld90miXRgPlWoxnK4INPEQMH0OEUIiQ0w7gIuOzjjYECbFNjImltLQ9V3iKGrlDRilGuES3+R3QWy/RiL7JmVm2zyoB35TMOprySd7JvnyU9/JZfhyYa2xzmF52uHsFVyW96ecXO98OZYXcc2z9YZHRPJCjiBtl6SHacvRx22rS1JhYxKj1izqjJ7v7nTvLzfFVeSNfqPO17MkHKo1GX/23S2r51X/69thzUT2i/8j+tdDqiHhPY+LmFgYWecxIg/xSGTOb8VMi9TLu2e/IDWrT9DRjsyPWMPqOVqOInKxR40y2/53ASWPleqs935pfutlYuFtO5xQu4QrmOIG3sID7WESXnQ7wAi+xW/3uXHbmnGtFaLVSci7gr+Xc+Amv4cG5</latexit>
f
c
<latexit sha1_base64="HDoCsqqHr2EFbC1lHRoDNPQThdg=">AAACuHichVFLLwNRFP46XvVssZHYCCq1aW5FkK4aNpYtqhKamhlXXeZlZtqoiT/gD1jYILEQe3/ARmJt4SeIZT02Fs5MmwgNzuTOPfc75zv3O/coliYcl7HHkNTS2tbeEe7s6u7p7YtE+wdWHbNsqzynmppprymywzVh8JwrXI2vWTaXdUXjeWVvwY/nK9x2hGmsuFWLF3S5ZIhtocouQYVMfEPRvYOj4u5kcbcYHWMJFthIs5NsOGPp8deb20r3W8aM3mMDWzChogwdHAZc8jXIcOhbRxIMFmEFeITZ5IkgznGELuKWKYtThkzoHv1LdFpvoAad/ZpOwFbpFo2WTcwRxNgDu2I1dseu2RP7+LWWF9TwtVRpV+pcbhUjx0PL7/+ydNpd7Hyx/tTsYhtzgVZB2q0A8btQ6/zK4UltObUU8ybYBXsm/efskd1SB0blRb3M8qVTqh77VREn3H83+4+X86hD3Y/TIJM/x9bsrE4lkjOJ6SxNdB51C2MYo4jT3GaRxiIyyFH9fZzgDOdSStqUSpKop0qhBmcQ30yyPwFvPaGg</latexit>
P(xj)j
<latexit sha1_base64="tBDIEPeY5CXTtrU34EKEl22tqAg=">AAACuHichVE7SwNBEP48X/EdtRFsxBCJCGFPRMUqaGMZHzGCCfHuXOPivby7BOORP2AtWNioYCH+DBvB2iI/QSwVbCycuwREQ+Icezv7zXyz3+yoti5cj7Fah9TZ1d3TG+nrHxgcGh6Jjo7tuFbJ0XhGs3TL2VUVl+vC5BlPeDrftR2uGKrOs+rxWhDPlrnjCsvc9io2zxtK0RSHQlM8gvLpRE41/NNqQcwWRCEaY0kW2lSzIzecWGo6N3dRS1XSVvQZORzAgoYSDHCY8MjXocClbw8yGGzC8vAJc8gTYZyjin7iliiLU4ZC6DH9i3Taa6AmnYOabsjW6BadlkPMKcTZC7tn7+yJPbBX9tWylh/WCLRUaFfrXG4XRs4ntj7/ZRm0ezj6YbXV7OEQy6FWQdrtEAm60Or88tnl+9bKZtyfYbfsjfTfsBp7pA7M8od2t8E3r6h6vKUiTnjwbk6bl/OpQyOI0yDlv2Nrdnbmk/JicmGDJrqKukUwiWkkaG5LSGEdaWSo/gkucY0baUXal4qSqKdKHQ3OOH6Z5HwD0wygZA==</latexit>
P(xi)i
<latexit sha1_base64="G64al792m6qC4AyNwF4JuoyaU3k=">AAAC4XichVHPS9xQEP421tZarVutUOhFKis9lOVFpBVBEHvxuK6uu2BsSF6fGje/SN4u6iN/gJ5KoQjuSaGH4j8h9GLbswf/BOlRwUsPnWQXiop2QjLzvplv8s0bO3SdWDJ2ltO6HnQ/fNTzuPdJX//TgfyzwaU4aERcVHjgBlHNtmLhOr6oSEe6ohZGwvJsV1Tt+vs0X22KKHYCf1FuhWLFs9Z8Z9XhliTIzE8YniXXueWqWmJuTBvKsD21mZhq4009McjXp/Xkg7pWlZj5UVZkmY3cDvROMDoz9ryafKkOl4L8Lxj4iAAcDXgQ8CEpdmEhpmcZOhhCwlagCIsocrK8QIJe4jaoSlCFRWidvmt0Wu6gPp3TnnHG5vQXl96ImCMosFP2jV2wE3bEztmfO3uprEeqZYu83eaK0BzYfbFw9V+WR15i/R/rXs0Sq5jMtDqkPcyQdAre5je39y4WpsoFNcYO2W/Sf8DO2HeawG9e8q/zotyi7oU7FQnC03uL7rk5RRN6aZ4Wqd9c2+1gabyovy1OzNNGZ9G2HrzEK7ymvb3DDOZQQoX67+MYP/BT49qO9kn73C7Vch3OEK6Z1voLlHyxkw==</latexit>
Xj = {xj,k}
Xj
k=1
<latexit sha1_base64="+zrhFHCeRbPnDSAODLSqUbGd4MQ=">AAAC4XichVFNSxxBEH1OTGI0xtUYCHiRyIoHWXpEogiC6CVHv9ZdcMww07ba7Hwx07tEm/kB8SQBCcSTQg6SPxHwkujZgz9BcjSQSw6pmV0IKmoNM1X9ql7Nqy438mSiGLtoMx61P37ytONZZ9fz7hc9hd6+lSSsx1yUeeiFcdV1EuHJQJSVVJ6oRrFwfNcTFbc2l+UrDREnMgyW1XYk1nxnM5AbkjuKILswbvmO2uKOp6upLactbbm+/pDaWo7WUot8bdpM3+trValdGGIlltvg7cBsBUMzw/2VdL/yaj4snMHCOkJw1OFDIICi2IODhJ5VmGCICFuDJiymSOZ5gRSdxK1TlaAKh9AafTfptNpCAzpnPZOczekvHr0xMQdRZOfsmF2xH+wbu2R/7+yl8x6Zlm3ybpMrIrtn9/XSnwdZPnmFrf+sezUrbGAy1ypJe5Qj2RS8yW/sfL5amlos6mF2xH6R/kN2wU5ogqDxm39dEIsH1L14pyJBeHZv8T03p2lCP8vTIs2ba7sdrIyVzLel8QXa6Cya1oEBvMEI7W0CM3iHeZSp/xd8x0+cGtz4aOwZn5qlRluL8xLXzDj4B41hsZA=</latexit>
Xi = {xi,k}Xi
k=1
Fair class
margin
penalty
Average
con
fi
dence score of
each individual
Overall average
con
fi
dence score
Set of
con
fi
dence scores

6. Margin coefficient
<latexit sha1_base64="Y8zmh9SJvkO0I8eEqm/56+c8DHo=">AAADeXicjVI9bxNBEH328RHMRxJokGgMllFEYY0hShBVBA1lEuMkUhJZd5eNWeW+dHc2BAt+ABItKahAokD8DBr+AEV+AqIMEgUUvN07QGCFZE93O/N23sy82/GSQGe5yH6l6pw4eer0xJna2XPnL0xOTV9cyeJB6quuHwdxuua5mQp0pLq5zgO1lqTKDb1ArXo798z56lClmY6jB/luojZDtx/pbe27OaHOVs/vTTWkJXbVx412aTRQrsV4uiLYwBZi+BgghEKEnHYAFxmfdbQhSIhtYkQspaXtucJT1MgdMEoxwiW6w2+f3nqJRvRNzsyyfVYJ+KZk1tGUT/JODuSjvJfP8uPQXCObw/Syy90ruCrpTT6/3Pl2JCvknuPhHxYZzUM5irhRlh6hLcc2bltNmhoTixi1flFn+GTvoHNnuTm6Lm/kC3W+ln35QKXR8Kv/dkktv/pP3x57LqpH9B/ZvxZaHRHvaUTc3ELfIo8ZaZBfKmNmM35KpF7GPfsduUFtmp5mbHbMGkbf8WoUkeM1apzJ9r8TOG6s3Gy151pzS7ONhbvldE7gCq5hhhM4jwXcxyK67LSPF3iJvep356oz49woQquVknMJfy3n1k+pt8G3</latexit>
dc
ℎ = 1
ℎ = 0
ℎ = 0.3
visualization of
<latexit sha1_base64="Y8zmh9SJvkO0I8eEqm/56+c8DHo=">AAADeXicjVI9bxNBEH328RHMRxJokGgMllFEYY0hShBVBA1lEuMkUhJZd5eNWeW+dHc2BAt+ABItKahAokD8DBr+AEV+AqIMEgUUvN07QGCFZE93O/N23sy82/GSQGe5yH6l6pw4eer0xJna2XPnL0xOTV9cyeJB6quuHwdxuua5mQp0pLq5zgO1lqTKDb1ArXo798z56lClmY6jB/luojZDtx/pbe27OaHOVs/vTTWkJXbVx412aTRQrsV4uiLYwBZi+BgghEKEnHYAFxmfdbQhSIhtYkQspaXtucJT1MgdMEoxwiW6w2+f3nqJRvRNzsyyfVYJ+KZk1tGUT/JODuSjvJfP8uPQXCObw/Syy90ruCrpTT6/3Pl2JCvknuPhHxYZzUM5irhRlh6hLcc2bltNmhoTixi1flFn+GTvoHNnuTm6Lm/kC3W+ln35QKXR8Kv/dkktv/pP3x57LqpH9B/ZvxZaHRHvaUTc3ELfIo8ZaZBfKmNmM35KpF7GPfsduUFtmp5mbHbMGkbf8WoUkeM1apzJ9r8TOG6s3Gy151pzS7ONhbvldE7gCq5hhhM4jwXcxyK67LSPF3iJvep356oz49woQquVknMJfy3n1k+pt8G3</latexit>
dc
<latexit sha1_base64="415X51qSNvuF78/R0QNbK7MeRKI=">AAADeXicjVI9bxNBEH328RHMRxJokGgMllFEYY0BBUQVQUOZxDiJlETW3bF2Vrkv3Z0NwYIfgERLCiqQKBA/g4Y/QJGfgCiDRAEFb/cOEFgh2dPdzrydNzPvdrwk0FkuslepOseOnzg5dap2+szZc9Mzs+dXsniY+qrrx0GcrnlupgIdqW6u80CtJalyQy9Qq972PXO+OlJppuPoQb6TqM3QHUS6r303J9Tp9/zeTENaYld90miXRgPlWoxnK4INPEQMH0OEUIiQ0w7gIuOzjjYECbFNjImltLQ9V3iKGrlDRilGuES3+R3QWy/RiL7JmVm2zyoB35TMOprySd7JvnyU9/JZfhyYa2xzmF52uHsFVyW96ecXO98OZYXcc2z9YZHRPJCjiBtl6SHacvRx22rS1JhYxKj1izqjJ7v7nTvLzfFVeSNfqPO17MkHKo1GX/23S2r51X/69thzUT2i/8j+tdDqiHhPY+LmFgYWecxIg/xSGTOb8VMi9TLu2e/IDWrT9DRjsyPWMPqOVqOInKxR40y2/53ASWPleqs935pfutlYuFtO5xQu4QrmOIG3sID7WESXnQ7wAi+xW/3uXHbmnGtFaLVSci7gr+Xc+Amv4cG5</latexit>
fc
and (γ=10)
‣ The margin coef
fi
cient dc is designed to
increase the margin's impact on classes
that are less favored (fc < 0)
‣ Enlarging the facial feature space, while
reducing the margin's impact on classes
that are more favored (fc ≧ 0)
‣ Parameter h adjusts the in
fl
uence on the
more favored classes.
‣ Parameter γ adjusts the sensitivity of the
output to the input.
less
favored
more
favored
Margin coef
fi
cient dc can be get from output of a function similar to the reversed sigmoid
function that takes Class Favoritism Level (fc)

7. Evaluation | Objectives
(2) Label-Independent Fairness Improvement
(1) Fairness-Accuracy Trade-off
Can our proposed method improve the fairness of certain sensitive attributes
(e.g. race or gender) while maintaining accuracy?
In our experiment, we evaluate the effectiveness of our proposed
LabellessFace framework by addressing the following two main questions.
Can our proposed method improve fairness independent of annotated labels?

8. Evaluation | Protocol
[9] Wang, Mei, and Weihong Deng. "Mitigating bias in face recognition using skewness-aware reinforcement learning." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2020.
[10] Wang, Mei, et al. "Racial faces in the wild: Reducing racial bias by information maximization adaptation network." Proceedings of the ieee/cvf international conference on computer vision. 2019.
[11] Gary B. Huang, Manu Ramesh, Tamara Berg, and Erik Learned-Miller. Labeled Faces in the Wild: A Database for Studying Face Recognition in Unconstrained Environments. University of Massachusetts, Amherst, Technical Report 07-49.
Dataset
‣ RFW for Fairness-Accuracy Trade-off Evaluation
‣ LFW for Label-Independent Fairness Improvement Evaluation
Model ResNet-34
Metric EER, AUC (Performance)
STD, Gini, SER (Fairness)
BUPT-Balancedface[9] (train), RFW[10] (test), LFW[11] (test)

9. Results | Fairness-Accuracy Trade-off
- CIFP achieved lower EER since it utilize a training algorithm that takes into
account pre-labeled racial information
- Proposed method exhibits better fairness performance (STD/Gini/SER) with
relatively lower EER indicating that proposed method improves fairness while
maintaining authentication performance
EER-Af(↓) EER-As(↑) EER-Ca(↑) EER-In(↑) STD(↓) Gini(↓) SER(↓)
ArcFace 0.1847 0.1975 0.1145 0.1621 0.03163 0.01031 0.1725
MagFace 0.2034 0.1905 0.0989 0.154 0.04054 0.1353 2.056
CIFP 0.1683 0.173 0.097 0.1293 0.03097 0.1175 1.78
MixFairFace 0.4661 0.2869 0.2928 0.3155 0.07349 0.1032 1.627
Proposed 0.181 0.1871 0.1163 0.1625 0.02775 0.08922 1.609
✓ Evaluated EER (performance)
✓ How much difference is there in EER across racial attributes? (STD/Gini/SER) (fairness)
✓ Compare with ArcFace, MagFace, CIFP and MixFairFace that take different races into account.
Af: African / As: Asian / Ca: Caucasian / In: Indian

10. Results | Label-Independent Fairness Improvement
- The proposed method performs best on all
performance and fairness metrics.
- Label-free fairness training at an individual level
can achieve a high trade-off between accuracy
and fairness even for unknown attributes. Fairness Heatmap of Proposed Method
EER(↓) AUC(↑) STD(↓) Gini(↓) SER(↓)
ArcFace 0.093 0.9665 0.0117 0.08292 2.766
MagFace 0.09867 0.959 0.01127 0.08279 2.766
CIFP 0.091 0.9614 0.01157 0.08845 3.038
Proposed 0.091 0.9681 0.01019 0.07398 2.525
✓ We selected 26 attributes that have more than 100 samples from LFW Dataset
✓ How much difference is there in EER across various unknown attributes? (fairness)
✓ Compare proposed method with ArcFace, MagFace and CIFP.

11. Discussions
How should we de
fi
ne the hyperparameters? (e.g. γ or h?)
Isn't the class favoritism level computationally expensive?
We can compute the class favoritism level sequentially during training, which
results in only a small increase in time and spatial computational complexity.
However, since the computation increases in proportion to the number of training
data, this aspect needs to be considered.
We set the hyper parameters γ=10 and h=1 through grid search to balance the
trade-off between fairness and accuracy. Larger value of h or γ could lead to
greater fairness improvement in case there exists signi
fi
cant latent attribute
biases.

12. Conclusion
✓ The proposed LabellessFace framework aims to achieve fairness in
facial recognition without requiring demographic group labeling.
✓ It introduces a "fair class margin penalty" based on class favoritism
levels, eliminating the need for demographic labels to ensure fairness.
✓ Experiments on facial benchmarks demonstrated the method's
effectiveness in achieving fairness compared to other baselines,
without requiring fairness considerations during training.
✓ We open up new possibilities for creating more equitable and scalable
face recognition systems
project site