This document discusses the role of artificial intelligence (AI) and machine learning (ML) in the beauty care industry. It provides examples of how companies are using AI/ML for skin and hair analysis, virtual makeup try-ons, and product recommendations. The techniques described include facial landmark extraction, lip masking, alpha compositing for virtual makeup effects, as well as challenges and future applications like personalized product design and home-based cosmetic testing. Live demos of virtual makeup tools are also presented.

1. Role of AI & ML in
Beauty Care Industry
GeekNight :: 14th November, 2018
Piyush Bhargava | (Hyderabad)

2. Agenda
● What is AI & ML ?
● AI & ML in Beauty Care
● Examples
● Skin Analysis
● Hair Analysis
● Virtual Makeup
● Live demo of Virtual Makeup
● Q&A
2

3. Artificial Intelligence vs Machine Learning ?
● AI is not a system, but a system
has AI
● ML is one of the ways to make a
system AI enabled
● ML is making a machine learn
○ from Experience (E) - Training Data
○ w.r.t some Task (T) - Problem
Statement
○ and a Performance Measure (P) -
Accuracy
when P improves with E
Artificial Intelligence
Machine Learning
Deep Learning
3

4. AI & ML in Beauty Care
● Product Recommendation
● Product Personalisation
● Product Validation & Research
● Retail Experiences
○ Virtual (Magic) Mirrors
○ Virtual Try On Apps (Web & Mobile)
● Constant Learning AI Systems
4

5. Some big p:)ayers | Unique So:)utions
Recommendation Engines
● Proven - Personalised Skin Care
● My Beauty Matches - Personalised Product Recommendation
● Function of Beauty - Personalised Hair Care
● Beauty.ai - Deep Learning based Beauty Contest
● Olay - Skin analysis from Selfie
Augmented Reality
● Modiface - SDKs for Beauty and Face, AR Mirror
● Perfect Corp. - Virtual Makeup Mobile app
● Coty - Magic Mirror for Bourjois Velvet Collection
5

6. Some big p:)ayers | Unique So:)utions
6Video Source: https://www.coty.com/in-the-news/press-release/coty-magic-mirror

7. Product Validation
● Objective - Improve product performance
● Experimental Cosmetics tested on subjects
● Skin / Hair analysis performed through ML / CV modules
● Based on the analysis, ingredients are finalised for the specific
product
● At product launch, statistical data used to back the performance
claims
7

8. Product Recommendation
8
Quiz / Customer
Profiling
Matching
Product
Recommendations
A sample product
markup table
Visitors profile
estimation

9. Skin Analysis
● Skin Advisor Applications
● Mobile Skin & Hair Analysis Devices
○ Octa Core Processor
○ 30 to 500 times high magnification lens
○ Normal and UV light sensors
○ Android based, WiFi mirroring
● Pore Analysis
● Moisture / Oiliness
● Wrinkles
● Melanin (Color) / Pigmentation
● Sebum / Acne (T & U Zone)
● Morphological Operations, Edge and Contour
analysis, Color based Segmentation, GLCM
9Image Source: http://www.aramhuvis.com/en/apm/

10. Skin Analysis :: Machine Learning Approach
10
Used for detection of symptoms and sometimes
even categorisation of symptoms based on
severity (high, medium, low)
● Crow’s feet
● Blackheads
● Acne
● Roughness / Dryness
● Eye bags
Image
Pre-processing
RoI Selection
Features Extraction
Classifier Training
Feature Vector
Classifier Model
Detection /
Recognition

11. Skin Color Analysis using ICA
Such a tool helps in
● Studying the current
pigmentation composition of a
subject’s skin
● Recommending specific skin
creams based on the analysis
above
● Simulate the proposed
improvements in skin through
reconstruction
● Observe and monitor the actual
changes over a period of time
11
Image
ICA
Haemoglobin (H) Melanin (M)
Modified (H’) Modified (M’)
Inverse
ICA/PCA
PCA
Reconstructed
Image

12. Hair / Scalp Surface Analysis
● Hair Cuticle Damage Analysis
○ Uplifted Cuticle
○ Chipped Cuticle
○ Missing Cuticle
● Broken and Split ends detection
● Hair Density Analysis
● Sebum / Acne analysis on Scalp Surface
● High Magnification Mobile Device used for
capturing the Hair / Scalp images for further
processing
● The tool helps in assessing the hair and scalp
surface, identifying a health index for hair and
recommend the most suitable Shampoo and
Conditioner (and even hair oil)
12
Image source: https://www.youtube.com/watch?v=FmSkffcqBxY&t=700s
Image source: https://www.cgtrader.com/3d-models/science/medical/damaged-hair-cuticle-layer-and-healthy-hair-cuticle-layer

13. Virtual Try On / Makeup
● Simulating makeup on self
● Google Search for “Virtual Makeup” shows many online portals
offering such services
● All major cosmetic brands now offer Virtual Makeup feature on
their website and mobile apps
○ Try makeup on Eyes, Lips and Face, Hair Color
○ Choose from different shades
○ Add product to cart
○ Order online or collect from nearest store
● Convenient and Comfortable
● More choices than a “brick and mortar” shop
● A better visualization of products on self
● Look Good, Feel Better factor increases popularity among common
people
● Steady source of revenue for companies like Modiface, Taaz,
Perfect365, etc offering Virtual Makeover Platform / Engine as a
product to many cosmetic companies
13
Image Source: http://www.flickr.com/photos/idhren/6236954553/

14. Shape Predictor / Face Landmark Extraction
● Shape predictor is used in analysis of
structures and sub regions of an object
● For faces, landmark points describe the
shape of the face attributes like: eyes,
eyebrows, nose, mouth, jawline & chin
● Wide range of applications, including:
face recognition, face filters (snapchat
style), face morphing, emotion
recognition, blink detection, ...
● Standard pre-trained detectors available
● New shape predictors can be trained
● New dataset can also be generated
68 landmark points generated by Dlib
14

15. Training a Shape Predictor using Dlib
● Create or Pick/Download a large dataset of images
○ Mirroring and Augmentation increase the dataset size further
● Generate landmark annotations (.xml file)
○ Face region :: <top, left, width, height>
○ N points :: <x, y> where N is the number of landmarks we want to train for
● CNN training options affect Size, Speed and Accuracy
● Manually fine-tuned in order to get the desired performance
● Face detection (like OpenCV Haar Cascade, Dlib HOG Detector, CNN detectors)
● The points must be
○ Manually labelled
○ Manually reviewed and corrected
● Example python code (http://dlib.net/train_shape_predictor.py.html)
● Example c++ code (http://dlib.net/train_shape_predictor_ex.cpp.html)
● Also, there is a blog post on Medium that explains training a shape predictor using Dlib
15

16. A generic pipeline for Virtual Makeup
1. Face Detection
a. (HAAR, SVM+HOG detector, CNN)
2. Facial Landmark Points extraction
a. (using DLib)
3. Lip Mask Creation through Interpolation
and Curve Fitting
4. Overlaying Lipstick color through Alpha
Compositing
5. Refinements (for more realistic effects)
6. Similar stages for other makeup like eye
liner, eye shadow, etc.
16

17. Facial Landmark Points Extraction using DLib
● Deep Learning based
implementation
● Can be trained for custom
number of landmark points
● Real time, good for Video / Live
processing
● Landmark points help in
○ Head Pose estimation,
○ Facial Shape analysis
○ Facial Region Extraction
and Analysis
○ Many more
Image source: https://ibug.doc.ic.ac.uk/resources/facial-point-annotations/ 17

18. Lip Mask Creation
Normal
Smiling
Open Mouth
Lip Contour
Lip landmark points
18
Extract lip points
Interpolate to get smoother
lip contour
Create a Mask
Blur the Mask
68 landmark points
Apply Lipstick

19. Alpha Compositing - Augmenting on Live Frame
● Alpha Blending ensures
○ Smoother edges
○ Lesser Artifacts
○ Transparency effect to retain
texture from original image
+
HSV Colored Lip
Region
Blurred Mask for
Lip Region
Without Alpha
Blending
With Alpha
Blending
19

20. Let’s have some fun :)
Choose a Lipstick shade
Choose a shade for Eyeshadow
Virtual Makeup in action ;)
20

21. Cha:((enges & Future Landscape
● Data
● Constant innovation to stay ahead of the Competition
● Voice based systems
● Rendering Realistic Effects
○ Using Skin Surface Reflective Properties
○ Using Ambient Lighting Information
○ Compute intensive
● Shift towards Natural and Organic Products
● Personalised Designing & Packaging of Products
● Shifting lab based cosmetic tests to home environments
○ Using cheaper but reliable sensors
○ Increases the data sources
21

22. Acknowledgements
● Adrian Rosebrock for online tutorials at PyImageSearch
● Intelligent Behavior Understanding Group (iBUG) for providing
annotated datasets
● Luca Anzalone for his Medium blog on Shape Predictor Training
● REES46 for their Medium blog post on Progressive Personalization
● ThoughtWorks Hyderabad
● The Awesome Audience .. that’s You !! :)
22

23. Q&A
23
Thank You !!
Piyush Bhargava | (Hyderabad)
piyush.bhargava@thoughtworks.com