This document outlines a project exploring methods for virtual clothing try-on systems. It discusses using U-Net and Mask R-CNN for cloth parsing, as well as a VITON model for virtual try-on. The goals were to survey existing technologies, perform cloth parsing, understand current systems, and estimate human sizes. Work included implementing U-Net and Mask R-CNN, as well as proposing a size predictor and CGAN-based color changer. Issues included lack of understanding of complex models, poor multi-class segmentation, and no size dataset. Future work focuses on improving models, adding cloth wrapping, collecting data, and optimizing for mobile/AR.

1. Virtual
Try-On
B18CSE039, B18CSE041, B18CSE043

2. 1. Problem Statement
2. Goals
3. Literature Survey
4. Our HandsOn Work
4.1. U-Net
4.2. Mask RCNN
4.3. Cloth Size Estimation
4.4. VITON
5. Proposals
6. Accomplishments
7. Issues Faced
8. Future Work
9. Concluding Remarks
Index

3. Problem Statement
To survey and explore some of the methods that can be
used to create a modular pipeline for a virtual try-on
system and to understand the existing one, to make a
productive solution for $1.5 Trillion eFashion Industry.
GOALS
We assigned the following goals for our project:
1. An extensive survey of the papers regarding
technologies that can be used in the fashion industry.
2. Perform cloth parsing on images using instance
segmentation models.
3. Understand the existing virtual-tryon systems.
4. Develop methods to perform human cloth size
estimation.
5. Deploying a feasible solution.

5. LITERATURE REVIEW
1. Depth Cameras
2. Depth Camera based size prediction
3. Camera Based
4. Parsing
5. Virtual Try On

6. DEPTH CAMERA
Structural Light and Coded Light Stereo Depth Time of Flight

7. DEPTH CAMERA BASED SIZE PREDICTION
Automatic human body feature
extraction and personal size
measurements,Tan Xiaohui,et al
Depth Camera, Random Forest
and Geodesic Distances
Excellent precision, average error
of 0.0617cm

8. Single CAMERA BASED size prediction

9. CLOTH PARSING
HUMAN PARSING
VIRTUAL TRY ON
UNets and Mask-RCNNs
OpenPose and Pose2Seg
VITON and ACGPN

10. ACGPN

11. Our Work
1. U-Net
2. Mask RCNN
3. Cloth Size Estimation
4. VITON

12. A Basic Pipeline

13. U-NEt
An Example Structure

14. U-NEt
Given
Input
Expected
Output

15. U-NEt
The structure we Used

16. U-NEt
Result 1 Result 2

17. U-NEt
The mean IoU with background
excluded was calculated for
1000 Random Images as 0.9284.
Result 3

18. Mask RCNN

19. Mask RCNN

20. Mask RCNN
For a given image I and class C,
1. C is predicted to be contained in I if IoUC
>= Ө, where Ө is the IoU
threshold
2. Precision and Recall are calculated over all classes for a set of test
images for some given values of Ө
3. MP and MR are used to calculate F1 score.

21. Mask RCNN

22. Size PreDictor

23. Size PreDictor

24. Size PreDictor

25. VITON

26. Proposals

27. PROPOSED SIZE PREDICTOR

28. PROPOSED SIZE PREDICTOR
Height, Weight, Age, Gender

29. CGAN BASED T-SHIRT COLOR CHANGER

30. Accomplishments
1. Learned and understood the fundamentals
of necessary technology.
2. Found and understood appropriate
technologies/research for our pipelines.
3. Implemented few methods based on UNet,
Mask-RCNN and OpenPose
4. Proposed few intermediate processes.

31. Issues Faced
1. The mathematical details regarding the
workings of the virtual tryon models
proved to be out of scope for us, and
hence we weren’t able to proceed to
create a version of our own.
2. The multi-class implementation of U-Net
gave poor performance on the test
images, and the faults weren’t clear, and
hence we proceeded with a single class
implementation only.
3. Good dataset was not available to check
to experiments, train and validated our
proposed body shape model.
4. Lack of depth sensing hardware.
5. Computational very expensive units.
6. Restrictions and challenges due to
uncertain events due to COVID-19.

32. Future work
1. To gain better mathematical understanding
behind VITON and ACGPN
2. To understand and design cloth wrapping
modules.
3. To prepare a good size dataset
4. To design and optimise the model such that it
can work on a mobile device with a primitive
GPUs.
5. To extend the capabilities to AR.

33. CONCLUDING REMARKS
1. This BTP was a great learning experience and
have gained much insights of the field.
2. Learning Outcomes Slides (GMMs, UNet, Mask
RCNN, Generative Models, AR) available at
http://bit.ly/VirtualTry
3. Report Available at http://bit.ly/VirtualTry
4. Special Thanks to Dr. Anand Mishra for the
constant support and encouragement.
5. We would be further continuing work in the
area, goal is to make it in hands of users.