TLDR (Twin Learning for Dimensionality Reduction) is an unsupervised dimensionality reduction method that combines neighborhood embedding learning with the simplicity and effectiveness of recent self-supervised learning losses.

1. TLDR
TWIN LEARNING FOR DIMENSIONALITY
REDUCTION

2. Unsupervised approach that learn low-dimensional spaces, where some properties of the
initial space, typically the notion of “neighborhood”, are preserved.
Dimension Reduction method is the technique of converting a set of data
having vast dimensions into data with lesser dimensions with an
assurance that it conveys similar information concisely.
Like other neighborhood embeddings, TLDR effectively and un-supervisedly learns low-
dimensional spaces where local neighborhoods of the input space are preserved.
Unlike other manifold learning methods, it simply consists of an offline nearest neighbor
computation step and a straightforward learning process.
It does not require mining negative samples to contrast, eigen decompositions, or
cumbersome optimization solvers.
TLDR is of broad applicability, simple, easy to implement and train.
It aims for scalability by focusing on improving the linear dimension reduction.
TLDR

3. TLDR Architecture
TLDR

4. Steps for TLDR:
1. Start with a set of unlabeled and high-dimensional features.
2. We will use nearest neighbors to define a set of feature pairs whose distance or
proximity we want to preserve.
3. Learn the parameters of a dimensionality reduction function i.e., the encoder,
using a loss that defines neighbors in the input space to have similar
representations.
4. Append a projector to the encoder that produces a representation in a very high
dimensional space.
5. At the end of the learning process, discard the projector.
TLDR

5. Preserving local neighborhoods:-
 Practically define the local neighborhood of each training sample as its k
nearest neighbors. It is experimentally shown that it is not only sufficient, but
also that TLDR algorithm is robust across a wide range of values for k.
Learning à la Barlow Twins:-
 Contrastive losses were proven highly successful for visual representation
learning, explicitly minimizing the redundancy of the output dimension is
highly desirable for dimensionality reduction.
 So, we choose to learn the parameters of our encoder by minimizing the Barlow
Twins loss function that suits the requirements perfectly.
The Encoder:-
 For encoder, we consider several different architectures:- Linear, Factorized
Linear and Multi-layer Perceptron (MLP).
 Since our main goal is to develop a scalable alternative to PCA for
dimensionality reduction, we will mostly focus in linear and factorized linear
encoders.
TLDR

6. The Projector:-
 Projector is present in several contrastive self-supervised learning methods.
 Unlike other methods where the projector takes the representations to an even
lower dimensional space for the contrastive loss to operate on.
 For the Barlow Twins objective, operating in large output dimensions is crucial.
TLDR

7. Conclusion
 TLDR performs brilliantly for dimensionality
reduction to mid-size outputs, especially when
dimension d is in range of 32 to 256 dimensions.
 Very useful in practice for Landmark Image Retrieval
/ Document Retrieval and a set of output dimensions
where most manifold learning methods are not
scalable.
 Enables to utilize a powerful learning framework that
was initially tailored for visual representation
learning in different domains like natural language.
TLDR