[SIGGRAPH 2021] Dynamic Closest Color Warping to Sort and Compare Palettes

THE PREMIER CONFERENCE & EXHIBITION IN
COMPUTER GRAPHICS & INTERACTIVE TECHNIQUES
DYNAMIC CLOSEST
COLOR WARPING TO
SORT AND COMPARE
PALETTES
© 2021 SIGGRAPH. ALL RIGHTS RESERVED.
SUZI KIM
SUNGHEE CHOI

WHAT IS A COLOR PALETTE?
A limited number of colors expressed in refined shapes
© 2021 SIGGRAPH. ALL RIGHTS RESERVED. 2

WHAT IS A COLOR PALETTE?
Intuitiveness
Generality
Easy-computation
Simplicity
A limited number of colors expressed in refined shapes

SORT THESE COLORS!

SORT THESE COLORS!
What if there are a lot of colors to be sorted?

SORT THESE COLORS!
What if there are a lot of colors to be sorted?
Or, what if you have to sort two palettes at the same time?

THIS PAPER INTRODUCES
considering both hue and lightness simultaneously
Single Palette Sorting
1
allowing palettes to be aligned concurrently
Multiple Palettes Sorting
2
reflecting sorted color tendencies
Similarity Measurement between Palettes
3

Lexicographical Sorting according to the Color Space
Previous Sorting Methods #1
LEXICOGRAPHICAL SORTING
RGB
HSV
VHS
LAB
Pros
Cons  Between lightness and hue, only one priority can be employed.
 Hue is a circular quantity with discontinuity at 0 and 360 degrees.
 Simple and easy-to-compute
Input Palette
Sort

IMAGE COMPRESSION
Purpose Reflecting pixels’ spatiality to reproduce the original as much as possible while
increasing compression ratio.
Methods  Geometric graph traversal by reducing to the traveling salesman problem (TSP).
- Greedy algorithm, Neural network, Simulated annealing, Adaptive particle swarm
optimization, Genetic algorithm, etc.
Cons  Visual plausibility is not taken into account.
 It cannot sort multiple palettes simultaneously.
Color Sorting for the Image Compression

BINARY PALETTE SORTING
Cons
Purpose To sort two sets of multiple palettes
Methods  Similar to merge sort
 (1) Partitioning two given palettes into a single palette and (2) repeatedly sorting
merging in a bottom-up manner.
 It assumes that each palette has a similar color distribution pattern.
 Because the colors are sorted only in a row-chunk unit, it is difficult to arrange the
multiple palettes naturally if they are composed of unorganized colors.
Binary Palette Sorting (BPS) [Phan et al. 2018]

METHODS

SORTING STRATEGY
• We define a natural order of colors as adjacent colors have
smooth hue and lightness changes.
Image source: mbaumi @Unsplash

SORTING STRATEGY
• To reflect both conditions, we use a multi-dimensional graph
traversal with colors as vertices and connecting orders as edges.
Image source: cosminserban @Unsplash

SORTING STRATEGY
• We adopt CIELAB color space which is more perceptually
uniform than other color spaces.
Image source: Holger kkk Everding
CC BY-SA @Wikipedia

SORTING STRATEGY
• We adopt CIELAB color space which is more perceptually
uniform than other color spaces.
• Graph traversal is reduced to symmetric TSP using CIEDE2000
distance.
Image source: n-voitkevich @Pexels

Method #1
SINGLE PALETTE SORTING

Method #1
TSP Solver
LKH

Method #1

Method #2
SORTING A PAIR OF PALETTES
What if you need to sort two palettes at the same time?

Method #2
What if you need to sort two palettes at the same time?
We arrange the color swatches to satisfy two conditions:
Overall color flow is similar when the palettes are placed side by side.
Each palette is naturally sorted.

Method #2

Method #2
TSP Solver
LKH
Where to Cut?

Method #2
SPS Cutting
Strategy
TSP Solver
LKH

Method #2
SPS Cutting
Strategy
Input Palettes
Sorted Palettes
Merged Palettes
The most different pair of consecutive colors.
Radical color change cannot be avoided!

Method #2
Input Palettes
Cutting Strategy for a Pair of Palettes
 Arranging the two palettes in a similar color flow.
 Preventing sudden color changes in each palette.
Starting from the point of view of individual palettes.
Purpose
Methods

Method #2
1. Find the longest edge connecting the same palette colors.

Method #2
𝒄𝒂
𝒄𝒃
2. Among the two subpaths connecting 𝑐𝑎 and 𝑐𝑏, we choose the
one with fewer color vertices.

Method #2
𝒄𝒂
𝒄𝒃
3. To decide which side the remaining colors should be bound
to, we find a point 𝑐𝑘 that minimizes
where 𝑎 ≤ 𝑘 ≤ 𝑏.
2. Among the two subpaths connecting 𝑐𝑎 and 𝑐𝑏, we choose the
one with fewer color vertices.
𝑖=𝑎+1
𝑘
𝑑 𝑐𝑎, 𝑐𝑖 +
𝑖=𝑘+1
𝑏−1
𝑑(𝑐𝑏, 𝑐𝑖) ,
𝒄𝒌
𝒄𝒌+𝟏

Method #2
Input Palettes
Sorted Palettes
Merged Palettes

Method #2
Input Palettes
Sorted Palettes
SPS Cutting Strategy
PPS Cutting Strategy

Method #2
Input Palettes PPS Cutting Strategy SPS Cutting Strategy

Method #2
Palettes with Different Length More than Two Palettes

Method #3
PALETTE SIMILARITY MEASUREMENT
How much do they look similar?

Method #3
Previous Similarity Measurements
Element-wise Similarity

Method #3
Element-wise Similarity Flow-based Similarity

Method #3
We proposed a Dynamic Closest Color Warping (DCCW).
− Geometric simularity
− Not element-wise, but flow-based measurement
− Regarding a palette as a sequence, and then measure
the distance between two sequences
Input Pair Sorted Pair
Geometric Arrangement

Method #3
Sort
Input Palettes
Sorted Palettes

Method #3
Sorted Palettes
𝐷𝐷𝐶𝐶𝑊 𝐴, 𝐵 =
𝑊 𝐴, 𝐵 + 𝑊(𝐵, 𝐴)
𝐴 + 𝐵
𝑊 𝐴, 𝐵 =
𝑎∈𝐴
min
𝑏𝑖∈𝐵
𝑑(𝑎, 𝑏𝑖𝑏𝑖+1)
A
B
𝒂𝟏 𝒂𝟐 𝒂𝟑 𝒂𝟒 𝒂𝟓 𝒂𝟔 𝒂𝟕 𝒂𝟖 𝒂𝟗 𝒂𝟏𝟎
𝒃𝟏 𝒃𝟐 𝒃𝟑 𝒃𝟒 𝒃𝟓 𝒃𝟔 𝒃𝟕 𝒃𝟖 𝒃𝟗 𝒃𝟏𝟎
𝑑(𝑎, 𝑏𝑖𝑏𝑖+1)
𝒃𝒊+𝟏
𝒃𝒊
𝒂
The shortest Euclidean distance between
CIELAB color 𝑎 and line segment bi𝑏𝑖+1.

EVALUATION

EVALUATION OVERVIEW
SPS
Palette Pair Sorting
PPS
Similarity Measurement
DCCW
 How naturally the colors are aligned
 How well the colors are aligned between palettes
 How naturally the colors are aligned in each palette
 How effectively DCCW rates similar palettes as similar
 How effectively DCCW rates dissimilar palettes as dissimilar

Evaluation #1
SPS
If there is a correctly ordered palette,
we can evaluate how well SPS aligned a shuffled palette comparing to a ground-truth palette.
Ground Truth
(Correctly Ordered)
Sorted Palette

Farnsworth-Munsell 100 Hue Test
• It is designed to test for color blindness, with 85 different hues.
• It tests the ability to distinguish and arrange minute hue differences under constant value
and chroma.
Dataset #1: Farnsworth-Munsell 100 Hue Test Palette Dataset (FM100P)
Evaluation #1

Evaluation #1
Farnsworth-Munsell 100 Hue Test
Dataset #1: Farnsworth-Munsell 100 Hue Test Palette Dataset (FM100P)
Correctly Sorted Palette
(Ground Truth)
We generated several FM100P types with
palette lengths 10, 15, 20, 25, 30, 35, and 40,
each comprising 100 palettes.
Maximum Distance: 5

📐 How can we measure the accuracy of the sorting result?
Ground
Truth
(GT)
Target
Success = The order of the GT and target palette is identical.
Success Rate = (# of successes) / (# of tests)
1. Success Rate
The longest subsequence length for the sorted elements, where
the subsequences are not necessarily contiguous in the sequence
3. Length of the Longest Increasing Subsequence (LLIS)
Measuring the difference between two sequences.
2. Levenshtein Distance
To measure differences
between two palettes
even if the sorted order
is not perfectly correct.
Evaluation #1
SINGLE PALETTE SORTING: ACCURACY

Evaluation #1
SINGLE PALETTE SORTING: RESULT
CIELAB HSV GA FIA SA ACO-50 ACO-10 ACO-2 LKH (Ours)
SR 0.001 0.829 0.201 0.011 0.020 0.733 0.631 0.540 0.789
LD 0.884 0.014 0.760 0.849 0.874 0.562 0.562 0.574 0.563
LLIS 0.488 0.993 0.693 0.623 0.463 0.967 0.954 0.941 0.966
ET 0.001 0.001 0.201 0.129 0.294 4.889 1.035 0.552 0.169
Since FM100P was originally sorted by
hue value, it is expected that HSV
lexicographical sorting performs best
Time Complexity of ACO: 𝑂(𝐼n2 𝑚)
Time Complexity of LKH: 𝑂(𝐼𝑛2.2
)
I: # of iterations, m: the number of ants
SR: Success Rate
LD: Levenshetin Distance
ET: Elapsed Time

EVALUATION OVERVIEW
SPS
PPS
DCCW

Evaluation #2
PALETTE PAIR SORTING: DATASET
PPS
FM100P for SPS Evaluation

Evaluation #2
hue
tone
Kobayashi Hue & Tone 130 System
• It is designed for color semantics based on the
Munsell color system and ISCC-NBS color
naming system.
Neutral
Dataset #2: Kobayashi Hue & Tone Palette Dataset (KHTP)

Evaluation #2
hue
tone
To avoid bipartite matching, we varied row-grouped
palettes by (1) mixing, (2) jittering, and (3) interpolation.
Original Palette Sampling

Mixing changes each palette color into a different color in
the same column with a maximum adjacent index
difference of 2.
Evaluation #2
hue
tone
Mixing Colors
Mixing

Evaluation #2
𝛼 = 5
𝛼 = 10
𝛼 = 15
Jittering
Jittering adds an arbitrary offset, Ω(−𝛼, 𝛼), to each color
to provide minute color changes.

Evaluation #2
Interpolation
Interpolation adds additional colors to
the palette to disrupt bipartite matching.
#Interpolation = 1
#Interpolation = 3
#Interpolation = 5

Evaluation #2
PALETTE PAIR SORTING: ACCURACY
📐 How can we measure whether a pair of palettes are correctly sorted?
Ground Truth Pair
Sorted Pair
A1 A2 A3 A4 A5 A6 A7 A8 A9 A10 A11
B1 B2 B3 B4 B5 B6 B7 B8 B9 B10 B11
A3 A7 A6 A9 A2 A5 A8 A10 A4 A11 A1
B9 B3 B5 B6 B11 B1 B7 B4 B8 B2 B10

Evaluation #2
 (Concurrency) How well the colors are aligned between palettes
LLCS indicates the length of the longest subsequence, which
is common to two sequences, and the subsequence is not
necessarily consecutive.
2. Length of the longest common subsequence (LLCS)
Measuring the difference between two sequences.
1. Levenshtein Distance
Sorted
Pair
A3 A7 A6 A9 A2 A5 A8 A10 A4 A11 A1
B9 B3 B5 B6 B11 B1 B7 B4 B8 B2 B10

Evaluation #2
 (Naturalness) How naturally the colors were aligned in each palette
Sorted A3 A7 A6 A9 A2 A5 A8 A10 A4 A11 A1
B9 B3 B5 B6 B11 B1 B7 B4 B8 B2 B10
Same metric with SPS
4. LLIS of Each Palette
Measuring the difference between sorted and ground truth
palette
3. Levenshtein Distance of Each Palette
Ground Truth
A1 A2 A3 A4 A5 A6 A7 A8 A9 A10 A11
B1 B2 B3 B4 B5 B6 B7 B8 B9 B10 B11
Sorted
Ground Truth

Evaluation #2
PALETTE PAIR SORTING: RESULT
Ours BPS I-BPS
Naturalness LD 0.631 0.783 0.651
LLIS 0.486 0.455 0.475
Concurrency LD 0.375 0.603 0.430
LLCS 0.769 0.560 0.738
Accuracy of the Palette Pair Sorting
LD: Levenshetin Distance

Evaluation #2
PALETTE PAIR SORTING: RESULT
w/o cutting strategy w/ cutting strategy
Δ𝑚𝑎𝑥 Δ𝑎𝑣𝑔 Δ𝑚𝑎𝑥 Δ𝑎𝑣𝑔
w/o jittering and interpolation 27.947 12.767 26.902 12.743
w/ jittering only 29.285 16.529 28.463 16.465
w/ interpolation only 28.438 13.076 27.561 13.071
w/ jittering and interpolation 29.015 16.346 27.901 16.257
𝛥𝑎𝑣𝑔: the average difference of adjacent colors
𝛥𝑚𝑎𝑥: the maximum color difference
Effects of the Cutting Strategy

EVALUATION OVERVIEW
SPS
PPS
DCCW

Evaluation #3
DCCW: DATASET
DCCW
FM100P for SPS Evaluation
KHTP for PPS Evaluation

Evaluation #3
DCCW: DATASET
Lin and Hanrahan’s Palettes Dataset
• [Lin and Hanrahan 2013] generated a color palette dataset to model human’s color palette
recognition from images.
• The dataset comprises unsorted palettes extracted from 40 images.
• by humans including artists and Amazon Mechanical Turk’s participants
• by machines using k-means, c-means clustering, and their extraction model.
Dataset #3: Lin and Hanrahan Similarity Palettes (LHSP)

Evaluation #3
DCCW: DATASET
Lin and Hanrahan’s Palettes Dataset
• [Lin and Hanrahan 2013] generated a color palette dataset to model human’s color palette
recognition from images.
• The dataset comprises unsorted palettes extracted from 40 images.
• by humans including artists and Amazon Mechanical Turk’s participants
• by machines using k-means, c-means clustering, and their extraction model.
Their
Models Amazon MTurk Artists
Source
Colors
Source Image

Evaluation #3
DCCW: DATASET
• To make palettes composed of similar but different colors
• Replacement: replaced 10% of the colors of the palette with arbitrary colors.
• Jittering: jittered the colors with jittering 𝛼 ranges = 0, 5, 10, 15
Source Image Example of Generated Palettes

Evaluation #3
DCCW: DATASET
• We divided the palettes from the same image into disjoint query and retrieval sets.
• We generated several LHSP types with palette lengths 5, 10, 15, 20, 25, and 30.
• LHSP with palette lengths 10~30:
• 50 query palettes (from 10 images)
• 400 retrieval palettes (from 40 images including 10 query images)
• LHSP with palette length 5
• 10 query palettes (from 10 images)
• 1,600 retrieval palettes (from 40 images including 10 query images)

Evaluation #3
DCCW: RESULT
📐 How well similar palettes from the same image are retrieved among the dataset.
DCCW MPD HD MHD LTS-HD MCD MPHSM CEMD MBME DTW
w/o jittering and
replacement
0.888 0.219 0.571 0.874 0.888 0.880 0.016 0.806 0.813 0.811
w/ jittering only 0.785 0.206 0.524 0.771 0.701 0.763 0.015 0.713 0.713 0.712
w/ replacement only 0.857 0.208 0.206 0.845 0.874 0.857 0.016 0.777 0.788 0.784
w/ jittering and
replacement
0.674 0.189 0.197 0.662 0.631 0.667 0.016 0.630 0.631 0.630
Mean Average Precision (mAP) for LHSP

Evaluation #3
DCCW: RESULT
DCCW MPD HD MHD LTS-HD MCD MPHSM CEMD MBME DTW
𝑘 = 5 0.618 0.152 0.345 0.571 0.509 0.588 0.017 0.454 0.457 0.418
𝑘 = 10 0.549 0.176 0.304 0.529 0.484 0.522 0.016 0.429 0.432 0.447
𝑘 = 15 0.683 0.189 0.293 0.678 0.654 0.671 0.015 0.626 0.629 0.632
𝑘 = 20 0.862 0.217 0.371 0.849 0.828 0.848 0.015 0.801 0.804 0.818
𝑘 = 25 0.917 0.237 0.418 0.923 0.904 0.923 0.015 0.924 0.925 0.924
𝑘 = 30 0.963 0.238 0.474 0.963 0.939 0.966 0.015 0.976 0.976 0.976
Mean Average Precision (mAP) for LHSP according to different palette size 𝑘
📐 How well similar palettes from the same image are retrieved among the dataset.

Evaluation #4
PERCEPTUAL STUDY
Ours
 Verifying that our method is superior for palette sorting
Purpose
Method  Amazon Mechanical Turk
Questionnaire SPS
Q. Which one do you think is arranged in the most natural order?
HSV-based Sorting
CIELAB-based Sorting

Evaluation #4
PERCEPTUAL STUDY
Ours
 Verifying that our method is superior for palette sorting
Purpose
Method  Amazon Mechanical Turk
Questionnaire PPS
Q. Which pair do you think is arranged in the most natural order?
HSV-based Sorting
CIELAB-based Sorting
BPS
I-BPS

Evaluation #4
PERCEPTUAL STUDY: RESULT
𝑘 = 10 𝑘 = 15 𝑘 = 20
SPS Ours > HSV-based sorting 79.9% 85.9% 86.7%
Ours > CIELAB-based sorting 69.3% 63.9% 64.4%
PPS Ours > HSV-based sorting 82.9% 88.8% 92.1%
Ours > CIELAB-based sorting 64.1% 60.9% 59.8%
Ours > BPS 86.5% 89.0% 91.4%
Ours > I-BPS 67.4% 64.9% 71.0%
* The effects of palette length and sorting methods were determined using a two-way repeated
measures ANOVA with a significance level of 𝑝 < 0.001 for both SPS and PPS results

APPLICATIONS

Application #1
PALETTE INTERPOLATION
HSV-based
CIELAB-based
DCCW-based

Application #2
PALETTE NAVIGATION
Kiki's Delivery Service
(1989)
Swallow (2019)
Call Me by Your Name
(2017)
The Place Beyond
the Pines (2012)
Pan's Labyrinth (2006)
Dunkirk (2017)
Query Palette Unsplash
WikiArt
COLOURlovers Commercial Movie

Django Unchained (2012)
Swallow (2019)
The Master (2012)
The Neon Demon (2016)
Titanic (1997)
World War Z (2013)
Application #2
PALETTE NAVIGATION
Query Palette Unsplash
WikiArt
COLOURlovers Commercial Movie

Application #3
PALETTE-BASED RECOLORING
Recoloring
Convex Hull
Deformation
Original Image
Extracted Palette
Convex Hull
Target Palette Simple Transformation Extending [Tan et al. 2018]

Application #3
Original Image
Target Palette
Improved Transformation
Recoloring Result
Convex Hull
Deformation
Source (Extracted)
Target
1
Sorted Source
Sorted target
2
Nearest Matching Using DCCW
Closest Colors
from Target to Source
3
Sorted Target
4
Buffered Palette
Merging

Application #3
Target Palette
Simple
Transformation
Improved
Transformation
Original Image Target Palette
Simple
Transformation
Improved
Transformation
Original Image

CONCLUSION

CONCLUSION
• We proposed a method to sort palette colors by traversing color graphs.
• We proposed a dynamic closest color warping (DCCW) similarity measurement for palette comparison.
• We evaluated our palette sorting and DCCW on various datasets using various criteria confirming ours
outperformed previous methods in terms of accuracy and computation time.
• We also evaluated our methods using a perceptual study, and the result clearly indicates that ours generate
perceptually natural sorting results, outperforming previous techniques.
• We demonstrate many compelling applications enabled by DCCW, including palette interpolation, palette
navigation, and image recoloring.
• Source code and dataset are available at:
• https://github.com/SuziKim/DCCW / https://github.com/SuziKim/DCCW-dataset

THANK YOU!
SUZI KIM AND SUNGHEE CHOI
GEOMETRIC COMPUTING LAB.
SCHOOL OF COMPUTING, KAIST
DYNAMIC CLOSEST COLOR WARPING
TO SORT AND COMPARE PALETTES

[SIGGRAPH 2021] Dynamic Closest Color Warping to Sort and Compare Palettes

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to [SIGGRAPH 2021] Dynamic Closest Color Warping to Sort and Compare Palettes

Similar to [SIGGRAPH 2021] Dynamic Closest Color Warping to Sort and Compare Palettes (17)

More from Suzi Kim

More from Suzi Kim (9)

Recently uploaded

Recently uploaded (20)

[SIGGRAPH 2021] Dynamic Closest Color Warping to Sort and Compare Palettes