MindFinder is an interactive sketch-based image search system that allows users to search over 2 million images using sketches, tags, and colors. It addresses limitations of prior sketch-based search by representing images as salient curves matched to user sketches. The system indexes sketches, tags, and colors for real-time multimodal search. An evaluation found sketch-based search to be more accurate and convenient than text for specific intents, demonstrating MindFinder's ability to help users better express search needs through simple sketches.

1. 2012.06.07
UX
Lab
LEE
NAMMIN
MM
'10
Proceedings
of
the
interna4onal
conference
on
Mul4media
MindFinder:
interac-ve
sketch-­‐based
image
search
on
millions
of
images
Yang
Cao,
Hai
Wang,
Changhu
Wang,
Zhiwei
Li,
Liquing
Zhang,
Lei
Zhang
@MicrosoI
Research
Keywords:
Query
formula4on Search
process User
interface
Informa4on
search Interac4ve
search
User
centered
design

2. MOTIVATION
| Is text-based query sufficient?
“어우. 이거 뭐라고 써야되지?”
a text query is usually too ambiguous to properly convey user’s search
intentions.

3. MOTIVATION
| 기존의 검색 방식들
•text-based : 많이 사용되고 있지만 심상 이미지를 텍스트 형태로 만들기란 어려움.
•content-based : 이미지 소스가 없으면 사용할 수 없음.
To overcome existing problems,
a natural solution is to enable users to flexibly express what they want.
| 어떻게?
•심상 이미지의 형태를 스케치하도록 해서,
•기존의 text-based 검색에서처럼 태그도 이용할 수 있게 해서,
•몇가지 지배적인 색을 지정할 수 있도록 해서,

4. INTRODUCTION
| Sketch-based
터치 디바이스 보편화 -> 스케치를 기반으로 한 기능이 동작하기에 충분한 환경이 마련됨.
among various query modalities, sketch is probably the most challenging one.
1990년부터 연구되어왔지만, 다음 세 가지 문제요소 때문에 뚜렷한 성과가 없었음.
| the 1st barrier
There is an unavoidable gap between the binary map sketched by the user and
full color natural images in the database.
| the 2nd barrier
the difficulty in meeting the requirement for both efficiency and accuracy when
matching a query sketch with curves in a natural image.
| the 3rd barrier
Most existing sketch-based search engines have no indexing mechanism.

5. INTRODUCTION
| revisit these problems, and have achieved very promising results.
| the 1st barrier : representation
To bridge the gap, we represent a natural image by its salient curves, which
has potential to be closer to the sketch queries from users.
| the 2nd barrier : matching
a raw curve-based algorithm is used to efficiently and precisely calculate the
similarity between the salient curve representation of natural images and a user’s
sketch query.
| the 3rd barrier : indexing
design an indexing strategy to speed up the matching process and make the
system scalable to millions of images.
: the first large-scale indexing framework

6. INTRODUCTION
| MindFinder
It is a real-time sketch-based image search engine.
the task suddenly becomes much easier.
by simply drawing the appearance of the pendant.
gives us the ability to better express our search intentions using a
simple sketch.

7. INTRODUCTION
| Mechanism
It enables users to flexibly express search intention by interactively
sketching, tagging and coloring.
over 2 million
web images

8. SYSTEM
OVERVIEW
| Interface

9. SYSTEM
OVERVIEW
| Sketch Querying
기본적으로 스케치로 찾는 기능이 메인.
스케치 패널에 심상 이미지의 윤곽선을 그리면 동적으로 결과패널이 업데이트 됨.
The sketch-based search in our system is precise and structure-sensitive, which
guarantees that the curves in natural images are highly matched with the strokes
drawn by users.

10. SYSTEM
OVERVIEW
| Sketch + Tag Querying
스케치 패널에서 텍스트 모드로 변경하고 타이핑하거나,
링바에 나타나있는 일반적으로 많이 사용하는 태그들을 스케치 패널에 드래그 앤 드랍
By only using the keyword-based search, top images were diverse and far from the
user’s intention. After adding a sketch query to confine the main structures, all top
results met the user’s requirement.

11. SYSTEM
OVERVIEW
| Sketch + Tag + Color Querying
지배적인 특정 색상에 대한 색상 쿼리를 입력하거나, 링 바에 있는 컬러 바에서 색을 선택
by specifying a certain dominant color, the user could easily find pictures of that in
any color she/he wants. Besides, the composition of the returned images still meet
the user’s search intention.

12. TECHNICAL
DETAILS
| back-end database
2,114,085 Flicker photos with tag information using the top 1000 hot queries.
downsampled each image to a suitable size
adopted a saliency discovering method to extract major curves
each full color images is transformed into a binary map
| matching
adopted an raw curve-based algorithm to achieve precise matching
| indexing
further speeded up by an index structure.
The tag and color features are indexed by inverted file structures, which totally
take less than 1GB memory. The sketch index occupies less than 7GB memory, and
thus MindFinder could be easily applied on a normal Intel machine.
all index structures were pre-built offline, by our well designed architecture, a
typical response time of a complex query is between 1 and 2 seconds.

13. TECHNICAL
DETAILS
| collaborate multimodal search conditions
A simple way,
각 조건에 따라 따로 찾아서 나중에 하나의 리스트로 머지하는 방법.
-> 규모가 큰 데이터베이스에서는 오버랩 되는 이미지들이 적을 것이기 때문에
머지했을 때 결과가 별로 없을 확률이 높음.
tag : to cross the semantic gap
To deal with this problem,
we use only one query type in the mixed query to retrieve an image set, and other
query types are worked as reranking conditions on this set.

14. RESULT
| search result

15. RESULT
| interactive searching

16. RESULT
| comparison between MindFinder & traditional search engine
the
sketch-­‐based
search
is
more
accurate
and
convenient
than
the
tradi-onal
search
when
a
user’s
search
intent
is
specific
and
complex.

17. CONCLUSIONS
| main contributions
•MindFinder
is
the
1st
sketch-­‐based
mul-modal
search
engine
for
more
than
two
million
web
images.
•Our
system
provides
a
convenient
interface
for
users
to
freely
express
their
search
inten-ons,
and
enables
real-­‐-me
interac-ons
for
users
to
more
efficiently
locate
their
desired
images.
•It
is
the
1st
index-­‐based
query-­‐by-­‐sketch
solu-on
for
million
level
database.

18. +
일단은,
뭔가 되게 반갑다!!

19. +
| approach
단순해 보이지만, 기본으로 돌아가서
“표현하기 어려우면 표현하는 것을 도와주면 되잖아!”
| process
왜 스케치 기반의 디자인을 생각하게 되었는지에 대해 궁금.
| mechanism
심상(목표)이 구체적이고 복잡한 경우로 한정지었음.
이미 상용화된거라면, 이 기능을 통해 이미지를 찾아가는 과정을 살폈다면
내 논문이 더 간결해졌을텐데! (심상 형성 -> 타협의 과정)
| contributions
간단한 서베이나 인터뷰 등의 UT를 통해 나온 데이터를 바탕으로 이야기했으면 좋았을 것!
우리가 쓴다면, 아마 이 논문의 앞단계의 사용자 조사와 디자인, 그리고 평가까지..
재밌었겠다는 상상

20. EOD
THANK
YOU