The document discusses a case study of building an open source search engine. It describes how an open source search engine was implemented for a company to address issues with decreasing click-through rates and lack of technical knowledge. Elasticsearch was chosen as it allows for real-time search and analytics, is distributed and highly available. The implementation involved using Zookeeper for node management, caching for performance improvements, and custom plugins for functions like scoring. This allowed the company to gain technical capabilities, provide differentiated search, reduce costs, and better manage search quality and service integration.

1. ©2014 Gruter, Inc. All rights reserved.
오픈소스검색엔진구축사례
2014.10.29
정호욱책임연구원

2. ©2014 Gruter, Inc. All rights reserved.
About me
•Product of Search
–Sprinter & Search Appliance
•그루터
–Search AdsMatchingPlatform
•삼성전자
–Instant Message/Mail Search
•NHN Technology Services
–Social Search
•야후! 코리아
–…
•Book
–실무예제로배우는Elasticsearch검색엔진-입문편
2

3. ©2014 Gruter, Inc. All rights reserved.
오늘의주제
•왜ElasticSearch인가? -짧게
•오픈소스검색엔진의도입과서비스적용사례
•SQL on ElasticSearch맛보기–시간남으면
3

4. 이거밖에없나???

6. ©2014 Gruter, Inc. All rights reserved.
왜ElasticSearch인가?
•Real time search & analytics
•Distributed & high availability
–cluster, node, data
•Multi tenancy
•Schema free & document oriented
•Data management
–Version, translog
•RESTfulAPI & pluggable modules
•Noconfiguration
6

7. 오픈소스검색엔진구축사례
살펴볼까요?

8. ©2014 Gruter, Inc. All rights reserved.
뭐가문제인가요?
•이용자는증가하고있는데CTR, CR은감소
•경쟁사대비차별성이없음
•검색서버증설시라이선스비용지출이큼
•자체사전데이터가없음
•기술과방법을잘모름
8

9. ©2014 Gruter, Inc. All rights reserved.
이런걸하고싶어요.
•기술내재화를하고싶어요
•차별화된검색서비스를하고싶어요
•서버증설에돈좀안들었으면좋겠어요
•한번구축한인프라로서비스통합및확장이가능하면좋겠어요
9

10. ©2014 Gruter, Inc. All rights reserved.
그럼해드려야죠!!
•검색서비스architecture구축
•Targeting 검색기능구현
•Function scoring 기능구현
•검색품질평가측정방법
–DCG 기법도입, A/B test 수행
•교육및기술이전
10

11. ©2014 Gruter, Inc. All rights reserved.
Sprinter를이용하였습니다.
•오픈소스기반인터넷서비스용검색엔진
11
Sprinter (Serving layer)
Processing layer
Search Engine
(Indexer/Searcher)
Document proc. pipe
API proxy/Cache
Presentation
(multi modules)
Smart Query
Multi fetcher
Smart Q
fetcher
Other search
modules
Result
optimizer
…
NLP
Meta
정보
Data Source Layer
Data Source
(Streaming
Or
Storage)
Front end
인덱스아카이빙
(낮은질의성능)
Back end
그루터솔루션
오픈소스
프로젝트개발

12. ©2014 Gruter, Inc. All rights reserved.
검색서비스에적용된구성입니다.
12
L4
질의분석및계획
Proxy
Query & Result Cache
Zookeeper
Elasticsearch
Cluster
+
Custom Plugin
www.….com
m.….com
auto completion
관리도구
Index Requester
Collector
쇼핑몰
DB
형태소분석기
그루터솔루션
오픈소스
프로젝트개발

13. ©2014 Gruter, Inc. All rights reserved.
특별히사용된기법이나기술은?
•Zookeeper를이용한node관리
•Cache cluster를이용한성능개선
•Plugin을이용한external analyzer 등록
•Plugin을이용한customfunction score 등록
•DCG를이용한검색품질평가
13

14. ©2014 Gruter, Inc. All rights reserved.
어려운점은없었나요?
•개인화추천데이터적용을위한모델링
•키워드추천데이터적용을위한모델링
14

15. ©2014 Gruter, Inc. All rights reserved.
어떻게바뀌었을까요?
15
상용엔진
오픈소스엔진

16. ©2014 Gruter, Inc. All rights reserved.
오픈소스검색엔진을도입후바뀐건?
•기술내재화가되었고요
•차별화된검색서비스를하게되었고요
•비용을절감했고요
•검색품질관리를직접하게되었고요
•서비스통합및연동을쉽게할수있게되었고요
16

17. ©2014 Gruter, Inc. All rights reserved.
SQL on ElasticSearch맛보기
•바로데모를보시겠습니다.
17

18. ©2014 Gruter, Inc. All rights reserved.
별첨. Upgraded Sprinter
•오픈소스기반인터넷서비스용검색엔진
18
Sprinter (Serving layer)
Processing layer
Search Engine
(Indexer/Searcher)
Document proc. pipe
API proxy
/ Cache
Presentation
(multi modules)
Smart Query
Multi fetcher
Smart Q
fetcher
Other search
modules
Result
optimizer
…
NLP
Meta
정보
Data Source Layer
Data Source
(Streaming
Or
Storage)
Front end
인덱스아카이빙
(낮은질의성능)
Back end
그루터솔루션
오픈소스
프로젝트개발
HDFS
ESBaseLoader
ESBase
JDBC/CLI

19. GRUTER: YOUR PARTNER
IN THE BIG DATA REVOLUTION
Phone +82-70-8129-2950
Fax+82-70-8129-2952
E-mail contact@gruter.com
Webwww.gruter.com
©2014 Gruter, Inc. All rights reserved.
Gruter, Inc.
5F SehwaOffice Building 889-70 Daechi-dong, Gangnam-gu, Seoul, South Korea 135-839