Presentation for meetup http://www.meetup.com/Liferay-Budapest-Tech-Meetup/events/229996198/ Liferay Hungary Kft., Budapest, 2016-04-20

1. In Search of...
A topic for a meetup :-)
Tibor Lipusz
Lead Software Engineer, SME of Search & Security @ Subscription Services
2016-04-20, Budapest

2. Agenda
Understanding Search
Technology Stack
Brief Overview of Search in Liferay

3. Understanding Search

4. Why do we search?

5. Understanding Search
Why do we search?
Because we don’t know
everything :-)

6. Understanding Search
Why do we search?
Basic Human instinct We want answers→

7. Understanding Search
Why do we search?
Basic Human instinct We want answers→
Humans are information-driven Knowledge→
is key to survive

8. Understanding Search
Why do we search?
Basic Human instinct We want answers→
Humans are information-driven Knowledge is key to survive→
Brainless Modeling the world→

9. Understanding Search
Why do we search?
Basic Human instinct We want answers→
Humans are information-driven Knowledge is key to survive→
Brainless Modeling the world→
The River and the Fish Information is fooding us, but most of→
the time we’re interested in only a specific slice of the“big cake”

10. What do we search for?

11. Understanding Search
What do we search for?
Everything! :-)

13. Understanding Search
What do we search for?
Places

14. Understanding Search
What do we search for?
Images

15. Understanding Search
What do we search for?
Books

16. Understanding Search
What do we search for?
Science (Answers)

17. Understanding Search
What do we search for?
Articles

18. Understanding Search
What do we search for?
Documents

19. Understanding Search
What do we search for?
Text

20. Understanding Search
What do we search for?
And many more… :-)

21. How do we search?

22. Understanding Search
How do we search?
The good-old (search) input box

23. Understanding Search
How do we search?
Search box with suggestions

24. Understanding Search
How do we search?
Faceted Search (guided navigation)

25. Understanding Search
“The Search”
Input: Search
Keywords MAGIC!
Output: Search Results

26. Understanding Search
“The Search”
It’s so simple, huh? :-)

27. Understanding Search
“The Search”
Unfortunately, it’s not… :-(

28. Understanding Search
“The Search”
Input: Search
Keywords MAGIC!
Output: Search Results

29. Understanding Search
“The Search”: User Input
What makes it complex?
- Creating an intuitive yet powerful frontend (UI)
- Expose syntax, but not overwhelm end-user
- Features to provide:
- Faceted Search
- Search suggestions, Live Search
- Related searches
- Default View: Basic or Advanced
- Type of user input: keywords, image, geolocation (coordinates)

30. Understanding Search
“The Search”
Input: Search
Keywords MAGIC!
Output: Search Results

31. Understanding Search
“The Search”: Magic
Magic = Actual Search (sorry, no real magic)

32. Understanding Search
“The Search”: Magic
What makes it complex?
- Different types of content → Abstraction, Transformation
- Different repositories → Integration
- Language characteristics ( Site & Content Localization)←
- Performance, Permissions, restrictions → Operation, Implementation, Business needs
- Mapping user-thinking → Queries
- Scaleability → Operation
- Finding the good platform → Architecture

33. Understanding Search
“The Search”
Input: Search
Keywords MAGIC!
Output: Search
Results

34. Understanding Search
“The Search”: Search Results
What makes it complex?
- Creating an intuitive yet powerful frontend (UI)
- Displaying results
- Filtering
- Pagination
- Ordering
- Highlighting
- Snippets (summaries)
- Navigation (to the actual content)

35. Understanding Search
“The Search”: ?
Didn’t we miss an important step?

36. Understanding Search
“The Search”: Content
The world is not ideal →

37. Understanding Search
“The Search”: Content
The world is not ideal →
Data comes in different forms →

38. Understanding Search
“The Search”: Content
The world is not ideal →
Data comes in different forms →
Search should operate on a common
form

39. Understanding Search
“The Search”: Content
The world is not ideal →
Data comes in different forms →
Search should operate on a common form
← Transparency for end-user

40. Understanding Search
Summary
What we need?
- Data Transformation → Pre-Search
- Search Backend → Search → Magic :-)
- Search Frontend → Couldn’t find a good word for this, but writing something just
to keep the format :-)

41. Technology Stack

42. Technology Stack

43. Technology Stack
LuceneTM
Full-featured text search engine library
Java-based
Document model
Provides indexing, scoring, spellchecking, hit highlighting and advanced
analysis/tokenization capabilities
Provides a wide range of queries
Open Source (Apache License 2.0)

44. Technology Stack
Lucene: The Life Inside the Search Engine
The (inverted) Index

45. Technology Stack
Lucene: The Life Inside the Search Engine
The Index Documents

46. Technology Stack
Lucene: The Life Inside the Search Engine
The Index Documents Fields

47. Technology Stack
Lucene
Document model Abstraction over data→
Main Phases
- Indexing: creating documents
- Analysis and Analyzers: tokenizing, stemming, punctuation etc.
- Searching: getting hits
- Analysis!
- Queries, Scoring (relevance), Boosting, Facets etc.

48. Technology Stack
Lucene
Why would we prefer this over databases?

49. Technology Stack
Lucene: Search Index vs. DB
- Different purposes
- DB: permanent storage, data-driven, application-centric
- Search Engine, Index: optimized for search, user-centric
- Search in search engine is about relevance
- Database searches does not provide us with fuzzy searching or any type of relevancy
- Can apply algorithms like“More Like This”to obtain similar content
- Advanced Features
- Geolocation
- Faceting of results
- Multi-lingual searching

50. Technology Stack

51. Technology Stack
Solr
Highly reliable, scalable and fault tolerant
Provides distributed indexing, replication and load-balanced querying, automated
failover and recovery, centralized configuration and more
Standalone enterprise search server (web application) with a REST-like API
Wide range of clients
Apache Foundation

52. Technology Stack
Solr: Features

53. Technology Stack
Solr: Features

54. Technology Stack
Solr: Features

55. Technology Stack

56. https://www.elastic.co/products/elasticsearch
Highly scalable, distributed, open-source full-text search and analytics engine (server)
Clustered by Design
Extensible with plugins
Technology Stack
Elasticsearch

57. Technology Stack
Elasticsearch: Features
(Near) Real-Time Data*: all data is immediately made available for search and
analytics.

58. Technology Stack
Elasticsearch: Features
Massively Distributed: ES allows you to start small and scale horizontally as you grow.
Simply add more nodes, and let the cluster automatically take advantage of the extra
hardware

59. Technology Stack
Elasticsearch: Features
High Availability: ES clusters are resilient — they will detect new or failed nodes, and
reorganize and rebalance data automatically

60. Technology Stack
Elasticsearch: Features
Multitenancy: A cluster may contain multiple indices that can be queried
independently or as a group. Index aliases.

61. Technology Stack
Elasticsearch: Features
Full-Text Search: ES builds distributed capabilities on top of Apache Lucene to provide
the most powerful full- text search capabilities

62. Technology Stack
Elasticsearch: Features
RESTful API: ES is API driven. Almost any action can be performed using a simple
RESTful API using JSON over HTTP

63. Technology Stack
Elasticsearch: Features
Open Source: Apache 2 License

64. Technology Stack
Elasticsearch: Features
Built on top of Lucene: Apache Lucene is a high performance, full-featured
Information Retrieval library, written in Java

65. Technology Stack
Elasticsearch: Architecture
Elasticsearch
Cluster
Application
DB
Search
Index

66. Technology Stack
Elasticsearch
Elasticsearch
Cluster

67. Technology Stack
Elasticsearch: Cluster

68. Technology Stack
Elasticsearch: Nodes

69. Technology Stack
Elasticsearch: Index

70. Technology Stack
Elasticsearch: Shards

71. Technology Stack
Elasticsearch: Lucene Index

72. Brief Overview of Search in Liferay

73. Search in Liferay
The Liferay Search Infrastructure
Liferay
Platform
Assets:
web content,
message boards,
wiki pages...
Search
infrastructure
(Magic
happens
here)
Search engine(s)
Indices,
documents,
analysis...

74. → 2004: Lucene: from the very beginnings
- Liferay Search APIs: built around/on top of Lucene, default engine
→ 2008 (5.1): abstracting out search mechanism → Solr integration (plugin)
→ 2011 (6.0): Faceted Search Support
Search in Liferay
The Evolution of Search

75. → 2011-2014: (6.0-6.2): enhancing search functionalities
- Suggestions,“Did You Mean”, filtering, improved permission awareness
→ 2016 (7.0): Elasticsearch integration
- Default search engine
- Generic queries
- OSGi module
- Partnership with Elastic
Search in Liferay
The Evolution of Search

76. Best of breed
Built for modern web applications
Distributed and clusterable by design
Lucene based
Great vendor support
Great monitoring tools: Marvel
Search in Liferay
Why Elasticsearch?

77. Open Source
Amazing documentation
High "just works" factor, e.g. zero-config indexing and clustering
REST for queries, health, admin - everything
Great Java Client API
Pretty JSON for talks ;-)
Search in Liferay
Great for Developers

78. Search in Liferay
Clustering with Liferay and Elasticsearch
Production mode
Dev mode

79. Security
Monitoring
Search in Liferay
Liferay 7 EE Search Features

80. Shield | Security for Elasticsearch
Protect your Liferay index with a username and password
SSL/TLS encryption for traffic within the Liferay Elasticsearch cluster
Elasticsearch plugin - no need for an external security solution
Restrict access to Liferay Portal instances with IP filtering
Liferay Integration
Search in Liferay
Liferay 7 EE Search Features: Security

81. Marvel | Monitor for Elasticsearch
Real-Time and Historical Analysis
Real-Time Cluster Health at a Glance
Multicluster Support
Liferay Integration
Search in Liferay
Liferay 7 EE Search Features: Monitoring

83. Details are coming soon...
Search in Liferay
Liferay 7 EE Search Features: Licensing

84. Resources
André de Oliveira: Harnessing the power of search, Liferay DEVCON, 2015
Michael Han: Search Reference documentation (Liferay Internal Documentation)
Tibor Lipusz: Introduction to Indexing & Searching, Faceted Search in Liferay (Liferay Internal Training)
Tibor Lipusz: Introduction to Search: Elasticsearch and Solr (Liferay Internal Training)
Source of Images, Figures:
www.liferay.com
https://dev.liferay.com
www.amazon.com
www.google.com
http://lucene.apache.org
www.elastic.co
http://lucene.apache.org/solr/

85. Thanks!
tibor.lipusz@liferay.com
https://github.com/lipusz