DBSight Instant Scalable Full Text Search on Any Databases
Some Customers <ul><ul><li>Websites </li></ul></ul><ul><ul><ul><li>Workopolis.com, biggest job site in Canada </li></ul></...
Features <ul><ul><li>Add Search when you want to </li></ul></ul><ul><ul><ul><li>Database Independent </li></ul></ul></ul><...
The Pain to Add Search <ul><ul><li>Do you have it? Common, but slow SQL,  </li></ul></ul><ul><ul><ul><li>select * from tab...
DBSight Design Goals <ul><ul><li>Very fast to create and adjust </li></ul></ul><ul><ul><li>“ Knobs” to tune search </li></...
Why DBSight? – Simple! <ul><ul><li>Create Search as 1-2-3 </li></ul></ul><ul><ul><ul><li>Setup JDBC Connection </li></ul><...
Simple to Create <ul><ul><li>Web UI to </li></ul></ul><ul><ul><ul><li>set SQL to retrieve content </li></ul></ul></ul>
Simple to Create <ul><ul><li>Scaffolding to generate search result template </li></ul></ul>
Simple to Create <ul><ul><li>Keep it DRY – Do not Repeat Yourself </li></ul></ul><ul><ul><ul><li>Make full use of existing...
Why DBSight? – Not So Simple! <ul><ul><li>A whole Solution </li></ul></ul><ul><ul><ul><li>No consulting fees </li></ul></u...
DBSight covers SDLC <ul><ul><li>No existing production ready solutions in the market covers the whole software development...
DBSight – Loosely Coupled <ul><ul><li>Database Independent </li></ul></ul><ul><ul><li>Programming language Independent </l...
DBSight – Enterprise Ready <ul><ul><li>Easy to move deployment environments </li></ul></ul><ul><ul><ul><li>Development, Te...
DBSight – SQL Friendly <ul><ul><li>Incremental Indexing </li></ul></ul><ul><ul><ul><li>Handle New/Updated/Deleted records ...
Easy to maintain <ul><ul><li>Scheduled Jobs </li></ul></ul><ul><ul><ul><li>Incremental indexing </li></ul></ul></ul><ul><u...
Facet Search <ul><ul><li>Not all facet search are the same! </li></ul></ul><ul><ul><ul><li>Single Value </li></ul></ul></u...
Fast Facet Search <ul><ul><li>In Memory Facets </li></ul></ul><ul><ul><li>Several cache for More Speed </li></ul></ul><ul>...
Examples of Facet Search <ul><ul><li>Basic Facet Search </li></ul></ul><ul><ul><ul><li>Example: Category </li></ul></ul></...
Feature: Seperated Indexing and Searching <ul><ul><li>Problem: Search Pauses! </li></ul></ul><ul><ul><ul><li>Indexing is C...
DBSight Architecture <ul><ul><li>Crawl database via user defined SQL </li></ul></ul><ul><ul><ul><li>Multiple database tabl...
 
DBSight Common Setup <ul><ul><li>Embedded </li></ul></ul><ul><ul><li>Multiple Indexes on single node </li></ul></ul><ul><u...
 
 
 
 
DBSight Introduction
DBSight Introduction
DBSight Introduction
Upcoming SlideShare
Loading in...5
×

DBSight Introduction

3,620

Published on

An excellent tool to create Database search quickly, with high performance and scalability, easy to maintain, flexible result rendering, api or javascript integration, etc.

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
3,620
On Slideshare
0
From Embeds
0
Number of Embeds
1
Actions
Shares
0
Downloads
10
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

DBSight Introduction

  1. 1. DBSight Instant Scalable Full Text Search on Any Databases
  2. 2. Some Customers <ul><ul><li>Websites </li></ul></ul><ul><ul><ul><li>Workopolis.com, biggest job site in Canada </li></ul></ul></ul><ul><ul><ul><li>Twenga.com, busy </li></ul></ul></ul><ul><ul><ul><li>Current.com </li></ul></ul></ul><ul><ul><li>Corporate </li></ul></ul><ul><ul><ul><li>eBay </li></ul></ul></ul><ul><ul><ul><li>Costco </li></ul></ul></ul><ul><ul><ul><li>FactSet </li></ul></ul></ul><ul><ul><ul><li>Genetech </li></ul></ul></ul><ul><ul><li>Consulting </li></ul></ul><ul><ul><ul><li>Computer Science Corporation </li></ul></ul></ul><ul><ul><li>Federal </li></ul></ul><ul><ul><ul><li>Federal Procurement Data System </li></ul></ul></ul><ul><ul><li>Banking </li></ul></ul><ul><ul><ul><li>European Central Bank, International Counterfeit Deterrence Centre(ECB-ICDC) </li></ul></ul></ul>
  3. 3. Features <ul><ul><li>Add Search when you want to </li></ul></ul><ul><ul><ul><li>Database Independent </li></ul></ul></ul><ul><ul><ul><li>Language Independent </li></ul></ul></ul><ul><ul><ul><li>Very Easy to Create </li></ul></ul></ul><ul><ul><ul><li>Very Easy to Modify </li></ul></ul></ul><ul><ul><ul><li>Very Easy to Monitor/Maintain </li></ul></ul></ul><ul><ul><li>Feature Rich </li></ul></ul><ul><ul><ul><li>Facet Search </li></ul></ul></ul><ul><ul><ul><li>Real Time Search </li></ul></ul></ul><ul><ul><ul><li>Flexible ranking </li></ul></ul></ul><ul><ul><ul><li>… </li></ul></ul></ul><ul><ul><li>High Performance </li></ul></ul><ul><ul><li>Linearly Scalable </li></ul></ul>
  4. 4. The Pain to Add Search <ul><ul><li>Do you have it? Common, but slow SQL, </li></ul></ul><ul><ul><ul><li>select * from tableA </li></ul></ul></ul><ul><ul><ul><li>where column1 like ‘%abc%’ </li></ul></ul></ul><ul><ul><ul><li>or column2 like ‘%abc%’ </li></ul></ul></ul><ul><ul><ul><li>or … </li></ul></ul></ul><ul><ul><li>Some database has search features, but still </li></ul></ul><ul><ul><ul><li>Not easy to customize </li></ul></ul></ul><ul><ul><ul><li>No facet search </li></ul></ul></ul><ul><ul><ul><li>Database specific solution </li></ul></ul></ul><ul><ul><li>Some Lucene based search, but </li></ul></ul><ul><ul><ul><li>Not cover development life cycle </li></ul></ul></ul><ul><ul><ul><li>Hard to maintain. </li></ul></ul></ul><ul><ul><ul><li>Will it scale if your data grows? </li></ul></ul></ul><ul><ul><ul><li>Too closely coupled with your program </li></ul></ul></ul>
  5. 5. DBSight Design Goals <ul><ul><li>Very fast to create and adjust </li></ul></ul><ul><ul><li>“ Knobs” to tune search </li></ul></ul><ul><ul><li>Features beyond basic search </li></ul></ul><ul><ul><ul><li>Facet Search </li></ul></ul></ul><ul><ul><ul><li>Results ranking by attributes like “price”, “time”! </li></ul></ul></ul><ul><ul><li>Minimal administration </li></ul></ul><ul><ul><li>Low total cost of ownership </li></ul></ul><ul><ul><ul><li>Off-the-shelf, no expensive consulting fees </li></ul></ul></ul><ul><ul><li>Flexible </li></ul></ul><ul><ul><ul><li>Customizable analyzers, similarities, data retrieval, UI scaffolding, different output formats, APIs. </li></ul></ul></ul>
  6. 6. Why DBSight? – Simple! <ul><ul><li>Create Search as 1-2-3 </li></ul></ul><ul><ul><ul><li>Setup JDBC Connection </li></ul></ul></ul><ul><ul><ul><li>Select with SQL </li></ul></ul></ul><ul><ul><ul><li>Generate Search Configuration, Results Template </li></ul></ul></ul><ul><ul><li>Java knowledge is not necessary </li></ul></ul>
  7. 7. Simple to Create <ul><ul><li>Web UI to </li></ul></ul><ul><ul><ul><li>set SQL to retrieve content </li></ul></ul></ul>
  8. 8. Simple to Create <ul><ul><li>Scaffolding to generate search result template </li></ul></ul>
  9. 9. Simple to Create <ul><ul><li>Keep it DRY – Do not Repeat Yourself </li></ul></ul><ul><ul><ul><li>Make full use of existing metadata </li></ul></ul></ul><ul><ul><li>From your SQL </li></ul></ul><ul><ul><ul><li>Generate most Lucene configuration </li></ul></ul></ul><ul><ul><ul><li>Generate most search options </li></ul></ul></ul><ul><ul><ul><li>Generate most rendering templates </li></ul></ul></ul><ul><ul><li>UI to fine-tune customization </li></ul></ul>
  10. 10. Why DBSight? – Not So Simple! <ul><ul><li>A whole Solution </li></ul></ul><ul><ul><ul><li>No consulting fees </li></ul></ul></ul><ul><ul><ul><li>UI to manage everything </li></ul></ul></ul><ul><ul><ul><li>Set it up, and leave it run. No babysitting. </li></ul></ul></ul><ul><ul><li>Agnostic of programming languages or frameworks </li></ul></ul><ul><ul><li>Create and maintain, with basic SQL </li></ul></ul><ul><ul><li>Built-in Usage Statistics </li></ul></ul><ul><ul><li>Scalable </li></ul></ul><ul><ul><ul><li>Linearly Scalable Sharded Search </li></ul></ul></ul><ul><ul><ul><li>Separated Indexing and Searching </li></ul></ul></ul><ul><ul><li>Many Customization point </li></ul></ul><ul><ul><ul><li>Search Results templates easy to customize </li></ul></ul></ul><ul><ul><ul><li>API for deep integration </li></ul></ul></ul>
  11. 11. DBSight covers SDLC <ul><ul><li>No existing production ready solutions in the market covers the whole software development life cycle. </li></ul></ul><ul><ul><ul><li>DBSight make search a separated concern </li></ul></ul></ul><ul><ul><ul><li>Change easily when database schema changes </li></ul></ul></ul><ul><ul><ul><li>Enterprise Ready </li></ul></ul></ul><ul><ul><ul><ul><li>Monitoring </li></ul></ul></ul></ul><ul><ul><ul><ul><li>Portable for Dev=>Test => Stage => Production environments </li></ul></ul></ul></ul>
  12. 12. DBSight – Loosely Coupled <ul><ul><li>Database Independent </li></ul></ul><ul><ul><li>Programming language Independent </li></ul></ul><ul><ul><li>Framework Independent </li></ul></ul><ul><ul><li>Works during the whole software life cycle </li></ul></ul><ul><ul><ul><li>Allows frequent adjustment </li></ul></ul></ul><ul><ul><ul><li>Easily re-create the whole index </li></ul></ul></ul><ul><ul><ul><li>Linearly scalable for high concurrency and for high data volume </li></ul></ul></ul>
  13. 13. DBSight – Enterprise Ready <ul><ul><li>Easy to move deployment environments </li></ul></ul><ul><ul><ul><li>Development, Testing(QA), Staging, Production </li></ul></ul></ul><ul><ul><li>Secure </li></ul></ul><ul><ul><ul><li>access control </li></ul></ul></ul><ul><ul><ul><li>sensitive database passwords </li></ul></ul></ul><ul><ul><li>Package-able Solution </li></ul></ul><ul><ul><ul><li>Import/Export configuration </li></ul></ul></ul><ul><ul><ul><li>Customizable enterprise-specific scaffolding </li></ul></ul></ul>
  14. 14. DBSight – SQL Friendly <ul><ul><li>Incremental Indexing </li></ul></ul><ul><ul><ul><li>Handle New/Updated/Deleted records </li></ul></ul></ul><ul><ul><ul><li>Find Deleted Records Efficiently! </li></ul></ul></ul><ul><ul><ul><ul><li>Support Hard Deleted Records </li></ul></ul></ul></ul><ul><ul><ul><ul><li>Support Soft Deleted Records </li></ul></ul></ul></ul><ul><ul><li>Flexible User Defined SQL </li></ul></ul><ul><ul><ul><li>Star-schema like content retrieval </li></ul></ul></ul><ul><ul><ul><li>No too-smart auto discovery </li></ul></ul></ul><ul><ul><li>Efficient </li></ul></ul><ul><ul><ul><li>Multi-threaded data retrieval </li></ul></ul></ul><ul><ul><ul><li>Caching to minimize database load </li></ul></ul></ul><ul><ul><ul><li>Customizable number of SQL connections </li></ul></ul></ul>
  15. 15. Easy to maintain <ul><ul><li>Scheduled Jobs </li></ul></ul><ul><ul><ul><li>Incremental indexing </li></ul></ul></ul><ul><ul><ul><li>Re-Create indexing </li></ul></ul></ul><ul><ul><ul><li>Build spell checking, synonyms, stop words dictionaries </li></ul></ul></ul><ul><ul><li>Web UI to </li></ul></ul><ul><ul><ul><li>Monitor Indexing process </li></ul></ul></ul><ul><ul><ul><li>Monitor Search Usage </li></ul></ul></ul>
  16. 16. Facet Search <ul><ul><li>Not all facet search are the same! </li></ul></ul><ul><ul><ul><li>Single Value </li></ul></ul></ul><ul><ul><ul><li>Multiple Value </li></ul></ul></ul><ul><ul><ul><li>Range Value </li></ul></ul></ul><ul><ul><li>Fast! </li></ul></ul><ul><ul><li>Memory Efficient! </li></ul></ul><ul><ul><li>More Features! </li></ul></ul><ul><ul><ul><li>List most-used facet according to usage! </li></ul></ul></ul><ul><ul><ul><li>Sum()/Avg() functions </li></ul></ul></ul>
  17. 17. Fast Facet Search <ul><ul><li>In Memory Facets </li></ul></ul><ul><ul><li>Several cache for More Speed </li></ul></ul><ul><ul><ul><li>Cache for top facets </li></ul></ul></ul><ul><ul><ul><li>Cache for recent facets </li></ul></ul></ul><ul><ul><li>Automatically Pre-warm up </li></ul></ul>
  18. 18. Examples of Facet Search <ul><ul><li>Basic Facet Search </li></ul></ul><ul><ul><ul><li>Example: Category </li></ul></ul></ul><ul><ul><ul><ul><li>Classic ( 23 matches) </li></ul></ul></ul></ul><ul><ul><li>Multi-Valued Facet Search </li></ul></ul><ul><ul><ul><li>Example: Tag, or Tag Cloud. Several tags for one record </li></ul></ul></ul><ul><ul><li>Configurable dynamic facet grouping </li></ul></ul><ul><ul><ul><li>Example: Price Range </li></ul></ul></ul><ul><ul><ul><ul><li>$2 ~$3 (7 matches) </li></ul></ul></ul></ul><ul><ul><li>Average/Sum for each facet </li></ul></ul><ul><ul><ul><li>Example: Average price for the Year Range </li></ul></ul></ul><ul><ul><ul><ul><li>1970~1980(6 matches, average price $1,245.34) </li></ul></ul></ul></ul>
  19. 19. Feature: Seperated Indexing and Searching <ul><ul><li>Problem: Search Pauses! </li></ul></ul><ul><ul><ul><li>Indexing is CPU and Disk intensive </li></ul></ul></ul><ul><ul><ul><li>Searching is CPU, Disk, and Memory intensive </li></ul></ul></ul><ul><ul><ul><li>Resource competition </li></ul></ul></ul><ul><ul><li>Solution: Separated Indexing and Searching </li></ul></ul><ul><ul><ul><li>Different JVM processes </li></ul></ul></ul><ul><ul><ul><ul><li>Easier to manage resources via JVM settings </li></ul></ul></ul></ul><ul><ul><ul><li>Indexing and Searching can be on different machines. </li></ul></ul></ul><ul><ul><ul><ul><li>Improve performance </li></ul></ul></ul></ul><ul><ul><ul><ul><li>No hiccup because of CPU, memory, disk resources contension. </li></ul></ul></ul></ul><ul><ul><ul><ul><li>Cluster of Searching nodes for better scalable performance! </li></ul></ul></ul></ul>
  20. 20. DBSight Architecture <ul><ul><li>Crawl database via user defined SQL </li></ul></ul><ul><ul><ul><li>Multiple database tables </li></ul></ul></ul><ul><ul><ul><li>Support star-schema like outer joins </li></ul></ul></ul><ul><ul><li>Create and Maintain Lucene index </li></ul></ul><ul><ul><ul><li>Incremental Indexing </li></ul></ul></ul><ul><ul><ul><li>Re-Creating Indexing </li></ul></ul></ul><ul><ul><li>Serve Search results via </li></ul></ul><ul><ul><ul><li>user defined templates </li></ul></ul></ul><ul><ul><ul><ul><li>XML/HTML/JSON/JSONP </li></ul></ul></ul></ul><ul><ul><ul><li>API, protocol buffer for Java and other languages </li></ul></ul></ul><ul><ul><li>Linear Scalability </li></ul></ul>
  21. 22. DBSight Common Setup <ul><ul><li>Embedded </li></ul></ul><ul><ul><li>Multiple Indexes on single node </li></ul></ul><ul><ul><li>Single Node, Indexing + Searching </li></ul></ul><ul><ul><li>Two Nodes, Separated Indexing and Searching </li></ul></ul><ul><ul><ul><li>Setup for LAN </li></ul></ul></ul><ul><ul><ul><li>Setup for WAN </li></ul></ul></ul><ul><ul><li>Cluster of Searching nodes via Replication, one Indexing nodes </li></ul></ul><ul><ul><li>Cluster of Sharded Nodes, each with Indexing +Searching </li></ul></ul><ul><ul><li>Cluster of Searching nodes via Sharding, one or several indexing nodes. </li></ul></ul>
  1. A particular slide catching your eye?

    Clipping is a handy way to collect important slides you want to go back to later.

×