A Scalable Architecture For Distributed Retrieval System In High Concurrency Environment

A Scalable Architecture for Distributed Retrieval System in High Concurrency Environment
Concepts and Implementations of ClusCom and FeDrs
Liu Liu

Motivation
Scale retrieval system to billions;
Accuracy depends on size of database;
High concurrency in real world.

What is ClusCom
A C++ lib for distributed computing in high concurrency environment;
Easy to use and employ;

Structure
Tree-based structure:
........................................................ Node 1 Node 3 Node 2

Lib Interface
virtual int handle ( char * to_request, char * to_reply);
virtual int synthesize (apr_uint32_t &node, char ** to_synthe, char * to_reply);
virtual int manipulate ( char * to_mani, apr_uint32_t& return_size, char * to_reply);

Some Facts of ClusCom
Based on Apache Portable Runtime Library;
Self-managed memory allocate;
Persistent socket connection pipe with backups;
epoll and multithreading hybird server socket.

How ClusCom Works
7 seperated threads ;
Request Pre-handler Dispatcher Handler Classifier Synthesizer Repeater Reply Pre-handler

Why
Seperate network I/O handling and actual data process;
7 threads are not only 7 threads;

Look into ClusCom
Network I/O handler using persistent socket pipe;
Handler and synthesizer using leader-follower thread pool;

Deal with High Concurrence REST Connection Persistent Connection ClusCom Server ClusCom Proxy Application Servers

What is ClusCom Proxy
establish non-state connection with application;
Maintain a persistent connection with ClusCom server;
ClusCom Proxy REST Connection Persistent Connection

FeDrs
Based on ClusCom;
Design for retrieval system;
Easy to use and employ.

FeDrs Interface
Expose only one function and can be changed on fly;
double evaluate_func( const char * input_a, const char * input_b, const char * config);

Some Facts of FeDrs
Data saved with radix tree and 128-bit key;
A PHP extension to access FeDrs;
Retrieval within groups;

A Scalable Architecture For Distributed Retrieval System In High Concurrency Environment

by Liu Liu on Dec 21, 2008

Accessibility

Categories

Tags

Upload Details

Usage Rights

Statistics

A Scalable Architecture For Distributed Retrieval System In High Concurrency Environment — Presentation Transcript