A Scalable Architecture for Distributed Retrieval System in High Concurrency Environment <ul><li>Concepts and Implementati...
Motivation <ul><li>Scale retrieval system to billions; </li></ul><ul><li>Accuracy depends on size of database; </li></ul><...
What is ClusCom <ul><li>A C++ lib for distributed computing in high concurrency environment; </li></ul><ul><li>Easy to use...
Structure <ul><li>Tree-based structure: </li></ul>........................................................ Node 1 Node 3 N...
Lib Interface <ul><li>virtual int   handle ( char * to_request,  char * to_reply); </li></ul><ul><li>virtual int   synthes...
Some Facts of ClusCom <ul><li>Based on Apache Portable Runtime Library; </li></ul><ul><li>Self-managed memory allocate; </...
How ClusCom Works <ul><li>7 seperated  threads ; </li></ul>Request Pre-handler Dispatcher Handler Classifier Synthesizer R...
Why <ul><li>Seperate network I/O handling and actual data process; </li></ul><ul><li>7  threads  are not only 7 threads; <...
Look into ClusCom <ul><li>Network I/O handler using persistent socket pipe; </li></ul><ul><li>Handler and synthesizer usin...
Deal with High Concurrence REST Connection Persistent Connection ClusCom Server ClusCom Proxy Application Servers
What is ClusCom Proxy <ul><li>establish non-state connection with application; </li></ul><ul><li>Maintain a persistent con...
FeDrs <ul><li>Based on ClusCom; </li></ul><ul><li>Design for retrieval system; </li></ul><ul><li>Easy to use and employ. <...
FeDrs Interface <ul><li>Expose only one function and can be changed on fly; </li></ul><ul><li>double  evaluate_func( const...
Some Facts of FeDrs <ul><li>Data saved with radix tree and 128-bit key; </li></ul><ul><li>A PHP extension to access FeDrs;...
Upcoming SlideShare
Loading in …5
×

A Scalable Architecture For Distributed Retrieval System In High Concurrency Environment

1,104 views

Published on

Published in: Technology
0 Comments
3 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
1,104
On SlideShare
0
From Embeds
0
Number of Embeds
59
Actions
Shares
0
Downloads
0
Comments
0
Likes
3
Embeds 0
No embeds

No notes for slide

A Scalable Architecture For Distributed Retrieval System In High Concurrency Environment

  1. 1. A Scalable Architecture for Distributed Retrieval System in High Concurrency Environment <ul><li>Concepts and Implementations of ClusCom and FeDrs </li></ul><ul><li>Liu Liu </li></ul>
  2. 2. Motivation <ul><li>Scale retrieval system to billions; </li></ul><ul><li>Accuracy depends on size of database; </li></ul><ul><li>High concurrency in real world. </li></ul>
  3. 3. What is ClusCom <ul><li>A C++ lib for distributed computing in high concurrency environment; </li></ul><ul><li>Easy to use and employ; </li></ul>
  4. 4. Structure <ul><li>Tree-based structure: </li></ul>........................................................ Node 1 Node 3 Node 2
  5. 5. Lib Interface <ul><li>virtual int handle ( char * to_request, char * to_reply); </li></ul><ul><li>virtual int synthesize (apr_uint32_t &node, char ** to_synthe, char * to_reply); </li></ul><ul><li>virtual int manipulate ( char * to_mani, apr_uint32_t& return_size, char * to_reply); </li></ul>
  6. 6. Some Facts of ClusCom <ul><li>Based on Apache Portable Runtime Library; </li></ul><ul><li>Self-managed memory allocate; </li></ul><ul><li>Persistent socket connection pipe with backups; </li></ul><ul><li>epoll and multithreading hybird server socket. </li></ul>
  7. 7. How ClusCom Works <ul><li>7 seperated threads ; </li></ul>Request Pre-handler Dispatcher Handler Classifier Synthesizer Repeater Reply Pre-handler
  8. 8. Why <ul><li>Seperate network I/O handling and actual data process; </li></ul><ul><li>7 threads are not only 7 threads; </li></ul>Request Pre-handler Dispatcher Handler Classifier Synthesizer Repeater Reply Pre-handler
  9. 9. Look into ClusCom <ul><li>Network I/O handler using persistent socket pipe; </li></ul><ul><li>Handler and synthesizer using leader-follower thread pool; </li></ul>Request Pre-handler Dispatcher Handler Classifier Synthesizer Repeater Reply Pre-handler
  10. 10. Deal with High Concurrence REST Connection Persistent Connection ClusCom Server ClusCom Proxy Application Servers
  11. 11. What is ClusCom Proxy <ul><li>establish non-state connection with application; </li></ul><ul><li>Maintain a persistent connection with ClusCom server; </li></ul>ClusCom Proxy REST Connection Persistent Connection
  12. 12. FeDrs <ul><li>Based on ClusCom; </li></ul><ul><li>Design for retrieval system; </li></ul><ul><li>Easy to use and employ. </li></ul>
  13. 13. FeDrs Interface <ul><li>Expose only one function and can be changed on fly; </li></ul><ul><li>double evaluate_func( const char * input_a, const char * input_b, const char * config); </li></ul>
  14. 14. Some Facts of FeDrs <ul><li>Data saved with radix tree and 128-bit key; </li></ul><ul><li>A PHP extension to access FeDrs; </li></ul><ul><li>Retrieval within groups; </li></ul>

×