Google's realtime search system stores tens of petabytes of data across thousands of machines and processes billions of updates per day. It uses a distributed storage system called Bigtable to store the large volume of indexing data but Bigtable cannot natively support incremental updates or enforce data integrity. Google developed Percolator, a new system built on top of Bigtable, to allow incremental processing of updates through transactions and notifications at large scale while maintaining data consistency. Percolator provides the ability to perform ACID transactions over Bigtable and organize incremental computations through observer notifications.

1. Realtime Search 罗磊

2. Reference A Billion Queries Per Day Bigtable: A Distributed Sorage System for Structured Data Large-scale Incremental Processing Using Distributed Transactions and Notifications

3. Realtime Search @ Twitter

6. Posting list format and early query terminal Only evaluate as few documents as possible before terminating the query Rank documents in reverse time order (newest documents first)

13. Index memory model

31. Lock-free algorithms and data structures

46. Realtime Search @ Twitter Q & A

47. Realtime Search @ Google

48. Arguement about MapReduce MapReduce: A major step backwards MapReduce and Parallel DBMSs: Friend or Foe MapReduce: A Flexible Data Processing Tool

49. Google's realtime search Google's indexing system stores tens of petabytes of data and processes billions of updates per day on thousands of machines. MapReduce and other batch-processing systems cannot process small updates individually as they rely on creating large batches for efficiency.

50. Avoid indexes duplicates MapReduce can't deal with any update MapReduce must be run again over the entire repository

51. Where to store index at Google DBMS Can't handle the sheer volume of data: Google's indexing System store tens of petabytes across thousands of machines. Bigtable Can't provide tools to help programmers maintain data invariants in the face of concurrent updates

52. Percolator Incremental processing model it maintains a very large repository of documents and update it efficiently Processing many updates concurrently

53. Percolator Percolator provides two main abstractions for performing incremental processing at large scale: ACID transactions over a random-access repository observers, a way to organize an incremental computation

54. Percolator

55. Transaction is a must

56. Transaction is a must

57. Transaction is a must

58. Transcation on Percolar

59. Transcation on Percolar

60. Transcation on Percolar

61. Transcation on Percolar

62. Transcation on Percolar

63. Transcation on Percolar Transcation processing is complicated by the possibility of client failure table server failure does not affect the system since Bigtable guarantees that written locks persist across talbe server failures A transaction will not clean up a lock unless it suspects that a lock belongs to a dead or stuck worker

64. Notifications Transactions let the user mutate the table while maintaining invariants, but users also need a way to trigger and run the transactions To provide notifications, Percolator maintans a special "observation" column.

65. Optimization Reduce RPC batching when reading from the same table server Hacking on Linux kernel to support high thread counts

66. Evaluation

67. Evalution

68. Realtime Search @ Google Q & A

69. Realtime Search Realtime Search @ Twitter Realtime Search @ Google Thank you