An Introduction to Google Percolator

3,815 views

Published on

An Introduction to Google Percolator, what is it and what is it used
for ? How is it different from Apache Hadoop MapReduce ?

Published in: Technology, Business

An Introduction to Google Percolator

  1. 1. Google Percolator ● What is it ? ● What is it used for ? ● Percolator Vs MapReduce ● Architecture www.semtech-solutions.co.nz info@semtech-solutions.co.nz
  2. 2. Percolator – What is it ? ● Incremental updates to Big Data ● Developed by Google ● Based on Google File System ( GFS ) ● Provides transactions and locking ● Faster than comparable Map Reduce ● Developed by Google due to MapReduce limitations www.semtech-solutions.co.nz info@semtech-solutions.co.nz
  3. 3. Percolator – What is it used for ? ● Iterative updates ● No need to batch process ● Update as data received ● Data in multi petabyte range ● Strong consistency needed ● Improved latency ( 100 x ) ● Reduced document age ( 50 % ) ● Random access to big data repository www.semtech-solutions.co.nz info@semtech-solutions.co.nz
  4. 4. Percolator Vs MapReduce Map Reduce ● Batch Processing ● No transactions ● Latency A ● Run time scales with data ● Code in C++ ● Open source ● Uses HDFS Percolator – Iterative – Transactions – Latency 100 x A – Incremental updates – Code in Java ( mainly ) – Google owned – Uses GFS www.semtech-solutions.co.nz info@semtech-solutions.co.nz
  5. 5. Percolator – Architecture ● Applications are a sequence of observers ● An observer is called via a notification ● A notification is triggered when table data changes ● Application calls TabletServer via RPC ● TabletServer calls GFS ChunkServer
  6. 6. Percolator – Architecture ● Applications – Series of observers ● Observer – Completes task – Updates table ● Next Observer called – Via notification ● Percolator worker – Scans for changes – Sends notifications
  7. 7. Percolator – Architecture Actual worker diagram including time stamping and locking via Chubby lock server
  8. 8. Contact Us ● Feel free to contact us at – www.semtech-solutions.co.nz – info@semtech-solutions.co.nz ● We offer IT project consultancy ● We are happy to hear about your problems ● You can just pay for those hours that you need ● To solve your problems

×