An introduction to Apache Chukwa

2,283 views

Published on

A introduction to Apache Chukwa, what is it and
how does it work ? Why is it important to monitor
Hadoop DFS and how can it help us ?

Published in: Technology
0 Comments
2 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total views
2,283
On SlideShare
0
From Embeds
0
Number of Embeds
3
Actions
Shares
0
Downloads
39
Comments
0
Likes
2
Embeds 0
No embeds

No notes for slide

An introduction to Apache Chukwa

  1. 1. Apache Chukwa ● What is it ? ● How does it work ? ● What can we collect ? ● Architecture www.semtech-solutions.co.nz info@semtech-solutions.co.nz
  2. 2. Chukwa – What is it ? ● For log collection and analysis ● Designed for big data ● Designed for Hadoop ● Uses HDFS and MapReduce ● Scaleable ● Robust ● Provides a tool kit to analyse logs www.semtech-solutions.co.nz info@semtech-solutions.co.nz
  3. 3. Chukwa – How does it work ? ● Chukwa agents on source nodes ● Transfer data to collectors which save data to HDFS ● Data sinks contain raw unsorted data ● Data sinks clean data ● Demux adds structure to create Chukwa records ● Chukwa records go to database ● Are ready to be analysed www.semtech-solutions.co.nz info@semtech-solutions.co.nz
  4. 4. Chukwa – What can we collect ? ● Metrics ● System logs – Defined format – Undefined format ● Low latency – Access to log data www.semtech-solutions.co.nz info@semtech-solutions.co.nz
  5. 5. Chukwa – Architecture ? www.semtech-solutions.co.nz info@semtech-solutions.co.nz
  6. 6. Chukwa – Architecture ? ● Chukwa agents – Reside on the Hadoop machines – Collect raw data – Use adaptors for data sources – Use http to transmit data – Operate on data chunks – Can fail over between collectors www.semtech-solutions.co.nz info@semtech-solutions.co.nz
  7. 7. Contact Us ● Feel free to contact us at – www.semtech-solutions.co.nz – info@semtech-solutions.co.nz ● We offer IT project consultancy ● We are happy to hear about your problems ● You can just pay for those hours that you need ● To solve your problems

×