Your SlideShare is downloading. ×
0
HBaseCon 2012 | Lessons learned from OpenTSDB - Benoit Sigoure, StumbleUpon
HBaseCon 2012 | Lessons learned from OpenTSDB - Benoit Sigoure, StumbleUpon
HBaseCon 2012 | Lessons learned from OpenTSDB - Benoit Sigoure, StumbleUpon
HBaseCon 2012 | Lessons learned from OpenTSDB - Benoit Sigoure, StumbleUpon
HBaseCon 2012 | Lessons learned from OpenTSDB - Benoit Sigoure, StumbleUpon
HBaseCon 2012 | Lessons learned from OpenTSDB - Benoit Sigoure, StumbleUpon
HBaseCon 2012 | Lessons learned from OpenTSDB - Benoit Sigoure, StumbleUpon
HBaseCon 2012 | Lessons learned from OpenTSDB - Benoit Sigoure, StumbleUpon
HBaseCon 2012 | Lessons learned from OpenTSDB - Benoit Sigoure, StumbleUpon
HBaseCon 2012 | Lessons learned from OpenTSDB - Benoit Sigoure, StumbleUpon
HBaseCon 2012 | Lessons learned from OpenTSDB - Benoit Sigoure, StumbleUpon
HBaseCon 2012 | Lessons learned from OpenTSDB - Benoit Sigoure, StumbleUpon
HBaseCon 2012 | Lessons learned from OpenTSDB - Benoit Sigoure, StumbleUpon
HBaseCon 2012 | Lessons learned from OpenTSDB - Benoit Sigoure, StumbleUpon
HBaseCon 2012 | Lessons learned from OpenTSDB - Benoit Sigoure, StumbleUpon
HBaseCon 2012 | Lessons learned from OpenTSDB - Benoit Sigoure, StumbleUpon
HBaseCon 2012 | Lessons learned from OpenTSDB - Benoit Sigoure, StumbleUpon
HBaseCon 2012 | Lessons learned from OpenTSDB - Benoit Sigoure, StumbleUpon
HBaseCon 2012 | Lessons learned from OpenTSDB - Benoit Sigoure, StumbleUpon
HBaseCon 2012 | Lessons learned from OpenTSDB - Benoit Sigoure, StumbleUpon
HBaseCon 2012 | Lessons learned from OpenTSDB - Benoit Sigoure, StumbleUpon
HBaseCon 2012 | Lessons learned from OpenTSDB - Benoit Sigoure, StumbleUpon
HBaseCon 2012 | Lessons learned from OpenTSDB - Benoit Sigoure, StumbleUpon
HBaseCon 2012 | Lessons learned from OpenTSDB - Benoit Sigoure, StumbleUpon
HBaseCon 2012 | Lessons learned from OpenTSDB - Benoit Sigoure, StumbleUpon
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

HBaseCon 2012 | Lessons learned from OpenTSDB - Benoit Sigoure, StumbleUpon

9,076

Published on

OpenTSDB was built on the belief that, through HBase, a new breed of monitoring systems could be created, one that can store and serve billions of data points forever without the need for destructive …

OpenTSDB was built on the belief that, through HBase, a new breed of monitoring systems could be created, one that can store and serve billions of data points forever without the need for destructive downsampling, one that could scale to millions of metrics, and where plotting real-time graphs is easy and fast. In this presentation we’ll review some of the key points of OpenTSDB’s design, some of the mistakes that were made, how they were or will be addressed, and what were some of the lessons learned while writing and running OpenTSDB as well as asynchbase, the asynchronous high-performance thread-safe client for HBase. Specific topics discussed will be around the schema, how it impacts performance and allows concurrent writes without need for coordination in a distributed cluster of OpenTSDB instances.

Published in: Technology, Economy & Finance
1 Comment
40 Likes
Statistics
Notes
  • The problem with HBase's implementation is that every single cell also stores the row key and a bunch of other redundant information.

    I think the compress may decrease the capacity in disk
       Reply 
    Are you sure you want to  Yes  No
    Your message goes here
No Downloads
Views
Total Views
9,076
On Slideshare
0
From Embeds
0
Number of Embeds
20
Actions
Shares
0
Downloads
326
Comments
1
Likes
40
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide

Transcript

  • 1. Lessons Learned from OpenTSDBOr why OpenTSDB is the way it is and how it changed iteratively tocorrect some of the mistakes made Benoît “tsuna” Sigoure tsuna@stumbleupon.com
  • 2. Key concepts• Data Points (time, value)• Metrics proc.loadavg.1m• Tags host=web42 pool=static• Metric + Tags = Time Series• Order of magnitude: >106 time series, >1012 data points put proc.loadavg.1m 1234567890 0.42 host=web42 pool=static
  • 3. OpenTSDB @ StumbleUpon• Main production monitoring system for ~2 years• Storing hundreds of billions of data points• Adding over 1 billion data points per day• 13000 data points/s → 130 QPS on HBase• If you had a 5 node cluster, this load would hardly make it sweat
  • 4. Do’s• Wider rows to seek faster before: ~4KB/row, after: ~20KB• Make writes idempotent and independent before: start rows at arbitrary points in time after: align rows on 10m (then 1h) boundaries• Store more data per KeyValue Remember you pay for the key along each value in a row, so large keys are really expensive
  • 5. Don’ts• Use HTable / HTablePool in app servers asynchbase + Netty or Finagle = performance++• Put variable-length fields in composite keys They’re hard to scan• Exceed a few hundred regions per RegionServer “Oversharding” introduces overhead and makes recovering from failures more expensive
  • 6. Use asynchbase HTable asynchbase scan sequential read sequential write50s 500s 200s38s 375s 150s25s 250s 100s13s 125s 50s 0s 0s 0s 4 8 16 24 32 4 8 16 24 32 4 8 16 24 32 # Threads # Threads # Threads
  • 7. How OpenTSDB came to be the way it isQuestions:• How to store time series data efficiently in HBase?• How to enable concurrent writes without synchronization between the writers?• How to save space/memory when storing hundreds of billions of data items in HBase?
  • 8. Ta Time Series Data in HBase ke 1 Col don’t care umn Key 1234567890 1 values 1234567892 2timestamps 1234567894 3Simplest design: only 1 time series, 1 row with asingle KeyValue per data point.Supports time-range scans.
  • 9. Ta Time Series Data in HBase ke 2 Colu mn Key foo 1234567890 1 foo 1234567892 3 metric name fool 1234567890 2Metric name first in row key for data locality.Problem: can’t store the metric as text in row keydue to space concerns
  • 10. Ta Time Series Data in HBase ke 3 Colu Separate mn Key Lookup Table: Key Value 0x1 1234567890 1 0x1 foo 0x1 1234567892 3 0x2 fool metric foo 0x1 ID 0x2 1234567890 2 fool 0x2Use a separate table to assign unique IDs tometric names (and tags, not shown here). IDs give us apredictable length and achieve desired data locality.
  • 11. Ta Time Series Data in HBase ke 4 Colu mn +0 +2 Key 0x1 1234567890 1 3 0x1 1234567892 3 0x2 1234567890 2Reduce the number of rows by storing multipleconsecutive data points in the same row.Fewer rows = faster to seek to a specific row.
  • 12. Ta Time Series Data in HBase ke 4 Colu mn +0 +2 Key 0x1 1234567890 1 3 Misleading table 0x1 1234567892 3representation 0x2 1234567890 2 Gotcha #1: wider rows don’t save any space* Key Colum Value le 0x1 1234567890 n +0 1 ab l t d 0x1 1234567890 ua re +2 3 * Until magic prefix ct to +0 2 compression happens in A s 0x2 1234567890 upcoming HBase 0.94
  • 13. Ta Time Series Data in HBase ke 4 Colu mn +0 +2 Key 0x1 1234567890 1 3 0x1 1234567892 3 0x2 1234567890 2Devil is in the details: when to start new rows?Naive answer: start on first data point, after sometime start a new row.
  • 14. Ta Time Series Data in HBase ke 4 Colu mn +0 Key 0x1 1000000000 1 0000 00 1 TSD1 1000 First data point: foo Start a new rowClient TSD2
  • 15. Ta Time Series Data in HBase ke 4 Colu mn +0 +10 ... Key 0x1 1000000000 1 2 ... 0000 10 2 TSD1 1000 Keep adding foo points until...Client TSD2
  • 16. Ta Time Series Data in HBase ke 4 Colu mn +0 +10 ... +599 Key 0x1 1000000000 1 2 ... 42 42 0000 0599 TSD1 ... some arbitrary fo o 10 limit, say 10minClient TSD2
  • 17. Ta Time Series Data in HBase ke 4 Colu mn +0 +10 ... +599 Key 0x1 1000000000 1 2 ... 42 0x1 1000000600 51 51 0000 0610 TSD1 Then start a new fo o 10 rowClient TSD2
  • 18. Ta Time Series Data in HBase ke 4 Colu mn +0 Key 0x1 1234567890 1 But this scheme fails with multiple TSDs 5678 90 1 TSD1 Create new row foo 1234Client TSD2
  • 19. Ta Time Series Data in HBase ke 4 Colu mn +0 +2 Key 0x1 1234567890 1 3 5678 92 3 TSD1 Add to row foo 1234Client TSD2
  • 20. Ta Time Series Data in HBase ke 4 Colu mn +0 +2 Key 0x1 1234567890 1 3 Oops! 0x1 1234567892 3 Maybe a connection failure occurred, client is retransmitting data to another TSD TSD1 Add to row foo 12345678 92 3Client TSD2 Create new row
  • 21. Ta Time Series Data in HBase ke 5 Colu mn +90 +92 Key Basetimestamp 0x1 1234567800 1 3 always amultiple of 0x2 1234567800 2 600In order to scale easily and keep TSD stateless,make writes independent & idempotent.New rule: rows are aligned on 10 min. boundaries
  • 22. Ta Time Series Data in HBase ke 6 Colu mn +1890 +1892 Key Basetimestamp 0x1 1234566000 1 3 always amultiple of 0x2 1234566000 2 36001 data point every ~10s => 60 data points / rowNot much. Go to wider rows to further increaseseek speed. One hour rows = 6x fewer rows
  • 23. Ta Time Series Data in HBase ke 6 Colu mn +1890 +1892 Key 0x1 1234566000 1 3 0x2 1234566000 2Remember: wider rows don’t save any space! Key Colum Value Key is easily 4x le 0x1 1234566000 n +1890 1 bigger than tab column + value al ed 0x1 1234566000 tu or +1892 3Ac st 0x2 1234566000 +1890 2 and repeated
  • 24. Ta Time Series Data in HBase ke 7 Colu mn +1890 +1890 +1892 +1892 Key 0x1 1234566000 1 1 3 3 0x2 1234566000 2Solution: “compact” columns by concatenation Key Column Value Space savings le 0x1 1234566000 +1890 1 on disk and in tab al ed 0x1 tu or 1234566000 +1890,+1892 1, 3 memory areAc st 0x1 1234566000 +1892 3 huge: data is 0x2 1234566000 +1890 2 4x-8x smaller!
  • 25. ¿ Questions ? ub tH Gi on opentsdb.net e m krFo Summary • Use asynchbase • Use Netty or Finagle • Wider table > Taller table • Short family names • Make writes idempotent • Make writes independent • Compact your data • Have predictable key sizes ool? Thin k this is c Benoît “tsuna” Sigoure W e’re hiring tsuna@stumbleupon.com

×