Snappy servay
Upcoming SlideShare
Loading in...5
×
 

Snappy servay

on

  • 3,341 views

 

Statistics

Views

Total Views
3,341
Views on SlideShare
2,901
Embed Views
440

Actions

Likes
2
Downloads
29
Comments
0

7 Embeds 440

http://d.hatena.ne.jp 238
http://diary.overlasting.net 177
http://twitter.com 11
http://webcache.googleusercontent.com 7
https://twitter.com 5
http://a0.twimg.com 1
https://si0.twimg.com 1
More...

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

Snappy servay Presentation Transcript

  • 1. google-snappy (machy)
  • 2. google-snappy Snappy is a compression/decompression library. It does not aim for maximumcompression, or compatibility with any other compression library; instead,it aims for very high speeds and reasonable compression. For instance,compared to the fastest mode of zlib, Snappy is an order of magnitude fasterfor most inputs, but the resulting compressed files are anywhere from 20% to100% bigger. (For more information, see "Performance", below.) README •  snappy / snappy zlib 20 100% 1 ”Performance”
  • 3. •  1.0.3•  http://code.google.com/p/snappy/•  google-gflags•  google-gflags, google-snappy WARNING: Compiled with assertions enabled, will be slow. ./configure CXXFLAGS=“-g -O2 –DNDEBUG” --with-gflags --with-gflags gflags configure gflags
  • 4. snappy_unittest •  snappy_unittest•  ./snappy_unittest•  ./snappy_unittest -run_microbenchmarks=false - write_compressed aaa.txt aaa.txt.comp•  ./snappy_unittest -run_microbenchmarks=false - write_uncompressed aaa.txt.comp aaa.txt.comp.uncomp•  zlib ./snappy_unittest -run_microbenchmarks=false -zlib testdata/*
  • 5. •  snappy alice29.txt 149KB html_x_4 400KB HTML 400KB urls.10K 1 URL 686KB baddata1.snappy (?) 27KB house.jpg 124KB
  • 6. zlib snappy 99.1 MB/s 293.3 MB/s 59.8 % alice29.txt zlib fastest 20.7 MB/s 81.5 MB/s 42.8 % zlib default 6.6 MB/s 90.4 MB/s 35.8 % snappy 230.2 MB/s 557.9 MB/s 23.6 % html_x_4 zlib fastest 45.6 MB/s 154.5 MB/s 16.5 % zlib default 20.7 MB /s 177.7 MB/s 13.0 % snappy 132.6 MB/s 411.2 MB/s 50.9 % urls.10K zlib fastest 24.7 MB/s 94.8 MB/s 36.1 % zlib default 12.2 MB/s 102.4 MB/s 31.7 % snappy 137.5 MB/s 1068.7 MB/s 97.0 %baddata1.snappy zlib fastest 12.3 MB/s 57.0 MB/s 84.1 % zlib default 10.8 MB/s 58.9 MB/s 83.4 % snappy 933.7 MB/s 7271.6 MB/s 99.9 % house.jpg zlib fastest 11.9 MB/s 89.6 MB/s 99.6 % zlib default 11.5 MB/s 122.4 MB/s 99.6 %
  • 7. zlib snappy 99.1 MB/s 293.3 MB/s 59.8 % alice29.txt zlib fastest 20.7 MB/s 81.5 MB/s 42.8 % zlib default 6.6 MB/s 90.4 MB/s 35.8 % snappy 230.2 MB/s 557.9 MB/s 23.6 % html_x_4 zlib fastest 45.6 MB/s 154.5 MB/s 16.5 % zlib default 20.7 MB /s 177.7 MB/s 13.0 % snappy 132.6 MB/s 411.2 MB/s 50.9 % urls.10K zlib fastest 24.7 MB/s 94.8 MB/s 36.1 % zlib default 12.2 MB/s 102.4 MB/s 31.7 % snappy 137.5 MB/s 1068.7 MB/s 97.0 %baddata1.snappy zlib fastest 12.3 MB/s 57.0 MB/s 84.1 % zlib fastest(level=1) 5 zlib default 3.510.8 MB/s 58.9 MB/s 83.4 % snappy 1.2 933.7 MB/s 1.4 7271.6 MB/s 99.9 % house.jpg zlib fastest 11.9 MB/s 89.6 MB/s 99.6 % zlib default 11.5 MB/s 122.4 MB/s 99.6 %
  • 8. zlib snappy 99.1 MB/s 293.3 MB/s 59.8 % alice29.txt zlib fastest 20.7 MB/s 81.5 MB/s 42.8 % zlib default 6.6 MB/s 90.4 MB/s 35.8 % snappy 230.2 MB/s 557.9 MB/s 23.6 % snappy html_x_4 zlib fastest 45.6 MB/s 154.5 MB/s 16.5 % zlib default 20.7 MB /s 177.7 MB/s 13.0 % snappy 132.6 MB/s 411.2 MB/s 50.9 % urls.10K zlib fastest 24.7 MB/s 94.8 MB/s 36.1 % zlib default 12.2 MB/s 102.4 MB/s 31.7 % snappy 137.5 MB/s 1068.7 MB/s 97.0 %baddata1.snappy zlib fastest 12.3 MB/s 57.0 MB/s 84.1 % zlib default 10.8 MB/s 58.9 MB/s 83.4 % snappy 933.7 MB/s 7271.6 MB/s 99.9 % house.jpg zlib fastest 11.9 MB/s 89.6 MB/s 99.6 % zlib default 11.5 MB/s 122.4 MB/s 99.6 %
  • 9. lzo snappy 85.9 MB/s 259.7 MB/s 59.8 % alice29.txt lzo 90.6 MB/s 178.2 MB/s 57.8 % snappy 206.7 MB/s 463.1 MB/s 23.6 % html_x_4 lzo 203.3 MB/s 421.6 MB/s 21.8 % snappy 119.4 MB/s 363.2 MB/s 50.9 % urls.10K lzo 125.3 MB/s 308.5 MB/s 49.3 % snappy 109.6 MB/s 1048.1 MB/s 97.0 %baddata1.snappy lzo 353.4 MB/s 2267.1 MB/s 100.4 % snappy 846.4 MB/s 6642.0 MB/s 99.9 % house.jpg lzo 672.6 MB/s 2024.4 MB/s 100.3 % hadoop lzo snappy snappy lzo
  • 10. snappy zlib zlib(deflate) snappy
  • 11. government_of_the_people,_by_the_people,_for_the_peoplegovernment_of_the_people,_by[15,13]for[16,11] 15 13 HAHAHAHAHA...HA[2,8]...
  • 12. 18bit15bit
  • 13. snappy LITERAL LITERAL 11byte 64byte 4byte 64byte
  • 14. snappy LITERAL Byte LITERAL 11byte 64byte 4byte 64byte
  • 15. snappy 16KB fragment fragment (16KB) fragment (16KB) fragmentfragment
  • 16. snappy 2 ( 8192 ) byte 4byte government_of_the_people,_by_the_people,_for_the_people 13 : Hash(“f_th”)=7 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 6 0 4 1 13 5 8 10 11 3government_of_the_people,_by_the_people,_for_the_people 14 : Hash(“_the”)=9 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 6 0 4 1 13 14 5 8 10 11 3
  • 17. snappy 3 government_of_the_people,_by_the_people,_for_the_people 29 : Hash(“_the”)=9 0 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 6 23 0 4 27 26 22 13 24 14 25 15 10 28 16 20 =13government_of_the_people,_by_the_people,_for_the_people[government_of_the_people,_by][15,13] 32 government_of_the_people,_by_the_people,_for_the_people
  • 18. static inline uint32 HashBytes(uint32 bytes, int shift) { uint32 kMul = 0x1e35a7bd; return (bytes * kMul) >> shift; }
  • 19. static inline uint32 HashBytes(uint32 bytes, int shift) { uint32 kMul = 0x1e35a7bd; return (bytes * kMul) >> shift; } 4byte 4byte 32bit CPU #if defined(__i386__) || defined(__x86_64__) || defined(__powerpc__) #define UNALIGNED_LOAD32(_p) (*reinterpret_cast<const uint32 *>(_p)) #else inline uint32 UNALIGNED_LOAD32(const void *p) { uint32 t; memcpy(&t, p, sizeof t); return t; }
  • 20. static inline uint32 HashBytes(uint32 bytes, int shift) { uint32 kMul = 0x1e35a7bd; return (bytes * kMul) >> shift; }shift 8192 (2 13 ) 32-13=19 8192
  • 21. JPEG(~5% performance, ~0.1% density) ...a93ecm2k39cn10xi10chakegueks16krpqw2453maheggubz......a93ecm2k39cn10xi10chakegueks16krpqw2453maheggubz... 32 2 ...a93ecm2k39cn10xi10chakegueks16krpqw2453maheggubz......a93ecm2k39cn10xi10chakegueks16krpqw2453maheggubz... 32 1 16KB 1008
  • 22. •  4byte• •  CPU• 
  • 23. •  ( zlib deflateBound(),compressBound() )
  • 24. •  snappy• •  16KB fragment CPU•  (zlib )