• Save
Lost in Translation:varnishlog, varnishtest(VUG7)
Upcoming SlideShare
Loading in...5
×
 

Lost in Translation:varnishlog, varnishtest(VUG7)

on

  • 2,033 views

 

Statistics

Views

Total Views
2,033
Slideshare-icon Views on SlideShare
593
Embed Views
1,440

Actions

Likes
2
Downloads
0
Comments
0

7 Embeds 1,440

http://blog.xcir.net 1421
http://www.linkedin.com 8
http://webcache.googleusercontent.com 4
https://twitter.com 3
http://labs.gree.jp 2
https://www.linkedin.com 1
http://translate.googleusercontent.com 1
More...

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

    Lost in Translation:varnishlog, varnishtest(VUG7) Lost in Translation:varnishlog, varnishtest(VUG7) Presentation Transcript

    • Copyright © GREE, Inc. All Rights Reserved.Copyright © GREE, Inc. All Rights Reserved.Lost in Translation:varnishlog, varnishtest~The Story of a Successful Migration~GREE, inc.Shohei TanakaVarnish User Group Meeting 7Friday, May 31st 2013
    • Copyright © GREE, Inc. All Rights Reserved.Who am IShohei Tanaka (@xcir)- Call me iwana-chan- GREE, inc.- Joined 2013/01~- Tokyo, Japan- Activities at varnish project- VMOD author- ldap, redirect, parsereq, awsrest and more..- Wrote Japanese Varnish guidebook- Other- blog http://blog.xcir.net/- github http://github.com/xcir/
    • Copyright © GREE, Inc. All Rights Reserved.Migration from legacy systemFinished without any downtime- Platform downtime was not an option- Domain / backend has more than a hundred columns- Very high- Traffic- Connection/sec- Request/sec- Many target device types- PC- Smartphone- Mobile phone (No support for keep-alive)
    • Copyright © GREE, Inc. All Rights Reserved.New systemServer layout- 3 layer model- Use Hash director- Objects wont beduplicatedacross L2 pool- ImageOperation- Apache + Varnish- Create thumbs- mod_small_light- Generate barcode(QR)- GREE original
    • Copyright © GREE, Inc. All Rights Reserved.New systemVCL layout- Common settings- Same for all server roles- Security, ACL, Normalize....- Individual server role settings- Backend, Rewrite...- Divide action by- Type (categorized domain)- Domain (cant categorize)
    • Copyright © GREE, Inc. All Rights Reserved.Why Varnish?GREE had never used it- VCL- Very powerful DSL- Fits into existing system- Healthcheck- Varnishtest- Can define the behavior of the server, client, varnish.- Useful tools- varnishncsa and varnishlog are very useful- libvarnishapi can connect to VSL easily- And I love Varnish ;-)
    • Copyright © GREE, Inc. All Rights Reserved.Testing, Testing, Testing!We used varnishtest- Different package for common and server roles- Unit tests for common settings- Integration tests for common + server role
    • Copyright © GREE, Inc. All Rights Reserved.varnishtest(failed log)I love it, but its difficult to read
    • Copyright © GREE, Inc. All Rights Reserved.varnishtest(failed log)I love it, but its difficult to read
    • Copyright © GREE, Inc. All Rights Reserved.vtctrans.pyHuman readable ;)- Sort by- Event- Error output- Header and expected result- Macro definehttps://github.com/xcir/vtctrans
    • Copyright © GREE, Inc. All Rights Reserved.ScreenshotEventvarnishtestvtctrans
    • Copyright © GREE, Inc. All Rights Reserved.ScreenshotEventvarnishtestvtctrans
    • Copyright © GREE, Inc. All Rights Reserved.ScreenshotMacro listvarnishtestvtctrans
    • Copyright © GREE, Inc. All Rights Reserved.ScreenshotMacro listvarnishtestvtctrans
    • Copyright © GREE, Inc. All Rights Reserved.ScreenshotHeader and expected resultvtctrans varnishtest
    • Copyright © GREE, Inc. All Rights Reserved.ScreenshotHeader and expected resultvtctrans varnishtest
    • Copyright © GREE, Inc. All Rights Reserved.Performance tuningMeasure, dont guess- Nevertheless, capacity planning is important :)- runtime parameters- server specs, placement- fault tolerance (keep service up even when a rack fails)- Measure with production servers, its the most importantthing- Useful commands- varnishncsa -m "txstatus:^[45]"- varnishstat- varnishhist -m "rxheader:Host: foo"- varnishlog -m "rxurl:/foo/bar"- Used varnishncsa and varnishlog alot
    • Copyright © GREE, Inc. All Rights Reserved.varnishlogugly
    • Copyright © GREE, Inc. All Rights Reserved.vsltrans.py(old:vltrans.php)beautiful!- Sort by- Error output- VCL action- variable- hash and vary- can use multiple -m optionhttps://github.com/xcir/vsltrans
    • Copyright © GREE, Inc. All Rights Reserved.ScreenshotGeneral, Error, Hash, Vary infovsltransand more...varnishlog
    • Copyright © GREE, Inc. All Rights Reserved.ScreenshotGeneral, Error, Hash, Vary infovsltransand more...varnishlog
    • Copyright © GREE, Inc. All Rights Reserved.ScreenshotActionvsltransvarnishlog
    • Copyright © GREE, Inc. All Rights Reserved.ScreenshotActionvsltransvarnishlog
    • Copyright © GREE, Inc. All Rights Reserved.ScreenshotVariablevsltransvarnishlog
    • Copyright © GREE, Inc. All Rights Reserved.ScreenshotVariablevsltransvarnishlog
    • Copyright © GREE, Inc. All Rights Reserved.More measurements- By different host- Cache hit rate- Transfer size per sec- Request per sec- By directory (specify depth)- Cache hit rate- Transfer size per sec- Request per sec- And more...
    • Copyright © GREE, Inc. All Rights Reserved.Why do we need a detailedmeasurements?- Hit rate is summarized by default- Different costs for different objects- Dynamic object: high cost- Static object: low cost- Its necessary to separate results by object cost- Need to consider system cost instead of justVarnish hit rate- However you cannot do this with just the defaultcommandsThats where libvarnishapi comes in
    • Copyright © GREE, Inc. All Rights Reserved.libvarnishapi.so- VSL access is easy with libvarnishapi- uses varnishlog, varnishncsa- Used Python ctypes and it proved to be very simple to use- Sample codehttps://github.com/xcir/python-varnishapi
    • Copyright © GREE, Inc. All Rights Reserved.python-varnishapi sample- VSL data output sample- Made it slightly more readable ;)
    • Copyright © GREE, Inc. All Rights Reserved.python-varnishapi sample
    • Copyright © GREE, Inc. All Rights Reserved.python-varnishapi sample
    • Copyright © GREE, Inc. All Rights Reserved.libvarnishapi: very easy to useHack on with libvarnishapi and Varnish!Better than making a measuring toolfrom scratch
    • Copyright © GREE, Inc. All Rights Reserved.Thank you for yourattention!Any Questions?@xcirhttp://blog.xcir.net/https://github.com/xcir/vtctranshttps://github.com/xcir/vsltranshttps://github.com/xcir/python-varnishapi