Http capturing

HTTP Capturing &
forwarding
Platform Architecture팀 안 병욱M
2015.11

references
• project
• tcpdump(http://rhelmer.org/blog/capture-and-replay-http-post-using-tcpdump)
• tcpreplay(http://tcpreplay.synfin.net/wiki/usage)
• gor(https://github.com/buger/gor)
• node : capture-proxy(https://github.com/socsieng/capture-proxy)
• request와 response를 저장할 수 있지만, 버그와 기능이 부족하여 상용에 적용하기 어려움
• blog
• http://leonsbox.com/blog/2013/06/04/improving-testing-by-using-real-traffic-from-production/
• https://www.igvita.com/2008/09/30/load-testing-with-log-replay/
• knowledge markets
• http://serverfault.com/questions/570761/how-to-duplicate-tcp-traffic-to-one-or-multiple-remote-servers-
for-benchmarking
• http://stackoverflow.com/questions/7247668/duplicate-tcp-traffic-with-a-proxy

• tcpdump
• tcpreplay
• gor

tcpdump 를 이용하기
production에서 사용하기위해 가장 원시적인 방법

$tcpdump -i eth0 dst port 81 -C 100 -z “gzip” -w output.dmp -Z webadmin
port 81번만 캡춰
-C -z : output.pcap 파일을 100 MB 단위로 rolling
output.dmp
output.dmp1.gz
output.dmp2.gz
“webadmin” id를 지정

$ tcpslice -w full.pcap 2015y11m25d15h30m 2015y11m25d16h40m output.dmp
output.dmp에서 특정 시간의 데이타를 얻어오고, 여
러개로 나뉜 파일을 새로운 단일 파일(full.pcap)로 저
장

전체 dump한 내용에서, 하나의 HTTP request를 한
개의 파일로 저장.

request마다 무작위의 이름으로 저장됨

cks2ckt_contents.dat 파일 내용

zy2zz_contents.dat 의 내용을 읽어
서 slodev-rhatf2 호스트에 8180 포트
로 전송
response 가 출력됨.

$ cat aaju2aajv_contents.dat | sed 's/Host: prodserver/Host: devserver/' | nc devserver 80
Host header 수정한 후 request를 보
낼 수 있음

tcpdump등 유틸리티 사용 필수
많은 디스크 공간이 필요함
매뉴얼 방식이라 불편함

tcpreplay
• 다양한 ﬁrewall, IDS, IPS등의 네트워킹 제조사와 대학
등에서 사용
• pcap 파일을 replay 하는데 이용
• 네트웍 장비의 성능 테스트를 위해 이용
• HTTP capture/replay에는 적당하지 않음.

root@pw29:~$ sudo tcpreplay -i eth7 -t -K --loop 5000 smallFlows.pcap
File Cache is enabled
Actual: 71305000 packets (46082655000 bytes) sent in 38.03 seconds.
Rated: 1201832266.1 Bps, 9614.65 Mbps, 1859629.17 pps
Flows: 1209 flows, 31.53 fps, 71215000 flow packets, 90000 non-flow
Statistics for network device: eth7
Attempted packets: 71305000
Successful packets: 71305000
Failed packets: 0
Truncated packets: 0
Retried packets (ENOBUFS): 0
Retried packets (EAGAIN): 0
smallFlows.pcap 을 eth7 네트웍 인터페이스
에 -t(테스트)를 하는데, cap 파일을 메모리로 로
딩해서(K), 5,000(-loop) 번 반복해서 트래픽을
생성하여라.

nginx:80
apache:9000
replay
gor listen -p 80
webserver A
webserver B
gor listen?

-ip="0.0.0.0": Specify IP address to listen
-p=80: Specify the http server port whose traffic you
want to capture
-r="localhost:28020": Address of replay server.
-verbose=false: Log requests
-cpuprofile="": write cpu profile to file
-memprofile="": write memory profile to this file

$ sudo gor listen -p 80 -r="localhost:9000" -verbose=true
Version: 0.3.3
Listening for HTTP traffic on 0.0.0.0:80
Forwarding requests to replay server: localhost:9000
2015/11/23 09:29:17 Forwarding request: &{GET / HTTP/1.1 1 1 map[Cache-Control:[max-age=0]
User-Agent:[Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_1) AppleWebKit/537.36 (KHTML, like
Gecko) Chrome/46.0.2490.86 Safari/537.36] Accept-Encoding:[gzip, deflate, sdch] Connection:[keep-
alive] Accept:[text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8] Upgrade-
Insecure-Requests:[1] Accept-Language:[en-US,en;q=0.8,ko;q=0.6] If-Modified-Since:[Mon, 04 Oct
2004 15:04:06 GMT]] 0xc2000d4400 0 [] false 192.168.100.10 map[] map[] <nil> map[] / <nil>}
80 포트를 감시하고, 9000 포트로
replay를 한다.

gor replay -p 80
apache:9000
forward
webserver A
webserver B
apache:9000
webserver Cforward
gor replay ?

-f="http://localhost:8080": http address to forward trafﬁc.
You can limit requests per second by adding `|num` after address.
If you have multiple addresses with different limits. For example:
http://staging.example.com|100,http://dev.example.com|10
-ip="0.0.0.0": ip addresses to listen on
-p=28020: specify port number
-verbose=false: Log requests

$ sudo gor replay -ip="192.168.100.10" -p=80 -f="http://192.168.100.10:9000" -verbose=true
Version: 0.3.3
Listening for HTTP trafﬁc on 192.168.100.10:80
Forwarding requests to replay server: 192.168.100.10:9000
80 포트로 띄우고, 9000 포트로
forward

nginx:80
gor listen & replay
gor listen 80
Production(Webserver)
API
Gateway
Production
(apache:
8000 or)
gor replay
API
Gateway
staging
API
Gateway
dev
HTTP 10 rps
HTTP 5 rps

gor 0.3.3
HTTP trafﬁc record/replay 기능이 없다.
단, HTTP 트래픽을 staging/dev 존으로 실시간 전달
하여 테스트하기 용이함.

-debug=false: Turn on debug output, shows all itercepted traffic. Works only when with `verbose` flag
-http-allow-header=[]: A regexp to match a specific header against. Requests with non-matching headers will be dropped:
gor --input-raw :8080 --output-http staging.com --http-allow-header api-version:^v1
-http-allow-method=[]: Whitelist of HTTP methods to replay. Anything else will be dropped:
gor --input-raw :8080 --output-http staging.com --http-allow-method GET --http-allow-method OPTIONS
-http-allow-url=[]: A regexp to match requests against. Filter get matched against full url with domain. Anything else will be dropped:
gor --input-raw :8080 --output-http staging.com --http-allow-url ^www.
-http-disallow-header=[]: A regexp to match a specific header against. Requests with matching headers will be dropped:
gor --input-raw :8080 --output-http staging.com --http-disallow-header "User-Agent: Replayed by Gor"
-http-disallow-url=[]: A regexp to match requests against. Filter get matched against full url with domain. Anything else will be forwarded:
gor --input-raw :8080 --output-http staging.com --http-disallow-url ^www.
-http-header-limiter=[]: Takes a fraction of requests, consistently taking or rejecting a request based on the FNV32-1A hash of a specific header:
gor --input-raw :8080 --output-http staging.com --http-header-imiter user-id:25%
-http-original-host=false: Normally gor replaces the Host http header with the host supplied with --output-http. This option disables that behavior, preserving the original Host header.
-http-param-limiter=[]: Takes a fraction of requests, consistently taking or rejecting a request based on the FNV32-1A hash of a specific GET param:
gor --input-raw :8080 --output-http staging.com --http-param-limiter user_id:25%
-http-rewrite-url=[]: Rewrite the request url based on a mapping:
gor --input-raw :8080 --output-http staging.com --http-rewrite-url /v1/user/([^/]+)/ping:/v2/user/$1/ping
-http-set-header=[]: Inject additional headers to http reqest:
gor --input-raw :8080 --output-http staging.com --http-set-header 'User-Agent: Gor'
-http-set-param=[]: Set request url param, if param already exists it will be overwritten:
gor --input-raw :8080 --output-http staging.com --http-set-param api_key=1
-input-dummy=[]: Used for testing outputs. Emits 'Get /' request every 1s
-input-file=[]: Read requests from file:
gor --input-file ./requests.gor --output-http staging.com
-input-http=[]: Read requests from HTTP, should be explicitly sent from your application:
# Listen for http on 9000
gor --input-http :9000 --output-http staging.com
-input-raw=[]: Capture traffic from given port (use RAW sockets and require *sudo* access):
# Capture traffic from 8080 port
gor --input-raw :8080 --output-http staging.com
-input-tcp=[]: Used for internal communication between Gor instances. Example:
# Receive requests from other Gor instances on 28020 port, and redirect output to staging
gor --input-tcp :28020 --output-http staging.com
-middleware="": Used for modifying traffic using external command
-output-dummy=[]: Used for testing inputs. Just prints data coming from inputs.
-output-file=[]: Write incoming requests to file:
gor --input-raw :80 --output-file ./requests.gor
-output-http=[]: Forwards incoming requests to given http address.
# Redirect all incoming requests to staging.com address
gor --input-raw :80 --output-http http://staging.com
-output-http-elasticsearch="": Send request and response stats to ElasticSearch:
gor --input-raw :8080 --output-http staging.com --output-http-elasticsearch 'es_host:api_port/index_name'
-output-http-header=[]: WARNING: `--output-http-header` DEPRECATED, use `--http-set-header` instead
-output-http-header-filter=[]: WARNING: `--output-http-header-filter` DEPRECATED, use `--http-allow-header` instead
-output-http-header-hash-filter=[]: WARNING: òutput-http-header-hash-filter` DEPRECATED, use `--http-header-hash-limiter` instead
-output-http-method=[]: WARNING: `--output-http-method` DEPRECATED, use `--http-allow-method` instead
-output-http-redirects=0: Enable how often redirects should be followed.
-output-http-rewrite-url=[]: WARNING: `--output-http-rewrite-url` DEPRECATED, use `--http-rewrite-url` instead
-output-http-stats=false: Report http output queue stats to console every 5 seconds.
-output-http-timeout=0: Specify HTTP request/response timeout. By default 5s. Example: --output-http-timeout 30s
-output-http-url-regexp=[]: WARNING: `--output-http-url-regexp` DEPRECATED, use `--http-allow-url` instead
-output-http-workers=0: Gor uses dynamic worker scaling by default. Enter a number to run a set number of workers.
-output-tcp=[]: Used for internal communication between Gor instances. Example:
# Listen for requests on 80 port and forward them to other Gor instance on 28020 port
gor --input-raw :80 --output-tcp replay.local:28020
-output-tcp-stats=false: Report TCP output queue stats to console every 5 seconds.
-split-output=false: By default each output gets same traffic. If set to `true` it splits traffic equally among all outputs.
-stats=false: Turn on queue stats output
-verbose=false: Turn on more verbose output
0.10.x 에는 많은 옵션이 존재함.
HTTP-*,
INPUT-*,
OUTPUT-*

1. 포트에서 캡춰하기
http://production.com
apache2:80
gor —input-raw :80
http://staging.com:9000
nginx:9000
$ cat /etc/hosts
192.168.100.10 production.com
192.168.100.10 staging.com
192.168.100.10 dev.com
192.168.100.10 replay.local
$ sudo service apache2 start
$ sudo service nginx start
$ sudo gor --input-raw :80 --output-http http://staging.com:9000 --verbose=true
input-raw : for capturing
gor process를 listening과 replaying으로 설정
CPU resource가 충분한 경우에 추천

2. 포트에서 캡춰하기 : relay 사용시
apache2:80
nginx:9000
$ sudo gor --input-raw :80 --output-tcp 127.0.0.1:28020
$ gor --input-tcp 127.0.0.1:28020 --output-http http://staging.com:9000
gor replay.local:28020
replay 구성을 추천
output-http : payload가 200kb보다 작아야한다.

$ sudo gor --input-raw :80 --output-http 127.0.0.1:28020
$ gor --input-http :28020 --output-http http://staging.com:9000
input-http 를 사용을 추천한다.
input-tcp : missing requests 발생

3. multiple address(not split)
apache2:80
nginx:9000
gor 127.0.0.1:28020
$ sudo gor --input-raw :80 --output-http 127.0.0.1:28020
$ gor --input-http 127.0.0.1:28020 --output-http “http://staging.com:9000” --output-http “http://dev.com:8000”
http://dev.com:8000
lighttpd:8000
4 requests
4 requests
4 requests

4. multiple address(split)
apache2:80
nginx:9000
gor 127.0.0.1:28020
$ sudo gor --input-raw :80 --output-http 127.0.0.1:28020 --verbose=true
$ gor --input-http 127.0.0.1:28020 --output-http “http://staging.com:9000” --
output-http “http://dev.com:8000” --split-output true
http://dev.com:8000
lighttpd:8000
4 requests
2 requests
2 requests
4 request는 대략, 2 request s + 2 requests
로 분리되어 처리한다.

5. HTTP output workers
$ gor --input-http 127.0.0.1:28020 --output-http “http://staging.com:9000” --
output-http “http://dev.com:8000” --output-http-workers=20
apache2:80
nginx:9000
default : output-http-workers는 10 이고,
dynamic 하게 변한다.
위와 같이 20 으로 고정할 수도 있다.
queue

The optimal number of output-http-workers???
output-workers =
(Average number of requests/s)
————————————————
(Average target response time/s)
70 /s
—————
50 ms
= 1,400OPi-web01 =>

2015/11/27 01:41:13 [HTTPClient] Connection error: dial tcp 192.168.100.10:9000: too many open ﬁles
too many open ﬁles…
default :1024 에서 조정해야함
$ ulimit -n 64000

gor 믿을 만한가?
대량의 트래픽 발생시?
( $ ab -n 300000 -c 10000 http://production.com )
gor? crash를 발생하는 경우를 경험했다.
Crash~

6. Follow redirect
$ gor --input-http 127.0.0.1:28020 --output-http http://staging.com --output-http-redirects 2
nginx:9000
apache2:80
2) status : 3xx
1)
3)
output-http에서 redirect 발생시 gor
가 redirect를 처리한다.

$ cat /var/wwwapache/.htaccess
RewriteEngine on
RewriteRule ^ move.html [R=302,L]
$ tail -f /var/log/apache/access.log
192.168.100.10 - - [27/Nov/2015:04:07:03 +0000] "GET / HTTP/1.0" 302 594 "-" "ApacheBench/2.3"
192.168.100.10 - - [27/Nov/2015:04:07:03 +0000] "GET http://staging.com:9000//var/wwwapache/move.html HTTP/1.1" 400 510 "-" "-"
--outout-http-redirects=2
--outout-http-redirects : 사용하지 않는다면, 아래
와 같이 redirect 요청을 staging.com:9000 으로
보내지 않는다

$ gor --input-http 127.0.0.1:28020 --output-http “http://staging.com:9000” —
output-http-timeout 30s
7. HTTP timeouts
default http timeout :
request : 5s
response : 5s
아래와 같이 30s로 변경할 수 있다.

Rate limiting
• dropping random requests
• dropping requests based on Header or URL param value

8. Rate limiting - absolute number
$ sudo gor --input-raw :80 --output-http 127.0.0.1:28020 --
verbose=true
$ gor --input-http 127.0.0.1:28020 --output-http “http://staging.com:
9000|10”
$ ab -n 100 -c 5 http://production.com/
매초 10 requests까지만을
staging.com 으로 relay
apache2:80
nginx:9000

9. Rate limiting - percentage
$ sudo gor --input-raw :80 --output-tcp “127.0.0.1:28020|10%” --
verbose=true
$ gor --input-tcp 127.0.0.1:28020 --output-http http://staging.com:
9000
trafﬁc이 많은 환경에서는 listener를
제한하는게 좋은 방법이다. 전체 요청중
10%만 받아서 처리함.
apache2:80
nginx:9000

10. Rate limiting - Header or URL param value
$ sudo gor --input-raw :80 --output-tcp “127.0.0.1:28020|10%” --http-header-limiter “X-API-KEY: 10%”
or $ sudo gor --input-raw :80 --output-tcp “127.0.0.1:28020|10%” --http-param-limiter “api_key=3: 10%”
$ gor --input-tcp 127.0.0.1:28020 --output-http “http://staging.com:9000”
$ ab -n 10 -H “X-API-KEY:” http://production.com/
$ ab -n 10 http://production.com?api_key=3
header or param 이 있으면 제한가능
아래 예에서,10 requests를 보내면, 1
request만 relay를 함, output-tcp(10%)
와 limiter(10%)를 동시에 설정해야하는데
버그인것 같다.
apache2:80
nginx:9000

Filtering
• Allow url regexp
• Disallow url regex
• Filter header or http method

11. Allow url regexp
$ sudo gor --input-raw :80 --output-tcp “127.0.0.1:28020” --http-allow-url /api
$ gor --input-tcp 127.0.0.1:28020 --output-http “http://staging.com:9000” --http-allow-url /api
$ ab -n 100 -c 5 http://production.com/api
apache2:80
nginx:9000
replay 전/후 모두 설정할 수 있다.
아래 예는, /api uri만 relay한다.

12. Disallow url regexp
$ sudo gor --input-raw :80 --output-tcp “127.0.0.1:28020” --http-disallow-url /api
$ gor --input-tcp 127.0.0.1:28020 --output-http “http://staging.com:9000” --http-disallow-url /api
nginx:80
apache2:9000
replay 전/후 모두 설정할 수 있다.
아래 예는, /api uri 는 relay처리를 안한다.

13. Filter(header)
$ sudo gor --input-raw :80 --output-tcp “127.0.0.1:28020” —http-allow-header “api-version:^1.0d”
$ ab -n 100 -c 5 -H “api-version:1.00” http://production.com/api
nginx:80
apache2:9000
http-allow-header
http-disallow-header
api-version 1.00xxx ~ 1.09xxx 허
용

14. Filter(http method)
$ sudo gor --input-raw :80 --output-tcp “127.0.0.1:28020” —http-allow-method GET
nginx:80
apache2:9000
GET method만을 허용한다.
(whitelist)

Rewriting original request
• rewrite-url
• URL param 설정하기
• Header 설정하기
• Host header 설정하기

15. Rewrite URL
$ sudo gor --input-raw :80 --output-tcp “127.0.0.1:28020” —http-rewrite-url /abc:/api/
$ ab -n 100 -c 5 http://production.com/abc
nginx:80
apache2:9000
/abc는 rewrite-url 조건에 따라, /api
를 호출하게 된다.

16. URL param 설정하기
$ sudo gor --input-raw :80 --output-tcp “127.0.0.1:28020” --http-set-param api_key=1
$ ab -n 100 -c 5 http://production.com/api_key=3
nginx:80
apache2:9000
api_key=3 로 요청하면,
=> api_key=1 로 변경하여
staging.com 으로 전달한다.

17. header 설정하기
$ sudo gor --input-raw :80 --output-tcp “127.0.0.1:28020” —http-set-header “User-Agent: Replayed
by Gor”
nginx:80
apache2:9000
임의의 header를 설정할 수 있다.

18. Middleware : bash/java/golang/python…
$ sudo gor --input-raw :80 --middleware "/opt/middleware_executable/echo.sh" --output-http
"http://staging.com:9000" --verbose --debug
$ sudo gor --input-raw :80 --middleware "java -cp /opt/middleware_executable/ echo" --
output-http "http://staging.com:9000" --verbose --debug

19. save and replay
$ sudo gor --input-raw :80 --output-file /tmp/requests.gor
// 100만 request => request.gor : 649M
$ sudo gor --input-file /tmp/requests.gor --output-http "http://staging.com:9000"
nginx:80
apache2:9000
--output-file / --input-file 사용
requests.gor

requests.gor : 649MB <- 100만 requests
$ sudo gor --input-ﬁle /tmp/requests.gor --output-http "http://staging.com:9000"

$ cat requests.gor
1 b6932617bd2e5bedc0aa0d5848f8fe7726b77cef 1448943857456852974
GET / HTTP/1.0^M
Host: production.com^M
User-Agent: ApacheBench/2.3^M
Accept: */*
2 b6932617bd2e5bedc0aa0d5848f8fe7726b77cef 141596
HTTP/1.1 200 OK^M
Date: Tue, 01 Dec 2015 04:24:17 GMT^M
Server: Apache/2.2.22 (Ubuntu)^M
Last-Modiﬁed: Tue, 01 Dec 2015 01:55:48 GMT^M
ETag: "3a0c09-b9-525cc74ec3d4f"^M
Accept-Ranges: bytes^M
Content-Length: 185^M
Vary: Accept-Encoding^M
Connection: close^M
Content-Type: text/html^M
^M
<html><body><h1>It works!, apache</h1>
<p>This is the default web page for this server.</p>
<p>The web server software is running but no content has been added, yet.</p>
</body></html>
…
…
Response payload :
Type+ ‘ ‘ + uuid(sha1 of time and ack) + ‘ ‘ + round-
trip time
Type =>
const (
RequestPayload = '1'
ResponsePayload = '2'
ReplayedResponsePayload = '3'
)
Request payload :
Type+ ‘ ‘ + uuid(sha1 of time and ack) + ‘ ‘ + start
time(nano 단위)
Type =>
const (
RequestPayload = '1'
ResponsePayload = '2'
ReplayedResponsePayload = '3'
)

20. Load testing
$ sudo gor --input-raw :80 --output-ﬁle requests.gor
$ gor --input-ﬁle “requests.gor|1%” --output-http “http://staging.com:9000"
nginx:80
apache2:9000
Flow control
1% : 느리게 relay를 처리함.
200% : 2x 속도로 relay를 처리함.
requests.gor

21. SSL replay
https://production.com
apache:80
proxy_pass http://localhost:8000
lighttpd:8000
production(backend)
• gor는 직접 SSL을 캡춰할 수 없다
• 아래와 같이 apache/nginx에서 proxy 설정을 해야한다.
$ sudo gor --input-raw :8000 --output-tcp 127.0.0.1:28020 --verbose=true
$ gor --input-tcp 127.0.0.1:28020 --output-http http://staging.com:9000 --verbose=true
nginx:9000

host header 설정하기??????
$ sudo gor --input-raw :80 --output-tcp “127.0.0.1:28020”
$ gor --input-tcp 127.0.0.1:28020 —http-original-host --output-http “http://staging.com:9000”
$ ab -n 100 -c 5 http://production.com/api_key=3
nginx:80
apache2:9000
http-original-host를 설정하면,
original header를 유지한다.

Http capturing

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Viewers also liked

Viewers also liked (12)

Similar to Http capturing

Similar to Http capturing (20)

Recently uploaded

Recently uploaded (20)

Http capturing