The document discusses ISUCON5 and its benchmark tools. It describes how the benchmark tools were developed with two main requirements - high performance and integrity checks. It then provides details on the key features and implementation of the benchmark tools using Java and the Jetty client library to send requests and check responses in parallel. Distributed benchmarking using multiple nodes is also covered. Performance results from ISUCON5 are shared showing it could handle around 194,000 requests per minute using up to 30 nodes.

1. Tales of ISUCON5
and Its Bench Tools
BizReach internal meetup (Nov 27, 2015)
Satoshi "Moris" Tagomori (@tagomoris)

2. Satoshi "Moris" Tagomori
(@tagomoris)
Fluentd, Norikra, Hadoop,
MessagePack-Ruby, Woothee, ...
Treasure Data, Inc.

4. http://www.treasuredata.com/

5. HQ
Branch

7. ISUCON5 Main Topics
• Qualify: "ISUxi"
• Good old SNS
• Friend relations, Footprint, Many N+1 queries, ...
• Final: "AirISU"
• API aggregate server
• Parallel requests, Application processes/threads,
Cache based on data/protocol, HTTP/2, ...

8. How to Get High Score
in ISUCON5
• Qualify
• Add index, Cache master data, Remove N+1, ...
• Final
• Massive threads, Cache invariable data, Async/
Parallel requests to APIs, If-Modified-Since,
HTTP/2, ...

9. BUG

10. Bugs in Organizer Side
• Qualify
• Nothing serious
• Last bug was fixed at 1st day 11:30am
• Final
• Some serious bugs in scenario to make effects
for top N players

11. Benchmark Tool

12. Why ISUCON Bench Tools
Should Be Written Newly?
• Two inconsistent requirements:
• high performance
• integrity check
• 1 request pattern for 2 requirements
• players can cheat w/ different requests for
purposes

13. Requirements in detail
• Performance
• throughput, concurrency, low latency
• Content check
• HTML parser, JSON parser, CSS/JS check, Image, other binaries, ...
• Complex scenario coding
• tools should simulate user behavior
• Protocol handling in detail
• HTTP protocols, HTTP headers, keepalive, cache control, timeouts, ...
• Variable source data
• disable "cache all requests/response" strategy

14. Features
• Sending GET/POST requests
• w/ various query params a/o content body
• w/ various HTTP headers
• Sending request series for a session
• Sending request series for several sessions
• Checking response integrity/consistency
• Skipping response check for performance if needed

15. Rough Sketch
• http_load + custom ruby script
• http_load: requests for performance
• ruby script: requests for checks

16. Sessions!
(http_load cannot handle sessions)

17. Overview
• Java: jetty-client + Java8 Lambda
• jetty-client for performance
• lambda for content check
• jackson to parse input data
• jsoup to parse response html (CSS selector)
• json-path to parse response json (JsonPath)

18. {
getAndCheck(session, "/", "GET INDEX BEFORE SIGNUP", (check) -> { check.isRedirect("/login"); });
getAndCheck(session, "/login", "GET LOGIN BEFORE SIGNUP", (check) -> {
check.isStatus(200);
check.isContentType("text/html");
if (! check.hasViolations()) {
check.exist("form.form-signin[action=/login]");
check.exist("input[name=email]");
check.exist("input[name=password]");
check.exist("button[type=submit]");
check.hasStyleSheet("/css/bootstrap.min.css");
check.hasStyleSheet("/css/signin.css");
}
});
getAndCheck(session, "/css/bootstrap.min.css", "BOOTSTRAP CSS", (check) -> {
check.isStatus(200);
if (! check.hasViolations()) {
check.isContentBodyChecksum("08df9a96752852f2cbd310c30facd934e348c2c5");
}
});
getAndCheck(session, "/css/signin.css", "SIGNIN CSS", (check) -> {
check.isStatus(200);
if (! check.hasViolations()) {
check.isContentBodyChecksum("702783cc5eff3d8d3532e339ddd15c57f7a08776");
}
});
}
Bootstrap.java

19. while (true) {
if (LocalDateTime.now().isAfter(stopAt))
break;
Session s = sessions.get(random.nextInt((int) sessions.size()));
get(s, "/login");
get(s, "/css/bootstrap.min.css");
get(s, "/css/signin.css");
post(s, "/login", formLogin(s));
if (LocalDateTime.now().isAfter(stopAt))
break;
get(s, "/");
get(s, "/css/bootstrap.min.css");
get(s, "/css/jumbotron-narrow.css");
get(s, "/js/jquery-1.11.3.js");
get(s, "/js/bootstrap.js");
get(s, "/js/airisu.js");
get(s, "/user.js");
if (LocalDateTime.now().isAfter(stopAt))
break;
for (int i = 0 ; i < 10 ; i++) {
get(s, "/data");
if (LocalDateTime.now().isAfter(stopAt))
break;
}
if (LocalDateTime.now().isAfter(stopAt))
break;
}
Load.java

20. @Override
public Result finishHook(Result result) {
long requests = result.requests;
if (result.responses.exception * 100.0 / (requests * 1.0) >= 1.0) {
result.addViolation("Too many exceptions", "通信エラー等の失敗が多過ぎます(1%以上)");
result.fail();
}
if (result.responses.error * 100.0 / (requests * 1.0) >= 1.0) {
result.addViolation("Too many errors", "ステータス 5xx のレスポンスが多過ぎます(1%以上)");
result.fail();
}
if (result.responses.failure * 100.0 / (requests * 1.0) >= 5.0) {
result.addViolation("Too many failures", "ステータス 4xx のレスポンスが多過ぎます(5%以上)");
result.fail();
}
return result;
}
@Override
public Step[] steps() {
Step[] steps = new Step[3];
steps[0] = new Step(35000L, Init.class);
steps[1] = new Step(60000L, Bootstrap.class);
steps[2] = new Step(
70000L,
Checker.class, ModifyLoader.class,
Load.class, Load.class, Load.class, Load.class, Load.class, Load.class, Load.class, Load.class,
Load.class, Load.class, Load.class, Load.class, Load.class, Load.class, Load.class, Load.class,
Load.class, Load.class, Load.class, Load.class, Load.class, Load.class, Load.class, Load.class,
Load.class, Load.class, Load.class, Load.class, Load.class, Load.class, Load.class, Load.class,
Load.class, Load.class, Load.class, Load.class, Load.class, Load.class, Load.class, Load.class,
Load.class, Load.class, Load.class, Load.class, Load.class, Load.class, Load.class, Load.class
);
return steps;
} Full.java

21. Code Example
• Simple scenario
• https://github.com/isucon/isucon5-final/blob/master/bench/src/main/java/net/isucon/
isucon5f/bench/Load.java
• https://github.com/isucon/isucon5-final/blob/master/bench/src/main/java/net/isucon/
isucon5f/bench/Checker.java
• https://github.com/isucon/isucon5-final/blob/master/bench/src/main/java/net/isucon/
isucon5f/bench/Bootstrap.java
• https://github.com/isucon/isucon5-final/blob/master/bench/src/main/java/net/isucon/
isucon5f/bench/Init.java
• Complex scenario
• https://github.com/isucon/isucon5-final/blob/master/bench/src/main/java/net/isucon/
isucon5f/bench/Full.java

22. Distributed Benchmarking
• N nodes for 1 benchmark
• Can a node perform fast enough? (CPU bounded)
• Y -> 1 vs 1, N -> 2 vs 1
• "GET /": 5000req/thread on localhost -> enough
• N nodes for many benchmarks
• Scale out strategy
• Queue/worker system

23. Scaling out nodes
ISUCON Portal (RoR)
Queue (MySQL)
Daemon script (ruby)
Bench (Java)
Daemon script (ruby)
Daemon script (ruby)
Daemon script (ruby)
Bench (Java)
ack req/set result
Daemon script (ruby)
Bench (Java)

24. Recorded Performance
• about 194,000 requests / 60 sec
• OK: 185,000
• Redirects: 9,300
• Peak 30 nodes (Qualify)
• 11,500 benchmarks in 2days

25. Far more: Scenario
• Checking for critical issues / non-critical issues
• Critical/non-critical mode of Checker class
• Checks w/ dependencies
• If a check fails, following checks throws NPE :(

26. Some More Topics
• Async
• Gigantic parallel requests
• 2 or more simultaneous requests under control

27. Java8: What and How I feel
about it

28. OSS: net.isucon.bench.*