Introduction httpClient on Java11 / Java11時代のHTTPアクセス再入門

tamtam180
tamtam180 Square Enix
Java11時代のHTTPアクセス再入門
JJUG CCC 2019 Spring
@tamtam180
2019/05/18
アジェンダ
• 自己紹介
• おさらい
– HttpUrlConnectionの話
– Apache HttpClientの話
• java.net.http.HttpClientの話
アジェンダ
• 話さないこと
– HTTP/2の基本的な話
自己紹介
• Name: Kiyotaka Suzuki
• Twitter: @tamtam180
• Main works
– Square Enix (5.5 Year)
• PlayOnline, FF-XIV, etc,
– SmartNews (5 Year)
• Cross functional Expert Team
• Senior Software Engineer
• Advertising System, PoC, SRE, etc,..
• I Love OSS about Datastore
• Mad performance tuner
その前に
• 検証に便利なWebサイト
– https://http2.pro/api/v1
• HTTP/2のテスト
– https://httpbin.org/
• HTTPメソッド、ステータス、Cookie、Cache、etc
• このサイトでほぼ事足りる
– https://httpstat.us/
• HTTPステータス
– https://badssl.com/
• HTTPSのエラーテスト
– https://www.websocket.org/echo.html
• WebSocketのテスト
おさらい
• 今までの主なHTTPアクセス
– HttpUrlConnection
– Apache HttpClient
– OkHttp
おさらい
• HttpUrlConnection
– 同期処理だけ
– HTTP1.1まで
– Cookieは一応扱える
• OnMemory
– Basic認証も出来る
• QueryStringを作るのが大変(個人的感想)
• Responseのgzip処理とか大変
おさらい
• Apache HTTPClient
– 5系からHTTP/2が使える(5系はβバージョン)
– 3系から4系でプログラムの構造が大きく変更
• Commons HttpClientは3系, もう古い
– IO Model
• Blocking I/O
– Classic Java IO
– Non blocking
• Event Driven I/O with JavaNIO
おさらい
• Apache HTTPClient
– Core
• v4.4
• v5.0 beta
– Client
• v4.5
• v5.0 beta
– HttpAsyncClient
• v4.1
おさらい
• OkHttp (v3.14.1)
– HTTP/2対応
– Androidも対応
• Android5.0+ (API Level 21+)
– Java8+
– TLS1.3対応
おさらい
• java.net.http / HttpClient
– HTTP/2対応
– WebSocket対応
– 同期/非同期 処理対応
– Reactive Streamとして
Request/Responseを処理
– Builderパターン
おさらい HttpUrlConnection
おさらい: HttpUrlConnection
• GET
var url = new URL("https://httpbin.org/get");
var conn = (HttpURLConnection) url.openConnection();
try (InputStream in = conn.getInputStream()) {
System.out.println(conn.getResponseCode());
System.out.println(IOUtils.toString(in, StandardCharsets.UTF_8));
}
conn.disconnect();
"headers": {
"Accept": "text/html, image/gif, image/jpeg, *; q=.2, */*; q=.2",
"Host": "httpbin.org",
"User-Agent": "Java/11.0.2"
}
おさらい: HttpUrlConnection
• メソッドを変更する(例えばHEAD)
conn.setRequestMethod("HEAD");
おさらい: HttpUrlConnection
• Request Headerを弄る
conn.setRequestProperty("User-Agent", "MyJava/11.0.0");
conn.setRequestProperty("Content-Type", "application/json");
おさらい: HttpUrlConnection
• BODYを送信する
conn.setDoOutput(true);
try (var writer = new OutputStreamWriter(conn.getOutputStream())) {
writer.write(new Gson().toJson(Map.of("hello", "world")));
writer.flush();
}
おさらい: HttpUrlConnection
• POST(組み合わせ)
var url = new URL("http://httpbin.org/post");
HttpURLConnection conn = (HttpURLConnection) url.openConnection();
conn.setDoOutput(true);
conn.setRequestMethod("POST");
conn.setRequestProperty("User-Agent", "MyJava/11.0.0");
conn.setRequestProperty("Content-Type", "application/json");
try (OutputStreamWriter writer = new OutputStreamWriter(conn.getOutputStream())) {
writer.write(new Gson().toJson(Map.of("hello", "world")));
writer.flush();
}
try (InputStream in = conn.getInputStream()) {
System.out.println(conn.getResponseCode());
System.out.println(IOUtils.toString(in, StandardCharsets.UTF_8));
}
conn.disconnect();
おさらい: HttpUrlConnection
• Cookie
– CookieManager, CookieHandlerを使う
– 世の中にあるサンプル、本当に動く??
– CookieManagerはThreadSafeじゃない
CookieManager cookieManager = new CookieManager(null, CookiePolicy.ACCEPT_ALL);
CookieHandler.setDefault(cookieManager);
System.out.println(conn.getResponseCode());
CookieHandler.getDefault().put(conn.getURL().toURI(), conn.getHeaderFields());
for (HttpCookie cookie : cookieManager.getCookieStore().get(conn.getURL().toURI())) {
System.out.println(cookie);
}
これが無いと動かないのだけど..
おさらい: HttpUrlConnection
• Proxy
var proxy = new Proxy(
Proxy.Type.SOCKS,
InetSocketAddress.createUnresolved("127.0.0.1", 7777));
HttpURLConnection conn = (HttpURLConnection)
url.openConnection(proxy);
ssh any-server –D7777
HTTP Proxyの場合は、 Proxy.Type.HTTP
おさらい: HttpUrlConnection
• Redirect
– デフォルトはredirectを追従
var url = new URL("https://httpbin.org/redirect/2");
var conn = (HttpURLConnection) url.openConnection();
conn.setInstanceFollowRedirects(true);
おさらい: HttpUrlConnection
• Basic認証
– Header直接指定
– java.net.Authenticator
• requestPasswordAuthenticationInstanceを
上書き
• getPasswordAuthenticationを上書き
おさらい: HttpUrlConnection
• Basic認証
– Header直接指定
var url = new URL("https://httpbin.org/basic-auth/test-user/test-pass");
var conn = (HttpURLConnection) url.openConnection();
conn.setRequestProperty(
"Authorization",
"Basic " + Base64.getEncoder().encodeToString("test-user:test-
pass".getBytes()));
おさらい: HttpUrlConnection
• Basic認証
– java.net.Authenticator その1
Authenticator.setDefault(new Authenticator() {
@Override
protected PasswordAuthentication getPasswordAuthentication() {
if (getRequestorType() != RequestorType.SERVER) return null;
if (!"basic".equals(getRequestingScheme())) return null;
if ("httpbin.org".equalsIgnoreCase(getRequestingHost())) {
return new PasswordAuthentication("test-user", "test-pass".toCharArray());
}
return null;
}
});
おさらい: HttpUrlConnection
• Basic認証
– java.net.Authenticator その2
Authenticator.setDefault(new Authenticator() {
@Override
public PasswordAuthentication requestPasswordAuthenticationInstance(String
host, InetAddress addr, int port, String protocol, String prompt, String scheme,
URL url, RequestorType reqType) {
if (reqType != RequestorType.SERVER) return null;
if (!"basic".equals(scheme)) return null;
if ("httpbin.org".equalsIgnoreCase(host)) {
return new PasswordAuthentication("test-user", "test-pass".toCharArray());
}
return null;
}
});
おさらい: HttpUrlConnection
• SSL Errorを無視する通信
SSLContext sslcontext = SSLContext.getInstance("SSL");
sslcontext.init(null, new TrustManager[]{new X509TrustManager() {
public void checkClientTrusted(X509Certificate[] x509Certificates, String s) {}
public void checkServerTrusted(X509Certificate[] x509Certificates, String s) {}
public X509Certificate[] getAcceptedIssuers() { return null; }
}}, null);
HttpsURLConnection.setDefaultHostnameVerifier((s, sslSession) -> true);
var url = new URL("https://expired.badssl.com/");
HttpURLConnection conn = (HttpURLConnection) url.openConnection();
if (conn instanceof HttpsURLConnection) {
((HttpsURLConnection) conn).setSSLSocketFactory(sslcontext.getSocketFactory());
}
SSLContext
Verifier
SocketFactory
おさらい: HttpUrlConnection
• デバッグの方法
– Logger指定
– java.net.debug
おさらい: HttpUrlConnection
• Logger指定
[logging.properties]
handlers = java.util.logging.ConsoleHandler
java.util.logging.ConsoleHandler.level = ALL
sun.net.www.protocol.http.HttpURLConnection.level = ALL
System.setProperty("java.util.logging.config.file", "logging.properties");
おさらい: HttpUrlConnection
• javax.net.debug
– Helpを指定するとhelpが表示される
System.setProperty("javax.net.debug", ”help");
URL url = new URL("https://www.google.com");
var conn = (HttpURLConnection) url.openConnection();
conn.getInputStream();
https://docs.oracle.com/javase/jp/8/docs/technotes/guides/security/jsse/JSSERefGuide.html
https://www.ibm.com/support/knowledgecenter/en/SSYKE2_7.0.0/com.ibm.java.security.component.70.doc/security-
all
ssl:record,handshake,keygen,session,defaultctx,sslctx,sessioncache,keymanager,trustmanager,plugga
bility
handshake:data,verbose
plaintext,packet
指定可能な値
例: ssl:handshake,session
例: all
カンマ区切りかコロン区切り
Apache HttpClient
ApacheHC/GET
• GET
var clientBuilder = HttpClientBuilder.create();
try (CloseableHttpClient client = clientBuilder.build()) {
var getMethod = new HttpGet(URI.create("http://httpbin.org/get"));
try (CloseableHttpResponse resp = client.execute(getMethod)) {
String body = EntityUtils.toString(resp.getEntity(), StandardCharsets.UTF_8);
System.out.println(resp.getStatusLine());
System.out.println(body);
}
}
ApacheHC/Accept Header
• 注意点
– デフォルトのヘッダはAccept Header無し
Host: httpbin.org
Connection: Keep-Alive
User-Agent: Apache-HttpClient/4.5.4 (Java/11.0.1)
Accept-Encoding: gzip,deflate
ApacheHC/Accept Header
• Accept Header
• (指定が無い場合は何でも受け付ける事を意
味する)
• と、書いてあるのに..
RFC7231: Section 5.3.2
A request without any Accept header field implies that the user
agent will accept any media type in response.
https://tools.ietf.org/html/rfc7231#section-5.3.2
ApacheHC/Accept Header
• サイトによっては正しく返却されない
– 例: http://httpstat.us/200
curl -H "Accept: */*" http://httpstat.us/200
# Content-Length: 6
200 OK
curl -H "Accept:" http://httpstat.us/200
# Content-Length: 0
ApacheHC/Accept Header
• 各ブラウザのAccept Header
– デフォルト値の一覧は以下のサイトを参照
https://developer.mozilla.org/en-
US/docs/Web/HTTP/Content_negotiation/List_of_default_Accept_values
ApacheHC/Gzip
• Gzipはデフォルトで処理される
– gzipもdeflateも
curl -s http://httpbin.org/gzip | gzip -d
var clientBuilder = HttpClientBuilder.create();
try (CloseableHttpClient client = clientBuilder.build()) {
var getMethod = new HttpGet("http://httpbin.org/gzip");
try (var resp = client.execute(getMethod)) {
String body = EntityUtils.toString(resp.getEntity(), StandardCharsets.UTF_8);
System.out.println(body);
}
}
本当はAccept Encodingを付ける必要がある。
httpbin.org/gzip や httpbin.org/deflate は
それを無視して送ってくるのでコードでは省略している。
ApacheHC
• Queryを作る時は URI Builder
var uri = new URIBuilder("http://httpbin.org/get")
.setCharset(StandardCharsets.UTF_8)
.addParameter("hello", "world")
.addParameter("sushi", "寿司")
.setScheme("https") // HTTPSに変更
.setFragment("hello")
.build();
https://httpbin.org/get?hello=world&sushi=%E5%AF%BF%E5%8F%B8#hello
ApacheHC
• Connection Pool
var connManager = new PoolingHttpClientConnectionManager();
connManager.setMaxTotal(100);
connManager.setDefaultMaxPerRoute(100);
var clientBuilder = HttpClientBuilder.create();
clientBuilder.setConnectionManager(connManager);
ApacheHC
• Request Headerをいじる
var clientBuilder = HttpClientBuilder.create();
var defaultHeaders = Arrays.asList(
new BasicHeader("Accept", "*/*")
);
clientBuilder.setDefaultHeaders(defaultHeaders);
var getMethod = new HttpGet(URI.create("http://httpbin.org/get"));
getMethod.setHeader("Accept", "*/*");
定数クラスがあるのでそれを使うと良い
HttpHeaders, ContentType
ApacheHC/POST
var clientBuilder = HttpClientBuilder.create();
try (CloseableHttpClient client = clientBuilder.build()) {
String jsonText = new Gson().toJson(Map.of("hello", "world"));
HttpEntity entity = EntityBuilder.create()
.setContentType(ContentType.APPLICATION_JSON)
.setText(jsonText).build();
HttpPost postMethod = new HttpPost(URI.create("http://httpbin.org/post"));
postMethod.setEntity(entity);
try (CloseableHttpResponse resp = client.execute(postMethod)) {
String body = EntityUtils.toString(resp.getEntity(), StandardCharsets.UTF_8);
System.out.println(resp.getStatusLine());
System.out.println(body);
}
}
ApacheHC
• やり方色々
– Client
• HttpClientBuilder.create().build();
• HttpClients.custom().build();
• HttpClients.createDefault();
– Entity
• EntityBuilder
• 直接生成
BasicHttpEntity
BufferedHttpEntity
ByteArrayEntity
FileEntity
InputStreamEntity
SerializableEntity
StringEntity
ApacheHC
• やり方色々
– execute
• 単純に実行
• HttpContext
• ResponseHandler
ApacheHC
• Responseの処理はEntityUtilsが便利
– toString()
– toByteArray()
– writeTo()
– consume()
– ...
ApacheHC/Redirect
• デフォルトで追従
• OFFにする場合は、RequestConfig
var requestConfig = RequestConfig.custom()
.setRedirectsEnabled(false) //追従しない
.build();
var clientBuilder = HttpClientBuilder.create();
clientBuilder.setDefaultRequestConfig(requestConfig);
var getMethod = new
HttpGet(URI.create("http://httpbin.org/redirect/3"));
getMethod.setConfig(requestConfig);
ApacheHC/Retry
• DefaultHttpRequestRetryHandler
– ConnectTimeoutExceptionはRetryしない
– InterruptedIOExceptionのSubClassだけど
https://hc.apache.org/httpcomponents-client-
ga/httpclient/apidocs/org/apache/http/impl/client/DefaultHttpRequestRetryHandler.html
ApacheHC/Cookie
var cookieStore = new BasicCookieStore();
var clientBuilder = HttpClientBuilder.create();
clientBuilder.setDefaultCookieStore(cookieStore);
// return Set-Cookie
try (CloseableHttpClient client = clientBuilder.build()) {
var get = new HttpGet("https://httpbin.org/cookies/set/hello/world");
try (CloseableHttpResponse resp = client.execute(get)) {}
}
// send Cookie: hello and sushi
var cookie = new BasicClientCookie("sushi", " 🍣");
cookie.setDomain("httpbin.org");
cookieStore.addCookie(cookie);
try (CloseableHttpClient client = clientBuilder.build()) {
var get = new HttpGet("https://httpbin.org/cookies");
try (CloseableHttpResponse resp = client.execute(get)) {}
}
GET /cookies HTTP/1.1
Host: httpbin.org
Connection: Keep-Alive
User-Agent: Apache-HttpClient/4.5.4 (Java/11.0.1
Cookie: hello=world; sushi= 🍣
Accept-Encoding: gzip,deflate
• Proxy
• 認証有りProxy
ApacheHC/Proxy
var clientBuilder = HttpClientBuilder.create();
var proxy = new HttpHost(proxyHost, proxyPort, "http");
clientBuilder.setProxy(proxy);
clientBuilder.setProxyAuthenticationStrategy(new ProxyAuthenticationStrategy());
var credsProvider = new BasicCredentialsProvider();
credsProvider.setCredentials(
AuthScope.ANY,
new UsernamePasswordCredentials(proxyUser, proxyPassword));
clientBuilder.setDefaultCredentialsProvider(credsProvider);
ApacheHC
• Basic認証
var u = URI.create("https://httpbin.org/basic-auth/kuma/pass");
var credProvider = new BasicCredentialsProvider();
credProvider.setCredentials(
new AuthScope(u.getHost(), u.getPort(), AuthScope.ANY_REALM,
AuthScope.ANY_SCHEME),
new UsernamePasswordCredentials("kuma", "pass"));
var clientBuilder = HttpClientBuilder.create();
clientBuilder.setDefaultCredentialsProvider(credProvider);
try (CloseableHttpClient client = clientBuilder.build()) {
var getMethod = new HttpGet(u);
try (CloseableHttpResponse resp = client.execute(getMethod)) {
System.out.println(resp.getStatusLine());
}
}
ApacheHC
• Basic認証 (ANYを使う:非推奨)
– 最低限、Host, Portは指定した方が良い
– Realmも本当は指定した方が良いけど
credProvider.setCredentials(
AuthScope.ANY,
new UsernamePasswordCredentials("kuma", "pass"));
ApacheHC/BadSSL
• BadSSLスルー
var sslContext = SSLContext.getInstance("SSL");
sslContext.init(null, new TrustManager[] { new X509TrustManager() {
public X509Certificate[] getAcceptedIssuers() { return null; }
public void checkClientTrusted(X509Certificate[] certs, String authType) { }
public void checkServerTrusted(X509Certificate[] certs, String authType) { }
} }, new SecureRandom());
var clientBuilder = HttpClientBuilder.create();
clientBuilder.setSSLSocketFactory(new SSLConnectionSocketFactory(sslContext));
try (CloseableHttpClient client = clientBuilder.build()) {
var getMethod = new HttpGet("https://expired.badssl.com/");
try (var resp = client.execute(getMethod)) {
String body = EntityUtils.toString(resp.getEntity(), StandardCharsets.UTF_8);
System.out.println(body);
}
}
ApacheHC/Async
• HttpAsyncClient
– Basicな方法
CloseableHttpAsyncClient client = HttpAsyncClients.createDefault();
client.start();
var request1 = new HttpGet("https://httpbin.org/get");
Future<HttpResponse> future = client.execute(request1, null);
// any processing
var resp = future.get();
System.out.println(resp.getStatusLine());
client.close();
ApacheHC/Async
• HttpAsyncClient
– Consumer / Producer
CloseableHttpAsyncClient client = HttpAsyncClients.createDefault();
client.start();
var producer = HttpAsyncMethods.create(new HttpGet("https://httpbin.org/get"));
var consumer = new AsyncCharConsumer<HttpResponse>() {
HttpResponse response;
protected void onCharReceived(CharBuffer charBuffer, IOControl ioControl) { }
protected void onResponseReceived(HttpResponse httpResponse) {
this.response = httpResponse;
}
protected HttpResponse buildResult(HttpContext httpContext) { return this.response; }
};
ApacheHC/Async
• HttpAsyncClient
– Consumer / Producer
var latch = new CountDownLatch(1);
client.execute(producer, consumer, new FutureCallback<>() {
public void completed(HttpResponse httpResponse) { latch.countDown(); }
public void failed(Exception e) { latch.countDown(); }
public void cancelled() { latch.countDown(); }
});
latch.await();
System.out.println(consumer.response.getStatusLine());
client.close();
ApacheHC/Debug
• DEBUG
– Loggerを設定する
– Lobackの例
<logger name="org.apache.http" additivity="true">
<level value="DEBUG"/>
</logger>
ApacheHC/不満点
• HTTP/2は まだ 対応していない
• WebSocket対応もまだ
java.net.http
おさらい
• java.net.http / HttpClient
– Java11で正式APIになった
– HTTP/2対応
– WebSocket対応
– 同期/非同期 処理対応
– Reactive Streamとして
Request/Responseを処理
– Builderパターン
java.net.http
• Clientの作り方
もしくは
[Header]
GET /get HTTP/1.1
Content-Length: 0
Host: httpbin.org
User-Agent: Java-http-client/11.0.1
var client = HttpClient.newHttpClient();
var request = HttpRequest.newBuilder()
.uri(URI.create("https://http2.pro/api/v1"))
.build();
var resp = client.send(request,
HttpResponse.BodyHandlers.ofString(StandardCharsets.UTF_8));
System.out.println(resp.statusCode());
System.out.println(resp.body());
var client = HttpClient.newBuilder().build();
[Body]
{"http2":1,"protocol":"HT
TP¥/2.0","push":1,"user_a
gent":"Java-http-
client¥/11.0.1"}
java.net.http
• デフォルトは、
– HTTP/2を勝手にやってくれる
• サイトが対応していない場合はHTTP/1.1で通信
– Redirectは勝手にしてくれない
– GZip/Deflateは勝手に処理してくれない
• 便利なURIBuilderは存在しない
java.net.http
• ClientBuilder
– authenticator
– connectTimeout
– cookieHandler
– executor
– followRedirects
– priority
– proxy
– sslContext
– version
– sslParameters
java.net.http
• BodyHandlers
– ofString
– ofByteArray
– ofByteArrayConsumer
– ofFile
– ofFileDownload
– ofInputStream
– ofLines
– ofPublisher
– ...
java.net.http/redirect
• Redirectの設定
var client = HttpClient.newBuilder()
.followRedirects(HttpClient.Redirect.NORMAL)
.build();
NEVER Redirectしない
ALWAYS 常にRedirectする
NORMAL 常にRedirectする(HTTPS->HTTPは除く)
java.net.http/Header
• Request Headerの編集
var request = HttpRequest.newBuilder()
.header("Accept", "*/*")
.header("User-Agent", "JJUG/1.0")
.uri(URI.create("https://httpbin.org/get"))
.build();
java.net.http/POST
• POSTとBody送信
var bodyString = new Gson().toJson(Map.of("hello", "world"));
var client = HttpClient.newBuilder().build();
var request = HttpRequest.newBuilder()
.header("Content-Type", "application/json")
.POST(HttpRequest.BodyPublishers.ofString(bodyString))
.uri(URI.create("https://httpbin.org/post"))
.build();
var resp = client.send(
request,
HttpResponse.BodyHandlers.ofString(StandardCharsets.UTF_8));
System.out.println(resp.statusCode());
System.out.println(resp.body());
java.net.http
• BodyPublishers
– noBody
– ofString
– ofByteArray
– ofByteArrays
– ofFile
– ofInputStream
– ...
java.net.http/method
• 他のメソッド
– GET(), POST(), DELETE(), PUT()
– HEADやPATCHは?
• methodを使う
.method("HEAD", HttpRequest.BodyPublishers.noBody())
.method("PATCH", HttpRequest.BodyPublishers.ofString(bodyString))
java.net.http/async
• 非同期: client.sendAsync
var client = HttpClient.newBuilder().build();
var request = HttpRequest.newBuilder(URI.create("https://httpbin.org/get")).build();
CompletableFuture<HttpResponse<String>> future =
client.sendAsync(request,
HttpResponse.BodyHandlers.ofString(StandardCharsets.UTF_8));
// CompletableFutureなので処理を繋げる事ができる
future.thenApply(HttpResponse::body)
.exceptionally(e -> "Error!! :" + e.getMessage())
.thenAccept(System.out::println);
//future.get();
java.net.http/content-encoding
• BodySubscribers.mappingを使う
– ※JDKの不具合で動きません
https://bugs.openjdk.java.net/browse/JDK-8217264
private HttpResponse.BodySubscriber<InputStream> gzippedBodySubscriber(
HttpResponse.ResponseInfo responseInfo) {
// 本当はheaderのContent-Encodingを確認する
return HttpResponse.BodySubscribers.mapping(
HttpResponse.BodySubscribers.ofInputStream(),
this::decodeGzipStream);
}
private InputStream decodeGzipStream(InputStream gzippedStream) {
try {
return new GZIPInputStream(gzippedStream);
} catch (IOException ex) { throw new UncheckedIOException(ex); }
}
var resp = client.send(request, this::gzippedBodySubscriber);
java.net.http/content-encoding
• Workaround-1
var client = HttpClient.newBuilder().build();
var request = HttpRequest.newBuilder()
.uri(URI.create("https://httpbin.org/gzip"))
.build();
var resp = client.send(request, HttpResponse.BodyHandlers.ofInputStream());
InputStream in = resp.body();
if ("gzip".equals(resp.headers().firstValue("Content-Encoding").get())) {
in = new GZIPInputStream(resp.body());
}
System.out.println(IOUtils.toString(in, StandardCharsets.UTF_8));
java.net.http/content-encoding
• Workaround-2
var future = client.sendAsync(request, HttpResponse.BodyHandlers.ofInputStream())
.thenApply(resp -> {
String ce = resp.headers().firstValue("Content-Encoding").get();
if ("gzip".equals(ce)) {
try {
return new GZIPInputStream(resp.body());
} catch (IOException e) { throw new UncheckedIOException(e); }
}
return resp.body();
});
var in = future.get();
System.out.println(IOUtils.toString(in, StandardCharsets.UTF_8));
java.net.http/content-encoding
• Workaround others
https://stackoverflow.com/questions/53379087/wrapping-
bodysubscriberinputstream-in-gzipinputstream-leads-to-hang
java.net.http/custom handler
• 例: json
public static class JsonBodyHandler<T> implements HttpResponse.BodyHandler<T> {
private Class<T> type;
private JsonBodyHandler(Class<T> type) { this.type = type; }
public static <T> JsonBodyHandler<T> jsonBodyHandler(Class<T> type) {
return new JsonBodyHandler<>(type);
}
@Override
public HttpResponse.BodySubscriber<T> apply( HttpResponse.ResponseInfo info) {
return HttpResponse.BodySubscribers.mapping(
HttpResponse.BodySubscribers.ofString(StandardCharsets.UTF_8),
s -> new Gson().fromJson(s, this.type));
}
}
var resp = client.send(request, JsonBodyHandler.jsonBodyHandler(MyJson.class));
MyJson body = resp.body();
java.net.http/query-param, form-encoded
• 用意されていない
– 便利なライブラリが他にあるのでそれを利用して
BodyにStringとして渡す
– (例) Spring関連のLibrary
– (例) Apache HttpClient 😨
• デフォルトでは無効化されている
java.net.http/cookie
var cookieHandler = new CookieManager();
var client = HttpClient.newBuilder().cookieHandler(cookieHandler).build();
var u = URI.create("https://httpbin.org/cookies/set/hello/world");
var request1 = HttpRequest.newBuilder(u).build();
client.send(request1, HttpResponse.BodyHandlers.ofString());
var cookie = new HttpCookie("sushi", "tenpura");
cookie.setDomain(u.getHost());
cookie.setPath("/");
cookie.setVersion(0);
cookieHandler.getCookieStore().add(u, cookie);
var request2 =
HttpRequest.newBuilder(URI.create("https://httpbin.org/cookies")).build();
var resp2 = client.send(request2, HttpResponse.BodyHandlers.ofString());
Version0が⼤事
Version0 -> Netscape style
Version1 -> RFC2965/2109
java.net.http/basic auth
• Basic認証
– HttpUrlConnectionのサンプルと同じ
– 利用クラスも同じなのでそちらを参照
var client = HttpClient.newBuilder()
.authenticator(authenticator)
.build();
java.net.http/download
• Download
var client = HttpClient.newBuilder()
.followRedirects(HttpClient.Redirect.ALWAYS)
.build();
var u = URI.create("https://github.com/AdoptOpenJDK/openjdk11-
binaries/releases/download/jdk-11.0.3%2B7/OpenJDK11U-
jdk_x64_linux_hotspot_11.0.3_7.tar.gz");
var request = HttpRequest.newBuilder().uri(u).build();
Path f = Paths.get("/tmp/").resolve(Path.of(u.getPath()).getFileName());
var resp = client.send(request, HttpResponse.BodyHandlers.ofFile(f));
System.out.println(resp.statusCode());
System.out.println(f.toFile().length());
java.net.http/proxy with authentication
System.setProperty("jdk.http.auth.tunneling.disabledSchemes", "");
var proxySelector = ProxySelector.of(
InetSocketAddress.createUnresolved(PROXY_HOST, PROXY_PORT));
var client = HttpClient.newBuilder()
.proxy(proxySelector)
.authenticator(new Authenticator() {
protected PasswordAuthentication getPasswordAuthentication() {
if (getRequestorType() == RequestorType.PROXY) {
return new PasswordAuthentication(PROXY_USER, PROXY_PASS);
}
return null;
}
}).build();
var u = URI.create("https://httpbin.org/get");
var request1 = HttpRequest.newBuilder(u).build();
var resp = client.send(request1, HttpResponse.BodyHandlers.ofString());
https://www.oracle.com/technetwork/java/javase/8u111-relnotes-3124969.html
jdk8u111より
これが無いとhttpsのurlの
認証処理がskipされる
Proxyの時だけ認証
java.net.http/badSSL
• SSLContextを設定できるので、ApacheHCと同じ
var sslContext = SSLContext.getInstance("TLS");
sslContext.init(null, new TrustManager[] { new X509TrustManager() {
public X509Certificate[] getAcceptedIssuers() { return null; }
public void checkClientTrusted(X509Certificate[] certs, String authType) { }
public void checkServerTrusted(X509Certificate[] certs, String authType) { }
} }, new SecureRandom());
var client = HttpClient.newBuilder().sslContext(sslContext).build();
var u = URI.create("https://expired.badssl.com/");
var request1 = HttpRequest.newBuilder(u).build();
var resp = client.send(request1, HttpResponse.BodyHandlers.ofString());
java.net.http/server-push
https://www.nginx.com/blog/nginx-1-13-9-http2-server-push/
java.net.http/server-push
var httpClient = HttpClient.newBuilder().version(HttpClient.Version.HTTP_2).build();
var pageRequest =
HttpRequest.newBuilder(URI.create("https://http2.golang.org/serverpush")).build();
AtomicInteger cnt = new AtomicInteger(1);
var futures = new CopyOnWriteArrayList<CompletableFuture<HttpResponse<String>>>();
HttpResponse.PushPromiseHandler<String> handler =
(initiatingRequest, pushPromiseRequest, acceptor) -> {
System.out.println("Promise request: " + pushPromiseRequest.uri());
var pushedFuture = acceptor.apply(HttpResponse.BodyHandlers.ofString());
pushedFuture = pushedFuture.thenApply(resp -> {
System.out.println("[" + cnt.getAndIncrement() + "] Pushed response: " + resp.uri());
return resp;
});
futures.add(pushedFuture);
}; applyPushPromise(
HttpRequest initiatingRequest,
HttpRequest pushPromiseRequest,
Function<HttpResponse.BodyHandler<String>,
CompletableFuture<HttpResponse<String>>> acceptor
)
java.net.http/server-push
var future = httpClient.sendAsync(
pageRequest,
HttpResponse.BodyHandlers.ofString(),
handler);
future.thenAccept(pageResponse -> {
System.out.println("Page response status code: " +
pageResponse.statusCode());
}).join();
var array = futures.toArray(new CompletableFuture[0]);
CompletableFuture.allOf(array).get();
Promise request: https://http2.golang.org/serverpush/static/jquery.min.js?1558076345232851844
Promise request: https://http2.golang.org/serverpush/static/godocs.js?1558076345232851844
Promise request: https://http2.golang.org/serverpush/static/playground.js?1558076345232851844
Promise request: https://http2.golang.org/serverpush/static/style.css?1558076345232851844
[1] Pushed response: https://http2.golang.org/serverpush/static/style.css?1558076345232851844
[2] Pushed response: https://http2.golang.org/serverpush/static/playground.js?1558076345232851844
[3] Pushed response: https://http2.golang.org/serverpush/static/godocs.js?1558076345232851844
Page response status code: 200
[4] Pushed response: https://http2.golang.org/serverpush/static/jquery.min.js?1558076345232851844
java.net.http/web-socket
var u = URI.create("wss://echo.websocket.org");
var client = HttpClient.newBuilder().build();
var listener = new WebSocket.Listener() {
public CompletionStage<?> onText(WebSocket webSocket,
CharSequence data, boolean last) {
System.out.println("receive:" + data + ", last=" + last);
return WebSocket.Listener.super.onText(webSocket, data, last);
}
};
WebSocket wsock = client.newWebSocketBuilder().buildAsync(u, listener).join();
wsock.sendText("hello world 1", false);
wsock.sendText("hello world 2", true);
TimeUnit.SECONDS.sleep(2);
wsock.sendClose(WebSocket.NORMAL_CLOSURE, "BYE");
java.net.http/debug
[class]
jdk.internal.net.http.common.Utils
jdk.internal.net.http.common.DebugLogger
jdk.internal.net.http.common.Log
jdk.internal.net.http.common.Logger
• システムプロパティの設定
jdk.internal.httpclient.debug=true
jdk.internal.httpclient.websocket.debug=true
jdk.internal.httpclient.hpack.debug=true
• もしくは
• もしくは
– LoggerName
java.net.http/debug
-Djava.net.HttpClient.log=
errors,requests,headers,frames[:control:data:window:all..],co
ntent,ssl,trace,channel
jdk.httpclient.HttpClient
java.net.http
• reactive streamの話をしていない?
java.net.http
• reactive streamの話をしていない?
– 今まで使っていたPublisher, SubScriberが
そうなのです。
public interface BodySubscriber<T> extends Flow.Subscriber<List<ByteBuffer>> {
CompletionStage<T> getBody();
void onSubscribe(Subscription subscription);
void onNext(List<ByteBuffer> item);
void onError(Throwable throwable);
void onComplete();
}
public interface BodyPublisher extends Flow.Publisher<ByteBuffer> {
long contentLength();
void subscribe(Subscriber<? super ByteBuffer> subscriber);
}
java.net.http
• もっといろいろな書き方が出来ます!!
– Javadocと公式サンプルを見て遊びましょう
続く(かも)
1 of 86

More Related Content

What's hot(20)

5分で分かるgitのrefspec5分で分かるgitのrefspec
5分で分かるgitのrefspec
ikdysfm15.8K views
Glibc malloc internalGlibc malloc internal
Glibc malloc internal
Motohiro KOSAKI62.1K views
Spring Boot × Vue.jsでSPAを作るSpring Boot × Vue.jsでSPAを作る
Spring Boot × Vue.jsでSPAを作る
Go Miyasaka14.9K views
今こそ知りたいSpring Batch(Spring Fest 2020講演資料)今こそ知りたいSpring Batch(Spring Fest 2020講演資料)
今こそ知りたいSpring Batch(Spring Fest 2020講演資料)
NTT DATA Technology & Innovation8.4K views
HTTP/2 入門HTTP/2 入門
HTTP/2 入門
Yahoo!デベロッパーネットワーク63.7K views
containerdの概要と最近の機能containerdの概要と最近の機能
containerdの概要と最近の機能
Kohei Tokunaga3K views
PostgreSQLバックアップの基本PostgreSQLバックアップの基本
PostgreSQLバックアップの基本
Uptime Technologies LLC (JP)24K views
What's new in Spring Batch 5What's new in Spring Batch 5
What's new in Spring Batch 5
ikeyat5.1K views
Google Cloud Dataflow を理解する - #bq_sushiGoogle Cloud Dataflow を理解する - #bq_sushi
Google Cloud Dataflow を理解する - #bq_sushi
Google Cloud Platform - Japan12.1K views
MetaspaceMetaspace
Metaspace
Yasumasa Suenaga24.9K views
こわくない Gitこわくない Git
こわくない Git
Kota Saito881.2K views
Dockerからcontainerdへの移行Dockerからcontainerdへの移行
Dockerからcontainerdへの移行
Kohei Tokunaga16.6K views

Similar to Introduction httpClient on Java11 / Java11時代のHTTPアクセス再入門

Presto anatomyPresto anatomy
Presto anatomyDongmin Yu
5K views50 slides
Solving anything in VCLSolving anything in VCL
Solving anything in VCLFastly
13.3K views62 slides

Similar to Introduction httpClient on Java11 / Java11時代のHTTPアクセス再入門(20)

More from tamtam180 (7)

Japanese font testJapanese font test
Japanese font test
tamtam180 590 views
jjugccc2018 app review postmortemjjugccc2018 app review postmortem
jjugccc2018 app review postmortem
tamtam180 16.1K views
PipelineDBとは?PipelineDBとは?
PipelineDBとは?
tamtam180 2.3K views
動画共有ツール動画共有ツール
動画共有ツール
tamtam180 6.9K views
Hive undocumented featureHive undocumented feature
Hive undocumented feature
tamtam180 7.9K views

Recently uploaded(20)

The Research Portal of Catalonia: Growing more (information) & more (services)The Research Portal of Catalonia: Growing more (information) & more (services)
The Research Portal of Catalonia: Growing more (information) & more (services)
CSUC - Consorci de Serveis Universitaris de Catalunya59 views
Liqid: Composable CXL PreviewLiqid: Composable CXL Preview
Liqid: Composable CXL Preview
CXL Forum120 views
Web Dev - 1 PPT.pdfWeb Dev - 1 PPT.pdf
Web Dev - 1 PPT.pdf
gdsczhcet49 views
Green Leaf Consulting: Capabilities DeckGreen Leaf Consulting: Capabilities Deck
Green Leaf Consulting: Capabilities Deck
GreenLeafConsulting177 views
CXL at OCPCXL at OCP
CXL at OCP
CXL Forum203 views

Introduction httpClient on Java11 / Java11時代のHTTPアクセス再入門

  • 2. アジェンダ • 自己紹介 • おさらい – HttpUrlConnectionの話 – Apache HttpClientの話 • java.net.http.HttpClientの話
  • 4. 自己紹介 • Name: Kiyotaka Suzuki • Twitter: @tamtam180 • Main works – Square Enix (5.5 Year) • PlayOnline, FF-XIV, etc, – SmartNews (5 Year) • Cross functional Expert Team • Senior Software Engineer • Advertising System, PoC, SRE, etc,.. • I Love OSS about Datastore • Mad performance tuner
  • 5. その前に • 検証に便利なWebサイト – https://http2.pro/api/v1 • HTTP/2のテスト – https://httpbin.org/ • HTTPメソッド、ステータス、Cookie、Cache、etc • このサイトでほぼ事足りる – https://httpstat.us/ • HTTPステータス – https://badssl.com/ • HTTPSのエラーテスト – https://www.websocket.org/echo.html • WebSocketのテスト
  • 7. おさらい • HttpUrlConnection – 同期処理だけ – HTTP1.1まで – Cookieは一応扱える • OnMemory – Basic認証も出来る • QueryStringを作るのが大変(個人的感想) • Responseのgzip処理とか大変
  • 8. おさらい • Apache HTTPClient – 5系からHTTP/2が使える(5系はβバージョン) – 3系から4系でプログラムの構造が大きく変更 • Commons HttpClientは3系, もう古い – IO Model • Blocking I/O – Classic Java IO – Non blocking • Event Driven I/O with JavaNIO
  • 9. おさらい • Apache HTTPClient – Core • v4.4 • v5.0 beta – Client • v4.5 • v5.0 beta – HttpAsyncClient • v4.1
  • 10. おさらい • OkHttp (v3.14.1) – HTTP/2対応 – Androidも対応 • Android5.0+ (API Level 21+) – Java8+ – TLS1.3対応
  • 11. おさらい • java.net.http / HttpClient – HTTP/2対応 – WebSocket対応 – 同期/非同期 処理対応 – Reactive Streamとして Request/Responseを処理 – Builderパターン
  • 13. おさらい: HttpUrlConnection • GET var url = new URL("https://httpbin.org/get"); var conn = (HttpURLConnection) url.openConnection(); try (InputStream in = conn.getInputStream()) { System.out.println(conn.getResponseCode()); System.out.println(IOUtils.toString(in, StandardCharsets.UTF_8)); } conn.disconnect(); "headers": { "Accept": "text/html, image/gif, image/jpeg, *; q=.2, */*; q=.2", "Host": "httpbin.org", "User-Agent": "Java/11.0.2" }
  • 15. おさらい: HttpUrlConnection • Request Headerを弄る conn.setRequestProperty("User-Agent", "MyJava/11.0.0"); conn.setRequestProperty("Content-Type", "application/json");
  • 16. おさらい: HttpUrlConnection • BODYを送信する conn.setDoOutput(true); try (var writer = new OutputStreamWriter(conn.getOutputStream())) { writer.write(new Gson().toJson(Map.of("hello", "world"))); writer.flush(); }
  • 17. おさらい: HttpUrlConnection • POST(組み合わせ) var url = new URL("http://httpbin.org/post"); HttpURLConnection conn = (HttpURLConnection) url.openConnection(); conn.setDoOutput(true); conn.setRequestMethod("POST"); conn.setRequestProperty("User-Agent", "MyJava/11.0.0"); conn.setRequestProperty("Content-Type", "application/json"); try (OutputStreamWriter writer = new OutputStreamWriter(conn.getOutputStream())) { writer.write(new Gson().toJson(Map.of("hello", "world"))); writer.flush(); } try (InputStream in = conn.getInputStream()) { System.out.println(conn.getResponseCode()); System.out.println(IOUtils.toString(in, StandardCharsets.UTF_8)); } conn.disconnect();
  • 18. おさらい: HttpUrlConnection • Cookie – CookieManager, CookieHandlerを使う – 世の中にあるサンプル、本当に動く?? – CookieManagerはThreadSafeじゃない CookieManager cookieManager = new CookieManager(null, CookiePolicy.ACCEPT_ALL); CookieHandler.setDefault(cookieManager); System.out.println(conn.getResponseCode()); CookieHandler.getDefault().put(conn.getURL().toURI(), conn.getHeaderFields()); for (HttpCookie cookie : cookieManager.getCookieStore().get(conn.getURL().toURI())) { System.out.println(cookie); } これが無いと動かないのだけど..
  • 19. おさらい: HttpUrlConnection • Proxy var proxy = new Proxy( Proxy.Type.SOCKS, InetSocketAddress.createUnresolved("127.0.0.1", 7777)); HttpURLConnection conn = (HttpURLConnection) url.openConnection(proxy); ssh any-server –D7777 HTTP Proxyの場合は、 Proxy.Type.HTTP
  • 20. おさらい: HttpUrlConnection • Redirect – デフォルトはredirectを追従 var url = new URL("https://httpbin.org/redirect/2"); var conn = (HttpURLConnection) url.openConnection(); conn.setInstanceFollowRedirects(true);
  • 21. おさらい: HttpUrlConnection • Basic認証 – Header直接指定 – java.net.Authenticator • requestPasswordAuthenticationInstanceを 上書き • getPasswordAuthenticationを上書き
  • 22. おさらい: HttpUrlConnection • Basic認証 – Header直接指定 var url = new URL("https://httpbin.org/basic-auth/test-user/test-pass"); var conn = (HttpURLConnection) url.openConnection(); conn.setRequestProperty( "Authorization", "Basic " + Base64.getEncoder().encodeToString("test-user:test- pass".getBytes()));
  • 23. おさらい: HttpUrlConnection • Basic認証 – java.net.Authenticator その1 Authenticator.setDefault(new Authenticator() { @Override protected PasswordAuthentication getPasswordAuthentication() { if (getRequestorType() != RequestorType.SERVER) return null; if (!"basic".equals(getRequestingScheme())) return null; if ("httpbin.org".equalsIgnoreCase(getRequestingHost())) { return new PasswordAuthentication("test-user", "test-pass".toCharArray()); } return null; } });
  • 24. おさらい: HttpUrlConnection • Basic認証 – java.net.Authenticator その2 Authenticator.setDefault(new Authenticator() { @Override public PasswordAuthentication requestPasswordAuthenticationInstance(String host, InetAddress addr, int port, String protocol, String prompt, String scheme, URL url, RequestorType reqType) { if (reqType != RequestorType.SERVER) return null; if (!"basic".equals(scheme)) return null; if ("httpbin.org".equalsIgnoreCase(host)) { return new PasswordAuthentication("test-user", "test-pass".toCharArray()); } return null; } });
  • 25. おさらい: HttpUrlConnection • SSL Errorを無視する通信 SSLContext sslcontext = SSLContext.getInstance("SSL"); sslcontext.init(null, new TrustManager[]{new X509TrustManager() { public void checkClientTrusted(X509Certificate[] x509Certificates, String s) {} public void checkServerTrusted(X509Certificate[] x509Certificates, String s) {} public X509Certificate[] getAcceptedIssuers() { return null; } }}, null); HttpsURLConnection.setDefaultHostnameVerifier((s, sslSession) -> true); var url = new URL("https://expired.badssl.com/"); HttpURLConnection conn = (HttpURLConnection) url.openConnection(); if (conn instanceof HttpsURLConnection) { ((HttpsURLConnection) conn).setSSLSocketFactory(sslcontext.getSocketFactory()); } SSLContext Verifier SocketFactory
  • 27. おさらい: HttpUrlConnection • Logger指定 [logging.properties] handlers = java.util.logging.ConsoleHandler java.util.logging.ConsoleHandler.level = ALL sun.net.www.protocol.http.HttpURLConnection.level = ALL System.setProperty("java.util.logging.config.file", "logging.properties");
  • 28. おさらい: HttpUrlConnection • javax.net.debug – Helpを指定するとhelpが表示される System.setProperty("javax.net.debug", ”help"); URL url = new URL("https://www.google.com"); var conn = (HttpURLConnection) url.openConnection(); conn.getInputStream(); https://docs.oracle.com/javase/jp/8/docs/technotes/guides/security/jsse/JSSERefGuide.html https://www.ibm.com/support/knowledgecenter/en/SSYKE2_7.0.0/com.ibm.java.security.component.70.doc/security- all ssl:record,handshake,keygen,session,defaultctx,sslctx,sessioncache,keymanager,trustmanager,plugga bility handshake:data,verbose plaintext,packet 指定可能な値 例: ssl:handshake,session 例: all カンマ区切りかコロン区切り
  • 30. ApacheHC/GET • GET var clientBuilder = HttpClientBuilder.create(); try (CloseableHttpClient client = clientBuilder.build()) { var getMethod = new HttpGet(URI.create("http://httpbin.org/get")); try (CloseableHttpResponse resp = client.execute(getMethod)) { String body = EntityUtils.toString(resp.getEntity(), StandardCharsets.UTF_8); System.out.println(resp.getStatusLine()); System.out.println(body); } }
  • 31. ApacheHC/Accept Header • 注意点 – デフォルトのヘッダはAccept Header無し Host: httpbin.org Connection: Keep-Alive User-Agent: Apache-HttpClient/4.5.4 (Java/11.0.1) Accept-Encoding: gzip,deflate
  • 32. ApacheHC/Accept Header • Accept Header • (指定が無い場合は何でも受け付ける事を意 味する) • と、書いてあるのに.. RFC7231: Section 5.3.2 A request without any Accept header field implies that the user agent will accept any media type in response. https://tools.ietf.org/html/rfc7231#section-5.3.2
  • 33. ApacheHC/Accept Header • サイトによっては正しく返却されない – 例: http://httpstat.us/200 curl -H "Accept: */*" http://httpstat.us/200 # Content-Length: 6 200 OK curl -H "Accept:" http://httpstat.us/200 # Content-Length: 0
  • 34. ApacheHC/Accept Header • 各ブラウザのAccept Header – デフォルト値の一覧は以下のサイトを参照 https://developer.mozilla.org/en- US/docs/Web/HTTP/Content_negotiation/List_of_default_Accept_values
  • 35. ApacheHC/Gzip • Gzipはデフォルトで処理される – gzipもdeflateも curl -s http://httpbin.org/gzip | gzip -d var clientBuilder = HttpClientBuilder.create(); try (CloseableHttpClient client = clientBuilder.build()) { var getMethod = new HttpGet("http://httpbin.org/gzip"); try (var resp = client.execute(getMethod)) { String body = EntityUtils.toString(resp.getEntity(), StandardCharsets.UTF_8); System.out.println(body); } } 本当はAccept Encodingを付ける必要がある。 httpbin.org/gzip や httpbin.org/deflate は それを無視して送ってくるのでコードでは省略している。
  • 36. ApacheHC • Queryを作る時は URI Builder var uri = new URIBuilder("http://httpbin.org/get") .setCharset(StandardCharsets.UTF_8) .addParameter("hello", "world") .addParameter("sushi", "寿司") .setScheme("https") // HTTPSに変更 .setFragment("hello") .build(); https://httpbin.org/get?hello=world&sushi=%E5%AF%BF%E5%8F%B8#hello
  • 37. ApacheHC • Connection Pool var connManager = new PoolingHttpClientConnectionManager(); connManager.setMaxTotal(100); connManager.setDefaultMaxPerRoute(100); var clientBuilder = HttpClientBuilder.create(); clientBuilder.setConnectionManager(connManager);
  • 38. ApacheHC • Request Headerをいじる var clientBuilder = HttpClientBuilder.create(); var defaultHeaders = Arrays.asList( new BasicHeader("Accept", "*/*") ); clientBuilder.setDefaultHeaders(defaultHeaders); var getMethod = new HttpGet(URI.create("http://httpbin.org/get")); getMethod.setHeader("Accept", "*/*"); 定数クラスがあるのでそれを使うと良い HttpHeaders, ContentType
  • 39. ApacheHC/POST var clientBuilder = HttpClientBuilder.create(); try (CloseableHttpClient client = clientBuilder.build()) { String jsonText = new Gson().toJson(Map.of("hello", "world")); HttpEntity entity = EntityBuilder.create() .setContentType(ContentType.APPLICATION_JSON) .setText(jsonText).build(); HttpPost postMethod = new HttpPost(URI.create("http://httpbin.org/post")); postMethod.setEntity(entity); try (CloseableHttpResponse resp = client.execute(postMethod)) { String body = EntityUtils.toString(resp.getEntity(), StandardCharsets.UTF_8); System.out.println(resp.getStatusLine()); System.out.println(body); } }
  • 40. ApacheHC • やり方色々 – Client • HttpClientBuilder.create().build(); • HttpClients.custom().build(); • HttpClients.createDefault(); – Entity • EntityBuilder • 直接生成 BasicHttpEntity BufferedHttpEntity ByteArrayEntity FileEntity InputStreamEntity SerializableEntity StringEntity
  • 41. ApacheHC • やり方色々 – execute • 単純に実行 • HttpContext • ResponseHandler
  • 42. ApacheHC • Responseの処理はEntityUtilsが便利 – toString() – toByteArray() – writeTo() – consume() – ...
  • 43. ApacheHC/Redirect • デフォルトで追従 • OFFにする場合は、RequestConfig var requestConfig = RequestConfig.custom() .setRedirectsEnabled(false) //追従しない .build(); var clientBuilder = HttpClientBuilder.create(); clientBuilder.setDefaultRequestConfig(requestConfig); var getMethod = new HttpGet(URI.create("http://httpbin.org/redirect/3")); getMethod.setConfig(requestConfig);
  • 44. ApacheHC/Retry • DefaultHttpRequestRetryHandler – ConnectTimeoutExceptionはRetryしない – InterruptedIOExceptionのSubClassだけど https://hc.apache.org/httpcomponents-client- ga/httpclient/apidocs/org/apache/http/impl/client/DefaultHttpRequestRetryHandler.html
  • 45. ApacheHC/Cookie var cookieStore = new BasicCookieStore(); var clientBuilder = HttpClientBuilder.create(); clientBuilder.setDefaultCookieStore(cookieStore); // return Set-Cookie try (CloseableHttpClient client = clientBuilder.build()) { var get = new HttpGet("https://httpbin.org/cookies/set/hello/world"); try (CloseableHttpResponse resp = client.execute(get)) {} } // send Cookie: hello and sushi var cookie = new BasicClientCookie("sushi", " 🍣"); cookie.setDomain("httpbin.org"); cookieStore.addCookie(cookie); try (CloseableHttpClient client = clientBuilder.build()) { var get = new HttpGet("https://httpbin.org/cookies"); try (CloseableHttpResponse resp = client.execute(get)) {} } GET /cookies HTTP/1.1 Host: httpbin.org Connection: Keep-Alive User-Agent: Apache-HttpClient/4.5.4 (Java/11.0.1 Cookie: hello=world; sushi= 🍣 Accept-Encoding: gzip,deflate
  • 46. • Proxy • 認証有りProxy ApacheHC/Proxy var clientBuilder = HttpClientBuilder.create(); var proxy = new HttpHost(proxyHost, proxyPort, "http"); clientBuilder.setProxy(proxy); clientBuilder.setProxyAuthenticationStrategy(new ProxyAuthenticationStrategy()); var credsProvider = new BasicCredentialsProvider(); credsProvider.setCredentials( AuthScope.ANY, new UsernamePasswordCredentials(proxyUser, proxyPassword)); clientBuilder.setDefaultCredentialsProvider(credsProvider);
  • 47. ApacheHC • Basic認証 var u = URI.create("https://httpbin.org/basic-auth/kuma/pass"); var credProvider = new BasicCredentialsProvider(); credProvider.setCredentials( new AuthScope(u.getHost(), u.getPort(), AuthScope.ANY_REALM, AuthScope.ANY_SCHEME), new UsernamePasswordCredentials("kuma", "pass")); var clientBuilder = HttpClientBuilder.create(); clientBuilder.setDefaultCredentialsProvider(credProvider); try (CloseableHttpClient client = clientBuilder.build()) { var getMethod = new HttpGet(u); try (CloseableHttpResponse resp = client.execute(getMethod)) { System.out.println(resp.getStatusLine()); } }
  • 48. ApacheHC • Basic認証 (ANYを使う:非推奨) – 最低限、Host, Portは指定した方が良い – Realmも本当は指定した方が良いけど credProvider.setCredentials( AuthScope.ANY, new UsernamePasswordCredentials("kuma", "pass"));
  • 49. ApacheHC/BadSSL • BadSSLスルー var sslContext = SSLContext.getInstance("SSL"); sslContext.init(null, new TrustManager[] { new X509TrustManager() { public X509Certificate[] getAcceptedIssuers() { return null; } public void checkClientTrusted(X509Certificate[] certs, String authType) { } public void checkServerTrusted(X509Certificate[] certs, String authType) { } } }, new SecureRandom()); var clientBuilder = HttpClientBuilder.create(); clientBuilder.setSSLSocketFactory(new SSLConnectionSocketFactory(sslContext)); try (CloseableHttpClient client = clientBuilder.build()) { var getMethod = new HttpGet("https://expired.badssl.com/"); try (var resp = client.execute(getMethod)) { String body = EntityUtils.toString(resp.getEntity(), StandardCharsets.UTF_8); System.out.println(body); } }
  • 50. ApacheHC/Async • HttpAsyncClient – Basicな方法 CloseableHttpAsyncClient client = HttpAsyncClients.createDefault(); client.start(); var request1 = new HttpGet("https://httpbin.org/get"); Future<HttpResponse> future = client.execute(request1, null); // any processing var resp = future.get(); System.out.println(resp.getStatusLine()); client.close();
  • 51. ApacheHC/Async • HttpAsyncClient – Consumer / Producer CloseableHttpAsyncClient client = HttpAsyncClients.createDefault(); client.start(); var producer = HttpAsyncMethods.create(new HttpGet("https://httpbin.org/get")); var consumer = new AsyncCharConsumer<HttpResponse>() { HttpResponse response; protected void onCharReceived(CharBuffer charBuffer, IOControl ioControl) { } protected void onResponseReceived(HttpResponse httpResponse) { this.response = httpResponse; } protected HttpResponse buildResult(HttpContext httpContext) { return this.response; } };
  • 52. ApacheHC/Async • HttpAsyncClient – Consumer / Producer var latch = new CountDownLatch(1); client.execute(producer, consumer, new FutureCallback<>() { public void completed(HttpResponse httpResponse) { latch.countDown(); } public void failed(Exception e) { latch.countDown(); } public void cancelled() { latch.countDown(); } }); latch.await(); System.out.println(consumer.response.getStatusLine()); client.close();
  • 53. ApacheHC/Debug • DEBUG – Loggerを設定する – Lobackの例 <logger name="org.apache.http" additivity="true"> <level value="DEBUG"/> </logger>
  • 54. ApacheHC/不満点 • HTTP/2は まだ 対応していない • WebSocket対応もまだ
  • 56. おさらい • java.net.http / HttpClient – Java11で正式APIになった – HTTP/2対応 – WebSocket対応 – 同期/非同期 処理対応 – Reactive Streamとして Request/Responseを処理 – Builderパターン
  • 57. java.net.http • Clientの作り方 もしくは [Header] GET /get HTTP/1.1 Content-Length: 0 Host: httpbin.org User-Agent: Java-http-client/11.0.1 var client = HttpClient.newHttpClient(); var request = HttpRequest.newBuilder() .uri(URI.create("https://http2.pro/api/v1")) .build(); var resp = client.send(request, HttpResponse.BodyHandlers.ofString(StandardCharsets.UTF_8)); System.out.println(resp.statusCode()); System.out.println(resp.body()); var client = HttpClient.newBuilder().build(); [Body] {"http2":1,"protocol":"HT TP¥/2.0","push":1,"user_a gent":"Java-http- client¥/11.0.1"}
  • 58. java.net.http • デフォルトは、 – HTTP/2を勝手にやってくれる • サイトが対応していない場合はHTTP/1.1で通信 – Redirectは勝手にしてくれない – GZip/Deflateは勝手に処理してくれない • 便利なURIBuilderは存在しない
  • 59. java.net.http • ClientBuilder – authenticator – connectTimeout – cookieHandler – executor – followRedirects – priority – proxy – sslContext – version – sslParameters
  • 60. java.net.http • BodyHandlers – ofString – ofByteArray – ofByteArrayConsumer – ofFile – ofFileDownload – ofInputStream – ofLines – ofPublisher – ...
  • 61. java.net.http/redirect • Redirectの設定 var client = HttpClient.newBuilder() .followRedirects(HttpClient.Redirect.NORMAL) .build(); NEVER Redirectしない ALWAYS 常にRedirectする NORMAL 常にRedirectする(HTTPS->HTTPは除く)
  • 62. java.net.http/Header • Request Headerの編集 var request = HttpRequest.newBuilder() .header("Accept", "*/*") .header("User-Agent", "JJUG/1.0") .uri(URI.create("https://httpbin.org/get")) .build();
  • 63. java.net.http/POST • POSTとBody送信 var bodyString = new Gson().toJson(Map.of("hello", "world")); var client = HttpClient.newBuilder().build(); var request = HttpRequest.newBuilder() .header("Content-Type", "application/json") .POST(HttpRequest.BodyPublishers.ofString(bodyString)) .uri(URI.create("https://httpbin.org/post")) .build(); var resp = client.send( request, HttpResponse.BodyHandlers.ofString(StandardCharsets.UTF_8)); System.out.println(resp.statusCode()); System.out.println(resp.body());
  • 64. java.net.http • BodyPublishers – noBody – ofString – ofByteArray – ofByteArrays – ofFile – ofInputStream – ...
  • 65. java.net.http/method • 他のメソッド – GET(), POST(), DELETE(), PUT() – HEADやPATCHは? • methodを使う .method("HEAD", HttpRequest.BodyPublishers.noBody()) .method("PATCH", HttpRequest.BodyPublishers.ofString(bodyString))
  • 66. java.net.http/async • 非同期: client.sendAsync var client = HttpClient.newBuilder().build(); var request = HttpRequest.newBuilder(URI.create("https://httpbin.org/get")).build(); CompletableFuture<HttpResponse<String>> future = client.sendAsync(request, HttpResponse.BodyHandlers.ofString(StandardCharsets.UTF_8)); // CompletableFutureなので処理を繋げる事ができる future.thenApply(HttpResponse::body) .exceptionally(e -> "Error!! :" + e.getMessage()) .thenAccept(System.out::println); //future.get();
  • 67. java.net.http/content-encoding • BodySubscribers.mappingを使う – ※JDKの不具合で動きません https://bugs.openjdk.java.net/browse/JDK-8217264 private HttpResponse.BodySubscriber<InputStream> gzippedBodySubscriber( HttpResponse.ResponseInfo responseInfo) { // 本当はheaderのContent-Encodingを確認する return HttpResponse.BodySubscribers.mapping( HttpResponse.BodySubscribers.ofInputStream(), this::decodeGzipStream); } private InputStream decodeGzipStream(InputStream gzippedStream) { try { return new GZIPInputStream(gzippedStream); } catch (IOException ex) { throw new UncheckedIOException(ex); } } var resp = client.send(request, this::gzippedBodySubscriber);
  • 68. java.net.http/content-encoding • Workaround-1 var client = HttpClient.newBuilder().build(); var request = HttpRequest.newBuilder() .uri(URI.create("https://httpbin.org/gzip")) .build(); var resp = client.send(request, HttpResponse.BodyHandlers.ofInputStream()); InputStream in = resp.body(); if ("gzip".equals(resp.headers().firstValue("Content-Encoding").get())) { in = new GZIPInputStream(resp.body()); } System.out.println(IOUtils.toString(in, StandardCharsets.UTF_8));
  • 69. java.net.http/content-encoding • Workaround-2 var future = client.sendAsync(request, HttpResponse.BodyHandlers.ofInputStream()) .thenApply(resp -> { String ce = resp.headers().firstValue("Content-Encoding").get(); if ("gzip".equals(ce)) { try { return new GZIPInputStream(resp.body()); } catch (IOException e) { throw new UncheckedIOException(e); } } return resp.body(); }); var in = future.get(); System.out.println(IOUtils.toString(in, StandardCharsets.UTF_8));
  • 71. java.net.http/custom handler • 例: json public static class JsonBodyHandler<T> implements HttpResponse.BodyHandler<T> { private Class<T> type; private JsonBodyHandler(Class<T> type) { this.type = type; } public static <T> JsonBodyHandler<T> jsonBodyHandler(Class<T> type) { return new JsonBodyHandler<>(type); } @Override public HttpResponse.BodySubscriber<T> apply( HttpResponse.ResponseInfo info) { return HttpResponse.BodySubscribers.mapping( HttpResponse.BodySubscribers.ofString(StandardCharsets.UTF_8), s -> new Gson().fromJson(s, this.type)); } } var resp = client.send(request, JsonBodyHandler.jsonBodyHandler(MyJson.class)); MyJson body = resp.body();
  • 72. java.net.http/query-param, form-encoded • 用意されていない – 便利なライブラリが他にあるのでそれを利用して BodyにStringとして渡す – (例) Spring関連のLibrary – (例) Apache HttpClient 😨
  • 73. • デフォルトでは無効化されている java.net.http/cookie var cookieHandler = new CookieManager(); var client = HttpClient.newBuilder().cookieHandler(cookieHandler).build(); var u = URI.create("https://httpbin.org/cookies/set/hello/world"); var request1 = HttpRequest.newBuilder(u).build(); client.send(request1, HttpResponse.BodyHandlers.ofString()); var cookie = new HttpCookie("sushi", "tenpura"); cookie.setDomain(u.getHost()); cookie.setPath("/"); cookie.setVersion(0); cookieHandler.getCookieStore().add(u, cookie); var request2 = HttpRequest.newBuilder(URI.create("https://httpbin.org/cookies")).build(); var resp2 = client.send(request2, HttpResponse.BodyHandlers.ofString()); Version0が⼤事 Version0 -> Netscape style Version1 -> RFC2965/2109
  • 74. java.net.http/basic auth • Basic認証 – HttpUrlConnectionのサンプルと同じ – 利用クラスも同じなのでそちらを参照 var client = HttpClient.newBuilder() .authenticator(authenticator) .build();
  • 75. java.net.http/download • Download var client = HttpClient.newBuilder() .followRedirects(HttpClient.Redirect.ALWAYS) .build(); var u = URI.create("https://github.com/AdoptOpenJDK/openjdk11- binaries/releases/download/jdk-11.0.3%2B7/OpenJDK11U- jdk_x64_linux_hotspot_11.0.3_7.tar.gz"); var request = HttpRequest.newBuilder().uri(u).build(); Path f = Paths.get("/tmp/").resolve(Path.of(u.getPath()).getFileName()); var resp = client.send(request, HttpResponse.BodyHandlers.ofFile(f)); System.out.println(resp.statusCode()); System.out.println(f.toFile().length());
  • 76. java.net.http/proxy with authentication System.setProperty("jdk.http.auth.tunneling.disabledSchemes", ""); var proxySelector = ProxySelector.of( InetSocketAddress.createUnresolved(PROXY_HOST, PROXY_PORT)); var client = HttpClient.newBuilder() .proxy(proxySelector) .authenticator(new Authenticator() { protected PasswordAuthentication getPasswordAuthentication() { if (getRequestorType() == RequestorType.PROXY) { return new PasswordAuthentication(PROXY_USER, PROXY_PASS); } return null; } }).build(); var u = URI.create("https://httpbin.org/get"); var request1 = HttpRequest.newBuilder(u).build(); var resp = client.send(request1, HttpResponse.BodyHandlers.ofString()); https://www.oracle.com/technetwork/java/javase/8u111-relnotes-3124969.html jdk8u111より これが無いとhttpsのurlの 認証処理がskipされる Proxyの時だけ認証
  • 77. java.net.http/badSSL • SSLContextを設定できるので、ApacheHCと同じ var sslContext = SSLContext.getInstance("TLS"); sslContext.init(null, new TrustManager[] { new X509TrustManager() { public X509Certificate[] getAcceptedIssuers() { return null; } public void checkClientTrusted(X509Certificate[] certs, String authType) { } public void checkServerTrusted(X509Certificate[] certs, String authType) { } } }, new SecureRandom()); var client = HttpClient.newBuilder().sslContext(sslContext).build(); var u = URI.create("https://expired.badssl.com/"); var request1 = HttpRequest.newBuilder(u).build(); var resp = client.send(request1, HttpResponse.BodyHandlers.ofString());
  • 79. java.net.http/server-push var httpClient = HttpClient.newBuilder().version(HttpClient.Version.HTTP_2).build(); var pageRequest = HttpRequest.newBuilder(URI.create("https://http2.golang.org/serverpush")).build(); AtomicInteger cnt = new AtomicInteger(1); var futures = new CopyOnWriteArrayList<CompletableFuture<HttpResponse<String>>>(); HttpResponse.PushPromiseHandler<String> handler = (initiatingRequest, pushPromiseRequest, acceptor) -> { System.out.println("Promise request: " + pushPromiseRequest.uri()); var pushedFuture = acceptor.apply(HttpResponse.BodyHandlers.ofString()); pushedFuture = pushedFuture.thenApply(resp -> { System.out.println("[" + cnt.getAndIncrement() + "] Pushed response: " + resp.uri()); return resp; }); futures.add(pushedFuture); }; applyPushPromise( HttpRequest initiatingRequest, HttpRequest pushPromiseRequest, Function<HttpResponse.BodyHandler<String>, CompletableFuture<HttpResponse<String>>> acceptor )
  • 80. java.net.http/server-push var future = httpClient.sendAsync( pageRequest, HttpResponse.BodyHandlers.ofString(), handler); future.thenAccept(pageResponse -> { System.out.println("Page response status code: " + pageResponse.statusCode()); }).join(); var array = futures.toArray(new CompletableFuture[0]); CompletableFuture.allOf(array).get(); Promise request: https://http2.golang.org/serverpush/static/jquery.min.js?1558076345232851844 Promise request: https://http2.golang.org/serverpush/static/godocs.js?1558076345232851844 Promise request: https://http2.golang.org/serverpush/static/playground.js?1558076345232851844 Promise request: https://http2.golang.org/serverpush/static/style.css?1558076345232851844 [1] Pushed response: https://http2.golang.org/serverpush/static/style.css?1558076345232851844 [2] Pushed response: https://http2.golang.org/serverpush/static/playground.js?1558076345232851844 [3] Pushed response: https://http2.golang.org/serverpush/static/godocs.js?1558076345232851844 Page response status code: 200 [4] Pushed response: https://http2.golang.org/serverpush/static/jquery.min.js?1558076345232851844
  • 81. java.net.http/web-socket var u = URI.create("wss://echo.websocket.org"); var client = HttpClient.newBuilder().build(); var listener = new WebSocket.Listener() { public CompletionStage<?> onText(WebSocket webSocket, CharSequence data, boolean last) { System.out.println("receive:" + data + ", last=" + last); return WebSocket.Listener.super.onText(webSocket, data, last); } }; WebSocket wsock = client.newWebSocketBuilder().buildAsync(u, listener).join(); wsock.sendText("hello world 1", false); wsock.sendText("hello world 2", true); TimeUnit.SECONDS.sleep(2); wsock.sendClose(WebSocket.NORMAL_CLOSURE, "BYE");
  • 83. • もしくは • もしくは – LoggerName java.net.http/debug -Djava.net.HttpClient.log= errors,requests,headers,frames[:control:data:window:all..],co ntent,ssl,trace,channel jdk.httpclient.HttpClient
  • 85. java.net.http • reactive streamの話をしていない? – 今まで使っていたPublisher, SubScriberが そうなのです。 public interface BodySubscriber<T> extends Flow.Subscriber<List<ByteBuffer>> { CompletionStage<T> getBody(); void onSubscribe(Subscription subscription); void onNext(List<ByteBuffer> item); void onError(Throwable throwable); void onComplete(); } public interface BodyPublisher extends Flow.Publisher<ByteBuffer> { long contentLength(); void subscribe(Subscriber<? super ByteBuffer> subscriber); }