SlideShare a Scribd company logo
1 of 33
Download to read offline
Advanced nginx in Mercari
「nginx実践入門」出版記念!執筆者らが語る nginx Tech Talks
Masahiro Nagano
How to handle over 1,200,000 HTTPS Reqs/Min
Me
• Masahiro Nagano @kazeburo
• Mercari, inc.
• Principal Engineer, Site Reliability Engineering Team
• BASE, inc. Technical Advisor
Mercari
Your Friendly Mobile MarketPlace
JP US
Mercari KPI
DOWNLOAD
GMV
ITEMS
3100万DL(JP+US)
月間数十億円
1日数十万品以上
API Traffic
1,200,000 reqs/min
(HTTPS only)
nginx in mercari
©2011 Amazon Web Services LLC or its affiliates. All rights reserved.
Users Client Multimedia Corporate
data center
Trad
se
Mobile Client
AWS Management
Console
IAM Add-on Example:
IAM Add-on
Tasks (HIT) TaskTurk
ice Specific
©2011 Amazon Web Services LLC or its affiliates. All rights reserved.
User
Users
Client
Multimedia
Corporatedata center
Mobile Client
net
AWS Management
Console
IAM Add-on
Example:IAM Add-on
Requester
azon
W
eb Services LLC
or its affiliates. All rights reserved.
ser
Users
Client
Multim
edia
Corporate
data
center
Tradit
server
Mobile
Client
AW
S
Managem
ent
Console
IAM
Add-on
Exam
ple:
IAM
Add-on
ence
HIT)
Assignm
ent/
Task
W
orkers
L7 load balancer
Reverse Proxy TLS Terminator
Contents Distribution©2011
Am
azon
W
eb
Services LLC
or its affiliates. All rights reserve
Users
Client
M
ultim
edia
Corporate
data
center
Tradition
server
M
obile
Client
Internet
AW
S
M
anagem
ent
Console
IAM
Add-on
Exam
ple:
IAM
Add-on
m
ent/
Task
Requester
W
orkers
nginx Extension in mercari
• ngx_dynamic_upstream
• Operate upsteam servers Up/Down dynamically
• OpenResty (ngx_lua)
• More powerful L7 load balancer
• Application log collector
Agenda
• Continuous updating nginx
• TLS Optimization
• Automated TLS session ticket key updating
• SPDY to HTTP/2
Continuous updating nginx
Keep nginx the latest
• Use nginx mainline version
• stableは安定しているという意味ではなく新機能が追加さ
れず、バグ修正も緊急な修正を除いて行われないというこ
と。すべての新機能やバグ修正はmainlineにて行われるの
で結果的にmainlineが安定している

https://www.nginx.com/blog/nginx-1-6-1-7-released/
• nginxは無停止でのアップデートが可能
• Periodic OpenSSL vulnerability report 😓
nginx-build
• nginx builder command written in Go

https://github.com/cubicdaiya/nginx-build
• Automate download, configure and make
• Download OpenSSL, zlib and PCRE. build them
statically ( independent from OS’s library version)
• Support Tengine and OpenResty
nginx-build
$ EXPORT $VERSION=1.9.10
$ nginx-build 
-clear 
-d work 
-v $VERSION 
-c nginx-build-conf/configure 
-m nginx-build-conf/modules3rd.ini 
-zlib 
-zlibversion=1.2.8 
-verbose 
-pcre 
-pcreversion=8.38 
-openssl 
-opensslversion=1.0.2f 
$ cd work/$VERSION/nginx-$VERSION
$ sudo make install
TLS Optimization
TLS Optimization
• PFS - Perfect Forward Secrecy
• TTFB - Time To First Byte
Perfect Forward Secrecy
• Mozilla Wiki

https://wiki.mozilla.org/Security/Server_Side_TLS
• Mozilla SSL Configuration Generator

https://mozilla.github.io/server-side-tls/ssl-config-
generator/
• *最新の情報を参照し、プロダクション環境に投
入する前にテストを行ってください
TTFB
• TLS Session Cache
• TLS Session Tickets
• OCSP Stapling
• SSL buffer size
TLS Session Cache
• Server side session cache
• Client software does not matter
• nginx could’t share session cache between servers.
ssl_session_cache shared:SSL:120m;
ssl_session_timeout 5m;
CONS
PROS
TLS Session Cache
• OpenResty will support it?

https://twitter.com/agentzh/status/
686655229828403201
TLS Session Tickets
• Client Side Session Cache

(like a CookieStore)
• No need to share cache between servers
• It’s required share ssl_session_ticket_key file
• Client support required
ssl_session_tickets on;
ssl_session_ticket_key /path/to/ssl_session_ticket;
PROS
CONS
TLS Session Tickets
User Agent Session Tickets
Android 2.3.7 No
Android 4.0.4 YES
Android 5 YES
Chrome 47 YES
Firefox 42 YES
IE11/Win7 No
IE11/Win10 YES
Edge/Win10 YES
Safari 9/iOS9 No
Safari 9/MacOSX10.11 No
!!!!
!!!!
https://www.ssllabs.com/ssltest/clients.html
OCSP Stapling
https://wiki.mozilla.org/Security/Server_Side_TLS
Client required to checking the
revocation status. If OCSP
Responder is down, page load is
failed.
Server retrieves OCSP record,
caches it and servers it to client
directly. Fast!
OSCP Stapling
• All Android version don’t support yet
• iOS < 9 doesn’t support it
CONS
CONS
ssl_stapling on;
ssl_stapling_verify on;
ssl_trusted_certificate /path/to/server.crt;
resolver 8.8.8.8 valid=30s;
resolver_timeout 5s;
TLS buffer size
• TLS record size. Default 16KB, it’s too large. That’s
effect to latency

https://www.igvita.com/2013/12/16/optimizing-nginx-tls-time-to-first-byte/
ssl_buffer_size 1k;
Automated
TLS session tickets key
updating
TLS Session Tickets
• Should share ssl_session_ticket_key file between
servers
• Mozilla recommends replace it every day for
forward secrecy
• Previously we changed the key file almost every
month in manual operation😓
Update Tickets with Consul
• Inspired by Stretcher (fujiwara👍)
• Key_file is very small (48byte), It can store into a payload
of consul event
• We’re updating key_file twice a day by using Consul
Update Tickets with Consul
JOB
Consul
Servers
nginx
Consul
Agent
$ consul event 
-name=“ssl-session-ticket-refresh" 
$(openssl rand 48 | base64)
nginx
Consul
Agent
nginx
Consul
Agent
cat | jq -r '.Payload' | 
base64 -d | base64 -d 
> /path/to/ticket.key
service nginx reload
❶
❷ ❷ ❷
❸
SPDY to HTTP/2
SPDY is alive
Request to a webview
728
1,009,268
270,527
HTTP/1.1 SPDY HTTP2
SPDY to HTTP/2
• In Mobile App, transition to HTTP/2 from SPDY is not
so quickly
• Android 4.x, iOS 8...
• nginx dropped SPDY in 1.9.4, but we need to
transition gracefully in a few and more years
• We need a server that speak both HTTP/2 and SPDY
Tengine
Now we’re trying Tengine (at a previous slide)
cloudflare patch?
https://blog.cloudflare.com/introducing-http2/
End

More Related Content

What's hot

Alfresco search services: Now and Then
Alfresco search services: Now and ThenAlfresco search services: Now and Then
Alfresco search services: Now and Then
Angel Borroy López
 

What's hot (20)

Hadoop Meetup Jan 2019 - Overview of Ozone
Hadoop Meetup Jan 2019 - Overview of OzoneHadoop Meetup Jan 2019 - Overview of Ozone
Hadoop Meetup Jan 2019 - Overview of Ozone
 
Terraform
TerraformTerraform
Terraform
 
GraalVM Native Images by Oleg Selajev @shelajev
GraalVM Native Images by Oleg Selajev @shelajevGraalVM Native Images by Oleg Selajev @shelajev
GraalVM Native Images by Oleg Selajev @shelajev
 
Introduction to Kafka
Introduction to KafkaIntroduction to Kafka
Introduction to Kafka
 
Terraform 0.12 + Terragrunt
Terraform 0.12 + TerragruntTerraform 0.12 + Terragrunt
Terraform 0.12 + Terragrunt
 
Building Data Pipelines for Solr with Apache NiFi
Building Data Pipelines for Solr with Apache NiFiBuilding Data Pipelines for Solr with Apache NiFi
Building Data Pipelines for Solr with Apache NiFi
 
Alfresco search services: Now and Then
Alfresco search services: Now and ThenAlfresco search services: Now and Then
Alfresco search services: Now and Then
 
Event-sourced architectures with Akka
Event-sourced architectures with AkkaEvent-sourced architectures with Akka
Event-sourced architectures with Akka
 
Ozone- Object store for Apache Hadoop
Ozone- Object store for Apache HadoopOzone- Object store for Apache Hadoop
Ozone- Object store for Apache Hadoop
 
Apache Calcite (a tutorial given at BOSS '21)
Apache Calcite (a tutorial given at BOSS '21)Apache Calcite (a tutorial given at BOSS '21)
Apache Calcite (a tutorial given at BOSS '21)
 
Why Scala?
Why Scala?Why Scala?
Why Scala?
 
Building your First gRPC Service
Building your First gRPC ServiceBuilding your First gRPC Service
Building your First gRPC Service
 
Alfresco Certificates
Alfresco Certificates Alfresco Certificates
Alfresco Certificates
 
What's New in Apache Hive
What's New in Apache HiveWhat's New in Apache Hive
What's New in Apache Hive
 
Apache Flink internals
Apache Flink internalsApache Flink internals
Apache Flink internals
 
Oracle GoldenGate and Apache Kafka: A Deep Dive Into Real-Time Data Streaming
Oracle GoldenGate and Apache Kafka: A Deep Dive Into Real-Time Data StreamingOracle GoldenGate and Apache Kafka: A Deep Dive Into Real-Time Data Streaming
Oracle GoldenGate and Apache Kafka: A Deep Dive Into Real-Time Data Streaming
 
Metadata Extraction and Content Transformation
Metadata Extraction and Content TransformationMetadata Extraction and Content Transformation
Metadata Extraction and Content Transformation
 
Intro to the Alfresco Public API
Intro to the Alfresco Public APIIntro to the Alfresco Public API
Intro to the Alfresco Public API
 
user Behavior Analysis with Session Windows and Apache Kafka's Streams API
user Behavior Analysis with Session Windows and Apache Kafka's Streams APIuser Behavior Analysis with Session Windows and Apache Kafka's Streams API
user Behavior Analysis with Session Windows and Apache Kafka's Streams API
 
Nifi workshop
Nifi workshopNifi workshop
Nifi workshop
 

Similar to Advanced nginx in mercari - How to handle over 1,200,000 HTTPS Reqs/Min

KTC Profile-august
KTC Profile-augustKTC Profile-august
KTC Profile-august
KTC Host
 
Hacking Tizen : The OS of Everything - Nullcon Goa 2015
Hacking Tizen : The OS of Everything - Nullcon Goa 2015Hacking Tizen : The OS of Everything - Nullcon Goa 2015
Hacking Tizen : The OS of Everything - Nullcon Goa 2015
Ajin Abraham
 

Similar to Advanced nginx in mercari - How to handle over 1,200,000 HTTPS Reqs/Min (20)

AD113 Speed Up Your Applications w/ Nginx and PageSpeed
AD113  Speed Up Your Applications w/ Nginx and PageSpeedAD113  Speed Up Your Applications w/ Nginx and PageSpeed
AD113 Speed Up Your Applications w/ Nginx and PageSpeed
 
E g innovations overview
E g innovations overviewE g innovations overview
E g innovations overview
 
Deploy secure, scalable, and highly available web apps with Azure Front Door ...
Deploy secure, scalable, and highly available web apps with Azure Front Door ...Deploy secure, scalable, and highly available web apps with Azure Front Door ...
Deploy secure, scalable, and highly available web apps with Azure Front Door ...
 
AWS re:Invent 2016: Amazon CloudFront Flash Talks: Best Practices on Configur...
AWS re:Invent 2016: Amazon CloudFront Flash Talks: Best Practices on Configur...AWS re:Invent 2016: Amazon CloudFront Flash Talks: Best Practices on Configur...
AWS re:Invent 2016: Amazon CloudFront Flash Talks: Best Practices on Configur...
 
Citirx Day 2013: Citrix Enterprise Mobility
Citirx Day 2013: Citrix Enterprise MobilityCitirx Day 2013: Citrix Enterprise Mobility
Citirx Day 2013: Citrix Enterprise Mobility
 
56k.cloud training
56k.cloud training56k.cloud training
56k.cloud training
 
Choosing a Citrix Monitoring Strategy: Key Capabilities and Pitfalls to Avoid
Choosing a Citrix Monitoring Strategy: Key Capabilities and Pitfalls to AvoidChoosing a Citrix Monitoring Strategy: Key Capabilities and Pitfalls to Avoid
Choosing a Citrix Monitoring Strategy: Key Capabilities and Pitfalls to Avoid
 
Architecting &Building Scalable Secure Web API
Architecting &Building Scalable Secure Web APIArchitecting &Building Scalable Secure Web API
Architecting &Building Scalable Secure Web API
 
KTC Profile-august
KTC Profile-augustKTC Profile-august
KTC Profile-august
 
Advantages of cPanel-based LiteSpeed Hosting
Advantages of cPanel-based LiteSpeed HostingAdvantages of cPanel-based LiteSpeed Hosting
Advantages of cPanel-based LiteSpeed Hosting
 
The Real World - Plugging the Enterprise Into It (nodejs)
The Real World - Plugging  the Enterprise Into It (nodejs)The Real World - Plugging  the Enterprise Into It (nodejs)
The Real World - Plugging the Enterprise Into It (nodejs)
 
Adobe Flash Platform for the Enterprise
Adobe Flash Platform for the EnterpriseAdobe Flash Platform for the Enterprise
Adobe Flash Platform for the Enterprise
 
Secure Content Delivery Using Amazon CloudFront and AWS WAF
Secure Content Delivery Using Amazon CloudFront and AWS WAFSecure Content Delivery Using Amazon CloudFront and AWS WAF
Secure Content Delivery Using Amazon CloudFront and AWS WAF
 
Hacking Tizen : The OS of Everything - Nullcon Goa 2015
Hacking Tizen : The OS of Everything - Nullcon Goa 2015Hacking Tizen : The OS of Everything - Nullcon Goa 2015
Hacking Tizen : The OS of Everything - Nullcon Goa 2015
 
High Volume Payments using Mule
High Volume Payments using MuleHigh Volume Payments using Mule
High Volume Payments using Mule
 
HTTP - The Other Face Of Domino
HTTP - The Other Face Of DominoHTTP - The Other Face Of Domino
HTTP - The Other Face Of Domino
 
Secure Content Delivery Using Amazon CloudFront and AWS WAF
Secure Content Delivery Using Amazon CloudFront and AWS WAFSecure Content Delivery Using Amazon CloudFront and AWS WAF
Secure Content Delivery Using Amazon CloudFront and AWS WAF
 
Secure Content Delivery Using Amazon CloudFront and AWS WAF
Secure Content Delivery Using Amazon CloudFront and AWS WAFSecure Content Delivery Using Amazon CloudFront and AWS WAF
Secure Content Delivery Using Amazon CloudFront and AWS WAF
 
SSL for SaaS Providers
SSL for SaaS ProvidersSSL for SaaS Providers
SSL for SaaS Providers
 
Securing MongoDB to Serve an AWS-Based, Multi-Tenant, Security-Fanatic SaaS A...
Securing MongoDB to Serve an AWS-Based, Multi-Tenant, Security-Fanatic SaaS A...Securing MongoDB to Serve an AWS-Based, Multi-Tenant, Security-Fanatic SaaS A...
Securing MongoDB to Serve an AWS-Based, Multi-Tenant, Security-Fanatic SaaS A...
 

More from Masahiro Nagano

ISUCON4 予選問題で(中略)、”my.cnf”に1行だけ足して予選通過ラインを突破するの術
ISUCON4 予選問題で(中略)、”my.cnf”に1行だけ足して予選通過ラインを突破するの術ISUCON4 予選問題で(中略)、”my.cnf”に1行だけ足して予選通過ラインを突破するの術
ISUCON4 予選問題で(中略)、”my.cnf”に1行だけ足して予選通過ラインを突破するの術
Masahiro Nagano
 
Isucon makers casual talks
Isucon makers casual talksIsucon makers casual talks
Isucon makers casual talks
Masahiro Nagano
 
blogサービスの全文検索の話 - #groonga を囲む夕べ
blogサービスの全文検索の話 - #groonga を囲む夕べblogサービスの全文検索の話 - #groonga を囲む夕べ
blogサービスの全文検索の話 - #groonga を囲む夕べ
Masahiro Nagano
 
Gazelle - Plack Handler for performance freaks #yokohamapm
Gazelle - Plack Handler for performance freaks #yokohamapmGazelle - Plack Handler for performance freaks #yokohamapm
Gazelle - Plack Handler for performance freaks #yokohamapm
Masahiro Nagano
 
Dockerで遊んでみよっかー YAPC::Asia Tokyo 2014
Dockerで遊んでみよっかー YAPC::Asia Tokyo 2014Dockerで遊んでみよっかー YAPC::Asia Tokyo 2014
Dockerで遊んでみよっかー YAPC::Asia Tokyo 2014
Masahiro Nagano
 
Web Framework Benchmarksと Perl の現状報告会 YAPC::Asia Tokyo 2014 LT
Web Framework Benchmarksと Perl の現状報告会 YAPC::Asia Tokyo 2014 LTWeb Framework Benchmarksと Perl の現状報告会 YAPC::Asia Tokyo 2014 LT
Web Framework Benchmarksと Perl の現状報告会 YAPC::Asia Tokyo 2014 LT
Masahiro Nagano
 
ISUCONで学ぶ Webアプリケーションのパフォーマンス向上のコツ 実践編 完全版
ISUCONで学ぶ Webアプリケーションのパフォーマンス向上のコツ 実践編 完全版ISUCONで学ぶ Webアプリケーションのパフォーマンス向上のコツ 実践編 完全版
ISUCONで学ぶ Webアプリケーションのパフォーマンス向上のコツ 実践編 完全版
Masahiro Nagano
 
Webアプリケーションの パフォーマンス向上のコツ 実践編
 Webアプリケーションの パフォーマンス向上のコツ 実践編 Webアプリケーションの パフォーマンス向上のコツ 実践編
Webアプリケーションの パフォーマンス向上のコツ 実践編
Masahiro Nagano
 
Webアプリケーションの パフォーマンス向上のコツ 概要編
 Webアプリケーションの パフォーマンス向上のコツ 概要編 Webアプリケーションの パフォーマンス向上のコツ 概要編
Webアプリケーションの パフォーマンス向上のコツ 概要編
Masahiro Nagano
 
Webアプリケーションとメモリ
WebアプリケーションとメモリWebアプリケーションとメモリ
Webアプリケーションとメモリ
Masahiro Nagano
 
最近作ったN個のCPANモジュール Yokohama.pm #10
最近作ったN個のCPANモジュール Yokohama.pm #10最近作ったN個のCPANモジュール Yokohama.pm #10
最近作ったN個のCPANモジュール Yokohama.pm #10
Masahiro Nagano
 

More from Masahiro Nagano (20)

Rhebok, High Performance Rack Handler / Rubykaigi 2015
Rhebok, High Performance Rack Handler / Rubykaigi 2015Rhebok, High Performance Rack Handler / Rubykaigi 2015
Rhebok, High Performance Rack Handler / Rubykaigi 2015
 
Big Master Data PHP BLT #1
Big Master Data PHP BLT #1Big Master Data PHP BLT #1
Big Master Data PHP BLT #1
 
Stream processing in Mercari - Devsumi 2015 autumn LT
Stream processing in Mercari - Devsumi 2015 autumn LTStream processing in Mercari - Devsumi 2015 autumn LT
Stream processing in Mercari - Devsumi 2015 autumn LT
 
メルカリのデータベース戦略 / PHPとMySQLの怖い話 MyNA会2015年8月
メルカリのデータベース戦略 / PHPとMySQLの怖い話 MyNA会2015年8月メルカリのデータベース戦略 / PHPとMySQLの怖い話 MyNA会2015年8月
メルカリのデータベース戦略 / PHPとMySQLの怖い話 MyNA会2015年8月
 
ISUCONの勝ち方 YAPC::Asia Tokyo 2015
ISUCONの勝ち方 YAPC::Asia Tokyo 2015ISUCONの勝ち方 YAPC::Asia Tokyo 2015
ISUCONの勝ち方 YAPC::Asia Tokyo 2015
 
Norikraで作るPHPの例外検知システム YAPC::Asia Tokyo 2015 LT
Norikraで作るPHPの例外検知システム YAPC::Asia Tokyo 2015 LTNorikraで作るPHPの例外検知システム YAPC::Asia Tokyo 2015 LT
Norikraで作るPHPの例外検知システム YAPC::Asia Tokyo 2015 LT
 
メルカリでのNorikraの活用、 Mackerelを添えて
メルカリでのNorikraの活用、 Mackerelを添えてメルカリでのNorikraの活用、 Mackerelを添えて
メルカリでのNorikraの活用、 Mackerelを添えて
 
Gazelle & CPAN modules for performance. Shibuya.pm Tech Talk #17 LT
Gazelle & CPAN modules for performance. Shibuya.pm Tech Talk #17 LTGazelle & CPAN modules for performance. Shibuya.pm Tech Talk #17 LT
Gazelle & CPAN modules for performance. Shibuya.pm Tech Talk #17 LT
 
Mackerel & Norikra mackerel meetup #4 LT
Mackerel & Norikra mackerel meetup #4 LTMackerel & Norikra mackerel meetup #4 LT
Mackerel & Norikra mackerel meetup #4 LT
 
ISUCON4 予選問題で(中略)、”my.cnf”に1行だけ足して予選通過ラインを突破するの術
ISUCON4 予選問題で(中略)、”my.cnf”に1行だけ足して予選通過ラインを突破するの術ISUCON4 予選問題で(中略)、”my.cnf”に1行だけ足して予選通過ラインを突破するの術
ISUCON4 予選問題で(中略)、”my.cnf”に1行だけ足して予選通過ラインを突破するの術
 
Isucon makers casual talks
Isucon makers casual talksIsucon makers casual talks
Isucon makers casual talks
 
blogサービスの全文検索の話 - #groonga を囲む夕べ
blogサービスの全文検索の話 - #groonga を囲む夕べblogサービスの全文検索の話 - #groonga を囲む夕べ
blogサービスの全文検索の話 - #groonga を囲む夕べ
 
Gazelle - Plack Handler for performance freaks #yokohamapm
Gazelle - Plack Handler for performance freaks #yokohamapmGazelle - Plack Handler for performance freaks #yokohamapm
Gazelle - Plack Handler for performance freaks #yokohamapm
 
Dockerで遊んでみよっかー YAPC::Asia Tokyo 2014
Dockerで遊んでみよっかー YAPC::Asia Tokyo 2014Dockerで遊んでみよっかー YAPC::Asia Tokyo 2014
Dockerで遊んでみよっかー YAPC::Asia Tokyo 2014
 
Web Framework Benchmarksと Perl の現状報告会 YAPC::Asia Tokyo 2014 LT
Web Framework Benchmarksと Perl の現状報告会 YAPC::Asia Tokyo 2014 LTWeb Framework Benchmarksと Perl の現状報告会 YAPC::Asia Tokyo 2014 LT
Web Framework Benchmarksと Perl の現状報告会 YAPC::Asia Tokyo 2014 LT
 
ISUCONで学ぶ Webアプリケーションのパフォーマンス向上のコツ 実践編 完全版
ISUCONで学ぶ Webアプリケーションのパフォーマンス向上のコツ 実践編 完全版ISUCONで学ぶ Webアプリケーションのパフォーマンス向上のコツ 実践編 完全版
ISUCONで学ぶ Webアプリケーションのパフォーマンス向上のコツ 実践編 完全版
 
Webアプリケーションの パフォーマンス向上のコツ 実践編
 Webアプリケーションの パフォーマンス向上のコツ 実践編 Webアプリケーションの パフォーマンス向上のコツ 実践編
Webアプリケーションの パフォーマンス向上のコツ 実践編
 
Webアプリケーションの パフォーマンス向上のコツ 概要編
 Webアプリケーションの パフォーマンス向上のコツ 概要編 Webアプリケーションの パフォーマンス向上のコツ 概要編
Webアプリケーションの パフォーマンス向上のコツ 概要編
 
Webアプリケーションとメモリ
WebアプリケーションとメモリWebアプリケーションとメモリ
Webアプリケーションとメモリ
 
最近作ったN個のCPANモジュール Yokohama.pm #10
最近作ったN個のCPANモジュール Yokohama.pm #10最近作ったN個のCPANモジュール Yokohama.pm #10
最近作ったN個のCPANモジュール Yokohama.pm #10
 

Recently uploaded

Hyatt driving innovation and exceptional customer experiences with FIDO passw...
Hyatt driving innovation and exceptional customer experiences with FIDO passw...Hyatt driving innovation and exceptional customer experiences with FIDO passw...
Hyatt driving innovation and exceptional customer experiences with FIDO passw...
FIDO Alliance
 

Recently uploaded (20)

Simplifying Mobile A11y Presentation.pptx
Simplifying Mobile A11y Presentation.pptxSimplifying Mobile A11y Presentation.pptx
Simplifying Mobile A11y Presentation.pptx
 
Continuing Bonds Through AI: A Hermeneutic Reflection on Thanabots
Continuing Bonds Through AI: A Hermeneutic Reflection on ThanabotsContinuing Bonds Through AI: A Hermeneutic Reflection on Thanabots
Continuing Bonds Through AI: A Hermeneutic Reflection on Thanabots
 
Observability Concepts EVERY Developer Should Know (DevOpsDays Seattle)
Observability Concepts EVERY Developer Should Know (DevOpsDays Seattle)Observability Concepts EVERY Developer Should Know (DevOpsDays Seattle)
Observability Concepts EVERY Developer Should Know (DevOpsDays Seattle)
 
الأمن السيبراني - ما لا يسع للمستخدم جهله
الأمن السيبراني - ما لا يسع للمستخدم جهلهالأمن السيبراني - ما لا يسع للمستخدم جهله
الأمن السيبراني - ما لا يسع للمستخدم جهله
 
AI in Action: Real World Use Cases by Anitaraj
AI in Action: Real World Use Cases by AnitarajAI in Action: Real World Use Cases by Anitaraj
AI in Action: Real World Use Cases by Anitaraj
 
ChatGPT and Beyond - Elevating DevOps Productivity
ChatGPT and Beyond - Elevating DevOps ProductivityChatGPT and Beyond - Elevating DevOps Productivity
ChatGPT and Beyond - Elevating DevOps Productivity
 
Human Expert Website Manual WCAG 2.0 2.1 2.2 Audit - Digital Accessibility Au...
Human Expert Website Manual WCAG 2.0 2.1 2.2 Audit - Digital Accessibility Au...Human Expert Website Manual WCAG 2.0 2.1 2.2 Audit - Digital Accessibility Au...
Human Expert Website Manual WCAG 2.0 2.1 2.2 Audit - Digital Accessibility Au...
 
Intro to Passkeys and the State of Passwordless.pptx
Intro to Passkeys and the State of Passwordless.pptxIntro to Passkeys and the State of Passwordless.pptx
Intro to Passkeys and the State of Passwordless.pptx
 
How to Check CNIC Information Online with Pakdata cf
How to Check CNIC Information Online with Pakdata cfHow to Check CNIC Information Online with Pakdata cf
How to Check CNIC Information Online with Pakdata cf
 
Design and Development of a Provenance Capture Platform for Data Science
Design and Development of a Provenance Capture Platform for Data ScienceDesign and Development of a Provenance Capture Platform for Data Science
Design and Development of a Provenance Capture Platform for Data Science
 
State of the Smart Building Startup Landscape 2024!
State of the Smart Building Startup Landscape 2024!State of the Smart Building Startup Landscape 2024!
State of the Smart Building Startup Landscape 2024!
 
Microsoft CSP Briefing Pre-Engagement - Questionnaire
Microsoft CSP Briefing Pre-Engagement - QuestionnaireMicrosoft CSP Briefing Pre-Engagement - Questionnaire
Microsoft CSP Briefing Pre-Engagement - Questionnaire
 
Event-Driven Architecture Masterclass: Integrating Distributed Data Stores Ac...
Event-Driven Architecture Masterclass: Integrating Distributed Data Stores Ac...Event-Driven Architecture Masterclass: Integrating Distributed Data Stores Ac...
Event-Driven Architecture Masterclass: Integrating Distributed Data Stores Ac...
 
WebRTC and SIP not just audio and video @ OpenSIPS 2024
WebRTC and SIP not just audio and video @ OpenSIPS 2024WebRTC and SIP not just audio and video @ OpenSIPS 2024
WebRTC and SIP not just audio and video @ OpenSIPS 2024
 
Working together SRE & Platform Engineering
Working together SRE & Platform EngineeringWorking together SRE & Platform Engineering
Working together SRE & Platform Engineering
 
Navigating the Large Language Model choices_Ravi Daparthi
Navigating the Large Language Model choices_Ravi DaparthiNavigating the Large Language Model choices_Ravi Daparthi
Navigating the Large Language Model choices_Ravi Daparthi
 
JohnPollard-hybrid-app-RailsConf2024.pptx
JohnPollard-hybrid-app-RailsConf2024.pptxJohnPollard-hybrid-app-RailsConf2024.pptx
JohnPollard-hybrid-app-RailsConf2024.pptx
 
Hyatt driving innovation and exceptional customer experiences with FIDO passw...
Hyatt driving innovation and exceptional customer experiences with FIDO passw...Hyatt driving innovation and exceptional customer experiences with FIDO passw...
Hyatt driving innovation and exceptional customer experiences with FIDO passw...
 
Introduction to FIDO Authentication and Passkeys.pptx
Introduction to FIDO Authentication and Passkeys.pptxIntroduction to FIDO Authentication and Passkeys.pptx
Introduction to FIDO Authentication and Passkeys.pptx
 
The Zero-ETL Approach: Enhancing Data Agility and Insight
The Zero-ETL Approach: Enhancing Data Agility and InsightThe Zero-ETL Approach: Enhancing Data Agility and Insight
The Zero-ETL Approach: Enhancing Data Agility and Insight
 

Advanced nginx in mercari - How to handle over 1,200,000 HTTPS Reqs/Min

  • 1. Advanced nginx in Mercari 「nginx実践入門」出版記念!執筆者らが語る nginx Tech Talks Masahiro Nagano How to handle over 1,200,000 HTTPS Reqs/Min
  • 2. Me • Masahiro Nagano @kazeburo • Mercari, inc. • Principal Engineer, Site Reliability Engineering Team • BASE, inc. Technical Advisor
  • 3. Mercari Your Friendly Mobile MarketPlace JP US
  • 6. nginx in mercari ©2011 Amazon Web Services LLC or its affiliates. All rights reserved. Users Client Multimedia Corporate data center Trad se Mobile Client AWS Management Console IAM Add-on Example: IAM Add-on Tasks (HIT) TaskTurk ice Specific ©2011 Amazon Web Services LLC or its affiliates. All rights reserved. User Users Client Multimedia Corporatedata center Mobile Client net AWS Management Console IAM Add-on Example:IAM Add-on Requester azon W eb Services LLC or its affiliates. All rights reserved. ser Users Client Multim edia Corporate data center Tradit server Mobile Client AW S Managem ent Console IAM Add-on Exam ple: IAM Add-on ence HIT) Assignm ent/ Task W orkers L7 load balancer Reverse Proxy TLS Terminator Contents Distribution©2011 Am azon W eb Services LLC or its affiliates. All rights reserve Users Client M ultim edia Corporate data center Tradition server M obile Client Internet AW S M anagem ent Console IAM Add-on Exam ple: IAM Add-on m ent/ Task Requester W orkers
  • 7. nginx Extension in mercari • ngx_dynamic_upstream • Operate upsteam servers Up/Down dynamically • OpenResty (ngx_lua) • More powerful L7 load balancer • Application log collector
  • 8. Agenda • Continuous updating nginx • TLS Optimization • Automated TLS session ticket key updating • SPDY to HTTP/2
  • 10. Keep nginx the latest • Use nginx mainline version • stableは安定しているという意味ではなく新機能が追加さ れず、バグ修正も緊急な修正を除いて行われないというこ と。すべての新機能やバグ修正はmainlineにて行われるの で結果的にmainlineが安定している
 https://www.nginx.com/blog/nginx-1-6-1-7-released/ • nginxは無停止でのアップデートが可能 • Periodic OpenSSL vulnerability report 😓
  • 11. nginx-build • nginx builder command written in Go
 https://github.com/cubicdaiya/nginx-build • Automate download, configure and make • Download OpenSSL, zlib and PCRE. build them statically ( independent from OS’s library version) • Support Tengine and OpenResty
  • 12. nginx-build $ EXPORT $VERSION=1.9.10 $ nginx-build -clear -d work -v $VERSION -c nginx-build-conf/configure -m nginx-build-conf/modules3rd.ini -zlib -zlibversion=1.2.8 -verbose -pcre -pcreversion=8.38 -openssl -opensslversion=1.0.2f $ cd work/$VERSION/nginx-$VERSION $ sudo make install
  • 14. TLS Optimization • PFS - Perfect Forward Secrecy • TTFB - Time To First Byte
  • 15. Perfect Forward Secrecy • Mozilla Wiki
 https://wiki.mozilla.org/Security/Server_Side_TLS • Mozilla SSL Configuration Generator
 https://mozilla.github.io/server-side-tls/ssl-config- generator/ • *最新の情報を参照し、プロダクション環境に投 入する前にテストを行ってください
  • 16. TTFB • TLS Session Cache • TLS Session Tickets • OCSP Stapling • SSL buffer size
  • 17. TLS Session Cache • Server side session cache • Client software does not matter • nginx could’t share session cache between servers. ssl_session_cache shared:SSL:120m; ssl_session_timeout 5m; CONS PROS
  • 18. TLS Session Cache • OpenResty will support it?
 https://twitter.com/agentzh/status/ 686655229828403201
  • 19. TLS Session Tickets • Client Side Session Cache
 (like a CookieStore) • No need to share cache between servers • It’s required share ssl_session_ticket_key file • Client support required ssl_session_tickets on; ssl_session_ticket_key /path/to/ssl_session_ticket; PROS CONS
  • 20. TLS Session Tickets User Agent Session Tickets Android 2.3.7 No Android 4.0.4 YES Android 5 YES Chrome 47 YES Firefox 42 YES IE11/Win7 No IE11/Win10 YES Edge/Win10 YES Safari 9/iOS9 No Safari 9/MacOSX10.11 No !!!! !!!! https://www.ssllabs.com/ssltest/clients.html
  • 21. OCSP Stapling https://wiki.mozilla.org/Security/Server_Side_TLS Client required to checking the revocation status. If OCSP Responder is down, page load is failed. Server retrieves OCSP record, caches it and servers it to client directly. Fast!
  • 22. OSCP Stapling • All Android version don’t support yet • iOS < 9 doesn’t support it CONS CONS ssl_stapling on; ssl_stapling_verify on; ssl_trusted_certificate /path/to/server.crt; resolver 8.8.8.8 valid=30s; resolver_timeout 5s;
  • 23. TLS buffer size • TLS record size. Default 16KB, it’s too large. That’s effect to latency
 https://www.igvita.com/2013/12/16/optimizing-nginx-tls-time-to-first-byte/ ssl_buffer_size 1k;
  • 25. TLS Session Tickets • Should share ssl_session_ticket_key file between servers • Mozilla recommends replace it every day for forward secrecy • Previously we changed the key file almost every month in manual operation😓
  • 26. Update Tickets with Consul • Inspired by Stretcher (fujiwara👍) • Key_file is very small (48byte), It can store into a payload of consul event • We’re updating key_file twice a day by using Consul
  • 27. Update Tickets with Consul JOB Consul Servers nginx Consul Agent $ consul event -name=“ssl-session-ticket-refresh" $(openssl rand 48 | base64) nginx Consul Agent nginx Consul Agent cat | jq -r '.Payload' | base64 -d | base64 -d > /path/to/ticket.key service nginx reload ❶ ❷ ❷ ❷ ❸
  • 29. SPDY is alive Request to a webview 728 1,009,268 270,527 HTTP/1.1 SPDY HTTP2
  • 30. SPDY to HTTP/2 • In Mobile App, transition to HTTP/2 from SPDY is not so quickly • Android 4.x, iOS 8... • nginx dropped SPDY in 1.9.4, but we need to transition gracefully in a few and more years • We need a server that speak both HTTP/2 and SPDY
  • 31. Tengine Now we’re trying Tengine (at a previous slide)
  • 33. End