This document summarizes four Japanese AdTech companies that use AWS: Dynalyst, fluct, IM-DMP, and UNICORN. Dynalyst uses AWS for real-time bidding and cross-region data processing. fluct is an SSP that processes 30 billion impressions per month on a serverless architecture. IM-DMP utilizes Amazon ECS and Spot Fleet to power its public DMP. UNICORN is a full automated marketing platform that uses AWS for real-time bidding, data analysis, and machine learning.

1. Accelerating AdTech on AWS in Japan
Pragmatic use-cases
Dynalyst / fluct / IM-DMP / UNICORN
Eiji Shinohara
Amazon Web Services Japan, Solutions Architect
August 30, 2017 at MARU180

2. 안녕하십니까!
@werner: Amazon CTO
Piljoong
Park-san
Eiji Shinohara (篠原 英治)
§ Twitter: @shinodogg
§ Blog: shinodogg.com
AWS Solutions Architect
§ Market: AdTech & Startup
§ Area of Depth: Search
Korean Cuisine Lover
§ 三겹살/삼겹살, 불고기, 김치,,,

3. Agenda
Japanese AdTech Industry
Japanese AdTech Community
AdTech on AWS use-cases in Japan
vDynalyst http://www.dynalyst.io
vfluct https://fluct.jp
vIM-DMP https://corp.intimatemerger.com
vUNICORN https://uncn.jp

4. Japanese AdTech Industry

5. Japanese AdTech Industry
JP 2016 Internet Ads Market Size Research by CCI
http://www.cci.co.jp/news/release/2017_04_17/1.html

6. Japanese AdTech Industry
JP 2016 Internet Ads Market Size Research by CCI
http://www.cci.co.jp/news/release/2017_04_17/1.html
$10 Billion
Market
Smartphone
Shift
SmartphoneDesktop

7. Japanese AdTech Industry
https://www.slideshare.net/HiroshiKondo/jp-chaosmap-20152016

8. Japanese AdTech Industry
https://www.slideshare.net/HiroshiKondo/jp-chaosmap-20152016
Fragmented and Competitive
same as other regions

9. Japanese AdTech Community

10. Japanese AdTech Community
AdTech Meetup by AWS in 2016 #AWSAdTechJP

11. Japanese AdTech Community
AdTech Meetup by AWS in 2016 #AWSAdTechJP
“Digital Marketing”
Trend
DialogOne
“LINE” Business Connect
“AdNetwork”
Admin Tools
http://aws.typepad.com/sajp/2016/07/aws-adtech-jp.html
Wrap-up
Blog Post

12. Japanese AdTech Community
Akiba Lab – Over 800 people in Facebook group

13. Japanese AdTech Community
Akiba Lab – Over 800 people in Facebook group
アドテク⇒AdTech
Akiba Lab is a Japanese AdTech community
Big year-end party in Dec 2016
Lightning Talks

14. Japanese AdTech Community
#CTONight powered by AWS

15. Japanese AdTech Community
#CTONight powered by AWS
Japanese Top-Notch
CTOs gathering

16. Japanese AdTech Community
#CTONight powered by AWS
v AdTech CTOs at CTO Night J

17. Japanese AdTech Community
AdTech CTOs Round Table
v Deeper AdTech Talks!

18. Japanese AdTech on AWS
Use-Cases

19. AdTech on AWS Use-Cases in Japan
Dynalyst http://www.dynalyst.io
v Re-Targeting / Re-Engaging
v Japan and U.S.
fluct https://fluct.jp
v SSP: 30 billion impressions in a month
v Ajitofm: Podcast @ VOYAGE GROUP in company bar
IM-DMP https://corp.intimatemerger.com
v Public DMP
v Small Engineering Team delivers Big Result
UNICORN https://uncn.jp
v Full Automated Marketing Platform
v International Engineers in Tokyo

20. CyberAgent - Internet Conglomerate
https://www.cyberagent.co.jp/en/

21. CyberAgent - Internet Conglomerate
http://pdf.cyberagent.co.jp/C4751/GEqq/Pzq8/RSrX.pdf

22. CyberAgent - Internet Conglomerate
http://pdf.cyberagent.co.jp/C4751/GEqq/Pzq8/RSrX.pdf
About 500 million USD
Ad Business

23. CyberAgent - AdTech Studio
https://adtech.cyberagent.io/
https://www.slideshare.net/InsightTechnology/dbts-
tokyo-2016-b15-bigdata-actian-matrix-mapr-cyberagent

24. CyberAgent - AdTech Studio
https://adtech.cyberagent.io/
https://www.slideshare.net/InsightTechnology/dbts-
tokyo-2016-b15-bigdata-actian-matrix-mapr-cyberagent

25. Dynalyst - Dynamic Retargeting for Game Apps
Massive Audience
Personalized Engagement
AWS Summit Tokyo 2014
AWS Summit Tokyo 2015

26. Dynalyst - Log Processing Architecture

27. Japan
US
ap-northeast-1
us-east-1
Up to
100 instances
Up to
80 shards
KCL on ECS
Docker Cluster
S3
Redshift
EMR
Up to
100 instances
Up to
80 shards
KCL on ECS
Docker Cluster
Dynalyst - Log Processing Architecture

28. Japan
US
ap-northeast-1
us-east-1
Up to
100 instances
Up to
80 shards
KCL on ECS
Docker Cluster
S3
Redshift
EMR
Up to
100 instances
Up to
80 shards
KCL on ECS
Docker Cluster
Dynalyst - Log Processing Architecture
Petabyte Scale

29. Dynalyst - Cross Region Replication
Real-Time Bidding
https://media.mopub.com/media/filer_public/30/1f/301ffdbc-1edb-4e8a-ab22-a3d5db57851e/mopub_dynalyst_case_study.pdf

30. Dynalyst - Cross Region Replication
Real-Time Bidding
https://media.mopub.com/media/filer_public/30/1f/301ffdbc-1edb-4e8a-ab22-a3d5db57851e/mopub_dynalyst_case_study.pdf
Network Latency is Critical
https://www.mopub.com/resources/mopub-demand/mopub-marketplace-overview/network-infrastructure/

31. Dynalyst - Cross Region Replication
DynamoDB Streams
Amazon RDS Aurora

32. Dynalyst - Real-Time Bidding
Train Model: Spark ML / Save Model: Redis
Quick Response to Bid Requests!
EMR ElastiCacheS3
Bid Request
Memcached
Redis Aurora
DynamoDB

33. Dynalyst - Go Global with AWS!
Shuhei Kimura
v Moving back and forth from Japan to U.S.
v Diving deeply into U.S. AdTech eco-system
v Planning to use another AWS region in US West

34. VOYAGE GROUP - fluct: SSP

35. fluct - Serverless Architecture in 2016
Serverless for Analyzing contents
vBetter Contents/Context matched Ad delivery
https://speakerdeck.com/suzuken/how-to-use-aws-lambda-in-document-processing-pipeline

36. fluct – SSP: 30billion impressions in a month
Kenta Suzuki
A. Advertising transparency
v Players are relying on each other
v Preventing unethical actions is an
entire industry problem!
v Letʼs make the Internet better
place J
Q. What is the trend in AdTech
industry?

37. fluct magazine https://magazine.fluct.jp
fluct – SSP: 30billion impressions in a month

38. fluct – SSP: 30billion impressions in a month
“ads.txt” aims to increase transparency
in the AdTech ecosystem
How do we introduce ads.txt?
fluct magazine https://magazine.fluct.jp

39. fluct - Ad Verification Architecture

40. Ad
tag
ALB ECS
Lambda
Lambda
Amazon ES
3rd Party
service
Kinesis
fluct - Ad Verification Architecture

41. Ad
tag
ALB ECS
Lambda
Lambda
Amazon ES
3rd Party
service
Kinesis
Speed Layer
Batch Layer
fluct - Ad Verification Architecture

42. Ad
tag
ALB ECS
Lambda
Lambda
Amazon ES
3rd Party
service
Kinesis
Running “Golang” application
on AWS Lambda w/ Apex
fluct - Serverless Architecture

43. Apex – Serverless Infrastructure
“build, deploy, and manage AWS Lambda functions with ease”
http://apex.run/

44. https://speakerdeck.com/kakakakakku/case-study-of-lambda-and-apex
Apex – Serverless Infrastructure
Running Golang on AWS Lambda

45. https://speakerdeck.com/kakakakakku/case-study-of-lambda-and-apex
Running Golang on AWS Lambda
Apex – Serverless Infrastructure

46. Tech Podcast - VOYAGE GROUP
https://www.instagram.com/p/BXWzZ9ngYLZ/HUGO (https://gohugo.io/) + Hosting on S3
Ajitofm https://ajito.fm/

47. https://ajito.fm/2/
Tech Podcast - VOYAGE GROUP
Running Golang on AWS Lambda
v Node.js -> Golang
Running Golang as a Child Process
Utilize STDIN and STDOUT
Sounds like “CGI” in Cloud ERA...
http://www.kent-web.com/

48. https://ajito.fm/2/
v Node.js -> Golang
Running Golang as a Child Process
Utilize STDIN and STDOUT
Sounds like “CGI” in Cloud ERA...
Popular CGI Examples
In 90s…
Tech Podcast - VOYAGE GROUP
Running Golang on AWS Lambda
http://www.kent-web.com/

49. AJITO – VOYAGE GROUP Inside Company Bar

50. Intimate Merger - IM-DMP

51. Intimate Merger - IM-DMP
Intimate Merger
v Founded in 2013 as a Joint Venture
FreakOut: The first DSP in Japan
Preferred Infrastructure: Cutting Edge Tech
v Shareholders in 2017
FreakOut Holdings: Global Marketing Tech group
Dentsu: Worldʼs leading Advertising Agency
YJCapital: Yahoo! Japan Corporate Venture Capital

52. Intimate Merger - IM-DMP
w/ dentsu
v Contribute to Public DMP ”dPublic” by dentsu
w/ Yahoo! Japan
v Connect to Yahoo! Japan DMP
https://corp.intimatemerger.com/archives/1855/

53. Intimate Merger - IM-DMP
w/ dentsu
v Contribute to Public DMP ”dPublic” by dentsu
w/ Yahoo! Japan
v Connect to Yahoo! Japan DMP
https://corp.intimatemerger.com/archives/1855/
PsychographicDemographic
400 million
Audience Data

54. IM-DMP - Architecture

55. IM-DMP - Architecture
ECS
RDS
ElastiCache
Spot Fleet
ECS
CSV
TSV
JSON
SQS
IM-DMP
UI&API Data
Processing
S3
3rd Party
Partners

56. ECS
RDS
ElastiCache
CSV
TSV
JSON
SQS
IM-DMP
UI&API Data
Processing
S3
3rd Party
Data Providers
Everything is started from S3 upload
Simple File-based Trigger
Spot Fleet
ECS
IM-DMP - S3 File-based Architecture

57. ECS
RDS
ElastiCache
CSV
TSV
JSON
SQS
IM-DMP
UI&API Data
Processing
S3
3rd Party
Partners
ECS & Spot Fleet
Spot Fleet
ECS
IM-DMP - Amazon ECS & EC2 Spot Fleet

58. AWS Summit Tokyo 2017
https://www.youtube.com/watch?v=vJZJY_t8JDE https://www.slideshare.net/im_docs/amazon-ecs-spotfleet
IM-DMP - Amazon ECS & EC2 Spot Fleet

59. AWS Summit Tokyo 2017
https://www.youtube.com/watch?v=vJZJY_t8JDE https://www.slideshare.net/im_docs/amazon-ecs-spotfleet
IM-DMP - Amazon ECS & EC2 Spot Fleet

60. Amazon EC2 Spot Instances
July 26, 2017 / ap-northeast-1 / Linux
On
Demand
Reserved Instances for 1 year
Spot
Instances
Spot Block
All
Upfront
Partial
Upfront
No
Upfront 1h 6h
c4.large $0.126
$0.084
(33%)
$0.086
(32%)
$0.090
(29%)
$0.029
(77%)
$0.077
(39%)
$0.098
(22%)
m4.large $0.129
$0.081
(37%)
$0.083
(36%)
$0.087
(32%)
$0.027
(79%)
$0.101
(21%)
$0.128
(0.7%)
r3.large $0.20
$0.127
(36%)
$0.130
(35%)
$0.149
(26%)
$0.031
(84%)
$0.116
(42%)
$0.147
(26%)

61. On
Demand
Reserved Instances for 1 year
Spot
Instances
Spot Block
All
Upfront
Partial
Upfront
No
Upfront 1h 6h
c4.large $0.126
$0.084
(33%)
$0.086
(32%)
$0.090
(29%)
$0.029
(77%)
$0.077
(39%)
$0.098
(22%)
m4.large $0.129
$0.081
(37%)
$0.083
(36%)
$0.087
(32%)
$0.027
(79%)
$0.101
(21%)
$0.128
(0.7%)
r3.large $0.20
$0.127
(36%)
$0.130
(35%)
$0.149
(26%)
$0.031
(84%)
$0.116
(42%)
$0.147
(26%)
July 26, 2017 / ap-northeast-1 / Linux
Amazon EC2 Spot Instances

62. ECS
RDS
ElastiCache
CSV
TSV
JSON
SQS
IM-DMP
UI&API Data
Processing
S3
3rd Party
Partners
Spot Instances
Spot Fleet
ECS
IM-DMP - Elasticsearch on Spot Instances

63. Elasticsearch: Approx. 400 million IDs
v Extract IDs with
v Keyword (by browsing history)
v Segment
v User Agent
v IP address
v Geo
https://www.slideshare.net/im_docs/elasticsearch-48873206
IM-DMP - Elasticsearch on Spot Instances

64. Elasticsearch on Spot Instances
v approx. 500vCPUs for Analytics workload
Over 8vCPUs i3 Instances
IM-DMP - Elasticsearch on Spot Instances

65. Intimate Merger
Small engineering team delivers Big results!
Masafumi Osogoe & Kazuki Matsuda
Kazuki Masafumi

66. Bulbit - Adways company
http://adways.kr/https://www.adways.net

67. Bulbit - Party Track & UNICORN

68. 68
UNICORN - Full Automated Marketing Platform

69. Greatly Skilled Engineers from China J
vHailin Hu
vXiaoyi Qu
UNICORN - Full Automated Marketing Platform
ü How do you feel about
working on AdTech in
Japan?
ü What are you focusing on?
HailinXiaoyi

70. Greatly Skilled Engineers from China J
vHailin Hu
vXiaoyi Qu
UNICORN - Full Automated Marketing Platform
Itʼs like a “Gold Mine”
ü Day-by-Day Evolution
ü Achieving Goals with latest
Big Data Technologies
ü Utilize “Amazon Athena”
in a massive way!
HailinXiaoyi

71. UNICORN - Architecture

72. Auto Scaling
Up to 200
instances
Athena
Redshift
Deep Learning
on EC2
S3
UNICORN - Architecture

73. Auto Scaling
Up to 200
instances
Athena
Redshift
Deep Learning
on EC2
S3
UNICORN - Real-Time Bidding
From Ruby to Golang
“Speed is King”
in Real-Time Bidding

74. Auto Scaling
Up to 200
instances
Athena
Redshift
Deep Learning
on EC2
S3
UNICORN - Data Analysis
v Extract data for Machine Learning every 30min
v Ad-Hoc Big Data Analysis

75. Auto Scaling
Up to 200
instances
Athena
Redshift
Deep Learning
on EC2
S3
UNICORN - Machine Learning
v w/ Minimum Libraries
ü No Heavy Framework
ü As Fast As Possible!!
v Making Steady Effort
ü Plan-Do-Check-Act
ü Parameter Tuning
ü A/B Testing

76. Auto Scaling
Up to 200
instances
Athena
Redshift
Deep Learning
on EC2
S3
UNICORN - Machine Learning
For Real-Time Bidding,
Bidding servers load “Trained Models” into Memory

77. Auto Scaling
Up to 200
instances
Athena
Redshift
Deep Learning
on EC2
S3
UNICORN - Big Data Technology
v Right Technology in the Right Place
v Recently in favor with “Apache Flink”

78. AdTech use-cases in Japan

79. Let's meet up at re:Invent 2017!!
https://reinvent.awsevents.com/

80. Let's meet up at re:Invent 2017!!
https://reinvent.awsevents.com/

81. Let's meet up at re:Invent 2017!!
https://reinvent.awsevents.com/
@Kimchi Korean BBQ
http://www.kimchilasvegas.com/

82. 감사합니다!!