SlideShare a Scribd company logo
1 of 29
Download to read offline
SCALA
@
SMARTNEWS
High Performance Ad Server with Finagle	

@taketon_
Who?
• @taketon_
• Ph.D in Psychology(Vision)
• TypeSafe and Beer Lover
SmartNews
• Mobile News App, which uses machine learning algorithms

to discover and aggregate news
• 10 million DL globally
Chikyukun
Ad
• Started Ad delivery(only in Japan) from 2014/Dec
• Policy: Ads as Content
Team members
Ad Server
• Implemented within about 2 months
Finagle
• OSS RPC Framework by Twitter, Inc.
• Using Future like Akka
• Wrapper of Netty
• "Server as a function"
Finagle
• We use Finagle for Movie Ad Server
• avg. 20ms latency / Max 2000qps
Pros
• easy to build server
• High Performance (backed by Netty)
• Async Handling by Future type
• shared model by thrift IDL
• provides metrics via twitter-server and Tracer
Cons
• slightly slow to update (e.g. upgrade to Scala 2.11)
• (waiting for finagle-redis update)
• Different Behavior between twitter.util.Future and
scala.concurrent.Future (pop-quiz)
• limited documentation
• not a good fit with NewRelic (instead, use Tracer)
How to implement

Ad Server
Handling Async
• Type as a document
• Future indicates IO call
• Procedures in AdServer
• Request -> JSON parse -> Data load ->
Campaign Filtering -> Response
Server as a function
• Server : Request => Response
!
class Service[-Req, +Rep] extends (Req => Future[Rep])
Ad Service
• Launches ListeningServer(backed by Netty)
on `:port`
!
class AdService extends Service[Request, Response] {
override def apply(request: Request): Future[Response] = {
// hogehoge
}
}
val server = Http.serve(":port", new AdService())
Await.result(server)
* caution: This code needs transformation of Request to HttpRequest

(also Response)
JSON Parse
• Jackson Streaming API
• AdRequest is defined in .thrift file
!
trait JsonParser {
def read(request: Request):Option[AdRequest]
}
Data Load
• Future type indicates the function causes
heavy IO task
!
trait ContextLoader {
def load(): Future[CampaignContext]
}
Data Load
• Campaign Master Info is stored in MySQL,
and be cached on JVM memories.
• Use AtomicReference to ConcurrentHashMap
as cache
!
trait ContextLoader {
def load(): Future[CampaignContext]
}
Campaign Filtering
• Filtering campaigns based on Context and
Request
!
trait CampaignFilter {
def filter(
request: AdRequest,
ctx: CampaignContext
): Option[AdResponse]
}
Future Chain
• add callback to the function with flatMap
• callback will receive the result of former
Future function, then returns new Future
!
def flatMap[B](f: A => Future[B]): Future[B]
Future Chain (flatMap)
!
def apply(request: Request): Future[Response] = {
val adReqOpt: Option[AdRequest] = JSONParser.read(request)
val fAdResOpt: Future[Option[AdResponse]] = adReqOpt.map { adReq =>
val fCtx: Future[CampaignContext] = ContextLoader.load()
fCtx.map { ctx =>
CampaignFilter.filter(adReq, ctx)
}
}.getOrElse {
Future.value(None)
}
!
Future Chain (flatMap)
!
fAdResOpt.flatMap { adResOpt =>
adResOpt.map { adRes =>
val fSessionKeyOpt = RedisService.getKey()
fSessionKeyOpt.flatMap { sessionKeyOpt =>
sessionKeyOpt.map { sessionKey =>
Future.value(mkResponse(adRes, sessionKey))
}.getOrElse {
Future.value(Response(NO_CONTENT))
}
}
}.getOrElse {
Future.value(Response(NO_CONTENT))
}
}
}
Future Chain (optionT)
• optionT[A]: Future[Option[A]] => OptionT[Future, A]
• need to make Future as instance of Monad (gist)
!
def apply(request: Request): Future[Response] = {
(for {
adReq <- optionT(JSONParser.read(request).point[Future])
ctx <- LocalCache.load().liftM[OptionT]
adRes <- optionT(CampaignFilter.filter(adReq, ctx).point[Future])
sessionKey <- optionT(RedisService.getKey())
} yield mkResponse(adRes, sessionKey)).run.map {
_.getOrElse(Response(NO_CONTENT))
}
}
!
Filter
• Set filter for application-agnostic behavior, like
timeout
class HttpTimeoutFilter (
val timeOutMs: Int
) extends TimeoutFilter[Request, Response](
timeOutMs.milliseconds,
new GlobalRequestTimeoutException(timeOutMs.milliseconds),
DefaultTimer.twitter){
}
!
Http.serve(":port", new HttpTimeoutFilter(100) andThen AdService)
!
Routing
val muxer = new HttpMuxer()
.withHandler("/ad", new AdService())
!
val server = Http.server(":port", muxer)
Await.result(server)
Tips for Performance
• Basically the server will not be CPU-bound
• Wrap IO-bound process by Future
Tips for Performance
• The cost for creating instances for Future or
Option can be ignored
• For constructing List, use mutable data structure
(e.g. ArrayBuffer) and finally call toList
• For transforming List, use Iterator and finally call
toList
• Practice for Performance tuning in Scala (Japanese)
Summary
• Finagle will be a good fit for Ad Server
• We used mutable for data construction
• However, no need for sensitive for tuning
• (́-`).。oO(Hope Finagle will become more popular)

More Related Content

What's hot

A unified analytics platform with Kafka and Flink | Stephan Ewen, Ververica
A unified analytics platform with Kafka and Flink | Stephan Ewen, VervericaA unified analytics platform with Kafka and Flink | Stephan Ewen, Ververica
A unified analytics platform with Kafka and Flink | Stephan Ewen, VervericaHostedbyConfluent
 
I/O intensiveなKafka ConsumerアプリケーションのスループットをLINE Ads Platformではどのように改善したか
I/O intensiveなKafka ConsumerアプリケーションのスループットをLINE Ads Platformではどのように改善したかI/O intensiveなKafka ConsumerアプリケーションのスループットをLINE Ads Platformではどのように改善したか
I/O intensiveなKafka ConsumerアプリケーションのスループットをLINE Ads Platformではどのように改善したかLINE Corporation
 
Managing Your Cloud Spend With PlanForCloud - RightScale Compute 2013
Managing Your Cloud Spend With PlanForCloud - RightScale Compute 2013Managing Your Cloud Spend With PlanForCloud - RightScale Compute 2013
Managing Your Cloud Spend With PlanForCloud - RightScale Compute 2013RightScale
 
Data / Streaming / Microservices Platform with Devops
Data / Streaming / Microservices Platform with DevopsData / Streaming / Microservices Platform with Devops
Data / Streaming / Microservices Platform with DevopsKidong Lee
 
Santa Cloud: How Netflix Does Holiday Capacity Planning - South Bay SRE Meetu...
Santa Cloud: How Netflix Does Holiday Capacity Planning - South Bay SRE Meetu...Santa Cloud: How Netflix Does Holiday Capacity Planning - South Bay SRE Meetu...
Santa Cloud: How Netflix Does Holiday Capacity Planning - South Bay SRE Meetu...Coburn Watson
 
Windows Azure Zero Downtime Upgrade
Windows Azure Zero Downtime UpgradeWindows Azure Zero Downtime Upgrade
Windows Azure Zero Downtime UpgradePavel Revenkov
 
Kafka and Stream Processing, Taking Analytics Real-time, Mike Spicer
Kafka and Stream Processing, Taking Analytics Real-time, Mike SpicerKafka and Stream Processing, Taking Analytics Real-time, Mike Spicer
Kafka and Stream Processing, Taking Analytics Real-time, Mike Spicerconfluent
 
How Netflix Uses Amazon Kinesis Streams to Monitor and Optimize Large-scale N...
How Netflix Uses Amazon Kinesis Streams to Monitor and Optimize Large-scale N...How Netflix Uses Amazon Kinesis Streams to Monitor and Optimize Large-scale N...
How Netflix Uses Amazon Kinesis Streams to Monitor and Optimize Large-scale N...Amazon Web Services
 
Kafka Case by getting data from SAP
Kafka Case by getting data from SAPKafka Case by getting data from SAP
Kafka Case by getting data from SAPTechrom Tecnologia
 
Autoscale without netscaler_ccceu13
Autoscale without netscaler_ccceu13Autoscale without netscaler_ccceu13
Autoscale without netscaler_ccceu13Nguyen Anh Tu
 
Serverless Reality
Serverless RealityServerless Reality
Serverless RealityLynn Langit
 
Administrative techniques to reduce Kafka costs | Anna Kepler, Viasat
Administrative techniques to reduce Kafka costs | Anna Kepler, ViasatAdministrative techniques to reduce Kafka costs | Anna Kepler, Viasat
Administrative techniques to reduce Kafka costs | Anna Kepler, ViasatHostedbyConfluent
 
Comparing Cloud VM Types and Prices: AWS vs Azure vs Google vs IBM
Comparing Cloud VM Types and Prices: AWS vs Azure vs Google vs IBMComparing Cloud VM Types and Prices: AWS vs Azure vs Google vs IBM
Comparing Cloud VM Types and Prices: AWS vs Azure vs Google vs IBMRightScale
 
Scaling Galaxy on Google Cloud Platform
Scaling Galaxy on Google Cloud PlatformScaling Galaxy on Google Cloud Platform
Scaling Galaxy on Google Cloud PlatformLynn Langit
 
Shift Remote: WEB - GraphQL and React – Quick Start - Dubravko Bogovic (Infobip)
Shift Remote: WEB - GraphQL and React – Quick Start - Dubravko Bogovic (Infobip)Shift Remote: WEB - GraphQL and React – Quick Start - Dubravko Bogovic (Infobip)
Shift Remote: WEB - GraphQL and React – Quick Start - Dubravko Bogovic (Infobip)Shift Conference
 
Visualizing AutoTrader Traffic in Near Real-Time with Spark Streaming-(Jon Gr...
Visualizing AutoTrader Traffic in Near Real-Time with Spark Streaming-(Jon Gr...Visualizing AutoTrader Traffic in Near Real-Time with Spark Streaming-(Jon Gr...
Visualizing AutoTrader Traffic in Near Real-Time with Spark Streaming-(Jon Gr...Spark Summit
 
Михаил Максимов ( Software engineer, DataArt. AWS certified Solution Architect)
Михаил Максимов ( Software engineer, DataArt. AWS certified Solution Architect)Михаил Максимов ( Software engineer, DataArt. AWS certified Solution Architect)
Михаил Максимов ( Software engineer, DataArt. AWS certified Solution Architect)DataArt
 
GraphQL API on a Serverless Environment
GraphQL API on a Serverless EnvironmentGraphQL API on a Serverless Environment
GraphQL API on a Serverless EnvironmentItai Yaffe
 
Grafana optimization for Prometheus
Grafana optimization for PrometheusGrafana optimization for Prometheus
Grafana optimization for PrometheusMitsuhiro Tanda
 
9 Ways to Reduce Cloud Storage Costs
9 Ways to Reduce Cloud Storage Costs9 Ways to Reduce Cloud Storage Costs
9 Ways to Reduce Cloud Storage CostsRightScale
 

What's hot (20)

A unified analytics platform with Kafka and Flink | Stephan Ewen, Ververica
A unified analytics platform with Kafka and Flink | Stephan Ewen, VervericaA unified analytics platform with Kafka and Flink | Stephan Ewen, Ververica
A unified analytics platform with Kafka and Flink | Stephan Ewen, Ververica
 
I/O intensiveなKafka ConsumerアプリケーションのスループットをLINE Ads Platformではどのように改善したか
I/O intensiveなKafka ConsumerアプリケーションのスループットをLINE Ads Platformではどのように改善したかI/O intensiveなKafka ConsumerアプリケーションのスループットをLINE Ads Platformではどのように改善したか
I/O intensiveなKafka ConsumerアプリケーションのスループットをLINE Ads Platformではどのように改善したか
 
Managing Your Cloud Spend With PlanForCloud - RightScale Compute 2013
Managing Your Cloud Spend With PlanForCloud - RightScale Compute 2013Managing Your Cloud Spend With PlanForCloud - RightScale Compute 2013
Managing Your Cloud Spend With PlanForCloud - RightScale Compute 2013
 
Data / Streaming / Microservices Platform with Devops
Data / Streaming / Microservices Platform with DevopsData / Streaming / Microservices Platform with Devops
Data / Streaming / Microservices Platform with Devops
 
Santa Cloud: How Netflix Does Holiday Capacity Planning - South Bay SRE Meetu...
Santa Cloud: How Netflix Does Holiday Capacity Planning - South Bay SRE Meetu...Santa Cloud: How Netflix Does Holiday Capacity Planning - South Bay SRE Meetu...
Santa Cloud: How Netflix Does Holiday Capacity Planning - South Bay SRE Meetu...
 
Windows Azure Zero Downtime Upgrade
Windows Azure Zero Downtime UpgradeWindows Azure Zero Downtime Upgrade
Windows Azure Zero Downtime Upgrade
 
Kafka and Stream Processing, Taking Analytics Real-time, Mike Spicer
Kafka and Stream Processing, Taking Analytics Real-time, Mike SpicerKafka and Stream Processing, Taking Analytics Real-time, Mike Spicer
Kafka and Stream Processing, Taking Analytics Real-time, Mike Spicer
 
How Netflix Uses Amazon Kinesis Streams to Monitor and Optimize Large-scale N...
How Netflix Uses Amazon Kinesis Streams to Monitor and Optimize Large-scale N...How Netflix Uses Amazon Kinesis Streams to Monitor and Optimize Large-scale N...
How Netflix Uses Amazon Kinesis Streams to Monitor and Optimize Large-scale N...
 
Kafka Case by getting data from SAP
Kafka Case by getting data from SAPKafka Case by getting data from SAP
Kafka Case by getting data from SAP
 
Autoscale without netscaler_ccceu13
Autoscale without netscaler_ccceu13Autoscale without netscaler_ccceu13
Autoscale without netscaler_ccceu13
 
Serverless Reality
Serverless RealityServerless Reality
Serverless Reality
 
Administrative techniques to reduce Kafka costs | Anna Kepler, Viasat
Administrative techniques to reduce Kafka costs | Anna Kepler, ViasatAdministrative techniques to reduce Kafka costs | Anna Kepler, Viasat
Administrative techniques to reduce Kafka costs | Anna Kepler, Viasat
 
Comparing Cloud VM Types and Prices: AWS vs Azure vs Google vs IBM
Comparing Cloud VM Types and Prices: AWS vs Azure vs Google vs IBMComparing Cloud VM Types and Prices: AWS vs Azure vs Google vs IBM
Comparing Cloud VM Types and Prices: AWS vs Azure vs Google vs IBM
 
Scaling Galaxy on Google Cloud Platform
Scaling Galaxy on Google Cloud PlatformScaling Galaxy on Google Cloud Platform
Scaling Galaxy on Google Cloud Platform
 
Shift Remote: WEB - GraphQL and React – Quick Start - Dubravko Bogovic (Infobip)
Shift Remote: WEB - GraphQL and React – Quick Start - Dubravko Bogovic (Infobip)Shift Remote: WEB - GraphQL and React – Quick Start - Dubravko Bogovic (Infobip)
Shift Remote: WEB - GraphQL and React – Quick Start - Dubravko Bogovic (Infobip)
 
Visualizing AutoTrader Traffic in Near Real-Time with Spark Streaming-(Jon Gr...
Visualizing AutoTrader Traffic in Near Real-Time with Spark Streaming-(Jon Gr...Visualizing AutoTrader Traffic in Near Real-Time with Spark Streaming-(Jon Gr...
Visualizing AutoTrader Traffic in Near Real-Time with Spark Streaming-(Jon Gr...
 
Михаил Максимов ( Software engineer, DataArt. AWS certified Solution Architect)
Михаил Максимов ( Software engineer, DataArt. AWS certified Solution Architect)Михаил Максимов ( Software engineer, DataArt. AWS certified Solution Architect)
Михаил Максимов ( Software engineer, DataArt. AWS certified Solution Architect)
 
GraphQL API on a Serverless Environment
GraphQL API on a Serverless EnvironmentGraphQL API on a Serverless Environment
GraphQL API on a Serverless Environment
 
Grafana optimization for Prometheus
Grafana optimization for PrometheusGrafana optimization for Prometheus
Grafana optimization for Prometheus
 
9 Ways to Reduce Cloud Storage Costs
9 Ways to Reduce Cloud Storage Costs9 Ways to Reduce Cloud Storage Costs
9 Ways to Reduce Cloud Storage Costs
 

Similar to Scala(finagle)@SmartNews_English

Event-Based API Patterns and Practices
Event-Based API Patterns and PracticesEvent-Based API Patterns and Practices
Event-Based API Patterns and PracticesLaunchAny
 
Actors or Not: Async Event Architectures
Actors or Not: Async Event ArchitecturesActors or Not: Async Event Architectures
Actors or Not: Async Event ArchitecturesYaroslav Tkachenko
 
An approach to responsive, realtime with Backbone.js and WebSockets
An approach to responsive, realtime with Backbone.js and WebSocketsAn approach to responsive, realtime with Backbone.js and WebSockets
An approach to responsive, realtime with Backbone.js and WebSocketsAndrei Sebastian Cîmpean
 
Timeline Service v.2 (Hadoop Summit 2016)
Timeline Service v.2 (Hadoop Summit 2016)Timeline Service v.2 (Hadoop Summit 2016)
Timeline Service v.2 (Hadoop Summit 2016)Sangjin Lee
 
Timeline service V2 at the Hadoop Summit SJ 2016
Timeline service V2 at the Hadoop Summit SJ 2016Timeline service V2 at the Hadoop Summit SJ 2016
Timeline service V2 at the Hadoop Summit SJ 2016Vrushali Channapattan
 
Kafka streams decoupling with stores
Kafka streams decoupling with storesKafka streams decoupling with stores
Kafka streams decoupling with storesYoni Farin
 
Building Scalable Stateless Applications with RxJava
Building Scalable Stateless Applications with RxJavaBuilding Scalable Stateless Applications with RxJava
Building Scalable Stateless Applications with RxJavaRick Warren
 
Deep learning and streaming in Apache Spark 2.2 by Matei Zaharia
Deep learning and streaming in Apache Spark 2.2 by Matei ZahariaDeep learning and streaming in Apache Spark 2.2 by Matei Zaharia
Deep learning and streaming in Apache Spark 2.2 by Matei ZahariaGoDataDriven
 
Predictable reactive state management - ngrx
Predictable reactive state management - ngrxPredictable reactive state management - ngrx
Predictable reactive state management - ngrxIlia Idakiev
 
M2M Protocol Interoperability using IoT Toolkit
M2M Protocol Interoperability using IoT ToolkitM2M Protocol Interoperability using IoT Toolkit
M2M Protocol Interoperability using IoT ToolkitMichael Koster
 
M2M Protocol Interoperability using IoT Toolkit
M2M Protocol Interoperability using IoT ToolkitM2M Protocol Interoperability using IoT Toolkit
M2M Protocol Interoperability using IoT ToolkitMichael Koster
 
Object models for interoperability
Object models for interoperabilityObject models for interoperability
Object models for interoperabilityMichael Koster
 
Developing Microservices using Spring - Beginner's Guide
Developing Microservices using Spring - Beginner's GuideDeveloping Microservices using Spring - Beginner's Guide
Developing Microservices using Spring - Beginner's GuideMohanraj Thirumoorthy
 
Integration Monday - BizTalk Migrator Deep Dive
Integration Monday - BizTalk Migrator Deep DiveIntegration Monday - BizTalk Migrator Deep Dive
Integration Monday - BizTalk Migrator Deep DiveBizTalk360
 
RichFaces: rich:* component library
RichFaces: rich:* component libraryRichFaces: rich:* component library
RichFaces: rich:* component libraryMax Katz
 

Similar to Scala(finagle)@SmartNews_English (20)

KFServing and Feast
KFServing and FeastKFServing and Feast
KFServing and Feast
 
Event-Based API Patterns and Practices
Event-Based API Patterns and PracticesEvent-Based API Patterns and Practices
Event-Based API Patterns and Practices
 
Actors or Not: Async Event Architectures
Actors or Not: Async Event ArchitecturesActors or Not: Async Event Architectures
Actors or Not: Async Event Architectures
 
An approach to responsive, realtime with Backbone.js and WebSockets
An approach to responsive, realtime with Backbone.js and WebSocketsAn approach to responsive, realtime with Backbone.js and WebSockets
An approach to responsive, realtime with Backbone.js and WebSockets
 
Timeline Service v.2 (Hadoop Summit 2016)
Timeline Service v.2 (Hadoop Summit 2016)Timeline Service v.2 (Hadoop Summit 2016)
Timeline Service v.2 (Hadoop Summit 2016)
 
Timeline service V2 at the Hadoop Summit SJ 2016
Timeline service V2 at the Hadoop Summit SJ 2016Timeline service V2 at the Hadoop Summit SJ 2016
Timeline service V2 at the Hadoop Summit SJ 2016
 
Ajax
AjaxAjax
Ajax
 
Angular or React
Angular or ReactAngular or React
Angular or React
 
Kafka streams decoupling with stores
Kafka streams decoupling with storesKafka streams decoupling with stores
Kafka streams decoupling with stores
 
Building Scalable Stateless Applications with RxJava
Building Scalable Stateless Applications with RxJavaBuilding Scalable Stateless Applications with RxJava
Building Scalable Stateless Applications with RxJava
 
Deep learning and streaming in Apache Spark 2.2 by Matei Zaharia
Deep learning and streaming in Apache Spark 2.2 by Matei ZahariaDeep learning and streaming in Apache Spark 2.2 by Matei Zaharia
Deep learning and streaming in Apache Spark 2.2 by Matei Zaharia
 
Ajax
AjaxAjax
Ajax
 
Data Pipeline at Tapad
Data Pipeline at TapadData Pipeline at Tapad
Data Pipeline at Tapad
 
Predictable reactive state management - ngrx
Predictable reactive state management - ngrxPredictable reactive state management - ngrx
Predictable reactive state management - ngrx
 
M2M Protocol Interoperability using IoT Toolkit
M2M Protocol Interoperability using IoT ToolkitM2M Protocol Interoperability using IoT Toolkit
M2M Protocol Interoperability using IoT Toolkit
 
M2M Protocol Interoperability using IoT Toolkit
M2M Protocol Interoperability using IoT ToolkitM2M Protocol Interoperability using IoT Toolkit
M2M Protocol Interoperability using IoT Toolkit
 
Object models for interoperability
Object models for interoperabilityObject models for interoperability
Object models for interoperability
 
Developing Microservices using Spring - Beginner's Guide
Developing Microservices using Spring - Beginner's GuideDeveloping Microservices using Spring - Beginner's Guide
Developing Microservices using Spring - Beginner's Guide
 
Integration Monday - BizTalk Migrator Deep Dive
Integration Monday - BizTalk Migrator Deep DiveIntegration Monday - BizTalk Migrator Deep Dive
Integration Monday - BizTalk Migrator Deep Dive
 
RichFaces: rich:* component library
RichFaces: rich:* component libraryRichFaces: rich:* component library
RichFaces: rich:* component library
 

More from Shigekazu Takei

More from Shigekazu Takei (7)

Scala@SmartNews_20150221
Scala@SmartNews_20150221Scala@SmartNews_20150221
Scala@SmartNews_20150221
 
Startprintf_2013May18
Startprintf_2013May18Startprintf_2013May18
Startprintf_2013May18
 
Pfds 9 2_2
Pfds 9 2_2Pfds 9 2_2
Pfds 9 2_2
 
Pfds 5 2+6_4_2
Pfds 5 2+6_4_2Pfds 5 2+6_4_2
Pfds 5 2+6_4_2
 
Ids ips
Ids ipsIds ips
Ids ips
 
Stanford ml neuralnetwork
Stanford ml neuralnetworkStanford ml neuralnetwork
Stanford ml neuralnetwork
 
Pfds ex3 9
Pfds ex3 9Pfds ex3 9
Pfds ex3 9
 

Recently uploaded

Bitdefender-CSG-Report-creat7534-interactive
Bitdefender-CSG-Report-creat7534-interactiveBitdefender-CSG-Report-creat7534-interactive
Bitdefender-CSG-Report-creat7534-interactivestartupro
 
Automation Ops Series: Session 3 - Solutions management
Automation Ops Series: Session 3 - Solutions managementAutomation Ops Series: Session 3 - Solutions management
Automation Ops Series: Session 3 - Solutions managementDianaGray10
 
Laying the Data Foundations for Artificial Intelligence!
Laying the Data Foundations for Artificial Intelligence!Laying the Data Foundations for Artificial Intelligence!
Laying the Data Foundations for Artificial Intelligence!Memoori
 
Transcript: Green paths: Learning from publishers’ sustainability journeys - ...
Transcript: Green paths: Learning from publishers’ sustainability journeys - ...Transcript: Green paths: Learning from publishers’ sustainability journeys - ...
Transcript: Green paths: Learning from publishers’ sustainability journeys - ...BookNet Canada
 
Landscape Catalogue 2024 Australia-1.pdf
Landscape Catalogue 2024 Australia-1.pdfLandscape Catalogue 2024 Australia-1.pdf
Landscape Catalogue 2024 Australia-1.pdfAarwolf Industries LLC
 
Genislab builds better products and faster go-to-market with Lean project man...
Genislab builds better products and faster go-to-market with Lean project man...Genislab builds better products and faster go-to-market with Lean project man...
Genislab builds better products and faster go-to-market with Lean project man...Farhan Tariq
 
Dublin_mulesoft_meetup_API_specifications.pptx
Dublin_mulesoft_meetup_API_specifications.pptxDublin_mulesoft_meetup_API_specifications.pptx
Dublin_mulesoft_meetup_API_specifications.pptxKunal Gupta
 
React Native vs Ionic - The Best Mobile App Framework
React Native vs Ionic - The Best Mobile App FrameworkReact Native vs Ionic - The Best Mobile App Framework
React Native vs Ionic - The Best Mobile App FrameworkPixlogix Infotech
 
Infrared simulation and processing on Nvidia platforms
Infrared simulation and processing on Nvidia platformsInfrared simulation and processing on Nvidia platforms
Infrared simulation and processing on Nvidia platformsYoss Cohen
 
Accelerating Enterprise Software Engineering with Platformless
Accelerating Enterprise Software Engineering with PlatformlessAccelerating Enterprise Software Engineering with Platformless
Accelerating Enterprise Software Engineering with PlatformlessWSO2
 
Digital Tools & AI in Career Development
Digital Tools & AI in Career DevelopmentDigital Tools & AI in Career Development
Digital Tools & AI in Career DevelopmentMahmoud Rabie
 
WomenInAutomation2024: AI and Automation for eveyone
WomenInAutomation2024: AI and Automation for eveyoneWomenInAutomation2024: AI and Automation for eveyone
WomenInAutomation2024: AI and Automation for eveyoneUiPathCommunity
 
Introduction-to-Wazuh-and-its-integration.pptx
Introduction-to-Wazuh-and-its-integration.pptxIntroduction-to-Wazuh-and-its-integration.pptx
Introduction-to-Wazuh-and-its-integration.pptxmprakaash5
 
Abdul Kader Baba- Managing Cybersecurity Risks and Compliance Requirements i...
Abdul Kader Baba- Managing Cybersecurity Risks  and Compliance Requirements i...Abdul Kader Baba- Managing Cybersecurity Risks  and Compliance Requirements i...
Abdul Kader Baba- Managing Cybersecurity Risks and Compliance Requirements i...itnewsafrica
 
Connecting the Dots for Information Discovery.pdf
Connecting the Dots for Information Discovery.pdfConnecting the Dots for Information Discovery.pdf
Connecting the Dots for Information Discovery.pdfNeo4j
 
Green paths: Learning from publishers’ sustainability journeys - Tech Forum 2024
Green paths: Learning from publishers’ sustainability journeys - Tech Forum 2024Green paths: Learning from publishers’ sustainability journeys - Tech Forum 2024
Green paths: Learning from publishers’ sustainability journeys - Tech Forum 2024BookNet Canada
 
Potential of AI (Generative AI) in Business: Learnings and Insights
Potential of AI (Generative AI) in Business: Learnings and InsightsPotential of AI (Generative AI) in Business: Learnings and Insights
Potential of AI (Generative AI) in Business: Learnings and InsightsRavi Sanghani
 
Decarbonising Buildings: Making a net-zero built environment a reality
Decarbonising Buildings: Making a net-zero built environment a realityDecarbonising Buildings: Making a net-zero built environment a reality
Decarbonising Buildings: Making a net-zero built environment a realityIES VE
 
Bridging Between CAD & GIS: 6 Ways to Automate Your Data Integration
Bridging Between CAD & GIS:  6 Ways to Automate Your Data IntegrationBridging Between CAD & GIS:  6 Ways to Automate Your Data Integration
Bridging Between CAD & GIS: 6 Ways to Automate Your Data Integrationmarketing932765
 
A Glance At The Java Performance Toolbox
A Glance At The Java Performance ToolboxA Glance At The Java Performance Toolbox
A Glance At The Java Performance ToolboxAna-Maria Mihalceanu
 

Recently uploaded (20)

Bitdefender-CSG-Report-creat7534-interactive
Bitdefender-CSG-Report-creat7534-interactiveBitdefender-CSG-Report-creat7534-interactive
Bitdefender-CSG-Report-creat7534-interactive
 
Automation Ops Series: Session 3 - Solutions management
Automation Ops Series: Session 3 - Solutions managementAutomation Ops Series: Session 3 - Solutions management
Automation Ops Series: Session 3 - Solutions management
 
Laying the Data Foundations for Artificial Intelligence!
Laying the Data Foundations for Artificial Intelligence!Laying the Data Foundations for Artificial Intelligence!
Laying the Data Foundations for Artificial Intelligence!
 
Transcript: Green paths: Learning from publishers’ sustainability journeys - ...
Transcript: Green paths: Learning from publishers’ sustainability journeys - ...Transcript: Green paths: Learning from publishers’ sustainability journeys - ...
Transcript: Green paths: Learning from publishers’ sustainability journeys - ...
 
Landscape Catalogue 2024 Australia-1.pdf
Landscape Catalogue 2024 Australia-1.pdfLandscape Catalogue 2024 Australia-1.pdf
Landscape Catalogue 2024 Australia-1.pdf
 
Genislab builds better products and faster go-to-market with Lean project man...
Genislab builds better products and faster go-to-market with Lean project man...Genislab builds better products and faster go-to-market with Lean project man...
Genislab builds better products and faster go-to-market with Lean project man...
 
Dublin_mulesoft_meetup_API_specifications.pptx
Dublin_mulesoft_meetup_API_specifications.pptxDublin_mulesoft_meetup_API_specifications.pptx
Dublin_mulesoft_meetup_API_specifications.pptx
 
React Native vs Ionic - The Best Mobile App Framework
React Native vs Ionic - The Best Mobile App FrameworkReact Native vs Ionic - The Best Mobile App Framework
React Native vs Ionic - The Best Mobile App Framework
 
Infrared simulation and processing on Nvidia platforms
Infrared simulation and processing on Nvidia platformsInfrared simulation and processing on Nvidia platforms
Infrared simulation and processing on Nvidia platforms
 
Accelerating Enterprise Software Engineering with Platformless
Accelerating Enterprise Software Engineering with PlatformlessAccelerating Enterprise Software Engineering with Platformless
Accelerating Enterprise Software Engineering with Platformless
 
Digital Tools & AI in Career Development
Digital Tools & AI in Career DevelopmentDigital Tools & AI in Career Development
Digital Tools & AI in Career Development
 
WomenInAutomation2024: AI and Automation for eveyone
WomenInAutomation2024: AI and Automation for eveyoneWomenInAutomation2024: AI and Automation for eveyone
WomenInAutomation2024: AI and Automation for eveyone
 
Introduction-to-Wazuh-and-its-integration.pptx
Introduction-to-Wazuh-and-its-integration.pptxIntroduction-to-Wazuh-and-its-integration.pptx
Introduction-to-Wazuh-and-its-integration.pptx
 
Abdul Kader Baba- Managing Cybersecurity Risks and Compliance Requirements i...
Abdul Kader Baba- Managing Cybersecurity Risks  and Compliance Requirements i...Abdul Kader Baba- Managing Cybersecurity Risks  and Compliance Requirements i...
Abdul Kader Baba- Managing Cybersecurity Risks and Compliance Requirements i...
 
Connecting the Dots for Information Discovery.pdf
Connecting the Dots for Information Discovery.pdfConnecting the Dots for Information Discovery.pdf
Connecting the Dots for Information Discovery.pdf
 
Green paths: Learning from publishers’ sustainability journeys - Tech Forum 2024
Green paths: Learning from publishers’ sustainability journeys - Tech Forum 2024Green paths: Learning from publishers’ sustainability journeys - Tech Forum 2024
Green paths: Learning from publishers’ sustainability journeys - Tech Forum 2024
 
Potential of AI (Generative AI) in Business: Learnings and Insights
Potential of AI (Generative AI) in Business: Learnings and InsightsPotential of AI (Generative AI) in Business: Learnings and Insights
Potential of AI (Generative AI) in Business: Learnings and Insights
 
Decarbonising Buildings: Making a net-zero built environment a reality
Decarbonising Buildings: Making a net-zero built environment a realityDecarbonising Buildings: Making a net-zero built environment a reality
Decarbonising Buildings: Making a net-zero built environment a reality
 
Bridging Between CAD & GIS: 6 Ways to Automate Your Data Integration
Bridging Between CAD & GIS:  6 Ways to Automate Your Data IntegrationBridging Between CAD & GIS:  6 Ways to Automate Your Data Integration
Bridging Between CAD & GIS: 6 Ways to Automate Your Data Integration
 
A Glance At The Java Performance Toolbox
A Glance At The Java Performance ToolboxA Glance At The Java Performance Toolbox
A Glance At The Java Performance Toolbox
 

Scala(finagle)@SmartNews_English

  • 1. SCALA @ SMARTNEWS High Performance Ad Server with Finagle @taketon_
  • 2. Who? • @taketon_ • Ph.D in Psychology(Vision) • TypeSafe and Beer Lover
  • 3. SmartNews • Mobile News App, which uses machine learning algorithms
 to discover and aggregate news • 10 million DL globally
  • 5. Ad • Started Ad delivery(only in Japan) from 2014/Dec • Policy: Ads as Content
  • 7. Ad Server • Implemented within about 2 months
  • 8.
  • 9. Finagle • OSS RPC Framework by Twitter, Inc. • Using Future like Akka • Wrapper of Netty • "Server as a function"
  • 10. Finagle • We use Finagle for Movie Ad Server • avg. 20ms latency / Max 2000qps
  • 11. Pros • easy to build server • High Performance (backed by Netty) • Async Handling by Future type • shared model by thrift IDL • provides metrics via twitter-server and Tracer
  • 12. Cons • slightly slow to update (e.g. upgrade to Scala 2.11) • (waiting for finagle-redis update) • Different Behavior between twitter.util.Future and scala.concurrent.Future (pop-quiz) • limited documentation • not a good fit with NewRelic (instead, use Tracer)
  • 14. Handling Async • Type as a document • Future indicates IO call • Procedures in AdServer • Request -> JSON parse -> Data load -> Campaign Filtering -> Response
  • 15. Server as a function • Server : Request => Response ! class Service[-Req, +Rep] extends (Req => Future[Rep])
  • 16. Ad Service • Launches ListeningServer(backed by Netty) on `:port` ! class AdService extends Service[Request, Response] { override def apply(request: Request): Future[Response] = { // hogehoge } } val server = Http.serve(":port", new AdService()) Await.result(server) * caution: This code needs transformation of Request to HttpRequest
 (also Response)
  • 17. JSON Parse • Jackson Streaming API • AdRequest is defined in .thrift file ! trait JsonParser { def read(request: Request):Option[AdRequest] }
  • 18. Data Load • Future type indicates the function causes heavy IO task ! trait ContextLoader { def load(): Future[CampaignContext] }
  • 19. Data Load • Campaign Master Info is stored in MySQL, and be cached on JVM memories. • Use AtomicReference to ConcurrentHashMap as cache ! trait ContextLoader { def load(): Future[CampaignContext] }
  • 20. Campaign Filtering • Filtering campaigns based on Context and Request ! trait CampaignFilter { def filter( request: AdRequest, ctx: CampaignContext ): Option[AdResponse] }
  • 21. Future Chain • add callback to the function with flatMap • callback will receive the result of former Future function, then returns new Future ! def flatMap[B](f: A => Future[B]): Future[B]
  • 22. Future Chain (flatMap) ! def apply(request: Request): Future[Response] = { val adReqOpt: Option[AdRequest] = JSONParser.read(request) val fAdResOpt: Future[Option[AdResponse]] = adReqOpt.map { adReq => val fCtx: Future[CampaignContext] = ContextLoader.load() fCtx.map { ctx => CampaignFilter.filter(adReq, ctx) } }.getOrElse { Future.value(None) } !
  • 23. Future Chain (flatMap) ! fAdResOpt.flatMap { adResOpt => adResOpt.map { adRes => val fSessionKeyOpt = RedisService.getKey() fSessionKeyOpt.flatMap { sessionKeyOpt => sessionKeyOpt.map { sessionKey => Future.value(mkResponse(adRes, sessionKey)) }.getOrElse { Future.value(Response(NO_CONTENT)) } } }.getOrElse { Future.value(Response(NO_CONTENT)) } } }
  • 24. Future Chain (optionT) • optionT[A]: Future[Option[A]] => OptionT[Future, A] • need to make Future as instance of Monad (gist) ! def apply(request: Request): Future[Response] = { (for { adReq <- optionT(JSONParser.read(request).point[Future]) ctx <- LocalCache.load().liftM[OptionT] adRes <- optionT(CampaignFilter.filter(adReq, ctx).point[Future]) sessionKey <- optionT(RedisService.getKey()) } yield mkResponse(adRes, sessionKey)).run.map { _.getOrElse(Response(NO_CONTENT)) } } !
  • 25. Filter • Set filter for application-agnostic behavior, like timeout class HttpTimeoutFilter ( val timeOutMs: Int ) extends TimeoutFilter[Request, Response]( timeOutMs.milliseconds, new GlobalRequestTimeoutException(timeOutMs.milliseconds), DefaultTimer.twitter){ } ! Http.serve(":port", new HttpTimeoutFilter(100) andThen AdService) !
  • 26. Routing val muxer = new HttpMuxer() .withHandler("/ad", new AdService()) ! val server = Http.server(":port", muxer) Await.result(server)
  • 27. Tips for Performance • Basically the server will not be CPU-bound • Wrap IO-bound process by Future
  • 28. Tips for Performance • The cost for creating instances for Future or Option can be ignored • For constructing List, use mutable data structure (e.g. ArrayBuffer) and finally call toList • For transforming List, use Iterator and finally call toList • Practice for Performance tuning in Scala (Japanese)
  • 29. Summary • Finagle will be a good fit for Ad Server • We used mutable for data construction • However, no need for sensitive for tuning • (́-`).。oO(Hope Finagle will become more popular)