Using WSO2 API Manager to deliver dynamic display advertising
Case Study: Using WSO2 API
Manager to deliver dynamic
Senior Software Engineer
Chuck Megivern is a Senior Software Engineer for MyWebGrocer, Inc.
He is the company technical lead for open source technologies. Chuck has been
developing advertising applications for MyWebGrocer and before that, Dealer.com,
for the last five years.
He has delivered Display Advertising, SEM, SEO and Digital Video applications and
APIs. He has also delivered multiple cloud based server deployments.
Earlier in his career, Chuck developed software for IBM and also had a number of
engineering assignments there.
Chuck has a BSEE degree from Lehigh University, and an MSEE degree from
About the author
• MyWebGrocer (MWG) was founded in 1999.
• We serve the grocery industry and consumer packaged goods industry.
• We provide an online grocery platform used by leading retailers
encompassing over 10,000 retail locations in the US.
• Expanding in Asia Pacific and Europe.
• We also have extensive eCommerce and eMarketing solutions.
• We provide digital display advertising on our own ad network, as well as on
public internet sites that participate in other online ad networks, such as
• My area provides APIs for internal company use, and for external business
• Our case study involves using WSO2 API Manager to enrich display advertising served to
consumers via the Google DoubleClick ad network.
• The ad network delivers an Adobe Flash based Rich Media ad to consumer’s desktops while
they are browsing internet sites.
• The ad campaign lasts for a number of weeks. The Rich Media ad, using Action Script,
calls a MWG API and dynamically pulls in data every time the ad is displayed.
• The API delivers product information that updates daily. That data
• comes from the retailer’s store that is closest to the consumer, and
• is currently advertising the products of interest.
• Most consumer advertising application use cases involve responding to a consumers 'click'
• However, this high traffic use case invokes the API every time that a display ad is
presented to a consumer.
• This presentation discusses our technical approach and application stack used to handle
this high volume.
Our case study
WSO2 Product Usage
• Case study usage
• API Manager 1.5
• Business Activity Monitor (BAM) 2.4.0
• Current Usage
• API Manager 1.6
• Business Activity Monitor (BAM) 2.4.1
Types of APIs served
• Case study: very high volume consumer browser API invokations.
• At the same time, this distributed API manager installation serves:
• external business partners using MWG API’s
• internal MWG servers invoking MWG API’s
• Heterogeneous API consumers and API implementing servers
• Java related technologies
• Microsoft .NET
• WSO2 is helping us be a “connected enterprise”
MWG WSO2 Product Usage and the MWG API
Three APIM Gateways. Two Primaries,
one is a backup without a BAM
connection. The F5 has an API health
nginx: preprocesses static content
Load balances application servers
API sync: rsync between gateways
MySQL DBs with
Circular replication (not shown)
F5 load balancer exposed to public
APIM Store / Publisher not shown,
Not accessible from public internet
Our WSO2 Product Deployment
• Consumer visits a website that is showing ads.
• Doubleclick network delivers the ad to the
consumer’s browser. There are multiple ads that
can be displayed. Each ad contains a set of
keywords relating to the item advertised, and the
• Rich media ad determines the consumer’s zip code.
• MWG API is called with the zip code, keywords,
• The API returned the nearest store and information
on a relevant item for sale. The consumer sees …
An ad is shown: step by step process
• Geolocation query for a particular retailer.
• bounding box by latitude, longitude
• Haversine function to sharpen results
• Returns a list of stores, ordered by increasing distance.
• Store circulars are queried
• keyword search
• Caching: database retrieval results
• geolocation query caching
• store circular data caching
• memcached used
Database loaded via background E-T-L processes
• API Manager requires an Authorization header.
• The Ad Unit, running Adobe ActionScript, will supply this header only with a
• ActionScript also supports client security by
demanding a crossdomain.xml file.
GET host:port/crossdomain.xml (no context root)
• nginx is used to intercept all the http calls
• serves the static crossdomain.xml file
• passes all other requests to API Manager.
• also used: load balance application server endpoints (needed for APIM 1.5)
Initial Technical Challenges
• Received only a total impression count from
marketing. No peak load estimate.
•API Manager / BAM integration vital.
• Essential: Used summarized BAM statistics from the
•Discovered with BAM: Workload was not random, or average. but very
• Bottleneck was our serving infrastructure, not API
• Added a memcached infrastructure on the fly.
•essential to boost for application server response time.
• Uncached API service time is 150 – 200ms.
• Object caching brings to 22ms average service time.
• Cache hit ratio: 85 – 90%.
• Record (with fast average service time): 376
responses per second at 5ms average service time.
• Peak responses per second: 402 responses /second
• A number of traffic bursts: 100 to 400 responses /
• Traffic slows in the overnight.
• Traffic peaks in late afternoon, evening.
• Hourly, traffic slows 25 minutes after the hour. It
can increase 50% at 55 minutes after the hour.
• Assume: ad network serving hourly adjustments or TV commercials.
• Overall mean: 4.5 responses per second.
percentile: 21 responses per second.
• Note: performance summarized by minute, based
on the BAM analytics toolbox for API manager.
Performance Highlights, cont’d.
• This plot range:
up to 100/sec
Responses per second distribution
• This plot range:
100-400 per sec.
• High end bursts
• Few instances.
Responses per second
Can spike if:
• Not enough processing
• Cache entries expired.
• Many cache misses and
• No API Manager
• Note: uncached service is
150ms – 200ms.
• With caching design
point, database is not
Average Service Time
• WSO2 BAM Cassandra storage
• Cassandra used as a ‘fast write’ repository for the streams of data
from the API manager gateways.
• BAM Hive scripts summarize on a per-minute basis, put summaries in
a relational db for reporting.
• Cassandra data does not seem to expire for us, so the Hive scripts
• Purging old, already summarized Cassandra data.
• WSO2 Api Manager: Publisher
• Provides cool displays of usage right in the publisher from the
• Publisher had trouble handling the data once we had a lot of data.
We turned off the display inside Publisher.
• Key Features
• API Design Wizard
• Support for OAuth2 Scopes
• Statistics Dashboard for developers
• Multi tenant workflow support
• Workflow integration for token generation
• URL templating support for API definition
• Customisable Store themes for tenants
• Forum Support - API Store
• Visibility Control for Documentation
• Statistics dashboard for southbound API calls
• Using Windows Authentication for OAuth2 Token generation
API Manager 1.7.0 is coming up...
• The WSO2 API Manager platform is well suited to
handle high volume traffic. It was very capable
• Essential: WSO2 Business Activity Monitor, using the
WSO2 provided API Manager BAM Analytics toolbox,
allowed us to
•Understand our traffic patterns
•Tune our server infrastructure
•Produce metrics for reporting
• Thanks for listening.
• Thanks: Ad unit graphic development: Kurrent
• Thanks: WSO2 for a flexible configurable product
set, based on open source standards.