I Love APIs 2015 : Zero to Thousands TPS Private Cloud Operations Workshop

Zero to Thousands TPS: !
Private Cloud Operations Workshop!
Maudrit Martinez
Apigee
Paul Mibus
Apigee

Agenda
9:30 – 10:00
Apigee Overview
10:00 – 11:00
Apigee Private Cloud Architecture
BREAK (15 MINS)
11:15 – 12:00
Apigee Private Cloud Architecture (cont)
BREAK – LUNCH (60 MINS)
13:00 – 15:45
Apigee Edge Install
BREAK (15 MINS)
16:00 – 17:30
Platform Operations
©2015 Apigee. All Rights Reserved.

Intelligent API platform enables the digital value chain
INTELLIGENT API
PLATFORM
Intelligent API Platform
User
Connected
Experience
Developer
API
API Team
Backend

Apigee Edge
Design"
Design first. Document Smart.
Full support for Swagger 2.0
Develop
Configuration: Over 30 ready-to-use and "
configurable policies
Code: Built-in support for Node, JavaScript and Java
extensibility BaaS
Secure
End-to-end security
Threat protection
Access control
Simple OAuth implementation for your APIs
PCI and HIPAA compliance
Publish
Turnkey developer portal
Monitor 
Centralized control, decentralized development
Multi-tenant architecture
Billions of API calls, including large spikes
Analyze
Complete visibility– from app end to backend
Automatically and continuously collect "
all API-traffic data out of the box
Monetize
Flexible rate plans
Internationalization support
Usage tracking
Limits and notifications
Scale
Self-service
State @ scale
Flexible deployment

Apigee Edge – API management
API Services
API Team
Developer Services
Monitor, manage, and measure success
Analytics Services
Increase adoption and value of APIs
Build, manage, scale, and secure APIs and Apps
API"
Gateway
OAuth &
Security
Policies &
Programmability
Versioning &
Governance
API Services
Developer"
Portal
Smart"
Docs
API"
BaaS
Monetization
Developer Services
Ops & Business
Metrics
Developer"
Analytics
App"
Performance
Custom"
Reports
Analytics Services
Flexible Deployment
Self-Service
State @ Scale
Connected
Experience
Business User
Developer

Conﬁg - Over 30 ready-to-use and conﬁgurable policies
Code - Built-in support for Node, JavaScript, Java and Python extensibility

Apigee Edge policies – Build APIs faster
Manage interactions with
API consumers and
optimize performance
Secure APIs and
protect back-end
systems from attack
Transform, translate and
reformat data for easy
consumption
Extend with
programming when
you need it

Public Cloud = Private Cloud

Flexible deployment options
Apigee Cloud
Multi-Datacenter Deployment
1+ Billion "
calls / month
7 regions
>99.9%"
API availability
19 availability zones
3,000+"
servers
~1/2 billion"
transactions / day
~20 ms"
average latency
Private Cloud"
leading Telco

Agenda
09:30 – 10:00
Apigee Overview
10:00 – 11:00
BREAK (15 MINS)
11:15 – 12:00
13:00 – 15:45
Apigee Edge Install
BREAK (15 MINS)
16:00 – 17:30
Platform Operations

Apigee Edge is comprised of several stateless components that use infrastructure services to persist data
•  Gateway: Routing and processing API calls
•  Apigee UIs: Enterprise UI, Developer Portal
•  Infrastructure Services: Persistence and queuing of run time data
•  Management Server: Provider of REST APIs for all conﬁguration tasks

Note: Monetization is part of Developer Services and leverages Gateway, Analytics Services and Management Server
Apigee Edge architecture – High level
ANALYTICS
SERVICES
API
SERVICES
BaaS"
Backend
DEVELOPER
SERVICES
Analytics
Backend
Management
Server
Gateway
Apigee "
UIs
Infrastructure
Services
Developer
Portal
Client
Backend

Apigee
Open Source
Legend:

Apigee Edge architecture – High level
Load
Balancer
Infrastructure
Services
BaaS
Backend
Developer
Portal
Analytics
Backend
Management
Server
Gateway
Infrastructure
Services
Apigee "
UIs
Gateway
Infrastructure
Services
Infrastructure
Services
BaaS
Backend
Developer
Portal
Analytics
Backend
Management
Server
Gateway
Infrastructure
Services
Apigee "
UIs
Gateway
Infrastructure
Services
DATAREPLICATION
DC-1
DC-2
Load
Balancer
Client
Backends

Apigee
Open Source
Legend:

Apigee Edge architecture – Component view

Apigee
Open Source
Legend:
•  Router handles all incoming API traffic and dispatches it. The
Router terminates the HTTP request, handles the SSL traffic,
and uses the virtual host name, port, and URI to steer
requests to the appropriate node.
•  Message Processor handles API traffic for a specific
organization and environment and which executes all
policies.
•  Apache Cassandra stores application configurations,
distributed quota counters, API keys, and OAuth tokens for
applications running on the gateway.
•  Apache ZooKeeper contains configuration data about all the
services of the zone and which notifies the different servers
of configuration changes.
•  OpenLDAP contains organization user and roles.
•  Management Server offers an API that is used by the Central
Services server to communicate with the servers in each on-
premises installation.
•  QPID Server manages queuing system for analytics data.
•  Postgres Server manages analytics database.
API SERVICES
DEVELOPER
SERVICES
Router
Enterprise "
UI
Qpid / Ingest
Server
postgreSQL
Postgres
Server
Developer
Portal
mySQL
BaaS Stack
BaaS UI
Management
Server
cassandra
openldap
zookeeper
qpidd
BAAS
ANALYTICS
SERVICES
Message
Processor
Client
Backend

Apigee Edge architecture – Technology Stack
Management
ZookeeperConsistency
Availability
Partition
Tolerance
Our customers value "
the ability to have highly
available API expanded
across geographically
dispersed sites.
Our customers value
accurate metrics to "
help drive data driven
business decisions.
Our customers value the ability to centralized
management of distributed components.
Apigee
Edge
CAP
Theorem
In theoretical computer science, the CAP theorem, also known as Brewer's theorem, states that it is impossible for a
distributed computer system to simultaneously provide all three of the following guarantees:
•  Consistency - all nodes see the same data at the same time.
•  Availability - a guarantee that every request receives a response about whether it succeeded or failed.
•  Partition tolerance - the system continues to operate despite arbitrary message loss or failure of part of the system.
Text Credits: Wikipedia


Apache Cassandra
15
DC 1
CS CS CS
DC 2
CS CS CS
“Apache Cassandra is an open source distributed database management system. It is an Apache Software
Foundation top-level project designed to handle very large amounts of data spread out across many
commodity servers while providing a highly available service with no single point of failure.” -- Wikipedia
http://cassandra.apache.org/
Cassandra characteristics:
•  All nodes are equal. Not master/slave or primary/secondary.
•  An application can read/write data from any node.
•  Data replication. Apigee Edge uses replication factor 3.
•  Consistency managed by application. Apigee Edge uses one
and local quorum.

Cassandra is used by Apigee for a variety of purposes, including:
•  Storage of developer, application and API Product data
•  Storage of access and refresh tokens
•  Storage of key-value map data
•  Audit logs
•  Custom analytics report models
QD
PG
OL
ZK
CS
MY
UI
R
MP
MS
QIS
PS
DP
BA
Developer Portal
BaaS Server
Router
Message Processor
Enterprise UI
Management Server
Postgres Server
Qpid/Ingest Server
Openldap
PostgreSQL
Apache Qpid
MySQL
Zookeeper
Cassandra
Server/Virtual Machine
POD
Legend:

Apache Zookeeper
16
DC 1
ZKV ZKV ZKV
DC 2
ZKV ZKV ZKO
“Apache ZooKeeper is a software project of the Apache Software Foundation, providing an open source
distributed configuration service, synchronization service, and naming registry for large distributed systems.” "

– Wikipedia

Zookeeper is used by Apigee as a distributed configuration registry, tracking component location, configuration
and status data. With some exceptions, it is NOT required to process API requests.
•  Leader: The node that controls coordination of writes
across distributed Zookeeper nodes
•  Voters: Nodes that can vote on change proposal
made by the Leader
•  Observers: Do not vote on change proposals and
must forward all writes to the Leader
http://zookeeper.apache.org/
QD
PG
OL
ZK
CS
MY
UI
R
MP
MS
QIS
PS
DP
BA
Developer Portal
BaaS Server
Router
Message Processor
Enterprise UI
Management Server
Postgres Server
Qpid/Ingest Server
Openldap
PostgreSQL
Apache Qpid
MySQL
Zookeeper
Cassandra
POD
Legend:

Apache Qpid
17
DC 1
QD QD
DC 2
QD QD
“Apache Qpid, an open-source (Apache 2.0 licensed) messaging system, implements the Advanced Message
Queuing Protocol. It provides transaction management, queuing, distribution, security, management, clustering,
federation and heterogeneous multi-platform support.” "

– Wikipedia

Qpid is used by Apigee Edge as messaging system for analytics and monetization data.
http://qpid.apache.org/
QD
PG
OL
ZK
CS
MY
UI
R
MP
MS
QIS
PS
DP
BA
Developer Portal
BaaS Server
Router
Message Processor
Enterprise UI
Management Server
Postgres Server
Qpid/Ingest Server
Openldap
PostgreSQL
Apache Qpid
MySQL
Zookeeper
Cassandra
POD
Legend:

•  Apigee Edge architecture offers great ﬂexibility when it comes to deployment options.
•  Edge can be deployed from a single VM to a multi-datacenter active/active conﬁguration.
•  Edge has been designed from the ground up to be a true cloud scale solution, capable of running on
both virtualized (including AWS) and physical hardware.
Apigee Edge architecture – Deployment options
QD
PG
OL
ZK
CS
MY
UI
R
MP
MS
QIS
PS
DP
BA
Developer Portal
BaaS Server
Router
Message Processor
Enterprise UI
Management Server
Postgres Server
Qpid/Ingest Server
Openldap
PostgreSQL
Apache Qpid
MySQL
Zookeeper
Cassandra
POD
Legend:
Node 1
MS
UI
R
MP
ZK
CS
AD
QIS
PS
QD
PG
Demo
Non-Production
Node 1
Node 2
Node 3
Node 4
Node 5
Node 7
Node 8
Node 9
Node 10
R R
QISMP MP
PS
QIS
ZKV
CS
ZKV
CS CS
QD QD
PGM
MS
UI
OL
Node 6
R
MP
ZKO
Production
Node 1
Node 2
Node 3
Node 4
Node 5
MS
UI R R
QISMP MP
PS
QIS
PS
ZK
CS
ZK
CS CS
ZK
OL
QD QD
PGM PGS

1. Know your business
Topology selection – Key considerations
Business SLAs Infrastructure
2. Understand trafﬁc patterns
3. Plan for growth (2X, 5X, 10X?)
4. Know your API proxies
5. Remember everything fails
6. API Trafﬁc, Analytics and
Management, Developer Service
components can and should scale
independently.
R R
MP MP
CS CS CS
PSQISQD PG
MSUI OL
Analytics
Management
DP MYDeveloper
R MPAPI Traffic CS
ZK
QD
PG
OL
ZK
CS
MY
UI
R
MP
MS
QIS
PS
DP
BA
Developer Portal
BaaS Server
Router
Message Processor
Enterprise UI
Management Server
Postgres Server
Qpid/Ingest Server
Openldap
PostgreSQL
Apache Qpid
MySQL
Zookeeper
Cassandra
POD
Legend:

•  Horizontally scalable. Components can be installed and conﬁgured to run from a single node "
(all in one setup) all the way to a multi DC centers, active/active, globally distributed setup.
•  Ability to stack components.
•  Distributed data replicated using eventual consistency.
•  Asynchronous analytics data capture and processing.
•  Centralized conﬁguration for distributed components.
•  Multitenant by design.
•  Management via APIs and UI console.
Apigee Edge architecture – Architecture characteristics
DC2
DC1
Node 1
Node 2
Node 3
Node 4
Node 5
Node 7
Node 8
Node 9
Node 10
R R
QISMP MP
PS
QIS
ZKV
CS
ZKV
CS CS
QD QD
PGM
MS
UI
ELB
OL
ELB
Load balancer
Node 6
R
MP
Node 11
Node 12
Node 13
Node 14
Node 15
Node 17
R R
QISMP MP
PS
QIS
ZKV
CS
ZKV
CS CS
QD QD
PGS
MS
UI
OL
Node 16
R
MP
ZKO
ZKV
Node 18
Node 19
Node 20
QD
PG
OL
ZK
CS
MY
UI
R
MP
MS
QIS
PS
DP
BA
Developer Portal
BaaS Server
Router
Message Processor
Enterprise UI
Management Server
Postgres Server
Qpid/Ingest Server
Openldap
PostgreSQL
Apache Qpid
MySQL
Zookeeper
Cassandra
POD
Legend:

•  Planet footprint is driven by customer requirements, transaction volumes, availability and reliability
among others drive component stacking and number of nodes.
•  R, MP and CS are critical components to handle live API trafﬁc.
•  Management and analytics components are not required on every DC.
•  For high availability, ZK and QD should be available in all DCs.
•  Analytics data replication via Master/Slave or Publish/Subscribe.
Apigee Edge architecture – Horizontal scalability
QD
PG
OL
ZK
CS
MY
UI
R
MP
MS
QIS
PS
DP
BA
Developer Portal
BaaS Server
Router
Message Processor
Enterprise UI
Management Server
Postgres Server
Qpid/Ingest Server
Openldap
PostgreSQL
Apache Qpid
MySQL
Zookeeper
Cassandra
POD
Legend:
DC2
DC2
DC2DC1
Node 1 Node 2 Node 3 Node 4 Node 5 Node 6 Node 7 Node 8 Node 9
R R
QISMP MP
PS
QIS
ZKV
CS
ZKV
CS CS
QD QD
PGM
MS
UI
OL
Node 10 Node 11 Node 12 Node 13 Node 14 Node 15 Node 16 Node 17 Node 18
R R
QISMP MP
PS
QIS
ZKV
CS
ZKV
CS CS
ZKV
QD QD
PGS
MS
UI
OL
DC<N>
Node 11 Node 12 Node 13 Node 14 Node 15 Node 16 Node 17
R R
QISMP MP QIS
ZKV
CS
ZKV
CS CS
ZKV
QD QD
ZKV

•  Given the responsibility and capabilities offered by
each component, scalability requirements and how
they are implemented may vary.
•  In most scenarios, scaling to accommodate higher
number of TPS or API calls may impact only
components serving live API trafﬁc.
•  Analytics and management components may grow
in number mostly driven by demanding high
availability requirements for those capabilities
provided by analytics and management
components.
Apigee Edge architecture – Scaling by capability
QD
PG
OL
ZK
CS
MY
UI
R
MP
MS
QIS
PS
DP
BA
Developer Portal
BaaS Server
Router
Message Processor
Enterprise UI
Management Server
Postgres Server
Qpid/Ingest Server
Openldap
PostgreSQL
Apache Qpid
MySQL
Zookeeper
Cassandra
POD
Legend:
QIS PSQIS
QD QD PGM
R R
MP MP
CS CS CS
MSUI OL
ZK ZK ZK
API Trafﬁc
Analytics
Management
DP MYDeveloper

Need correct heart
15 mins break

Apigee Edge architecture – Multitenant by design
DC2
DC1
Node 1
Node 2
Node 3
Node 4
Node 5
Node 7
Node 8
Node 9
Node 10
R R
QISMP MP
PS
QIS
ZKV
CS
ZKV
CS CS
QD QD
PGM
MS
UI
ELB
OL
ELB
Load balancer
Node 6
R
MP
Node 11
Node 12
Node 13
Node 14
Node 15
Node 17
R R
QISMP MP
PS
QIS
ZKV
CS
ZKV
CS CS
QD QD
PGS
MS
UI
OL
Node 16
R
MP
ZKO
ZKV
Node 18
Node 19
Node 20
QD
PG
OL
ZK
CS
MY
UI
R
MP
MS
QIS
PS
DP
BA
Developer Portal
BaaS Server
Router
Message Processor
Enterprise UI
Management Server
Postgres Server
Qpid/Ingest Server
Openldap
PostgreSQL
Apache Qpid
MySQL
Zookeeper
Cassandra
POD
Legend:

Multitenancy
“Multitenancy is a reference to the mode of operation of software where multiple independent instances of
one or multiple applications operate in a shared environment. The instances (tenants) are logically isolated,
but physically integrated. The degree of logical isolation must be complete, but the degree of physical
integration will vary. The more physical integration, the harder it is to preserve the logical isolation. The
tenants (application instances) can be representations of organizations that obtained access to the
multitenant application (this is the scenario of an ISV offering services of an application to multiple
customer organizations). The tenants may also be multiple applications competing for shared underlying
resources (this is the scenario of a private or public cloud where multiple applications are offered in a
common cloud environment).” - Gartner
Multitenancy = Software capabilities to support shared infrastructure "
while providing tenants with data and processing partitioning.

Apigee Edge Architecture – Multitenancy
Region

Pod

Org

Virtual
Host

Environment

Regional Services are shared by many pods across a single geographical region and maintain
state and provide an API that works across the pods in a region
Pods are a collection of servers that share logical functions such as a Gateway Pod or an
Analytics Pod.
Orgs provide logical grouping to secure access to API management services. Orgs are associated
with pods for servers – they can either share hardware or be isolated
Environments are virtual routes that allow API bundles to be deployed and tested within and Org.
Environments can be associated with pods for servers independently of the Org.
Virtual Hosts are similar to Apache Virtual hosts and route trafﬁc to environments based on ports
and domain names.
Planet
Planets represent an entire physical environment and can encompass multiple regions and pods

Apigee Edge Architecture – Multitenancy
Data Center 1
Data Center 2
Data Center N
Planet

Org
1

Environment
1

Environment
N

Org
N

Environment
1

Environment
N

•  A Planet can expand multiple DCs. Organization and Environment can expand across the planet.
•  Data partitioning by Organization and Environment. Processing partitioning by Org+Env can be conﬁgured.

Apigee Edge Architecture – Organizational structure
Organiza3on

Environment

Cache
Resource

KVM
Resource

Target
Server

User

Role

Permission

Permission

Extension
Resource

Key
/
Trust
Store

Vault

API
Product

Company

Developer

Applica3on

Key
/
Token

KVM
Resource

Virtual
Host

API
Proxy

Extension
Resource

Vault

Deployment


Apigee Edge Architecture – Physical partitioning
Tenant A
Tenant B
Tenant C
Tenant A
Tenant B
Tenant C
Tenant D
Tenant E
Tenant D
Tenant E
Tenant A
Tenant B
Tenant C
Tenant D
Tenant E
Inbound Routing API Call Processing Data Storage

Apigee Edge Architecture – Organizational structure
DC2
DC1
Node 1
Node 2
Node 3
Node 4
Node 5
Node 7
Node 8
Node 9
Node 10
R R
QISMP MP
PS
QIS
ZKV
CS
ZKV
CS CS
QD QD
PGM
MS
UI
OL
Node 6
R
MP
Node 11
Node 12
Node 13
Node 14
Node 15
Node 17
R R
QISMP MP
PS
QIS
ZKV
CS
ZKV
CS CS
QD QD
PGS
MS
UI
OL
Node 16
R
MP
ZKO
ZKV
Node 18
Node 19
Node 20
Organization
Environment
Central
POD
Gateway
POD
Analytics
POD
Planet
QD
PG
OL
ZK
CS
MY
UI
R
MP
MS
QIS
PS
DP
BA
Developer Portal
BaaS Server
Router
Message Processor
Enterprise UI
Management Server
Postgres Server
Qpid/Ingest Server
Openldap
PostgreSQL
Apache Qpid
MySQL
Zookeeper
Cassandra
POD
Legend:

Apigee Edge Architecture – API traffic data flow
32
Legend:
•  Routers send requests to Message
Processors in their Gateway pod.
•  If there are two or more gateway
pods in a region, then routers will
ignore message processors in the
other gateway pods.
•  Message Processors respect the
region as their scope.
•  For two data center the same rules
apply as for one datacenter.
•  All Apigee components are
configured to only use the Cassandra
nodes in their region / data center.
Region / DC 1
R
MP
R
MP
Load balancer
Client
CS CSCS
Backend
GatewayPod1
R
MP
R
MP
GatewayPod2
QD
PG
OL
ZK
CS
MY
UI
R
MP
MS
QIS
PS
DP
BA
Developer Portal
BaaS Server
Router
Message Processor
Enterprise UI
Management Server
Postgres Server
Qpid/Ingest Server
Openldap
PostgreSQL
Apache Qpid
MySQL
Zookeeper
Cassandra
POD
Legend: API call flow
Analytics flow

Apigee Edge Architecture – API traffic data flow
33
R+MP Configuration options
R
MP
R
MP
Load balancer
Tenant Aware Routing
•  Rs direct traffic to appropriate MPs, load balancing
between them.
•  Default behavior.
•  Health check heartbeat allows R to automatically take
MP out of/into rotation if unresponsiveness is detected.
R
MP
R
MP
Load balancer
Server Affinity
•  R can also be configured to connect to a dedicated MP.
Using server affinity, all traffic handled by a R is exclusively
sent to its corresponding MP.
•  This configuration option offers customer the ability to
isolate R/MP for dedicated use cases without impacting
API traffic flowing across other R/MP within the same pod.
QD
PG
OL
ZK
CS
MY
UI
R
MP
MS
QIS
PS
DP
BA
Developer Portal
BaaS Server
Router
Message Processor
Enterprise UI
Management Server
Postgres Server
Qpid/Ingest Server
Openldap
PostgreSQL
Apache Qpid
MySQL
Zookeeper
Cassandra
POD
Analytics flow

Apigee Edge Architecture – Analytics data ﬂow
34
•  Ingest services in Qpid Ingest Server will collect Analytics
data from all queues and store in PostgreSQL.
•  Postgres Server aggregates analytics data
asynchronously.
•  Message Processors respect the region as their scope
and will ofﬂoad analytics data to all Apache Qpid queues
in their region / data center.
R
MP
R
MP
Load balancer
GatewayPod
QD QD
CentralPod
PGM
QIS QIS
PS
QD
PG
OL
ZK
CS
MY
UI
R
MP
MS
QIS
PS
DP
BA
Developer Portal
BaaS Server
Router
Message Processor
Enterprise UI
Management Server
Postgres Server
Qpid/Ingest Server
Openldap
PostgreSQL
Apache Qpid
MySQL
Zookeeper
Cassandra
POD
Analytics flow

DC 2
DC 2
DC 1
DC 1
Apigee Edge Architecture – Analytics data flow
35
QD
PG
OL
ZK
CS
MY
UI
R
MP
MS
QIS
PS
DP
BA
Developer Portal
BaaS Server
Router
Message Processor
Enterprise UI
Management Server
Postgres Server
Qpid/Ingest Server
Openldap
PostgreSQL
Apache Qpid
MySQL
Zookeeper
Cassandra
POD
Analytics flow
QIS
PS PS
QD
PGM PGS
MP
Writes
Writes
Reads
Writes
QISQD
MP
Writes
Writes
Reads
QIS
PS
QD
PGM
MP
Writes
Reads
Writes
QD
MP
Writes
QIS
PS
PGM
Writes
Reads
Reads Reads
Reads/Writes
Reads/Writes Reads/Writes
•  Analytics data is partitioned per Environment.
•  Analytics data size generated by MPs per transaction is about 1kb.
•  PostgreSQL contains Raw Data Tables and Aggregated Data Tables. Raw Data Tables grow as
analytics data is collected. Appropriate data retention policy and purge processes are required.
•  Custom reports allow customers to define user-defined queries which run against raw data.

Apigee Edge Architecture – Network Zoning
36
QD
PG
OL
ZK
CS
MY
UI
R
MP
MS
QIS
PS
DP
BA
Developer Portal
BaaS Server
Router
Message Processor
Enterprise UI
Management Server
Postgres Server
Qpid/Ingest Server
Openldap
PostgreSQL
Apache Qpid
MySQL
Zookeeper
Cassandra
POD
Analytics flow
Data
Data
DMZ
App
DMZ
Core
R R
QIS
MP MP
PS
QIS
ZKV
CS
ZKV
CS CS
QD QD
PGM
MS
UI
OL
R
MP
ZKV
DP
MY
QIS
MP MP
QIS
QD QD
MS
UI
OL
MP
R RR
PS
ZKV
CS
ZKV
CS CS
PGM
ZKV
DP
MY
QIS
MP MP
QIS
QD QDMS
UI
OL
MP
R RR
PS
ZKV
CS
ZKV
CS CS
PGM
ZKV
DP
MY
Single Zone
Two Zones
Three Zones
•  Edge architecture does not impose
network zone requirements. Network
zoning will be driven by customer
operation and security requirements.
•  Firewalls and security appliances
between zones should
accommodate the connectivity "
and trafﬁc characteristics of Apigee
components without adding "
latency overhead.
•  Keep API trafﬁc as direct as possible.
Keep MP dependencies close.
•  Pay attention to cross-zone
connections.

Apigee Edge Architecture – Components Connectivity
37
QD QIS
CS ZK
CS ZK
CS ZK
AD / OL
PG-standby PS
Target Backend
PG: 5432
QD:
5672
customer
specific
10389
ZK: 2181
CS: 9160,
7199
PG: 5432
PG-master PS
ZK: 2181
CS: 9160,
7199
UI MS
QIS:
4529
PS: 4530
PG: 5432
PS
PS
MP
MS
MS
MS
MS
QIS
MP
QIS
MS
ZK: 2181
2888, 3888
CS: 7000
7199, 9160
PG: 5432*
PG
PG
QIS:
8083
1102
PS: 8084
1103
8081
1100
MS:
8080
UI:
9000
4528
M
M
M
MP
8082
1101
M
8998, 4528
R
customer specific
MP
MS
4527
MYDP 3306
MS:
8080
DP
DP
client / load balancer
R

R Router
MP Message Processor
CS Cassandra
MS Management Server
OL OpenLDAP
ZK Zookeeper
QD Apache Qpid
QS Qpid Server
PG PostgreSQL
PS Postgres Server
DP Developer Portal
MY MySQL
APItraffic
Configuration
Analytics
Dev
UI Management UI
Legend: from
svc:port
1-way
2-way
to
management
calls HTTP or
JMX)
M
Notes:
•  For CS all nodes
talk to all other CS
nodes
•  For ZK all nodes talk
to all other ZK nodes
•  * both directions to
enable PostgreSQL
failover
QD:
5672
R
Single DC View

Apigee Edge Architecture – Components Connectivity
38

R Router
MP Message Processor
CS Cassandra
MS Management Server
OL OpenLDAP
ZK Zookeeper
QD Apache Qpid
QS Qpid Server
PG PostgreSQL
PS Postgres Server
DP Developer Portal
MY MySQL
APItraffic
Configuration
Analytics
Dev
UI Management UI
Legend: from
svc:port
1-way
2-way
to
management
calls HTTP or
JMX)
M
Notes:
•  For CS all nodes
talk to all other CS
nodes
•  For ZK all nodes talk
to all other ZK nodes
•  * both directions to
enable PostgreSQL
failover
Multi DC View
^
Data Center 1 Data Center 2
MP
Target Backend
R
QD QIS
CS ZK
CS ZK
CS ZK
OL
PG-master
PS
UI MS
CS ZK
CS ZK
CS ZK
OL
QD QIS
PG-standby
PS
R
MP
UI MS
PG: 5432
10389
PG: 5432
4527
1100
4528
1101
4527
1100
4528
1101
ZK: 2181, 2888, 3888
CS: 7000, 7199, 9160
PG
QIS
MS MS
^Target Backend

Need correct heart
60 mins break

System Requirements – Hardware Speciﬁcation
•  Cassandra
–  Local storage with SSD or fast HDD supporting 2000 IOPS
•  PostgreSQL
–  Local storage with SSD or fast HDD supporting 1000 – 8000 IOPS
–  CPU, Memory and Size the storage according to the data retention
requirements. Implement archive / purge process and keep less
data for best performance.
–  Operations Guide provides formula to estimate storage needs.
Type C
ZKV
CS
Type G
QIS
QD
Type H
PS
PGM
Type A
MS
UI
Type B
OL
Type
Components
CPU
Memory
Disk

A
UI,
MS
2
core
4
GB
60
GB

B
OL
2
core
4
GB
60
GB

C
ZK,
CS
8
core
16
GB
250
GB
local
storage
with
SSD
or
fast
HDD
supporOng
2000
IOPS

D
R,
MP
4/8
core
8/16
GB
100
GB

E
R
4
core
8
GB
60
GB

F
MP
8
core
16
GB
100
GB

G
QIS,
QD
8
core
16
GB
500
GB
local
storage
with
SSD
or
fast
HDD
supporOng
1000
IOPS

H
PS,
PG
8
core
16
GB
500
GB
to1
TB
local
storage
with
SSD
or
fast
HDD
supporOng
4000-‐8000

IOPS

Type F
MP
Type D
R
MP
Type E
R
QD
PG
OL
ZK
CS
MY
UI
R
MP
MS
QIS
PS
DP
BA
Developer Portal
BaaS Server
Router
Message Processor
Enterprise UI
Management Server
Postgres Server
Qpid/Ingest Server
Openldap
PostgreSQL
Apache Qpid
MySQL
Zookeeper
Cassandra
POD
Legend:

System Requirements – Software Dependencies
https://apigee.com/docs/api-services/reference/supported-software
Operating System
•  Red Hat Enterprise Linux (64-bit):
o  6.3, 6.4, 6.5, 6.6, 7.0
•  CentOS (64-bit):
o  6.3, 6.4, 6.5, 6.6, 7.0
•  Oracle Linux (64-bit):
o  6.5

JDK
•  Oracle JDK 1.7
•  OpenJDK 7
SSL/TLS
•  1.0
•  1.2

Other software
•  Cassandra 2.0.15
•  Zookeeper 3.4.5
•  QPID 0.14
•  PostgreSQL 9.3
•  Play (UI) 2.3.4
•  OpenLDAP 2.4

Private Cloud Installer – Software Dependencies
•  awk
•  basename
•  bash
•  chkconﬁg
•  curl
•  date
•  dirname
•  echo
•  expr
•  grep
•  hostname
•  id
•  ls
•  perl
•  pgrep (from procps)
•  ps
•  pwd
•  python
•  rpm
•  rpm2cpio
•  sed
•  sudo
•  tar
•  tr
•  uname
•  unzip
•  useradd
•  wc
•  yum
Complete list of prerequisites can be found in Apigee Edge Install and Conﬁguration Guide, Page 19.

In addition to the tools above, some nodes require the installation of additional software components such as:
•  ntp (all nodes)
•  rsync (PostgreSQL nodes)
•  openldap-clients openldap-servers (Management Server node)

Apigee Edge Installation Process
DC1
Node 1
Node 2
Node 3
Node 4
Node 5
Node 7
Node 8
Node 9
Node 10
R R
QISMP MP
PS
QIS
ZKV
CS
ZKV
CS CS
QD QD
PGM
MS
UI
OL
Node 6
R
MP
ZKV
QD
PG
OL
ZK
CS
MY
UI
R
MP
MS
QIS
PS
DP
BA
Developer Portal
BaaS Server
Router
Message Processor
Enterprise UI
Management Server
Postgres Server
Qpid/Ingest Server
Openldap
PostgreSQL
Apache Qpid
MySQL
Zookeeper
Cassandra
POD
Legend:
1.  Install data store hosts
2.  Install management hosts
•  Install LDAP ﬁrst if using standalone LDAP
hosts
3.  Install router, message processor
4.  Qpid hosts
5.  PostgreSQL hosts
6.  Conﬁgure PostgreSQL replication, if needed
7.  Create organization(s) and environment(s)

If applicable, install:

7.  Developer portal
8.  Install monetization
9.  App services (BaaS)
1
2
3
4
5

Private Cloud – Software Install
Apigee Private Cloud (apigee-edge-4.15.07.00.zip) has everything required to install and conﬁgure Apigee Edge
and BaaS components except for the Developer Portal. The Developer Portal is distributed using a different
package (DeveloperServices-4.15.07.00.tar).
Apigee Edge installation steps:
1.  Acquire a license key and copy it to all nodes.
2.  Download the Apigee Private Cloud from ftp.apigee.com and copy it to all nodes.
3.  Installation downloads and installs required system software via Yum. It requires Internet connection or local
repository.
4.  Unzip apigee-edge-4.<YY>.<MM>.<V>.zip
5.  Run the primary installation script:

/<unzip-‐location>/apigee-‐edge-‐4.<YY>.<MM>.<V>/apigee-‐install.sh
-‐j
/usr/java/default

-‐r
<inst-‐root>
-‐d
<data-‐root>


Private Cloud – Software Setup
1.  Once Apigee binaries are installed, run:
/<inst-‐root>/apigee4/share/installer/apigee-‐setup.sh

2.  Setup script requires a proﬁle type and information about the system, provided interactively or via response
ﬁle (silent install).
3.  Create an organization, environment(s) and organization administrator, run:
/<inst-‐root>/apigee4/bin/setup-‐org.sh


Private Cloud – Silent Install
•  Silent install provides a way for specifying, in advance, all necessary values used by apigee-setup.sh
•  Silent install provides a way for specifying, in
advance, all necessary values used by apigee-
setup.sh
/<inst-‐root>/apigee4/share/installer/
apigee-‐setup.sh
-‐p
ds
-‐f
<response-‐
file-‐name>

•  The response files contain a number of variables
definitions to be used by apigee-setup.sh. A
response file per DC/Region will be needed since
some values are unique per DC/Region.
Single machine setups:
•  aio = All In One (Gateway and Analytics Standalone)
Cluster node setup for ZooKeeper and Cassandra (min 3 nodes):
•  ds = Datastore Cluster Node

LDAP setup for OpenLDAP:
•  ld = LDAP Node

Separate components setup:
•  ld = LDAP Node
•  ms = Gateway Management Server
•  r = Gateway Router
•  mp = Gateway Message Processor
•  rmp = Gateway Router and Message Processor
•  qs = Analytics Qpid Server
•  ps = Analytics Postgres Server
•  mo = Monetization Server
Note: The list above doesn’t show all the available options. Refers to Apigee Edge
Install and Configuration Guide, section Basic Host Installation, page 35 for details.

Private Cloud – Silent Response File
•  IP1=
•  IP2=
•  IP3=
•  IP4=
•  IP5=
•  HOSTIP=$(hostname -i)
•  MSIP=$IP1
•  ADMIN_EMAIL=
•  APIGEE_ADMINPW=
•  LICENSE_FILE=/root/opdk/license.txt
•  USE_LDAP_REMOTE_HOST=n
•  LDAP_TYPE=1
•  APIGEE_LDAPPW=
•  ENABLE_AX=y
•  MP_POD=gateway
•  REGION=dc-1
•  USE_ZK_CLUSTER=y
•  ZK_HOSTS="$IP1 $IP2 $IP3"
•  ZK_CLIENT_HOSTS="$IP1 $IP2 $IP3"
•  USE_CASS_CLUSTER=y
•  CASS_HOSTS="$IP1:1,1 $IP2:1,1 $IP3:1,1"
•  CASS_USERNAME=
•  CASS_PASSWORD=
•  SKIP_SMTP=y
•  SMTPHOST=
•  SMTPPORT=25
•  SMTPUSER=
•  SMTPPASSWORD=
•  SMTPSSL=n
•  BIND_ON_ALL_INTERFACES=y

Apigee Edge Install Lab
Scope
•  5 nodes Apigee Edge Install
•  VMs instances running in Amazon AWS
•  Installation and setup using silent install process.
•  Setup order DS, MS, RMP, SAX
AWS Instances IP
•  http://iloveapis.com/workshops/ops/

Documentation
•  ApigeeEdgePrivateCloud-Install-Conﬁg-Guide.pdf
o  Page 35 - Basic Host Installation
o  Page 47 – Onboarding
o  Page 120 - Appendix A: Silent Installation
o  Page 121 - 5-host Clustered Installation
Node 1
Node 2
Node 3
Node 4
Node 5
MS
UI R R
QISMP MP
PS
QIS
PS
ZK
CS
ZK
CS CS
ZK
OL
QD QD
PGM PGS
2: MS
1: DS
3: RMP
4: SAX

Apigee Edge Install Lab - Prerequisites
Preparation
A.  Software, sample install file and JDK are located in /root/opdk/
B.  Install prerequisites
o  In all nodes:
§  Install
JDK (rmp -ivh <rmp>)

§  yum
-‐y
install
ntp

o  In node 1:
§  yum
-‐y
install
openldap-‐clients
openldap-‐servers

o  In node 4 and 5:
§  yum
-‐y
install
rsync

C.  In all nodes, unzip
apigee-‐edge-‐4.15.07.00.zip

D.  Update silent install file with private IP addresses corresponding to your
AWS instances. Copy silent install files to all nodes.

Additional considerations:
•  For the purpose of the lab, we took care of some prerequisites. When planning for real world installation, prerequisites described in Apigee
Edge Install and Configuration Guide, Page 19-35 should be completed.

Apigee Edge Install Lab – Installation and Setup
Install Steps

A.  Unzip apigee-edge-4.15.07.00.zip
B.  ./apigee-‐install.sh
-‐j
/usr/java/default
-‐r
/opt
-‐d
/opt

Setup Steps

1.  /opt/apigee4/share/installer/apigee-‐setup.sh
-‐p
ds
-‐f
/root/opdk/response.txt

-‐p
ms
-‐f

-‐p
rmp
-‐f

-‐p
sax
-‐f

Node 1
Node 2
Node 3
Node 4
Node 5
MS
UI R R
QISMP MP
PS
QIS
PS
ZK
CS
ZK
CS CS
ZK
OL
QD QD
PGM PGS
2: MS
1: DS
3: RMP
4: SAX

Apigee Edge Install Lab – Installation and Setup
Setup Master-Standby Replication for PostgreSQL
•  Apigee Edge Install and Conﬁguration Guide, Page 59

Onboarding
•  Apigee Edge Install and Conﬁguration Guide, Page 47

•  /opt/apigee4/bin/setup-org.sh
Access Apigee Edge
•  http://<management-server-public-ip>:9000

Agenda
09:30 – 10:00
Apigee Overview
10:00 – 11:00
BREAK (15 MINS)
11:15 – 12:00
13:00 – 15:45
Apigee Edge Install Lab
BREAK (15 MINS)
16:00 – 17:30
Platform Operations

Platform Operations - Walkthrough
•  Directory Structure
o  /<inst-‐root>/apigee4

•  Start / Stop / Status
o  /<inst-‐root>/apigee4/bin/all-‐start.sh

o  /<inst-‐root>/apigee4/bin/all-‐status.sh

o  /<inst-‐root>/apigee4/bin/all-‐stop.sh

•  Logs ﬁle
o  /<inst-‐root>/apigee4/var/log

•  Management UI
o  http://<management-‐server-‐host>:9000

•  Management API
o  http://apigee.com/docs/management/apis

o  http://<management-‐server-‐host>:8080


Platform Operations – Backup and Restore
•  Backup/restore
o  Edge includes backup scripts for:
§  LDAP
§  Cassandra
§  ZooKeeper
§  PostgreSQL
§  UI (custom reports)
§  Component UUIDs
o  <inst-root>/apigee4/bin/backup.sh

•  High Availability and Disaster Recovery
o  Multiple copies of the data
o  Eventual consistency
o  Horizontal scalability and resiliency
DC1
Node 1
Node 2
Node 3
Node 4
Node 5
Node 7
Node 8
Node 9
Node 10
R R
QISMP MP
PS
QIS
ZKV
CS
ZKV
CS CS
QD QD
PGM
MS
UI
OL
Node 6
R
MP
ZKV

Platform Operations – Recurring Maintenance
•  Cassandra nodes repair
/opt/apigee4/share/apache-‐cassandra/bin/nodetool
repair
-‐pr

•  Periodically prune detailed analytics data based on retention requirements
/opt/apigee4/bin/pg-‐data-‐purge.sh


Platform Operations – Component Level Monitoring
JVM Statistics –
Threads,
heap, GC.
Platform APIs.
System Statistics –
CPU, Memory, Disk,
Network.
DC1
Node 1
Node 2
Node 3
Node 4
Node 5
Node 7
Node 8
Node 9
Node 10
R R
QISMP MP
PS
QIS
ZKV
CS
ZKV
CS CS
QD QD
PGM
MS
UI
OL
Node 6
R
MP
ZKV
Monitoring Tool
QD
PG
OL
ZK
CS
MY
UI
R
MP
MS
QIS
PS
DP
BA
Developer Portal
BaaS Server
Router
Message Processor
Enterprise UI
Management Server
Postgres Server
Qpid/Ingest Server
Openldap
PostgreSQL
Apache Qpid
MySQL
Zookeeper
Cassandra
POD
Legend:
•  System-Level Checks
–  CPU, Memory, Disk, Load, Network
•  Process/Application Checks
–  Thread statistics, Memory utilization
•  Components monitoring
–  JMX
–  Metrics
–  Management API (http://<host>:<port>/v1/
servers/self/up)
–  Router health check of the Message
Processer
–  Log monitoring
•  HeartBeat
•  Mark down/mark up events
•  Logging Policy

•  API-Level Checks
–  API calls 20X responses
–  API call error within Apigee
–  API call error on backend
–  JMX
–  Metrics
•  Message Flow Checks
–  Number of active clients
–  Number of responses (10X, 20X, 30X, 40X and 50X)
–  Connect failures
•  API Health - https://pages.apigee.com/api-health
Platform Operations – APIs Monitoring
QD
PG
OL
ZK
CS
MY
UI
R
MP
MS
QIS
PS
DP
BA
Developer Portal
BaaS Server
Router
Message Processor
Enterprise UI
Management Server
Postgres Server
Qpid/Ingest Server
Openldap
PostgreSQL
Apache Qpid
MySQL
Zookeeper
Cassandra
POD
Legend:
LB
Backend System(s)
R+MP
R
MP
R
MP
Monitoring
Tool
Monitoring Tool
Edge
Analytics

Takeaway
Thank you!
•  Public Cloud = Private Cloud
•  Architecture purpose built and proven

•  Horizontally scalable, components can be scaled independently of each other
•  Multi-tenant by design

I Love APIs 2015 : Zero to Thousands TPS Private Cloud Operations Workshop

More Related Content

What's hot

Viewers also liked

Similar to I Love APIs 2015 : Zero to Thousands TPS Private Cloud Operations Workshop

More from Apigee | Google Cloud

Recently uploaded

I Love APIs 2015 : Zero to Thousands TPS Private Cloud Operations Workshop