Austin Scales - Nexus - Bazaarvoice's Cloud Infrastructure

Nexus
Bazaarvoice Cloud Infrastructure
Victor Trac
Cloud Architect, Bazaarvoice

Bazaarvoice
Not bizarre boys

Austin-based company founded in 2005 Basic stats:
Thousands of clients
SaaS serving software that collects
and displays user generated content, Hundreds of millions of pieces of
crunches analytics, and extracts content
data.
Hundreds of millions of unique
Engineering offices in Austin, NYC, visitors per month
London, and San Francisco
Tens of billions of page-views per
month

EC2, S3, VPC,
Regions, Autoscale,
CloudFormation,
ELB...?
Does this mean anything to you?

VPC & Subnets
VPC allows us to choose our internal IP space.

Public: Default route via IGW
Default Route for All Subnets to IGW
● Let's call these subnets all "Public"
● Requires all instances to have EIPs before
talking to the internet
● EIPs are a limited resource

Private: Default route via instance(s) in Public Subnets

Advantage: Most instances in the private subnet can
talk to the internet without dealing with an EIP.

Elastic Load Balancing

● Only Round Robin and Sticky Sessions
● Supports HTTP Response code or basic TCP connection Health Checks

{
"AWSTemplateFormatVersion" : "2010-09-09",
"Description" : "A text description for the template usage",
"Parameters": {
// A set of inputs used to customize the template per
deployment
},
"Mappings": {
// Mappings match a key to a corresponding set of named values
},
"Resources" : {
// The set of AWS resources and relationships between them
},
"Outputs" : {
// A set of values to be made visible to the stack creator
}
}

CloudFormation Instance Example

{
"AWSTemplateFormatVersion" : "2010-09-09",
"Description" : "Create an EC2 instance running the Amazon Linux 32 bit AMI.”,
"Parameters" : {
"KeyPair" : {
"Description" : "The EC2 Key Pair to allow SSH access to the instance",
"Type" : "String"
}
},
"Resources" : {
"Ec2Instance" : {
"Type" : "AWS::EC2::Instance",
"Properties" : {
"KeyName" : { "Ref" : "KeyPair" },
"ImageId" : "ami-75g0061f",
"InstanceType" : "m1.small"
}
}
},
"Outputs" : {
"InstanceId" : {
"Description" : "The InstanceId of the newly created EC2 instance",
"Value" : { "Ref" : "Ec2Instance" }
}
}
}

IAM and Console Access
Sign-on Credentials
● IAM Console login
○ Username, Password, and MFA Time token
Access Credentials
● AWS has three API types: REST, Query, & SOAP.
● Each API uses one or more Access Credentials
○ Access Keys for REST and Query APIs
○ x.509 Certificates for SOAP API
○ EC2 KeyPairs for instance SSH authentication

In the beginning...

A Java application server + a MySQL
Database

Scaled by adding in another
application server.

Then we just duplicated this entire
stack, giving us two "clusters".

Scaled more by adding more and more
clusters.

Decentralization --> Everyone goes fast!

Goals

Full control over AWS resources
EC2 resources, Autoscale, ELB, S3, etc.

Team Isolation
Resources created by one team can only be modified/terminated by
that team

3rd Party Solution
enStratus, RightScale, asgard, etc

Good Bad

● enStratus & RightScale ● No AWS API Access
provide cloud-agnostic ● No AWS CLI Tools & SDKs
tools ● Locked into only supported
services

Multiple Accounts
Good Bad
● Provides for full resource ● Inter-team network
control with direct API communications can become
access very complicated, relying
● Protects teams from one on VPN between VPCs ->
another Reduced Reliability
● Allows for easy accounting ● Management of networking is
on a per-team basis a possible bottleneck
● May make it easier for ● Shared resources may need
external auditors to to be redundantly built in
determine which teams have every VPC: LDAP, DNS,
"production" access Monitoring

Single Shared Account
Good Bad

● Sharing of resources will ● No built-in protections
be simple - just open between teams, even with
access via security groups IAM
between teams ● Creates a centralized
● Reliable networking between resource that someone has
teams without need for VPN to maintain
● Possibly better performance ● Requires us to build tools
due to fewer hops to use long-term
● Certain resources can be
shared: LDAP, DNS,
Monitoring, etc.

In more detail...
Nexus is:

● AWS Infrastructure designed with best practices:
○ secure
○ highly available
○ multi-region
○ repeatable
● Cloud building blocks and recipes for all of Engineering
● A Single Account Solution

Philosophy: Engineering teams at Bazaarvoice are free to choose
their own stack, but we want to make Nexus so compelling that it
is the default choice.

(some) Batteries Included
Included: Dev teams provide
● Bastion Hosts anything required to
● NAT Instances
run their app, which
● VPN Connectivity between
Regions probably means:
● Internal DNS
● Monitoring* ● Puppet/Chef/etc
● Centralized Logging* ● Your actual app
● Services Discovery* ● Deployment process
● Scripts & CloudFormation in
Github to create ephemeral
VPCs that look like a
Managed Environment

Autoscaling with CloudFormation

External DNS (Route53)

Records in the .nexus.bazaarvoice.com zone are for you to use.

Conformity -> Measurable Efficiency

Limitations & Risks

● Danger! Single Shared Account
○ You can wipe out all of a region with a
bad script.
● Single NAT per AZ
○ Someone else downloading lots of data from
the internet will affect all other
instances sharing your private subnet.
● Single VPN Instance per VPN Destination
○ Similar to NAT problem, but worse.
○ Avoid VPN when possible
○ If not possible, make your VPN dependency
resilient to lack of bandwidth and network
blips

Nexus is a catalyst:

old and busted new and shiny
waterfall agile

centralized development distributed teams

8-10 week release cycle release anytime

monolithic app services oriented architecture

mysql cassandra

solr elasticsearch

java whatever

dev + ops devops

Email: victor@bazaarvoice.com
Twitter: @victortrac

Thanks!

Austin Scales - Nexus - Bazaarvoice's Cloud Infrastructure

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to Austin Scales - Nexus - Bazaarvoice's Cloud Infrastructure

Similar to Austin Scales - Nexus - Bazaarvoice's Cloud Infrastructure (20)

Austin Scales - Nexus - Bazaarvoice's Cloud Infrastructure