A high level overview of MeteorJS, Amazon Web Services, and how to scale MeteorJS on Amazon's cloud to handle tends of thousands of concurrent websocket connections.

1. Scaling a MeteorJS SaaS app on
AWS
Brett McLain
Application Systems Architect
PotashCorp

2. Agenda
• Overview of my application.
• Overview of MeteorJS (architecture,
deployment tools, clustering).
• Overview of Amazon (EC2, ELB, EBS, Route 53,
Auto Scaling).
• My journey of trying to scale MeteorJS on
Amazon Web Services.

3. What should you get out of this talk?
• A better understanding of MeteorJS.
• A better understanding of Amazon Web
Services.
• How to deploy, cluster, and scale a MeteorJS
web app.

4. Application Overview
• SaaS application that uses Twilio to send SMS
and phone calls to devices that connect to the
cell phone network.
• MeteorJS (runs on Node.js)
• MongoDB
• Flot Charts
• Twitter Bootstrap
• Blah blah blah.

5. What is MeteorJS?
• "Meteor is a full-stack JavaScript platform for
developing modern web and mobile applications.“
• Out of the box tight integration with MongoDB + full
data reactivity.
• Install MeteorJS and have an app written in minutes
with well established and easily understood patterns.
• Hot code push.
• Full data reactivity.
• Easily build for Android and iPhone (hot code push).

6. Meteor Architecture
• Production deployments are NodeJS applications.
• Meteor has it’s own package manager called
isobuild. Extremely similar to NPM.
• Meteor can also use NPM packages now on the
client or server (as of version 1.3).
• Supports ECMAScript 6 (ES 2015).
• Meteor Up (mup) - Meteor deployment plugin.
• Meteor Cluster - Meteor clustering plugin.

7. Meteor Architecture

8. Meteor Deployments
• I use Meteor Up (mup) for deployments.
• Single command to build and deploy code to an
unlimited number of servers in a rolling fashion.
• Installs all necessary software automatically
(NodeJS, npm, MeteorJS, docker, nginx, mongo).
• Setups up an nginx reverse proxy in front of a
docker instance for meteor and a docker instance
for mongo.
• 10 seconds of downtime per instance!

10. Meteor Cluster
• Provides clustering capability within Meteor JS
apps.
• Meteor Cluster is a regular meteor package.
• Turns each meteor server into a load balancer
or a DDP instance (backend server).New
nodes register with MongoDB backend and
are auto-discovered. Traffic begins routing to
them immediately.

11. Round Robin DNS Load Balancing

12. 1 Load Balancer, 2 DDP Servers

13. 2 Load Balancers, 2 DDP Servers

14. Meteor Clustering
• Load balancer servers are pointed to by DNS
records.
• Once the client connects to a load balancer
server, it *might* receive the address of a DDP
server to share out the load.
• Supports clustering of microservices too!

15. So…now what?
• Meteor has some great deployment and
clustering tools, but how do we deploy it in a
scalable way?We need to pick a host that fits
well with our deployment tools and utilize the
clustering capabilities of Meteor.

16. Initial Objectives
• Low Cost!
• Availability
• Performance
• Low Maintenance

17. Where to start?
• Originally hosted everything (prod, dev,
mongo, reverse proxy) on a single Virtual
Private Server (VPS).
• I have past experience with Amazon, it’s well
documented, tons of tools.
• Google *appears* pricier, less tools.
• Digital Ocean…?

18. So what does Amazon offer?
• EC2 (Elastic Compute Cloud)
• Elastic Bean Stalk
• Elastic Load Balancers
• Route 53 (DNS)
• OpsWorks
• CloudFormation
• Elastic Block Store
• The list goes on!

20. Elastic Compute Cloud (EC2)
• On demand infrastructure.
• Amazon Machine Image (AMI) of many popular
operating systems, platforms, community images,
etc.
• Reserved Instances - t2.micro = $0.006/hour - 3
year contract. Relatively cheap.
• Spot instances (bid for unused servers @ 80% of
cost).
• Can apply auto-scaling groups.
• Fairly basic...let's see what else Amazon has!

21. EC2 Availability Zones + Regions
• 11 Regions (i.e. US
East)
• 33 Availability
Zones
• us-east-1a
• us-east-1b, etc are
availability zones
while US East (N.
Virginia), US West
(Oregon), Asia
Pacific (Tokyo)

22. EC2 Regions

23. Elastic Beanstalk
• Elastic Beanstalk is a Platform as a Service
(PaaS) that handles the complexities of your
underlying infrastructure.
• Manages load balancing (via ELBs), instance
scaling, instance health monitoring, and
deployments.
• Supports Docker, Go, Java, .NET, Node, PHP,
Python, Ruby.

24. Let's see how this goes...
• Elastic Beanstalk seemed like a great solution.
Handles scaling and load balancing (with ELB).
• Ran into problems very quickly...
• MeteorJS settings are a JSON file exported as an
environment variable. Beanstalk strips
environment variables of a number of special
characters!
• Quickly realized that it is not possible to do
websockets, sticky sessions, and HTTPS with
Elastic Load Balancers.

25. How about OpsWorks?
• OpsWorks is a configuration management
service that helps you configure and operate
applications of all shapes and sizes using Chef.
• You can define the application’s architecture
and the specification of each component
including package installation, software
configuration and resources such as storage.

30. Nope!
• Stood up a MongoDB cluster and two separate
Meteor clusters.
• Autoscaling and application stack health-checks
are nice!
• Didn't bother using Chef as I have my own
deployment tools.
• Ran into problems very quickly...
• "running_setup" never ended. Servers would
hang. No support from Amazon!

31. Ok, uh, what else is left?
CloudFormation?
• CloudFormation allows you to define
infrastructure and services as a configuration
template.
• Provision servers, storage, networks, NATs,
relationships, run scripts, etc. all from a JSON
config.
• Rapid deployment of an entire stack,
extremely powerful and complicated.

34. CloudFormation
• I used the MongoDB CloudFormation
template provided by Amazon (only on github,
not via the CloudFormation interface).
• Over time I abandoned it as the scripts they
used for deployment contained a race
condition that would cause deployment to fail
about 50% of the time.
• For my purposes, this was overkill.

35. So, EC2 then?
• Setup 3 layers: (1) Load Balancers, (1) DDP
Servers, (3) Mongo Replica Sets (no shards).
• Created 5 DNS records for the load balancers to
do round robin DNS.
• Used Route 53 latency weighted routing to
failover load balancers that are offline to load
balancers that are online.
• 2 auto scaling groups, one for load balancers, one
for DDP servers.
• Lets Encrypt for SSL/TLS.

36. Route 53
• Route 53 is a highly available and scalable
DNS.Routing policies are:
– Simple Routing
– Weighted Routing
– Latency Routing
– Failover Routing

37. Down the rabbit hole…
• Auto scaling occurs when average CPU > 70%
for 2 minutes.
• When auto scaling occurs in either auto
scaling group, it creates a new instance using
the same image.
• That image runs a script on boot...

38. do_all_the_things.sh
• Step 1: git pull to get latest code.
• Step 2: Determine if an elastic IP address is
already assigned (if not, then this is a fresh
instance).
• Step 3a: If fresh instance, and DDP server, use
auto-assigned public IP.
• Step 3b: If fresh instance, and load balancer,
check to see which DNS elastic IP's are unused
and assign one.Deploy to self (build code and
deploy to localhost).
• Step 4: Done!

42. Testing Methodology
• Used MeteorDown for load testing. It connects to
an endpoint, opens up a DDP connect and
completes some action (in this case a 200kb
document transfer) before closing the
connection.
• Scale up connections incrementally and monitor
when and how fast instances are created.
• Determine total capability of 5 load balancers and
10 DDP servers (each instance is t2.micro, 1
vCPU, 512mb ram).

43. Testing Results
• First auto scaling event: provisioning of
instance, boot, git pull, build, and deploy is 2
minutes 30 seconds.
• Second+ auto scaling event: 1 minute 10
seconds.
• Concurrent subscriptions achieved: 38,000
• Maximum subscriptions per instance is
reduced by approximately 1% per added
instance.

44. What I learned…
• Meteor has very powerful tools out of the box,
however these tools do not lend themselves well
to solutions such as Elastic Beanstalk or Google
App Engine.
• DNS is a finicky mistress.
• Scaling load balancers is hard; scaling a web app
using sticky websockets is even harder!
• Diving in head first has some serious drawbacks -
it's difficult to navigate this space without
someone to guide you.

45. What did I end up with?
• One line command to do rolling deployments to all my
servers.
• 10 second downtime when deploying.
• Highly available (across AZ's), scalable architecture
(auto-scaling + MongoDB sharding/replicas).
• 38,000 concurrent connections constantly re-
establishing websockets and sending data.
• Great support for microservices.
• Reactive data and UI, hot code push, mobile
deployment.

46. What was the point of this?
• Hopefully you understand that it IS possible to
scale MeteorJS applications, and that it isn't
that hard (I just made it hard for myself).
• High level understanding of a few of Amazon's
infrastructure and deployment services.