Node.js in Production
at Aviary
NYC Node.js Meetup
March 5, 2014
Aviary
Photo-Editing
SDK & Apps
Fully-Baked UI
Configurable, High-Quality Tools
Over 6,500 Partners
Over 70 Million Monthl...
Who Are We?
Nir

Jack

Lead Serverside

Director of

Engineer

Engineering

Likes:
●
●
●

Automated deployment
Big-O notat...
Who Are We?
Jeff

Ari

Serverside

Developer

Engineer

Evangelist

Likes:
●
●
●

Performance Profiling
Spaces, not tabs
B...
How Do We Use Node?
● In Production:
○
○
○
○
○

● Future:

Analytics Dashboard
○ Server-side Rendering API
Content Deliver...
Why Do We Use Node?
●
●
●
●
●
●
●
●

Extremely fast and lightweight
Easy to iterate on
Common language for client and serv...
Setting Up
Your Server
Request Routing
Our API servers all require(routes.json)
{
“home”: {
“getVersion”: {
“verb”: “get”,
“path”: “/version”
}
}...
Authentication: Overview
Request

Server
listens

Middleware

Logged in?

No
Yes

Response

Request handler takes
over

Do...
Authentication: Login
passport.use(new GoogleStrategy({ returnURL: DOMAIN + '/auth/return' },
function (identifier, profil...
Working with
JSON
Validation - JSON Schema
SCHEMA

● JSON-based

JSON

{

{
“type”: “object”
“additionalProperties”: false
“properties”: {
“...
Advanced JSON - Content
Effects

Frames

Stickers

Messages
The One-To-Many Problem
Android expects responses to look like this:
iOS 1.0 expects responses to look like this:
iOS 2.0 ...
Response Formatting - The Model
Content Entry

Response Formats

Responses

JSON document describing
content item

JSON do...
Response Formatting - Details
The Single Content Entry
"identifier": "com.aviary.stickers.234fe"

The Response Format
"id"...
Code Sample (Dumbed Down)
var formattedResponse = {};

for (var propName in responseFormat) {
var val = contentEntry[respo...
Interacting with
External Processes
Image Rendering
Challenge: Use our existing image rendering .NET/C++
process from node server
Solution:
require(‘child_pro...
Code Sample
var spawn = require(‘child_process’).spawn;

var renderer = spawn(‘renderer.exe’, [‘-i’, ‘inputImage.jpg’, … ]...
Going Live
Testing Philosophy
● Unit tests (sparingly)
● End-to-end integration tests
● Mocha
● Enforced before push
○ (master / deve...
Example Integration Test
#!/bin/bash

● Bash script

mocha scopecreation &&
mocha cmsformatcreation &&
mocha crfcreation &...
Automated Deployment: Overview
Git

S3
2) Jenkins
polls for repo
changes

1) Code is
pushed to
master

3) Code is zipped
a...
Automated Deployment: Bootstrap
5) SSH into each
server and run
the bootstrap script

#!/bin/bash
ZIP_LOCATION="s3://aviar...
Summary
Lessons Learned (1)
● Integration tests!
● Watch out for node and npm updates
○ Hardcode the node version you’re using
○ I...
Lessons Learned (2)
● Always callback in async functions
● Always return after a callback
● Node doesn’t always run the sa...
Conclusion
Today, our production node servers:
● serve dynamic content to 20MM people (soon 70MM)
● power our website: avi...
Questions?
Comments also welcome
nir@aviary.com - jack@aviary.com - ari@aviary.com - jeff@aviary.com
…and by the way, WE’R...
Upcoming SlideShare
Loading in...5
×

Node in Production at Aviary

1,593

Published on

Aviary's customizable SDK powers cross-platform photo editing for over 6,500 partners and over 70 million monthly active users across the globe. Some of our notable partners include Walgreens, Squarespace, Yahoo Mail, Flickr, Photobucket, and Wix. At Aviary, we use node.js for several mission-critical projects in production and have seen extremely positive results. In this talk, we will discuss how we approach some common situations that developers deploying node.js projects will likely need to tackle. We will walk you through our routing mechanism, our automated deployment system, some of our custom middleware, and our testing philosophy.

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total Views
1,593
On Slideshare
0
From Embeds
0
Number of Embeds
2
Actions
Shares
0
Downloads
10
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

Node in Production at Aviary

  1. 1. Node.js in Production at Aviary NYC Node.js Meetup March 5, 2014
  2. 2. Aviary Photo-Editing SDK & Apps Fully-Baked UI Configurable, High-Quality Tools Over 6,500 Partners Over 70 Million Monthly Users Over 6 Billion Photos Edited iOS, Android, Web, Server J
  3. 3. Who Are We? Nir Jack Lead Serverside Director of Engineer Engineering Likes: ● ● ● Automated deployment Big-O notation Brainteasers Hates: ● Cilantro Likes: ● ● ● Parallelizing processes DRY code Seltzer Hates: ● Food after the sell-by date
  4. 4. Who Are We? Jeff Ari Serverside Developer Engineer Evangelist Likes: ● ● ● Performance Profiling Spaces, not tabs Bikes Hates: ● His Photo Likes: ● ● ● Empowering Developers Refactoring/Patterns Dancing Hates: ● Forrest Gump
  5. 5. How Do We Use Node? ● In Production: ○ ○ ○ ○ ○ ● Future: Analytics Dashboard ○ Server-side Rendering API Content Delivery (CDS) ○ Automated billing Public Website Receipts Collection Monitoring Tools
  6. 6. Why Do We Use Node? ● ● ● ● ● ● ● ● Extremely fast and lightweight Easy to iterate on Common language for client and server JSON Cross Platform npm express module Active Community
  7. 7. Setting Up Your Server
  8. 8. Request Routing Our API servers all require(routes.json) { “home”: { “getVersion”: { “verb”: “get”, “path”: “/version” } }, “queue”: { “updateContent”: { “verb”: ”put”, “path”: “/content/:id”, “permissions”: [“content”] } } } for (var controllerName in routes) { var controller = require(ctrlrDir + controllerName); for (var endpointName in routes[controllerName]) { var endpoint = routes[controllerName][endpointName]; var callback = controller[endpointName]; app[endpoint.verb]( endpoint.path, ensurePermissions(endpoint.permissions), callback ); } }
  9. 9. Authentication: Overview Request Server listens Middleware Logged in? No Yes Response Request handler takes over Does user have permission? Redirected to login page Authenticated user saved in cookie
  10. 10. Authentication: Login passport.use(new GoogleStrategy({ returnURL: DOMAIN + '/auth/return' }, function (identifier, profile, done) { var userInfo = { name: profile.email.value, fullName: profile.name }; userRepository.findUserByName(userInfo.name, function (findErr, foundUser) { // ... if (foundUser.length === 0) { done('Invalid user', null); return; } userInfo.userId = foundUser.user_id; userInfo.permissions = foundUser.permissions; done(null, userInfo); }); } ));
  11. 11. Working with JSON
  12. 12. Validation - JSON Schema SCHEMA ● JSON-based JSON { { “type”: “object” “additionalProperties”: false “properties”: { “status”: { “type”: “string”, “enum”: [“ok”, “error”], “required”: true }, “data”: { “type”: “object”, “required”: false } } ● Like XML Schema ● Validation modules ● Used throughout Aviary’s systems } “status”: “ok” } { “status”: “error”, “data”: { “reason”: “hoisting” } } { “status”: “gladys”, “node”: “meetup” }
  13. 13. Advanced JSON - Content Effects Frames Stickers Messages
  14. 14. The One-To-Many Problem Android expects responses to look like this: iOS 1.0 expects responses to look like this: iOS 2.0 expects responses to look like this:
  15. 15. Response Formatting - The Model Content Entry Response Formats Responses JSON document describing content item JSON documents defining mappings from entry to responses Actual JSON responses delivered to devices
  16. 16. Response Formatting - Details The Single Content Entry "identifier": "com.aviary.stickers.234fe" The Response Format "id": { "dataKey": "identifier" "icon": { "path": "cds/hats/icon.png" "path-100": "cds/hats/icon100.png" }, "isPaid": { "isPaid": true, "iconImagePath": "cds/hats/icon100.png" "stickers": [ { "type": "boolean", }, "dataKey": "isFree", { "identifier": "1" "transformations": ["negateBool"] "items": [ "imageUrl": "cds/hats/1.png" }, "identifier": "1" } "iconImagePath": { "imageUrl": "cds/hats/1.png" "type": "string", } ] "id": "com.aviary.stickers.234fe", "type": "string", "isFree": false, The Response "dataKey": "icon.path-100" }, "stickers": { "type": "array", "dataKey": "items" ]
  17. 17. Code Sample (Dumbed Down) var formattedResponse = {}; for (var propName in responseFormat) { var val = contentEntry[responseFormat[propName].dataKey]; for (var transformation in responseFormat[propName].transformations) { val = transformationModule[transformation](val); } formattedResponse[propName] = val; } return formattedResponse;
  18. 18. Interacting with External Processes
  19. 19. Image Rendering Challenge: Use our existing image rendering .NET/C++ process from node server Solution: require(‘child_process’).spawn(‘renderer.exe’) Benefits: Easy IPC, asynchronous workflow
  20. 20. Code Sample var spawn = require(‘child_process’).spawn; var renderer = spawn(‘renderer.exe’, [‘-i’, ‘inputImage.jpg’, … ]); // read text renderer.stderr.setEncoding(‘utf8’); renderer.stderr.on(‘data’, function (data) { json += data; }); // or binary data renderer.stdout.on(‘data’, function (data) { buffers.push(data); }); renderer.on(‘close’, function (code, signal) { // respond to exit code, signal (e.g. ‘SIGTERM’), process output var diagnostics = JSON.parse(json); var img = Buffer.concat(buffers); });
  21. 21. Going Live
  22. 22. Testing Philosophy ● Unit tests (sparingly) ● End-to-end integration tests ● Mocha ● Enforced before push ○ (master / development)
  23. 23. Example Integration Test #!/bin/bash ● Bash script mocha scopecreation && mocha cmsformatcreation && mocha crfcreation && ● Independent files mocha mrfcreation && mocha rflcreation && mocha appcreation && ● Shared configuration mocha contentcreation && mocha manifestcreation && mocha push && ● Single failure stops process mocha cmsformatupdate && mocha crfaddition && mocha rfladdition && mocha contentupdate && mocha manifestupdate
  24. 24. Automated Deployment: Overview Git S3 2) Jenkins polls for repo changes 1) Code is pushed to master 3) Code is zipped and uploaded to S3 Jenkins 4) Get a list of live servers in this group AWS API 5) SSH into each server and run the bootstrap script
  25. 25. Automated Deployment: Bootstrap 5) SSH into each server and run the bootstrap script #!/bin/bash ZIP_LOCATION="s3://aviary/projectX/deployment.zip"; cd ~/projectX; sudo apt-get -y -q install nodejs@0.10.0; sudo apt-get -y -q install s3cmd; sudo npm install -g forever@0.10.8; Goals of the bootstrap.sh: 1. Ensure all dependencies are installed 2. Download and extract project 3. Ensure HTTP traffic is routed to the proper port 4. Keep the old version of the project live until the moment the new one is ready to go live # Missing step: create s3 configuration file s3cmd -c /usr/local/s3cfg.config get "$ZIP_LOCATION" theCds.zip; unzip -q -o deployment.zip iptables -t nat -A PREROUTING -p tcp --dport 80 -j REDIRECT --to-ports 8142; forever stopall; forever start server.js;
  26. 26. Summary
  27. 27. Lessons Learned (1) ● Integration tests! ● Watch out for node and npm updates ○ Hardcode the node version you’re using ○ If you’re using package.json, version everything ● Node.js + MongoDb are a great couple ● Make sure you understand hoisting
  28. 28. Lessons Learned (2) ● Always callback in async functions ● Always return after a callback ● Node doesn’t always run the same on all platforms ● Use middleware only when necessary ● Always store dates as Unix Timestamps ○ Timezones are a pain in your future ● Throwing unhandled errors will crash your process
  29. 29. Conclusion Today, our production node servers: ● serve dynamic content to 20MM people (soon 70MM) ● power our website: aviary.com ● log real-time receipt data for every in-app purchase ● allow us to analyze hundreds of millions of events daily ● power quick scripts and one-off internal tools
  30. 30. Questions? Comments also welcome nir@aviary.com - jack@aviary.com - ari@aviary.com - jeff@aviary.com …and by the way, WE’RE HIRING!
  1. ¿Le ha llamado la atención una diapositiva en particular?

    Recortar diapositivas es una manera útil de recopilar información importante para consultarla más tarde.

×