June 3-5, 2014 | Berlin, Germany
Robin Fernandes, Dev Team Lead, Atlassian
Building a Production-
Ready Connect Add-On
Who’s
Looking?
19500 users
950 hosts
Overview / Stack
× many!
× many!
Who’s Looking?
Multi-tenancy
Scalability
https://bitbucket.org/atlassian/whoslooking-conn...
What Connect Provides
Tenant registration
Authentication & tenant context
Webhooks
Integrated look & feel
Local dev & test...
A Production-Ready Add-On
Hosting
Upgrading
Scaling
Monitoring
…
Lots of decisions…
Hosting
Stateless Request Processing
Load Balancer
Data Store
Hosting on a PaaS can encourage
statelessness, which enables
horizon...
Rich ecosystem of
external services
3rd Party Services
Basic deployment is just a git push
Low Barrier to Entry
> git push heroku master
Free tiers allow you to experiment at no...
Testing
Testing
Unit testing: business as usual
Browser tests: similar to in-process add-ons
Run add-on and host app locally
Atlas...
Testing & Muti-tenancy
Mock out products’ REST APIs to test multi-tenant interactions
(see Who’s Looking source on Bitbuck...
Load Testing
Load test on free tier, and extrapolate
Use e.g. New Relic to monitor the add-on under load
Load Testing
Tool: ac-load-test
Sets up Gatling-based load tests for Connect add-ons
Simulates authenticated requests from...
Load Testing
> ./sbt run
…
================================================================================
2014-05-17 17:...
Deployment
Staged Deployments
whoslooking-dev.
herokuapp.com
dev
run PDV tests
deploy
Staged Deployments
whoslooking-stg.
herokuapp.com
stg
run PDV tests
promote from
dev to stg
(using pipelines)
Green dev
de...
Staged Deployments
whoslooking.
herokuapp.com
run PDV tests
All good on stg?
prod
promote from
stg to prod
(using pipeline...
Further Precautions
the goggles…
Things can still go wrong!
Log management
Set up alerts on log patterns, search logs
Add-ons: Logentries, FlyData, Papertr...
Healthcheck & Monitoring
http://whoslooking.herokuapp.com/healthcheck
Implement a Health Check
REST resource that runs san...
Healthcheck & Monitoring
To build a production-ready add-on…
Design for multi-tenancy and horizontal scalability
Ensure deploying is easy
Make the ...
Who’s Looking Stats
2-4 dynos (average ~3)
5 external services (3 free, 2 paid)
19800 active users last week
960 active in...
Upcoming SlideShare
Loading in …5
×

AtlasCamp 2014: Building a Production Ready Connect Add-on

549 views
477 views

Published on

Atlassian Connect add-ons are SaaS applications. Building and running them means planning with operations in mind: where should you host your add-on? What's the best way to deploy it? How can you monitor it once it's live? How much will it cost to run? We'll draw from Atlassian's experiences building Who's Looking for OnDemand (a production Connect add-on installed in 750 instances with 15,000 active users) to explore tips and best practices to help answer these questions and more.

Published in: Technology
0 Comments
0 Likes
Statistics
Notes
  • Be the first to comment

  • Be the first to like this

No Downloads
Views
Total views
549
On SlideShare
0
From Embeds
0
Number of Embeds
224
Actions
Shares
0
Downloads
7
Comments
0
Likes
0
Embeds 0
No embeds

No notes for slide

AtlasCamp 2014: Building a Production Ready Connect Add-on

  1. 1. June 3-5, 2014 | Berlin, Germany
  2. 2. Robin Fernandes, Dev Team Lead, Atlassian Building a Production- Ready Connect Add-On
  3. 3. Who’s Looking? 19500 users 950 hosts
  4. 4. Overview / Stack × many! × many! Who’s Looking? Multi-tenancy Scalability https://bitbucket.org/atlassian/whoslooking-connect
  5. 5. What Connect Provides Tenant registration Authentication & tenant context Webhooks Integrated look & feel Local dev & test … Handled by Connect!
  6. 6. A Production-Ready Add-On Hosting Upgrading Scaling Monitoring … Lots of decisions…
  7. 7. Hosting
  8. 8. Stateless Request Processing Load Balancer Data Store Hosting on a PaaS can encourage statelessness, which enables horizontal scalability and improves resilience. Requests app processes
 (dynos) whoslooking.herokuapp.com
  9. 9. Rich ecosystem of external services 3rd Party Services
  10. 10. Basic deployment is just a git push Low Barrier to Entry > git push heroku master Free tiers allow you to experiment at no cost
  11. 11. Testing
  12. 12. Testing Unit testing: business as usual Browser tests: similar to in-process add-ons Run add-on and host app locally Atlassian SDK, WebDriver, Atlassian Page Objects… Add post-deployment verification tests Reliable sanity tests, run after every deployment Must be runnable against a production instance
  13. 13. Testing & Muti-tenancy Mock out products’ REST APIs to test multi-tenant interactions (see Who’s Looking source on Bitbucket for examples) ac-mock-tenants.herokuapp.com Connect add-on
  14. 14. Load Testing Load test on free tier, and extrapolate Use e.g. New Relic to monitor the add-on under load
  15. 15. Load Testing Tool: ac-load-test Sets up Gatling-based load tests for Connect add-ons Simulates authenticated requests from 1000s of hosts & users Connect add-on ac-mock-tenants> ./sbt run ac-load-test (Gatling) heavy load bitbucket.org/atlassianlabs/ac-load-test.g8 Mock OD instances JWT-signed requests
  16. 16. Load Testing > ./sbt run … ================================================================================ 2014-05-17 17:36:22 20s elapsed ---- A mixture of iframe and XHR requests -------------------------------------- [--------------------------------------------------------------------------] 0% waiting: 0 / running: 200 / done:0 ---- Requests ------------------------------------------------------------------ > Global (OK=899 KO=0 ) > iframe-request (OK=200 KO=0 ) > token-based-xhr-request (OK=699 KO=0 ) ================================================================================ … bitbucket.org/atlassianlabs/ac-load-test.g8
  17. 17. Deployment
  18. 18. Staged Deployments whoslooking-dev. herokuapp.com dev run PDV tests deploy
  19. 19. Staged Deployments whoslooking-stg. herokuapp.com stg run PDV tests promote from dev to stg (using pipelines) Green dev deployment? + soak time
  20. 20. Staged Deployments whoslooking. herokuapp.com run PDV tests All good on stg? prod promote from stg to prod (using pipelines)
  21. 21. Further Precautions the goggles…
  22. 22. Things can still go wrong! Log management Set up alerts on log patterns, search logs Add-ons: Logentries, FlyData, Papertrail… Feature flags Turn new features on & off via configuration Metrics / analytics Get insight into application behaviour to detect issues Libraries for in-memory metrics: Simon, Metrics, Servo… Add-ons for externalised analytics: New Relic, StatsMix, Hosted Graphite…
  23. 23. Healthcheck & Monitoring http://whoslooking.herokuapp.com/healthcheck Implement a Health Check REST resource that runs sanity-checks & reports results Monitor it (Pingdom, New Relic…) Example: (Checks Postgres & Redis connections, reports config settings, resource usage, weekly/daily active users/hosts)
  24. 24. Healthcheck & Monitoring
  25. 25. To build a production-ready add-on… Design for multi-tenancy and horizontal scalability Ensure deploying is easy Make the most of 3rd party services Load test to help estimate running cost Implement a health check resource & monitor it Run post-deployment verification tests Start simple!
  26. 26. Who’s Looking Stats 2-4 dynos (average ~3) 5 external services (3 free, 2 paid) 19800 active users last week 960 active instances last week 3500 average rpm (~7500 peak) <1s average page load time ! $89.50 / month (~$0.005 / user / month) Slides: go.atlassian.com/prod-connect robin@atlassian.com / @rewbs

×