AtlasCamp 2014: Building a Production Ready Connect Add-On


Published on

Atlassian Connect add-ons are SaaS applications. Building and running them means planning with operations in mind: where should you host your add-on? What's the best way to deploy it? How can you monitor it once it's live? How much will it cost to run? We'll draw from Atlassian's experiences building Who's Looking for OnDemand (a production Connect add-on installed in 950 instances with 20,000 active users) to explore tips and best practices to help answer these questions and more.

Published in: Technology
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

AtlasCamp 2014: Building a Production Ready Connect Add-On

  1. 1. June 3-5, 2014 | Berlin, Germany
  2. 2. Robin Fernandes, Dev Team Lead, Atlassian Building a Production- Ready Connect Add-On
  3. 3. Who’s Looking? 19500 users 950 hosts
  4. 4. Overview / Stack × many! × many! Who’s Looking? Multi-tenancy Scalability
  5. 5. What Connect Provides Tenant registration Authentication & tenant context Webhooks Integrated look & feel Local dev & test … Handled by Connect!
  6. 6. A Production-Ready Add-On Hosting Upgrading Scaling Monitoring … Lots of decisions…
  7. 7. Hosting
  8. 8. Stateless Request Processing Load Balancer Data Store Hosting on a PaaS can encourage statelessness, which enables horizontal scalability and improves resilience. Requests app processes
  9. 9. Rich ecosystem of external services 3rd Party Services
  10. 10. Basic deployment is just a git push Low Barrier to Entry > git push heroku master Free tiers allow you to experiment at no cost
  11. 11. Testing
  12. 12. Testing Unit testing: business as usual Browser tests: similar to in-process add-ons Run add-on and host app locally Atlassian SDK, WebDriver, Atlassian Page Objects… Add post-deployment verification tests Reliable sanity tests, run after every deployment Must be runnable against a production instance
  13. 13. Testing & Muti-tenancy Mock out products’ REST APIs to test multi-tenant interactions (see Who’s Looking source on Bitbucket for examples) Connect add-on
  14. 14. Load Testing Load test on free tier, and extrapolate Use e.g. New Relic to monitor the add-on under load
  15. 15. Load Testing Tool: ac-load-test Sets up Gatling-based load tests for Connect add-ons Simulates authenticated requests from 1000s of hosts & users Connect add-on ac-mock-tenants> ./sbt run ac-load-test (Gatling) heavy load Mock OD instances JWT-signed requests
  16. 16. Load Testing > ./sbt run … ================================================================================ 2014-05-17 17:36:22 20s elapsed ---- A mixture of iframe and XHR requests -------------------------------------- [--------------------------------------------------------------------------] 0% waiting: 0 / running: 200 / done:0 ---- Requests ------------------------------------------------------------------ > Global (OK=899 KO=0 ) > iframe-request (OK=200 KO=0 ) > token-based-xhr-request (OK=699 KO=0 ) ================================================================================ …
  17. 17. Deployment
  18. 18. Staged Deployments whoslooking-dev. dev run PDV tests deploy
  19. 19. Staged Deployments whoslooking-stg. stg run PDV tests promote from dev to stg (using pipelines) Green dev deployment? + soak time
  20. 20. Staged Deployments whoslooking. run PDV tests All good on stg? prod promote from stg to prod (using pipelines)
  21. 21. Further Precautions the goggles…
  22. 22. Things can still go wrong! Log management Set up alerts on log patterns, search logs Add-ons: Logentries, FlyData, Papertrail… Feature flags Turn new features on & off via configuration Metrics / analytics Get insight into application behaviour to detect issues Libraries for in-memory metrics: Simon, Metrics, Servo… Add-ons for externalised analytics: New Relic, StatsMix, Hosted Graphite…
  23. 23. Healthcheck & Monitoring Implement a Health Check REST resource that runs sanity-checks & reports results Monitor it (Pingdom, New Relic…) Example: (Checks Postgres & Redis connections, reports config settings, resource usage, weekly/daily active users/hosts)
  24. 24. Healthcheck & Monitoring
  25. 25. To build a production-ready add-on… Design for multi-tenancy and horizontal scalability Ensure deploying is easy Make the most of 3rd party services Load test to help estimate running cost Implement a health check resource & monitor it Run post-deployment verification tests Start simple!
  26. 26. Who’s Looking Stats 2-4 dynos (average ~3) 5 external services (3 free, 2 paid) 19800 active users last week 960 active instances last week 3500 average rpm (~7500 peak) <1s average page load time ! $89.50 / month (~$0.005 / user / month) Slides: