Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Building a PaaS at HubSpot

1,650 views

Published on

How we migrated the HubSpot product from legacy hardware to a Mesos-based Platform-a-a-Service system.

Published in: Engineering
  • Be the first to comment

Building a PaaS at HubSpot

  1. 1. Building a PaaS at HubSpot Tom Petr — @tpetr
  2. 2. Blogging SEO Social Media CMS Lead Management Landing Pages Calls-to-Action Marketing Automation Email Analytics CRM Sync
  3. 3. Over 100 engineers Over 1,600 deployables ~300 daily deploys From git push to Production in about 10 minutes
  4. 4. Engineers own the end-to-end success of their products. They make most tech decisions; they wear the pagers. We don’t have an “ops” team.
  5. 5. 1. Develop locally 2. Provision QA hardware 3. Deploy via local Python script 4. Provision PROD hardware 5. Deploy via local Python script … 6. Replace hardware at 4am
  6. 6. PaaS is all about empowering engineers. Give them good tools and a solid foundation so they can focus on what they do best.
  7. 7. First Mesos cluster Sept 2013
  8. 8. Abstracts away machines. Promotes homogenous environment. Ability to scale out specific processes. Centralized service registry.
  9. 9. Singularity Oct 2013
  10. 10. Web services Background workers Cron jobs One-off tasks
  11. 11. Thomass-MacBook-Pro:~ tpetr$ logtail -r BlazarService-web Fetching tasks ctrl+c to exit 67.192.253.80 - - [21/Aug/2015:00:58:28 +0000] "GET /blazar/v1 HTTP/1.0 67.192.253.80 - - [21/Aug/2015:00:58:28 +0000] "GET /blazar/v1/ HTTP/1. 67.192.253.80 - - [21/Aug/2015:00:58:28 +0000] "GET /blazar/v1/static/c 67.192.253.80 - - [21/Aug/2015:00:58:28 +0000] "GET /blazar/v1/static/c 67.192.253.80 - - [21/Aug/2015:00:58:28 +0000] "GET /blazar/v1/static/j 67.192.253.80 - - [21/Aug/2015:00:58:30 +0000] "GET /blazar/v1/static/i 67.192.253.80 - - [21/Aug/2015:00:58:30 +0000] "GET /blazar/v1/%3C/blaz 67.192.253.80 - - [21/Aug/2015:00:58:30 +0000] "GET /blazar/v1/static/i 67.192.253.80 - - [21/Aug/2015:00:58:30 +0000] "GET /blazar/v1/static/i
  12. 12. Baragon Load Balancer API Nov 2013
  13. 13. Amazon ELB NGINX NGINX NGINX NGINX Baragon Singularity
  14. 14. QA Migration Dec 2013 to June 2014
  15. 15. x ∞
  16. 16. Local filesystem state singleProcessRunning Stationary hosts Memory isolation
  17. 17. 💥 💥
  18. 18. Max heap + (Stack size * expected max threads) + (GC overhead pct * max heap) + JVM Overhead + Extra off-Heap Memory = No More OOMs!
  19. 19. Biggest fear: Inconsistency
  20. 20. “My service will be the last to move to Mesos PROD.”
  21. 21. Singularity Baragon hubapi.com Agents “Add host(s) to hubapi.com” “Request XX is WAITING” “Update config” “Applied successfully” “Status of request XX?” “Request XX is WAITING” “Status of request XX?” “Request XX is SUCCESS”
  22. 22. PROD QA ✨ TEST ✨
  23. 23. Rebranding with “v2”
  24. 24. The Amazon Rebootpocalypse Sept 2014
  25. 25. PROD Migration Feb 2015
  26. 26. #
  27. 27. “But does this all work?”
  28. 28. Yes, it powers the HubSpot product. (and other companies too!)
  29. 29. 8,000,000 tasks launched in 2015
  30. 30. No critical situations caused by infrastructure.
  31. 31. PROD Machines 0 200 400 600 800 May 2013 Dec 2013 Jul 2014 Feb 2015 App Servers Mesos Slaves PROD migrated Singularity first deploy
  32. 32. Rainmaker Users 0 10 20 30 40 50 May 2013 Dec 2013 Jul 2014 Feb 2015 PROD migrated
  33. 33. Wormhole Client-side load balancing Sept 2014
  34. 34. Avoid hitting load balancers Preemptively translates URLs into actual hosts
  35. 35. Facsimile Near-realtime email automation + intelligence July 2015
  36. 36. Persistent IMAP connections to 200k+ inboxes Spread over 60 instances
  37. 37. Ghidorah Mesos-based Load Balancers Jan 2015
  38. 38. NGINX + Baragon Agent in a Docker image Hosted in Singularity Absorbed 40 standalone instances in QA
  39. 39. Blazar An out-of-this-world build system!™ July 2015
  40. 40. GitHub Webhooks Singularity Heroku-like buildpacks
  41. 41. TL;DR We run our product on Mesos, and you should too.
  42. 42. We invested in a custom scheduler because we started early.
  43. 43. CTA!
  44. 44. Try Singularity today! github.com/HubSpot/Singularity

×