• Share
  • Email
  • Embed
  • Like
  • Save
  • Private Content
Tuenti Release Workflow v1.1
 

Tuenti Release Workflow v1.1

on

  • 2,054 views

At Tuenti, we do 3 code pushes per week, sometimes modifying thousands of files and running thousands of automated tests and build operations before, to ensure not only that the code works but also ...

At Tuenti, we do 3 code pushes per week, sometimes modifying thousands of files and running thousands of automated tests and build operations before, to ensure not only that the code works but also that proper localization is applied, bundles are generated and files get deployed to hundreds of servers as fast and reliable as possible.

We use opensource tools like Mercurial, MySQL, Jenkins, Selenium, PHPUnit and Rsync among our own in-house ones, and have different development, testing, staging and production environments.
We had to fight with problems like statics bundling and versioning, syntax errors and of course the fact that we have +100 engineers working on the codebase, merging and releasing more than a 15 branches the same day. We also switched from Subversion to Mercurial to obtain more flexibility and faster branching operations.
With this talk we will explain the process of how code changes in ourcode repository end up in live code, detailing some practices and tips that we apply, problems we had and how we solved them.

Statistics

Views

Total Views
2,054
Views on SlideShare
2,052
Embed Views
2

Actions

Likes
4
Downloads
0
Comments
0

2 Embeds 2

http://us-w1.rockmelt.com 1
http://ristrettonews.com 1

Accessibility

Categories

Upload Details

Uploaded via as Microsoft PowerPoint

Usage Rights

© All Rights Reserved

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment
  • TDD: Backend nearer, FE hard once you enter visual tests (acceptance)
  • Monday: Too much trafficFriday: Weekend next day, safer not to just in case something happens. (First redesign story)
  • Shared Gdocs spreadsheet in which QA add bugs and engineers check and mark
  • We use Singletons. PHP running PHP and thus keeping things between test batteries is the problem.We’re working on adding more testeability features to the FEFW
  • Yes, we use Singletons. the problem is PHP running PHP and thus keeping things between test batteriesWe’re working on adding more testeability features to the FEFW
  • We use XEN (http://xen.org/) for Windows virtualization on Jenkins buildsNodes are not virtualized because 20% less performance
  • We use the Percona server instead of vanilla MySQLMock everything (unit/integration)Reuse data if possible (browser)
  • Recommendation: Build one cluster, experiment, learn, use it. Then build another one better tunned with whatever you learned
  • Explain the 0,3 WTF
  • Indexed in [5,15] min normal scenario, worst case 1h max/limit
  • Old was a SH + PHP non-multithreaded script
  • JS & CSS
  • JS & CSS
  • Solution: easy chat server tutorial. Now all Comms team engineers have their own VM chat server setupNow resharded to 2 chat clusters with LB and 12 machines each (thanks to improvements)

Tuenti Release Workflow v1.1 Tuenti Release Workflow v1.1 Presentation Transcript

  • Release WorkflowDiego Muñozdiego@tuenti.comhttp://twitter.com/Kartones V1.1
  • Agenda• Numbers• Release workflow• Tools
  • Numbers• +13M users (~10,4M active users dec 2011)• +100 usage minutes / day (avg)• +400M chat messages / day• +4M photos uploaded / day (peaks)• +41,000M page views / month• +35K requests / sec (peaks)• +1,3K servers• +250 employees (~60% techies)• +15K files in the repositories• +10K Tests
  • Release WorkflowBranch Code Test Integrate Release Stabilize
  • Release Workflow: BranchBranch Code Test Integrate Release Stabilize• Avg. 15 branches per release• Current record: 29 branches• Repository per functional area (be, fe, stats, …)• Avg. lines modified per release: 63K
  • Release Workflow: Code + TestBranch Code Test Integrate Release Stabilize• Scrum (or at least Agile)• As TDD as possible• Labs• A/B Testing• PoCs• Dark launch
  • Release Workflow: IntegrateBranch Code Test Integrate Release Stabilize• Repo always available• Specific release date given by devops – Merge & wait for target• Only merge if 100% tests ok or specific approval• QA Regression & manual tests• Fix possible integration problems ASAP
  • Release Workflow: ReleaseBranch Code Test Integrate Release Stabilize• 3 releases per week – DevOps goal: All weekdays• Latest stable changeset from Integration taken previous working day morning• Release doc, pre-release meetings• Staging servers to test with live data
  • Release Workflow: StabilizeBranch Code Test Integrate Release Stabilize• First code push: 8 AM – DevOps Goal: single push + release closed• Release window: 1-2 h – DevOps goal: < 30 minutes• Error stabilization or release rollback• Representatives from all involved teams
  • Tools
  • DVCS: Mercurial• http://mercurial.selenic.com/• Syntax similar to SVN (our old system)• Easy API to plug our plugins and hooks• Cross-platform• Tuenti Addons: – Commit hooks to check syntax, push ticket #...• Problems: – Push/pulls through VPN are slow – Handling multiple repos still slow – Only one level of rollback!
  • Issue Tracking: Trac• http://trac.edgewall.org/• User Stories tasks + Bugs• Wiki (now also internal Google Sites)• Plugins and extensible• Tuenti Addons: – Master/Slave architecture – Tons of tweaks and source code integration hooks• Problems: – Slow, limited, code viewing sucks• Migration to JIRA planned
  • Testing: PHPUnit• http://www.phpunit.de• Some caveats – Mocking just „works‟ – PHP process spawning PHP tests• Tuenti Addons: – Vastly improved mocking framework – Shell scripts that isolate test batteries – Better integration with Selenium• Problems: – Our current FEFW does not cope perfectly with PHPUnit/Selenium
  • Testing: Selenium• http://seleniumhq.org/• Running browser tests in FFox and IE• Tuenti Addons: – Custom build with some fixes• Problems: – Javascript handling/detection not perfect – AJAX far from optimal – IE runner is an iframe• Planned migration to Webdriver
  • CI: Jenkins• http://jenkins-ci.org/• Previously Hudson too• Specialized farm (master + 22 nodes)• Tuenti Addons: • Parallelization (up to 6 nodes) • Special reports • “Smart” runs (try first last failed tests, etc.)• Problems: • Browser tests slow (due to Selenium) • Unstable (mainly due to Selenium)
  • Storage: MySQL• http://www.mysql.com/ | http://www.percona.com• Live site storage• Dev. env. storage – 1 DB per user (to run tests) – 1 shared DB (common faked data)• Clusters of master/slave DBs• Problems: – Slow when running tests – Shared dev DB has old-time inconsistencies
  • Storage: Hadoop• http://hadoop.apache.org/• Dedicated cluster• Pig scripts: Stats, other non-realtime data• HBase: Async. data storage• Hive: SQL-like querying• Problems: – Complex configuration for newcomers
  • Caching: Memcached• http://memcached.org/• Avg. DB querys/pageview: 0.3• Dev. Behaviour == live behaviour• Tuenti Addons (https://github.com/tuenti): – UDP + multi-ports• Problems: – 32GB RAM / machine practical limit – Remember to warm-up data or MC will kill the DB!
  • Configuration: Puppet• http://puppetlabs.com/• Production machines• Jenkins nodes• VM management / Dev web servers config• Problems: – Wipes user config if not puppetized
  • Search: Sphinx• http://sphinxsearch.com/• Non-realtime (index based)• Very fast• Problems – Index re-generation on dev & test env. – Could be more friendly to add new data
  • Build: Our build script• http://ant.apache.org/• Localization• Minification + Bundling + Versioning• Statics deployment to CDNs• Fast: 2-3 minutes full build – Multithreading + parallelization• Allows partial, component based builds• Problems: – Under heavy CPU load, build time goes up :(
  • Build: RSync• http://rsync.samba.org/• Deployment of code (live & dev)• Sends deltas/diffs• Really fast
  • Statics bundling: YUI• Less files == faster download & deploy• Big text file == better HTTP Gzip• HTML/JS/CSS Minification• Ultra fast build: ~4 seconds in Dev.• On demand JS loading!• Nice typical framework features• Wonderful & simple events system
  • Statics bundling: YUI (II)• Tuenti Addons: – Caching builds – Line breaks each # characters (easier IE debugging) – CDNs handling• Problems: – Change in JS requires rebuilding even in dev. – Requires small migrations/changes in existing JS
  • Chat Server: Ejabberd• http://www.ejabberd.im/• Erlang XMPP (Jabber) server• 400M msgs/day, 1M concurrent users peak,…• 20 machines, ~5 instances per machine• Tuenti Addons: – Ejjabberd codebase tweaked (3,5x faster) – Protocol tweaks to optimize for our architecture• Problems: – Same behaviour dev/live is critical
  • Compilation: HipHop• Migrating old code to fully support HipHop – With PHP 5.3• Obvious speed improvements• Also nice for static code analysis
  • exit(0); Sounds interesting?http://jobs.tuenti.com