At Tuenti, we do 3 code pushes per week, sometimes modifying thousands of files and running thousands of automated tests and build operations before, to ensure not only that the code works but also that proper localization is applied, bundles are generated and files get deployed to hundreds of servers as fast and reliable as possible. We use opensource tools like Mercurial, MySQL, Jenkins, Selenium, PHPUnit and Rsync among our own in-house ones, and have different development, testing, staging and production environments. We had to fight with problems like statics bundling and versioning, syntax errors and of course the fact that we have +100 engineers working on the codebase, merging and releasing more than a 15 branches the same day. We also switched from Subversion to Mercurial to obtain more flexibility and faster branching operations. With this talk we will explain the process of how code changes in ourcode repository end up in live code, detailing some practices and tips that we apply, problems we had and how we solved them.

1. Release Workflow
Diego Muñoz
diego@tuenti.com
http://twitter.com/Kartones
V1.1

2. Agenda
• Numbers
• Release workflow
• Tools

3. Numbers
• +13M users (~10,4M active users dec 2011)
• +100 usage minutes / day (avg)
• +400M chat messages / day
• +4M photos uploaded / day (peaks)
• +41,000M page views / month
• +35K requests / sec (peaks)
• +1,3K servers
• +250 employees (~60% techies)
• +15K files in the repositories
• +10K Tests

4. Release Workflow
Branch Code Test Integrate Release Stabilize

5. Release Workflow: Branch
Branch Code Test Integrate Release Stabilize
• Avg. 15 branches per release
• Current record: 29 branches
• Repository per functional area (be, fe, stats, …)
• Avg. lines modified per release: 63K

6. Release Workflow: Code + Test
Branch Code Test Integrate Release Stabilize
• Scrum (or at least Agile)
• As TDD as possible
• Labs
• A/B Testing
• PoCs
• Dark launch

7. Release Workflow: Integrate
Branch Code Test Integrate Release Stabilize
• Repo always available
• Specific release date given by devops
– Merge & wait for target
• Only merge if 100% tests ok or specific approval
• QA Regression & manual tests
• Fix possible integration problems ASAP

8. Release Workflow: Release
Branch Code Test Integrate Release Stabilize
• 3 releases per week
– DevOps goal: All weekdays
• Latest stable changeset from Integration taken
previous working day morning
• Release doc, pre-release meetings
• Staging servers to test with live data

9. Release Workflow: Stabilize
Branch Code Test Integrate Release Stabilize
• First code push: 8 AM
– DevOps Goal: single push + release closed
• Release window: 1-2 h
– DevOps goal: < 30 minutes
• Error stabilization or release rollback
• Representatives from all involved teams

10. Tools

11. DVCS: Mercurial
• http://mercurial.selenic.com/
• Syntax similar to SVN (our old system)
• Easy API to plug our plugins and hooks
• Cross-platform
• Tuenti Addons:
– Commit hooks to check syntax, push ticket #...
• Problems:
– Push/pulls through VPN are slow
– Handling multiple repos still slow
– Only one level of rollback!

12. Issue Tracking: Trac
• http://trac.edgewall.org/
• User Stories tasks + Bugs
• Wiki (now also internal Google Sites)
• Plugins and extensible
• Tuenti Addons:
– Master/Slave architecture
– Tons of tweaks and source code integration hooks
• Problems:
– Slow, limited, code viewing sucks
• Migration to JIRA planned

13. Testing: PHPUnit
• http://www.phpunit.de
• Some caveats
– Mocking just „works‟
– PHP process spawning PHP tests
• Tuenti Addons:
– Vastly improved mocking framework
– Shell scripts that isolate test batteries
– Better integration with Selenium
• Problems:
– Our current FEFW does not cope perfectly with
PHPUnit/Selenium

14. Testing: Selenium
• http://seleniumhq.org/
• Running browser tests in FFox and IE
• Tuenti Addons:
– Custom build with some fixes
• Problems:
– Javascript handling/detection not perfect
– AJAX far from optimal
– IE runner is an iframe
• Planned migration to Webdriver

15. CI: Jenkins
• http://jenkins-ci.org/
• Previously Hudson too
• Specialized farm (master + 22 nodes)
• Tuenti Addons:
• Parallelization (up to 6 nodes)
• Special reports
• “Smart” runs (try first last failed tests, etc.)
• Problems:
• Browser tests slow (due to Selenium)
• Unstable (mainly due to Selenium)

16. Storage: MySQL
• http://www.mysql.com/ | http://www.percona.com
• Live site storage
• Dev. env. storage
– 1 DB per user (to run tests)
– 1 shared DB (common faked data)
• Clusters of master/slave DBs
• Problems:
– Slow when running tests
– Shared dev DB has old-time inconsistencies

17. Storage: Hadoop
• http://hadoop.apache.org/
• Dedicated cluster
• Pig scripts: Stats, other non-realtime data
• HBase: Async. data storage
• Hive: SQL-like querying
• Problems:
– Complex configuration for newcomers

18. Caching: Memcached
• http://memcached.org/
• Avg. DB querys/pageview: 0.3
• Dev. Behaviour == live behaviour
• Tuenti Addons (https://github.com/tuenti):
– UDP + multi-ports
• Problems:
– 32GB RAM / machine practical limit
– Remember to warm-up data or MC will kill the DB!

19. Configuration: Puppet
• http://puppetlabs.com/
• Production machines
• Jenkins nodes
• VM management / Dev web servers config
• Problems:
– Wipes user config if not puppetized

20. Search: Sphinx
• http://sphinxsearch.com/
• Non-realtime (index based)
• Very fast
• Problems
– Index re-generation on dev & test env.
– Could be more friendly to add new data

21. Build: Our build script
• http://ant.apache.org/
• Localization
• Minification + Bundling + Versioning
• Statics deployment to CDNs
• Fast: 2-3 minutes full build
– Multithreading + parallelization
• Allows partial, component based builds
• Problems:
– Under heavy CPU load, build time goes up :(

22. Build: RSync
• http://rsync.samba.org/
• Deployment of code (live & dev)
• Sends deltas/diffs
• Really fast

23. Statics bundling: YUI
• Less files == faster download & deploy
• Big text file == better HTTP Gzip
• HTML/JS/CSS Minification
• Ultra fast build: ~4 seconds in Dev.
• On demand JS loading!
• Nice typical framework features
• Wonderful & simple events system

24. Statics bundling: YUI (II)
• Tuenti Addons:
– Caching builds
– Line breaks each # characters (easier IE debugging)
– CDNs handling
• Problems:
– Change in JS requires rebuilding even in dev.
– Requires small migrations/changes in existing JS

25. Chat Server: Ejabberd
• http://www.ejabberd.im/
• Erlang XMPP (Jabber) server
• 400M msgs/day, 1M concurrent users peak,…
• 20 machines, ~5 instances per machine
• Tuenti Addons:
– Ejjabberd codebase tweaked (3,5x faster)
– Protocol tweaks to optimize for our architecture
• Problems:
– Same behaviour dev/live is critical

26. Compilation: HipHop
• Migrating old code to fully support HipHop
– With PHP 5.3
• Obvious speed improvements
• Also nice for static code analysis

27. exit(0);
Sounds interesting?
http://jobs.tuenti.com