Your SlideShare is downloading. ×
0
Tuenti Release Workflow
Tuenti Release Workflow
Tuenti Release Workflow
Tuenti Release Workflow
Tuenti Release Workflow
Tuenti Release Workflow
Tuenti Release Workflow
Tuenti Release Workflow
Tuenti Release Workflow
Tuenti Release Workflow
Tuenti Release Workflow
Tuenti Release Workflow
Tuenti Release Workflow
Tuenti Release Workflow
Tuenti Release Workflow
Tuenti Release Workflow
Tuenti Release Workflow
Tuenti Release Workflow
Tuenti Release Workflow
Tuenti Release Workflow
Tuenti Release Workflow
Tuenti Release Workflow
Tuenti Release Workflow
Tuenti Release Workflow
Tuenti Release Workflow
Tuenti Release Workflow
Tuenti Release Workflow
Tuenti Release Workflow
Tuenti Release Workflow
Tuenti Release Workflow
Upcoming SlideShare
Loading in...5
×

Thanks for flagging this SlideShare!

Oops! An error has occurred.

×
Saving this for later? Get the SlideShare app to save on your phone or tablet. Read anywhere, anytime – even offline.
Text the download link to your phone
Standard text messaging rates apply

Tuenti Release Workflow

2,545

Published on

At Tuenti, we do two code pushes per week, sometimes modifying thousands of files and running thousands of automated tests and build operations before, to ensure not only that the code works but also …

At Tuenti, we do two code pushes per week, sometimes modifying thousands of files and running thousands of automated tests and build operations before, to ensure not only that the code works but also that proper localization is applied, bundles are generated and files get deployed to hundreds of servers as fast and reliable as possible.

We use opensource tools like Mercurial, MySQL, Jenkins, Selenium, PHPUnit and Rsync among our own in-house ones, and have different development, testing, staging and production environments.
We had to fight with problems like statics bundling and versioning, syntax errors and of course the fact that we have +100 engineers working on the codebase, sometimes merging and releasing more than a dozen branches the same day. We also switched from Subversion to Mercurial to obtain more flexibility and faster branching operations.
With this talk we will explain the process of how code changes in ourcode repository end up in live code, detailing some practices and tips that we apply.

Published in: Technology
0 Comments
5 Likes
Statistics
Notes
  • Be the first to comment

No Downloads
Views
Total Views
2,545
On Slideshare
0
From Embeds
0
Number of Embeds
0
Actions
Shares
0
Downloads
0
Comments
0
Likes
5
Embeds 0
No embeds

Report content
Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
No notes for slide
  • TDD: Backend nearer, FE hard once you enter visual tests (acceptance)
  • Monday: Too much trafficFriday: Weekend next day, safer not to just in case something happens. (First redesign story)
  • Shared Gdocs spreadsheet in which QA add bugs and engineers check and mark
  • Partially due to our architecture: We’re working on master-slave architectures to make writes on master but reads on slaves, failover…
  • Yes, we use Singletons. the problem is PHP running PHP and thus keeping things between test batteriesWe’re working on adding more testeability features to the FEFW
  • We use XEN (http://xen.org/) for Windows virtualization on Jenkins buildsNodes are not virtualized because 20% less performance
  • Mock everything (unit/integration)Reuse data if possible (acceptance)
  • Now all dev laptops have 8GB RAM and old ones can ask for an upgrade
  • Indexed in 5-15 min normal scenario, worst case 1h max/limit
  • UDP + random ports instead of TCP biggest win
  • JS & CSS
  • IE 32 CSS files limitation best example of dev. Only problemsSplit in multiple lines instead of just one (ease IE debugging)
  • Migration from single SH + PHP script to Ant parallelized script almost done. Build time: 1 minute!
  • Transcript

    • 1. Release WorkflowDiego Muñozdiego@tuenti.comhttp://twitter.com/Kartones
    • 2. Agenda• Some numbers• Release workflow• Some tools• Some rants
    • 3. Some numbers• +12M users• +100 usage minutes / day (avg)• +200M chat messages / day• +4M photos uploaded / day (peaks)• +40,000M page views / month• +35K requests / sec (peaks)• +1K servers• +250 employees (~60% techies)• +15K files in the repositories• +8K Tests
    • 4. Release WorkflowBranch Code Test Integrate Release Stabilize
    • 5. Release Workflow: BranchBranch Code Test Integrate Release Stabilize• 8-12 branches per release• Current record: 29 branches• Repository per functional area (be, fe, stats, …)• Avg. # lines modified per release: 63K
    • 6. Release Workflow: Code + TestBranch Code Test Integrate Release Stabilize• Scrum (or at least Agile)• As TDD as possible• Labs• A/B Testing• PoCs• Dark launch
    • 7. Release Workflow: IntegrateBranch Code Test Integrate Release Stabilize• Repo always available• Only merge with 100% tests ok / TFW approval• QA Regression & manual tests• Fix possible merge/integration problems ASAP
    • 8. Release Workflow: ReleaseBranch Code Test Integrate Release Stabilize• 2 releases per week (tuesdays & thursdays)• Latest stable changeset from Integration taken previous day 11 AM• Release doc, pre-release meetings• Staging servers to test with live data• We are searching for a Release Manager ;)
    • 9. Release Workflow: StabilizeBranch Code Test Integrate Release Stabilize• First code push: 8 AM• Release window: 3 hours (normal scenario)• Live error stabilization or rollback• Representatives from all involved teams
    • 10. Some tools
    • 11. DVCS: Mercurial• http://mercurial.selenic.com/• Syntax similar to SVN (our old system)• Easy API to plug our plugins and hooks• 100% cross-platform (now Git also but not before)• Commit hooks to check syntax, coding standards…• Bottleneck: – Push/pulls through VPN are slow
    • 12. Issue Tracking: Trac• http://trac.edgewall.org/• User Stories tasks• Bugs• Wiki• Plugins and extensible• Bottleneck: Sometimes slow, code viewing not optimal
    • 13. Testing: PHPUnit• http://www.phpunit.de• Some caveats – Mocking just ‘works’ – PHP process spawning PHP tests• We have: – Vastly improved mocking framework – Shell scripts that isolate test batteries – Better integration with Selenium• Bottleneck: Our current FEFW does not cope perfectly with PHPUnit/Selenium
    • 14. CI: Jenkins• http://jenkins-ci.org/• Previously Hudson too• CI servers farm• Parallelization, plugins, special reports, custom tunnings• Bottleneck: Acceptance tests
    • 15. Storage: MySQL• Live site storage• Dev. env. storage – 1 DB per user (to run tests) – 1 shared DB (common faked data)• Bottleneck: Slow when running tests
    • 16. File copying: RSync• http://rsync.samba.org/• Deployment of code (live & dev)• Sends deltas/diffs• Really fast
    • 17. Configuration: Puppet• http://puppetlabs.com/• Production machines• Jenkins nodes• VM management / Dev web servers config
    • 18. Stats: Hadoop• http://hadoop.apache.org/• Dedicated cluster• HBase• Hive
    • 19. VirtualBox• https://www.virtualbox.org/• Multi-platform• Accurate and fast emulation of Windows OS• Bottleneck: RAM of host machine
    • 20. Search: Sphinx• http://sphinxsearch.com/• Non-realtime (index based)• Very fast• Bottleneck: Index generation on dev & test env.
    • 21. Caching: Memcached• http://memcached.org/• Dev. Behaviour == live behaviour• Tuenti has sent improvements & patches – UDP patch, random ports• Bottleneck: 32GB RAM / machine practical limit
    • 22. Some rants
    • 23. Statics versioning: The good• Browser caching problems gone• Transparent & easy to use by developers• Easy to deactivate for testing
    • 24. Statics versioning: The bad• Dangerous if not careful with non- production ready files• Waste of bandwith• No dev versioning == browser caching problems
    • 25. File bundling: The good• Less files == faster download & deploy• Big text file == better HTTP Gzip• Build time only (no need for dev)• Combines perfectly with minification and versioning
    • 26. File bundling: The bad• Firebug debugging harder (big single files)• One syntax error breaks multiple bundled files – Minimized + Bundled + Error == Real pain• Hierarchy needed to avoid bundling multiple times same files. – Web, external/public page, mobile…• Dev. only errors !
    • 27. Our Build script• Localization• Minification• Bundling• Versioning• Statics deployment to CDNs• Deltas of changes or full build• …• Bottleneck: Build time
    • 28. HipHop• Migrating old code to fully support HipHop – With PHP 5.3• Obvious speed improvements• Also nice for static code analysis
    • 29. Our Chat• Erlang (server) + Javascript (client)• Jabber protocol• Ejjabberd tweaked (3,5x faster)• 200M msgs/day, 1M concurrent users peak,…• 20 machines, ~5 instances per machine• Same behaviour dev/live is critical• We’re searching for Erlang experts ;)
    • 30. exit(0); Sounds interesting?http://jobs.tuenti.com

    ×