How Badoo Saved $1M
Switching to PHP7
Nikolay Krapivnyy
phpday.it, 2016
About me
✦ Nikolay Krapivnyy
✦ 10+ years PHP experience
✦ 5 years with Badoo,
now backend team lead
✦ This is my first talk in English :)
46languages
data
300M
users190countries
“
BADOO IS THE
LARGEST DATING
NETWORK IN
THE WORLD
— Forbes
Badoo tech:
✦ 4 datacenters, ~3,000 servers overall
✦ up to 80,000 RPS for php backend
✦ ~200 engineers (~70 php devs)
✦ >2M PHP LoC
✦ and growing….
Today I will:
✦ Share our practical experience
✦ Talk about some of our tools
✦ Tell you a million dollar story :)
1. Problem
2. Transition
3. Results
1. Problem
DBs
CLOUD SERVICES
LTM
How it works:
APP
HTTPHTTP
API
API
9 %
15 %
22 %
22 %
32 %
php
mysql
other
photos
c/go
Hardware
32% or 1000
of our servers
are running php
PHP Cluster Growth
0
125
250
375
500
Jan, 2014 Jul, 2014 Jan, 2015 Jul, 2015 Jan, 2016
servers in 1DC users
CPU is the main
bottleneck
for us
How do we
measure php
performance?
Pinba!
✦ Php is not a bottleneck anymore
✦ RT performance monitoring tool
✦ Extension for php + plugin for mysql
✦ Open source! http://pinba.org/
Request time
✦ Some API call: GET_USER
✦ avg request time = 0.12 sec
✦ request time = cpu time + i/o time
✦ i/o time = mysql + services + FS i/o +…
Pinba
Particular requests time
Pinba report
= +
Request time
✦ 0.12 sec = 0.07 (cpu) + 0.05 (i/o)
✦ 0.05 (i/o) = 0.012 + 0.05 + 0.03 + …
✦ 0.012 (memcache) = ~26*0.0005
✦ Realtime!
CPU usage
takes ~50% of
our request time
PHP7 alternatives?
✦ HHVM (Facebook, http://hhvm.com/)
✦ kPHP (VK.com, https://github.com/vk-com/kphp-kdb)
+ JIT, PHP5 compatible
— C++, extensions API
+ faster then HHVM
— no OOP support at all
2. Transition
Platform team:
✦ C developers
✦ Php developers
✦ Badoo infrastructure support
Transition steps:
2.1. Extensions
2.2. Codebase
2.3. Deploy
PHP7: extensions
xdebug
apcu
lua
imagick
✦ ~30 of them
✦ Already supported:
PHP7: extensions
blitz
xhprof
handlersocketi
pinba
leptonica
tarantool
sphinx
memtrack
judy
&& 15 more…
PHP7: extensions
https://github.com/tony2001
All php7 versions
available:
Check for php7
branch
We thought
we were done,
but…
Runkit
What is runkit?
✦ runkit_method_add/redefine
✦ runkit_constant_add/redefine
✦ in runtime
✦ but why?
Unit testing!
but why?
Unit testing:
✦ 60K Tests now
✦ Coverage >50%
✦ Introduced only a few years ago
✦ Runkit: to isolate old code
Runkit: summary
✦ essential for us
✦ no support for php7
Runkit for php7?
✦ It`s different
✦ Alternatives?
• UOPZ (https://github.com/krakjoe/uopz)
✦ Lots of changes anyway
Let`s remove
this dependency
Soft mocks!
✦ Soft means no extension
✦ Plain PHP, compatible with all PHP
✦ Replaces Runkit for us
✦ Open source!
(https://github.com/badoo/soft-mocks)
How it works:
✦ Rewrites code on the fly
✦ Inserts "interceptors" in methods
✦ Wraps all constant usages
✦ Uses PHP-Parser
1 require_once(__DIR__ . '/path.inc');
2 ini_set(‘pinba.enabled', 1);
3 echo RRD_BASE_PATH;
1 require_once SoftMocks::rewrite(
ROOT_PATH . '/path' . '/path.inc'
);
2 SoftMocks::call(
'ini_set',
array('pinba.enabled', 1’)
);
3 echo SoftMocks::getConst(‘RRD_BASE_PATH');
Original
After SoftMocks::rewrite()
1 class A
2 {
3 public function getValue()
4 {
5 return 10;
6 }
7 }
1 class A
2 {
3 public function getValue()
4 {
if (SoftMocks::isMocked(A::class, __FUNCTION__)) {
return SoftMocks::getResult(…);
}
5 return 10;
6 }
7 }
Original
After SoftMocks::rewrite()
So…
✦ No extension dependency
✦ 1.5 weeks for development
✦ A bit longer to adapt tests
✦ More stable tests in general
https://github.com/YuriyNasretdinov
Credits
More cool stuff:
Soft mocks:
https://github.com/badoo/soft-mocks
2.2. Codebase
php7mar
✦ https://github.com/Alexia/php7mar
✦ Migration Assistant Report
php7mar
469 * foreachByReference
230 * oldClassConstructors
156 * funcGetArg
22 * hexadecimalString
19 * variableInterpolation
14 * yield
7 * deprecatedFunctions
4 * reservedNames
3 * arrayValueByReference
2.3. Deploy
Deploy
✦ Devel: separate sandbox
✦ Staging
✦ Production
Deploy: production
✦ Several hosts in each cluster
✦ Several weeks of quarantine
✦ CLI: OK, WEB: ERROR
✦ WEB: fix for opcache reset in FPM
3. Results
CPU usage
cluster CPU usage
memory usage
We got:
2x CPU usage
9x memory usage
✦ 700 servers running FPM
✦ 1/2 required
✦ ~$4K each
✦ 350*4K = $1,4M
Let's calculate
Links
✦ Pinba: http://pinba.org/
✦ Soft mocks: https://github.com/badoo/soft-mocks
✦ php7mar: https://github.com/Alexia/php7mar
✦ php7 exts: https://github.com/tony2001
Summary:
✦ PHP7 rocks!
✦ Know your bottleneck
✦ Find your way
✦ Share!
Grazie!
Questions?
Twitter: @BadooTech
Blog: https://techblog.badoo.com
LinkedIn: https://linkedin.com/in/nkrapivnyy
Email: cyberklin@gmail.com
Joind.in: https://joind.in/talk/948d4

How Badoo Saved $1M Switching to PHP7 - Nikolay Krapivnyy - PHPDay Verona 2016