The journey of a symfony app from 150ms to 20ms

2. Alex Bumbacea
● Programmer since 2007
● tackled a bit of sysadmin
● telecom/gaming/e-commerce
● Symfony user since 1.4

3. The journey of a Symfony
app from 150ms to 20ms

5. More context
● Recommandations engine
● API with 2 endpoints
○ /bulk
○ /one

7. xhprof

8. Prep work
● Smoke tests - fast deployment
● Stable env
● Canary server (⅓ servers at least)
● Monitoring tools

9. Monitoring tools
● NewRelic
● Grafana
● ELK - monitor error rates

10. Daily status update
Now Target
99% 4000ms 300ms
95% 3000ms N/A
50% - median 150ms N/A
Throughput 100rps 800rps

12. Take 1
Gearman to parallelize bulk

13. Findings for take 1
● Increase in throughput
● Bulk response time decreased
● Hard to scale fast
● Consuming a lot of memory

14. Take 2
Benchmark against java & node js
(with gearman jobs in php)

15. Original stats

16. Take 2.1
Upgrade from php 5.6 to 7.0

17. Some more stats

19. Problems of take 2.1
● Mongo driver support dropped
● Upgrade doctrine mongo bundle
● XHProf tools incompatibility

20. Findings take 2.1
25% decrease in request timings
Memory usage reduction

21. Daily status update
Before Now Target
99% 4000ms 3200ms 300ms
95% 3000ms 2000ms N/A
50% - median 150ms 100ms N/A
Throughput 100rps 130rps 800rps

22. Take 3
Drop gearman and move to multi curl

23. Findings for take 3
● Increase in throughput
● Better monitoring - bulk was hiding stuff
● Better http cache for our API -> we could use varnish in
front of it

24. Daily status update
Before Now Target
99% 3200ms 3000ms 300ms
95% 2000ms 2000ms N/A
50% - median 100ms 100ms N/A
Throughput 130rps 200rps 800rps

25. Connecting / Reconnecting
● Redis
● Mongo
● MySQL
● etc
● Consider connection state (ex: redis selected db)

27. Consider your timeouts
● Solr / Mongo / MySQL / Redis etc
● None of your timeouts should be larger
than your SLA

28. Network connection: Connection timeout!

30. ● XHProf
● NewRelic

31. Look into dispatched events
● It hides a lot of processing
● Not everything is required every time

32. Get rid of garbage
- We removed bundles
- SecurityBundle, SwiftMailerBundle, SensioFrameworkExtraBundle, TwigBundle,
AsseticBundle
- Only required services (that don’t have lazy
loading)
- Only required routes

33. Dedicated kernel (ApiKernel)
- And dedicated cache folder :)

34. Autoload optimizations
Composer optimize or not?
For our project:
NOT

35. IO

36. IO

37. IO

38. Queue it!
If you don’t need it now… queue it :)

39. Slaves. Do you use them?

40. Config tweaks
● FPM children ++
● Mongo max connections ++

41. Daily status
Before Now Target
99% 4000ms 280ms 300ms
95% 3000ms 200ms N/A
50% - median 150ms 20ms N/A
Throughput 100rps 500rps 800rps

42. Add servers
X servers * 2

43. Daily status - END
Before Target Now
99% 4000ms 300ms 280ms
95% 3000ms N/A 200ms
50% - median 150ms N/A 20ms
Throughput 100rps 800rps 1100rps

46. Summary
● Gearman
● Java -> PHP 7
● Gearman -> curl_multi_get
● Connections and timeouts
● ApiKernel
● IO / Memory / FPM
● Horizontal scaling

48. Don’t use those numbers on your app.
Make your own tests.

50. Q&A

51. https://medium.com/techlabs-emag
https://www.linkedin.com/in/alexbumbacea/
https://github.com/alexbumbacea
Thank you!