Let's get acquainted
➢ Solutions Architect at Silpo (E-commerce)
➢ In IT since 2008 (17 years of experience)
➢ Has experience in various roles (full-stack, backend,
team lead, architect, CTO)
➢ Prints on a 3d printer for PrintArmy
➢ 10+ million orders during this last 3 years
➢ 6+ million unique Guests
Silpo E-commerce in Numbers
➢ IT specialists: 370+
➢ Teams: 35 (26 product teams + 9 service
teams)
➢ Releases: 30K+
➢ Merge requests: 50K+
Offline Shops
Silpo E-commerce
and Ecosystem
Today we’ll talk about…
➢ Products: 14+
➢ Service and microservices: 400+
➢ RPS: 4k+
➢ Requests per month: 6B+
➢ Total Data Served per month: 30TB+
Courier platform Picking platform
Merchant platform
SuperAPP LOKO SuperWEB
Core services:
PIM, OMS
Payment Gateway
GEO
services
PHP in microservices
world
Create a few PHP services
● Loose Coupling
● Scalability
● Parallel
Development
● Increased Complexity
● Higher Infrastructure
Costs
● Security Challenges
Pros: Cons:
PHP Frameworks hell
● Team have the
ability to choose
the framework
● Only team with
knowledge of specific
framework can efficiently
make changes in service
● Cross teams code review
will not be efficient
● The issue solved in one
service can appear in the
other
Pros: Cons:
PHP Frameworks hell
Symfony Framework
● One of the most
popular PHP
Frameworks
● Big and active
community
● Single approach
through all teams,
only business logic
is different
● Efficient cross teams
code review
● Performance overhead
(if out of the box)
● Configuration-heavy
● Development Speed
Pros: Cons:
Fixing Cons (Configuration-heavy and Development Speed)
Service Pack:
● Provides a set of required public and custom
Symfony bundles and configs with meaningful
default values (Configuration-heavy)
● Creating a new service: just require the Service
Pack via Composer and you’re ready to go
(Development Speed)
● Centralized fixes applied to all services at once
(Development Speed)
Service Pack secret (please, don't tell anyone)
Client interaction
● Client can get
access to any data
(can save time
during
development)
● Client can get access to
any data (security issue)
● Client knows about
backend architecture
● Refactoring or any
change on the backend
must be supported with
frontend
● Each service can be
accessed from the
outside
Pros: Cons:
Client interaction
Client interaction: API Gateway
● Single endpoint can
merge data from
different services
● Services are not
accessible from the
outside
● Client doesn’t know
about backend
architecture
● Single point for
monitoring and
alerting
● Can be a bottleneck
● Needs add proxy for
service endpoints
(development
overhead)
Pros: Cons:
Service to service sync interaction
● Any service can
get any
information from
other service
● Chaotic service to
service connections
● Very hard to
understand all
dependencies
Pros: Cons:
Service to service sync interaction
Service to service sync interaction: API Gateway
● Strict service to
service interaction
pattern
● Each service
knows only about
API Gateway
● Single point for
monitoring and
alerting
● Can be a bottleneck
● Needs add proxy for
service endpoints
(development overhead)
Pros: Cons:
Service to service async interaction
Typical PHP service
Typical PHP service structure
API web server
Keep in mind:
● One process per container in
k8s pod
● Each OpenSwoole worker has
its own php environment and
keeps the code in memory
● !!! Each worker maintains its
own persistent PHP state
across requests !!!
OpenSwooleBundle
Main features:
● Controls OpenSwoole server
(start/stop/reload/status)
● Integrates OpenSwoole request/response
handling with Symfony
● Ensures all state-dependent services
reset their state on each request
● Enables parallel execution of functions via
OpenSwoole coroutines
…
Clear state code example:
Swoole vs OpenSwoole
Metrics: Prometheus
Metrics: Mechanics
● PrometheusMetricsBundle - collects
Symfony metrics
● Custom collector - collects
OpenSwoole metrics
● Prometheus scrapes API service
metrics
● Workers and Cron jobs push metrics
to Pushgateway
Metrics: Visualization
Metrics: Visualization Examples
Symfony + Kubernetes
OpenSwoole
Alerts
Alerts: Prometheus Example
Alerts as a code
Prometheus rule
Alerts: PagerDuty & Slack Examples
PagerDuty incident
Slack alert
Logs
Logs: Implementation
● Filebeat Logstash Writes logs
directly to ElasticSearch
● Async write of logs via OpenSwoole
task workers
● Symfony Messenger component
integration with OpenSwoole task
transport
● Doesn’t overwhelm pods with a
huge volume of stdout logs
● Doesn’t affect request/response
latency
Errors and tracing
Errors and tracing: Sentry
● Async error & trace export via
OpenSwoole task workers
● Custom implementation of Sentry
HttpClientInterface to use OpenSwoole
task workers under the hood
● Doesn’t affect request/response latency
● Track 100% of errors and warnings
● Sample traces for 1% of requests, with
the ability to enforce tracing for specific
request
Errors and tracing: Sentry Alert Example
Alerts as a code Sentry alert
Slack alert
Errors and tracing: Sentry Tracing Example
Autoscaling
Autoscaling: First Iteration
Formula for KEDA
triggers:
query: sum(rate(http_requests_total{service="ecom--order-service"}[1m]))
threshold: "20"
How it works:
● Metric from Symfony
● Each pod can handle ~20 RPS safely
● If traffic grows → more pods
● If traffic drops → fewer pods
Traffic increase (RPS)
Pods scaled up to 13
Autoscaling: Second Iteration
Formula for KEDA
triggers:
query: ((sum(openswoole_workers_total - openswoole_workers_idle) / sum(openswoole_workers_total)) + 0.5) * kube_deployment_status_replicas_available
threshold: "1"
How it works:
● Metric from OpenSwoole
● Scales pods by worker load: when
workers are near full, add pods
Development Gates
● Cross-team code reviews
● Tests are part of the Definition of Done
● Static analyzers: PHPStan, PHPMD, PHPCP, etc.
● SonarQube rules and gates
● Security checks: dependency vulnerabilities, OWASP, SAST, secret detection
Our custom Symfony bundles are public now
https://github.com/silpo-tech
● OpenSwooleBundle
● RestBundle
● ExceptionHandlerBundle
● PaginatorBundle
● HealthCheckBundle
● FilterBundle
● … and more on GitHub
300 UAH promo code for SilpoApp https://jobs.dou.ua/companies/silpo/vacancies/
Silpo (E-commerce) vacancies

"How to run 200+ PHP services in production without losing your mind?", Yurii Panaiotov

  • 2.
    Let's get acquainted ➢Solutions Architect at Silpo (E-commerce) ➢ In IT since 2008 (17 years of experience) ➢ Has experience in various roles (full-stack, backend, team lead, architect, CTO) ➢ Prints on a 3d printer for PrintArmy
  • 3.
    ➢ 10+ millionorders during this last 3 years ➢ 6+ million unique Guests Silpo E-commerce in Numbers ➢ IT specialists: 370+ ➢ Teams: 35 (26 product teams + 9 service teams) ➢ Releases: 30K+ ➢ Merge requests: 50K+
  • 4.
    Offline Shops Silpo E-commerce andEcosystem Today we’ll talk about… ➢ Products: 14+ ➢ Service and microservices: 400+ ➢ RPS: 4k+ ➢ Requests per month: 6B+ ➢ Total Data Served per month: 30TB+
  • 5.
    Courier platform Pickingplatform Merchant platform SuperAPP LOKO SuperWEB Core services: PIM, OMS Payment Gateway GEO services
  • 6.
  • 7.
    Create a fewPHP services ● Loose Coupling ● Scalability ● Parallel Development ● Increased Complexity ● Higher Infrastructure Costs ● Security Challenges Pros: Cons:
  • 8.
    PHP Frameworks hell ●Team have the ability to choose the framework ● Only team with knowledge of specific framework can efficiently make changes in service ● Cross teams code review will not be efficient ● The issue solved in one service can appear in the other Pros: Cons:
  • 9.
  • 10.
    Symfony Framework ● Oneof the most popular PHP Frameworks ● Big and active community ● Single approach through all teams, only business logic is different ● Efficient cross teams code review ● Performance overhead (if out of the box) ● Configuration-heavy ● Development Speed Pros: Cons:
  • 11.
    Fixing Cons (Configuration-heavyand Development Speed) Service Pack: ● Provides a set of required public and custom Symfony bundles and configs with meaningful default values (Configuration-heavy) ● Creating a new service: just require the Service Pack via Composer and you’re ready to go (Development Speed) ● Centralized fixes applied to all services at once (Development Speed)
  • 12.
    Service Pack secret(please, don't tell anyone)
  • 13.
    Client interaction ● Clientcan get access to any data (can save time during development) ● Client can get access to any data (security issue) ● Client knows about backend architecture ● Refactoring or any change on the backend must be supported with frontend ● Each service can be accessed from the outside Pros: Cons:
  • 14.
  • 15.
    Client interaction: APIGateway ● Single endpoint can merge data from different services ● Services are not accessible from the outside ● Client doesn’t know about backend architecture ● Single point for monitoring and alerting ● Can be a bottleneck ● Needs add proxy for service endpoints (development overhead) Pros: Cons:
  • 16.
    Service to servicesync interaction ● Any service can get any information from other service ● Chaotic service to service connections ● Very hard to understand all dependencies Pros: Cons:
  • 17.
    Service to servicesync interaction
  • 18.
    Service to servicesync interaction: API Gateway ● Strict service to service interaction pattern ● Each service knows only about API Gateway ● Single point for monitoring and alerting ● Can be a bottleneck ● Needs add proxy for service endpoints (development overhead) Pros: Cons:
  • 19.
    Service to serviceasync interaction
  • 20.
  • 21.
  • 22.
    API web server Keepin mind: ● One process per container in k8s pod ● Each OpenSwoole worker has its own php environment and keeps the code in memory ● !!! Each worker maintains its own persistent PHP state across requests !!!
  • 23.
    OpenSwooleBundle Main features: ● ControlsOpenSwoole server (start/stop/reload/status) ● Integrates OpenSwoole request/response handling with Symfony ● Ensures all state-dependent services reset their state on each request ● Enables parallel execution of functions via OpenSwoole coroutines … Clear state code example:
  • 24.
  • 25.
  • 26.
    Metrics: Mechanics ● PrometheusMetricsBundle- collects Symfony metrics ● Custom collector - collects OpenSwoole metrics ● Prometheus scrapes API service metrics ● Workers and Cron jobs push metrics to Pushgateway
  • 27.
  • 28.
  • 29.
  • 30.
    Alerts: Prometheus Example Alertsas a code Prometheus rule
  • 31.
    Alerts: PagerDuty &Slack Examples PagerDuty incident Slack alert
  • 32.
  • 33.
    Logs: Implementation ● FilebeatLogstash Writes logs directly to ElasticSearch ● Async write of logs via OpenSwoole task workers ● Symfony Messenger component integration with OpenSwoole task transport ● Doesn’t overwhelm pods with a huge volume of stdout logs ● Doesn’t affect request/response latency
  • 34.
  • 35.
    Errors and tracing:Sentry ● Async error & trace export via OpenSwoole task workers ● Custom implementation of Sentry HttpClientInterface to use OpenSwoole task workers under the hood ● Doesn’t affect request/response latency ● Track 100% of errors and warnings ● Sample traces for 1% of requests, with the ability to enforce tracing for specific request
  • 36.
    Errors and tracing:Sentry Alert Example Alerts as a code Sentry alert Slack alert
  • 37.
    Errors and tracing:Sentry Tracing Example
  • 38.
  • 39.
    Autoscaling: First Iteration Formulafor KEDA triggers: query: sum(rate(http_requests_total{service="ecom--order-service"}[1m])) threshold: "20" How it works: ● Metric from Symfony ● Each pod can handle ~20 RPS safely ● If traffic grows → more pods ● If traffic drops → fewer pods Traffic increase (RPS) Pods scaled up to 13
  • 40.
    Autoscaling: Second Iteration Formulafor KEDA triggers: query: ((sum(openswoole_workers_total - openswoole_workers_idle) / sum(openswoole_workers_total)) + 0.5) * kube_deployment_status_replicas_available threshold: "1" How it works: ● Metric from OpenSwoole ● Scales pods by worker load: when workers are near full, add pods
  • 41.
    Development Gates ● Cross-teamcode reviews ● Tests are part of the Definition of Done ● Static analyzers: PHPStan, PHPMD, PHPCP, etc. ● SonarQube rules and gates ● Security checks: dependency vulnerabilities, OWASP, SAST, secret detection
  • 43.
    Our custom Symfonybundles are public now https://github.com/silpo-tech ● OpenSwooleBundle ● RestBundle ● ExceptionHandlerBundle ● PaginatorBundle ● HealthCheckBundle ● FilterBundle ● … and more on GitHub
  • 44.
    300 UAH promocode for SilpoApp https://jobs.dou.ua/companies/silpo/vacancies/ Silpo (E-commerce) vacancies

Editor's Notes

  • #11 Розповісти про те що у нас є Service Pack. Такий собі бандл бандлів для Symfony. Маючи це, ми сильно спрощуємо створення нових сервісів і пришвидшуємо швидкість розробки.
  • #12 Розповісти про те що у нас є підключення стандартних бандлів та ліб (можуть бути публічні, або наші кастомні). Далі у нас є структура підключення бандлів і конфігурацій аналогічна сімфоні. Ну і найголовніший секрет/хак. Ми екстендимо кернел сімфоні і додаємо йому можливість завантаження бандлів та конфігурації сервіс паку. В тому числі якщо розробник у своєму сервісі захоче перевизначити дефолтні конфігурації, то він це зможе робити, бо логіка завантаження сімфоні завжди має пріорітет над бандлами сервіс паку.
  • #22 Обов’язково сказати про те що workers - для обробки реквестів/респонсіт, а task workers - для важких асинхронних операцій
  • #23 + сказати про чистку доктріни
  • #26 API - скрепінг метрик через енд /metrics, Worker and Cron - пушінг через push-gateway
  • #27 API - скрепінг метрик через енд /metrics, Worker and Cron - пушінг через push-gateway
  • #28 API - скрепінг метрик через енд /metrics, Worker and Cron - пушінг через push-gateway
  • #29 API - скрепінг метрик через енд /metrics, Worker and Cron - пушінг через push-gateway
  • #30 Не забути про ранбуки під кожен алерт.
  • #31 API - скрепінг метрик через енд /metrics, Worker and Cron - пушінг через push-gateway
  • #35 В terminate евенті
  • #37 Приклад трейсінгу. Це приклад доволі складного ендпоінта в якому є ланцюжок сервісів які викликають один одного, але ми все одно маємо можливість бачити вкладені трейси завдяки шаред хедеру X-Trace-ID та стандарту OpenTelementry. Якщо вам здалося що в даному прикладі є паралельні спани (тобто паралельні виконання деяких функцій), то вам не здалося. Це також реалізовано завдяки OpenSwoole корутінам. Цей фукнціонал називається батч раннер. Його було створено одним з наших розробників і він є частиною нашого кастомного OpenSwoole Symfony Bundle
  • #42 Впевнений, майже у кожного з вас виникло до мене питання - Юра, ти розповідаєш як у вас все класно, але ж ви написали багато кастомного коду? Ти просто хизуєшся чи шо? Я підготувався до цього питання і у мене є аж 3 відповіді: 1. Ну, тепер ви знаєте що так можна і це перевірено продакшеном одного з найбільших еком проектів України… Не дуже, відповідь, так? 2. Ви можете зааплаїтися на наші вакансії і працювати в такому стеку (куар код та посилання будуть на останньому слайді)... Вже трошки краще, але все ж таки не ідеально, так? Добре, тоді у мене є третя відповідь - переключаюсь на наступний слайд
  • #43 Всі наші кастомні базові бандли та бібліотеки ми віддаємо в опен сорс і тепер ви можете спробувати наші підходи. Тестуйте, експеремінтуйте, контріб’ютьте, створюйте іщюси. Будемо раді взаємодії та новим ідеям.