The focus of my presentation are patterns, principles and especially practical solutions how to implement an ecommerce platform that consists of hundreds decoupled and coherent modules. This talk is suitable for senior developers who design large scale systems and are interested in architectural insights and inspirations.
Keywords: SOLID, Packaging Principles, Design Patterns, Macro- and Microservices, Semantic Versioning, Composer
3. 2017 Spryker Systems GmbH
About me
Fabian Wesner
CTO @ Spryker Systems
TwiGer: @FabianWesner
hGp://start.spryker.com
4. 2017 Spryker Systems GmbH
What is Spryker?
SPRYKER is a Commerce Opera>ng System
for customer or machine ini>ated transac>ons.
5. 2017 Spryker Systems GmbH
Modularity
One of Spryker’s key characteris>cs:
Modularity
This talk is about learnings, tradeoffs, pa5erns, tools and
principles to build a truly modular ecommerce pla9orm.
8. 2017 Spryker Systems GmbH
Microservices are compelling
• Independent teams, hos>ng and deployments
• Reduced complexity for single services
• Free choice of technologies….
• Big challenge. New tools to discover.
• Great for conference talks
9. 2017 Spryker Systems GmbH
But: Distributed compu>ng is hard
• Asynchronous calls
• Network is not reliable and has latency
• Debuging via network
• Performance issues
• API versioning
• Authen>fica>on
• Service discovery
• Distributed logging
• Eventual consistency, no simple transac>ons
• Distributed sets of data, no straight-‐forward joins
• No protec>on by foreign key constraints
• …
10. 2017 Spryker Systems GmbH
Good use case for Microservices
Microservices can work very well when there are very clean boundaries and communica>on direc>ons.
Example: Pla,orm for Online Games
Free choice of technology per game.
Independent deployment, hos>ng, data-‐
storage, etc.
Basic func>onality is implemented in a shared
service (e.g. Customer, Payment, …)
In this scenario Microservice is natural choice,
even for companies that are not the scale of
Google or Amazon.
11. 2017 Spryker Systems GmbH
Bad use case for Microservices
Microservices become pain in the ass when the system has high amount of internal dependencies
which usually becomes visible in the ER diagram of the database schema.
Example: Online-‐Shop
The diagram shows a typical
database schema of a online shop.
All rela>onships are meaningful and
need to be protected by the
database.
It’s not even big!
13. 2017 Spryker Systems GmbH
Prinicples of modularity
Strong encapsula/on
Hide implementa>on details inside components, leading to low coupling between different parts.
Teams can work in isola>on on decoupled parts of the system.
Well-‐defined interfaces
You can't hide everything (or else your system won't do anything meaningful), so well-‐defined and
stable APIs between components are a must. A component can be replaced by any implementa>on
that conforms to the interface specifica>on.
Explicit dependencies
Having a modular system means dis>nct components must work together. You'd beGer have a good
way of expressing (and verifying) their rela>onships.
Quoted from hGps://www.oreilly.com/ideas/modules-‐vs-‐microservices
14. 2017 Spryker Systems GmbH
Bad news!
PHP does not have built-‐in support for modules!
So we need to solve it with conven>ons, paGerns and the help of composer.
16. 2017 Spryker Systems GmbH
What is a bundle (~module)?
Spryker’s defini/on of a bundle
• “A bundle is a func>onal unit!”
• It defines its internal API
• It defines its data structures
• It defines its dependencies
• It contains all required layers:
• Presenta>on
• Communica>on
• Business
• Persistence
• It contains all unit tests
Bundles are small (~ micro),
coherent and loose coupled.
There will be a high numer
of bundles! (> 100)
We need a consistent way
how to communicate
between bundles.
è
17. 2017 Spryker Systems GmbH
Facades as internal API for bundles
Client
Facade!
Cart!
Facade pa5ern is structural pa5ern by the GOF.
19. 2017 Spryker Systems GmbH
Bundle to Bundle Communica>on
Facade! Facade!
Cart! Calculation!
Facade!
Discount!
calculateDiscounts()!recalculate()!add()!
20. 2017 Spryker Systems GmbH
Facade interface challenge
Internal class CartOpera>on programs against interface of Calcula>onFacade
Where are the bundle boundaries?
21. 2017 Spryker Systems GmbH
Facade interface challenge
Internal class CartOpera>on programs against interface of Calcula>onFacade
Where are the bundle boundaries?
22. 2017 Spryker Systems GmbH
Facade interface challenge
Approach 1: cart-‐opera>on class and interface are in same bundle.
Problem: The facade needs to implement an interface from another bundle.
23. 2017 Spryker Systems GmbH
Facade interface challenge
Approach 2: Facade and interface are in the same bundle
Problem: The Opera>on class has a hard-‐coded dependency to a specific
bundle.
24. 2017 Spryker Systems GmbH
Solu>on with bridge paGern
Solu/on: Add a bridge-‐paGern and a second interface to solve the problem!
The CartToCalcula>on contains only the required methods (ISP).
25. 2017 Spryker Systems GmbH
Solu>on with bridge paGern
The bridge allows us to
use any class that fits to
the interface.
Coupling in run-‐>me only.
Not in development-‐>me.
26. 2017 Spryker Systems GmbH
Solu>on with bridge paGern
Facade of Calcula>on-‐bunde is located and becomes wrapped into the
CartToCalcula>on-‐bridge.
27. 2017 Spryker Systems GmbH
Shared data structure
Challenge: Data needs to be exchanged between bundles.
Approach: Usage of arrays!?
public function recalculate(array $quote)
{
return $this->getFactory()->createStackExecutor()->recalculate($quote);
}
Not expressive.
No clear structure.
28. 2017 Spryker Systems GmbH
Shared data structure
Challenge: Data needs to be exchanged between bundles.
Approach: Data transfer objects (DTOs)
29. 2017 Spryker Systems GmbH
Shared data structure
Challenge: Data needs to be exchanged between bundles.
Approach: Data transfer objects (DTOs)
30. 2017 Spryker Systems GmbH
Shared data structure
Challenge: Where to put these DTOs?
Approaches:
• Have the same DTO in both bundle
• Have the DTO in the “main” bundle (Which one is it?)
31. 2017 Spryker Systems GmbH
Shared data structure
Solu/on: Define the DTO in XML in all related bundles, merge them and
generate the object.
è
32. 2017 Spryker Systems GmbH
Summary Communica>on
Bundles talk to each other via
Facades (~ internal APIs)
Bridges and interfaces connect
the bundles in run-‐>me but keep
them decoupled in development-‐>me.
Data is transported via Data transfer objects
33. 2017 Spryker Systems GmbH
Modular database
In a perfect world every bundle / service has it’s own database and data is
only transmiGed via messages.
But…
34. 2017 Spryker Systems GmbH
Modular database
In an e-‐commerce applica>on things
are connected.
Real-‐world use cases
• Show all customers with their number of sales-‐orders
• Discounts are valid only for products with a high stock
• Discounts are valid only for specific customers
• Products have specific prices per customers (Typical B2B feature)
• …
What we need is normalized database with protected foreign-‐key rela>ons!
35. 2017 Spryker Systems GmbH
Modular database
Challenge: How to handle cross-‐bundle rela>ons?
Example: “A product has stock”
• Table spy_product belongs to Product-‐bundle
• Tables spy_stock_product and spy_stock belong to Stock-‐bundle
36. 2017 Spryker Systems GmbH
Modular database
Solu/on: Each bundle ships with it’s own schema defini>on!
All XML-‐files are merged, diffed and migrated to the database.
When a bundle is removed , all the related tables and rela>ons are gone as well.
37. 2017 Spryker Systems GmbH
Modular database
Solu/on: Each bundle ships with it’s own schema defini>on!
Discussion:
• Every bundle can query it’s own data only.
• Cross-‐bundle joins are possible but they create a >ght coupling.
• Therefore we prefer communica>on via messages.
• The main reason to do cross-‐bundle joins is performance. It’s a tradeoff.
39. 2017 Spryker Systems GmbH
Designing dependencies
• Product bundle holds the product en>>es
• Stock bundle manages the inventory
What are their dependencies?
40. 2017 Spryker Systems GmbH
Designing dependencies
Considera>ons:
• There can be products without stock
• There is never a stock without a product
è Stock-‐bundle requires Product-‐bundle
(~ Product-‐bundle is responsible for Stock-‐bundle)
41. 2017 Spryker Systems GmbH
Another simple example (availability)
Availability of a product = Stock – Sold Items
Bundle Responsibility
Availability-‐bundle Calculates the availability of a product
Oms-‐bundle Order Management System knows the state of an order-‐item
Product-‐bundle
Holds the product en>>es
Stock-‐bundle
Holds the stock of a product
We need several bundles:
42. 2017 Spryker Systems GmbH
Another simple example (availability)
Availability of a product = Stock – Sold Items
What are their dependencies?
43. 2017 Spryker Systems GmbH
Another simple example (availability)
Availability of a product = Stock – Sold Items
44. 2017 Spryker Systems GmbH
Another simple example (availability)
Availability of a product = Stock – Sold Items
45. 2017 Spryker Systems GmbH
Another simple example (availability)
Availability of a product = Stock – Sold Items
46. 2017 Spryker Systems GmbH
Some>mes it‘s not that simple
Use case: Discounts are valid for specific customer-‐groups
What are the dependencies?
47. 2017 Spryker Systems GmbH
Some>mes it‘s not that simple
What are the dependencies?
• Both bundles can be used separately!
• Discounts are possible without any customer-‐groups and vice-‐versa.
Run-‐/me call: Discount-‐bundle needs to check if current customer belongs to
the related Customer-‐Group
Does that mean that Discount-‐bundle requires Customer-‐Group? No!
48. 2017 Spryker Systems GmbH
Some>mes it‘s not that simple
We put a Connector in the middle!
Now both bundles are independent from each other BUT can s>ll work
together.
How does that technically work?
49. 2017 Spryker Systems GmbH
Some>mes it‘s not that simple
✔ The connector makes a direct call to the CustomerGroupFacade
? But how can we make a call from discount-‐bundle to the connector with a
direct dependy?
50. 2017 Spryker Systems GmbH
Inversion of control
Solu=on: Inversion of control with Plugins
51. 2017 Spryker Systems GmbH
Plugin configura>on
Solu=on: Inversion of control with Plugins
The plugin is configured into the Discount-‐bundle!
52. 2017 Spryker Systems GmbH
Learning: Dependencies != Coupling
Conceptual dependencies are given and need to be discovered by the team
• Stock requires Product
• Product-‐OpTon requires Product
• Customer-‐Group requires Customer
• Etc.
Technical coupling between bundles doesn’t just happen!
• Dependencies can be inverted if needed.
• Dependencies can be mandatory or op>onal
53. 2017 Spryker Systems GmbH
Experience
Although we managed our dependencies very carefully, we ended up with
this:
54. 2017 Spryker Systems GmbH
Learning
We need to obey the Principles of Package Design!
Principles of cohesion
• The Release/reuse equivalence principle
• The Common reuse principle
• The Common closure principle
Principles of coupling
• The Acyclic dependencies principle
• The Stable dependencies principle
• The Stable abstrac>ons principle
55. 2017 Spryker Systems GmbH
Recommended books
The classic book about agile
sooware design from Robert Mar>n.
A modernized explana>on of SOLID
and Package Principles for modern PHP
development by MaGhias Noback.
56. 2017 Spryker Systems GmbH
The Acyclic dependencies principle
The dependency graph of packages must have no cycles.
58. 2017 Spryker Systems GmbH
The Stable dependencies principle
Depend in the direc>on of stability.
59. 2017 Spryker Systems GmbH
What is Stability?
A bundle is more stable the more bundles requires it.
Stability = Outgoing Dependencies / (Outgoing + Incoming Dependencies)
63. 2017 Spryker Systems GmbH
Benefits
A clean dependency tree has a lot of advantages:
• Reduced system complexity. Avoidance of side-‐effects.
• Team can work independently even with large systems.
• It becomes possible to split the applica>on into services
• Single bundles can be released, replaced and extended
67. 2017 Spryker Systems GmbH
Composer.json
We use Composer.json in the following way:
Require For direct dependencies (~ Calls to a facade)
Suggest For plugins that offer addi>onal func>onality
Require-‐Dev For dependencies that only exist in tests.
68. 2017 Spryker Systems GmbH
Challenge: Slow composer update
Spryker releases in >120
bundles now!
And many more in the
future!
Problem: Composer
update takes ages….
69. 2017 Spryker Systems GmbH
Solu>on: Toran Proxy
We run a public Toran Proxy that pre-‐fetches the composer.json
files from Github and allows to execute composer update in < 1
minute.
In case you prefer SaaS, then public or private Packagist will do the same job.
70. 2017 Spryker Systems GmbH
Challenge: Core development
How does core development work with > 120 bundles?
Should we commit to all bundles?
What about branches and tags?
What about conflicts?
How do release features?
71. 2017 Spryker Systems GmbH
Solu>on: Git Subtree Split
Git has a handy tool for this problem. Git Subtree Split:
• The core team works with a single (private repository).
• There is a liGle app installed on a server that reacts on every
commit to master and executes the subtree split opera>on.
• The opera>on maps single directories to other repositories and
forwards the commits.
72. 2017 Spryker Systems GmbH
Solu>on: Git Subtree Split
You can start with the open source bash script from Fabien Potencier:
hGps://github.com/splitsh/lite
73. 2017 Spryker Systems GmbH
How to release > 120 bundles?
There are two approaches:
Tags in the source-‐repository are forwarded to all splits
• Every bundle has all versions
• Some versions contain no changes for a single bundle (Phantom release)
• Advantage: There is a global version number (e.g. Symfony 3.0.0)
• This is how Symfony does it
Tags in the source-‐repository are forwarded only to these splits that contain a change
• Bundles have different versions
• Advantage: Bundles can be released independently (Atomic Release)
• This is how Spryker does it
74. STRICTLY CONFIDENTIAL
74
You cannot solve 21st century e-‐commerce problems with 20th century technology
www.spryker.com
@sprysys