How are Mage-OS packages built? This presentation gives an overview over the history so far. The presentation was given at MageTitans 2023 in Manchester (UK), organized by Space48. Mage-OS is a community owned fork of Magento Open Source. It ensures the accessibility, longevity and success of the Magento platform.

1. Mage-os.org 22. Apr. 2023
Building
Mage-OS
Or: DIY Magento Releases

2. Mage-os.org 22. Apr. 2023
Mage-OS is an open
initiative to ensure the
accessibility, longevity,
and success of the
Magento® platform
and ecosystem.
The Mage-OS
Distribution
is a community
owned fork
of Magento Open
Source.
Building a
Distribution

3. Mage-os.org 22. Apr. 2023
How are Mage-OS release packages built?

4. Mage-os.org 22. Apr. 2023
Back when we started: the first ideas…
Magento is many modules in a mono-repo
> Symfony uses splitsh/lite, maybe we can use it!
> There also are other existing tools…

5. Mage-os.org 22. Apr. 2023
First experiments
Splitting the magento repo into subrepos took ~36 hours!

6. Mage-os.org 22. Apr. 2023
Okay, something custom then!
“The Plan”
- Clone repo
- List all tags
- For each "version" tag…
- Iterate over all folders in app/code/Magento
- Zip the folder as a composer package using the name and version in the
composer.json file in each directory
- Use composer/satis to use the directory with the zips as an artifact type repo

7. Mage-os.org 22. Apr. 2023
Time to build!
Ivan: Let’s create a Rust repo-splitter that keeps the git repo
in-memory so no working copy is needed!
Vinai: Let’s keep it simple and write a shell script version, too!
> The more implementations we get, the better our understanding of the problem will be and the more
people will be able to work on it (rust, bash, js, …) <

8. Mage-os.org 22. Apr. 2023
One tool made it to production 🎉
The JavaScript implementation started as an
experiment.
Turned out to be good enough!
Also available as a docker image
magece/mageos-repo-js
to build your own Mage-OS repositories!

9. Mage-os.org 22. Apr. 2023
Beside the module & framework packages
magento/project-community-edition metapackage (e.g. 2.4.5-p2)
depends on ⬇️
magento/product-community-edition metapackage (e.g. 2.4.5-p2)
depends on ⬇️
EVERY MAGENTO PACKAGE included in 2.4.5-p2 in a specific version (e.g. 103.0.2)
and also ⬇️
magento/magento2-base (2.4.5-p2)

10. Mage-os.org 22. Apr. 2023
The magento2-base package
Contains base directory filesystem structure, i.e. app/code, bin/, dev/, ...

11. Mage-os.org 22. Apr. 2023
The magento2-base package
It depends on magento/magento-composer-installer.
It declares extra.map in composer.json.
The magento/magento-composer-installer
copies the extra.map files and folders
from the package into the project base.

12. Mage-os.org 22. Apr. 2023
There is no way to determine the extra.map contents programmatically.
From what is visible outside of Magento/Adobe, the map seems to be maintained manually.
The magento2-base package

13. Mage-os.org 22. Apr. 2023
Solution for mirror packages:
copy the magento2-base composer.json of
all past releases from upstream
The magento2-base package

14. Mage-os.org 22. Apr. 2023
There be Dragons Inconsistencies!
The git repos don’t match the old Magento releases exactly…
- Version mismatch in package composer.json and base package composer.json
- No version in package composer.json
- Missing release tags
- Branches with version name instead of a tag (did that branch change after the release??)
- For some releases, no commit in public repo matches the released code exactly
(file missing or file is different)

15. Mage-os.org 22. Apr. 2023
Workarounds
Each problem requires a
different workaround,
e.g. adding missing tags or files.

16. Mage-os.org 22. Apr. 2023
You did this for all versions of Magento 2?
Nope.
We only support releases since 2.3.7-p2.

17. Mage-os.org 22. Apr. 2023
Finally: it installs!

18. Mage-os.org 22. Apr. 2023
Why is there no 2FA for Admin login?

19. Mage-os.org 22. Apr. 2023
Ups! There is more than one git repo!
- https://github.com/mage-os/mirror-security-package
- https://github.com/mage-os/mirror-inventory
- https://github.com/mage-os/mirror-magento2-page-builder
- https://github.com/mage-os/mirror-inventory-composer-installer
- https://github.com/mage-os/mirror-adobe-ims
- https://github.com/mage-os/mirror-adobe-stock-integration
- https://github.com/mage-os/mirror-magento-composer-installer
- https://github.com/mage-os/mirror-composer-root-update-plugin
- https://github.com/mage-os/mirror-composer-dependency-version-audit-plugin
- https://github.com/mage-os/mirror-magento2-sample-data (not a direct dep)
- https://github.com/mage-os/mirror-commerce-data-export (not a direct dep)

20. Mage-os.org 22. Apr. 2023
Different repos, different Structures
- Single package repos, like mage-os/mirror-inventory-composer-installer
- Multiple packages in base, like mage-os/mirror-security-package
- Meta-package in _metapackage dir, like mage-os/mirror-security-package
- Special case subdirectory package, like
mage-os/mirror-magento2/tree/2.4-develop/lib/internal/Magento/Framework/Amqp
- Repos that change the directory structure, like mage-os/mirror-inventory
(first a clone of magento/magento2, then only app/code/Magento/* folders + _metapackage)

21. Mage-os.org 22. Apr. 2023
Additional dependencies!
The dependency on the packages from those repos are added to the magento/product-
community-edition package.

22. Mage-os.org 22. Apr. 2023
Bundled Core Extensions
- amzn/amazon-pay-and-login-magento-2-module
- dotmailer/dotmailer-magento2-extension
- klarna/m2-payments
- paypal/module-braintree
- temando/module-shipping
- vertex/product-magento-module
- yotpo/magento2-module-yotpo-reviews

23. Mage-os.org 22. Apr. 2023
Bundled Core Extensions
- Magento forked the public upstream repos
- These forks are not public
- Solution for mirror:
copy released packages from upstream
- Solution for Mage-OS distro:
no core bundled modules

24. Mage-os.org 22. Apr. 2023
Are you sure everything is working now?

25. Mage-os.org 22. Apr. 2023
Quality Assurance
- Install release from mirror.mage-os.org
- Install release from repo.magento.com
- Compare versions and files

26. Mage-os.org 22. Apr. 2023
We have mirror releases ✅

27. Mage-os.org 22. Apr. 2023
We have mirror releases ✅
Next step: nightly builds!

28. Mage-os.org 22. Apr. 2023
What is different from a mirror build?
Use the head of a branch instead of a git tag.
Figure out a nightly-build version for each package instead of relying on a version
in composer.json.

29. Mage-os.org 22. Apr. 2023
Nightly Package Versions
After considering many options, we now use
- get the latest release tag of the repo
- get the version of all packages
- if less than 4 parts are present: increment last part of the current version,
otherwise increment the last part
- if there is no stability suffix, add an -aYYYYMMDD suffix to version
- otherwise keep the stability suffix and add YYYYMMDD

30. Mage-os.org 22. Apr. 2023
What deps to use in the base package?
- clone repo
- run composer install
- use github.com/mage-os/php-dependency-list to scan every .php, .phtml and di.xml file
- For each referenced PHP class
- Use the composer autoloader to determine which package contains the class
- Record the package name and version as a composer dependency for the base package

31. Mage-os.org 22. Apr. 2023
Upstream Nightly Releases ✅
composer create-project magento/project-community-edition
--repository-url https://upstream-nightly.mage-os.org
--stability alpha ./current-nightly

32. Mage-os.org 22. Apr. 2023
Mage-OS Nightly Releases ✅
The same process as upstream-nightly works
for the mageos-nightly builds, too!

33. Mage-os.org 22. Apr. 2023
New upstream Magento Open Source release?
- copy project-community-edition, product-community-edition and base-package composer.json into
the respective resources/history/magento/* folder with the release version as the name
- run the build
- install and validate new version
- add workarounds for missing tags and such to build config, re-build and test until everything works
- PR to github.com/mage-os/generate-mirror-repo-js, run action against preview-mirror.mage-os.org
- Merge PR, new packages are built and deployed to mirror.mage-os.org

34. Mage-os.org 22. Apr. 2023
GitHub Actions

35. Mage-os.org 22. Apr. 2023
Time for a wee bit of code?

36. Mage-os.org 22. Apr. 2023
The Build Config
Everything is declared in
“the build config”.
Base config extended by build type
(mirror, upstream-nightly or mageos-nightly)
Config is processed by the build script to
generate the package ZIP files.

37. Mage-os.org 22. Apr. 2023
The Build Config

38. Mage-os.org 22. Apr. 2023
Next steps
Come! Pls halp!

39. Mage-os.org 22. Apr. 2023
Next steps
Active Projects:
- Generate a real Mage-OS release
- Faster Integration & Unit Tests - (hi Adam + Damien!)
- New Features - (hey Gowri + Ivan!)
- Content!

40. Mage-os.org 22. Apr. 2023
Join our Community
Adam Crossland (UK)
Alexander Buch (DE)
Andreas Mautz (DE)
Anton Siniorg (EE)
Arron Moss (UK)
Barbaros Selim Büyükelçi (TR)
Benno Lippert (DE)
Bernard Robbins (US)
Brent Peterson (US)
Christian Münch (DE)
Damien Retzinger (US)
Daniel Fahlke (DE)
Daniel Sloof (NL)
David Lambauer (DE)
Dominik Rammert (DE)
Erik Hanson (US)
Gowri Sankar (AU)
Jisse Reitsma (NL)
John Prendergast (IE)
Nicole Mentzen (DE)
Roland Haselager (NL)
Sergei Derzap (BY)
Tim Bucciarelli (US)
Timon de Groot (NL)
Vitaliy Golomozy (UA)
Willem Wigman (NL)
Wouter Steenmeijer (NL)
Thank you to the contributors!

41. Mage-os.org 22. Apr. 2023
Mage-OS
Fork it, Own it, Grow it, Better!
TOGETHER