This document discusses optimizing the Magento upgrade process by applying multi-processing techniques. The standard Magento upgrade from 1.10 to 1.11 can take a disproportionate amount of time, causing system downtime. The author developed a prototype that breaks the upgrade into parallel parts that are executed by multiple child processes simultaneously. Testing on a production store reduced the upgrade time from 34 hours to 4 hours for a database with 2.5 million orders.

1. Applying Multi-processing
techniques in Magento for
optimizing the upgrade process
Kirill Morozov
Expert Consulting Group
kirill.morozov@magento.com
Github: kirmorozov
Kaprun, Austria
20 March 2013

2. Outline
Problem
• Magento Upgrade 1.10-1.11.X takes a disproportionate
amount of time
• PHP does not have native Threads
Results from production
• Concept
• Execution and Parallel Parts
• Building prototype
• Multi-process upgrade

3. Problem
Magento Upgrade 1.10-1.11.X
System is idle

4. System Downtime means loss of business
Problem
Magento Upgrade 1.10-1.11.X

5. Problem
Magento Upgrade 1.10-1.11.X
Start Point
•70 Gb DB size, 2.5 millions of orders in database,
•Upgrade from 1.9 to 1.11 EE (1.4 - 1.6 CE) takes 34 hours.
Goal
•Protect Data Integrrity
•Minimize downtime
Available tools
•PHP, Shell, Mysql, Web-Server

6. Staging
• Prepare for logging
Staging
• Upgrade
• Collect Log
Production
• Read Log
• Filter Overhead
• Optimize sequence
• Merge queries
• Execute Queries
• Run native data-
upgrades
Results from Production
Concept

7. Remove
Foreign Keys
•Table 1
•Table 2
•…..
Add/Rename
Columns
•Table 1
•Table 2
•…
Create
Foreign Keys
•Table 1
•Table 2
•…
Drop
Columns
Data
modifications
•Sequential
Results from Production
Execution and Parallel Parts

8. Master Initialization
• Create FIFO
• Open Child processes
• Send tasks to child
Child Loop
• Read STDIN
• Process request
• Write response to STDOUT
• Write Id to FIFO
Master Loop
• Reading next Child Id from FIFO
• Checks response from process STDOUT
• In case of no tasks exits loop.
• Sends new task to child
Finalize
• Close Child processes
Results from Production
Building prototype

9. Results from Production
Multi-process upgrade
• DB: 2.5 millions of orders
• Original time: 34 hours
• Optimized replay time: 4 hours
Result 1 (1.9 EE -> 1.11 EE)
• DB: 700k products, ~1kk of orders
• Original time: 15 hours
• Optimized replay time: 2 hours
Result 2 (1.4 CE -> 1.12 EE)

10. References
https://github.com/kirmorozov/Magento-Upgrade-Replay
GitHub
• dalehurley / PHP-Process-Manager
• pfeigl/ZendX
Google
• php multithreading (top 20)
Prior prototypes were done via
• Erlang
• Shell

11. Q/A?
https://github.com/kirmorozov/Magento-Upgrade-Replay
Kirill Morozov
Expert Consulting Group
kirill.morozov@magento.com
Kaprun, Austria
20 March 2013