Download as PDF, PPTX
Uploaded byAOE
Multithreaded XML Import (San Francisco Magento Meetup)
This document discusses Aoe_Import, an open source PHP library for importing XML files into Magento in a multithreaded manner. It describes how Aoe_Import uses forking and the Threadi library to process imports across multiple processes to avoid memory issues. It also addresses some challenges with multithreading like database connections and thread safety. The presenter suggests other use cases for this approach like processing scheduler jobs, queues, and indexes in a batched, multithreaded way.
More Related Content
Multithreaded XML Import (San Francisco Magento Meetup)
- 1. Multithreaded XML Import …for Magento SanFrancisco Magento Meetup Group - October 23, 2013
- 2.
- 3.
- 4.
- 5. Will Aoe_Import bethe fastest product importer around? YES, of course! Well, maybe… Actually, Aoe_Import is only a XML Importer “Framework”. It’s up to you to decide how to handle the xml snippets…
- 6. for large XMLfiles XML! Not CSV. full flexibility in processor implementation Aoe_Import multi-thread support! Subscribe your “Processors” to xpaths Stream processing (XMLReader) “event” driven
- 7.
- 8.
- 9. Beat the memoryLeak by forking Waiting for other thread to terminate Threading overhead Process import
- 10. Forking? In PHP? $pid =pcntl_fork(); if ($pid) { // parent process runs what is here echo "parentn"; } else { // child process runs what is here echo "childn"; }
- 11.
- 12. Clean OOP interfacefor PHP to forking and process management Threadi
- 13. Batch Processor Collect abunch of imports … …fork… …and process them in a child process.
- 14. No imports areprocessed in the main thread. So there’s no memory leak happing here Main thread memory Memory limit time Create process collection Waiting for other thread to terminate Threading Process imports in process collection overhead Forks Every fork starts with the low memory footprint of the main thread Find the number of imports that can be processed at a time without hitting the memory limit
- 15. Multi-threading? Sure! Number of threads processed inparallel Number of items in a batch
- 16. Problems? Database Connection Database connection doesn’t liketo be cloned! Mage::getSingleton('core/resource') ->getConnection('core_write') ->closeConnection();
- 17.
- 18. Problems? Thread Safety --- a/app/code/core/Enterprise/Catalog/Model/Index/Action/Catalog/Category/Product/Refresh.php +++b/app/code/core/Enterprise/Catalog/Model/Index/Action/Catalog/Category/Product/Refresh.php @@ -326,7 +326,7 @@ class Enterprise_Catalog_Model_Index_Action_Catalog_Category_Product_Refresh ->setComment('Catalog Category Product Index Tmp'); $this->_connection->dropTable($this->_getMainTmpTable()); $this->_connection->createTable($table); $this->_connection->createTemporaryTable($table); + } /**
- 19.
- 20. Thank you! Any questions? Myblog http://www.aoemedia.com http://www.fabrizio-branca.de @fbrnc Follow me on twitter!