Download as PDF, PPTX
Uploaded bynetlogix
Building a large eCommerce application with Extbase, Fluid and Apache Solr
This document discusses the development of a new e-commerce system for IBC SOLAR built on TYPO3. It describes some of the challenges in previous systems and decisions made in developing the new system. Key points include using the BMEcat standard for the domain model, leveraging Extbase and the kickstarter to generate model objects, and using Apache Solr for faceted search of product data. Data integration was handled with Pentaho ETL and initial Extbase performance issues were addressed.
Building a large eCommerce application with Extbase, Fluid and Apache Solr
- 1. October 2, 2010 1 Clemens Kalb, Lienhart Woitok, T3CON10 media.netlogix.de
- 2. October 2, 2010 2 Clemens Kalb, Lienhart Woitok, T3CON10 media.netlogix.de
- 3. IBC SOLAR: Selling 857 million turnover everything from a in 2009 solar module to 4-figure number of complete solar plants products x product Working with retail variants x retail partners in Germany partners and throughout the world October 2, 2010 3 Clemens Kalb, Lienhart Woitok, T3CON10 media.netlogix.de
- 4. „Build yet anotherShop System for TYPO3 – are you insane?“ All of us & the customer (before the start of the project) October 2, 2010 4 Clemens Kalb, Lienhart Woitok, T3CON10 media.netlogix.de
- 5. Previous system basedon Commerce • scalability issues tt_products? Hell no! ;-) Specialized functionality lacking in all systems Magento looked fairly nice – but not integrated in TYPO3 October 2, 2010 5 Clemens Kalb, Lienhart Woitok, T3CON10 media.netlogix.de
- 6. October 2, 2010 6 Clemens Kalb, Lienhart Woitok, T3CON10 media.netlogix.de
- 7. October 2, 2010 7 Clemens Kalb, Lienhart Woitok, T3CON10 media.netlogix.de
- 8. October 2, 2010 8 Clemens Kalb, Lienhart Woitok, T3CON10 media.netlogix.de
- 9. October 2, 2010 9 Clemens Kalb, Lienhart Woitok, T3CON10 media.netlogix.de
- 10. All data comesfrom the company‘s ERP system Custom import functionality for complete orders Individual prices for customers No need for retail functions like online payment processing October 2, 2010 10 Clemens Kalb, Lienhart Woitok, T3CON10 media.netlogix.de
- 11. standing on theshoulders of giants October 2, 2010 11 Clemens Kalb, Lienhart Woitok, T3CON10 media.netlogix.de
- 12. How do youdesign a generic domain model for a shop system? October 2, 2010 12 Clemens Kalb, Lienhart Woitok, T3CON10 media.netlogix.de
- 13. You don‘t … (unless you have too much time on your hands) October 2, 2010 13 Clemens Kalb, Lienhart Woitok, T3CON10 media.netlogix.de
- 14. …because it alreadyexists: October 2, 2010 14 Clemens Kalb, Lienhart Woitok, T3CON10 media.netlogix.de
- 15. BMEcat October 2, 2010 15 Clemens Kalb, Lienhart Woitok, T3CON10 media.netlogix.de
- 16. Proven & openstandard for product catalogues Very well documented (specification >600 pages) Originally an XML data exchange format October 2, 2010 16 Clemens Kalb, Lienhart Woitok, T3CON10 media.netlogix.de
- 17. The Extbase kickstarter: a great way to create lots of model objects October 2, 2010 17 Clemens Kalb, Lienhart Woitok, T3CON10 media.netlogix.de
- 18. October 2, 2010 18 Clemens Kalb, Lienhart Woitok, T3CON10 media.netlogix.de
- 19. October 2, 2010 19 Clemens Kalb, Lienhart Woitok, T3CON10 media.netlogix.de
- 20. The kickstarter isnot an editor! October 2, 2010 20 Clemens Kalb, Lienhart Woitok, T3CON10 media.netlogix.de
- 21. Using SCM tocherry-pick changes: 1. Save in kickstarter 2. svn revert <fileThatShouldNotBeTouched> 3. Eclipse diff view to manually merge files October 2, 2010 21 Clemens Kalb, Lienhart Woitok, T3CON10 media.netlogix.de
- 22. Apache Solr asa Storage Backend October 2, 2010 22 Clemens Kalb, Lienhart Woitok, T3CON10 media.netlogix.de
- 23. Java based search server Searches millions of documents (products, text entries, prices…) in very little time Provides faceted search facilities October 2, 2010 23 Clemens Kalb, Lienhart Woitok, T3CON10 media.netlogix.de
- 24. The naive way: writing direct querys to the Solr server from the application code October 2, 2010 24 Clemens Kalb, Lienhart Woitok, T3CON10 media.netlogix.de
- 25. NO Faceted Extbase Repository Query FACETED persistence Query QUERIES layer Background storage Solr storage Storage Backend backend (DB, Solr…) October 2, 2010 25 Clemens Kalb, Lienhart Woitok, T3CON10 media.netlogix.de
- 26. Tx_Extbase_Persistence_Query $query->equals(‘foobar’, $foobar) $query->logicalAnd($statement1, $statement2) $query->in(‘foobar’, array($item1, $item2)) October 2, 2010 26 Clemens Kalb, Lienhart Woitok, T3CON10 media.netlogix.de
- 27. Tx_Nxsolrbackend_Persistence_Backend_ SolrBackend foobar:$foobar ($statement1 AND $statement2) foobar:($item1 $item2) October 2, 2010 27 Clemens Kalb, Lienhart Woitok, T3CON10 media.netlogix.de
- 28. Tx_Nxsolrbackend_Persistence_FacetedQuery extends Tx_Extbase_Persistence_Query $query->facet($facetConfiguration) $query->execute(): Tx_Nxsolrbackend_Persistence_ FacetedQueryResult $facetedQueryResult->getObjects() $facetedQueryResult->getFacets() October 2, 2010 28 Clemens Kalb, Lienhart Woitok, T3CON10 media.netlogix.de
- 29. Structural differences: databasevs. index Data duplication: database and Solr index Extbase knows only one StorageBackend October 2, 2010 29 Clemens Kalb, Lienhart Woitok, T3CON10 media.netlogix.de
- 30. Pentaho Data Integration October2, 2010 30 Clemens Kalb, Lienhart Woitok, T3CON10 media.netlogix.de
- 31. How to feeddata into the database from external sources? October 2, 2010 31 Clemens Kalb, Lienhart Woitok, T3CON10 media.netlogix.de
- 32. (Photo courtesy oftrenttsd/flickr) October 2, 2010 32 Clemens Kalb, Lienhart Woitok, T3CON10 media.netlogix.de
- 33. Looking for theright tool: Integrate different data sources homogenously Source data from Microsoft Dynamics NAV (MSSQL) CSV file to specify additional files for products like datasheets and the like Need to understand it 3 years later October 2, 2010 33 Clemens Kalb, Lienhart Woitok, T3CON10 media.netlogix.de
- 34. Pentaho Data Integration Extract, Transform and Load JDBC + plugins for different data sources/targets October 2, 2010 34 Clemens Kalb, Lienhart Woitok, T3CON10 media.netlogix.de
- 35. Configurability: import intodifferent sysfolders, configure parameters and default values Where needed: Use JS and external code for special transformations October 2, 2010 35 Clemens Kalb, Lienhart Woitok, T3CON10 media.netlogix.de
- 36. How fast isExtbase? October 2, 2010 36 Clemens Kalb, Lienhart Woitok, T3CON10 media.netlogix.de
- 37. First implementations werepainfully slow due to… Working with thousands of objects Bug in Extbase: most objects considered dirty at any time October 2, 2010 37 Clemens Kalb, Lienhart Woitok, T3CON10 media.netlogix.de
- 38. Build first, optimizelater! (Might be scary at times, but the approach does actually work…) October 2, 2010 38 Clemens Kalb, Lienhart Woitok, T3CON10 media.netlogix.de
- 39. Optimization #1: Make heavy use of lazy loading, load objects only when needed October 2, 2010 39 Clemens Kalb, Lienhart Woitok, T3CON10 media.netlogix.de
- 40. Optimization #2: Fix Extbase: Objects no longer dirty ;-) October 2, 2010 40 Clemens Kalb, Lienhart Woitok, T3CON10 media.netlogix.de
- 41. Optimization #3: Not everything needs to be an object: If you don‘t need domain logic, you might just render raw query results October 2, 2010 41 Clemens Kalb, Lienhart Woitok, T3CON10 media.netlogix.de
- 42. Optimization #4: Use the good old TYPO3 caching mechanisms, but be especially careful October 2, 2010 42 Clemens Kalb, Lienhart Woitok, T3CON10 media.netlogix.de
- 43. October 2, 2010 43 Clemens Kalb, Lienhart Woitok, T3CON10 media.netlogix.de
- 44. October 2, 2010 44 Clemens Kalb, Lienhart Woitok, T3CON10 media.netlogix.de