WebSphere Commerce v7 Data Load


Published on

Brief review of WebSphere Commerce Data Load focused on development and customization.

Published in: Technology
No Downloads
Total Views
On Slideshare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

WebSphere Commerce v7 Data Load

  1. 1. WebSphere Commerce v7 An overviewFrancesco Schettini @FranSchettiniE-Commerce Solution ArchitectFreelance http://fschettini.com
  2. 2. Overview
  3. 3. Scope CSVs Trasformation WebSphere & Loading Commerce • Products XML • Attributes • Prices Legacy • Filters • Inventory • Promotions • Categories • …
  4. 4. Transforming and Loading data WebSphere Commerce provides utilities for preparing and loading data into a WebSphere Commerce database. The loading utilities are flexible enough to handle customization that are made to the WebSphere Commerce schema. The two main loading utilities are  the mass load utility  the Data Load utility The Data Load utility is the recommended loading utility.
  5. 5. Tasks The Data Load utility performs the following functions in a single operation:  Reads data from a source.  Transforms the source data to WebSphere Commerce business objects.  Allocates and resolves WebSphere Commerce business objects to physical data.  Loads the physical data into the database. Data BO PD DB
  6. 6. Tip CSV format is the suggested and most used way in data load flow!Why? Cause WCS provides OutOfTheBox a CSV reader
  7. 7. Simple use case 3 Products.CSV Transform & Load
  8. 8. Not so simple use case cat.CSV prod.CSV color.CSV offer.CSV size.CSV … Transform & Load
  9. 9. Complex use case Cat_01012013.CSV Prod_01012013.CSV Color_01012013.CSV Offer_01012013.CSV Size_01012013.CSV … Transform & Load Every Day!
  10. 10. Points of complexity Duplication Correctness Consistency Integrity Big data Performance Data Knowledge…
  11. 11. Architectural Overview Data WCS Source Loader DB Data BO BO Data Reader Builder Mediator Writer Java Business Phisical Objects Objects Objects
  12. 12. Data Load planification 1 Development Phase Developer 1 week 1 year 2 Load Phase 1 second Repeat as PM System Admin 2 months needed
  13. 13. Question A customer needs to load products data from its old e-commerce platform. Which approach would you suggest? A. Export the data as CSV and use dataload tool to load the data contained in the files; B. Export the data as CSV and use massload tool to load the data contained in the files; C. Run custom queries to load the data directly in the Commerce DB;
  14. 14. Development and Run
  15. 15. Classic ETL development tasks Create reader component Create transformer component Create loader component Extraction Transformation Loading
  16. 16. WCS loves you! CSV Extraction 1 Reader com.ibm...CSVReader BO Trasformation 2 Builder & com.ibm...BaseBusinessObjectBuilder Loader com.ibm…BusinessObjectLoader Mediator Loading 3 & Data Writer com.ibm...CatalogGroupMediator com.ibm...JDBCDataWriter
  17. 17. Developer’s Tasks Create Business Object templates files Create Environment’s configuration files Create Load order configuration files
  18. 18. Business Object Template’s content CSV reader configuration Business Object Builder Data Mapping Mediator configuration Data Writer configuration
  19. 19. Tip Create a different Template for each Business Object
  20. 20. BO Template, an example
  21. 21. Configuring Environment Default Store Default Language Default Currency Data Base connection coordinates IDResolver configuration Default Data Writer
  22. 22. Configuring Environment, anexample
  23. 23. Configuring Loading Order Environment configuration file Business object loading order  CSV file  Business Object Template File
  24. 24. Configuring Loading Order, anexample
  25. 25. Run!!!!!! (suggested approach) Log in to local WebSphere Commerce instance (shell) Run the script dataload.sh including the indication of the Configuration Loading Order file. WCS creates multiple logs- by default- for each run: • Main log file • A detailed log file for each Business Object
  26. 26. Basic skills required Deep understanding of WCS DB Schema Deep knowledge of SQL and DBMS Confidence using  Management Center  Accelerator  Organizational Administration Console Experience with ETL procedures and XML configuration tasks Troubleshooting skills Java development
  27. 27. Question What’s the minimum amount of configuration files I need If I want to follow the suggested approach? A. 1 B. 3 C. 5 D. Cannot be defined
  28. 28. Customization
  29. 29. Data Load Customizations OOB extension Data Load 0 customizationsUse genericTableObject 1 2 3 Data Column BO Reader Handler Mediator
  30. 30. Use TableObject 0 Extension to the concept of a business object builder; The table object builder is responsible for accepting data read by the data reader and constructing value objects representing physical tables (table data objects) in the data base; The partially populated table data objects will then be passed to the business object mediator for additional population.
  31. 31. Extension Table, an example 0 Personal Wish List population; No OOB Business Object Mediators; Use generic TableObjectBuilder and Mediator; Tables involved:  IITEMLIST  IITEM
  32. 32. Extension Table, example 1 0<_config:BusinessObjectBuilder className="com…TableObjectBuilder" ><_config:Table name="IITEMLIST" excludeUnListedColumns="true"> <_config:Column name="IITEMLIST_ID" value="itemlist_id” /> <_config:Column name="MEMBER_ID" value="cust_id” /></_config:Table><_config:Table name="IITEM" excludeUnListedColumns="true"> <_config:Column name="CATENTRY_ID" value="catentry_id” /> <_config:Column name="IITEMLIST_ID" value="itemlist_id” /> <_config:Column name="STOREENT_ID" value="storeent_id” /> <_config:Column name="MEMBER_ID" value="cust_id” /> <_config:Column name="QUANTITY" value="quantity” /></_config:Table><_config:BusinessObjectMediator className="com...TableObjectMediator" /></_config:BusinessObjectBuilder>
  33. 33. Extension Table, example 2 0<_config:BusinessObjectBuilder className="com...TableObjectBuilder" > <_config:Table name="USERREG" excludeUnListedColumns="true"> <_config:Column name="USERS_ID" value="USERS_ID" valueFrom="IDResolve" > <_config:IDResolve tableName="USERREG" generateNewKey="false"> </_config:IDResolve> </_config:Column> <_config:Column name="LOGONID" value="login" /> <_config:Column name="PLCYACCT_ID" value="-1" valueFrom="Fixed" /> </_config:Table>… … <_config:BusinessObjectMediator className="com...TableObjectMediator" /></_config:BusinessObjectBuilder>
  34. 34. Supported File Formats 1 CSV XML (from FEP6)http://pic.dhe.ibm.com/infocenter/wchelp/v7r0m0/topic/com.ibm.commerce.data.doc/refs/rmldataloadsourcefile.htm
  35. 35. Custom data reader 1 It is not necessary to change the format of your existing data to work with the Data Load utility. Create custom data reader  Create new class extends AbstractDataReader;  Implement next() abstract method;  Override the init(), close(), and getSourcePosition() methods;  In the <_config:DataReader> template files use the custom implementation.
  36. 36. Custom column handler 2 Resolve data from database columns based on input values that cannot be mapped to the column value through the configuration; Create custom column handler  Create new class extends AbstractColumnHandler;  Implement resolveColumnValue() abstract method;  Override the init(), close() methods;  In the <_config:ColumnHandler> template files-using TableObject- use the custom implementation.
  37. 37. Custom column handler, anexample 2 Password column in the database is a clear text password; Password column in the database table expects an encrypted value; Create a custom column handler to encrypt the clear text password and return an encrypted password from the resolveColumnValue() method.
  38. 38. Custom column handler, anexample 2<_config:Column name="LOGONPASSWORD" value="LOGONPASSWORD" valueFrom="ColumnHandler" > <_config:ColumnHandlerclassName="com.mycompany...PasswordEncryptionHandler"> <_config:Parameter name="logonPassword"value="logonPassword" /> <!-- WC-server.xml is the file to store the merchant key --> <_config:Parameter name="instanceConfigLocation" value="c:IBMWCDE_ENT70workspaceWCxmlconfigwc-server.xml" valueFrom="Fixed" /> </_config:ColumnHandler></_config:Column>
  39. 39. Default BO Mediators 3 Catalog Mediators Inventory Mediators Price Mediatorshttp://pic.dhe.ibm.com/infocenter/wchelp/v7r0m0/topic/com.ibm.commerce.data.doc/refs/rmlmediators.htm
  40. 40. Custom business objectmediator 3 Create custom business object mediator  Create new class extends AbstractBusinessObjectMediator;  Implement transform() abstract method to transform the custom business objects into a list of physical objects;  Override the methods as necessary to achieve mediate functionality;  In the <__config:BusinessObjectMediator> template files use the custom implementation.
  41. 41. A real common case
  42. 42. The legacy architecture 3rd party Live System Stores Product Managers Marketing Managers …
  43. 43. Why do not introduce WCS? 3rd party System Stores Product Managers Marketing Managers … CSV files WCS
  44. 44. Why do not use WCS? 3rd party System Stores WCS Product Managers Marketing Managers …
  45. 45. Involve DL tasks 3rd party System Stores Product Managers Marketing Managers … CSV Data WCS files Load Data
  46. 46. The WCS architecture Staging Production WCS WCS instance Cluster FE Store CMC propagation Production Staging DB Production DB Main WS DB WS1
  47. 47. The WCS architecture + DL Staging Production WCS WCS instance ClusterCSV FE Store CSV CMCfiles files propagation Production DL Staging Production DL DBCatalog DB Main WS DB Inventory WS1 1 2 3
  48. 48. Problems to overcome Complex CSV data structure Complex mapping with WCS BOs Daily loading of huge CSV files One single file containing whole data set Not clean data…
  49. 49. References http://pic.dhe.ibm.com/infocenter/wchelp/v7r0m0/in dex.jsp
  50. 50. The end!Thanks for your attention Francesco Schettini @FranSchettini E-Commerce Solution Architect Freelance http://fschettini.com