Transforming and Loading data WebSphere Commerce provides utilities for preparing and loading data into a WebSphere Commerce database. The loading utilities are flexible enough to handle customization that are made to the WebSphere Commerce schema. The two main loading utilities are the mass load utility the Data Load utility The Data Load utility is the recommended loading utility.
Tasks The Data Load utility performs the following functions in a single operation: Reads data from a source. Transforms the source data to WebSphere Commerce business objects. Allocates and resolves WebSphere Commerce business objects to physical data. Loads the physical data into the database. Data BO PD DB
Tip CSV format is the suggested and most used way in data load flow!Why? Cause WCS provides OutOfTheBox a CSV reader
Simple use case 3 Products.CSV Transform & Load
Not so simple use case cat.CSV prod.CSV color.CSV offer.CSV size.CSV … Transform & Load
Complex use case Cat_01012013.CSV Prod_01012013.CSV Color_01012013.CSV Offer_01012013.CSV Size_01012013.CSV … Transform & Load Every Day!
Points of complexity Duplication Correctness Consistency Integrity Big data Performance Data Knowledge…
Architectural Overview Data WCS Source Loader DB Data BO BO Data Reader Builder Mediator Writer Java Business Phisical Objects Objects Objects
Data Load planification 1 Development Phase Developer 1 week 1 year 2 Load Phase 1 second Repeat as PM System Admin 2 months needed
Question A customer needs to load products data from its old e-commerce platform. Which approach would you suggest? A. Export the data as CSV and use dataload tool to load the data contained in the files; B. Export the data as CSV and use massload tool to load the data contained in the files; C. Run custom queries to load the data directly in the Commerce DB;
Run!!!!!! (suggested approach) Log in to local WebSphere Commerce instance (shell) Run the script dataload.sh including the indication of the Configuration Loading Order file. WCS creates multiple logs- by default- for each run: • Main log file • A detailed log file for each Business Object
Basic skills required Deep understanding of WCS DB Schema Deep knowledge of SQL and DBMS Confidence using Management Center Accelerator Organizational Administration Console Experience with ETL procedures and XML configuration tasks Troubleshooting skills Java development
Question What’s the minimum amount of configuration files I need If I want to follow the suggested approach? A. 1 B. 3 C. 5 D. Cannot be defined
Data Load Customizations OOB extension Data Load 0 customizationsUse genericTableObject 1 2 3 Data Column BO Reader Handler Mediator
Use TableObject 0 Extension to the concept of a business object builder; The table object builder is responsible for accepting data read by the data reader and constructing value objects representing physical tables (table data objects) in the data base; The partially populated table data objects will then be passed to the business object mediator for additional population.
Extension Table, an example 0 Personal Wish List population; No OOB Business Object Mediators; Use generic TableObjectBuilder and Mediator; Tables involved: IITEMLIST IITEM
Supported File Formats 1 CSV XML (from FEP6)http://pic.dhe.ibm.com/infocenter/wchelp/v7r0m0/topic/com.ibm.commerce.data.doc/refs/rmldataloadsourcefile.htm
Custom data reader 1 It is not necessary to change the format of your existing data to work with the Data Load utility. Create custom data reader Create new class extends AbstractDataReader; Implement next() abstract method; Override the init(), close(), and getSourcePosition() methods; In the <_config:DataReader> template files use the custom implementation.
Custom column handler 2 Resolve data from database columns based on input values that cannot be mapped to the column value through the configuration; Create custom column handler Create new class extends AbstractColumnHandler; Implement resolveColumnValue() abstract method; Override the init(), close() methods; In the <_config:ColumnHandler> template files-using TableObject- use the custom implementation.
Custom column handler, anexample 2 Password column in the database is a clear text password; Password column in the database table expects an encrypted value; Create a custom column handler to encrypt the clear text password and return an encrypted password from the resolveColumnValue() method.
Custom column handler, anexample 2<_config:Column name="LOGONPASSWORD" value="LOGONPASSWORD" valueFrom="ColumnHandler" > <_config:ColumnHandlerclassName="com.mycompany...PasswordEncryptionHandler"> <_config:Parameter name="logonPassword"value="logonPassword" /> <!-- WC-server.xml is the file to store the merchant key --> <_config:Parameter name="instanceConfigLocation" value="c:IBMWCDE_ENT70workspaceWCxmlconfigwc-server.xml" valueFrom="Fixed" /> </_config:ColumnHandler></_config:Column>
Custom business objectmediator 3 Create custom business object mediator Create new class extends AbstractBusinessObjectMediator; Implement transform() abstract method to transform the custom business objects into a list of physical objects; Override the methods as necessary to achieve mediate functionality; In the <__config:BusinessObjectMediator> template files use the custom implementation.