4. Transforming and Loading data
WebSphere Commerce provides utilities for
preparing and loading data into a WebSphere
Commerce database.
The loading utilities are flexible enough to handle
customization that are made to the WebSphere
Commerce schema.
The two main loading utilities are
the mass load utility
the Data Load utility
The Data Load utility is the recommended loading
utility.
5. Tasks
The Data Load utility performs the following functions
in a single operation:
Reads data from a source.
Transforms the source data to WebSphere Commerce
business objects.
Allocates and resolves WebSphere Commerce business
objects to physical data.
Loads the physical data into the database.
Data BO PD DB
6. Tip
CSV format is the suggested and most used way in
data load flow!
Why? Cause WCS provides OutOfTheBox a
CSV reader
8. Not so simple use case
cat.CSV prod.CSV color.CSV
offer.CSV size.CSV …
Transform & Load
9. Complex use case
Cat_01012013.CSV Prod_01012013.CSV Color_01012013.CSV
Offer_01012013.CSV Size_01012013.CSV …
Transform & Load
Every Day!
10. Points of complexity
Duplication
Correctness
Consistency
Integrity
Big data
Performance
Data Knowledge
…
11. Architectural Overview
Data WCS
Source
Loader DB
Data BO BO Data
Reader Builder Mediator Writer
Java Business Phisical
Objects Objects Objects
12. Data Load planification
1 Development
Phase
Developer 1 week
1 year
2 Load
Phase
1 second Repeat as
PM
System Admin 2 months needed
13. Question
A customer needs to load products data from its old
e-commerce platform. Which approach would you
suggest?
A. Export the data as CSV and use dataload tool to load
the data contained in the files;
B. Export the data as CSV and use massload tool to load
the data contained in the files;
C. Run custom queries to load the data directly in the
Commerce DB;
21. Configuring Environment
Default Store
Default Language
Default Currency
Data Base connection coordinates
IDResolver configuration
Default Data Writer
25. Run!!!!!! (suggested approach)
Log in to local WebSphere Commerce instance (shell)
Run the script dataload.sh including the indication of
the Configuration Loading Order file.
WCS creates multiple logs- by
default- for each run:
• Main log file
• A detailed log file for each
Business Object
26. Basic skills required
Deep understanding of WCS DB Schema
Deep knowledge of SQL and DBMS
Confidence using
Management Center
Accelerator
Organizational Administration Console
Experience with ETL procedures and XML
configuration tasks
Troubleshooting skills
Java development
27. Question
What’s the minimum amount of configuration files I
need If I want to follow the suggested approach?
A. 1
B. 3
C. 5
D. Cannot be defined
29. Data Load Customizations
OOB extension Data
Load
0
customizations
Use generic
TableObject
1 2 3
Data Column BO
Reader Handler Mediator
30. Use TableObject 0
Extension to the concept of a business object builder;
The table object builder is responsible for accepting
data read by the data reader and constructing value
objects representing physical tables (table data
objects) in the data base;
The partially populated table data objects will then
be passed to the business object mediator for
additional population.
31. Extension Table, an example 0
Personal Wish List population;
No OOB Business Object Mediators;
Use generic TableObjectBuilder and Mediator;
Tables involved:
IITEMLIST
IITEM
35. Custom data reader 1
It is not necessary to change the format of your
existing data to work with the Data Load utility.
Create custom data reader
Create new class extends AbstractDataReader;
Implement next() abstract method;
Override the init(), close(), and getSourcePosition()
methods;
In the <_config:DataReader> template files use the
custom implementation.
36. Custom column handler 2
Resolve data from database columns based on input
values that cannot be mapped to the column value
through the configuration;
Create custom column handler
Create new class extends AbstractColumnHandler;
Implement resolveColumnValue() abstract method;
Override the init(), close() methods;
In the <_config:ColumnHandler> template files-using
TableObject- use the custom implementation.
37. Custom column handler, an
example 2
Password column in the database is a clear text
password;
Password column in the database table expects an
encrypted value;
Create a custom column handler to encrypt the clear
text password and return an encrypted password from
the resolveColumnValue() method.
38. Custom column handler, an
example 2
<_config:Column
name="LOGONPASSWORD"
value="LOGONPASSWORD"
valueFrom="ColumnHandler" >
<_config:ColumnHandler
className="com.mycompany...PasswordEncryptionHandler">
<_config:Parameter name="logonPassword"
value="logonPassword" />
<!-- WC-server.xml is the file to store the merchant key -->
<_config:Parameter name="instanceConfigLocation"
value="c:IBMWCDE_ENT70workspaceWCxmlconfigwc-
server.xml"
valueFrom="Fixed" />
</_config:ColumnHandler>
</_config:Column>
40. Custom business object
mediator 3
Create custom business object mediator
Create new class extends
AbstractBusinessObjectMediator;
Implement transform() abstract method to transform
the custom business objects into a list of physical
objects;
Override the methods as necessary to achieve mediate
functionality;
In the <__config:BusinessObjectMediator> template
files use the custom implementation.
43. Why do not introduce WCS?
3rd party
System Stores
Product Managers
Marketing Managers
…
CSV
files WCS
44. Why do not use WCS?
3rd party
System Stores
WCS
Product Managers
Marketing Managers
…
45. Involve DL tasks
3rd party
System Stores
Product Managers
Marketing Managers
…
CSV Data WCS
files Load Data
46. The WCS architecture
Staging Production
WCS WCS
instance Cluster
FE Store
CMC
propagation
Production
Staging DB Production
DB Main WS DB
WS1
47. The WCS architecture + DL
Staging Production
WCS WCS
instance Cluster
CSV FE Store CSV
CMC
files files
propagation
Production
DL Staging Production DL
DB
Catalog DB Main WS DB Inventory
WS1
1 2 3
48. Problems to overcome
Complex CSV data structure
Complex mapping with WCS BOs
Daily loading of huge CSV files
One single file containing whole data set
Not clean data
…