Brief review of WebSphere Commerce Data Load focused on development and customization.

1. WebSphere Commerce v7
An overview
Francesco Schettini @FranSchettini
E-Commerce Solution Architect
Freelance
http://fschettini.com

2. Overview

3. Scope
CSVs
Trasformation WebSphere
& Loading Commerce
• Products
XML • Attributes
• Prices
Legacy • Filters
• Inventory
• Promotions
• Categories
• …

4. Transforming and Loading data
 WebSphere Commerce provides utilities for
preparing and loading data into a WebSphere
Commerce database.
 The loading utilities are flexible enough to handle
customization that are made to the WebSphere
Commerce schema.
 The two main loading utilities are
 the mass load utility
 the Data Load utility
 The Data Load utility is the recommended loading
utility.

5. Tasks
 The Data Load utility performs the following functions
in a single operation:
 Reads data from a source.
 Transforms the source data to WebSphere Commerce
business objects.
 Allocates and resolves WebSphere Commerce business
objects to physical data.
 Loads the physical data into the database.
Data BO PD DB

6. Tip
 CSV format is the suggested and most used way in
data load flow!
Why? Cause WCS provides OutOfTheBox a
CSV reader

7. Simple use case
3 Products.CSV
Transform & Load

8. Not so simple use case
cat.CSV prod.CSV color.CSV
offer.CSV size.CSV …
Transform & Load

9. Complex use case
Cat_01012013.CSV Prod_01012013.CSV Color_01012013.CSV
Offer_01012013.CSV Size_01012013.CSV …
Transform & Load
Every Day!

10. Points of complexity
 Duplication
 Correctness
 Consistency
 Integrity
 Big data
 Performance
 Data Knowledge
…

11. Architectural Overview
Data WCS
Source
Loader DB
Data BO BO Data
Reader Builder Mediator Writer
Java Business Phisical
Objects Objects Objects

12. Data Load planification
1 Development
Phase
Developer 1 week
1 year
2 Load
Phase
1 second Repeat as
PM
System Admin 2 months needed

13. Question
 A customer needs to load products data from its old
e-commerce platform. Which approach would you
suggest?
A. Export the data as CSV and use dataload tool to load
the data contained in the files;
B. Export the data as CSV and use massload tool to load
the data contained in the files;
C. Run custom queries to load the data directly in the
Commerce DB;

14. Development and Run

15. Classic ETL development tasks
 Create reader component
 Create transformer component
 Create loader component
Extraction
Transformation
Loading

16. WCS loves you!
CSV Extraction
1 Reader com.ibm...CSVReader
BO Trasformation
2 Builder & com.ibm...BaseBusinessObjectBuilder
Loader com.ibm…BusinessObjectLoader
Mediator Loading
3 & Data
Writer
com.ibm...CatalogGroupMediator
com.ibm...JDBCDataWriter

17. Developer’s Tasks
 Create Business Object templates files
 Create Environment’s configuration files
 Create Load order configuration files

18. Business Object Template’s content
 CSV reader configuration
 Business Object Builder Data Mapping
 Mediator configuration
 Data Writer configuration

19. Tip
 Create a different Template for each Business
Object

20. BO Template, an example

21. Configuring Environment
 Default Store
 Default Language
 Default Currency
 Data Base connection coordinates
 IDResolver configuration
 Default Data Writer

22. Configuring Environment, an
example

23. Configuring Loading Order
 Environment configuration file
 Business object loading order
 CSV file
 Business Object Template File

24. Configuring Loading Order, an
example

25. Run!!!!!! (suggested approach)
 Log in to local WebSphere Commerce instance (shell)
 Run the script dataload.sh including the indication of
the Configuration Loading Order file.
WCS creates multiple logs- by
default- for each run:
• Main log file
• A detailed log file for each
Business Object

26. Basic skills required
 Deep understanding of WCS DB Schema
 Deep knowledge of SQL and DBMS
 Confidence using
 Management Center
 Accelerator
 Organizational Administration Console
 Experience with ETL procedures and XML
configuration tasks
 Troubleshooting skills
 Java development

27. Question
 What’s the minimum amount of configuration files I
need If I want to follow the suggested approach?
A. 1
B. 3
C. 5
D. Cannot be defined

28. Customization

29. Data Load Customizations
OOB extension Data
Load
0
customizations
Use generic
TableObject
1 2 3
Data Column BO
Reader Handler Mediator

30. Use TableObject 0
 Extension to the concept of a business object builder;
 The table object builder is responsible for accepting
data read by the data reader and constructing value
objects representing physical tables (table data
objects) in the data base;
 The partially populated table data objects will then
be passed to the business object mediator for
additional population.

31. Extension Table, an example 0
 Personal Wish List population;
 No OOB Business Object Mediators;
 Use generic TableObjectBuilder and Mediator;
 Tables involved:
 IITEMLIST
 IITEM

32. Extension Table, example 1 0
<_config:BusinessObjectBuilder className="com…TableObjectBuilder" >
<_config:Table name="IITEMLIST" excludeUnListedColumns="true">
<_config:Column name="IITEMLIST_ID" value="itemlist_id” />
<_config:Column name="MEMBER_ID" value="cust_id” />
</_config:Table>
<_config:Table name="IITEM" excludeUnListedColumns="true">
<_config:Column name="CATENTRY_ID" value="catentry_id” />
<_config:Column name="IITEMLIST_ID" value="itemlist_id” />
<_config:Column name="STOREENT_ID" value="storeent_id” />
<_config:Column name="MEMBER_ID" value="cust_id” />
<_config:Column name="QUANTITY" value="quantity” />
</_config:Table>
<_config:BusinessObjectMediator className="com...TableObjectMediator" />
</_config:BusinessObjectBuilder>

33. Extension Table, example 2 0
<_config:BusinessObjectBuilder className="com...TableObjectBuilder" >
<_config:Table name="USERREG" excludeUnListedColumns="true">
<_config:Column name="USERS_ID" value="USERS_ID" valueFrom="IDResolve" >
<_config:IDResolve tableName="USERREG" generateNewKey="false"
>
</_config:IDResolve>
</_config:Column>
<_config:Column name="LOGONID" value="login" />
<_config:Column name="PLCYACCT_ID" value="-1" valueFrom="Fixed" />
</_config:Table>…
…
<_config:BusinessObjectMediator className="com...TableObjectMediator" />
</_config:BusinessObjectBuilder>

34. Supported File Formats 1
 CSV
 XML (from FEP6)
http://pic.dhe.ibm.com/infocenter/wchelp/v7r0m0
/topic/com.ibm.commerce.data.doc/refs/rmldatalo
adsourcefile.htm

35. Custom data reader 1
 It is not necessary to change the format of your
existing data to work with the Data Load utility.
 Create custom data reader
 Create new class extends AbstractDataReader;
 Implement next() abstract method;
 Override the init(), close(), and getSourcePosition()
methods;
 In the <_config:DataReader> template files use the
custom implementation.

36. Custom column handler 2
 Resolve data from database columns based on input
values that cannot be mapped to the column value
through the configuration;
 Create custom column handler
 Create new class extends AbstractColumnHandler;
 Implement resolveColumnValue() abstract method;
 Override the init(), close() methods;
 In the <_config:ColumnHandler> template files-using
TableObject- use the custom implementation.

37. Custom column handler, an
example 2
 Password column in the database is a clear text
password;
 Password column in the database table expects an
encrypted value;
 Create a custom column handler to encrypt the clear
text password and return an encrypted password from
the resolveColumnValue() method.

38. Custom column handler, an
example 2
<_config:Column
name="LOGONPASSWORD"
value="LOGONPASSWORD"
valueFrom="ColumnHandler" >
<_config:ColumnHandler
className="com.mycompany...PasswordEncryptionHandler">
<_config:Parameter name="logonPassword"
value="logonPassword" />
<!-- WC-server.xml is the file to store the merchant key -->
<_config:Parameter name="instanceConfigLocation"
value="c:IBMWCDE_ENT70workspaceWCxmlconfigwc-
server.xml"
valueFrom="Fixed" />
</_config:ColumnHandler>
</_config:Column>

39. Default BO Mediators 3
 Catalog Mediators
 Inventory Mediators
 Price Mediators
http://pic.dhe.ibm.com/infocenter/wchelp/v7r0m0/topic/co
m.ibm.commerce.data.doc/refs/rmlmediators.htm

40. Custom business object
mediator 3
 Create custom business object mediator
 Create new class extends
AbstractBusinessObjectMediator;
 Implement transform() abstract method to transform
the custom business objects into a list of physical
objects;
 Override the methods as necessary to achieve mediate
functionality;
 In the <__config:BusinessObjectMediator> template
files use the custom implementation.

41. A real common case

42. The legacy architecture
3rd party Live
System Stores
Product Managers
Marketing Managers
…

43. Why do not introduce WCS?
3rd party
System Stores
Product Managers
Marketing Managers
…
CSV
files WCS

44. Why do not use WCS?
3rd party
System Stores
WCS
Product Managers
Marketing Managers
…

45. Involve DL tasks
3rd party
System Stores
Product Managers
Marketing Managers
…
CSV Data WCS
files Load Data

46. The WCS architecture
Staging Production
WCS WCS
instance Cluster
FE Store
CMC
propagation
Production
Staging DB Production
DB Main WS DB
WS1

47. The WCS architecture + DL
Staging Production
WCS WCS
instance Cluster
CSV FE Store CSV
CMC
files files
propagation
Production
DL Staging Production DL
DB
Catalog DB Main WS DB Inventory
WS1
1 2 3

48. Problems to overcome
 Complex CSV data structure
 Complex mapping with WCS BOs
 Daily loading of huge CSV files
 One single file containing whole data set
 Not clean data
…

49. References
 http://pic.dhe.ibm.com/infocenter/wchelp/v7r0m0/in
dex.jsp

50. The end!
Thanks for your attention
Francesco Schettini @FranSchettini
E-Commerce Solution Architect
Freelance
http://fschettini.com