OpenERP data integration in an entreprise context: a war story

4,945 views

Published on

OpenERPDays 2013 presentation: techniques used and lessons learned from almost 2 years experience integrating data into OpenERP in an enterprise context.

Published in: Technology

OpenERP data integration in an entreprise context: a war story

  1. 1. OpenERP OpenERP Days 2013OpenERP Days 2013 OpenERP enterprise integration: A war story
  2. 2. Daniel Reis IT Applications Manager Securitas Portugal @reis_pt daniel.reis@securitas.pt https://launchpad.net/~dreis-pt
  3. 3. OpenERP as an Enterprise App platform OpenERP Corporate ERP Customers Contracts Analytic accounts HRMS Employees Departments Op Systems Serviced sites Serviced systems
  4. 4. The basic method: import CSV file id, name, ref, categ_id product_product_aa, Prod A, AA, product_categ_X product_product_bb, Prod B, BB, product_categ_X product_product_cc, Prod C, CC, product_categ_Y
  5. 5. External Identifiers aka XML IDs XML ID Model Database ID product_product_aa product.product 879 product_product_bb product.product 880 product_product_cc product.product 881 product_product_AAA product.product 879 product_categ_X product.categ 45 product_categ_y product.categ 56
  6. 6. ORM API: load() product_categ.load(cr, uid, # import_data() on v6.1 [id , name ], # field names [ ['product_categ_X', 'Categ X'], # data rows ['product_categ_Y', 'Categ Y'], ] product_product.load(cr, uid, [id , name , ref , categ_id.id ], [ ['product_product_aa', 'Prod A', 'AA', 'product_categ_X'], ['product_product_bb', 'Prod B', 'BB', 'product_categ_X'], ['product_product_cc', 'Prod C', 'CC', 'product_categ_Y'], ] product_categ.load(cr, uid, # import_data() on v6.1 [id , name ], # field names [ ['product_categ_X', 'Categ X'], # data rows ['product_categ_Y', 'Categ Y'], ] product_product.load(cr, uid, [id , name , ref , categ_id.id ], [ ['product_product_aa', 'Prod A', 'AA', 'product_categ_X'], ['product_product_bb', 'Prod B', 'BB', 'product_categ_X'], ['product_product_cc', 'Prod C', 'CC', 'product_categ_Y'], ]
  7. 7. Import CSV file summary Official method Uses ORM External identifiers Many-to-one support
  8. 8. Automate ?
  9. 9. import_odbc
  10. 10. SQL import module solution OpenOpenERPERP 1. Query external database 2. Build import data structure 3. Call load()
  11. 11. Build "CSV file" from SQL id, name, ref, categ_id product_product_AA, Prod A, AA, product_categ_X product_product_BB, Prod B, BB, product_categ_X product_product_CC, Prod C, CC, product_categ_Y SELECT product.id as “None”, -- generates id column product.name as “name”, product.id as “ref”, 'product_categ_' + product.category_id as “categ_id” FROM product WHERE change_timestamp >= ? -- last sync date
  12. 12. import_odbc module
  13. 13. SQL import module summary OpenOpenERPERP lp:openobject-extension (https://code.launchpad.net/~extra-addons-commiter/openobject-extension ) Concerns Runs in OE server DB middleware installation SQL sources only SQL transforms only
  14. 14. import_odbc etl table loading
  15. 15. Another class of solutions: ETL & Pentaho Kettle OpenOpenERPERP
  16. 16. ETL allows for complex transformations and validations...
  17. 17. The naive way: direct table loading. (Don't do this!)
  18. 18. Pentaho Kettle ETL summary http://kettle.pentaho.com Concerns Learning curve OpenOpenERPERP
  19. 19. import_odbc etl table loading import_sqlpush
  20. 20. OpenOpenERPERP import_sqlpush: push and load solution 1. ETL push data a. Get data b. Build import line c. Write to queue table 2. OE scheduled import a. Read queue table b. Run import_data() c. Update queue status
  21. 21. An example: import categorized products
  22. 22. Import line built by the "data" column
  23. 23. import_sqlpush: push and load summary https://code.launchpad.net/~dreis-pt/reis-openerp-addons/trunk Concerns Two step latency Process in OpenERP server Data rows building logic OpenOpenERPERP
  24. 24. import_odbc etl table loading import_sqlpush openerp steps
  25. 25. OpenOpenERPERP OpenERP output step solution ETL Steps 1. Get data 2. Transformations 3. OpenERP output
  26. 26. An example: Customers and Contracts.
  27. 27. OpenOpenERPERP OpenERP output step issues: roundtrips & no XML Ids 1. search() all model ids 2. read() all lookup fields 3. Find ids for input rows 4. Call import_data()
  28. 28. Pentaho Kettle ETL summary http://tinyurl.com/openerpstep https://bugs.launchpad.net/openobject-server/+bug/1186977 Concerns Slow (roundtrips) No XML ID support Needs server patch on v7 OpenOpenERPERP
  29. 29. import_odbc etl table loading import_sqlpush openerp steps ... better steps...
  30. 30. Concerns OpenOpenERPERP OpenERP output step: evolution is in the works ... ? http://jira.pentaho.com/browse/PDI-9974 http://jira.pentaho.com/browse/PDI-9975
  31. 31. Conclusions: things we should pay attention to ... ● How fast? ● Uses OpenERP ORM? ● How easy to use? ● Interferes with other OE server processes? ● What data source types can be used? ● What type of transformations can be used?
  32. 32. Integration techniques wrap up ● import_odbc: simple, maybe too simple. ● Table loading: don't do this at home kids... ● import_sqlpush: good, but could be simpler. ● OpenERP Output Step: great, but slow ...
  33. 33. http://www.mulesoft.com/mule-esb-features Going further: why not use an ESB?
  34. 34. Also have a look at other options ● TerminatOOR: Kettle Openerp step using Ruby script https://github.com/akretion/ooor ● Talend Data Integration: ETL task code generator (Java) http://www.talend.com/products/data-integration
  35. 35. Questions? @reis_pt daniel.reis@securitas.pt https://launchpad.net/~dreis-pt

×