Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.
Sascha Wenninger
Integration Architect, Blue T
The CSV File Strikes Back
• Technology Architect
• Integration Focus
• Co-Founder, Blue T
• @sufw
About Me
• RESTful APIs
• Modern Message Queues
• Microservices
• Event Stream Processing
• Polyglot Programming
Integration Trends
Diversity
Image by Alexandre Dulaunoy
Use the Right Tool for the Job!
Image by Bruce Murray
Interaction?
A
B
B
A
A pushes to B
B fetches from A
Data Volume?
File Transfer
Messaging
SOA
REST
100B 1kB 10kB 100kB 1MB 10MB 100MB 1GB
Frequency?
File Transfer
Messaging
SOA
REST
week day hour minute second millisecond
K.I.S.S.
Image by Andreas
Focus on what’s important
Core
Context
Generic
Defines your company, makes money
Never “done”, always being refined
Direct...
Focus on what’s important
Core
Context
Generic
See Subdomains from Domain-Driven Design for more…
If applied to one system...
Focus on what’s important
Core
Context
Generic
Best of
Breed
Just
Do It
Example: Successfactors Employee Master Data
SAP ECC Successfactors
Environmental Factors
SaaS app managed by HR
➧ Design for Observability
Non-technical support
Used for Performance & Goals...
Looks like ‘Generic’ Subdomain…
Core
Context
Generic
Best of
Breed
Just
Do It
• SAP-delivered ABAP code for ERP data extraction
• Creates a CSV file
• File must be uploaded to SFTP server
Option 1: SA...
• SAP-delivered ABAP code for ERP data extraction
• Creates XML messages for sending via SOAP
• Requires middleware to inv...
Different Data Models
OU
Position
Person
1
1
n
n
Person
OU
Manager
Position
1
1
1
SAP HCM Org Structure Successfactors Rep...
• Fit for purpose
• Great usability and observability
• Outsource testing to HR
• Empower HR to service support calls with...
Our Solution
SAP ECC Successfactors
SAP Code
Z Code CSV file CSV file
Process Orchestration
✉️
Post-processing: remove
con...
Do: Files as Messages
SAP Business SuiteProcess Orchestration
ABAP
Proxy
Load
Program
store
invoke
read
CSV
file
CSV
file
...
• SAP PO’s adapters for FTP(S) and SFTP, large file support
• Guaranteed Delivery
• Failure detection and alerting
• Retry...
• FTP Scripts
– Difficult to inventory
– Difficult to monitor
– Password management
• Filesystem mounts everywhere
– The “...
Do: Atomic File Handling
~file.tmp
open
write
write
close
…
file.csvrename
file.csv
check size
check size
read contents
mo...
• Trigger files
• Start/End files
• Trailers and checksums
Don’t: Work-arounds
• Commas or tabs as separators
• Meaningful headers
• 1 line == 1 record
Do: Simple Data Structures
• Different Record types
• Fixed-width files
• Application Logic
– “01” means insert, “02” delete, etc.
• Trailers, Checks...
Image by idreamlikecrazy
Image by Alexandre Dulaunoy
Image by Adam Cohn
</presentation>
E: sascha@BlueT.com.au
M: +61 403 933 472
Me: http://about.sufw.me
@sufw
The CSV File Strikes Back
The CSV File Strikes Back
The CSV File Strikes Back
You’ve finished this document.
Download and read it offline.
Upcoming SlideShare
Xml
Next
Upcoming SlideShare
Xml
Next
Download to read offline and view in fullscreen.

Share

The CSV File Strikes Back

Download to read offline

Yes, CSV files are a blast from the past but sometimes they are still the best tool for the job. However there are some definite do's and dont's here when you need to, or want to, use file based interfaces.

Presented at Mastering SAP Technologies 2016

Related Books

Free with a 30 day trial from Scribd

See all

Related Audiobooks

Free with a 30 day trial from Scribd

See all
  • Be the first to like this

The CSV File Strikes Back

  1. 1. Sascha Wenninger Integration Architect, Blue T The CSV File Strikes Back
  2. 2. • Technology Architect • Integration Focus • Co-Founder, Blue T • @sufw About Me
  3. 3. • RESTful APIs • Modern Message Queues • Microservices • Event Stream Processing • Polyglot Programming Integration Trends
  4. 4. Diversity Image by Alexandre Dulaunoy
  5. 5. Use the Right Tool for the Job! Image by Bruce Murray
  6. 6. Interaction? A B B A A pushes to B B fetches from A
  7. 7. Data Volume? File Transfer Messaging SOA REST 100B 1kB 10kB 100kB 1MB 10MB 100MB 1GB
  8. 8. Frequency? File Transfer Messaging SOA REST week day hour minute second millisecond
  9. 9. K.I.S.S. Image by Andreas
  10. 10. Focus on what’s important Core Context Generic Defines your company, makes money Never “done”, always being refined Directly supports the Core Tends to be bespoke to some extent Necessary to stay in business. Reduce costs, drive to commodity See Subdomains from Domain-Driven Design for more…
  11. 11. Focus on what’s important Core Context Generic See Subdomains from Domain-Driven Design for more… If applied to one system: 5%of the code base 20%of value 80%of the effort
  12. 12. Focus on what’s important Core Context Generic Best of Breed Just Do It
  13. 13. Example: Successfactors Employee Master Data SAP ECC Successfactors
  14. 14. Environmental Factors SaaS app managed by HR ➧ Design for Observability Non-technical support Used for Performance & Goals ➧ Inconsistent, non-constant usage Tolerates latency In 2012, expected other modules to be added “later” ➧ YAGNI Don’t predict the future
  15. 15. Looks like ‘Generic’ Subdomain… Core Context Generic Best of Breed Just Do It
  16. 16. • SAP-delivered ABAP code for ERP data extraction • Creates a CSV file • File must be uploaded to SFTP server Option 1: SAP-Standard File-Based Integration Pros Cons • Unfashionable • SFSF import job run exactly once per day • Easy to inspect • Easy to post-process • Efficient • Full data set every day • Undemanding on middleware
  17. 17. • SAP-delivered ABAP code for ERP data extraction • Creates XML messages for sending via SOAP • Requires middleware to invoke SFSF APIs Option 2: SAP-Standard Web Services Integration Pros Cons • More complex delta processing • Requires IT support to manage sequence • Requires XML display/editing tools • Not easy to inspect in Excel • APIs! Yay! • Faster updates due to delta processing
  18. 18. Different Data Models OU Position Person 1 1 n n Person OU Manager Position 1 1 1 SAP HCM Org Structure Successfactors Reporting Relationships 🤔
  19. 19. • Fit for purpose • Great usability and observability • Outsource testing to HR • Empower HR to service support calls without IT help • One file in Excel = whole data set CSV Files FTW!
  20. 20. Our Solution SAP ECC Successfactors SAP Code Z Code CSV file CSV file Process Orchestration ✉️ Post-processing: remove contractors unless they manage permanent staff Send to SF support inbox for inspection if required Load via standard import job Guaranteed Delivery via SFTP
  21. 21. Do: Files as Messages SAP Business SuiteProcess Orchestration ABAP Proxy Load Program store invoke read CSV file CSV file XI msg CSV file
  22. 22. • SAP PO’s adapters for FTP(S) and SFTP, large file support • Guaranteed Delivery • Failure detection and alerting • Retry handling • Many standard-SAP import interface still require files… Do: Files as Messages
  23. 23. • FTP Scripts – Difficult to inventory – Difficult to monitor – Password management • Filesystem mounts everywhere – The “shared database” of integration Don’t: Muddling through
  24. 24. Do: Atomic File Handling ~file.tmp open write write close … file.csvrename file.csv check size check size read contents move to archive … Writing Reading
  25. 25. • Trigger files • Start/End files • Trailers and checksums Don’t: Work-arounds
  26. 26. • Commas or tabs as separators • Meaningful headers • 1 line == 1 record Do: Simple Data Structures
  27. 27. • Different Record types • Fixed-width files • Application Logic – “01” means insert, “02” delete, etc. • Trailers, Checksums Don’t: Anything else
  28. 28. Image by idreamlikecrazy
  29. 29. Image by Alexandre Dulaunoy
  30. 30. Image by Adam Cohn
  31. 31. </presentation>
  32. 32. E: sascha@BlueT.com.au M: +61 403 933 472 Me: http://about.sufw.me @sufw

Yes, CSV files are a blast from the past but sometimes they are still the best tool for the job. However there are some definite do's and dont's here when you need to, or want to, use file based interfaces. Presented at Mastering SAP Technologies 2016

Views

Total views

969

On Slideshare

0

From embeds

0

Number of embeds

15

Actions

Downloads

9

Shares

0

Comments

0

Likes

0

×