Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

The CSV File Strikes Back


Published on

Yes, CSV files are a blast from the past but sometimes they are still the best tool for the job. However there are some definite do's and dont's here when you need to, or want to, use file based interfaces.

Presented at Mastering SAP Technologies 2016

Published in: Technology
  • Hello! I have searched hard to find a reliable and best research paper writing service and finally i got a good option for my needs as ⇒ ⇐
    Are you sure you want to  Yes  No
    Your message goes here
  • You have to choose carefully. offers a professional writing service. I highly recommend them. The papers are delivered on time and customers are their first priority. This is their website:
    Are you sure you want to  Yes  No
    Your message goes here
  • who will win this game? get free picks and predictions. ▲▲▲
    Are you sure you want to  Yes  No
    Your message goes here
  • Be the first to like this

The CSV File Strikes Back

  1. 1. Sascha Wenninger Integration Architect, Blue T The CSV File Strikes Back
  2. 2. • Technology Architect • Integration Focus • Co-Founder, Blue T • @sufw About Me
  3. 3. • RESTful APIs • Modern Message Queues • Microservices • Event Stream Processing • Polyglot Programming Integration Trends
  4. 4. Diversity Image by Alexandre Dulaunoy
  5. 5. Use the Right Tool for the Job! Image by Bruce Murray
  6. 6. Interaction? A B B A A pushes to B B fetches from A
  7. 7. Data Volume? File Transfer Messaging SOA REST 100B 1kB 10kB 100kB 1MB 10MB 100MB 1GB
  8. 8. Frequency? File Transfer Messaging SOA REST week day hour minute second millisecond
  9. 9. K.I.S.S. Image by Andreas
  10. 10. Focus on what’s important Core Context Generic Defines your company, makes money Never “done”, always being refined Directly supports the Core Tends to be bespoke to some extent Necessary to stay in business. Reduce costs, drive to commodity See Subdomains from Domain-Driven Design for more…
  11. 11. Focus on what’s important Core Context Generic See Subdomains from Domain-Driven Design for more… If applied to one system: 5%of the code base 20%of value 80%of the effort
  12. 12. Focus on what’s important Core Context Generic Best of Breed Just Do It
  13. 13. Example: Successfactors Employee Master Data SAP ECC Successfactors
  14. 14. Environmental Factors SaaS app managed by HR ➧ Design for Observability Non-technical support Used for Performance & Goals ➧ Inconsistent, non-constant usage Tolerates latency In 2012, expected other modules to be added “later” ➧ YAGNI Don’t predict the future
  15. 15. Looks like ‘Generic’ Subdomain… Core Context Generic Best of Breed Just Do It
  16. 16. • SAP-delivered ABAP code for ERP data extraction • Creates a CSV file • File must be uploaded to SFTP server Option 1: SAP-Standard File-Based Integration Pros Cons • Unfashionable • SFSF import job run exactly once per day • Easy to inspect • Easy to post-process • Efficient • Full data set every day • Undemanding on middleware
  17. 17. • SAP-delivered ABAP code for ERP data extraction • Creates XML messages for sending via SOAP • Requires middleware to invoke SFSF APIs Option 2: SAP-Standard Web Services Integration Pros Cons • More complex delta processing • Requires IT support to manage sequence • Requires XML display/editing tools • Not easy to inspect in Excel • APIs! Yay! • Faster updates due to delta processing
  18. 18. Different Data Models OU Position Person 1 1 n n Person OU Manager Position 1 1 1 SAP HCM Org Structure Successfactors Reporting Relationships 🤔
  19. 19. • Fit for purpose • Great usability and observability • Outsource testing to HR • Empower HR to service support calls without IT help • One file in Excel = whole data set CSV Files FTW!
  20. 20. Our Solution SAP ECC Successfactors SAP Code Z Code CSV file CSV file Process Orchestration ✉️ Post-processing: remove contractors unless they manage permanent staff Send to SF support inbox for inspection if required Load via standard import job Guaranteed Delivery via SFTP
  21. 21. Do: Files as Messages SAP Business SuiteProcess Orchestration ABAP Proxy Load Program store invoke read CSV file CSV file XI msg CSV file
  22. 22. • SAP PO’s adapters for FTP(S) and SFTP, large file support • Guaranteed Delivery • Failure detection and alerting • Retry handling • Many standard-SAP import interface still require files… Do: Files as Messages
  23. 23. • FTP Scripts – Difficult to inventory – Difficult to monitor – Password management • Filesystem mounts everywhere – The “shared database” of integration Don’t: Muddling through
  24. 24. Do: Atomic File Handling ~file.tmp open write write close … file.csvrename file.csv check size check size read contents move to archive … Writing Reading
  25. 25. • Trigger files • Start/End files • Trailers and checksums Don’t: Work-arounds
  26. 26. • Commas or tabs as separators • Meaningful headers • 1 line == 1 record Do: Simple Data Structures
  27. 27. • Different Record types • Fixed-width files • Application Logic – “01” means insert, “02” delete, etc. • Trailers, Checksums Don’t: Anything else
  28. 28. Image by idreamlikecrazy
  29. 29. Image by Alexandre Dulaunoy
  30. 30. Image by Adam Cohn
  31. 31. </presentation>
  32. 32. E: M: +61 403 933 472 Me: @sufw