Properly automating your data pipelines, in a robust, scalable way, can eliminate these risks and save a significant amount of time. See how data integration tools like CloverDX can help you: Save time writing data manipulations scripts by switching to visual representation of data flows Handle a growing complexity of data transformation and movement scenarios with integrated jobflow management and business process monitoring Handle potentially hundreds of data feeds in a manageable manner by easily adopting templates and pre-made components

1. Automating Data
Pipelines: Moving
away from Scripts
and Excel
Kevin Scott
Director of Sales Engineering

2. Homegrown ETL solutions are common
Excel Excel, Python, SQL *-SQL, Java, C#
Manual Process Scripts Custom Applications

3. Naive assessment of the task
o “This is simple, we just need to…”
Urgency
o tight project deadline, no time for research/selection of third-party tools
Exceptional Requirements
o too challenging for a commercial off-the-shelf solution
Exceptional Team
o you have a highly skilled and available dev team eager to DIY
Historical Precedent
o you’ve always done it this way
Motivation for choosing homegrown solutions

4. Feature Gaps
o new end points, new DQ issues
Lack of transparency
o Logging, alerting, auditing, error reporting
Age
o Needs age-related overhaul, or has accumulated cruft
Maintenance Costs
o dev team has moved on (or you need the dev to move on…)
o maintenance costs ripple beyond that actual maintenance task – what else
could team be working on?
Scaling Issues
o can’t keep up with increased demand
Risks of choosing homegrown solutions

5. Designed in-house to solve specific in-house data problems
Use some combination of
o Manual processes
o Desktop tools
o Scripts
o Libraries
o Programs
o Data storage
o Operating System Services
Homegrown ETL Solutions

6. Using a Modern Data
Integration Platform to
properly automate your
data pipelines, in a robust,
scalable way, can eliminate
these risks and save a
significant amount of time.

7. In cloud — On premise — Hybrid
CloverDX Data Integration Platform
Automation of data
workloads from A to Z
One place for solving the
mundane and the complex
Productivity and trust
for the enterprise
Data self-service for everyone

8. CloverDX Data Integration Platform helps with..
Replacing legacy/home-grown tooling
Data ingestion/onboarding
Operational data and application integration
Data migration
Data quality
Data for BI and reporting

9. CloverDX High-level Architecture

10. Case Study
Ingesting data from many sources for analysis

11. Fintech Vertical
Business provides analysis services to credit unions
Accept input files from many client institutions
o Variable format
o Variable quality
Transform into standard format
Assess quality
Load into a warehouse for subsequent analysis
Case Study Scenario

12. As a manual process?

14. As a scripted process?

16. Using the CloverDX
Data Integration Platform…

17. Steps include:
o Detecting arrival of client files to be ingested
o Detecting format and layout of client files
o Reading client files
o Transforming/Mapping
o Assessing quality
o Loading to target
o Detecting/Logging at every step
End-to-end oversight of the ingest process

18. Steps include:
o Detecting arrival of client files to be ingested
o Detecting format and layout of client files
o Reading client files
o Transforming/Mapping
o Assessing quality
o Loading to target
o Detecting/Logging at every step
End-to-end oversight of the ingest process
Detect data
available for ingest
Match with
client-specific
processing rules
Read
Transform
Map
Validate
Load to warehouse
Update
ingestion log

19. Orchestrating the ingest process

20. Orchestrating the ingest process

21. Orchestrating the ingest process

22. Ingest process details
Read, validate,
transform,
write, log error

23. Run ingest jobs automatically, unattended
o Schedule jobs that look for files to onboard
o Listen for arrival of files to onboard
o Launch the onboarding process on-demand
Record all ingest activity
o Alerts when jobs fail
o Logs of every execution
o Graphical inspection of any run
CloverDX automates the ingest process

24. Run ingest jobs automatically and unattended

25. (Re)run ingest jobs on demand

26. Continually monitor ingest jobs

27. Visually inspect ingest job failures

28. Eliminate risks of using homegrown Scripts and Excel
Visually design your data jobs
Automate Execution
Instill confidence in operations
Save a significant amount of time
Use a Modern Data Integration Platform

29. More on automated data ingestion with CloverDX:
www.cloverdx.com/solutions/data-ingest
Request a CloverDX demo:
www.cloverdx.com/demo
Q&A
www.cloverdx.com/webinars