Advanced Spark Leverage : For converting Informatica mapping XML into pyspark code

1. AdSL
Advanced Spark Leverage
Pyspark job developer

2. Motivation
As part of informatica rewrite
Primary task : Convert
Challenges:
Heap of Manual work
Complex expression deduction order
Handling huge number of columns
Human errors 
Less time 

3. Current State
• Visualize the job from Informatica xml to an interactable SVG image
• Handle propagation logics between transformations and represent in the
pyspark convention
• Build the pyspark code for few static transformations like:
Expression
Look up
Router
Target Merge
and few other code specific details

4. AdSL in 3 steps
Quick intro to ADSL
Choose the required mapping from the Folder and start Leverage
1

5. AdSL in 3 steps
2
The chosen mapping is leveraged into svg image as shown

6. AdSL in 3 steps
3
Transformation logic
converted into pyspark
reference code
The data propagation logic
between transformations in
pyspark convention.

7. AdSL - techStack

8. Road Map
Target State – spyIDE
• Project Management - Architecture
• UI for code editior
• Handle Scheduling and Orchestration
• Dedicated blogs for technical issues discussion
(ADSL as Online IDE)
• Providing the UI enabled look for the existing pyspark code for easy
understanding
• Analysis of the code through UI enabling the Data lineage column level.
• Analyzing the dependency tables wise for the entire project.
• Separate page for the ideas of discussion for the technical and functional
challenegs