The document outlines the presentation on data integration techniques focusing on AIXM datasets, specifically delta calculations and comparison tools. It discusses the challenges of detecting changes efficiently between datasets, and introduces FME/Python based solutions for generating user-friendly reports and visualizations. The conclusion emphasizes the benefits of using FME for versatile data processing and automation.

1. AIXM – from calculating delta to
ensuring data compliance

2. Presenters
The
Peak
of
Data
Integration
20
23
Żaneta
Markiewicz
System&DataAnalyst
Boeing
Jakub
Olech
System&DataAnalyst
Boeing

3. The
Peak
of
Data
Integration
20
23
Agenda
1. Aeronauticalobstacledata (AIXM 5.1)
2. Delta calculation for two AIXM datasets
3. Comparisontoolfor DB tables

4. The
Peak
of
Data
Integration
20
23
AIXM delta

5. The
Peak
of
Data
Integration
20
23
AIXM VerticalStructure

6. The
Peak
of
Data
Integration
20
23
"Ever grappled with spotting
changes in AIXM datasets?
Join us for a deep dive."

7. The
Peak
of
Data
Integration
20
23
Why could we need to compare AIXM datasets?
• Completedata files each cycle
• Detectingadded,changedand decommissionedrecords
• No need to load all data again
• Less data impact
• Better performance

8. The
Peak
of
Data
Integration
20
23
Challenge: efficiently catch changes
betweentwo AIXM datasets
• No automated solutions for such comparison
• Data structure complexity
• Moderate comparison sensitivity
• Performance

9. The
Peak
of
Data
Integration
20
23
AIXM delta: FME/Python based solution
• Reads AIXM datasets
• Compareskey attributes& extensions
• Producesuser-friendlydifferences list
• Writes delta-onlyAIXM file
• GeneratesSHP for delta visualization

10. The
Peak
of
Data
Integration
20
23
AIXM delta: Reading
• XMLFragmenter
transformer
• Both point and line
geometry support

11. The
Peak
of
Data
Integration
20
23
AIXM delta: Comparing
• PythonCaller
• Based on UUID
• Ignoringirrelevant AIXM
properties

12. The
Peak
of
Data
Integration
20
23
AIXM delta: Presenting
• Logs all changes in FME log
• Loadsdifferences into DB
• Presentsoveralldelta statistics

13. The
Peak
of
Data
Integration
20
23
AIXM delta: Writing
• Filtering out UNCHANGED
• SavingADDED, REMOVED and CHANGED

14. The
Peak
of
Data
Integration
20
23
AIXM delta: Visualization

15. The
Peak
of
Data
Integration
20
23
Comparisontool

16. The
Peak
of
Data
Integration
20
23
„The deep dive gets deeper.
Can cross-tablecomparisonbe
done efficiently and clearly?”

17. The
Peak
of
Data
Integration
20
23
Why report in Excel?
• Visualizing data discrepancies in macro and micro scale
• Auditing and data compliance
• Customizable analysis approach

18. The
Peak
of
Data
Integration
20
23
Challenge is to effectively compare millionsof records
and then save the result in Excel
• Performance
• Excel limitations
• Overall statistics
• Customizablecomparisonparameters

19. The
Peak
of
Data
Integration
20
23
Comparisontool: FME/Python based solution
• Reads data in smaller batches
• Comparescustomerdefined attributes
• Generatesreport per batchand for all batches
1 *
P A R E N T C H I L D

20. The
Peak
of
Data
Integration
20
23
Comparisontool: Reading
• Calculates numberof
batches
• Reads data in chunks
• Triggers child jobs

21. The
Peak
of
Data
Integration
20
23
Comparisontool: Comparing
• Merges same records by UUID
• Comparescustomerdefined
attributes
• Gathersstatistics
• Defines cell formatting

22. The
Peak
of
Data
Integration
20
23
Comparisontool:
Writing
• Zipped report
• Overall statistics
with errors per
batch
• Differences per
record

23. The
Peak
of
Data
Integration
20
23
Conclusion

24. The
Peak
of
Data
Integration
20
23
Why use FME?
• Versatile data processing
• Automation and reporting
• Scalability and Efficiency

25. The
Peak
of
Data
Integration
20
23
Resources
• https://aixm.aero/sites/aixm.aero/files/imce/AIXM51HT
ML/index.html

26. Thank You!
jakub.olech@boeing.com
zaneta.markiewicz@boeing.com