1. Telco Data Pipelines in
the Cloud:
Architecture and Use
Cases
www.croz.net
Miro Miljanić
Data Architect

2. Agenda
• Introduction
• Use Cases:
• Reporting and data migration
• File processing offload
• User Feedback Anonymization
• Conclusion and Q&A

3. What do all
Cloud initiatives
have in
common?
They are all structured as a teen Rom-Com
1. Act One: Setup
- Introduction of the protagonist
- Establish the status quo
- Inciting incident
2. Act Two: Conflict
- Rising action
- Complications
- The false victory
- Emotional low point
3. Act Three: Resolution
- Climax
- Resolution
- Happily ever after
They also involve (over) optimistic assumptions…

4. Reporting and data migration

5. Reporting and
data migration
Act One
Setup
Business problem:
• We want to move the reporting and its data in the cloud:
• To have Single view of data, in Cloud (operational and analytical)
• Cloud Reimplementation of reporting – source and tool replacement
• Initial Governance setup

6. Reporting and
data migration
Act One
Setup
What does it really mean:
• (Delta) Replication of data from several on-prem DB into one
Cloud DB
• Reimplementation of reporting logic from legacy reports and
DB procedures
• Data catalogue, lineage, data access…
• Strict data security rules

7. Reporting and
data migration
Act One
Setup
On Premise
Data
Reports
Governance
Security
Business logic
reimplementation
Initial setup
Delta Data replication

8. Reporting and
data migration
Act One
Setup
Off we go
• Thorough analysis and Proofs of Concept (POC)
• Sources - Multiple DB and RDBMS, >10k objects
• Target architecture
• Delta load scenarios
• DB code reimplementation scenarios
• Reporting logic reimplementation scenarios and reporting
optimizations
• Data maintenance scenarios
• …

9. Reporting and
data migration
Act Two
Conflict
That’s all nice, but this is a bit too much for us now,
get for X days?

10. Reporting and
data migration
Act Three
Resolution
What did we do:
• Simpler scenario which was manageable in X days:
• No delta replication of data – initial DDLs and then loops and inserts
inserts from views and tables using data dictionary
• No code and report reimplementation
• Setup of all environments, systems and replication
• Security setup
• Development templates
• Cookbooks

11. Reporting and
data migration
Act One
Setup
On Premise
Data
Reports
Governance
Security
Business logic
reimplementation
Initial setup
Delta Data replication

12. File processing offload

13. File processing
offload
Act One
Setup
Business problem:
• File processing offload to Cloud
• We want to reduce our resources - DB, storage, ETL
• We want to be more flexible in terms of scaling
• We want to learn how this differs from our current process (and
have new solution as similar as possible to previous)

14. File processing
offload
Act One
Setup
What does it really mean:
• Large number of raw files
• Possible DQ and format issues – human intervention needed
• Time constraints for processing
• Performance requirements
• Detailed logging – per each file
• Preferred set of tools

15. File processing
offload
Act One
Setup
• Off we go
• POC
• Similar loading and processing logic as on prem: iterative, high
logging, multiple step processing
• Processing in ETL tool
• Detailed DQ checks per each file

16. File processing
offload
Act One
Setup
DQ checks
File preprocessing
Error Bucket Verify
and fix
File landing area
Error
Loop until ready for aggregation
Clean files
Aggregation
Export Archive

17. File processing
offload
Act Two
Conflict
The test run and billing estimate were too high!
Way too high…

18. File processing
offload
Act Three
Resolution
What did we do:
• Various load, processing and logging scenarios until we found the
found the solution
• Where to process files and how?
• What can be sequential and what can be parallel?
• How to log file processing?
• How to handle and pinpoint errors?
• Database, ETL or Code file processing?
• Database, ETL or Code data logging?
…or combination?

19. File processing
offload
Act Three
Resolution
• Preparation and DQ:
• Iterative, sequential, high logging
• Direct filesystem access
• Error pinpointing
• Orchestration
• Parallel processing of clean files

20. User Feedback Anonymization

21. User Feedback
Anonymization
Act One
Setup Business problem:
• User feedbacks contain free form entry in which users
sometime enter comments which contain personal information.
• We want to keep those comments, but we can keep this
information for only 90 days, after that we would like to
anonymize it.

22. User Feedback
Anonymization
Act One
Setup
• Hi, my name is Miro Miljanić from CROZ, I contacted your
agent yesterday about the problem on my address –
Marohnićeva 1, Zagreb. I checked today and it seems that
problem still exists. Please contact me on 099887766 or on this
e-mail - mmiljanic@croz.net. Thanks in advance.
• Hi, my name is <Person> from <Organization>, I contacted
your agent yesterday about the problem on my address –
<Location>, <Location>. I checked today and it seems that
still exists. Please contact me on <Phone Number> or on this
e-mail <e-mail>. Thanks in advance.
• Hi, my name is James Bond from Microsoft, I contacted your
agent yesterday about the problem on my address – 5th
Avenue, New York, NY. I checked today and it seems that
exists. Please contact me on 01999289029 or on this e-mail
dj.example2234@example.com. Thanks in advance.

23. User Feedback
Anonymization
Act One
Setup
What does it really mean:
• Application and company specific lingo
• Additional checks needed in some cases (human)
• E2E pipeline which can be reused for other purposes

24. User Feedback
Anonymization
Act One
Setup

25. User Feedback
Anonymization
Act Two
Conflict
This is great! Could you reimplement it in Azure 

26. User Feedback
Anonymization
Act Three
Resolution
Azure vs AWS – Human in the loop – custom integration

27. Conclusion

28. Conclusion • In more than one way, Clouds are teens, also. They are:
• Sometimes immature
• Sometimes unpredictable
• Have a logic of their own (different than adults)
(But also, with a fresh vision, and part of our future)

29. Conclusion Our first job is to explain this to our customers, and try to manage their
expectations
• How it is (not) going to solve all their problems
• How it differs from what they are used to
• Cost, Scope of Work, Way of Work, Maintenance, Performance …
And use POCs as much as possible, for all new topics for your customer.

30. sales@croz.net | www.croz.net