In the modern IT life whenever systems are upgraded, replaced, or consolidated the first and most important thing that comes into picture is the “Existing Data” in the system. Before the thought of moving to a new and advanced system for better efficiency in work, can flourish fully one has to take necessary measures to maintain the data as it and have it up and running in the new system as well. This is where Data-Migration Techniques play a vital role

1. Automation in Data Migration and Data Validation
Inthe modernITlife wheneversystemsare upgraded,replaced,orconsolidatedthefirstandmost
importantthingthatcomesintopicture isthe “ExistingData”inthe system.Beforethe thoughtof moving
to a new and advancedsystemfor betterefficiencyinwork,can flourishfullyone hasto take necessary
measuresto maintainthe data as it and have it up and runningin the new systemas well.Thisiswhere
Data-Migration Techniques play a vital role. At first, the word Data-Migration might mean very less to
mostof you…“all thatone needstodoismaintainabackupof the dataand restore itinthe new system!!”
Well unfortunatelyit's not as simple as this. Data Migration entails more challenges,to name a few are
high volume data, complex mappings and transformations, sometimes it also demands real time
synchronization between the old and the new systems. There are many tools introduced in the market
that ensure accurate data transfers.Talend Open Studio is one of the mostcost effective tool available.
Talend is fast and has the ability to support a wide range of business application software that include
Accounting,ERP,Ecommerce,CRMmoreoverTalendhasabilitytoconnect to BigData infrastructuresas
well. Talend provides rich functionality that dramatically speeds project implementation and boosts
developer productivity.
Inthe past,undertakingreal time datamigrationprojectsinvolvedintensivelabor,manualcoding,
which could be extremely time-consuming and expensive. Consequently, even after realizing that real
time datasynchronizationwasthe needof the hour,manyorganizationshadtostill keepitontheir"wish
list". We at Bista Solutions have explored the use of Talendand achieved data migration, be it one time
migrationorbe ita real time migrationwithverylessoralmostnohumaninterferencebyautomatingthe
processes.
Major challenges in a data migration projects is being able to validate thousands of tables
containinghuge numberof recordsina definedtimeframe.Infact, Gartnerhas reportedthatmore than
50% of data migration projects will exceed the budget and end up not achieving the end result due to
flawed execution.
Amid data migration there is dependably a probability of missing data or data debasement.
Therefore, it is necessary to ensure whether the given data set has migrated completely and correctly,
consideringbothhistorical andincremental datamigration.Incremental loadingof data,specifically,can
be challengingonthe groundsthateverydatabase hasitsownparticularstructure.Thismakesitessential
to test that the fieldsand jobs are loadedeffectivelyand that files/databases are not corrupted. Having
said this manually testing billions of records is also not feasible and realistic. And so, automating the
validation processes to check accuracy in migrated data is the best solution.
To automate the validation processes we at Bista Solutions make use of Talend Open Studioto
write scripts that compare and validate the data present in both the legacy and new database. Through
this post we are sharing few of these techniquesof achieving automation in data migration and data
validation.

2. Automation Tips and Tricks using Talend Open Studio for Data Migration:
Usecase1:The firststepof automationbeginwithhavingseveral migrationscripts (talendjobs) placedin
one job designwithoutyouhavingto finddifferentjobsfordifferentpurposesinvariousfolders.Talend
OpenStudio facilitatesthisbyprovidingthe “tRunJob” component. The Talend tRunJobComponentalso
letsyouto put one TalendJob within anothersothattwo or more talendjobscan be executedone after
another in a row.