Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Payment Gateway Live hadoop project

  • Login to see the comments

  • Be the first to like this

Payment Gateway Live hadoop project

  1. 1. Kamal A Certified BigData, SOA andJavaArchitect
  2. 2. Agile project scope details – User story , Scrum cycles. TechnologyStack details.  SetupVirtual machine based Hadoop cluster setup. Installation of Hadoop , Hive and Sqoop. ONLINETRANING2011@GMAIL.COM
  3. 3. Analyze the payment data xmls. Parse xml data using choice of technology(DOM , JAXB etc). Load data in RDBMS tables in incremental mode.  Schedule the preprocessing job to run for every 30 min run ( Javascheduler Quartz- source 1 every 15 min, crontab - source 2 : every 1 hour ) Add multithreading / parallel process model. (To handle large volumes . ONLINETRANING2011@GMAIL.COM|
  4. 4. Build data migration flow from RDBMS into Hadoop/ Hive usingSqoop. Create Import tables in Hive. Create Sqoop - Hive data import script. Verify data import records and write error for records mismatch. ONLINETRANING2011@GMAIL.COM|
  5. 5. RunHive analytic query and store output data in result table.( Schedule the job to run)  Execute Hive joins for complex queries. Write UDF for data normalization. Use Sqoop to resend data from Hive to RDBMS through shell script . ONLINETRANING2011@GMAIL.COM
  6. 6.  Visualize output data in RDBMS table using open source/commercialtools like Tableau. Create report using Bar graph to show the trends for issue rate. . Create report using Pie chart for payment data distribution on issues. Use Hiveserver2 to connect and generate live analytic results. ONLINETRANING2011@GMAIL.COM|
  7. 7.  Email:  Skype:onlinetraining2011  Some live sessionvideos: HadoopInstallation: HDFS File system Lab:  Linkedingroup : Blog: ProjectDuration: 20 hours TrainingMedium: OnlineviaGotomeeting ONLINETRANING2011@GMAIL.COM|