Loading data into S3 bucket and process it using lambda function to save the process data in RDS

1. Problem Statement: Load JSON files in AWS RDS using Lambda function.
Thisdocumentstatesthe approachfollowedandthe explanation of the thoughtprocessbehindit.
1.Data ObservationandData Processing:
The givendata has 24 folders(named00 to 23) containingdifferentnumberof .zipfileswhichcontainsafile
havingJSON formattedtweetdata.
I wrote a pythonscriptto read eachfile tomake sure the structure of JSON is similarornot.
As the structure were different,Ithoughtof making.jsonfileswithsimilarstructure sothataccordinglyit
can be used create table of similarstructure forRDS.
2.Loadingdata inRDS:
Followedstepsasfollows:
1.CreatedS3 bucketto loadthe .jsonfile.
2.Createdlambdafunction withatriggeron S3.i.e assoonas the .jsonfile getsuploadedinthe S3lambda
functionwill execute.
3.MySql database instance wascreatedinRDS withinboundaccesstoexternal resources.(MySQL
workbench)
4. ConnectedtoRDS via MySQL Workbenchusingpython.
Note:I couldn’tuploadthe whole data. Ihave useda jsonfile extractedfromabove mentionedapproach(of
one structure) anduploadedthe same.
Findsome screenshotsforyourreference.
1. S3: To load.jsonfile inthe bucket.

2. 2.Lambda: To accessdata from S3 bucketand to connectto RDS.
3.RDS: Db instance tostore jsondata.

3. 4.MySql Workbench