Automated data extraction and transformation on raw weather data using R Reim package and reduced 90% manual work

1. Dynamic Weather Data Scraping
Solution Brief
Requirement of a system through which we can fetch the historic temperature information on
Day level with minimum or no manual efforts.
Weather data should be contextual taking into consideration inputs like Date, City and State.
The data would be accurate and will also give Day Minimum and Day Maximum Temperature
which will be the average of Hourly Minimum and Hourly Maximum, also it handles if there is
no weather data available for certain Hours.
Solution Approach
Our solution approach comprises several distinct components:
1. Finding the State ids from master data with state name provided in the data
2. Searching for all the Station ids and matching it with the right City Name
3. Fetching the Hourly temperature and averaging it to get daily minimum and maximum
4. Dealing with few NA issues
5. Having a master sheet for the State ids and Station ids for future scalability
Future Goals:
1. API it to our Dashboard
2. Expanding the master data file to incorporate other countries (Currently it has all the
major cities of United States) – this will also require UOM handling
3. Should fetch if the State Abbreviations are provided rather just full name
4. Much more validations need to be in place
5. Testing with false or abnormal inputs
6. Store master file on cloud so that it becomes platform independent
7. Search by only city if there is no state name mentioned (Issue with this is some city
names are in multiple states)
8. Code Optimization as the current system will slower the dashboard
9. Including Wind Speed and Humidity data along with temperature

2. Sample Master Data:
Sample Input Data:

3. Sample Output:
Snapshot of R Environment: