SlideShare a Scribd company logo
CleanAir: Air Quality Dashboard
Puspal Hore
Insight Data Engineering Fellow
Air Pollution
• Air Pollution is a Health Hazard
• Many Types of Pollutants
–Ozone
–Particulate matter (pm2.5, pm10)
–NO2
–SO2
–CO
• Sensitive Groups: Children, Elderly, Patients
with Lung and Heart Disease etc.
Data Source: EPA
• EPA tracks air quality
• Monitoring stations across the country
• Hourly, daily data feed available as flat
files with delay (only data up to 2015 is
available at present)
AQI
• Air Quality Index
• Composite Index based on
–Ozone, pm2.5, pm10, NO2, SO2, CO
Data
• Location Data : State, County, Site
• Time Data : Date, Hour, Minute
• Value : AQI, individual pollutant concentration
Data Volume
• Daily data : 52 MB/ year for each pollutant
• Hourly data : 1.3 – 3 GB / year / pollutant
• 2 – 6 million rows
Data Pipeline
Simulated Air
Pollution Data
Demo: https://youtu.be/6ILrpyf8zPQ
Challenges: Forecasting
• Poor support in python compared to R
• SparkR vs Python / rpy2
• Python / Statsmodels.tsa
• Cloudera Spark-TS
About Me
• Electrical Engineer, IIT, India
• DBA, Systems Engineer
• MD, Rutgers NJMS
• Books, Travel, Photography
Influxdb
• Time Series DB with clustering
• CRud
• SQL like query language, HTTP, JSON
• Continuous Queries and Downsampling
• Built-in data retention policy
Versions…
• Kafka v0.8.2.2 with Scala 2.10
• Zookeeper v3.4.6
• Spark v1.5.2 with Hadoop v2.4+
• Hadoop v2.7.1
• Influxdb 0.10.0
• Grafana 2.6.0

More Related Content

Similar to Air Quality Dashboard

2013-04-30 EE DSS Approach and Demo
2013-04-30 EE DSS Approach and Demo2013-04-30 EE DSS Approach and Demo
2013-04-30 EE DSS Approach and Demo
Rudolf Husar
 
AQPF Slides July 2022.pptx
AQPF Slides July 2022.pptxAQPF Slides July 2022.pptx
AQPF Slides July 2022.pptx
CapitolTechU
 
Air Pollution Monitoring in India using Aeroqual Compact Air Quality Monitori...
Air Pollution Monitoring in India using Aeroqual Compact Air Quality Monitori...Air Pollution Monitoring in India using Aeroqual Compact Air Quality Monitori...
Air Pollution Monitoring in India using Aeroqual Compact Air Quality Monitori...
Carl Beck
 
CaBA Lancaster 01 June Jones Environment Agency
CaBA Lancaster 01 June Jones Environment AgencyCaBA Lancaster 01 June Jones Environment Agency
CaBA Lancaster 01 June Jones Environment Agency
CaBASupport
 
0411 Spec Nat Assess
0411 Spec Nat Assess0411 Spec Nat Assess
0411 Spec Nat Assess
Rudolf Husar
 
2004-11-24 Assessment of the Speciated PM Network (Initial Draft, November 20...
2004-11-24 Assessment of the Speciated PM Network (Initial Draft, November 20...2004-11-24 Assessment of the Speciated PM Network (Initial Draft, November 20...
2004-11-24 Assessment of the Speciated PM Network (Initial Draft, November 20...
Rudolf Husar
 
0411 Spec Nat Assess Tmp
0411 Spec Nat Assess Tmp0411 Spec Nat Assess Tmp
0411 Spec Nat Assess Tmp
Rudolf Husar
 
2005-01-28 Assessment of the Speciated PM Network (Initial Draft, November 2004)
2005-01-28 Assessment of the Speciated PM Network (Initial Draft, November 2004)2005-01-28 Assessment of the Speciated PM Network (Initial Draft, November 2004)
2005-01-28 Assessment of the Speciated PM Network (Initial Draft, November 2004)
Rudolf Husar
 
Citizen Sensing Efforts November 2011
Citizen Sensing Efforts   November 2011Citizen Sensing Efforts   November 2011
Citizen Sensing Efforts November 2011
timsdye
 
iMicrobe_ASLO_2015
iMicrobe_ASLO_2015iMicrobe_ASLO_2015
iMicrobe_ASLO_2015
Bonnie Hurwitz
 
Baseline study for EIA
Baseline study for EIABaseline study for EIA
Baseline study for EIA
Jenson Samraj
 
Manure management and temperature impacts on gas concentrations in monoslope ...
Manure management and temperature impacts on gas concentrations in monoslope ...Manure management and temperature impacts on gas concentrations in monoslope ...
Manure management and temperature impacts on gas concentrations in monoslope ...
LPE Learning Center
 
Save the Hills 3-2015
Save the Hills 3-2015Save the Hills 3-2015
Save the Hills 3-2015
Pang Houa Xiong- Yang
 
ICOS Services and Products
ICOS Services and Products ICOS Services and Products
ICOS Services and Products
Integrated Carbon Observation System (ICOS)
 
2014 LOOP Verification - Members' Briefing
2014 LOOP Verification - Members' Briefing2014 LOOP Verification - Members' Briefing
2014 LOOP Verification - Members' Briefing
loop_wwf
 
City Ambient Air Quality Monitoring
City Ambient Air Quality MonitoringCity Ambient Air Quality Monitoring
City Ambient Air Quality Monitoring
Ekonnect Knowledge Foundation
 
Air quality index by usama younas
Air quality index by usama younasAir quality index by usama younas
Air quality index by usama younas
USAMAYOUNAS11
 
04 radon
04 radon04 radon
04 radon
Chris Lutes
 
C1.06: US IOOS: Eyes on the Ocean - Zdenka Willis
C1.06: US IOOS: Eyes on the Ocean - Zdenka WillisC1.06: US IOOS: Eyes on the Ocean - Zdenka Willis
C1.06: US IOOS: Eyes on the Ocean - Zdenka Willis
Blue Planet Symposium
 
Ambient Air Quality Monitoring in Southern Alberta
Ambient Air Quality Monitoring in Southern AlbertaAmbient Air Quality Monitoring in Southern Alberta
Ambient Air Quality Monitoring in Southern Alberta
Kepa2014
 

Similar to Air Quality Dashboard (20)

2013-04-30 EE DSS Approach and Demo
2013-04-30 EE DSS Approach and Demo2013-04-30 EE DSS Approach and Demo
2013-04-30 EE DSS Approach and Demo
 
AQPF Slides July 2022.pptx
AQPF Slides July 2022.pptxAQPF Slides July 2022.pptx
AQPF Slides July 2022.pptx
 
Air Pollution Monitoring in India using Aeroqual Compact Air Quality Monitori...
Air Pollution Monitoring in India using Aeroqual Compact Air Quality Monitori...Air Pollution Monitoring in India using Aeroqual Compact Air Quality Monitori...
Air Pollution Monitoring in India using Aeroqual Compact Air Quality Monitori...
 
CaBA Lancaster 01 June Jones Environment Agency
CaBA Lancaster 01 June Jones Environment AgencyCaBA Lancaster 01 June Jones Environment Agency
CaBA Lancaster 01 June Jones Environment Agency
 
0411 Spec Nat Assess
0411 Spec Nat Assess0411 Spec Nat Assess
0411 Spec Nat Assess
 
2004-11-24 Assessment of the Speciated PM Network (Initial Draft, November 20...
2004-11-24 Assessment of the Speciated PM Network (Initial Draft, November 20...2004-11-24 Assessment of the Speciated PM Network (Initial Draft, November 20...
2004-11-24 Assessment of the Speciated PM Network (Initial Draft, November 20...
 
0411 Spec Nat Assess Tmp
0411 Spec Nat Assess Tmp0411 Spec Nat Assess Tmp
0411 Spec Nat Assess Tmp
 
2005-01-28 Assessment of the Speciated PM Network (Initial Draft, November 2004)
2005-01-28 Assessment of the Speciated PM Network (Initial Draft, November 2004)2005-01-28 Assessment of the Speciated PM Network (Initial Draft, November 2004)
2005-01-28 Assessment of the Speciated PM Network (Initial Draft, November 2004)
 
Citizen Sensing Efforts November 2011
Citizen Sensing Efforts   November 2011Citizen Sensing Efforts   November 2011
Citizen Sensing Efforts November 2011
 
iMicrobe_ASLO_2015
iMicrobe_ASLO_2015iMicrobe_ASLO_2015
iMicrobe_ASLO_2015
 
Baseline study for EIA
Baseline study for EIABaseline study for EIA
Baseline study for EIA
 
Manure management and temperature impacts on gas concentrations in monoslope ...
Manure management and temperature impacts on gas concentrations in monoslope ...Manure management and temperature impacts on gas concentrations in monoslope ...
Manure management and temperature impacts on gas concentrations in monoslope ...
 
Save the Hills 3-2015
Save the Hills 3-2015Save the Hills 3-2015
Save the Hills 3-2015
 
ICOS Services and Products
ICOS Services and Products ICOS Services and Products
ICOS Services and Products
 
2014 LOOP Verification - Members' Briefing
2014 LOOP Verification - Members' Briefing2014 LOOP Verification - Members' Briefing
2014 LOOP Verification - Members' Briefing
 
City Ambient Air Quality Monitoring
City Ambient Air Quality MonitoringCity Ambient Air Quality Monitoring
City Ambient Air Quality Monitoring
 
Air quality index by usama younas
Air quality index by usama younasAir quality index by usama younas
Air quality index by usama younas
 
04 radon
04 radon04 radon
04 radon
 
C1.06: US IOOS: Eyes on the Ocean - Zdenka Willis
C1.06: US IOOS: Eyes on the Ocean - Zdenka WillisC1.06: US IOOS: Eyes on the Ocean - Zdenka Willis
C1.06: US IOOS: Eyes on the Ocean - Zdenka Willis
 
Ambient Air Quality Monitoring in Southern Alberta
Ambient Air Quality Monitoring in Southern AlbertaAmbient Air Quality Monitoring in Southern Alberta
Ambient Air Quality Monitoring in Southern Alberta
 

Recently uploaded

Champions of Health Spotlight On Leaders Shaping Germany's Healthcare.pdf
Champions of Health Spotlight On Leaders Shaping Germany's Healthcare.pdfChampions of Health Spotlight On Leaders Shaping Germany's Healthcare.pdf
Champions of Health Spotlight On Leaders Shaping Germany's Healthcare.pdf
eurohealthleaders
 
Hypotension and role of physiotherapy in it
Hypotension and role of physiotherapy in itHypotension and role of physiotherapy in it
Hypotension and role of physiotherapy in it
Vishal kr Thakur
 
MBC Support Group for Black Women – Insights in Genetic Testing.pdf
MBC Support Group for Black Women – Insights in Genetic Testing.pdfMBC Support Group for Black Women – Insights in Genetic Testing.pdf
MBC Support Group for Black Women – Insights in Genetic Testing.pdf
bkling
 
Hypertension and it's role of physiotherapy in it.
Hypertension and it's role of physiotherapy in it.Hypertension and it's role of physiotherapy in it.
Hypertension and it's role of physiotherapy in it.
Vishal kr Thakur
 
CCSN_June_06 2024_jones. Cancer Rehabpptx
CCSN_June_06 2024_jones. Cancer RehabpptxCCSN_June_06 2024_jones. Cancer Rehabpptx
CCSN_June_06 2024_jones. Cancer Rehabpptx
Canadian Cancer Survivor Network
 
PET CT beginners Guide covers some of the underrepresented topics in PET CT
PET CT  beginners Guide  covers some of the underrepresented topics  in PET CTPET CT  beginners Guide  covers some of the underrepresented topics  in PET CT
PET CT beginners Guide covers some of the underrepresented topics in PET CT
MiadAlsulami
 
Top Rated Massage Center In Ajman Chandrima Spa
Top Rated Massage Center In Ajman Chandrima SpaTop Rated Massage Center In Ajman Chandrima Spa
Top Rated Massage Center In Ajman Chandrima Spa
Chandrima Spa Ajman
 
一比一原版(EUR毕业证)鹿特丹伊拉斯姆斯大学毕业证如何办理
一比一原版(EUR毕业证)鹿特丹伊拉斯姆斯大学毕业证如何办理一比一原版(EUR毕业证)鹿特丹伊拉斯姆斯大学毕业证如何办理
一比一原版(EUR毕业证)鹿特丹伊拉斯姆斯大学毕业证如何办理
gjsma0ep
 
Letter to MREC - application to conduct study
Letter to MREC - application to conduct studyLetter to MREC - application to conduct study
Letter to MREC - application to conduct study
Azreen Aj
 
Comprehensive Rainy Season Advisory: Safety and Preparedness Tips.pdf
Comprehensive Rainy Season Advisory: Safety and Preparedness Tips.pdfComprehensive Rainy Season Advisory: Safety and Preparedness Tips.pdf
Comprehensive Rainy Season Advisory: Safety and Preparedness Tips.pdf
Dr Rachana Gujar
 
PrudentRx: A Resource for Patient Education and Engagement
PrudentRx: A Resource for Patient Education and EngagementPrudentRx: A Resource for Patient Education and Engagement
PrudentRx: A Resource for Patient Education and Engagement
PrudentRx Program
 
Let's Talk About It: Breast Cancer (What is Mindset and Does it Really Matter?)
Let's Talk About It: Breast Cancer (What is Mindset and Does it Really Matter?)Let's Talk About It: Breast Cancer (What is Mindset and Does it Really Matter?)
Let's Talk About It: Breast Cancer (What is Mindset and Does it Really Matter?)
bkling
 
HUMAN BRAIN.pptx.PRIYA BHOJWANI@GAMIL.COM
HUMAN BRAIN.pptx.PRIYA BHOJWANI@GAMIL.COMHUMAN BRAIN.pptx.PRIYA BHOJWANI@GAMIL.COM
HUMAN BRAIN.pptx.PRIYA BHOJWANI@GAMIL.COM
priyabhojwani1200
 
NEEDLE STICK INJURY - JOURNAL CLUB PRESENTATION - DR SHAMIN EABENSON
NEEDLE STICK INJURY - JOURNAL CLUB PRESENTATION - DR SHAMIN EABENSONNEEDLE STICK INJURY - JOURNAL CLUB PRESENTATION - DR SHAMIN EABENSON
NEEDLE STICK INJURY - JOURNAL CLUB PRESENTATION - DR SHAMIN EABENSON
SHAMIN EABENSON
 
The Power of Superfoods and Exercise.pdf
The Power of Superfoods and Exercise.pdfThe Power of Superfoods and Exercise.pdf
The Power of Superfoods and Exercise.pdf
Dr Rachana Gujar
 
Can coffee help me lose weight? Yes, 25,422 users in the USA use it for that ...
Can coffee help me lose weight? Yes, 25,422 users in the USA use it for that ...Can coffee help me lose weight? Yes, 25,422 users in the USA use it for that ...
Can coffee help me lose weight? Yes, 25,422 users in the USA use it for that ...
nirahealhty
 
chatgptfornlp-230314021506-2f03f614.pdf. 21506-2f03f614.pdf
chatgptfornlp-230314021506-2f03f614.pdf. 21506-2f03f614.pdfchatgptfornlp-230314021506-2f03f614.pdf. 21506-2f03f614.pdf
chatgptfornlp-230314021506-2f03f614.pdf. 21506-2f03f614.pdf
marynayjun112024
 
U Part Wigs_ A Natural Look with Minimal Effort Jokerwigs.in.pdf
U Part Wigs_ A Natural Look with Minimal Effort Jokerwigs.in.pdfU Part Wigs_ A Natural Look with Minimal Effort Jokerwigs.in.pdf
U Part Wigs_ A Natural Look with Minimal Effort Jokerwigs.in.pdf
Jokerwigs arts and craft
 
DRAFT Ventilator Rapid Reference version 2.4.pdf
DRAFT Ventilator Rapid Reference  version  2.4.pdfDRAFT Ventilator Rapid Reference  version  2.4.pdf
DRAFT Ventilator Rapid Reference version 2.4.pdf
Robert Cole
 
FACIAL NERVE
FACIAL NERVEFACIAL NERVE
FACIAL NERVE
aditigupta1117
 

Recently uploaded (20)

Champions of Health Spotlight On Leaders Shaping Germany's Healthcare.pdf
Champions of Health Spotlight On Leaders Shaping Germany's Healthcare.pdfChampions of Health Spotlight On Leaders Shaping Germany's Healthcare.pdf
Champions of Health Spotlight On Leaders Shaping Germany's Healthcare.pdf
 
Hypotension and role of physiotherapy in it
Hypotension and role of physiotherapy in itHypotension and role of physiotherapy in it
Hypotension and role of physiotherapy in it
 
MBC Support Group for Black Women – Insights in Genetic Testing.pdf
MBC Support Group for Black Women – Insights in Genetic Testing.pdfMBC Support Group for Black Women – Insights in Genetic Testing.pdf
MBC Support Group for Black Women – Insights in Genetic Testing.pdf
 
Hypertension and it's role of physiotherapy in it.
Hypertension and it's role of physiotherapy in it.Hypertension and it's role of physiotherapy in it.
Hypertension and it's role of physiotherapy in it.
 
CCSN_June_06 2024_jones. Cancer Rehabpptx
CCSN_June_06 2024_jones. Cancer RehabpptxCCSN_June_06 2024_jones. Cancer Rehabpptx
CCSN_June_06 2024_jones. Cancer Rehabpptx
 
PET CT beginners Guide covers some of the underrepresented topics in PET CT
PET CT  beginners Guide  covers some of the underrepresented topics  in PET CTPET CT  beginners Guide  covers some of the underrepresented topics  in PET CT
PET CT beginners Guide covers some of the underrepresented topics in PET CT
 
Top Rated Massage Center In Ajman Chandrima Spa
Top Rated Massage Center In Ajman Chandrima SpaTop Rated Massage Center In Ajman Chandrima Spa
Top Rated Massage Center In Ajman Chandrima Spa
 
一比一原版(EUR毕业证)鹿特丹伊拉斯姆斯大学毕业证如何办理
一比一原版(EUR毕业证)鹿特丹伊拉斯姆斯大学毕业证如何办理一比一原版(EUR毕业证)鹿特丹伊拉斯姆斯大学毕业证如何办理
一比一原版(EUR毕业证)鹿特丹伊拉斯姆斯大学毕业证如何办理
 
Letter to MREC - application to conduct study
Letter to MREC - application to conduct studyLetter to MREC - application to conduct study
Letter to MREC - application to conduct study
 
Comprehensive Rainy Season Advisory: Safety and Preparedness Tips.pdf
Comprehensive Rainy Season Advisory: Safety and Preparedness Tips.pdfComprehensive Rainy Season Advisory: Safety and Preparedness Tips.pdf
Comprehensive Rainy Season Advisory: Safety and Preparedness Tips.pdf
 
PrudentRx: A Resource for Patient Education and Engagement
PrudentRx: A Resource for Patient Education and EngagementPrudentRx: A Resource for Patient Education and Engagement
PrudentRx: A Resource for Patient Education and Engagement
 
Let's Talk About It: Breast Cancer (What is Mindset and Does it Really Matter?)
Let's Talk About It: Breast Cancer (What is Mindset and Does it Really Matter?)Let's Talk About It: Breast Cancer (What is Mindset and Does it Really Matter?)
Let's Talk About It: Breast Cancer (What is Mindset and Does it Really Matter?)
 
HUMAN BRAIN.pptx.PRIYA BHOJWANI@GAMIL.COM
HUMAN BRAIN.pptx.PRIYA BHOJWANI@GAMIL.COMHUMAN BRAIN.pptx.PRIYA BHOJWANI@GAMIL.COM
HUMAN BRAIN.pptx.PRIYA BHOJWANI@GAMIL.COM
 
NEEDLE STICK INJURY - JOURNAL CLUB PRESENTATION - DR SHAMIN EABENSON
NEEDLE STICK INJURY - JOURNAL CLUB PRESENTATION - DR SHAMIN EABENSONNEEDLE STICK INJURY - JOURNAL CLUB PRESENTATION - DR SHAMIN EABENSON
NEEDLE STICK INJURY - JOURNAL CLUB PRESENTATION - DR SHAMIN EABENSON
 
The Power of Superfoods and Exercise.pdf
The Power of Superfoods and Exercise.pdfThe Power of Superfoods and Exercise.pdf
The Power of Superfoods and Exercise.pdf
 
Can coffee help me lose weight? Yes, 25,422 users in the USA use it for that ...
Can coffee help me lose weight? Yes, 25,422 users in the USA use it for that ...Can coffee help me lose weight? Yes, 25,422 users in the USA use it for that ...
Can coffee help me lose weight? Yes, 25,422 users in the USA use it for that ...
 
chatgptfornlp-230314021506-2f03f614.pdf. 21506-2f03f614.pdf
chatgptfornlp-230314021506-2f03f614.pdf. 21506-2f03f614.pdfchatgptfornlp-230314021506-2f03f614.pdf. 21506-2f03f614.pdf
chatgptfornlp-230314021506-2f03f614.pdf. 21506-2f03f614.pdf
 
U Part Wigs_ A Natural Look with Minimal Effort Jokerwigs.in.pdf
U Part Wigs_ A Natural Look with Minimal Effort Jokerwigs.in.pdfU Part Wigs_ A Natural Look with Minimal Effort Jokerwigs.in.pdf
U Part Wigs_ A Natural Look with Minimal Effort Jokerwigs.in.pdf
 
DRAFT Ventilator Rapid Reference version 2.4.pdf
DRAFT Ventilator Rapid Reference  version  2.4.pdfDRAFT Ventilator Rapid Reference  version  2.4.pdf
DRAFT Ventilator Rapid Reference version 2.4.pdf
 
FACIAL NERVE
FACIAL NERVEFACIAL NERVE
FACIAL NERVE
 

Air Quality Dashboard

  • 1. CleanAir: Air Quality Dashboard Puspal Hore Insight Data Engineering Fellow
  • 2. Air Pollution • Air Pollution is a Health Hazard • Many Types of Pollutants –Ozone –Particulate matter (pm2.5, pm10) –NO2 –SO2 –CO • Sensitive Groups: Children, Elderly, Patients with Lung and Heart Disease etc.
  • 3. Data Source: EPA • EPA tracks air quality • Monitoring stations across the country • Hourly, daily data feed available as flat files with delay (only data up to 2015 is available at present)
  • 4. AQI • Air Quality Index • Composite Index based on –Ozone, pm2.5, pm10, NO2, SO2, CO
  • 5. Data • Location Data : State, County, Site • Time Data : Date, Hour, Minute • Value : AQI, individual pollutant concentration
  • 6. Data Volume • Daily data : 52 MB/ year for each pollutant • Hourly data : 1.3 – 3 GB / year / pollutant • 2 – 6 million rows
  • 7. Data Pipeline Simulated Air Pollution Data Demo: https://youtu.be/6ILrpyf8zPQ
  • 8. Challenges: Forecasting • Poor support in python compared to R • SparkR vs Python / rpy2 • Python / Statsmodels.tsa • Cloudera Spark-TS
  • 9. About Me • Electrical Engineer, IIT, India • DBA, Systems Engineer • MD, Rutgers NJMS • Books, Travel, Photography
  • 10. Influxdb • Time Series DB with clustering • CRud • SQL like query language, HTTP, JSON • Continuous Queries and Downsampling • Built-in data retention policy
  • 11. Versions… • Kafka v0.8.2.2 with Scala 2.10 • Zookeeper v3.4.6 • Spark v1.5.2 with Hadoop v2.4+ • Hadoop v2.7.1 • Influxdb 0.10.0 • Grafana 2.6.0