Stormwater Data Single View POC
•Download as ODP, PDF•
1 like•215 views
Use MongoDB and Pentaho to rapidly evaluate a use case for the City of Charlotte's Stormwater Management System by creating "A Single View of a Raindrop".
Recommended
Recommended
More Related Content
Similar to Stormwater Data Single View POC
Similar to Stormwater Data Single View POC (20)
More from Dave Callaghan
Recently uploaded
Recently uploaded (20)
Stormwater Data Single View POC
- 1. Sparks Ignite, Inc. A technology consulting firm. We build outcomes. Sparks Ignite, Inc. MongoDB Build, Adapt, Reduce, Improve
- 2. Sparks Ignite, Inc. A technology consulting firm. We build outcomes. Single View of a Raindrop StormWater Analytics A Single View of a Raindrop
- 3. Sparks Ignite, Inc. A technology consulting firm. We build outcomes. Single View of a Raindrop Sparks Ignite, Inc. ● Data Integration and Analytics ● Data Management and Governance ● Enterprise Storage and Compute ● Consulting Services Offerings Introduction
- 4. Sparks Ignite, Inc. A technology consulting firm. We build outcomes. Single View of a Raindrop Introduction
- 5. Sparks Ignite, Inc. A technology consulting firm. We build outcomes. Single View of a Raindrop Stormwater Services is responsible for ● Improving Water Quality ● Reducing Flood Risks Introduction
- 6. Sparks Ignite, Inc. A technology consulting firm. We build outcomes. Single View of a Raindrop Stormwater Data ● Rainfall Data ● Rain Gauges ● ALERT ● FINS ● Stream Data ● CMANN ● SUSI Vision
- 7. Sparks Ignite, Inc. A technology consulting firm. We build outcomes. Single View of a Raindrop Stormwater Data ● Internet of Data ● Internet of Things ● Internet of Ideas ● Internet of People Vision
- 8. Sparks Ignite, Inc. A technology consulting firm. We build outcomes. ● Resiliency ● Stability ● Adaptability of Data Model ● Performance ● Configuration Flexibility ● Administrator Functionality ● Training ● Data Migration and Movement ● Conformance with Standards ● Quality & Availability of Support VisionSingle View of a Raindrop
- 9. Sparks Ignite, Inc. A technology consulting firm. We build outcomes. Single View of a Raindrop Implement a POC Data Munging ● Rainfall Data ● Rain Gauges ● ALERT ● FINS ● Stream Data ● CMANN ● SUSI
- 10. Sparks Ignite, Inc. A technology consulting firm. We build outcomes. Single View of a Raindrop Implement a POC
- 11. Sparks Ignite, Inc. A technology consulting firm. We build outcomes. Single View of a Raindrop ● Resiliency ● Stability ● Adaptability of Data Model ● Performance ● Configuration Flexibility ● Administrator Functionality ● Training ● Data Migration and Movement ● Conformance with Standards ● Quality & Availability of Support Implement a POC
- 12. Sparks Ignite, Inc. A technology consulting firm. We build outcomes. Single View of a Raindrop ● Single View ● Internet of Things ● Mobile ● Real-Time Analytics ● Personalization ● Catalog ● Content Management Implement a POC
- 13. Sparks Ignite, Inc. A technology consulting firm. We build outcomes. Single View of a Raindrop ● Financial Services ● Government ● Retail ● High Tech ● Media and Entertainment ● Healthcare ● Telecommuncations Implement a POC
- 14. Sparks Ignite, Inc. A technology consulting firm. We build outcomes. Single View of a Raindrop ● MongoDB ● Build ● Adapt ● Reduce ● Improve Stages of Adoption
- 15. Sparks Ignite, Inc. A technology consulting firm. We build outcomes. Single View of a Raindrop Stages of Adoption
- 16. Sparks Ignite, Inc. A technology consulting firm. We build outcomes. Single View of a Raindrop Questions? david@sparksignite.net
Editor's Notes
- Since 1993, Charlotte and Mecklenburg County have had a Storm Water Services division to manage storm water runoff and eliminate sources of water pollution.
- We research, evaluate, design, build & deploy innovative information technology outcomes. DATA INTEGRATION & ANALYTICS Embedded Analytics, IoT Analytics, Native Hadoop Integration, Mobile BI, Customer 360. DATA MANAGEMENT & GOVERNANCE Modern Day Enterprise BU & DR, Compliance, Cloud & EndPoint Protection, Active Archive CONSULTING SERVICES OFFERINGS Assessments, Software Development, Design for Storage, Cloudy Apps, Data Integration & Installs
- I bring twenty years of development experience to today's modern data environments. I am certified in and work extensively with Hadoop, HBase, Cassandra and MongoDB.. I am working with and contributing to the new Big Data governance initiative platform applications like Falcon and Ranger and Atlas.
- Since 1993, Charlotte and Mecklenburg County have had a Storm Water Services division to manage storm water runoff and eliminate sources of water pollution.
- Rainfall data comes from 73 rain guages which measure rain in increments of 0.1”. Data is transferred in real-time using the ALERT system & sent in real-time to the U.S. Geological Survey and to CharMeck Storm Water Services. Data goes into DIADvisor by OneRain. Routine rainfall data is recorded in 5 min intervals. If alert level data sent to National Weather Services. The Flood Information & Notification System (FINS) alerts fire, police emergency management and MEDIC at 13 of 73 sites. CMANN Continuous Monitoring and Alert Notification Network (CMANN) has 37 monitoring sites . Stream Use-Support Index (SUSI) reported water quality in non-technical and comprehensive terms.
- Address data silos and various data types (maps come to mind here Devices Make the framework open available and easily extensible Things need to be mobile and social. Not everyone knows to call so use the twitter feed
- robust enough to be a critical component? If failures occur, how does MongoDB respond? high-quality without unexpected failures? data model changes quickly? How do changes impact the applications and services that consume it? perform enough to serve a massive app and user base? hybrid deployment with both cloud-based and on-premise ? How long does it take to install and deploy ? How difficult is it to administer?backups, adding and removing indexes, and changing out hardware What current and ongoing training ops & developers How should we migrate data.load data? Retrieve well Is MongoDB an industry de-facto standard? If not, is it well positioned to become one? Fit to our culture? if something goes wrong , can we get qualified, top-notchassistance - even in the middle of the night or on a holiday?
- Data → USGS/NC real-time rain gauge information → Rainfall Data-> Water Data For the Nation → scroll to automated retrievals → Can I get USGS water data using a web service? → http://waterservices.usgs.gov/rest/ Why no water quality data? → Information → Follow a sample link → goes nowhere → just go to http://qwwebservices.usgs.gov/ → web services The other web services link is to the rainfall data, so I am good
- Data → USGS/NC real-time rain gauge information → Rainfall Data-> Water Data For the Nation → scroll to automated retrievals → Can I get USGS water data using a web service? → http://waterservices.usgs.gov/rest/ Why no water quality data? → Information → Follow a sample link → goes nowhere → just go to http://qwwebservices.usgs.gov/ → web services The other web services link is to the rainfall data, so I am good
- can survive multiple-node and site-wide failures. The replica set feature works well and is easy to use. stable and easy to integrate one of biggest strengths over relational databases as well as or even better than the Oracle system. Amazon EC2: cloud-based, on-premise, & hybrid first configuration within hours. Chef/Puppet spin up dozens of nodes in minutes two-day training from the MongoDB University developers proficient in MongoDB with just a half day populated by exporting from Oracle using Python met criteria for future inclusion on the standards list enterprise-grade support, large community
- Single View Real-time views of your business that integrate all of your siloed data. Internet of Things 40 billion sensors. $19 trillion in revenue. You’re gonna need a bigger database. Mobile Ship killer mobile apps. Scale to millions of users. Do it in weeks, not months. Real-Time Analytics Lightweight, low-latency analytics. Integrated into your operational database. In real time. Personalization Greet your customers like old friends so they’ll treat you like one, too. Catalog Catalogs change constantly. That’s an RDBMS nightmare. But it’s easy with MongoDB. Content Management Store and serve any type of content, build any feature, serve it any way you like. From a single database.
- Financial Services MetLife single view of 100M customers across 70 systems in 90 days. 8 yrs of prior failure Citigroup realtime event subscription RMS cloud-based risk analytics w/ Tbs of data Retail Ebay stores metadata for every single item Gap built their supply chain Government Chicago geospatial realtime analytics from30+ dept Natioanal Archive stores 1000 yrs worth of docs Healthcare AstraZeneca, Genetech and Medtronic do research
- Using MongoDB as a data store sets you up for a successful ecosystem Build – The codebase, deployment and management are easily prototyped and scaled Changing the schema does not break the application. That's a new way of thinking Reduce cost at the top and bottom. Reduce overhead. Developer friction. Operations time. Project time. Refactor without rebuilding with schema on read rather than schema on write for continuous improvement
- Typically at a MUG, we have 1. Build New Applications That Were Not Possible Before 2. Adapt And Develop Competitive Advantages 3. Improve Customer Satisfaction 4. Reduce Costs For the purpose of this meeting, we will assume that everyone here has a problem that they don't know how to fix but need to fix it anyway.
- We research, evaluate, design, build & deploy innovative information technology outcomes. DATA INTEGRATION & ANALYTICS Embedded Analytics, IoT Analytics, Native Hadoop Integration, Mobile BI, Customer 360. DATA MANAGEMENT & GOVERNANCE Modern Day Enterprise BU & DR, Compliance, Cloud & EndPoint Protection, Active Archive CONSULTING SERVICES OFFERINGS Assessments, Software Development, Design for Storage, Cloudy Apps, Data Integration & Installs