The DOHMH (NYC Department of Mental Health and Hygiene) uses MongoDB for their internal document management system called DocSpace. This presentation outlines -the system -how they came to adopt MongoDB -migrating from a relational DB to a document-oriented one -the advantages and disadvantages we’ve encountered and how we have managed them -Next steps with MongoDB

1. Why EDP chose
Artyom Diky
William Biesty
Mark Velez

2. Agenda
• Who are we?
• Evolution of Document Management
• File system to relational DB
• Relational to document-oriented DB
• Paper to electronic
• Advantages and Challenges
• Questions?

3. Who Are We?
• New York City Department of Health and Mental Hygiene
• Environmental Health Services (EHS)
• Environmental Disease Prevention (EDP)
• Lead Poisoning Prevention Program (LPPP)
• MIS Unit we are here
• We support many programs within EDP
• Who are our stakeholders?
• Inspectors
• Researchers
• Clinical Staff
• Lawyers (FOIL)

4. Evolution of Document Management
Paper
• A lot of legal documents on paper
• Historic - from the '70s and up
• Current (ongoing)
• Problems with Paper
• Time and Labor Intensive
• Locate, Copy, Redact, Copy, Mail (Repeat….)
• Storage Space
• Disaster Recovery

5. Evolution of Document Management
eFiles
• VB6
• Scanning utilities
• File-system based storage
• Millions of files
• Identifiers based on child ID

6. Evolution of Document Management
eFiles Issues
• Technical
• VB6 phased out
• Outdated 3rd party tools changed API
• License expired
• Security
• Documents have been redacted permanently
• No access control to private information
• Scalability
• New document types
• New indexing (tagging) mechanisms for search

7. Evolution of Document Management
• Need for better document management
• Paperless offices mandate
• Expand searchable attributes and document text
• Update technology
• Improved security
• HIPAA compliance
• Platform for future applications

8. File System to Relational DB
• Challenges:
• 1M+ historical documents as image files
• Need for document metadata
• Various and evolving schemas
• Security
• Updates and migration
• Fail-safe storage

9. Technologies
• We use Microsoft technologies
• SQL Server
• .NET
• We are a small team that develop and support
dozens of data collection apps (forms)
• Risk assessments • Case Management
• Inspection Reports
• Research

10. Example Documents
event_date document_type child ID
me_num

11. File System to Relational DB
FileStream
• MSSQL 2008
o Data storage with FileStream
o Metadata with Entity-Attribute-Value
 sql_variant
o Data-driven application design
• Rich service-oriented API through WCF
• Search engine
• Added features
o Versioning
 Change and revert

12. DocSpace SQL Architecture

13. Limitations of Relational Model
• Need faster development cycle
• Double effort for development and maintenance
• On application and database level
• Document definition (metadata) first, content
later
• Changing schema
• Rigid document structure
• Not amenable to change
• No support for non-primitive values

14. Effects on Development Cycle
• SQL Waterfall-like approach
• Fully develop requirements before
implementation
• Gotta get the schema right to avoid hassle
• Change discouraged
• MongoDB Rapid Application Development
• Prototyping
• Change accommodated

15. Document Management System Done Right
• Faster development cycles
• No translation of complex document structure
into relational model
• Application driven schema
• Document content first, metadata later
• Flexible document structure driven by user
requirements
• GridFS for large documents

16. DocSpace MongoDB Architecture

17. Case Study - Traffic Fatalities
• A study of traffic-related fatalities in NYC
• Injury Surveillance and Prevention
• Offline data collection
• 330+ data points
• Multiple weekly changes to schema
o Add/remove fields
o Value types
• Developed in 500 hrs (3 months)
• 1 intermediate developer, 1 novice

18. Evolving Use of MongoDB
• Single Node with Database Security
• Nightly Dump for Backup Archiving
• Master – Slave Nodes
• Replica Sets – 3 Nodes
• Distributed across Metropolitan Area Network
• Bare Iron Primary, VMware ESX and Hyper-V VM Secondaries
• Hurricane Sandy – No downtime, one node
failed

19. Thank you!
Questions

20. Contact Us
William Biesty, Database Administrator, wbiesty@health.nyc.gov
Art Diky, Software Engineer, adiky@health.nyc.gov
Mark Velez, Software Engineer, mvalez@health.nyc.gov
nyc.gov/health