Software Engineering of NLP-Based Computer Assisted Coding ...


Published on

  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Software Engineering of NLP-Based Computer Assisted Coding ...

  1. 1. Software Engineering of NLP-Based Computer Assisted Coding Applications Mark Morsch, MS; Carol Stoyla, BS,CLA; Ronald Sheffer, Jr., MA; Brian Potter, PhD A-Life Medical, Inc. – San Diego, CA
  2. 2. Presentation Overview <ul><li>Introduction and Motivations </li></ul><ul><li>Background </li></ul><ul><ul><li>What is CMM (or CMMI)? </li></ul></ul><ul><ul><li>Can CMM be applied to NLP Software Development? </li></ul></ul><ul><li>Development Process </li></ul><ul><ul><li>Key Practice Areas </li></ul></ul><ul><ul><li>Example Development Schedule </li></ul></ul><ul><li>Testing Model for NLP-based CAC Software </li></ul><ul><li>NLP and Scalability </li></ul><ul><li>Conclusion – Focus on Results </li></ul>
  3. 3. Introduction – NLP and CAC <ul><li>Natural Language Processing (NLP) software applications “read” physician notes and extract facts for coding </li></ul><ul><li>NLP for CAC requires electronic text – human transcription or speech recognition, OCR of typed text also possible </li></ul><ul><li>Primary applications today – CPT and ICD-9 coding for certain outpatient specialties, for example </li></ul><ul><ul><li>Radiology -> it’s the biggest </li></ul></ul><ul><ul><li>Emergency Medicine -> includes E/M coding </li></ul></ul><ul><ul><li>Pathology </li></ul></ul><ul><li>Emerging applications </li></ul><ul><ul><li>Inpatient coding -> work with current workflow, tools? </li></ul></ul><ul><ul><li>Quality measures -> JCAHO, CMS, etc. </li></ul></ul><ul><ul><li>Outcomes analysis </li></ul></ul>
  4. 4. Motivations <ul><li>Why do we want structured processes for NLP software development? </li></ul><ul><ul><li>Users should be confident in the results </li></ul></ul><ul><ul><li>Over time, CAC software should consistently improve </li></ul></ul><ul><ul><li>Medical coding is difficult and constantly changing, and the amount of prerequisite knowledge is massive </li></ul></ul><ul><li>What is (medical coding) or (NLP software development) like? </li></ul><ul><ul><li>“ I see mysteries and complications wherever I look, and I have never met a steadily logical person.” - Martha Gelhorn </li></ul></ul><ul><li>Structured processes bring order and logic to development </li></ul><ul><ul><li>Deliver updates on schedule </li></ul></ul><ul><ul><li>More confidence that what is promised can be delivered </li></ul></ul><ul><ul><li>Verify that the application works as intended </li></ul></ul>
  5. 5. Capability Maturity Model (CMM) <ul><li>Using a ranking system, measures the maturity of an organization’s software development processes </li></ul><ul><ul><li>Developed by the Software Engineering Institute (SEI) at Carnegie Mellon University </li></ul></ul><ul><ul><li>Defines best practices for organization involved in product development </li></ul></ul><ul><ul><li>Started in the 1980s to assess the capability of government contractors </li></ul></ul><ul><ul><li>Originally published in 1989, updates halted in 1997 in favor of CMMI </li></ul></ul><ul><li>Capability Maturity Model Integration (CMMI) is the successor to CMM </li></ul><ul><ul><li>Integrates models from various disciplines – software development, systems engineering, integrated product development and software acquisition </li></ul></ul><ul><ul><li>Better fit to iterative development methods, versus the traditional waterfall approach </li></ul></ul>
  6. 6. Software Hall of Shame (Source – Charette, Robert. Why Software Fails. IEEE Spectrum , September 2005)
  7. 7. CMMI Overview <ul><li>Level 1: Initial – Ad hoc processes, results are unpredictable and primarily driven by the skill of the team </li></ul><ul><li>Level 2: Managed – Core software development activities followed primarily at the project level </li></ul><ul><li>Level 3: Defined – Development activities are implemented and managed across multiple projects, performance improved through training, verification & validation and integrated project management </li></ul><ul><li>Level 4: Quantitatively Managed – Measures of business results such as cost, quality and timeliness utilized to improve organization performance, statistical quality control </li></ul><ul><li>Level 5: Optimized – Continuous, quantitative and proactive process improvement allowing an organization to learn, adapt and improve </li></ul>
  8. 8. Applying CMM <ul><li>NLP, like other Artificial Intelligence (AI) software, is often not developed following a software development process </li></ul><ul><ul><li>Input requirements very difficult to fully specify </li></ul></ul><ul><ul><li>Complex algorithms require special knowledge </li></ul></ul><ul><ul><li>Development is often evolutionary or experimental </li></ul></ul><ul><ul><li>Individuals in NLP development often do not have experience in software engineering </li></ul></ul><ul><li>Criticisms of CMM </li></ul><ul><ul><li>Emphasis of process over the individual </li></ul></ul><ul><ul><li>Lack of emphasis on innovation </li></ul></ul><ul><ul><li>Emphasis on activities over results </li></ul></ul>
  9. 9. Development Process <ul><li>Performance is consistent and continuously improving over time </li></ul><ul><li>Foster innovative thinking </li></ul><ul><li>Robust testing model for measuring results </li></ul><ul><li>Combine efforts of three skill areas: </li></ul><ul><ul><li>Computer science </li></ul></ul><ul><ul><li>Linguistics </li></ul></ul><ul><ul><li>Medical coding </li></ul></ul>
  10. 10. Five Practice Areas <ul><li>Requirements Management </li></ul><ul><li>Rapid Development Cycle </li></ul><ul><li>Verification and Validation </li></ul><ul><li>Complete Configuration Control </li></ul><ul><li>Formal Build and Installation Process </li></ul>
  11. 11. 1. Requirements Management <ul><li>Using a defect tracking tool, domain experts file bug reports and enhancement requests </li></ul><ul><li>Example medical documents and details of the desired output </li></ul><ul><li>Items are assigned a severity and frequency </li></ul><ul><li>Priority list is defined at the start of each update cycle </li></ul>
  12. 12. 2. Rapid Development Cycle <ul><li>Relatively short time lines between the publication date and the implementation date of coding changes </li></ul><ul><ul><li>6 to 12 weeks typically </li></ul></ul><ul><li>Weekly build and unit test cycles </li></ul><ul><ul><li>Verification of changes taking place with each unit test </li></ul></ul><ul><li>For new product development, iterations are extended to accommodate more significant development </li></ul>
  13. 13. 3. Verification and Validation <ul><li>Verification ensures the changes work correctly </li></ul><ul><ul><li>With zero or minimal regressions </li></ul></ul><ul><li>Validation ensures that the right changes have been done </li></ul><ul><li>Verification is done at both the unit and system testing levels </li></ul><ul><ul><li>Unit testing is performed at the component level and is executed by the NLP Development team </li></ul></ul><ul><ul><li>QA team may assist in the analysis of unit test results </li></ul></ul><ul><li>Independent quality assurance team, separate from the NLP development team, performs system verification and validation </li></ul>
  14. 14. 4. Complete Configuration Control <ul><li>All source code and system knowledge base files are maintained within a configuration control system </li></ul><ul><ul><li>Examples: VSS, CVS, ClearCase </li></ul></ul><ul><li>Development Lead is responsible for coordinating source code check-in and setting build checkpoints </li></ul><ul><li>All changes are recorded and documented, and past build configurations can be recovered </li></ul>
  15. 15. 5. Formal Build and Installation Process <ul><li>Installation packages are used with written installation instructions </li></ul><ul><li>Installation package records all component names and version numbers </li></ul><ul><li>Used to install releases into both the QA and production environments </li></ul><ul><li>Greatly reduces the likelihood of errors during the installation process </li></ul>
  16. 16. Example Development Schedule <ul><li>Phase 1: Requirements Analysis </li></ul><ul><ul><li>Weeks 1 -2: Bug reports and enhancement requests analyzed and prioritized </li></ul></ul><ul><li>Phase 2: Development and Unit Testing </li></ul><ul><ul><li>Weeks 3 – 9: Changes implemented, may overlap with Phase 1 if final code updates are not known </li></ul></ul><ul><li>Phase 3: System Testing </li></ul><ul><ul><li>Weeks 10 – 11: System installation package is built and delivered to the QA team </li></ul></ul><ul><li>Phase 4: Production Deployment and Documentation </li></ul><ul><ul><li>Week 12: NLP software installed into the production environment </li></ul></ul>
  17. 17. Tracking The Process
  18. 18. Testing Model <ul><li>For NLP software, it’s difficult to determine the appropriate level of testing </li></ul><ul><li>Regression Testing </li></ul><ul><ul><li>Ensures development does not break current behavior </li></ul></ul><ul><ul><li>Large scale test with a statistically significant sample (5% to 7%) of monthly production data </li></ul></ul><ul><ul><li>For A-Life in radiology, over 150,000 documents per batch </li></ul></ul><ul><li>Progression Testing </li></ul><ul><ul><li>Verifies changes are functioning as expected </li></ul></ul><ul><ul><li>Scale is much smaller, hundreds of documents </li></ul></ul><ul><ul><li>Use examples identified by domain experts during Requirements Analysis phase </li></ul></ul>
  19. 19. Test Execution and Analysis <ul><li>Automation of execution and analysis is essential to reach this scale </li></ul><ul><li>Unit Testing Platform </li></ul><ul><ul><li>Encapsulates the core NLP processing </li></ul></ul><ul><ul><li>Used by the NLP development team </li></ul></ul><ul><li>System Testing Platform </li></ul><ul><ul><li>Copy of the production system, including all pre- and post-processing stages </li></ul></ul><ul><li>Analysis Platform </li></ul><ul><ul><li>Scripts compare differences between any two test runs </li></ul></ul><ul><ul><li>Visual evaluation tool allows coding experts to score each change </li></ul></ul>
  20. 20. Coder Change Statistics
  21. 21. Analysis Platform
  22. 22. NLP and Scalability <ul><li>Even with large-scale testing, NLP software will encounter new or unfamiliar language </li></ul><ul><li>Two qualities of graceful behavior: </li></ul><ul><ul><li>Understand more than patterns of words but also model the underlying semantics – use ontologies </li></ul></ul><ul><ul><li>Detect situations when the content of the document is not adequately recognized by the NLP software </li></ul></ul><ul><li>Can NLP CAC software scale across medical domains? </li></ul><ul><ul><li>Most current applications are focused on medical specialties </li></ul></ul><ul><ul><li>Verification and validation to address an even larger scale </li></ul></ul><ul><ul><li>Tighter focus on a single coding system, such as ICD-9, that can be validated in narrower context </li></ul></ul>
  23. 23. Conclusion – Focus on Results <ul><li>Structured software development works for NLP </li></ul><ul><ul><li>Gives confidence to both the developer and user </li></ul></ul><ul><ul><li>Iterative, results-driven approach is best </li></ul></ul><ul><li>CAC application require means of verification that are transparent, repeatable and scalable </li></ul><ul><li>Onus on NLP software developers to verify performance on a large scale </li></ul><ul><li>Acceptable performance is difficult to quantify </li></ul><ul><li>Ongoing work - Define a verification process applicable across multiple medical domains </li></ul>
  24. 24. More Information <ul><li>Mark Morsch, VP NLP/Software Engineering, [email_address] </li></ul><ul><li>CMMI Web Site - </li></ul><ul><li>Why Software Fails. IEEE Spectrum, Sept 2005 - </li></ul><ul><li>LifeCode ® NLP technology - http:// </li></ul>