Successfully reported this slideshow.
We use your LinkedIn profile and activity data to personalize ads and to show you more relevant ads. You can change your ad preferences anytime.

Words that Matter

67 views

Published on

  • Be the first to comment

  • Be the first to like this

Words that Matter

  1. 1. 1 Words that Matter Application of Text Analytics
  2. 2. Topics  Business Questions  Success Strategy  Project Steps  Technical Solution  Analytic Requirements  Results  Business Application  Lessons Learned 2
  3. 3. Business Questions  How well has the Office of the Inspector General (OIG) fulfilled its mission?  How can the OIG prioritize final rule reviews? • Did common terms in public comments appear in final rules? • What sentiment did public comments express? 3
  4. 4. Success Strategy  Sizing the Project • Data – Available, Processable, Standardized • Security Concerns – factor in information security governance  Seeking an Executive Champion • Do they support the answer value? • To what extent will they fund the project (budgetary considerations)?  Repeating a Quick Win • Is the project repeatable to gain support for subsequent projects? 4
  5. 5.  Engaged management buy-in for questions  Assessed security concerns for public facing data  Contracted technical support and quantitative and qualitative statistical expertise  Used Amazon Web Services for infrastructure support  Used Amazon Marketplace for selecting text mining tool  Documented repeatable technical tasks 5 Project Steps
  6. 6. Technical Solution 6
  7. 7.  MarkLogic – platform enabled ability to parse unstructured text and calculate term frequencies  Term Frequency Normalization – where N is equal to the total number of terms within a document or set of documents 𝑡𝑡𝑡𝑡 𝑡𝑡 = 𝑤𝑤𝑖𝑖 𝑓𝑓 𝑁𝑁  Gap Concept – differences between normalized frequencies of baseline terms and corpus documents 7 Analytic Requirement #1
  8. 8. OIG Standards of Work  Business Question: How well has the Office of the Inspector General (OIG) fulfilled its mission?  Answer: OIG could improve its standards of audit work. 8 -0.04 -0.02 0 0.02 0.04 0.06 0.08 0.1 Baseline Terms Gap
  9. 9. OIG Mission Results 9 -0.015 -0.01 -0.005 0 0.005 0.01 Gap Baseline Terms risk Audit Mission Term Gap Analysis  “Risk” stood out for key mission terms. This suggests that the OIG generally balances workload to meet its mission.  Since “risk” is typically associated with “control” work, the OIG either has to emphasize more internal control work or the impact of the work.
  10. 10.  Utilize TeamMate software to standardize audit planning and execution  Emphasize internal control risks with project starts  Emphasize the impact associated with business question 10 Strategic Planning Application
  11. 11.  Business Question: Did common terms in public comments appear in final rules?  Answer: Yes, with varying degrees of intensity enabling differentiation. 11 Rule Review Results 0% 20% 40% 60% 80% 100% 75FR55410 76FR41398 76FR43851 76FR53172 76FR71626 76FR80674 77FR20128 77FR30596 77FR42559 81FR636 Gap Distribution Gap => + 1% Gap <= -1% -1% < Gap < 1%
  12. 12.  IBM AlchemyAPI – Natural Language Processing platform, learning algorithm  Scoring Mechanism – Positive, Neutral, Negative  Sentiment Attributes – Mixed Sentiment  Limitations of Exercise • Number of Available Comments for Each Rule • Data Quality – Data Capture, PDF’s, Noise • Document Level vs Entity Level • False Positives 12 Analytic Requirement #2
  13. 13.  Business Question: What sentiment did public comments express?  Answer: The majority of public comments are positive towards proposed rules. 13 Rule Review Results 0% 10% 20% 30% 40% 50% 60% 70% 80% 90% 100% 76FR41398 76FR43851 76FR53172 76FR80674 77FR20128 77FR30596 77FR42559 81FR636 76FR71626 Sentiment Distribution by Dodd-Frank Rule positive negative neutral
  14. 14.  Text mining tools, with some limitations, are useful in prioritizing OIG reviews of final rules.  Three rules in the negative quadrants should be considered for further study. 14 Strategic Planning Application Negative Positive Positive 77FR20128 76FR41398 76FR43851 76FR71626 77FR30596 Negative 77FR42559 76FR53172 75FR55410 Sentiment Term Frequency Gap 81FR636 76FR80674
  15. 15. Lessons Learned—Success Strategy 15 ?  Sizing the Project • Data – Available, Processable, Standardized • Security Concerns – factor in information security governance  Seeking an Executive Champion • Do they support the answer value? • To what extent will they fund the project (budgetary considerations)?  Repeating a Quick Win • Is the project repeatable to gain support for subsequent projects?

×