Using HPCC Systems ML to Map Thousands of Public Records Data Descriptions to Standard Codes

1. Innovation and Reinvention Driving Transformation OCTOBER 9, 2018 2018 HPCC Systems® Community Day Gus Reyna , Lili Xu Using HPCC Systems ML to Map Thousands of Public Records Data Descriptions to Standard Codes

2. Introduction • Background • Approach • Exploratory Analysis • Next steps Using HPCC Systems Machine Learning to map thousands of violation descriptions to Standard Violation Codes 4

3. Background • Public records data: birth/marriage/death certificates, business/professional/contractor licenses, foreclosures and tax liens, etc. • Businesses that use this data in their information technology processes must account for state variations of similar events. • LexisNexis maps public record data from different states to standard categories. • Businesses use the standard categories to create one system that can be used in all states. Using HPCC Systems Machine Learning to map thousands of violation descriptions to Standard Violation Codes 5

4. Problem • Team of Subject Matter Experts (SME) map public record data to standardized categories. • Data grew faster than team’s mapping capacity. • Data products time to market increased. Using HPCC Systems Machine Learning to map thousands of violation descriptions to Standard Violation Codes 6

5. Solution • Grow the team, train the team. • BUT, Trainers are SMEs who are not mapping when they’re training new team members • AND, It takes time to learn how to map public records data to standard categories Using HPCC Systems Machine Learning to map thousands of violation descriptions to Standard Violation Codes 7

6. Solution • Use HPCC Systems machine learning to generate 3 recommended standard categories for public record data … • Which shortens the time new team members become effective mappers and … • Reduces the time required for SMEs to train new team members Using HPCC Systems Machine Learning to map thousands of violation descriptions to Standard Violation Codes 8

7. Approach Using HPCC Systems Machine Learning to map thousands of violation descriptions to Standard Violation Codes 9 Categorize d Data Clean & Build Vocabulary Training Data Validation data Build Models New Data Models** Run Throug h Model Run Through Model Top 3 Category Recommendations Mappers ** Support Vector Machine (SVM) Naïve Bayes Category Description Category 1 DOGS RUNNING AT LARGE Category 1 CANINE RUNNING AT LARGE PROHIBITED Category 2 RUNNING A RED LIGHT Category 3 FISHING WITHOUT LICENSE

8. Build Vocabulary Using HPCC Systems Machine Learning to map thousands of violation descriptions to Standard Violation Codes 10 Categorize d Data Clean & Build Vocabulary WORD COUNT RUN 3 LARG 2 CANIN 1 …. …. http://textanalysisonline.com/nltk-porter-stemmer Category Description Category 1 DOGS RUNNING AT LARGE Category 1 CANINE RUNNING AT LARGE PROHIBITED Category 2 RUNNING A RED LIGHT Category 3 FISHING WITHOUT LICENSE

9. Build & Validate Model Using HPCC Systems Machine Learning to map thousands of violation descriptions to Standard Violation Codes 11 Build Models Models* Run Through Model Category Description Category 1 CANIN RUN AT LARG PROHIBIT Category 2 RUN A RED LIGHT Category 3 FISH WITHOUT LICENS * Support Vector Machine (SVM) Naïve Bayes Category Description Category 1 DOG RUN AT LARG Recommendations Category Description Category 1 Category 1 DOGS RUNNING AT LARGECategory 2 Category 3 Category Description Category 1 DOGS RUNNING AT LARGE Category 1 CANINE RUNNING AT LARGE PROHIBITED Category 2 RUNNING A RED LIGHT Category 3 FISHING WITHOUT LICENSE Training Data Validation data

10. Process New Data Using HPCC Systems Machine Learning to map thousands of violation descriptions to Standard Violation Codes 12 Models* Top 3 Category Recommendations Mappers * Support Vector Machine (SVM) Naïve Bayes Description CATS RUNNING AT LARGE HUNTING WITHOUT LICENSE Training Data Category Description Category 1 CANINE RUNNING AT LARGE PROHIBITED Category 2 RUNNING A RED LIGHT Category 3 FISHING WITHOUT LICENSE Recommende d Categories Description 1, 2, 3 CATS RUNNING AT LARGE 3, 1, 2 HUNTING WITHOUT LICENSE NEW Data

11. Approach Using HPCC Systems Machine Learning to map thousands of violation descriptions to Standard Violation Codes 13 Categorize d Data Clean & Build Vocabulary Training Data Validation data Build Models New Data Models** Run Throug h Model Run Through Model Top 3 Category Recommendations Mappers ** Support Vector Machine (SVM) Naïve Bayes Category Description Category 1 DOGS RUNNING AT LARGE Category 1 CANINE RUNNING AT LARGE PROHIBITED Category 2 RUNNING A RED LIGHT Category 3 FISHING WITHOUT LICENSE

12. Outcome • Backlog of public record data to standard category mapping eliminated. • Time to market for data products shortened, no more delays from data mapping to categories • Happy mapping team – could work on data enhancement projects. Using HPCC Systems Machine Learning to map thousands of violation descriptions to Standard Violation Codes 14

13. Exploratory Analysis 15 Using HPCC Systems Machine Learning to map thousands of violation descriptions to Standard Violation Codes

14. NLP Toolkits on HPCC Systems Platform Using HPCC Systems Machine Learning to map thousands of violation descriptions to Standard Violation Codes 16 RECORD DESCRIPTION OPERATING/VEH/OVER MAX HGT RECORD OPERATING VEH OVER MAX HGT RECORD OPERATING VEHICLE OVER MAX HEIGHT RECORD OPER VEHICL MAX HEIGHT TOKENIZO R STOP-WORDS REMOVER SEMANTIC ANALYZOR N-GRAM RECORD OPER VEHICL VEHICL MAX MAX HEIGHT RECORD OPER VEHICL OVER MAX HEIGHT STEMMER

15. Latent Dirichlet Allocation - Topic Model Using HPCC Systems Machine Learning to map thousands of violation descriptions to Standard Violation Codes 17 • Unsupervised Natural Language Processing(NLP) Algorithm • Explore the Topics in Documents. • Each topic is a distribution over words • Each document is a mixture of topics • Each word is drawn from the topics LDA Topic Model

16. TOPIC 1 TOPIC 2 TOPIC 3 TOPIC 4 TOPIC 5 TOPIC 6 TOPIC 7 TOPIC 8 TOPIC 9 TOPIC 10 OPER UNLA W PROO F DRIVE IMPRO P PARK VEHIC L SPEED INSUR MOTOCY CL INSUR REQUI R SPEED DRIVE UNLAW SPEED SPEED SPEED IMPRO P REQUIR PROOF VEHICL PROO F OPER OPER UNLAW VEHIC L EY PARK IMPROP INSUR REQUIR UNLAW REQUI R UNLA W SPEED VEHIC L EY UNLA W VEHICL PROO F INSUR MOTORCY CL PARK UNLA W INSUR REQUI R IMPRO P MOTORCY CL FAIL EY PROTE CT FAIL MOTORCY CL IMPROP OPER Scalable LDA on HPCC Systems Platform • Massive Parallel Topic Modeling • Flexible Hyper-Parameter Setup • Experiments Topic Range [10 – 103] Using HPCC Systems Machine Learning to map thousands of violation descriptions to Standard Violation Codes 18 LDA TOPIC MODEL RESULT

17. Next Steps • Continue exploratory analysis • Additional algorithms • Automatically map data to categories, not just make recommendations • Refine existing and build new models • Solve other business problems with HPCC Systems Machine Learning • Uniform language • Ease of data access • High productivity 19 Using HPCC Systems Machine Learning to map thousands of violation descriptions to Standard Violation Codes

Using HPCC Systems ML to Map Thousands of Public Records Data Descriptions to Standard Codes

You are reading a preview.

Check these out next

Using HPCC Systems ML to Map Thousands of Public Records Data Descriptions to Standard Codes

More Related Content

Similar to Using HPCC Systems ML to Map Thousands of Public Records Data Descriptions to Standard Codes (20)

More from HPCC Systems (20)

Recently uploaded (20)