Downloaded 28 times
Uploaded byDaniyar Mukhanov
Weka.arff
This document discusses using two classifiers, JRip and J48, on a zoo dataset containing animal attributes. It explores various preprocessing and evaluation techniques in Weka, including removing duplicates and useless attributes, cross validation, percentage splits, and comparing the classifiers using Experimenter. J48 slightly outperforms JRip on this dataset when run with default options and 10-fold cross validation. The document concludes that while J48 may generate more complex trees, it beats JRip for this small dataset without preprocessing. Experimenter and Knowledge Flow are useful Weka tools for comparing classifiers and understanding intermediate steps.
Weka.arff
- 1. Data Analysis withWeka zoo.arff Done by Clement Robert H. Daniyar M. Web and Social Computing
- 2. Dataset Zoo.arff: A simple databasecontaining 17 Boolean-valued attributes. The "type" attribute appears to be the class attribute. Here is a breakdown of which animals are in which type.
- 3. Objectives ● Select Dataset ●Learn and Explain the two classifiers ● Apply them to the selected Dataset ● Compare the results
- 4. Outline ● DataSet Visualization ●Preprocessing ● Classification ● JRip Implementation and Results ● J48 Implementation and Results ● Knowledge Flow Implementation ● Experimenter (Comparison of the Two Classifiers) ● Conclusions ● Questions and Answers
- 5. Visualization mammal, bird, reptile,fish, amphibian, insect, invertebrate.
- 6. Basic statistics mammal, bird,reptile, fish, amphibian, insect, invertebrate.
- 7. Hair mammal, bird, reptile,fish, amphibian, insect, invertebrate. mammal, bird, reptile, fish, amphibian, insect, invertebrate.
- 8. Feathers mammal, bird, reptile,fish, amphibian, insect, invertebrate. mammal, bird, reptile, fish, amphibian, insect, invertebrate.
- 9. Eggs mammal, bird, reptile,fish, amphibian, insect, invertebrate. mammal, bird, reptile, fish, amphibian, insect, invertebrate.
- 10. Milk mammal, bird, reptile,fish, amphibian, insect, invertebrate. mammal, bird, reptile, fish, amphibian, insect, invertebrate.
- 11. Catsize mammal, bird, reptile,fish, amphibian, insect, invertebrate. mammal, bird, reptile, fish, amphibian, insect, invertebrate.
- 12. Pre-Processing Why? Incomplete Data Noisy Data InconsistentData How? Discretize Remove Duplicates RemoveUseless Etc
- 13. Classification “Data mining techniqueused to predict group membership for data instances” Types of classifiers: Decision Trees J48 Rule Based Classifiers JRip Bayes Naive Bayes
- 14. Classifiers Options Explained-Appliedon DataSet Training Data: We use the whole dataset as training set. It gives the best results for the data set itself but does not guarantee the best test for unseen data. Cross-Validation: Divide the Data Set into K subsamples and use k-1 subsamples as training data and one subsample as test data. Percentage Split: We divide the dataset into two parts: the first X% of the data set is used as training and the rest is used as the test set.
- 15. JRIP Classifier Algorithm Basedon RIPPER Algorithm RIPPER: Repeated Incremental Pruning to Produce Error Reduction Incremental Pruning Error Reduction Produces Rules Works fine for: Class: Missing classes values, binary and Nominal Classes Advantages As highly expressive as decision trees Easy to interpret Easy to generate Can classify new instances rapidly “If Age is greater than 35 and Status is Married Then He/she Does not Cheat”
- 16. Classifier Evaluation-Default OptionsJRip Pre-Processed?-> NO Results
- 17. PreProcessed?-> No Results->NOBIG CHANGE Classifier Evaluation->Cross Validation Increased JRip ● Same Options as Previous ● Except ● Cross-Validation= 20 Folds “If the Cross-Validation Increases, the correctly Classified Instances Decreasing” Why? Next Cross-Validation=40 etc
- 18. PreProcessed?-> No Results->NOBIG CHANGE Classifier Evaluation->Cross Validation JRip ● No Relation among those values ● Why 10? Extensive experiments have shown that this is the best choice to get an accurate estimate for small Data Set ( we have 101 instances and 17 attributes) ● The more we increase the number of FOLDS in cross-Validation, the more we decrease the size of training subsets! (eg: for k=80, we wanted to have 80 training subsets, in 101 instances!) Cross Validation Correctly Classified Inst. 10 87.14 20 85.14 30 86.13 40 84.15 50 87.12 60 82.17 70 85.14 80 87.12 90 84.15
- 19. PreProcessed?-> No Results ClassifierEvaluation->Training Set JRip
- 20. PreProcessing-> Not Yet ClassifierEvaluation->Training Set JRip Cross Validation (K) Correctly Classified Inst. 10 87.14 20 85.14 30 86.13 40 84.15 50 87.12 60 82.17 70 85.14 80 87.12 90 84.15 VS Training Set 92.07 Training Set ● works with the totality of the data set as training and test Data ● It works better for the given data but does not gives us confidence to use it the unseen cases (prediction, etc)
- 21. Pre-Processing- Why? JRip ● Rulegenerated based on useless attribute ○ Filter with RemoveUseless ● Possible Duplicates->can lead to biased Rules-> We never know ○ Filter with RemoveDuplicates ● Other Filters ○ Discretize, etc
- 22. ● RemoveDuplicates: Removes allduplicate instances from the first batch of data it receive ● RemoveUseless: This filter removes attributes that do not vary at all or that vary too much Pre-Processing-> RemoveDuplicates (instances), RemoveUseless (attributes) JRip
- 23.
- 24. Classifier Evaluation-> PercentageSplit-> Default (66%) JRip PreProcessed?-> Yes Results ● With this split, the test samples do not have Amphibians! ● Has good statistics because test is done on small test set.
- 25. Classifier Evaluation-> PercentageSplit-> 80% JRip PreProcessed?-> Yes Results Percentage Split ● Takes samples randomly ● Chance for some representatives to be left out http://www.cs.waikato.ac.nz/ml/weka/mooc/dataminingwithweka/
- 26. What Next? JRip ●Done with Test Options Applied to Data ● JRIP has its own parameters to change in order to make it work better ○ Folds, MinWeights, Seeds of randomization, Error Rate and Pruning, ● Pruning ○ Pre-Pruning: Cut the classifier to continue growing when condition is met ○ Post-Pruning: Let the Classifier grow, and the reduce it to make it small so as to cover more unseen data. ● Without Pruning, the classifier is more detailed which makes it to be limited on unseen cases. ● JRip uses Post Pruning Method (based on REP) ● Next, JRip with Pruning, Vs JRip without Pruning
- 27. Classifier Evaluation->Pruning-TRUE VsFALSE JRip PreProcessed?-> no Results with Experimenter*
- 28. J48 Classifier Algorithm Basedon C4.5 Algorithm Modified ID3 continuous attributes missing attributes attributes with differing costs post-pruning trees Produces Decision Tree Works fine for: The advantages of the C4.5 are: • Builds models that can be easily interpreted • Easy to implement • Can use both categorical and continuous values • Deals with noise
- 29.
- 30.
- 31. Cross validation =20 J48 Nomajor changes ROC and PRC area dropped insignificantly Confusion matrix and Decision tree remains same
- 32. Changing cross validationvalue J48 Cross Validation Correctly Classified Inst. 10 92.079 20 92.079 30 92.079 40 92.079 50 92.079 60 92.079 70 92.079 80 92.079 90 92.079 Changes nothing including ROC, PRC, confusion matrix and decision tree This is because number of instances are relatively small for J48 (101)
- 33. Classifier by trainingset J48
- 34.
- 35.
- 36.
- 37. Comparing two setsof rules
- 38. Getting same resultsby Knowledge Flow
- 39. Getting same resultsby Knowledge Flow (con’t)
- 40. J48? or JRipon Zoo.arff-> Comparison with Experimenter Feature Experimenter is Suitable for: ● Large Scale Experiments ● Automation ● Statistics can be stored in .arff format ● ….. ● Classifiers Comparison
- 41. J48? or JRip?on Zoo.arff?-> Results With: ● Significance of 10% on Percentage Correct Instances ● Cross-Validation=10 ● Default Classifiers Options We see that: ● J48 is the Winner on Zoo.arff
- 42. Lesson Learnt &Conclusions ● Output Readability ○ JRip outputs are easy to read and understand ○ J48 Trees can be more complex to read ● Performance ○ J48 beats JRip, as it generate general tree without Pre-Processing ○ J48 generates Tree whose precision is high ○ No big difference for small data set like zoo.arff ● Weka test options ○ Can mislead someone who only looks for good results. ○ No good/bad classifier. It depends on many characteristics (dataset size, options used, classifier used etc). ● Weka ○ Experimenter gives a good way to compare classifiers ○ Knowledge Flow helps to get the intermediary steps in the generation of classifier (helps to understand how some options work like cross validation)
- 43. Q & A Thankyou