This document discusses algorithm-independent machine learning techniques. It introduces concepts like bias and variance, which can quantify how well a learning algorithm matches a problem without depending on a specific algorithm. Methods like cross-validation, bootstrapping, and resampling can be used with different algorithms. While no algorithm is inherently superior, such techniques provide guidance on algorithm use and help integrate multiple classifiers.

1. Algorithm-Independent Machine Learning Shyh-Kang Jeng Department of Electrical Engineering/ Graduate Institute of Communication/ Graduate Institute of Networking and Multimedia, National Taiwan University

9. No Free Lunch Theorem

12. Example 1 No Free Lunch for Binary Data -1 1 1 111 -1 1 1 110 -1 1 -1 101 -1 1 1 100 -1 1 -1 011 1 1 1 010 -1 -1 -1 001 D 1 1 1 000 h 2 h 1 F x

13. No Free Lunch Theorem

16. Venn Diagram Representation of Features as Predicates

33. Bayesian Perspective of MDL Principle

37. Bias and Variance for Regression

38. Bias-Variance Dilemma

44. Bias and Variance for Classification

45. Bias and Variance for Classification

46. Bias and Variance for Classification

47. Bias and Variance for Classification

49. Error rates and optimal K vs. N for d = 20 in KNN

50. Estimation and classification error vs. d for N = 12800 in KNN d

51. Boundary Bias-Variance Trade-Off

52. Leave-One-Out Method (Jackknife)

54. Jackknife Bias Estimate

55. Example 2 Jackknife for Mode

57. Bootstrap Bias and Variance Estimate

63. Training Data and Weak Learner

69. AdaBoost Algorithm

71. Ensemble Training Error

77. Active Learning Example

88. 95% Confidence Intervals for a Given Estimated p

90. Jackknife Estimation of Classification Accuracy

93. Maximum-Likelihood Comparison

94. Scientific Process *D. J. C. MacKay, “Bayesian interpolation,” Neural Computation, 4(3), 415-447, 1992

95. Bayesian Model Comparison

96. Concept of Occam Factor

98. Evidence for Gaussian Parameters

102. Analytical Analysis

103. Analytical Analysis

104. Analytical Analysis

105. Results of Simulation Experiments

108. Test Errors vs. Number of Training Patterns

109. Test and Training Error

110. Power Law

111. Sum and Difference of Test and Training Error

112. Fraction of Dichotomies of n Points in d Dimensions That are Linear

113. One-Dimensional Case f ( n = 4, d = 1) = 0.5 X 1111 X 0111 X 1110 0110 1101 0101 X 1100 0100 1011 X 0011 1010 0010 1001 X 0001 X 1000 X 0000 Linearly Separable? Labels Linearly Separable? Labels

116. Mixture Model for Producing Patterns

117. Mixture-of-Experts Architecture

118. Ensemble Classifiers

119. Maximum-Likelihood Estimation

122. Heuristics to Convert Outputs to Discrimunant Values

123. Illustration Examples 0.0 0 3/21=0.143 4th 0.111 0.1 0.0 0 5/21=0.238 2nd 0.129 0.2 0.0 0 6/21=0.286 1st 0.143 0.3 0.0 0 2/21=0.095 5th 0.260 0.9 1.0 1 1/21=0.048 6th 0.193 0.6 0.0 0 4/21=0.194 3rd 0.158 0.4 g i g i g i One-of- c Rank Order Analog