• Save
Peter Norvig - NYC Machine Learning 2013
Upcoming SlideShare
Loading in...5
×
 

Peter Norvig - NYC Machine Learning 2013

on

  • 1,047 views

 

Statistics

Views

Total Views
1,047
Views on SlideShare
1,047
Embed Views
0

Actions

Likes
2
Downloads
0
Comments
0

0 Embeds 0

No embeds

Accessibility

Categories

Upload Details

Uploaded via as Adobe PDF

Usage Rights

CC Attribution-NonCommercial LicenseCC Attribution-NonCommercial License

Report content

Flagged as inappropriate Flag as inappropriate
Flag as inappropriate

Select your reason for flagging this presentation as inappropriate.

Cancel
  • Full Name Full Name Comment goes here.
    Are you sure you want to
    Your message goes here
    Processing…
Post Comment
Edit your comment

Peter Norvig - NYC Machine Learning 2013 Presentation Transcript

  • 1. Learning Programming (by Humans by Machine) Peter Norvig
  • 2. Ng Koller Norvig Thrun MOOCs and Machine Learning
  • 3. For when you don’t know what’s going on
  • 4. Hal Abelson (SIGCSE Outstanding Contribution Award, 2012)
  • 5. The average under tutoring [with mastery learning] was about two standard deviations above the average of the control. (Benjamin Bloom, 1984)
  • 6. Python llll lll C ll Scheme l But: for “vocational”, not “academic”: Javascript dominates
  • 7. Analyzing Intro CS MOOC Data 101 Student Name Final Grade Peter Norvig
  • 8. Analyzing Intro CS MOOC Data 101 Student Name Final Grade Peter Norvig Incomplete
  • 9. 160,000 students; 209 countries
  • 10. Data Sources • Enrollment / Dropout • Quizzes and Exams • Video Interactions • Discussion Forums • Program submissions
  • 11. Syntax Error Messages Vague if x = 1: ^ SyntaxError: invalid syntax SyntaxError: Use “==“ for comparison, use “=“ for assignment.
  • 12. Inputs to Compiler Error Routine Source Code Compiler Error Message Ignored: - Past program submission histories - Problem Description - Comments - Test Cases
  • 13. CFG: Python Non-Python
  • 14. PCFG: Python-ish Probably Approximately Correct Grammar and Intent
  • 15. Match Programs to Corpus S1 suggest C2C1Corpus: Submission:
  • 16. Syntax Error Messages Vague if spam == 42 print('Hello!') ^ SyntaxError: invalid syntax ^ IndentationError: unexpected indent SyntaxError: Colon required, use “if spam == 42:“
  • 17. Attractive Nuisance >>> table = [[]] * 5 >>> table [[], [], [], [], []] >>> def add(key, table): table[hash(key) % len(table)].append(key) >>> add(’x', table) >>> table [[’x'], [’x'], [’x'], [’x'], [’x']] >>> table = listof(5, list) def listof(n, kind): "Make a list of n elements, each kind()." return [kind() for _ in range(n)]
  • 18. Faulty Student Models • Given: • nums = [1, 2, -3] • max(nums) == 2 • max(nums, key=abs) == -3 • len(‘word’) == 4 • wordlist = file(‘words’).read().split() • What is the longest word in wordlist? • ? • ?? • max(wordlist, key=len)
  • 19. DWIM Attractive Nuisance • def qsort(A): if len(A) <= 1: return A LT, EQ, GT = [] p = A[len(A)/2] for x in A: bin = (LT if x<p else GT if x>p else EQ) bin.append(x) return qsort(LT) + EQ + qsort(GT) LT = EQ = GT = [] LT, EQ, GT = [], [], []
  • 20. DWIM Attractive Nuisance • spam = ['cat', 'dog', 'mouse'] • for i in range(spam): print(spam[i]) • for i in len(spam): print(spam[i]) • for i in range(len(spam)): print(spam[i]) for x in spam: print(x) • for i in range(len(spam)): spam[i] = spam[i].upper() for x in spam: x = x.upper()
  • 21. Language Too General y = (x + 1) (x - 1) TypeError: unsupported operand type(s) for +: 'object' and 'int' SyntaxError: ‘*’ operator required. use “(x + 1) * (x – 1)”
  • 22. Multi-Language Confusion File ”example.py”, line 1 root = new Node() ^ SyntaxError: invalid syntax Test.java:6: cannot find symbol symbol : class awt location: package java java.awt.Color(r, g, b); ^ Error: use “Node()” in Python; use “new Color(…)” in Java
  • 23. Multi-Language Confusion int total = 0; for (int x : A) { total += x; } total = None for x in A: total = total + x
  • 24. Why do students stay or go?
  • 25. the movie in which no babies are born
  • 26. `
  • 27. Motivation, not Information • Willpower • Due dates • Peer support (forums, outside groups) • Faculty encouragement (email) • Pride of accomplishment • Authenticity • Early adopter
  • 28. Timeline (successful Coursera course) #usersonsite Real Couse 100K 50K ramp up + exponential decay + due dates synchronous, evergreen, semi-synchronous, bus route
  • 29. Learning Programming
  • 30. Representing Programs • Bag of words • N-grams of tokens (tagged?) • Static syntax tree • Dynamic call graph
  • 31. Representing Program Sequences HMM
  • 32. Alpha, Beta (steady progress) significantly better than Gamma (thrash, then solution) on midterm. No significant difference on score for this assignment.
  • 33. What is Learning?
  • 34. • Learning results from what the student does and thinks and only from what the student does and thinks. The teacher can advance learning only by influencing what the student does to learn. - Herb Simon
  • 35. • Problem first, then explanation (Peer) • Make students to prediction (see it is wrong) • (They didn’t like that.) • Experiments with more open-ended problems • Experiments with peer grading
  • 36. • “Richard Hamming told me his secret: First get together the problem sets and exams that you want the students to be able to solve. Then write a book that will teach them how to solve them.” -- Hal Varian (1993) T D D
  • 37. Do we trust Bloom?
  • 38. 0 0.5 1 1.5 2 2.5 Drill Intelligent Tutoring System Human Tutor Expected Actual
  • 39. 0 0.5 1 1.5 2 2.5 Drill Intelligent Tutoring System Human Tutor Expected Actual More Mastery than Tutoring
  • 40. 0 0.5 1 1.5 2 2.5 Drill Intelligent Tutoring System Human Tutor Expected Actual N = 33 More Mastery than Tutoring
  • 41. How do we make it one to one tutoring and mastery learning?
  • 42. “Books will soon be obsolete in the public schools. Scholars will be instructed through the eye. It is possible to teach every branch of human knowledge with the motion picture. Our school system will be completely changed inside of ten years”- Edison, 1913
  • 43. “Lectures were once useful, but now, when all can read, and books so numerous, lectures are unnecessary. If your attention fails, and you miss part of a lecture, it is lost; you cannot go back as you do upon a book.” - 1791
  • 44. Many Types of Individualization • Student control over rewind (multiplay) • Student choice of where to go next (multipath) • Flipped classroom (Koller, etc.) • Peer instruction (Mazur) • Peer feedback • Forums and other one-to-one interactions • Machine learning? • Courseware Engineering?
  • 45. Open-Ended Work Peer Grade Self Grade 0 10 20 30 40 50 60 70 80 90 100 0 10 20 30 40 50 60 70 80 90 100 Teacher Grade StudentGrade “The Impact of Self-and Peer-Grading on Student Learning”. P. Sadler, E. Good. Educational Assessment (2006).
  • 46. 20 bits per year 192K bits per sec
  • 47. Data Threshold?
  • 48. Modern sequencing methods (11 min) Genomic economics (8 min) Personalized medicine (13 min) History: The Human Genome Project (12 min) Modern GenomicsGenes and alleles (10 min) Basic Genetics Refresher (OPTIONAL) Mendelian inheritance (13 min) What is DNA? (12 min) Genetic testing in the commercial world (11 min) Protecting privacy (9 minutes) Case study: direct- to-consumer genetics (12 min) Case study: family planning (13 min) Commercial Genomics (OPTIONAL) Authored Multipath Sequence
  • 49. Courseware Engineering?
  • 50. One on One Tutoring; Mastery Learning Social; Motivation Large-Scale Courseware Engineering Machine Learning; Continual Improvement