Learning Programming
(by Humans
by Machine)
Peter Norvig
Ng Koller
Norvig Thrun
MOOCs
and
Machine
Learning
For when you don’t
know what’s going on
Hal Abelson
(SIGCSE Outstanding Contribution Award, 2012)
The average under tutoring [with mastery learning]
was about two standard deviations above the
average of the control. (Be...
Python llll lll
C ll
Scheme l
But: for “vocational”, not “academic”:
Javascript dominates
Analyzing Intro CS MOOC Data 101
Student Name Final Grade
Peter Norvig
Analyzing Intro CS MOOC Data 101
Student Name Final Grade
Peter Norvig Incomplete
160,000 students; 209 countries
Data Sources
• Enrollment / Dropout
• Quizzes and Exams
• Video Interactions
• Discussion Forums
• Program submissions
Syntax Error Messages Vague
if x = 1:
^
SyntaxError: invalid syntax
SyntaxError: Use “==“ for comparison,
use “=“ for assi...
Inputs to Compiler Error Routine
Source
Code
Compiler
Error
Message
Ignored:
- Past program submission histories
- Problem...
CFG:
Python
Non-Python
PCFG:
Python-ish
Probably Approximately Correct
Grammar and Intent
Match Programs to Corpus
S1 suggest
C2C1Corpus:
Submission:
Syntax Error Messages Vague
if spam == 42
print('Hello!')
^
SyntaxError: invalid syntax
^
IndentationError: unexpected ind...
Attractive Nuisance
>>> table = [[]] * 5
>>> table
[[], [], [], [], []]
>>> def add(key, table):
table[hash(key) % len(tab...
Faulty Student Models
• Given:
• nums = [1, 2, -3]
• max(nums) == 2
• max(nums, key=abs) == -3
• len(‘word’) == 4
• wordli...
DWIM Attractive Nuisance
• def qsort(A):
if len(A) <= 1: return A
LT, EQ, GT = []
p = A[len(A)/2]
for x in A:
bin = (LT if...
DWIM Attractive Nuisance
• spam = ['cat', 'dog', 'mouse']
• for i in range(spam):
print(spam[i])
• for i in len(spam):
pri...
Language Too General
y = (x + 1) (x - 1)
TypeError: unsupported operand
type(s) for +: 'object' and
'int'
SyntaxError: ‘*’...
Multi-Language Confusion
File ”example.py”, line 1
root = new Node()
^
SyntaxError: invalid syntax
Test.java:6: cannot fin...
Multi-Language Confusion
int total = 0;
for (int x : A) {
total += x;
}
total = None
for x in A:
total = total + x
Why do students stay or go?
the movie in which no babies are born
`
Motivation, not Information
• Willpower
• Due dates
• Peer support (forums, outside groups)
• Faculty encouragement (email...
Timeline (successful Coursera course)
#usersonsite
Real Couse
100K
50K
ramp up + exponential decay + due dates
synchronous...
Learning Programming
Representing Programs
• Bag of words
• N-grams of tokens (tagged?)
• Static syntax tree
• Dynamic call graph
Representing Program Sequences
HMM
Alpha, Beta (steady progress) significantly better than Gamma (thrash, then
solution) on midterm. No significant differenc...
What is Learning?
• Learning results from
what the student does
and thinks and only
from what the student
does and thinks. The
teacher can a...
• Problem first, then explanation (Peer)
• Make students to prediction (see it is wrong)
• (They didn’t like that.)
• Expe...
• “Richard Hamming told me his
secret: First get together the problem
sets and exams that you want the
students to be able...
Do we trust Bloom?
0
0.5
1
1.5
2
2.5
Drill Intelligent
Tutoring
System
Human
Tutor
Expected
Actual
0
0.5
1
1.5
2
2.5
Drill Intelligent
Tutoring
System
Human
Tutor
Expected
Actual
More Mastery than Tutoring
0
0.5
1
1.5
2
2.5
Drill Intelligent
Tutoring
System
Human
Tutor
Expected
Actual
N = 33
More Mastery than Tutoring
How do we make it
one to one tutoring
and
mastery learning?
“Books will soon be obsolete in the public schools.
Scholars will be instructed through the eye.
It is possible to teach e...
“Lectures were once useful,
but now, when all can read,
and books so numerous,
lectures are unnecessary.
If your attention...
Many Types of Individualization
• Student control over rewind (multiplay)
• Student choice of where to go next (multipath)...
Open-Ended Work
Peer Grade
Self Grade
0
10
20
30
40
50
60
70
80
90
100
0 10 20 30 40 50 60 70 80 90 100
Teacher Grade
Stud...
20 bits per year
192K bits per sec
Data Threshold?
Modern sequencing
methods (11 min)
Genomic
economics (8
min)
Personalized
medicine (13 min)
History: The
Human Genome
Proj...
Courseware Engineering?
One on One Tutoring; Mastery Learning
Social;
Motivation
Large-Scale Courseware Engineering
Machine Learning; Continual Im...
Peter Norvig - NYC Machine Learning 2013
Peter Norvig - NYC Machine Learning 2013
Peter Norvig - NYC Machine Learning 2013
Peter Norvig - NYC Machine Learning 2013
Peter Norvig - NYC Machine Learning 2013
Peter Norvig - NYC Machine Learning 2013
Peter Norvig - NYC Machine Learning 2013
Peter Norvig - NYC Machine Learning 2013
Peter Norvig - NYC Machine Learning 2013
Peter Norvig - NYC Machine Learning 2013
Peter Norvig - NYC Machine Learning 2013
Peter Norvig - NYC Machine Learning 2013
Peter Norvig - NYC Machine Learning 2013
Peter Norvig - NYC Machine Learning 2013
Peter Norvig - NYC Machine Learning 2013
Peter Norvig - NYC Machine Learning 2013
Peter Norvig - NYC Machine Learning 2013
Peter Norvig - NYC Machine Learning 2013
Peter Norvig - NYC Machine Learning 2013
Peter Norvig - NYC Machine Learning 2013
Peter Norvig - NYC Machine Learning 2013
Peter Norvig - NYC Machine Learning 2013
Peter Norvig - NYC Machine Learning 2013
Peter Norvig - NYC Machine Learning 2013
Peter Norvig - NYC Machine Learning 2013
Peter Norvig - NYC Machine Learning 2013
Peter Norvig - NYC Machine Learning 2013
Peter Norvig - NYC Machine Learning 2013
Peter Norvig - NYC Machine Learning 2013
Peter Norvig - NYC Machine Learning 2013
Peter Norvig - NYC Machine Learning 2013
Peter Norvig - NYC Machine Learning 2013
Peter Norvig - NYC Machine Learning 2013
Peter Norvig - NYC Machine Learning 2013
Peter Norvig - NYC Machine Learning 2013
Upcoming SlideShare
Loading in …5
×

Peter Norvig - NYC Machine Learning 2013

1,288 views

Published on

Published in: Technology, Education

Peter Norvig - NYC Machine Learning 2013

  1. 1. Learning Programming (by Humans by Machine) Peter Norvig
  2. 2. Ng Koller Norvig Thrun MOOCs and Machine Learning
  3. 3. For when you don’t know what’s going on
  4. 4. Hal Abelson (SIGCSE Outstanding Contribution Award, 2012)
  5. 5. The average under tutoring [with mastery learning] was about two standard deviations above the average of the control. (Benjamin Bloom, 1984)
  6. 6. Python llll lll C ll Scheme l But: for “vocational”, not “academic”: Javascript dominates
  7. 7. Analyzing Intro CS MOOC Data 101 Student Name Final Grade Peter Norvig
  8. 8. Analyzing Intro CS MOOC Data 101 Student Name Final Grade Peter Norvig Incomplete
  9. 9. 160,000 students; 209 countries
  10. 10. Data Sources • Enrollment / Dropout • Quizzes and Exams • Video Interactions • Discussion Forums • Program submissions
  11. 11. Syntax Error Messages Vague if x = 1: ^ SyntaxError: invalid syntax SyntaxError: Use “==“ for comparison, use “=“ for assignment.
  12. 12. Inputs to Compiler Error Routine Source Code Compiler Error Message Ignored: - Past program submission histories - Problem Description - Comments - Test Cases
  13. 13. CFG: Python Non-Python
  14. 14. PCFG: Python-ish Probably Approximately Correct Grammar and Intent
  15. 15. Match Programs to Corpus S1 suggest C2C1Corpus: Submission:
  16. 16. Syntax Error Messages Vague if spam == 42 print('Hello!') ^ SyntaxError: invalid syntax ^ IndentationError: unexpected indent SyntaxError: Colon required, use “if spam == 42:“
  17. 17. Attractive Nuisance >>> table = [[]] * 5 >>> table [[], [], [], [], []] >>> def add(key, table): table[hash(key) % len(table)].append(key) >>> add(’x', table) >>> table [[’x'], [’x'], [’x'], [’x'], [’x']] >>> table = listof(5, list) def listof(n, kind): "Make a list of n elements, each kind()." return [kind() for _ in range(n)]
  18. 18. Faulty Student Models • Given: • nums = [1, 2, -3] • max(nums) == 2 • max(nums, key=abs) == -3 • len(‘word’) == 4 • wordlist = file(‘words’).read().split() • What is the longest word in wordlist? • ? • ?? • max(wordlist, key=len)
  19. 19. DWIM Attractive Nuisance • def qsort(A): if len(A) <= 1: return A LT, EQ, GT = [] p = A[len(A)/2] for x in A: bin = (LT if x<p else GT if x>p else EQ) bin.append(x) return qsort(LT) + EQ + qsort(GT) LT = EQ = GT = [] LT, EQ, GT = [], [], []
  20. 20. DWIM Attractive Nuisance • spam = ['cat', 'dog', 'mouse'] • for i in range(spam): print(spam[i]) • for i in len(spam): print(spam[i]) • for i in range(len(spam)): print(spam[i]) for x in spam: print(x) • for i in range(len(spam)): spam[i] = spam[i].upper() for x in spam: x = x.upper()
  21. 21. Language Too General y = (x + 1) (x - 1) TypeError: unsupported operand type(s) for +: 'object' and 'int' SyntaxError: ‘*’ operator required. use “(x + 1) * (x – 1)”
  22. 22. Multi-Language Confusion File ”example.py”, line 1 root = new Node() ^ SyntaxError: invalid syntax Test.java:6: cannot find symbol symbol : class awt location: package java java.awt.Color(r, g, b); ^ Error: use “Node()” in Python; use “new Color(…)” in Java
  23. 23. Multi-Language Confusion int total = 0; for (int x : A) { total += x; } total = None for x in A: total = total + x
  24. 24. Why do students stay or go?
  25. 25. the movie in which no babies are born
  26. 26. `
  27. 27. Motivation, not Information • Willpower • Due dates • Peer support (forums, outside groups) • Faculty encouragement (email) • Pride of accomplishment • Authenticity • Early adopter
  28. 28. Timeline (successful Coursera course) #usersonsite Real Couse 100K 50K ramp up + exponential decay + due dates synchronous, evergreen, semi-synchronous, bus route
  29. 29. Learning Programming
  30. 30. Representing Programs • Bag of words • N-grams of tokens (tagged?) • Static syntax tree • Dynamic call graph
  31. 31. Representing Program Sequences HMM
  32. 32. Alpha, Beta (steady progress) significantly better than Gamma (thrash, then solution) on midterm. No significant difference on score for this assignment.
  33. 33. What is Learning?
  34. 34. • Learning results from what the student does and thinks and only from what the student does and thinks. The teacher can advance learning only by influencing what the student does to learn. - Herb Simon
  35. 35. • Problem first, then explanation (Peer) • Make students to prediction (see it is wrong) • (They didn’t like that.) • Experiments with more open-ended problems • Experiments with peer grading
  36. 36. • “Richard Hamming told me his secret: First get together the problem sets and exams that you want the students to be able to solve. Then write a book that will teach them how to solve them.” -- Hal Varian (1993) T D D
  37. 37. Do we trust Bloom?
  38. 38. 0 0.5 1 1.5 2 2.5 Drill Intelligent Tutoring System Human Tutor Expected Actual
  39. 39. 0 0.5 1 1.5 2 2.5 Drill Intelligent Tutoring System Human Tutor Expected Actual More Mastery than Tutoring
  40. 40. 0 0.5 1 1.5 2 2.5 Drill Intelligent Tutoring System Human Tutor Expected Actual N = 33 More Mastery than Tutoring
  41. 41. How do we make it one to one tutoring and mastery learning?
  42. 42. “Books will soon be obsolete in the public schools. Scholars will be instructed through the eye. It is possible to teach every branch of human knowledge with the motion picture. Our school system will be completely changed inside of ten years”- Edison, 1913
  43. 43. “Lectures were once useful, but now, when all can read, and books so numerous, lectures are unnecessary. If your attention fails, and you miss part of a lecture, it is lost; you cannot go back as you do upon a book.” - 1791
  44. 44. Many Types of Individualization • Student control over rewind (multiplay) • Student choice of where to go next (multipath) • Flipped classroom (Koller, etc.) • Peer instruction (Mazur) • Peer feedback • Forums and other one-to-one interactions • Machine learning? • Courseware Engineering?
  45. 45. Open-Ended Work Peer Grade Self Grade 0 10 20 30 40 50 60 70 80 90 100 0 10 20 30 40 50 60 70 80 90 100 Teacher Grade StudentGrade “The Impact of Self-and Peer-Grading on Student Learning”. P. Sadler, E. Good. Educational Assessment (2006).
  46. 46. 20 bits per year 192K bits per sec
  47. 47. Data Threshold?
  48. 48. Modern sequencing methods (11 min) Genomic economics (8 min) Personalized medicine (13 min) History: The Human Genome Project (12 min) Modern GenomicsGenes and alleles (10 min) Basic Genetics Refresher (OPTIONAL) Mendelian inheritance (13 min) What is DNA? (12 min) Genetic testing in the commercial world (11 min) Protecting privacy (9 minutes) Case study: direct- to-consumer genetics (12 min) Case study: family planning (13 min) Commercial Genomics (OPTIONAL) Authored Multipath Sequence
  49. 49. Courseware Engineering?
  50. 50. One on One Tutoring; Mastery Learning Social; Motivation Large-Scale Courseware Engineering Machine Learning; Continual Improvement

×