The document discusses learning programming through MOOCs and machine learning. It provides data on a MOOC with over 160,000 students from 209 countries. It analyzes student error messages, submissions, and interactions to improve programming instructions. However, programming languages can be ambiguous and students struggle with different concepts. The document advocates for mastery learning through one-on-one tutoring and continual course improvements using data and machine learning.

1. Learning Programming
(by Humans
by Machine)
Peter Norvig

2. Ng Koller
Norvig Thrun
MOOCs
and
Machine
Learning

4. For when you don’t
know what’s going on

7. Hal Abelson
(SIGCSE Outstanding Contribution Award, 2012)

9. The average under tutoring [with mastery learning]
was about two standard deviations above the
average of the control. (Benjamin Bloom, 1984)

24. Python llll lll
C ll
Scheme l
But: for “vocational”, not “academic”:
Javascript dominates

25. Analyzing Intro CS MOOC Data 101
Student Name Final Grade
Peter Norvig

26. Analyzing Intro CS MOOC Data 101
Student Name Final Grade
Peter Norvig Incomplete

28. 160,000 students; 209 countries

29. Data Sources
• Enrollment / Dropout
• Quizzes and Exams
• Video Interactions
• Discussion Forums
• Program submissions

30. Syntax Error Messages Vague
if x = 1:
^
SyntaxError: invalid syntax
SyntaxError: Use “==“ for comparison,
use “=“ for assignment.

31. Inputs to Compiler Error Routine
Source
Code
Compiler
Error
Message
Ignored:
- Past program submission histories
- Problem Description
- Comments
- Test Cases

32. CFG:
Python
Non-Python

34. PCFG:
Python-ish
Probably Approximately Correct
Grammar and Intent

35. Match Programs to Corpus
S1 suggest
C2C1Corpus:
Submission:

36. Syntax Error Messages Vague
if spam == 42
print('Hello!')
^
SyntaxError: invalid syntax
^
IndentationError: unexpected indent
SyntaxError: Colon required,
use “if spam == 42:“

37. Attractive Nuisance
>>> table = [[]] * 5
>>> table
[[], [], [], [], []]
>>> def add(key, table):
table[hash(key) % len(table)].append(key)
>>> add(’x', table)
>>> table
[[’x'], [’x'], [’x'], [’x'], [’x']]
>>> table = listof(5, list)
def listof(n, kind):
"Make a list of n elements, each kind()."
return [kind() for _ in range(n)]

38. Faulty Student Models
• Given:
• nums = [1, 2, -3]
• max(nums) == 2
• max(nums, key=abs) == -3
• len(‘word’) == 4
• wordlist = file(‘words’).read().split()
• What is the longest word in wordlist?
• ?
• ??
• max(wordlist, key=len)

39. DWIM Attractive Nuisance
• def qsort(A):
if len(A) <= 1: return A
LT, EQ, GT = []
p = A[len(A)/2]
for x in A:
bin = (LT if x<p else GT if x>p else EQ)
bin.append(x)
return qsort(LT) + EQ + qsort(GT)
LT = EQ = GT = []
LT, EQ, GT = [], [], []

40. DWIM Attractive Nuisance
• spam = ['cat', 'dog', 'mouse']
• for i in range(spam):
print(spam[i])
• for i in len(spam):
print(spam[i])
• for i in range(len(spam)):
print(spam[i])
for x in spam:
print(x)
• for i in range(len(spam)):
spam[i] = spam[i].upper()
for x in spam:
x = x.upper()

41. Language Too General
y = (x + 1) (x - 1)
TypeError: unsupported operand
type(s) for +: 'object' and
'int'
SyntaxError: ‘*’ operator required.
use “(x + 1) * (x – 1)”

42. Multi-Language Confusion
File ”example.py”, line 1
root = new Node()
^
SyntaxError: invalid syntax
Test.java:6: cannot find symbol
symbol : class awt
location: package java
java.awt.Color(r, g, b);
^
Error: use “Node()” in Python;
use “new Color(…)” in Java

43. Multi-Language Confusion
int total = 0;
for (int x : A) {
total += x;
}
total = None
for x in A:
total = total + x

44. Why do students stay or go?

46. the movie in which no babies are born

47. `

49. Motivation, not Information
• Willpower
• Due dates
• Peer support (forums, outside groups)
• Faculty encouragement (email)
• Pride of accomplishment
• Authenticity
• Early adopter

51. Timeline (successful Coursera course)
#usersonsite
Real Couse
100K
50K
ramp up + exponential decay + due dates
synchronous, evergreen,
semi-synchronous, bus route

52. Learning Programming

54. Representing Programs
• Bag of words
• N-grams of tokens (tagged?)
• Static syntax tree
• Dynamic call graph

55. Representing Program Sequences
HMM

57. Alpha, Beta (steady progress) significantly better than Gamma (thrash, then
solution) on midterm. No significant difference on score for this assignment.

58. What is Learning?

59. • Learning results from
what the student does
and thinks and only
from what the student
does and thinks. The
teacher can advance
learning only by
influencing what the
student does to
learn. -
Herb Simon

60. • Problem first, then explanation (Peer)
• Make students to prediction (see it is wrong)
• (They didn’t like that.)
• Experiments with more open-ended problems
• Experiments with peer grading

61. • “Richard Hamming told me his
secret: First get together the problem
sets and exams that you want the
students to be able to solve. Then
write a book that will teach them how
to solve them.” -- Hal Varian (1993)
T D D

62. Do we trust Bloom?

65. 0
0.5
1
1.5
2
2.5
Drill Intelligent
Tutoring
System
Human
Tutor
Expected
Actual

66. 0
0.5
1
1.5
2
2.5
Drill Intelligent
Tutoring
System
Human
Tutor
Expected
Actual
More Mastery than Tutoring

67. 0
0.5
1
1.5
2
2.5
Drill Intelligent
Tutoring
System
Human
Tutor
Expected
Actual
N = 33
More Mastery than Tutoring

68. How do we make it
one to one tutoring
and
mastery learning?

70. “Books will soon be obsolete in the public schools.
Scholars will be instructed through the eye.
It is possible to teach every branch of human knowledge
with the motion picture. Our school system will be
completely changed inside of ten years”- Edison, 1913

71. “Lectures were once useful,
but now, when all can read,
and books so numerous,
lectures are unnecessary.
If your attention fails,
and you miss part of a lecture,
it is lost; you cannot go back
as you do upon a book.”
- 1791

72. Many Types of Individualization
• Student control over rewind (multiplay)
• Student choice of where to go next (multipath)
• Flipped classroom (Koller, etc.)
• Peer instruction (Mazur)
• Peer feedback
• Forums and other one-to-one interactions
• Machine learning?
• Courseware Engineering?

73. Open-Ended Work
Peer Grade
Self Grade
0
10
20
30
40
50
60
70
80
90
100
0 10 20 30 40 50 60 70 80 90 100
Teacher Grade
StudentGrade
“The Impact of Self-and Peer-Grading on Student Learning”.
P. Sadler, E. Good. Educational Assessment (2006).

74. 20 bits per year
192K bits per sec

75. Data Threshold?

76. Modern sequencing
methods (11 min)
Genomic
economics (8
min)
Personalized
medicine (13 min)
History: The
Human Genome
Project (12 min)
Modern GenomicsGenes and alleles
(10 min)
Basic Genetics
Refresher (OPTIONAL)
Mendelian
inheritance
(13 min)
What is DNA?
(12 min)
Genetic testing in
the commercial
world (11 min)
Protecting privacy
(9 minutes)
Case study: direct-
to-consumer
genetics (12 min)
Case study:
family planning
(13 min)
Commercial Genomics
(OPTIONAL)
Authored Multipath Sequence

79. Courseware Engineering?

85. One on One Tutoring; Mastery Learning
Social;
Motivation
Large-Scale Courseware Engineering
Machine Learning; Continual Improvement