The document presents an approach called SPoC that uses search-based techniques to synthesize code from pseudocode. SPoC treats each line of pseudocode as a discrete portion of the program and uses a translation model and best-first search to map pseudocode to code. Experimental results show the approach achieves a maximum success rate of 55.2% on one test set and 71.4% on another. The approach also uses error localization techniques to improve results by identifying likely error locations when programs fail to compile.

1. SPoC: Search-based
Pseudocode to Code
• Sumith Kulal
• Panupong Pasupat
• Kartik Chandra
• Mina Lee
• Oded Padon
• Alex Aiken
• Percy Liang

2. Presented By
Minhazul Arefin

3. Abstract
 mapping pseudocode to long programs that are functionally correct
 treat the translation of each pseudocode line as a discrete portion of
the program
 If program fails to compile, an error localization method tries to
identify the portion of the program responsible for the failure
3

4. Introduction
 map natural language descriptions
 Perform non-trivial computations
 input-output pairs usually give little information about the
intermediate states of the program
4

5. Data collection
 Programs and test cases: NAPS dataset
 Decomposition: decompose each program into code lines
 Pseudocode: recruited 59 workers on Amazon Mechanical Turk to
write pseudocode
 Statistics: dataset contains 18,356 programs
 Training and test sets: they created two test sets
5

6. Problem Statement
• The system is given (a) a sequence x of L pseudocode lines 𝑥1; 𝑥2; : : : ; 𝑥𝐿, where
 𝑥𝑖 is a string with indentation level `I
 K public test cases in the form of
input-output string pairs (𝑇𝑖𝑛1 ; 𝑇𝑜𝑢𝑡1);
: : : ; (𝑇𝑖𝑛𝑘 ; 𝑇𝑜𝑢𝑡𝑘).
▸ The task is to synthesize a program y consisting of L code lines 𝑦1; 𝑦1; : : : ; 𝑦𝐿
▸ The program is accepted if it successfully compiles and passes all public test cases
6

7. Base approach (Translation)
• They used a standard seq2seq translation model with an LSTM
encoder and decoder
• An attention-based copying mechanism
• A coverage vector
7

8. Base approach (Best-first search)
1. Create a priority Queue pqueue.
2. insert ‘start’ in pqueue : pqueue.insert(start)
3. delete all elements of pqueue one by one.
1) if, the element is goal . Exit.
2) else, traverse neighbor’s and mark the node examined.
4. End.
8

9. Base approach (Best-first search)
Advantages:
▸ Best first search can switch between BFS and DFS by gaining the
advantages of both the algorithms.
▸ This algorithm is more efficient than BFS and DFS algorithms.
Disadvantages:
▸ It can behave as an unguided depth-first search in the worst case
scenario.
▸ It can get stuck in a loop as DFS.
▸ This algorithm is not optimal.
9

10. Experiments (Translation accuracy)
 Used BELU score in best first search
 only 18.2% of programs in TESTP and 32.0% of programs in TESTW is
correct in every line.
 the top candidate of each line have an even lower success rates of
17.8% on TESTP and 30.7% on TESTW
10

11. Experiments (Translation accuracy)
11
Figure: (a) While the translation accuracy is high at the line level, we need to consider the
result at the program level. For each program, we count the number of lines i where
(b) the top candidate ci1 is incorrect
(c) none of the candidates 𝑐𝑖𝑗 ꞓ 𝑐𝑖 is correct.

12. Experiments (Oracle success rate)
▸ count the number of lines i where the candidate list 𝑐𝑖 does not have
any correct candidate
▸ 44.8% of programs in TESTP and 28.6% of programs in TESTW have
least one difficult line where the translation model does not produce
a correct prediction
▸ It achieve a maximum success rate of 55.2% on TESTP and 71.4% on
TESTW
12

13. Experiments (Synthesis results)
 Error line, 𝑖∗ = 𝑖𝑒𝑟𝑟 - ∆i
 Where,
 𝑖∗
= predicted line
 𝑖𝑒𝑟𝑟 = error in line
 ∆i = corresponding line
13

14. Experiments (Synthesis results)
14
Figure: Success rates at budgets B of best-first search with different error localization methods

15. Experiments (Synthesis results)
15
Table: Effects of using error localization methods on all test examples

16. Experiments (Error analysis)
▸ s is half = “s == "half“
▸ interpreted as “s / 2 == 0”
or “s % 2 == 0”
▸ This causes the search to ultimately fail whereas best-first search finds a
correct program in 80 search iterations
16

17. Related work (Program synthesis)
▸ formulate synthesis as a constraint satisfaction problem
▸ requires that the synthesis problem can be translated to a theory
with effective constraint solvers
▸ brute force enumeration of programs works surprisingly well
▸ when the search space is too large for enumeration, randomized
search guided by a cost function can be effective
17

18. Related work (Semantic parsing)
 It is a task of converting a NLP text into a logical form (a machine-
understandable format)
 One of its traditional tasks is to parse a given into an executable
database query
 Instead of a single query, some work aims to parse a sequence of
queries that can be sequentially executed (max 5 sentences)
18

19. Related work (Error localization)
 uses neural models to localize errors focused on localizing and
correcting syntax errors
 semantic errors such as variable misuse and variable replace
 Their work identifies error locations by interpreting compiler error
messages
19