A talk on choosing coding questions for technical interviews, from when I worked in the Engineering Excellence team at Microsoft in 2006.

1. Choosing Technical Interview
Questions
Adam Barr
Development Engineering Excellence
November 14, 2006

2. Microsoft Engineering Excellence
Introductory Quiz
“Why are manhole covers round?”
Was this:
A. A question Microsoft used when evaluating college
graduates for full-time employment at Microsoft
B. A topic that was discussed in my son’s second-grade class

3. Microsoft Engineering Excellence
Introductory Quiz #2
“Write the code for Kanji backspace”
Was this:
A. A question asked of college seniors to determine if they
should be flown back to Redmond for further interviews
B. The final technical challenge given to a candidate with
industry experience to determine if they should be hired at
the senior (level 63-64) or principal (level 65-67) level

4. Microsoft Engineering Excellence
About this presentation
Ask great questions to
assess the candidate
Behaviora
l
Situational Role play
Probing Technical
Self-
Appraisal
Microsoft interview guidance
1. Have an interview strategy
2. Be independent of others
3. Don’t ask illegal questions
4. Ask great questions to assess the candidate
5. End with a positive feeling
6. No tentative Hire/No Hires
7. Prompt interview feedback
Technical
Write
code
Arch.
Design
Other

5. Microsoft Engineering Excellence
What this talk will cover
What candidates know before the interview
Advice on choosing questions
Various miscellaneous topics
Not everything has a clear answer
Progress through the talk
A related story

6. Microsoft Engineering Excellence
What do candidates know? Books
There are
12 billiard
balls
Count
in base
-2
How
many
pay
phones
Design
a spice
rack Four travelers
come to a bridge at
night
Insert an
element
into a
linked
list
Two
fuses
and a
lighter

7. Microsoft Engineering Excellence
What do candidates know? Websites
http://www.sellsbrothers.com/fun/msiview/
http://www.emicrosoftinterview.com
http://www.softwareinterview.com/
http://voltinsider.com/
http://www.coolinterview.com/type.asp?iType=3
http://www.techinterview.org/
http://en.wikipedia.org/wiki/Microsoft_interview

8. Microsoft Engineering Excellence
Internal advice (part 1)
“I often like to ask candidates a question
like ‘how are virtual functions dispatched
by the C++ compiler’”
“Some things can be learned…including
languages”

9. Microsoft Engineering Excellence
Internal advice (part 2)
“Choose two or three real problems you or
your group have worked on over the past
18 months”
“Almost all of the questions I ask I have
myself been asked in interviews or mock-
interviews; try to remember how you did
and where your instincts took you in order
to empathize with the candidate.”

10. Microsoft Engineering Excellence
Internal advice (part 3)
“All of the best software engineers I know have
at least a list of [personal coding] projects in
their heads that they would like to do ‘some
day when they have time’”
“It doesn’t bother me if a candidate doesn’t
write code in their spare time at home. I’m
interviewing them for what they will do at
work.”

11. Microsoft Engineering Excellence
Internal advice (part 4)
“Avoid instantaneous judgments”
“Your gut instinct is often right – figure out
why”

12. Microsoft Engineering Excellence
Internal advice (part 5)
“It is okay to include ambiguous information in
problems as long as you tell candidates that
they should ask clarifying questions in return.
Your question shouldn't be entirely based on
whether or not the candidate asks the right
clarifying questions”
“I ask the question vaguely to prompt
clarification questions. A good candidate
won’t “dive-in” and start coding”

13. Microsoft Engineering Excellence
Internal advice (part 6)
“I rotate among the routines I ask for the test,
but don’t concern myself with whether or not
people have seen them before.”
“Discard problems that are over two or three
years old. There are always new problems
you encounter, so rotate old problems out. It
makes it…much harder for candidates to read
about the answers on the internet.”

14. Microsoft Engineering Excellence
My advice
“An interview is not about showing off how smart you are”

15. Microsoft Engineering Excellence
A little HPT theory
Human Performance Technology is a set of principles about
how individuals and organizations can achieve results
In Engineering Excellence, we use the 6 boxes model:
Information Resources Incentives
Environment
Individual
Expectations Tools Compensation
Knowledge Capacity Motivation
The “Individual” aspects are what the candidate
could provide once they are working for Microsoft

16. Microsoft Engineering Excellence
Don’t worry so much about
knowledge…
Most things can be learned on the job
Domain expertise (e.g. SQL)
Language syntax
Admittedly, some basic concepts like arrays,
linked lists, loops can reasonably be
expected…
But what about recursion? Or O(n) analysis?
Or pointers? Or bit manipulation? Or two’s
complement representation?

17. Microsoft Engineering Excellence
…focus on capacity and motivation
Capacity determines their ability to acquire knowledge, to
grow skills, and ultimately to perform and produce
Motivation determines their desire to do all of that
These are what the candidate brings with them
They usually cannot be learned on the job

18. Microsoft Engineering Excellence
Capacity and motivation
"Creative thinkers make many false starts, and continually waver
between unmanageable fantasies and systematic attack.“
Harry Hepner, psychologist and personnel researcher
"To deal effectively with puzzles (and with the bigger problems for
which they may be a model), you must operate on two or more
levels simultaneously. One thread of consciousness tackles the
problem while another, higher-level thread monitors the progress.
You need to keep asking yourself 'Is this approach working? How
much time have I spent on this approach, and how likely is it to
produce an answer soon? Is there something else I should be
trying?'“
William Poundstone, How Would You Move Mount Fuji?

19. Microsoft Engineering Excellence
Attributes of a good technical
question
Solution
Explanation
Non-trivial
Algorithm
Testable
Expandable
(And, it has to be small enough that the explanation, solution,
expansion, etc. can be done in the time allotted)

20. Microsoft Engineering Excellence
Solution
The solution offers choices, and is something the candidate
can work out incrementally (not “Aha!” they get or don’t)
You have to be open to multiple solutions

21. Microsoft Engineering Excellence
Explanation
The program should require some explanation on your part (to
see if they have the capacity to learn it), but not too much
Be careful of questions if the candidate doesn’t know
something basic to solving it (such as pointers)

22. Microsoft Engineering Excellence
Non-trivial
Question has to present a reasonable challenge to solve and
explain
Wisdom for coding questions is it should require 3 or more
variables

23. Microsoft Engineering Excellence
Algorithm
Something they can explain
Can generate questions about “what is true at different points”

24. Microsoft Engineering Excellence
Testable
Offers an opportunity to talk about what unit tests they would
write, and what complete tests
Ideally could discuss possible security concerns

25. Microsoft Engineering Excellence
Expandable
Can follow up by introducing new constraints, discussing
optimizations, scaling up, etc.
This is the ultimate guard against “The candidate already
knew the question”

26. Microsoft Engineering Excellence
Attributes of a bad technical question
It depends on an “Aha!” moment or has a single solution
Problem is too trivial or too complicated to explain
It isn’t complicated enough
Can’t discuss the algorithm
Doesn’t lead to a test discussion
No opportunity to expand the problem

27. Microsoft Engineering Excellence
Analyzing a technical question
This is where the “real” interview happens
You have to follow up on their answer
Even if they didn’t finish
First step: Be quiet
Let the candidate solve it in peace
Politely ask them to think out loud
Watching what they do is very valuable
If you see a problem, wait to see if they catch it first
Step two: understand their answer
For coding question, understand what the code does
A language you aren’t familiar with is fine as long as
you can understand their code
If you need more time, ask for it
|X|

28. Microsoft Engineering Excellence
Analyzing a technical question (part 2)
Ask candidate to explain it
Ask a question to probe their understanding
Ask if they are confident their solution is correct
If there is a bug they missed, ask increasingly obvious
questions about it
For coding questions, optionally ask about proper syntax
Can even type it in to your machine and compile/run
Remember, your goal is not to embarrass the candidate
Now expand further (optimize, constrain, test, security, etc)
Special note: If they call a library function to do all the work,
that’s cute, but it shows knowledge, not capacity/motivation

29. Microsoft Engineering Excellence
Sample technical questions
Sort an array
Pretty good, but easy to explain
Strlen(), strcmp()
Not non-trivial; algorithm usually obvious
Remove duplicates from an array
Not too difficult, but no obvious weakness
Linked list manipulation
Algorithm may be too easy
Convert day in year to month and day
Hard to expand beyond basic problem
Draw a line between two points
May need too much explanation
Etc…
Solution
More than one!
Explanation
Some, not too much
Non-trivial
3 variables?
Algorithm
Requires
explanation
Testable
Unit, complete
Expandable
Constrain, optimize

30. Microsoft Engineering Excellence
What about estimation questions?
“To spot these talents, Executive Vice-President Steven A.
Ballmer came up with a technique that has become legend:
seemingly off-the-wall, brain-teaser questions. John Neilson,
regional general manager of Microsoft's New York office,
recalls his first sampling. He and Ballmer were jogging in
Central Park four years ago when Ballmer turned and said:
‘How many gas stations do you think there are in the United
States?’ Neilson spent 20 minutes puzzling it out. Now he
uses the question on job candidates”
- Business Week, February 24, 1992

31. Microsoft Engineering Excellence
What about “design” questions?
Design == “design an elevator/airport/house/venetian blind”
They are OK if you know what you are looking for
Hint: Capacity and motivation
Don’t play the “clarification” game

32. Microsoft Engineering Excellence
What about brainteasers, after all?

33. Microsoft Engineering Excellence
Two things you should not think
“I was hired, so the system must work”
“I was hired, so I must be a good interviewer”

34. Microsoft Engineering Excellence
Special note about experienced
candidates
“A couple days ago I did 'the interview loop' at that leading
online retailer. Over the course of six hours I was repeatedly
introduced to a guy in his early twenties, who would then
ask me to write out code on a white-board for a problem that
you might find in the study guide for a 200-level computer
science class. I have 20 years of experience in
programming and systems design. And in several cases the
interviewers were vague, semantically incorrect, or self-
contradictory.”
- Asked on Slashdot, August 25, 2006

35. Microsoft Engineering Excellence
Experienced candidate opinions
“All your questions assume a certain mentality that's probably
similar to your own mentality. Are you sure you're selecting
for quality programmers, and not just people that think like
yourself?”
“I am interviewing you, too, and I expect you to know your
stuff. I would not be here if I wasn't interested in your
business, but I am confident of my own abilities, including
my ability to find another job quickly if yours isn't up to
scratch”
- Comments on Slashdot article

36. Microsoft Engineering Excellence
Do you define the problem clearly or
not?
This is an “Aha!” trick
Too many ways to claim they missed part of the definition
E.g. “Well did you consider that this is a multi-threaded application so
you need to guard the array with a lock…”
My advice: define it clearly
Or remind them to ask for clarification

37. Microsoft Engineering Excellence
Should you ask everybody the same
question?
Allows better comparisons
Gets away from gut instinct
Reduces “spinning” of results
Remember: Have questions that work even if
the candidate knows them beforehand
Related topic: Should you choose a question
that is drawn from your current work?
Answer: It depends…
In either case, you will probably know more about the
question than the candidate

38. Microsoft Engineering Excellence
Final advice: practice
Sign up to do interviews
Read other people’s feedback
Do on-campus interviews
Some resources:
Channel 9 mock whiteboarding session
HRWeb sample technical interview
James Rodrigues “The Mathematics of Hiring” ThinkWeek paper

39. Microsoft Engineering Excellence
Call to action
Interview for capacity and motivation, not knowledge
Think about SENATE when choosing questions
(And feel free to suggest a new mnemonic!)
Be methodical in analyzing answers to questions
Practice
Estimation and design and brainteaser questions…
…you’ve heard my opinion, form your own opinion
Handling experienced candidates and asking unclear
questions and using the same question…
…you’ve heard my opinion, form your own opinion

40. Questions?

41. Microsoft Engineering Excellence
Resources used
“Technical Interviewing” slide deck by Leonardo Blanco, Sujal
Parikh, and Paul David
“Marc’s Interviewing Guide” by Marc Miller
Emails on interviewing from Tomasz Kasperkiewicz
Eric Brechner’s Hardcode column “Out of the interview loop”
“Office Interview Training” slide deck by Carl Tostevin
HRWeb “Types of Interview Questions” section of “Interviewer
Toolkit” site
“Campus Recruiting on a 30-minute Schedule” by Jeff Johnson
“Interviewing Tips and Tricks” by Altaf Gilani and Alvaro Peon
“Security Interview Questions” by Vikas Ahuja
James Hamilton Interviewing with Insight blog post
Malcolm Gladwell “The New-Boy Network” article