Dealing with negative results
Research methods and proposal writing
Dr M Benini
Università degli Studi dell’Insubria
kindly hosted by Dr L Longo
marco.benini@uninsubria.it
March 1st, 2016

Science is positive
The vast majority of scientific results are stated as answers to
questions of the kind
why
how
The first question asks for an explanation of some phenomenon;
the second one requires to provide a justified method to achieve
some result of interest
( 2 of 19 )

Science is positive
Consider the negated questions:
why not
how not
The first question requires to provide a way to falsify an
explanation; the second question asks to show a faulty way to
achieve a result, and to put in evidence its failure
As a matter of fact, we have a huge number of negative answers
in our experience, and we need them!
( 3 of 19 )

Science is positive
Consider the negated questions:
why not
how not
The first question requires to provide a way to falsify an
explanation; the second question asks to show a faulty way to
achieve a result, and to put in evidence its failure
As a matter of fact, we have a huge number of negative answers
in our experience, and we need them!
( 3 of 19 )

Science is positive
When we do research, we collect many negative answers:
when an experiment fails, we have to understand why it did
not succeed
when a (mathematical) proof fails to be found, we have to ask
ourselves what prevents it, eventually searching for a
counterexample
when a procedure fails to achieve the desired result, we have
to understand what went wrong, and why
( 4 of 19 )

Science is positive
When we do research, we collect many negative answers:
when an experiment fails, we have to understand why it did
not succeed
when a (mathematical) proof fails to be found, we have to ask
ourselves what prevents it, eventually searching for a
counterexample
when a procedure fails to achieve the desired result, we have
to understand what went wrong, and why
( 4 of 19 )

Science is positive
When we do research, we collect many negative answers:
when an experiment fails, we have to understand why it did
not succeed
when a (mathematical) proof fails to be found, we have to ask
ourselves what prevents it, eventually searching for a
counterexample
when a procedure fails to achieve the desired result, we have
to understand what went wrong, and why
( 4 of 19 )

Science is positive
When we do research, we collect many negative answers:
when an experiment fails, we have to understand why it did
not succeed
when a (mathematical) proof fails to be found, we have to ask
ourselves what prevents it, eventually searching for a
counterexample
when a procedure fails to achieve the desired result, we have
to understand what went wrong, and why
( 4 of 19 )

Science is positive
When we reach the final stage of our research efforts, publishing
the results, we focus on the positive achievements, And we
discard the negative experience we have accumulated
An experienced researcher, instead, should be more effective,
trying to use the negative results and to maximise the outcomes
of his efforts
( 5 of 19 )

Science is positive
When we reach the final stage of our research efforts, publishing
the results, we focus on the positive achievements, And we
discard the negative experience we have accumulated
An experienced researcher, instead, should be more effective,
trying to use the negative results and to maximise the outcomes
of his efforts
( 5 of 19 )

Negative content, positively presented
Although some results we have found are negative, they can
often be positively presented
when a procedure fails, precisely delimit the domain of
applicability, excluding the cases we know are bad: the limit
becomes a necessary condition, and the failure, a
counterexample to a direct generalisation
when a mathematical proof fails, find a counterexample, and
use it to add a condition that prevents its applicability: the
additional hypothesis becomes necessary because of the
counterexample
( 6 of 19 )

Negative content, positively presented
Although some results we have found are negative, they can
often be positively presented
when a procedure fails, precisely delimit the domain of
applicability, excluding the cases we know are bad: the limit
becomes a necessary condition, and the failure, a
counterexample to a direct generalisation
when a mathematical proof fails, find a counterexample, and
use it to add a condition that prevents its applicability: the
additional hypothesis becomes necessary because of the
counterexample
( 6 of 19 )

Negative content, positively presented
Although some results we have found are negative, they can
often be positively presented
when a procedure fails, precisely delimit the domain of
applicability, excluding the cases we know are bad: the limit
becomes a necessary condition, and the failure, a
counterexample to a direct generalisation
when a mathematical proof fails, find a counterexample, and
use it to add a condition that prevents its applicability: the
additional hypothesis becomes necessary because of the
counterexample
( 6 of 19 )

Negative content, positively presented
When stating a result, use a positive language
Theorem [Gödel] Peano Arithmetic is not complete.
Theorem [Gödel] There is a sentence G in Peano Arithmetic
such that both G and ¬G are not provable.
Theorem [Gödel] There is a sentence G in Peano Arithmetic
such that G is true on natural numbers, but G is not provable.
( 7 of 19 )

Negative content, positively presented
When stating a result, use a positive language
Theorem [Gödel] Peano Arithmetic is not complete.
Theorem [Gödel] There is a sentence G in Peano Arithmetic
such that both G and ¬G are not provable.
Theorem [Gödel] There is a sentence G in Peano Arithmetic
such that G is true on natural numbers, but G is not provable.
( 7 of 19 )

Negative content, positively presented
When stating a result, use a positive language
Theorem [Gödel] Peano Arithmetic is not complete.
Theorem [Gödel] There is a sentence G in Peano Arithmetic
such that both G and ¬G are not provable.
Theorem [Gödel] There is a sentence G in Peano Arithmetic
such that G is true on natural numbers, but G is not provable.
( 7 of 19 )

Negative content, positively presented
When stating a result, use a positive language
Theorem [Gödel] Peano Arithmetic is not complete.
Theorem [Gödel] There is a sentence G in Peano Arithmetic
such that both G and ¬G are not provable.
Theorem [Gödel] There is a sentence G in Peano Arithmetic
such that G is true on natural numbers, but G is not provable.
( 7 of 19 )

Negative content, positively presented
When writing a statement, provide a positive content
Theorem An arbitrary list cannot be sorted in linear time.
Theorem An arbitrary list can be sorted in O(nlogn) steps.
Theorem The heap sort algorithm sorts any list in at most
O(nlogn) steps.
( 8 of 19 )

Negative content, positively presented
When writing a statement, provide a positive content
Theorem An arbitrary list cannot be sorted in linear time.
Theorem An arbitrary list can be sorted in O(nlogn) steps.
Theorem The heap sort algorithm sorts any list in at most
O(nlogn) steps.
( 8 of 19 )

Negative content, positively presented
When writing a statement, provide a positive content
Theorem An arbitrary list cannot be sorted in linear time.
Theorem An arbitrary list can be sorted in O(nlogn) steps.
Theorem The heap sort algorithm sorts any list in at most
O(nlogn) steps.
( 8 of 19 )

Negative content, positively presented
When writing a statement, provide a positive content
Theorem An arbitrary list cannot be sorted in linear time.
Theorem An arbitrary list can be sorted in O(nlogn) steps.
Theorem The heap sort algorithm sorts any list in at most
O(nlogn) steps.
( 8 of 19 )

Negative content, positively presented
When stating a counterexample, be positive
The real-valued function f (x) = |x| does not admit a derivative in
x = 0.
The real-valued function f (x) = |x| admits f (x) = −1 as
derivative on the interval (−∞,0), and f (x) = 1 on the interval
(0,+∞). Thus, f (0) is not defined.
( 9 of 19 )

Negative content, positively presented
When stating a counterexample, be positive
The real-valued function f (x) = |x| does not admit a derivative in
x = 0.
The real-valued function f (x) = |x| admits f (x) = −1 as
derivative on the interval (−∞,0), and f (x) = 1 on the interval
(0,+∞). Thus, f (0) is not defined.
( 9 of 19 )

Negative content, positively presented
When stating a counterexample, be positive
The real-valued function f (x) = |x| does not admit a derivative in
x = 0.
The real-valued function f (x) = |x| admits f (x) = −1 as
derivative on the interval (−∞,0), and f (x) = 1 on the interval
(0,+∞). Thus, f (0) is not defined.
( 9 of 19 )

Negative content, positively presented
When stating a limit to a procedure, be positive
Using a 12,800 ISO film speed in a digital camera prevents
getting a noise-free photograph when shooting in a dark
environment.
Using a 12,800 ISO film speed in a digital camera provides a
noisy photograph when shooting in a dark environment, which
can be useful to calculate the correct exposure at a lower
sensibility, and get a noise-free image.
( 10 of 19 )

Negative content, positively presented
When stating a limit to a procedure, be positive
Using a 12,800 ISO film speed in a digital camera prevents
getting a noise-free photograph when shooting in a dark
environment.
Using a 12,800 ISO film speed in a digital camera provides a
noisy photograph when shooting in a dark environment, which
can be useful to calculate the correct exposure at a lower
sensibility, and get a noise-free image.
( 10 of 19 )

Negative content, positively presented
When stating a limit to a procedure, be positive
Using a 12,800 ISO film speed in a digital camera prevents
getting a noise-free photograph when shooting in a dark
environment.
Using a 12,800 ISO film speed in a digital camera provides a
noisy photograph when shooting in a dark environment, which
can be useful to calculate the correct exposure at a lower
sensibility, and get a noise-free image.
( 10 of 19 )

Negative content, positively presented
When stating a limit to a procedure, be positive
Using a 12,800 ISO film speed in a digital camera prevents
getting a noise-free photograph when shooting in a dark
environment.
Using a 12,800 ISO film speed in a digital camera provides a
noisy photograph when shooting in a dark environment, which
can be useful to calculate the correct exposure at a lower
sensibility, and get a noise-free image.
( 10 of 19 )

Impossibility results
Some results are inherently negative. If they are also significant,
it is worth stating them in a negative way
Theorem [Turing] Deciding whether a program terminates on
all inputs, is not a computable problem.
Theorem [Turing] There is no computer program that, taking
as input a computer program, decides whether it terminates on
all inputs.
( 11 of 19 )

Impossibility results
Some results are inherently negative. If they are also significant,
it is worth stating them in a negative way
Theorem [Turing] Deciding whether a program terminates on
all inputs, is not a computable problem.
Theorem [Turing] There is no computer program that, taking
as input a computer program, decides whether it terminates on
all inputs.
( 11 of 19 )

Impossibility results
Some results are inherently negative. If they are also significant,
it is worth stating them in a negative way
Theorem [Turing] Deciding whether a program terminates on
all inputs, is not a computable problem.
Theorem [Turing] There is no computer program that, taking
as input a computer program, decides whether it terminates on
all inputs.
( 11 of 19 )

Dead ends
When we do research, experiments fail, proofs do not come
through, and procedures give crazy results
We use these failures to better understand our problems, and
eventually to fix what did not work properly
Sometimes, we get in a situation where there seem to be no way
out, in which we are unable to find a way to fix the incorrect
behaviours we encounter. We call these situation dead ends
( 12 of 19 )

Dead ends
When we do research, experiments fail, proofs do not come
through, and procedures give crazy results
We use these failures to better understand our problems, and
eventually to fix what did not work properly
Sometimes, we get in a situation where there seem to be no way
out, in which we are unable to find a way to fix the incorrect
behaviours we encounter. We call these situation dead ends
( 12 of 19 )

Dead ends
When you get into a dead end, you suffer from a lack of
alternatives. There are many ways to try to get out
Write down what you got as clearly as possible. The effect to
organise the knowledge collected so far often shows
alternatives
Show your attempts to a colleague or to your supervisor. A
fresh look often provides a new point of view
Leave the problem apart for a while. When we do not think
consciously to a problem, our mind follows different paths, and
we leave room for new intuitions
( 13 of 19 )

Dead ends
When you get into a dead end, you suffer from a lack of
alternatives. There are many ways to try to get out
Write down what you got as clearly as possible. The effect to
organise the knowledge collected so far often shows
alternatives
Show your attempts to a colleague or to your supervisor. A
fresh look often provides a new point of view
Leave the problem apart for a while. When we do not think
consciously to a problem, our mind follows different paths, and
we leave room for new intuitions
( 13 of 19 )

Dead ends
When you get into a dead end, you suffer from a lack of
alternatives. There are many ways to try to get out
Write down what you got as clearly as possible. The effect to
organise the knowledge collected so far often shows
alternatives
Show your attempts to a colleague or to your supervisor. A
fresh look often provides a new point of view
Leave the problem apart for a while. When we do not think
consciously to a problem, our mind follows different paths, and
we leave room for new intuitions
( 13 of 19 )

Dead ends
When you get into a dead end, you suffer from a lack of
alternatives. There are many ways to try to get out
Write down what you got as clearly as possible. The effect to
organise the knowledge collected so far often shows
alternatives
Show your attempts to a colleague or to your supervisor. A
fresh look often provides a new point of view
Leave the problem apart for a while. When we do not think
consciously to a problem, our mind follows different paths, and
we leave room for new intuitions
( 13 of 19 )

Dead ends
It is tempting to get some result out of a long period of hard
work. But, when the result is a dead end, do not publish it!
A dead end leads to nowhere, by definition. You have few
chances to motivate your partial achievements
You will spend time writing, submitting the paper, and receiving
comments. You know in advance that the results are not
complete, and that the direction leads nowhere: honestly, any
reviewer will recognise these weaknesses, lowering your chances.
Are you sure you are not wasting more time?
( 14 of 19 )

Dead ends
It is tempting to get some result out of a long period of hard
work. But, when the result is a dead end, do not publish it!
A dead end leads to nowhere, by definition. You have few
chances to motivate your partial achievements
You will spend time writing, submitting the paper, and receiving
comments. You know in advance that the results are not
complete, and that the direction leads nowhere: honestly, any
reviewer will recognise these weaknesses, lowering your chances.
Are you sure you are not wasting more time?
( 14 of 19 )

Dead ends
It is tempting to get some result out of a long period of hard
work. But, when the result is a dead end, do not publish it!
A dead end leads to nowhere, by definition. You have few
chances to motivate your partial achievements
You will spend time writing, submitting the paper, and receiving
comments. You know in advance that the results are not
complete, and that the direction leads nowhere: honestly, any
reviewer will recognise these weaknesses, lowering your chances.
Are you sure you are not wasting more time?
( 14 of 19 )

Publishing or discarding?
A basic rule in Economy says to buy when the market is low, and
to sell when the market is high
Consider pursuing a scientific result as an investment. Publishing
is the act which provides you with a revenue. It makes no sense
to submit an article when you know in advance that it will not
get published. You spend time and effort with no revenue
( 15 of 19 )

Publishing or discarding?
A positive result has bigger chances to get published than a
negative one. But negative results have some chances, too
Try to positively present your negative outcomes
Strong impossibility results are landmarks. Prepare yourself to
defend them, but be sure they will be important in the long
term
Weak impossibility results are interesting only for specialists.
You should carefully evaluate if your audience is wide enough
to invest time and effort to try to publish such a negative
achievement
( 16 of 19 )

Publishing or discarding?
A positive result has bigger chances to get published than a
negative one. But negative results have some chances, too
Try to positively present your negative outcomes
Strong impossibility results are landmarks. Prepare yourself to
defend them, but be sure they will be important in the long
term
Weak impossibility results are interesting only for specialists.
You should carefully evaluate if your audience is wide enough
to invest time and effort to try to publish such a negative
achievement
( 16 of 19 )

Publishing or discarding?
A positive result has bigger chances to get published than a
negative one. But negative results have some chances, too
Try to positively present your negative outcomes
Strong impossibility results are landmarks. Prepare yourself to
defend them, but be sure they will be important in the long
term
Weak impossibility results are interesting only for specialists.
You should carefully evaluate if your audience is wide enough
to invest time and effort to try to publish such a negative
achievement
( 16 of 19 )

Publishing or discarding?
A positive result has bigger chances to get published than a
negative one. But negative results have some chances, too
Try to positively present your negative outcomes
Strong impossibility results are landmarks. Prepare yourself to
defend them, but be sure they will be important in the long
term
Weak impossibility results are interesting only for specialists.
You should carefully evaluate if your audience is wide enough
to invest time and effort to try to publish such a negative
achievement
( 16 of 19 )

Publishing or discarding?
Dead ends are the result of our efforts. What matters in Science
are our findings, not the effort you made
You should carefully judge what parts of your efforts are
significant, and what are irrelevant to other people, with no
regards for the time you invested
Do not try to publish what you think is irrelevant! If you think
so, how could you possibly convince a reader? You have to
believe in the value of your results before trying to explain them
( 17 of 19 )

Experience
Get into the habit of writing down, as clearly as possible,
everything you find during your research
you can reuse this pieces when writing the final paper
you can use them to help a new collaborator to enter in your
team
you have an evidence of your work
you can use them to illustrate the point you reached to, e.g., a
colleague or to your supervisor
you keep track of what you have understood. In the future,
you may want to reuse these partial results elsewhere, and you
do not want to rework all the details from scratch
( 18 of 19 )

Experience
Get into the habit of writing down, as clearly as possible,
everything you find during your research
you can reuse this pieces when writing the final paper
you can use them to help a new collaborator to enter in your
team
you have an evidence of your work
you can use them to illustrate the point you reached to, e.g., a
colleague or to your supervisor
you keep track of what you have understood. In the future,
you may want to reuse these partial results elsewhere, and you
do not want to rework all the details from scratch
( 18 of 19 )

Experience
Get into the habit of writing down, as clearly as possible,
everything you find during your research
you can reuse this pieces when writing the final paper
you can use them to help a new collaborator to enter in your
team
you have an evidence of your work
you can use them to illustrate the point you reached to, e.g., a
colleague or to your supervisor
you keep track of what you have understood. In the future,
you may want to reuse these partial results elsewhere, and you
do not want to rework all the details from scratch
( 18 of 19 )

Experience
Get into the habit of writing down, as clearly as possible,
everything you find during your research
you can reuse this pieces when writing the final paper
you can use them to help a new collaborator to enter in your
team
you have an evidence of your work
you can use them to illustrate the point you reached to, e.g., a
colleague or to your supervisor
you keep track of what you have understood. In the future,
you may want to reuse these partial results elsewhere, and you
do not want to rework all the details from scratch
( 18 of 19 )

Experience
Get into the habit of writing down, as clearly as possible,
everything you find during your research
you can reuse this pieces when writing the final paper
you can use them to help a new collaborator to enter in your
team
you have an evidence of your work
you can use them to illustrate the point you reached to, e.g., a
colleague or to your supervisor
you keep track of what you have understood. In the future,
you may want to reuse these partial results elsewhere, and you
do not want to rework all the details from scratch
( 18 of 19 )

Experience
Get into the habit of writing down, as clearly as possible,
everything you find during your research
you can reuse this pieces when writing the final paper
you can use them to help a new collaborator to enter in your
team
you have an evidence of your work
you can use them to illustrate the point you reached to, e.g., a
colleague or to your supervisor
you keep track of what you have understood. In the future,
you may want to reuse these partial results elsewhere, and you
do not want to rework all the details from scratch
( 18 of 19 )

Experience
In the long term, negative results will form the fundamental part
of your experience
you will know what directions and tools have few hopes to
lead to a result
you have a track of open problems, not-yet-attempted
alternatives, variants to explore
negative results, well documented, can speed up your
achievements when you hear about a new result that
overcomes a difficulty which blocked you in the past
All these aspects of experience qualify you as an expert
( 19 of 19 )

Experience
In the long term, negative results will form the fundamental part
of your experience
you will know what directions and tools have few hopes to
lead to a result
you have a track of open problems, not-yet-attempted
alternatives, variants to explore
negative results, well documented, can speed up your
achievements when you hear about a new result that
overcomes a difficulty which blocked you in the past
All these aspects of experience qualify you as an expert
( 19 of 19 )

Experience
In the long term, negative results will form the fundamental part
of your experience
you will know what directions and tools have few hopes to
lead to a result
you have a track of open problems, not-yet-attempted
alternatives, variants to explore
negative results, well documented, can speed up your
achievements when you hear about a new result that
overcomes a difficulty which blocked you in the past
All these aspects of experience qualify you as an expert
( 19 of 19 )

Experience
In the long term, negative results will form the fundamental part
of your experience
you will know what directions and tools have few hopes to
lead to a result
you have a track of open problems, not-yet-attempted
alternatives, variants to explore
negative results, well documented, can speed up your
achievements when you hear about a new result that
overcomes a difficulty which blocked you in the past
All these aspects of experience qualify you as an expert
( 19 of 19 )

Experience
In the long term, negative results will form the fundamental part
of your experience
you will know what directions and tools have few hopes to
lead to a result
you have a track of open problems, not-yet-attempted
alternatives, variants to explore
negative results, well documented, can speed up your
achievements when you hear about a new result that
overcomes a difficulty which blocked you in the past
All these aspects of experience qualify you as an expert
( 19 of 19 )