An attempt to indoctrinate graduate students with some philosophy of science and good practices in their research. Some references are included to disturbing trends known from poor practices that appear common to some fields, to make clear the importance of reliable methods, in particular the Scientific Method. Trigger warning: not trying to be nice to everybody.

1. On practical philosophy of research
in science and technology
Seppo Karrila
August 2016 (2559 Thai)

2. Executive summary
• Why do we pursue research?
• The Magic Spell for organizing (almost)
everything
• Is “science” getting sick?
• What is knowledge, what is science?
• How can you approach a given research topic?
• Included: some practical hints and tricks that help
you in your research, marked with a star as
are the otherwise most important slides.

3. Why we do research?
• Today’s research may be tomorrow’s
technology
• Teachers who don’t follow research might not
prepare you for your future!
• You do research to
– Demonstrate you know how research is done (MS)
– Demonstrate mastery of techniques (MS)
– Demonstrate mostly independent contribution to
science (PhD)

4. PhD: driver’s license to doing science
• Doing a PhD is like a test drive.
– You have support from an advisor. Your scope of
research is pre-defined. You need to come up with
some “new science” and defend your thesis.
• Once you have your PhD
– You are expected to be an independent researcher
who no longer needs an advisor!
– Don’t expect your thesis is your career. When you
have your license, you will not stay on the test track,
you need to learn about other roads and go to places.

5. Industry or academic job?
• Academia produces PhD’s who go to academia
to produce more PhD’s who go to academia…
– It could become a big club, as long as someone
pays for the fun. In the long run this is impossible,
academia feeding its own growth and not feeding
the outside world.
– Go outside to see what is done in the “real world”
with science. Maintain contact with academics,
perhaps come back to tell what you learned.

6. Reality restricts your options
• http://www.theatlantic.com/business/archive/2013/02/how-many-phds-actually-get-to-become-
college-professors/273434/

7. Here is a Magic Spell
• Issue
• Significance
• Approach
• Results
• Conclusions

8. The Magic Spell applies to many things
• A one-page project proposal
• A full project proposal
• A summary of someone else’s work
• Summarizing your project plan
• Writing a poster
• Giving a presentation
• Writing an article

9. Discussion with the Boss man
• Yes, you wanted to see me. What is this about?
– Issue: what is this about
• Well, I have other more important things…
– Significance: why this is important
• OK, but what can we do about it?
– Approach: actions to take
• So if we do all that, what do we get?
– (expected) results: products of the actions
• And what is the value of the results?
– Conclusions: effects on decisions, practices, …

10. Be prepared to see the Magic Spell
several times today
• Issue
• Significance
• Approach
• Results
• Conclusions

11. Trust in “science” is suffering
"
Ninety-seven percent of original studies
had significant results (P < .05). Thirty-
six percent of replications had
significant results…"

12. Note that this is rather sick
• If only 36% of reported “statistically
significant” results can be replicated, then
64% of the results were NOT replicated as
“significant”.
• About TWO OUT OF THREE papers in
psychological science are making claims that
you can’t trust.

13. https://www.timeshighereducation.com/news/papers-in-economics-not-reproducible

14. And it just gets curiouser and curiouser…
Why, sometimes I've believed as many as
six impossible things before breakfast.

15. Depressed?
• Has the World gone bananas?
• Did you pick Science because it is the ultimate
in fairness: right is right and wrong is wrong,
impersonally and without politicking?
• Hold on, the plague only affects some fields,
there will be healthy survivors.

16. The happy exceptions, solid healthy
science• Engineering
– The models used must be correct and reasonably accurate, otherwise
your bridge falls, your building collapses, etc.
– The products must deliver on promises!
• Math
– Accepts ONLY what has been proven beyond any doubt, in little steps
all of which are known to be correct. If a mistake is found later, the
result is labeled either “unproven hypothesis” or “conjecture”, or
actually proven false with a counterexample.
• Physics & chemistry
– Only what can be reproduced and confirmed is worth keeping.
• Common features:
– heavy reliance on numbers, logic, and equations
– strict rejection of unreliable models, or rules that “seem to work once
in a while” – attempt to record only “absolutely correct” constructs
– accountability and discipline enforced, falsehoods resolutely rejected

17. The Scientific Method

18. Note that you start with a question
and a hypothesis
• The question should have importance
– Otherwise any results will also be unimportant. You can become
an expert in unimportant matters…
– Importance usually comes from economy: productivity (yield,
processing speed, investment cost, operating cost), or product
quality.
• Naïve hypotheses are still better than nothing
– By searching through all possible choices, we can find the best
alternative. Not “deep science” but straightforward with useful
results.
• A “clever” type of hypothesis
– Two treatments or technologies have complementary strong
and weak points. Combining them could perhaps give an overall
improvement.

19. We now go back to the fundamentals
• … and discuss knowledge, understanding and
science.
• It may seem an abstract and “philosophical”
remote aspect, but if more people paid some
attention, the trustworthiness of science
would be much better than it is today.

20. Theory of knowledge
• Has been studied for a very long time, called
“epistemology”
• Such theories have been unable to reach practical
usefulness
– The question “how do we know what we know, and
do we really know?” is important
– Time travel would also be important…
– Companies are not struggling to find and hire
epistemologists

21. What is “understanding”?
• In common language it often just means to sympathize. “I feel
for you, I understand.”
• In science there are different levels of understanding
– Forming concepts and nomenclature, and general rules: an apple
falls if you let it go
– Naming the ghost behind the action: gravitation. Being able to name
the ghost gives a “feeling of understanding it”.
• Why does the apple fall – gravitation! This is nonsense and word magic,
just another name that sounds more learned than “it falls because it falls”.
– Quantitative understanding: how fast does it fall, how hard does it
hit. Can be calculated accurately with classical mechanics and ITS
EQUATIONS FOR GRAVITATION. Now we are getting somewhere!
(Some went to the moon, satellites for GPS and communications.)
– Application models based on experiments: when the apple hits the
floor, will it break, how much is it damaged, does this affect its price
or shelf-life, how should we package apples… These are clearly
things you can’t calculate accurately from classical mechanics, you
have to do experiments AND MAKE MODELS.

22. Stages of development
• A “young” scientific discipline
– Names and documents things
– Creates taxonomies
– Is descriptive
• Becomes more quantitative with time, a “teenager”
– Finds general rules that are equations, like “conservation
of mass”, thermodynamics of equilibrium
• Strives to replace experiments with predictive
computations, becoming an “adult mature discipline”
– This transition is currently happening in chemistry…
• In other words a somethingology tries to advance to a
somethingonomy, from qualitative to quantitative

23. “Predictive” is the keyword!
• There are theories that explain everything afterwards, but
predict nothing
– Pyramidology predicts history accurately but future always
poorly
– There is a BIG difference between a regression fit and a
predictive equation. Ability to match what is known does not
equal ability to predict the unknown.
• Engineering design is based on sufficiently accurate
predictions. Design equations may be mandated in official
standards.
– You can’t know the strength of a concrete mix accurately, so
engineers use safety margins in designs
– Still a lot of computations are useful and used in mechanical
design, inaccuracies don’t make a model useless

24. Training, validation and test data
• Here is the correct way to fit predictive models to
experimental data.
• The data are split to three sets, or generated at different
times for each of the three
– All candidate models are fit to the training data
– The validation data is “predicted” with every type of model, and
the best model type is chosen
• Now it can be fit to training + validation data
– The chosen model is tested for performance
• It has not “seen” the test data before, if you test with previously seen
data you are looking at “fit”, not “prediction ability” !
– Now you can pass on the model, and make some claim about its
prediction accuracy

25. How to make pyramidology look good
• Make predictions about the starting year in January.
• Check predictions next January
– Ups, all wrong again
• Quickly update the model
– Publish how the new model fits history
– Never discuss results from testing old predictions with new data, erase
the old models quickly
• Effect: the public model is always an untested fit, there is never
embarrassing numeric evidence about predictiveness (actually
complete lack of it)
– This game is played all the time. Various companies have forecasters
who don’t want to look incompetent. So does the government. The
forecasters want to keep their jobs and maintain credibility, and the
purpose of the forecasts is not to be right but to influence decisions.
Once the decisions are made, why look back…

26. Intermediate summary
• In the natural sciences, the highest level of understanding is a
quantitative predictive model
– Then you can predict effects of choices or actions on future function
and performance, this is called “engineering design”
• Such models must be based on reproducible experiments, the
models must be validated, and their accuracies known. The
models can only use validated reproducible characterizations
whose accuracy is also known.
– Characterizing material properties of cement mixes, or steel, or
rubber, or plastics – check the official standards.
• Fundamental theories are not yet good enough
– You only learn EQUILIBRIUM thermodynamics
– Even “viscosity” becomes difficult with polymeric liquids, so don’t
expect accurate flow calculations in a complicated geometry
– Flow  mixing  reaction rates  …

27. How does science progress?
• According to Karl Popper
– there is an accepted “scientific paradigm” that scientist use,
until enough evidence accumulates to refine or update it, and
that is a “paradigm shift” (like classical mechanics  relativity,
continuum  quanta)
• Negative evidence = false predictions ! Without them there will be no
paradigm shift
– Various fields like sociology would not fit his definition of
science at all, since they seldom predict anything. There are
no testable hypotheses, but an ideology may “explain”
anything in past history.
– Thomas Kuhn became popular because he essentially
proposed that science is whatever scientists do. This made
sociologists and various others happy again, they loved Kuhn.
– Popper’s views are most appropriate in the natural sciences
and engineering that have developed to a quantitative stage.

28. The scientific method!
• Make a hypothesis or conjecture
• Design experiments that could show it is false
– In fact, the hypothesis should predict something that is
extremely unlikely without it being correct, so when that
happens it is strong support to the hypothesis
• If the hypothesis survives the tests, keep it. If it also
makes useful predictions, others will adopt it and
teach it as current wisdom. (If it can’t be used for
anything of importance, others will not waste their
time. Unless they are political activists.)
– It becomes part of the current scientific paradigm

29. Science is NOT a still image
• Scientists are people who push the boundaries of
knowledge, so the coverage tends to expand,
except when new knowledge collapses an old
theory without a ready replacement. Even then
knowledge expanded and we learned an old
theory is wrong/insufficient.
• Some parts of science are rock solid. Much of the
engineering mathematics of today comes from
Gauss, who lived about 200 years ago. But
“modern mathematics” is evolving actively.

30. The big point
• Science looks for general truths that summarize
experiences in a useful way, and that allow making
predictions.
– Instead of learning every sentence you use, you learn
grammar. Summarization and general rules enable learning
and doing your own things instead of just copying and
repeating.
– Measurements may be needed, but doing measurements
is not the same a doing science.
– The most valuable hypotheses have a wide generality. Very
specific and restricted hypotheses are “doing the right
motions” but amount to little in scientific learning. They
provide a data point, not a useful summary. However, they
are the necessary less glorious grassroots of science, and
that is where most scientists live. It is the big things that
are historic with fame.

31. Other points to note
• If the hypothesis is such that nothing can falsify it,
then it only predicts things that would happen
anyway (or nothing at all)
– It cannot usefully predict anything new and surprising
• But the testing is based on predictions!
– A theory that predicts nothing does not enable any
decision, design, or action. These are the only things
that give value to scientific theories.
– This is why “theories” that predict nothing and can’t
be tested are pseudoscience.

32. About “mistakes”
• Progress of science depends on being wrong
– Falsification of old paradigm is the only way to a new
paradigm
• If you are very afraid of mistakes, you can’t do anything
– But it hurts less to learn from mistakes of others
– Admit mistakes quickly, correct them quickly before they
can take effect, avoid repeating them. It is not honorable
to hide a mistake.
– A man who never made a mistake has done nothing
• However, don’t study mistakes, study successes
– First, this way there is less to study
– Second, repeating a success is better than repeating a
mistake

33. You can never prove that something is
true – you can prove something false
• Mathematics is based on axioms, assumed truths. Its
proofs are valid IF the axioms are valid. In other words
the absolute truths of mathematics are confined within
mathematics.
– Still, logic and mathematics are the tools of clear thinking.
We want equations, and quantitative predictions!
• However many red roses you see, it does not prove
that all roses are red.
– It only takes ONE white rose to prove they are not ALL red
– This is why a hypothesis must be FALSIFIABLE, that is the
best we can do. You cannot have a PROVABLE general
truth, the generality extends only as far as current
experiences.

34. Null hypothesis
• Statistical testing is based on the same ideas.
• You make a NULL HYPOTHESIS “some roses are
not red”, that means falsification of the actual
hypothesis “all roses are red” (the general truth
claimed)
• You show that the null hypothesis is very unlikely
to be true
• That supports your actual hypothesis as
“significant”, meaning that we can keep it for now

35. Back to the magic spell…
• Your scientific story is “good” if you have
– Issue, significance, approach, results, conclusions
– The question must have importance, the conclusions
state effects of your results on theory or on practice!
– It is really good if the results are unexpected and
surprising
• A technically useful engineering result usually
includes a quantitative model
– How do you make a model? Where can you start?
How is “science” done in practice?

36. You are not Newton
• We don’t expect you to come up with new
general principles of fundamental science
• But recall that our science is limited, we need
experimental models
– You have equilibrium thermodynamics, but kinetic
reactions, flows, multiphase materials
– Rheology of a polymeric liquid is difficult enough
• Now add solids, perhaps nanomaterials, reactive species,
electromagnetic fields, …
– Or just examine if an apple breaks when it hits the
floor

37. For modeling
• Define concepts included in the model
– Some need to link to reality through direct
measurements
• Prefer physical measurements over industry technical
standards, the former will stay as they are today
– Others can be computed intermediate variables
• These may come from physics, physical chemistry, etc.
• Dimensionless groups!
• You should use these in statistical modeling

38. The problem with indirect
measurements
• Does “happiness” relate to “wealth”?
– A sociology problem where no concept can be directly
measured. Does wealth include your future inheritance from
grandfather, possibly winning lottery, or your relative who works
for PTT where you may get a good job? A rich girlfriend? How
do you measure “happiness”? Are you feeling 6.3 happy or 8.5
happy, on average this week?
– By adjusting definitions, you can get anything you want. The
result is theories that live as long as the professor who started
them, and who sits on committees of the National Research
Fund while alive. So you better agree with his theory while he
still lives, if you want to do related research. Oh, he is also the
Editor or on board of all relevant journals…
– Stick with the real sciences, we do things better. I like Karl
Popper.

39. Assume you got a topic from your
advisor
• Were you given a hypothesis, or do you need to
come up with one?
• Learn the basics, read review articles, find out
about techniques used in experimental
determinations
– For example, nanomaterials are modern and difficult
exactly because they are too small to “see”. Some
might also have nasty effects that are slow and
delayed, like asbestos fibers, so take precautions.
Don’t rush into exciting new things unless you can also
measure and detect.

40. About managing references
• Commercial option: EndNote
• Free high-quality option: Zotero
• Mostly you will download pdf files. You can feed them to
Zotero, which pulls in the “metadata” and formats it in your
reference list. You can install an add-in to MS-Word.
• Under your “project” folder, create a subfolder “References”.
Keep all your downloaded references in this one place.
• Make sure your project folder gets automatically backed up:
Dropbox, OneDrive, Google Drive,… When your computer
dies and you get a new one, your years of work can be pulled
back from the cloud. (If you know your password…)

41. Discipline…
• Organize your project folder with subfolders
– References
– Proposal and planning
– Materials and methods
• Raw materials, data sheets, measurement devices, sampling and
sample sources
– Experimental designs
– Experimental data
• Precious results of experimental time and cost, your family jewels.
Guard them and keep them pure.
– Analysis of data
• Do NOT mix this with raw data! If you do, soon nobody knows
what was measured and what is estimated or calculated or
imputed or corrected…
– Reporting

42. Finding literature
• Google:
– (ultrasound OR sonication) AND rice AND filetype:pdf
• Google patents
– Selection effect from cost of patenting: economic
importance. Patents can be much better for understanding
the WHY than academic publications. The motivation is
explained explicitly.
• Google scholar
– Check a well-known researcher
– Or search for references with keywords as usual
• Sometimes useful
– View only image results of search (to get chemical
structure, process scheme, etc.)

43. Hypothesis from review of literature
• Are there important gaps in knowledge?
– Turn these into hypotheses that seem reasonable
• Can they be studied with YOUR available
equipment and techniques?
– Can you estimate or guess sizes of effects?
• What are the factors affecting results?
– Which ones can you manipulate
– Which ones can you observe/measure
– Which ones can you limit or select, essentially
defining the scope of your study

44. Exercise that you can do later
• The “outline” feature in MS Word is very good for
making a hierarchical structure, to list factors that
affect.
• If you enjoy drawing and visuals, you can make a
“fishbone diagram” or a “mindmap”.
• Do this with your own project, trying to list everything!
– In plant-based raw materials: variety (of rice, oil palm,
rubber tree), age of plant, maturity of fruit, recent
weather, use of fertilizer, storage conditions, time delays,…
– Note that you can avoid variance from these factors by
pooling and homogenizing an initial batch, then using it
across all experiments. Then only your experimental
conditions cause differences in results. One key goal in an
experimental plan is to reduce or prevent “uncontrolled
noise”.

45. Example: factors influencing gas
mileage of a given car
• Weight
– Car itself
– Passengers
• Number, weight
– Cargo loading
• Weight
– Gas tank full/empty
• Weight of gas
• Route
– Uphill, downhill, level?
– Stops at lights, intersections?
– Distance to drive
– Type or road
• Asphalt
• Unpaved dirt road
• Muddy
• Off-road
– You might have many alternatives…
• Speed
• Condition of car
– Old or new car?
– Engine runs well?
– Maintenance, oil, air
filter, lubrication?
– Type and condition of
tires, tire pressure?
• Weather
– Rain?
– Sunny
• Air conditioning on?
– Windy
• Type of gasoline
– You might have many
alternatives…

46. Decisions and actions, once you have
your hypothesis
• Select scope, manipulated and observed variables
– Do you have the technical skills?
– Do you need to design and construct devices?
• Do you need preliminary or “pilot” experiments?
– Instead of gambling a long-term plan with uncertainties, can you
quickly check for some effects or phenomena?
• Design of experiments, after selecting a minimal set of
main manipulated variables
– Can be complicated, better to stay with some standard design
(e.g. Plackett-Burman), otherwise consult a statistician
– If task is to optimize something like yield, check out response
surfaces and Box-Behnken design
• Statistical analysis of results
– Significant effects AND their effect sizes !

47. About effect size
• Opening your car windows changes
aerodynamics, probably for the worse
– The top speed may go down by 3 km/h if you
open the windows
– If in experiments you would repeatedly find this is
so (using GPS to measure speed), then this effect
is statistically significant
– However, the effect size of 3 km/h is marginal, you
don’t need to care. If it dropped the top speed by
40 km/h, that would be practically important.

48. A key observation
• Statistically significant is not the same as
significant!
– Statistically significant means, it is likely there is
some detectable difference. A detectable
difference can be marginal in size.
– Significant Breakthrough Discovery: with a small
effort and at a low cost, you get a large effect (on
yield, quality, production rate, …)
– This is why you should pay attention to effect
size!

49. The arts and sciences cherish novelty
• But it is not enough. Pay attention to significance!
– If I paint with a banana, it is a novelty but nobody will
buy my painting
– If I multiply two 50-digit numbers and subtract
1234567, nobody else ever did that calculation with
the same numbers: it is a novelty!
• No insight, nothing of interest, can’t be published
– In descriptive sciences, things are published just to
document
• Asteroid number 123456 photographed through telescope –
it goes to a database as an entry, but nobody really cares.
This is their grassroot science, a data point.

50. Art, science and engineering, a
caricature
• When something is done for the first time, it can
be art or science
• When it is done repeatedly and efficiently, it can
be plagiarism, forgery, or engineering research
• In science, if you know the result you should not
do it, because you are looking to expand
knowledge
• In engineering, if you don’t know it works, you
should not do it, because most likely it will not
work.

51. Engineering and science are
intermixed
• The exploration exploitation dilemma in learning
– The caricature was about extremes in exploration and
exploitation
– If you only exploit existing knowledge, that may be convenient
and predictable but you learn nothing new
– If you only explore, you are wandering aimlessly and not
productive
– How much of time or budget should be exploration?
– Some companies have been very good at leaving explorations to
others, then quickly doing a technically reliable good job once a
new technology has been demonstrated. This is a cost
advantage.
– Similar opportunities abound in science, where you can transfer
techniques established in another field to your field. You will
not be “the first”, but if you are quick you can be the first in a
specific context. So keep your eyes open, read widely.

52. In STEM you should use precise
language
• You drive by a field and see a black bull in
profile view.
• Layman’s statement: ”See, they have black
bulls here!”
• Scientific statement: “In this region there is at
least one bull that is black on at least one
side.”

53. On evaluating your own work, or that
of others
• Use the magic spell
– Issue, significance, approach, results, conclusions
– If any of these are missing, thumbs down
• Do this also when you are planning your work
– What kind of results do you expect?
– How can the results affect theory or practice?

54. Research proposal
• Surprise:
– Issue, significance, approach, expected results,
expected effects (conclusions)
• Now approach needs to include
– An experimental design
• How many samples, what measurements, what
experiments, how many replications
– Time – how long would it take?
– Budget – how much would it cost?
– Risks – what can go wrong?

55. Dealing with risk
• Can you prevent it from happening?
– Vaccination may prevent infection
– Do you need to check for impurities of raw materials,
sterilize samples, remove suspended solids before optical
measurements or chromatography, de-aerate liquids, …
– Start with small doses of an additive, see if trouble arises.
Increase dose if all goes well. Large dose first is risky.
• What will you do if the bad thing happens?
– Go to doctor or hospital when you get sick
– Do you have alternatives or backups, for sources of
samples, for determination techniques, …

56. Conclusions
• Most likely you will run experiments and do some statistical
analysis of the results
– Planning of experiments should be based on a hypothesis, the
most “ignorant” just assuming that some factor has an effect
• After literature research, you should have a hypothesis and
an experimental approach, convert these to a detailed
research proposal
• When reporting results, or checking results of others,
statistical significance is worth little, effect size is worth a
lot
• The Magic Spell gives structure to any communication, or
analysis of communication. If an item in it is missing, then
the proposal/presentation/manuscript is incomplete.

57. One final note and warning on
science vs. pseudoscience
• We have not defined science, but it clearly seeks knowledge
and understanding that are useful and predictive
– The best we can do is accept a useful hypothesis until it has been
falsified. Useful ones predict something, so they can always be
tested.
• We can identify much of pseudoscience from this already
– Either the hypothesis can’t be falsified ever, by anything
– or its proponents bitterly oppose any such test that could falsify it,
their interest is not truth and knowledge but politics
• But… everybody wants to have the clout of science. If you point
a finger at pseudoscience, many people will be upset.
– It suffices that you know the truth, don’t waste your time in an
argument. Just keep it between you, me, and Sir Karl Popper. And
that guy in the toothpaste commercial who wears a white lab coat,
he knows, too.

58. Further reading
• Search for “PLOS ten simple rules”
– Lots of valuable advice in short format
• Search for “Hamming you and your research”
• “The art of doing SCIENCE and engineering”
by Richard Hamming is a book worth getting
and reading