Validation of Design Tools-PPT for CEDAR Meeting-04-15-2016
1. CED RClemson Engineering Design Applications and Research
Varun Kumar
kumar3@clemson.edu
Committee:
Dr. Gregory Mocko (Chair)
Dr. Joshua Summers
Dr. George Fadel
Similarity assessment of design problems
used in creativity research
2. 2 of 26
kumar3@clemson.eduCED RClemson Engineering Design Applications and Research
Research background
Research objectives
Design problems in creativity research
Design problem network
Problem similarity assessment
– Based on structural elements
– Based on Latent Semantic Analysis
Conclusions
Other tasks completed
Future work
References
Outline
3. 3 of 26
kumar3@clemson.eduCED RClemson Engineering Design Applications and Research
Research Overview
Study 1
Problem
1
Method
1
Metric 1Subject
Protocol
1
Study 2
Problem
2
Method
2
Metric 2Subject
Protocol
2
Evaluate the impact of different
methods on design creativity
- Various sources
of difference
between studies
are present
- What prohibits
comparison of two
methods based
on published
literature alone?
- Design problems
and their place in
creativity research
4. 4 of 26
kumar3@clemson.eduCED RClemson Engineering Design Applications and Research
Title Author, Year
Cognitive processes and ill-defined problems: A case study
from design
Eastman, C. M., 1969
Dilemmas in a General Theory of Planning Rittel, H. W. J., & Webber, M.
M., 1973
The structure of ill-structured problems Simon, H. A., 1977
A Proposed Taxonomy of Mechanical Design Problems Dixon, J. R., 1988
The structure of design problem spaces Goel, V., and Pirolli, 1992
A Suggested Taxonomy for Engineering Design Problems Frost, R. B., 1994
Mechanical Engineering Design Complexity Metrics: Size,
Coupling, and Solvability
Summers, J. D., and Shah, J.
J., 2010
In search of effective design problems for design research Durand et al., 2015
Research background
5. 5 of 26
kumar3@clemson.eduCED RClemson Engineering Design Applications and Research
Research opportunities
Conceptual
Problems
Use in protocol/
study new tool
or method
Analyze
creativity
metrics
Compare
problem similarity
based on metric
measures
Research Question 2
How to assess problem similarity based
solely on problem representation?
Summers and Shah’s approach Durand and Linsey’s approach
Opportunity
a. How to illustrate the application in
conceptual problems?
b. Can natural language
representation be used to compare
problems for similarity?
Opportunity
a. How to assess problem similarity
based solely on problem
representation?
6. 6 of 26
kumar3@clemson.eduCED RClemson Engineering Design Applications and Research
Understand the pattern of design problem usage.
Enable similarity comparison between conceptual design problems.
Evaluate the impact of problem size on between-study treatment effects.
Research Objectives
7. 7 of 26
kumar3@clemson.eduCED RClemson Engineering Design Applications and Research
Statement of requirement, needs, functions or objectives.
Most problems are ill-defined and co-evolve with solutions.
Conceptual design problems differ from real life design problems.
Examples:
Design Problems in creativity research
Design Problem 1 (Linsey, 2012)
Design and build a low-cost, easy to
manufacture peanut shelling machine that
will increase the productivity of the African
peanut farmers. Target throughput is
approximately 50 Kg per hour. The goals
include: a) Must remove the shell with
minimal damage to peanuts b) Electrical
outlets are not available as a power source
c) A large quantity of peanuts must be
quickly shelled.
Design Problem 2 (Mulet, 2012)
It is asked to design a new table for
offices that allows alternate sitting and
stand up work. There are a lot of people
who must work on sitting position the full
day. The possibility to alternate positions
during working time could drive to an
improvement in health and productivity.
The current tables that allow combining
positions in work have limited surface, not
enough for design, architecture and
engineering needs.
8. 8 of 26
kumar3@clemson.eduCED RClemson Engineering Design Applications and Research
Problem Usage Network
Problem
Author/s
9. 9 of 26
kumar3@clemson.eduCED RClemson Engineering Design Applications and Research
Metric Value Conclusions
Network density 0.0751 Poor connectivity between problems or vice-versa (between
researchers)
No. of nodes with
degree 0 or 1
20* High percentage of problems which have not been re-used
No. of
communities
20** - Presence of sub-groups
- Problem sharing within sub-groups
Network Analysis
* Total nodes = 50
** Includes isolated nodes
10. 10 of 26
kumar3@clemson.eduCED RClemson Engineering Design Applications and Research
Lack of collaboration
Absence of guidelines for formulating conceptual problems
No ‘benchmark’ problems in practice
Absence of methods to compare problems
Impact of problems on creativity results not understood fully
Possible reasons for disconnectedness
11. 11 of 26
kumar3@clemson.eduCED RClemson Engineering Design Applications and Research
Why compare problems?
– Help reduce variability in structure of design problems
– Enable problem replacement in pre-post test experiment designs
– Provide metric for assessing the choice of problem
– Enable use of ‘similar’ problems in practice
Methods for problem similarity assessment
– Approach 1: Identification of structural elements in problem
statement
– Approach 2: Latent Semantic analysis of problem statements
Objective is to compare similarity based on problem representation
Assessment of problem similarity
12. 12 of 26
kumar3@clemson.eduCED RClemson Engineering Design Applications and Research
Approach 1: Element identification
Design
Problem
Goals of the problem
Functional
Requirements
Non-functional
requirements
Information about end
user
Reference to an existing
product
13. 13 of 26
kumar3@clemson.eduCED RClemson Engineering Design Applications and Research
Goal of the problem
– Final objective of design task
– Problem may have more than one goal
– Ratio variable
Functional requirements
– Primary functions of designed object
– Should not insinuate about method for achieving function
– Generally action verbs associated with objects
– Can be nouns derived from verbs (eg. washing machine)
Non-functional requirements
– Do not describe primary function
– Determine shape, size, operation and design selection
Approach 1: Element definitions
14. 14 of 26
kumar3@clemson.eduCED RClemson Engineering Design Applications and Research
Information about end user
– Knowledge of end user in problem statement
– Should be explicitly stated
– Categorical variable (Yes/No)
Reference to an existing product
– Whether design problem hints at an existing product known to
reader.
– May be a ‘conundrum’ due to cultural differences
– Categorical number (Yes/No)
Approach 1: Element definitions (contd.)
15. 15 of 26
kumar3@clemson.eduCED RClemson Engineering Design Applications and Research
Approach 1: Element identification example 1
16. 16 of 26
kumar3@clemson.eduCED RClemson Engineering Design Applications and Research
Approach 1: Element identification example 2
17. 17 of 26
kumar3@clemson.eduCED RClemson Engineering Design Applications and Research
Based on elements identified and their ‘counts’ problems can be
compared
Approach 1: Comparing two problems
Design Problem
Design Problem Element Example 1 Example 2
No. of goals 1 1
No. of Functional requirements 1 2
No. of non-functional requirements 2 4
End user info. (Yes=1/No=0) 1 1
Reference to an existing product
(Yes=1/No=0)
1 1
18. 18 of 26
kumar3@clemson.eduCED RClemson Engineering Design Applications and Research
Definition of elements always subject to subjective interpretation
Test whether people identify the same elements and their quantities.
Method used:
– 4 evaluators/raters chosen
– Evaluators are presented with definitions of the 5 elements
– An identification example is shown to all evaluators separately
– 4 evaluators asked to identify the 5 elements from 4 design problems
– Inter-rater agreement evaluated
Approach 1: Verifying similarity comparison protocol
Elements
No. of
goal
Functional
requirement
Non-
Functional
requirement
End user
info
Ref. to
existing
product
Krippendorff’s
alpha
1.0 1.0 0.863 0.184 0.598
19. 19 of 26
kumar3@clemson.eduCED RClemson Engineering Design Applications and Research
Pros
– Useful as a starting guide for problem formulation
– Offers useful and simple mean for comparing 2 problems
– Helps identify ‘information content’ of problem statements
– Method shows good correlation between evaluators for most
elements
– Compares problems based on representation
Cons
– Subjectivity of human interpretation still involved
– Cannot compare ‘knowledge content’ and skill needed in problem
solving.
Approach 1: Conclusions
20. 20 of 26
kumar3@clemson.eduCED RClemson Engineering Design Applications and Research
Linguistic similarity of design problem representations
Relies on extracting contextual meanings from large text corpus.
Assumes that words and phrases with similar meaning are used in
similar context
Generates similarity scores between -1 and 1
LSA tool available at http://lsa.colorado.edu/ used
45 design problem statements tested for LSA similarity
Approach 2: Latent Semantic Analysis
21. 21 of 26
kumar3@clemson.eduCED RClemson Engineering Design Applications and Research
Approach 2: LSA results
Problem statements
Design an urban
bi or tri cycle
for use by white
collar workers
Design a concrete
mixer which can
operate using a
bicycle pedal
mechanism.
Design a device
which can
compost waste
vegetables.
Design a reading
device for old people
which can read the
newspaper for them.
Design a
water
lifting
device.
Propose alternative solution to coal
pile problem at thermal plant, since the
plant may not have enough land
nearby to store the coal on ground.
0.02 0.01 0.06 0.01 0.03
Design of a next generation alarm
clock which ensures easy operations
like change of time and alarm stop
unlike conventional clocks.
0.09 0.26 0.15 0.13 0.12
Design of a litter collection device for
volunteers.
0.23 0.48 0.6 0.25 0.39
Redesign an electric toothbrush for
increased portability.
0.01 0 -0.01 -0.01 0.03
Design alternative means to manually
propel boats which are easy to
maneuver, don't rock the boat or splash
water.
0.09 0.15 0.15 0.06 0.52
A portion of the LSA result matrix
22. 22 of 26
kumar3@clemson.eduCED RClemson Engineering Design Applications and Research
Pros
– Objective way of comparing two problem statements
– Based on problem representation only
– Backed by statistical computation, avoids subjectivity
– Can detect contextual changes between text bodies even with slight
modifications (viz. design vs redesign etc.)
Cons
– Results only as good as the corpus of text used
– Based solely on representation, problem ‘solvability’ can’t be
compared
Approach 2: Conclusions
23. 23 of 26
kumar3@clemson.eduCED RClemson Engineering Design Applications and Research
Evaluate the impact of problem size on between-study treatment effects.
– Meta-Analysis of different studies to assess treatment effect.
– Analysis of heterogeneity between treatment effect of various studies
– Meta-regression approach to understand impact of problem size on
heterogeneity
Other works completed
24. 24 of 26
kumar3@clemson.eduCED RClemson Engineering Design Applications and Research
Write Thesis
Defend
Enjoy Summer
Future work
25. 25 of 26
kumar3@clemson.eduCED RClemson Engineering Design Applications and Research
Linsey, J. S., Clauss, E. F., Kurtoglu, T., Murphy, J. T., Wood, K. L., and Markman, a. B., 2011, “An
Experimental Study of Group Idea Generation Techniques: Understanding the Roles of Idea
Representation and Viewing Methods,” J. Mech. Des., 133(3), p. 031008.
Chulvi, V., Sonseca, A., Mulet, E., and Chakrabarti, A., 2012, “Assessment of the Relationships Among
Design Methods, Design Activities, and Creativity,” J. Mech. Des., 134(11), p. 111004.
Ameri, F., Summers, J. D., Mocko, G. M., and Porter, M., 2008, “Engineering design complexity: An
investigation of methods and measures,” Res. Eng. Des., 19(2-3), pp. 161–179.
Rittel, H. W. J., and Webber, M. M., 1973, “Dilemmas in a General Theory of Planning,” Policy Sci.,
4(December 1969), pp. 155–169.
Simon, H. A., 1977, “The structure of ill-structured problems,” Models of discovery, Springer, pp. 304–
325.
Dixon, J. R., Duffey, M. R., Irani, R. K., Meunier, K. L., Orelup, M. F., 1988, “A Proposed Taxonomy of
Mechanical Design Problems,” Comput. Eng., 1, pp. 41–46.
Goel, V., and Pirolli, P., 1989, “Motivating the notion of generic design within information-processing
theory: the design problem space,” AI Mag., 10(1), pp. 18–36.
Goel, V., and Pirolli, P., 1992, “The structure of design problem spaces,” Cogn. Sci., 16(3), pp. 395–
429.
Frost, R. B., 1994, “A Suggested Taxonomy for Engineering Design Problems,” J. Eng. Des., 5(4), pp.
399–410.
Summers, J. D., and Shah, J. J., 2010, “Mechanical Engineering Design Complexity Metrics: Size,
Coupling, and Solvability,” J. Mech. Des., 132(2), p. 021004.
Tsenn, J., Helms, M. E., Linsey, J. S., and Mcadams, D. A., 2015, “In search of effective design
problems for design research,” pp. 1–10
References
27. 27 of 26
kumar3@clemson.eduCED RClemson Engineering Design Applications and Research
Appendix A: Design problem used for experimental verification
of protocol
Problem 1: Design a new system for gathering together and hiding the wires of the electronic equipment in an
office table. Currently the work in the field of design, architecture and engineering needs of a personal computer,
printers, and scanners. Each of these devices needs of electrical supply and the wires on table surface are annoying.
Actually, there are simple solutions to gather them, but it is difficult to extract or introduce a wire, or they leave the
wires hanging behind the table.
Problem 2: Design an automatic recycler device that can automatically sort plastic bottles, glass containers,
aluminum cans, and tin cans. The major differentiation between types of materials lay with the given dimensions of
the products: plastic bottles are the tallest, glass containers are very short and heavy, and aluminum cans are
lightweight. Devices are given strict requirements to adhere to such as volume and weight constraints, safety
requirements, and most importantly, have to operate autonomously once a master shutoff switch is toggled.
Problem 3: A mechanical system is required which, in the event of a fire, will enable people to escape from a
six-storey building by lowering themselves to the ground from windows. The system, which might make use of a 5
mm diameter steel cable, must be capable of lowering either a small child or a heavy adult at approximately the
same constant speed.
Problem 4: Design and develop an artifact to facilitate grocery shopping in a typical French/Singapore
Chinatown fresh market. The artifact should facilitate carrying of groceries from fresh market to home.
28. 28 of 26
kumar3@clemson.eduCED RClemson Engineering Design Applications and Research
S.No. Characteristic Definition Scoring system
1.
Reference to existing
product
Whether or not the problem statement contains a
reference to an existing product.
Questions to be asked
Does the problem statement contain reference to
an existing product?
Does the problem statement ask you to re-design
an existing product?
If a reference to existing product exists in the
problem statement, assign a score of 1, else 0
2.
Functional
requirements (FR)
These are the things which the product needs to do, or
the tasks that you want the product to perform without
any reference to how it should be done. Judgement
should be based only on explicitly stated texts, and not
on the implied meaning of a sentence. To identify FRs,
look for:
1. Action verbs like move, work etc. associated with
objects (objects include nouns on which the
action verbs act like throw stones, gather fruits
etc.)
2. Primary functions of the design (eg. move
objects, lift, transport etc.).
3. These could also be nouns derived from verbs
(eg. washing machine, toaster etc.)
4. If there are two objects associated with one
primary function, count it as 2 separate FRs (eg.
move object X & object Y is counted as 2 FRs)
Questions to be asked
What are the primary functions of the product?
What are the expected outputs/tasks which the
product needs to perform?
Count the number of functional requirements
given in the problem statement. There can be 2
cases:
1: When a new product design is desired: In
this case, FR should be specified in the
problem statement. Else, give a score of 0.
2: When a re-design or a new design for an
existing product is desired: FR count in this
case is already 1 to start off, since atleast 1
product function is known.
Appendix B: Element definitions
29. 29 of 26
kumar3@clemson.eduCED RClemson Engineering Design Applications and Research
S.No. Characteristic Definition Scoring system
3.
Non-functional
requirements
These are 'non-functional' requirements, which do not determine
the primary functions of the product, but cast a bound on the
overall shape, size, cost, operation and selection of the design.
Judgement should be based only on explicitly stated texts, and not
on the implied meaning of a sentence. Typical NFRs include:
1. Any restrictions on what the product or system shall or shall
not do apart from the primary functions (eg. should not
overheat, should be easy to use, should be manufacturable
etc.)
2. Any restriction on how the product shall fulfil its intended
functions (eg. device should move using rollers, device
should work using sliding mechanism etc.)
3. Any qualities that the product must possess (eg. easy to
make, easy to use, cheap, etc.)
Questions to be asked
• What things cast a limit or a bound on the solution space?
What are the qualities that the product should possess as a
whole?
How the overall product 'shall be' like?
Count the number of Non-functional requirements
given in the problem statement.
4. Number of goals
These are goals or final objectives associated with the design
problem.
Questions to be asked
What is the final objective of the problem statement, or
Does the problem statement ask only to design or do
something else?
Count the number of goals or objectives mentioned
in the problem statement.
5.
Information about end
user
Information about who is going to use the product or who is the
customer. It should be explicitly stated in the problem statement.
Questions to be asked
Who is going to be the end user of the product?
Check the problem statement to see if any
information about the end user is provided or not. If yes,
give a score of 1, else a 0
Appendix B: Element definitions
30. 30 of 26
kumar3@clemson.eduCED RClemson Engineering Design Applications and Research
Authors
Eastman
(1969)
Rittel and
Webber
(1973)
Dixon
(1988)
Goel
(1992)
Frost
(1994)
Summer
s and Shah
(2010)
Durand
and Linsey
(2015)
Lack of
well -defined
specifications
for goals
Lack of
formal
representation
language
Lack of
definitive
formulation
Lack of
known solution
states
Lack of
immediate tests
for solutions
Lack of
objectivity in
solution
selection
Lack of
learning
opportunities by
trial and error
Lack of
exhaustive set of
solutions
Perceived
need
Function
Phenomenon
Embodiment
Artefact
type
Artefact
instance
Lack of
problem
structuring
Presence of
distinct
problem
solving
phases
Reversal of
direction of
transformatio
n functions
Modular
structure
Incremental
solution
transformatio
n
Personalized
stopping rules
Abstraction
hierarchies
Type of
entity being
designed
Degree of
innovation
involved
Extent of
possible
decomposition of
designed entity
Availability
of adaptable
solutions
Complexity
of designed entity
Degree of
interaction within
solution
Looseness
or tightness of
constraints
Number of
artefacts to be
built
Number
of independent
design
variables
Number
of dependent
design
variables
Number
of design
relations
Number
of measures of
goodness
Number
of functional
requirements
and design
parameters
Number
of design
constraints
Size
Connectedness
Participant's
familiarity with
problem
Participant's
familiarity with
solution and
principles
Size of
potential solution
space
Assumed
constraints due to
known solutions
Effort
required
Fixation
Problem
domain
31. 31 of 26
kumar3@clemson.eduCED RClemson Engineering Design Applications and Research
DP1 Toy to transport basket
DP2 Subway improvement
DP3 New drawing table
DP4 Tubular map case
DP5
System to collect and
hide electronic wires
DP6 New table for offices
DP7 Rover device
DP8 Automatic recycler
DP9 Wearable binoculars
DP10 Bi/tri-cycle
DP11
Bicycled Concrete
Mixer
DP12 Vegetable Composter
DP13 Reading Device
DP14 Water lifting device
DP15 Traffic light using LED
32. 32 of 26
kumar3@clemson.eduCED RClemson Engineering Design Applications and Research
DP16 Counting and packaging device
DP17 Milk frothing device
DP18 Peanut shelling machine
DP19 Automatic casting system
DP20 Robotic vacuum
DP21 Portable washing machine
DP22 Solar heating and cooking device
DP23 Rigless abandonment tool
DP24 Wheelchair simulator
DP25 Counter top stand
DP26 Split pin design
DP27 Device to transport ping pong ball
DP28 Tool for alien species
DP29 Biomass based cooking system
DP30 System for UAVs
DP31 Remote controller
DP32 Doodle toaster
33. 33 of 26
kumar3@clemson.eduCED RClemson Engineering Design Applications and Research
DP33
Combined toaster and coffee
maker
DP34 Horizontal toaster
DP35 Crumb tray toaster
DP36 Shopping cart
DP37 Outdoor customer product
DP38 Manual boat propulsion device
DP39 Plastic and paper sorting device
DP40 Lunar dust protection device
DP41 Desk elevator
DP42 Ship guitar
DP43 Bottle cap regiser machine
DP44 Soda drink maker
DP45 Concept to center mating parts
DP46 Concept to use snow as insulator
DP47 Coal pile solution
DP48 Next gen alarm clock
DP49 Litter control device
DP50 Electric tooth brush
Editor's Notes
Hello everyone, My name is …….,
Going to talk about part of my research which deals with similarity assessement of design problems
Mocko, Summers and Fadel are committee chairs
-Outline of things I will be talking about today.
This is a brief overview of what I am looking into.
Various methods are tested, and there are lot of differences between each study
Some differences like protocol and method are inherent, but can we reduce some other variations like DP or experimental time.
My focus is on understanding design problems.
Problem characterization has been undertaken for quite some time now.
The overall idea found is that design problems are ill-defined, and co-evolve with solution.
Designers work towards a solution, and keep structuring the problem according to the solution found.
People have tried to add structure to design problems
Eastman: showed nature of ill-structured problems in architectural design
Rittel: characteristics of wicked problems in planning
Simon: what are the characteristics of ill-structured problems? Playing chess is well structured for a single move but ill structured for whole game
Dixon: Define problem based on initial and final states of knowledge, physical type, coupling between parameters, design assessment criteria
Goel: Deal with nature of design task environment
Frost: Taxonomy with examples of everyday products, not very useful for small problems like those found in creativity
Summers: quantify size, coupling and solvability. Needs more effort in identifying these metrics
Durand: 9 characteristics were defined. However, they did not showcase any characteristic in any problem. Some characteristics were not entirely
Design problem related.
Natural language representation have been used for identifying size.
Can we use them to compare problems or characterize them
Durand and others try to compare problem similarity based on results obtained from experiments. Can the loop be eliminated and problems
be compared based on their representation
Most people call them ill-defined since they lack proper structure.
Problems in maths are examples of well define problems since the correct solutions state is known
They lack a final solution, and the path that one needs to take to attain the solution.
Conceptual problems are small and abstract in order to evoke idea generation in participants, and not create a mental block
Two examples are shown here to illustrate what kind of problems are typically used.
Problems are not very big or intriguing,
One of the objective was to analyze how design problems have been used over the years.
Network graphs shows the connectivity between problems based on authors using them.
Problems which have atleast one common author are connected.
The problems used here are from user studies published in past 15 years in creativity and ideation.
Network is not dense, and connectivity is poor apart from some sub-groups which seem to be working within themselves.
The network statistics also indicate the loose-connectivity between design problems.
Communities are also present, which means sub-sections of research community seem to be working within a group
-people do not believe in re-using problems which have been used.
You are free to use any problem that you feel is good enough for the task. Considerable researcher-researcher variation
Lack of easy to use metric which can help researcher compare their problem with some of the others that have been used.
This is the major motivation as to why I am talking about problem similarity
You may feel you should have the liberty to choose the problem that you want.
However, the choice of problem has impact on results from creativity study.
Problems are another covariate in experiments. Linsey have found that for different problems, results can be different.
If problems are also a source of variation, it makes it difficult to compare 2 different methods with each other.
Based on prior work and by reading through several problem statements used in past, I managed to compile a list of elements which are more useful for conceptual problems.
I will go over some of these elements in the next 2 slides
- For analyzing problems, we need a set of rules or definitions to enable coherence.
-Goal: what does the problem ask you to do? Are you supposed to design an object?
Problems may have more than one goal, when you are asked to deign more than one artifact
-FR: Generally we find FR’s are associated with verb+ object combination
-NFR: Quality attributes, and not associated with primary functions
-cultural differences: first time I was asked to do a Axiomatic design of coffee maker, I was not aware of what a coffee maker is like, how it works. Coffees are
Generally tea drinkers, we don’t see coffee makers in our homes.
-FR’s can be nouns derived from verbs as well
No matter how well you define these terms, people will always have a subjective prejudice about how they see it.
To test whether there is atleast some usability of these definitions, I did a protocol study to see whether people can identify these elements similarly in
Problems or not
We see perfect coherence between 4 raters as far as identifying the goal and FR’s is concerned. However, perfect correlation would rarely be seen if this study is conducted with a large group and problem set,
Low end user info agreement was probably because the problem statements did not clearly state who the end user is going to be. They were more implied and probably a reason for this low agreement.
Extracts contextual meaning of phrases from a large corpus.
I filled in all the 45 problem statements into this tool to see if it can identify similar problems based on contextual usage of the phrases.
Here is a snippet from the 45x45 matrix that we got from LSA.
I do not have a cut-off score using which I can say that
Here for analysis I had used collection of readings upto 1st year of college. This may not be apt for this task.
We cant say whether the problems are solvable or not. Car to accelerate in 60 sec or in 1 min are the same, however LSA gives a difference in similarity.
Use a concept called meta-analysis which is a staitsitcal way of combining the results of different studies.
Also trying to understand if the choice of design problem are a probable cause for the differences between treatment effects of different studies.