Academic Paper Writing Service http://StudyHub.vip/An-Analysis-Of-Collaborative-Problem-So

1. An analysis of collaborative problem-solving
activities mediated by individual-based and
collaborative computer simulations
C.-J. Chang,* M.-H. Chang,* C.-C. Liu,* B.-C. Chiu,* S.-H. Fan Chiang,* C.-T. Wen,* F.-K. Hwang,†
P.-Y. Chao,‡ Y.-L. Chen§ & C.-S. Chai¶
*Graduate Institute of Network Learning Technology, National Central University, Taiwan
†Department of Physics, National Taiwan Normal University, Taiwan
‡Department of Information Communication, Yuan Ze University, Taiwan
§Department of Information Management, National Central University, Taiwan
¶National Institute of Education, Nanyang Technological University, Singapore
Abstract Researchers have indicated that the collaborative problem-solving space afforded by the
collaborative systems significantly impact the problem-solving process. However, recent
investigations into collaborative simulations, which allow a group of students to jointly
manipulate a problem in a shared problem space, have yielded divergent results regarding their
effects on collaborative learning. Hence, this study analysed how students solved a physics
problem using individual-based and collaborative simulations to understand their effects on
science learning. Multiple data sources including group discourse, problem-solving activities,
learning test scores, and questionnaire feedback were analysed. Lag sequential analysis on the
data found that students using the two simulations collaborated with peers to solve the problem
in significantly different patterns. The students using the collaborative simulations demonstrated
active engagement in the collaborative activity; however, they did not transform discussions into
workable problem-solving activities. The students using the individual-based simulation showed
a lower level of collaboration engagement, starting with individual exploration of the problem
with the simulation, followed by group reflection. The two groups also showed significant
differences in their learning test scores. The findings and pedagogical suggestions are discussed
in the hope of addressing critical activity design issues in using computer simulations for
facilitating collaborative learning.
Keywords collaborative problem solving, computer simulations, discourse analysis, lag sequential analysis,
learning analytics.
Introduction
As collaboration and problem solving are considered to
be core competencies of the 21st century (OECD,
2013), collaborative problem solving (CPS) has been
increasingly highlighted by researchers and educators
to address the contemporary education objectives (Care,
Scoular, & Griffin, 2016; Lin, Duh, Li, Wang, & Tsai,
2013; Lin, Hou, Wu, & Chang, 2014). CPS stemmed
from social constructivism and emphasizes knowledge
advancement through joint problem exploration, collab-
orative planning, and monitoring of problem-solving
processes to obtain a better solution to problems (Barron,
2000). Students participating in CPS activities are situ-
ated in a collaborative context in which they need to
Accepted: 02 July 2017
Correspondence: Chen-Chung Liu, Graduate Institute of Network
Learning Technology, National Central University, No. 300, Jhongda
Rd., Jhongli City, Taoyuan County 32001, Taiwan. Email: ccliu@cl.ncu.
edu.tw
© 2017 John Wiley & Sons Ltd Journal of Computer Assisted Learning 1
doi: 10.1111/jcal.12208
Original article
bs_bs_banner

2. integrate multiple perspectives and reflect through peer
discussions to develop effective solutions (Dillenbourg
& Traum, 2006). Owing to their social and constructive
nature, CPS activities have become an integral instruc-
tional approach to facilitating science knowledge
construction and enhancing students’ understanding of
science knowledge (Hmelo-Silver, Jordan, Liu, &
Chernobilsky, 2011; Lin et al., 2013).
Numerous studies have indicated that using
computer simulations could effectively augment the
CPS process for supporting science learning (Care
et al., 2016; Lin et al., 2013). Computer simulations
provide students with an observable world as an aid
to generating and testing hypotheses (Rutten,
van Joolingen, & van der Veen, 2012). In recent years,
computer simulations have been integrated with online
text chatrooms to help students solve science problems
collaboratively (Hao, Liu, von Davier, & Kyllonen,
2015; Lin et al., 2014). Computer simulations can serve
as group cognitive tools to help small groups of
students achieve mutual understanding of the problem
during peer discussion (Gijlers, Saab, Van Joolingen,
De Jong, & Van Hout-Wolters, 2009). It has been
found that computer simulations could not only effec-
tively enhance students’ learning performance during
the CPS process (Hao et al., 2015) but also facilitate
their high-level cognitive skills, such as their applica-
tion and analysis skills (Lin et al., 2014).
Despite the potential affordances of computer simula-
tions for supporting CPS, it has been found that the inter-
actions between students tend to be quite limited, and
that students can often only collaborate with each other
on a superficial level (Barron, 2003; Lin et al., 2014).
A great number of students tend to solve science
problems individually rather than collaboratively (Saab,
van Joolingen, & van Hout-Wolters, 2012; Sun & Looi,
2013). This may be because the computer simulations
applied in these studies were individual based, meaning
that student dependency was loosely coupled, such that
the students did not need to coordinate with their peers
to jointly solve the problems. In such collaborative
contexts, students face difficulties establishing common
references, resolving discrepancies in understanding,
and negotiating individual and collective actions on the
way to joint understanding (Barron, 2003). Moreover,
the free-riding or social loafing effect (Brush, 1998;
Swaray, 2012) may also impede the collaborative
process, with some students not actively participating in
the activities owing to the loosely tied interdependence
between students in individual-based simulations.
Recent developments in computer simulations have
made it possible to allow multiple students to work
jointly on a collaborative simulation to address the afore-
mentioned issues. These collaborative simulations afford
a shared problem space in which a group of students can
jointly manipulate a problem, and all the manipulations
are synchronized through the Internet. With such collab-
orative simulations, multiple students can jointly explore
a science problem in a shared simulation session (Jara,
Candelas, Torres, Dormido, & Esquembre, 2012). Stu-
dents’ actions in such collaborative simulations will si-
multaneously influence the shared problem state.
Numerous collaborative simulations have been imple-
mented to support CPS activities in diverse domains in-
cluding physics (Care et al., 2016; de la Torre, Heradio,
& Dormido, 2013; Gijlers et al., 2009; Jara et al.,
2012), earth science (Bell, Urhahne, Schanze, &
Ploetzner, 2010) and control systems (Jermann &
Dillenbourg, 2008).
These collaborative simulations offer new affordances
to better facilitate CPS processes. However, recent inves-
tigations into such shared problem spaces have yielded
divergent results. The shared problem space afforded
by the collaborative simulations provides group mem-
bers with information about the most up-to-date problem
situations and thus facilitates the establishment of shared
references (Barron, 2000) and joint attention (Çakır,
Zemel, & Stahl, 2009; Stahl, 2006). It was found that
the shared problem space could improve students’ artic-
ulation processes and promote their engagement in a
high level of peer discussion to construct shared under-
standing of the problem (Chung, Lee, & Liu, 2013;
Liu, Chung, Chen, & Liu, 2009). These collaborative
simulations could also enact a stronger level of interde-
pendence by giving each student responsibility for the
manipulation of the simulation so as to avoid the social
loafing effect and enhance the level of collaboration
(Brush, 1998; Zurita & Nussbaum, 2007).
However, it has been stated that the individual and
shared problem space may lead to unequal distribution
of reflective and collaborative activities. A previous
study by Liu, Tao, Chen, Chen, and Liu (2013) pointed
out that students tended to perform less in-depth activi-
ties in the shared space as they did not have individual
space to reflect upon the learning task. Chang, Liu, and
Shen (2012) also found that students exhibited less
2 C.-J. Chang et al.
© 2017 John Wiley & Sons Ltd

3. exploratory activities in the shared problem space,
although they demonstrated more discussion activities
than they did when an individual space was available.
It is thus necessary to investigate the distribution and
the patterns of students’ problem-solving activities in
different CPS settings to understand students’ participa-
tion in the learning task.
In this vein, this study aimed to investigate the impact
of an individual-based and collaborative simulation on
the CPS process. In order to understand the problem-
solving patterns and to show the dynamic transitions at
a macro level, we applied the lag sequential analysis
(LSA) (Bakeman & Gottman, 1997) technique to ana-
lyse the students’ discussion and problem-solving activ-
ities. LSA has been utilized to identify significant
behavioural sequential patterns, such as web traverse
analysis (Liu, Fan Chiang, Chou, & Chen, 2010) and
communication pattern analysis (Yamauchi, Yokozawa,
Shinohara, & Ishida, 2000). LSA was applied in this
study to identify significant CPS patterns that display im-
portant CPS moves, helping researchers understand the
progress of CPS activities facilitated by individual-based
and collaborative simulations. Data were collected from
83 senior high school students who engaged in a CPS
activity to solve a science problem. Multiple data sources
including group discussions, problem-solving activities
in the simulations, questionnaire feedback and the
students’ learning performance were analysed to answer
the following research questions:
RQ1: How did the distribution of students’ problem-
solving activities differ in the individual-based
and collaborative simulations?
RQ2: Did students show any distinctive CPS pattern in
their discussion and activities? If yes, did students
using individual-based and collaborative simula-
tions show significantly different CPS patterns?
RQ3: Did the individual-based and collaborative simu-
lations have a significant impact on the students’
learning performance?
Method
Participants
This study adopted a quasi-experimental design to inves-
tigate how students collaboratively solved a physics
problem using an individual-based simulation (IS) and
a collaborative simulation (CS). The participants in this
study were 83 eleventh-grade students from two classes
at a senior high school in northern Taiwan. Their ages
ranged from 17 to 18 years, and their native language is
Mandarin. The two intact classes instructed by the same
physics teacher were randomly assigned to the IS and CS
groups. The IS groups consisted of 44 students (16 boys
and 28 girls) divided into 14 groups of three and one
dyad owing to the limitation of student numbers. The
CS groups included 39 students (20 boys and 19 girls)
divided into 13 groups of three. There was no significant
statistical difference in their physics scores in their
mid-term examination (t = 1.04, p > 0.05), indicating
that the two groups had similar prior knowledge. In addi-
tion, none of the students in either group had experience
of using computer simulations in science learning.
Hence, the participants were suitable for this study,
which aimed to understand how they would solve a
scientific problem using the two types of simulations.
The CPS activity and procedure
Both the IS and CS groups were required to solve the
same target problem, a kinematics game in which they
needed to apply kinematics concepts to achieve the goal
of the simulation. In this game, an aircraft flying horizon-
tally at a constant velocity (Vx) will drop a bomb to
destroy the radar station located at a certain distance
(L). The game allowed the students to manipulate three
variables: the height of the aircraft (H), the constant
flying velocity (Vx) of the aircraft in the horizontal direc-
tion and the distance of the bomb-release point from the
radar station (L). Only when the time that the bomb needs
to fall to the ground is equal to the time that the aircraft
needs to fly to the horizontal position of the radar station
can the problem be solved. The students need to know
the physics rules that govern the movement of each
model, and also closely coordinate and discuss with each
other to successfully complete the task.
The CPS activity was conducted in a computer class-
room in which each student used a desktop computer to
participate in the activity. Before the activity, a learning
pre-test was conducted to examine the students’ prior
knowledge of the physics concepts involved in the target
problem. An orientation session was then given to help
the students understand the operation of the CPS
platform and practiced using the simulations. The orien-
tation session lasted 20 min. After the orientation
Collaborative problem solving 3
© 2017 John Wiley & Sons Ltd

4. session, the students participated in the CPS activities for
60 min. The IS groups used individual-based simulation,
while the CS groups adopted collaborative simulation.
The students in each group were then intentionally
located in different positions of the computer classroom
to ensure that they could only communicate with each
other via the online text chatroom. After the CPS activity,
the students filled up a questionnaire regarding the qual-
ity of the teamwork. Meanwhile, a learning post-test was
implemented to examine their understanding of the phys-
ics concepts after the CPS activity. The aforementioned
activities were implemented in a day in order to prevent
the students from unnecessary interference.
The individual-based simulation and collaborative
simulation
This study developed a platform (Figure 1) for students
of IS and CS groups to participate in the CPS activity.
The IS groups in the platform could (1) view the prob-
lem description and learning materials related to the
target program, (2) use a shared note panel to record
and exchange information, (3) record the results of
their CPS activity and the explanations in a group
report panel and (4) communicate with each other in
a text chatroom. The platform enabled students to
discuss with peers and develop a shared group report
but did not allow to collaboratively manipulate the
simulation. In other words, their manipulations of the
simulation were independent and did not influence
the state of the problem in others’ simulations, while
they could jointly discuss with each other via the
online text chatroom to solve the problem.
The collaborative simulations exhibited two main
differences in supporting problem solving. Firstly, the
collaborative simulation synchronized all the action that
all students performed in the simulation to help them to
be aware of the ongoing simulations, while such
synchronization was not supported in the individual-
based simulation. Secondly, the collaborative simulation
provided a mechanism to strengthen collaboration and
avoid the social loafing effect. The simulation distributed
Figure 1 The Simulation Platform Interface for the Target Problem [Colour figure can be viewed at wileyonlinelibrary.com]
4 C.-J. Chang et al.
© 2017 John Wiley & Sons Ltd

5. the control of the variables to different students, and each
student could only control a part of variables in the
simulation. The students could solve the problem only
when they closely coordinated with each other. The
individual-based simulation did not support such a
mechanism.
Data collection
Group discourse and problem-solving activities
The conversations between students in the text chatroom
were collected as discourse data to reflect the peer inter-
actions during the CPS activity. Moreover, the five
problem-solving activities on the CPS platform, includ-
ing browsing the problem description, note-taking,
manipulating the simulation, reading materials and
writing a group report were automatically recorded in
the activity logs. The group discourse data and the activ-
ity log were further analysed to understand the students’
CPS patterns.
Learning pre-test and post-test
The tests consisted of five conceptual questions in a
multiple-choice format to assess the students’ under-
standing of the science concepts related to the target
problem. The tests also included an application question
that asked the students to solve a new science problem,
which also involved both linear motion in the horizontal
direction and free fall motion. Therefore, the results of
the tests can reflect students’ conceptual understanding
of the problem, and also their ability to transfer what they
had learned to solve a new problem.
The pre-test and post-test questions were different,
but they followed a similar structure. Both the
pre-tests and post-tests aims to examine the same phys-
ics concept about the relations between three variables
[i.e. height (H), distance (D) and velocity (V)]. How-
ever, the context and content of the pre-tests and
post-tests were different. In the context of the pre-test,
an archer standing at a hill wants to shoot the target that
is located at a certain distance on the floor. The post-
tests asked students to throws two balls with two differ-
ent horizontal velocities to compare the relationship
between the locations and velocities of the two balls.
As the context of the pre-test, post-test and the CPS ac-
tivity was different, the pre-test is unlikely to interfere
with the CPS activity, and the pre-tests and post-tests
could reveal how students learned after participating
in the CPS activity. To ensure the validity of the test,
each question was literally revised by the physics
teacher of the two classes.
The conceptual and application questions were scored
separately. Students correctly answering a conceptual
question received one point. Therefore, the maximum
score for the conceptual questions was five. Students’
answers to the application question were evaluated by
the four criteria, including generating appropriate
hypothesis, listing correct equation, giving one correct
answer and giving the second correct answer. Fulfilling
any of the criteria was given one point. Therefore, the
maximum score for the application question was four.
An overall score was calculated by summing students’
score in the conceptual and application questions to
represent their overall understanding of the target
problem.
Teamwork quality questionnaire
The teamwork questionnaire (TWQ) developed by
Hoegl and Gemuenden (2001) was used in this study as
the questions of the questionnaire are closely related to
the context of this study. The TWQ was an inventory that
was presented using a five-point Likert scale (ranging
from 1, ‘strongly disagree’ to 5, ‘strongly agree’) probing
students’ engagement in the collaborative activity from
multiple dimensions. The questionnaire used in this
study involved the dimensions of communication (eight
items), coordination (four items), mutual support (six
items) and effort (four items). These dimensions could
reflect how the students perceived the collaboration qual-
ity during the CPS activity. The TWQ was translated to
Chinese, and the items were slightly modified to fit the
context of the CPS activity in this study. The translated
questionnaire has been used by a previous study by
Chung et al. (2013) to examine students’ perceptions of
teamwork quality, showing that the questionnaire was
adequately reliable. A factor analysis with the principle
components method was applied to examine the
construct validity of the modified questionnaire. The
total explained variance for the four factors was
75.00%, which was adequate for explanation. Moreover,
the overall reliability Cronbach’s α value was 0.91.
Hence, these results indicated that the modified question-
naire has adequate reliability and validity for measuring
the students’ perceptions of the collaboration experience.
Collaborative problem solving 5
© 2017 John Wiley & Sons Ltd

6. Data analysis
One of the goals of this study was to investigate the
significant CPS patterns that the students performed to
solve the problem. This study adopted a learning analyt-
ics approach to uncover students CPS patterns. The liter-
ature of CPS has applied diverse learning analytics
techniques to analyse student learning records and has
identified several critical challenges when students are
engaged in CPS activities (e.g. Lin et al., 2014; Tan,
Caleon, Jonathan, & Koh, 2014). The data of both the
student discourse and the activity logs were analysed
together to achieve this goal. The discourse data
contained utterances in sequential order. Thread analysis
(Cakir, Xhafa, Zhou, & Stahl, 2005) was applied to
segment-related utterances into threads based on group
problem-solving moves. A sequence of utterances in
the discourse was considered as a thread when these
utterances were coherently related to a specific
problem-solving sub-task based on the OECD CPS
framework (OECD, 2013). For instance, students may
show several utterances while discussing the result of a
simulation run, and these utterances were segmented as
a single monitoring/reflecting thread. The segmentation
processes of the groups were performed by two
researchers, and the agreement of the segmentation was
0.93, indicating an appropriate level of reliability. The
segmentation identified a total of 208 discourse threads
from the discourses of the IS groups and 398 discourse
threads from the CS groups. It should be noted that stu-
dents may greet and joke with peers. Such discourse
threads were not related to the problem-solving activity
and therefore were categorized as off-task threads and
excluded from the analysis. After excluding the off-task
discourse threads, the discourse threads of the IS groups
was 107, while that of the CS groups was 298.
The OECD CPS framework (OECD, 2013) was used
to categorize each discourse thread according to the
sub-tasks of the problem solving. The problem-solving
dimension involves four sub-tasks including
exploring/understanding, representing/formulating,
planning/executing and monitoring/reflecting. Table 1
lists the coding schema based on the four sub-tasks for
the discourse threads. The discourse threads were coded
by two researchers, with an inter-coder κ reliability of
0.88, indicating that the coding result was adequately
reliable. In addition, a chi-square test was used to analyse
the differences between the two groups in the thread
distributions and problem-solving activities.
To identify the significant CPS patterns, the discourse
threads and activity logs were aligned in chronological
order as a sequential dataset. The dataset was further
analysed with LSA to identify the significant CPS
patterns the students performed. Based on the probability
theory, the LSA computes the transition probability from
activity a to activity b (a → b) according to the occur-
rence of the transition. The z-score of each transition
probability was then calculated according to the expected
Table 1. The Coding Schema of Problem-Solving Dimension and Example Discourse Threads
Process Purpose Example threads
Exploring and understanding Exploring relevant information about the
scientific questions and understanding the
constraints and hints of the task.
A: What is the minimum value of the variable
Vx?
B: It’s 80. The maximum value is 150.
Representing and formulating Representing scientific concepts or
hypotheses with formulas or text.
L * L/Vx * Vx is equal to H/5.
Planning and executing Developing strategies to solve scientific
questions and executing actions in the
simulation system
A: I should observe the height of bomb that the
radar station can shoot.
B: Fix two of three variables and adjust one.
A: If the height of the aircraft is great than that
of the bomb, it definitely will be not shot. We
set up variable L and then calculate the time of
the aircraft, which is at the H height. Finally, we
calculate the value of variable V.
B: You are so professional.
Monitoring and reflecting Monitoring the outcome of executing the
simulation system or reflecting on the
possible causes of failure.
B: The aircraft was shot again; adjust the height
lower.
C: Try to increase velocity.
6 C.-J. Chang et al.
© 2017 John Wiley & Sons Ltd

7. probability of the transition. Among the transitions, only
the significant ones (z > 1.96) are displayed in the transi-
tion diagram, which illustrates the students’ CPS patterns.
We further analysed the students’ learning perfor-
mance before and after the activity to understand the
impact of the two simulations on the learning of kinemat-
ics. More specifically, a paired t-test was used to examine
the change on the students’ learning performance for
both IS and CS groups. Moreover, one-way analysis of
covariance (ANCOVA) was applied to compare the
learning performance of the IS and CS groups to under-
stand the impact of the two simulations on the learning
of kinematics. The ANCOVA, using the pre-tests scores
as the covariate, compared the post-test scores of the IS
and CS groups. Moreover, students’ feedback on the
TWQ was analysed with an independent t-test. The ques-
tionnaire results together with the students’ discourse
threads and problem-solving activities can thoroughly
display students’ engagement in the CPS activity.
Results
Descriptive data for discourse threads and
problem-solving activities
Table 2 displays the statistical results of the discourse
threads in the four problem-solving sub-tasks for the IS
and CS groups. The themes of the discussion of the
two groups were different. The CS groups showed higher
frequencies in their discourse threads than the IS groups
did (CS = 298, IS = 107). Furthermore, the chi-square
test showed that there was a significant difference be-
tween the two groups in the distribution of the four
problem-solving sub-tasks (x2
= 46.17, df = 3,
p < 0.05). The IS groups demonstrated a higher percent-
age of threads of monitoring/reflecting (78.5%) but
rarely discussed with each other the other problem-
solving sub-tasks. In contrast, the CS groups showed a
more balanced distribution in their discourse threads,
with higher frequencies of exploring/understanding
(16.4%), planning/executing (38.6%) and monitoring/
reflecting (42.6%). These results suggested that the two
simulations significantly influenced the students’ pat-
terns of engagement in the problem-solving process.
While the IS group only focused on the discussion of
the simulation results, the CS group went through all
the problem-solving sub-tasks. However, such descrip-
tive results cannot reflect the effect of the two simulations
on the students’ learning performance and their CPS pat-
terns. Hence, the discourse threads was further analysed
with LSA to investigate what types of CPS patterns the
students applied using the two simulations.
The frequencies of the five problem-solving activities
applied by the IS and CS groups are shown in Table 3.
The IS groups performed a total of 1865 problem-solving
activities, while the CS groups demonstrated only about
half (903) the number of such activities. The chi-square
test showed that there was a significant difference be-
tween the two groups in the distribution of the five
problem-solving activities (x2
= 22.53, df = 4,
p < 0.05). The IS groups mainly demonstrated the
highest frequency in the note-taking activity to keep
track of the use of the simulation (31.4%) to record the
results of their simulations as well as their reflections
on the simulation. On the contrary, the CS groups
showed the highest frequency of writing the report
(32.7%). Such results reflect that the type of simulation
had a significant impact on the students’ problem-solving
Table 2. The Distribution of Discourse Threads in the Problem-Solving Sub-tasks for the Two Groups
Groups
(n = number
of groups
and dyad)
Problem-solving process
Total χ
2
Exploring and
understanding
Representing and
formulating
Planning and
executing
Monitoring and
reflecting
IS groups
(n = 15)
10 (9.3%) 4 (3.7%) 9 (8.4%) 84 (78.5%) 107 (100%) 46.17***
CS groups
(n = 13)
49 (16.4%) 7 (2.3%) 115 (38.6%) 127 (42.6%) 298 (100%)
Total 59 11 124 211 405
Note. CS = collaborative simulation; IS = individual-based simulation.
***p < 0.001.
Collaborative problem solving 7
© 2017 John Wiley & Sons Ltd

8. activities. The IS group demonstrated more active
problem-solving activities to apply the knowledge and
keep track of the use of the simulation, while the CS
groups extensively engaged in group discussion.
LSA of the problem-solving processes of the two groups
The CPS patterns of the LSA are illustrated in Figures 2
and 3. The circles represent the problem-solving activi-
ties recorded in the activity log, while the rectangles
represent the discourse threads that appeared in the text
chatroom.
Several transitions were found in the LSA results of
the IS groups. The use of the simulation triggered many
types of activities. It often triggered them to browse the
problem description so as to understand the meaning of
the problem and further revisit the simulation. Such a
CPS pattern can be seen in the frequent transition of
‘simulating’ ➔ ‘browsing problem’ ➔ ‘simulating’.
However, after using it, individual students often sent
their peers the simulation results, which triggered the
students to collaboratively monitor and reflect upon the
problem-solving process. Such a pattern can be seen in
the frequent transition from ‘simulating’ to ‘monitoring/
reflecting’. The monitoring/reflecting discussion
promoted students’ discussion with peers to explore the
problem (‘exploring/understanding’). Such a pattern
was shown in the frequent transition from
‘monitoring/reflecting’ ➔ ‘exploring/understanding’.
These frequent transitions reveal that the IS groups expe-
rienced two stages of learning: individual exploration
with the simulation and then collaborative reflection
and exploration of the problem. The students first
explored the problem with the simulation on an individ-
ual basis to generate an initial solution to solve the prob-
lem. Following their individual exploration, they then
proceeded to the collaborative learning stage in which
they discussed their solutions to reflect upon the
problem-solving process and explore the problem to
advance their understanding.
Figure 3 displays the CPS patterns of the CS groups. It
was shown that the whole process centred on
‘planning/executing’, and frequent mutual transitions be-
tween ‘exploring/understanding’, ‘planning/executing’
and ‘monitoring/reflecting’ were detected by the LSA.
Such frequent transitions reveal that the students in the
CS group immediately proceeded to the CPS stage: in
order to find a solution, they discussed with each other,
monitored and reflected upon the process, represented
and formulated the problem, and explored the problem
Table 3. The Distribution of Problem-Solving Activities in the Simulations for the Two Groups
Groups (n = number
of groups and
dyad)
Problem-solving activities
Total (%) χ
2
Reading
materials
Browsing
problem Simulating
Note-
taking
Writing
report
IS group (n = 15) 18 (1.0%) 412 (22.1%) 284 (15.2%) 585 (31.4%) 566 (30.3%) 1865 (100%) 22.53***
CS group (n = 15) 30 (3.3%) 185 (20.5%) 129 (14.3%) 264 (29.2%) 295 (32.7%) 903 (100%)
Total 48 597 413 849 861 2768
Note. CS = collaborative simulation; IS = individual-based simulation.
***p < 0.001.
Figure 2 The Problem-Solving Pattern of the IS Groups (Individual-Based Simulation)
8 C.-J. Chang et al.
© 2017 John Wiley & Sons Ltd

9. to advance their understanding. These results reflect that
while using the collaborative simulation, the students
demonstrated multiple transitions among the problem-
solving sub-tasks.
However, it should be noted that the discussions of the
problem-solving sub-tasks were not triggered by the use
of the simulation and did not prompt the students to use
the simulation. It can be seen that the discourse threads
did not link to ‘simulating’. This means that the discus-
sion during the CPS process was not grounded on the
concrete simulation results. Instead, their discussion was
centred around collaboratively ‘planning/executing’, and
the main focus was on the coordination among peers to
find a solution. On the contrary, with the individual-based
simulation, students in the IS group tended to anchor on
the simulation results obtained by individual exploration,
and then further explored the results to find a group
solution.
These results reflect that the two simulations had a
substantial influence on the students’ CPS patterns,
suggesting the following remarkable observations:
• The IS groups started with individual exploration of
the problem with the simulation, followed by group
reflection on the results obtained from the simulation.
The CS groups did not go through an individual explo-
ration stage but immediately proceeded to the CPS
sub-tasks.
• The CPS patterns of the IS groups centred on the sim-
ulation, which prompted them to further reflect on the
problem-solving process, and triggered them to
explore the problem.
• Although the CS groups demonstrated sophisticated
transitions among various discussions of the CPS
sub-tasks, their discourse was not linked to any
problem-solving activities. Transformation of discus-
sions into workable problem-solving activities was
not seen in the CS groups’ interactions.
Learning pre-test and post-test
The learning pre-test and post-test were analysed using
paired t-test for IS and CS groups. Table 4 reveals that
Figure 3 The Problem-Solving Pattern of the CS Groups (Collaborative Simulation)
Table 4. The Results of a Paired t-Test of the IS and CS groups’ learning performance
Groups Category Test M SD t
IS group (N = 44) Overall Pre-test 3.45 2.08 7.73***
Post-test 5.84 2.62
Conceptual question Pre-test 3.07 1.28 3.13**
Post-test 3.70 1.30
Application question Pre-test 0.39 1.13 6.13***
Post-test 2.14 1.95
CS group (N = 39) Overall Pre-test 3.74 1.86 1.24
Post-test 4.21 2.12
Conceptual question Pre-test 3.15 1.31 1.35
Post-test 3.41 1.35
Application question Pre-test 0.59 1.02 0.68
Post-test 0.79 1.40
Note. CS = collaborative simulation; IS = individual-based simulation.
*p < 0.05.
**p < 0.01.
***p < 0.001.
Collaborative problem solving 9
© 2017 John Wiley & Sons Ltd

10. the IS groups demonstrated a significant improvement
in their overall performance (t = 7.73, p < 0.05).
More specifically, the improvement could be seen in
their scores in the conceptual test (t = 3.13,
p < 0.05) and application test (t = 6.13, p < 0.05).
The results indicated that the IS groups not only
enhance their conceptual understanding of the problem
but also were able to apply the concepts they learned
to solve a similar problem. However, such improve-
ment was not found in the CS groups in terms of the
overall performance (t = 1.24, p > 0.05), conceptual
test (t = 1.35, p > 0.05) and application test
(t = 0.68, p > 0.05).
The post-test scores of the IS and CS groups were
further analysed with one-way ANCOVA using pre-test
scores as the covariate. Table 5 reveals that the IS groups
demonstrated significantly higher scores in the overall
test than the CS groups did (F = 14.28, p < 0.05). More
specifically, the IS groups exhibited significantly higher
scores in the application test than the CS groups did
(F = 13.50, p < 0.05). However, no significant difference
was not found in the conceptual test between the two
groups (F = 1.86, p > 0.05). These results revealed that
students using individual-based simulation demonstrated
a significant enhancement in their ability to apply the
concepts they learned to solve a similar problem after
experiencing the CPS activity.
Students’ feedback on the TWQ
The students’ feedback obtained from the TWQ showed
that no significant differences were found between the
two groups in terms of communication (t = 0.86,
p > 0.05), mutual support (t = 0.65, p > 0.05) or group
effort (t = 1.16, p > 0.05). However, the CS groups
perceived a significantly higher level of coordination
than the IS groups did (MCS = 4.03, MIS = 3.73,
t = 1.97, p ≤ 0.05). The students in the CS groups
perceived a higher level of coordination in the two items:
‘My team members and I fully comprehended the goals
of the sub-tasks and each individual’s responsibility
within our team’ (t = 3.74, p < 0.05) and ‘Team mem-
bers accepted the goals of the sub-tasks and tried our best
to achieve the goals’ (t = 1.87, p = 0.07). These results
revealed that the collaborative simulation strengthened
the interdependency between members and prompted
the students to coordinate with each other to jointly solve
the target problem. Such a result implies that the collab-
orative simulations demonstrated a positive effect on the
group coordination during the CPS activity.
Discussion and implications
The results of this study found both affordances and lim-
itations of individual-based and collaborative simula-
tions in answering the research questions of this study.
Firstly (RQ1), the IS group demonstrated more active
problem-solving activities to apply and keep track of
the use of the simulation, while the CS groups exten-
sively engaged in group discussion. Secondly (RQ2),
the students using the collaborative simulation demon-
strated sophisticated transitions among various discus-
sions of CPS sub-tasks, and also perceived and showed
a higher level of coordination in their discussion threads
than the students using the individual-based simulation.
Lastly (RQ3), the IS groups showed a significant
enhancement in their conceptual understanding and
application ability after the CPS activity while the CS
group did not. The results of the three research questions
may reflect the pedagogical issues of CPS activities to
support science learning including positive interdepen-
dence, problem-solving strategy and the integration of
individual learning and collaborative learning.
Table 5. The Results of One-Way ANCOVA of the Test Scores
Category Groups Adjusted mean Standard error F
Overall IS (N = 44) 5.92 0.33 14.28***
CS (N = 39) 4.12 0.35
Conceptual question IS (N = 44) 3.73 0.17 1.86
CS (N = 39) 3.39 0.18
Application question IS (N = 44) 2.16 0.26 13.50***
CS (N = 39) 0.77 0.27
Note. ANCOVA = analysis of covariance; CS = collaborative simulation; IS = individual-based simulation.
***p < 0.001.
10 C.-J. Chang et al.
© 2017 John Wiley & Sons Ltd

11. Positive interdependence (RQ1)
The first noticeable difference is the CPS patterns they
demonstrated in solving the target problem. The results
indicated that the IS groups demonstrated higher
frequencies of problem-solving activities (IS = 1,865,
CS = 903) such as using the simulation and taking notes.
However, they demonstrated only a very limited level of
coordination to plan and execute their simulation. On the
contrary, the CS groups displayed more intensive discus-
sion threads (CS = 298, IS = 107). The students using the
collaborative simulations frequently coordinated with
their peers in order to plan actions to solve the problem.
Such result is consistent with the result of the TWQ,
showing that the CS groups perceived a higher level of
coordination than the IS groups did.
These results reflect that the collaborative simulation
may have facilitated students’ coordination, which may
have been the result of the collaboration protocol and
the joint attention afforded by the collaborative simula-
tion. Previous studies by Brush (1998) and Zurita and
Nussbaum (2007) indicated that giving clear individual
accountability to each student can enhance positive inter-
dependence that triggers students’ coordination with
each other to achieve the group goals. When using the
collaborative simulations, each student participated in
the collaborative activity with different capacities and
accountability in the simulation. The literature has
suggested that it is necessary to maintain a between-
person state of engagement to intertwine social and
cognitive factors in collaborative learning (Barron,
2003). Collaborative simulation might have been helpful
for establishing joint attention among students to the
shared problem space, as they closely coordinated each
other’s actions to operate the collaborative simulation
and demonstrated a higher level of coordination in the
process than the IS groups did.
Problem-solving strategy (RQ2)
The results of this study revealed a remarkable difference
between the IS groups and CS groups in their problem-
solving process. IS groups considerably centred on the
manipulation of the simulation, which informed them
to further reflect on the problem-solving process and trig-
gered them to explore the problem. In contrast with IS
groups, the discussion of the CS groups did not anchor
on a concrete simulation result.
A previous study by Lin et al. (2013) identified three
main epistemic activities for solving problems: firstly,
constructing a conceptual space to establish the concep-
tual relations among variables; secondly, constructing a
problem space to understand the problem condition,
constraints, and the goal of the problem; and thirdly,
constructing relations between the conceptual and prob-
lem space to generate a solution to solve the problem.
Another study by Lin et al. (2014) found that certain
problem-solving patterns may lead to students’ inability
to apply the three epistemic activities. They identified
two main problem-solving strategies including
manipulation-centred strategy and discussion-centred
strategy. More specifically, their study found that
students who used manipulation-centred strategy to solve
problems spent more time conducting tests with simula-
tions, and that their discussion included understanding,
application and analysis of the simulation results. On
the contrary, the discussion of the students who applied
discussion-centred strategy lacks analysis of the simula-
tion results to link their conceptual understanding to the
manipulation of the simulation.
The problem-solving process that the IS groups
performed resembles a manipulation-centred strategy as
identified by Lin et al. (2013), which contrasts with the
discussion-centred strategy. On the contrary, the CS
groups adopted a discussion-centred strategy to solve
the problem, and the connection between the discussion
and simulations was not observed. Although they
demonstrated frequent discussion on the problem-
solving sub-tasks, more coordinative discussion does
not necessarily indicate better collaboration (Meier,
Spada, & Rummel, 2007). The discussion-centred
collaboration was often not grounded on concrete simu-
lation results, and transfers between different levels of
problem-solving tasks were seldom observed (Lin
et al., 2013).
It was found that the CS groups were unable to transfer
their discussion into executable problem-solving steps,
although they demonstrated extensive discussion on the
problem-solving tasks. Such a result may be related to
the lack of effective ‘transactivity’ (Weinberger, 2011),
indicating students’ inability to reciprocally build their
understanding based on the argument of their peers and
a lack of appropriate scaffolding. In this study, the
collaborative simulation only provide scaffolding to
guide the students to perform the major tasks (problem
description, simulation, group report, notes, and learning
Collaborative problem solving 11
© 2017 John Wiley & Sons Ltd

12. material) at a scene level according to the scaffolding
levels identified by Vogel, Wecker, Kollar, and Fischer
(2016). It did not effectively guide students to recipro-
cally improve their understanding about the problem
based on the simulation results and peers’ reactions. It
is suggested that appropriately designed microscripts
are necessary to support the use of group discussion that
is anchored around the problem of understanding. Within
a knowledge-building activity, the contributions of indi-
viduals and the exchange of views need to be knowledge
based rather than activity based (Nussbaum et al., 2009).
The integration of individual learning and collaborative
learning (RQ3)
The results of the study found that the IS groups exhib-
ited a significant enhancement in their ability to apply
the concepts they learned to solve a similar problem.
Furthermore, the study found that the IS groups went
through individual exploration with the simulation,
followed by group reflection on the results obtained from
the simulation. On the contrary, the CS groups did not go
through an individual exploration stage but immediately
proceeded to the discussion of the CPS sub-tasks. A
previous study indicated that most students when partic-
ipating in collaborative learning need to go through indi-
vidual learning activities before proceeding to
collaborative learning activities at the initial stage (Sun
& Looi, 2013). It was also found that efficient collabora-
tive learning involves frequent transition between indi-
vidual learning, such as self-exploration and reading,
and collaborative learning such as discussion and joint
works (Chang et al., 2012). In this sense, an individual
space that supports individual students to learn, and a
joint space where a group of students can explore a prob-
lem together are both crucial in the process of collabo-
rative learning (Jeong & Chi, 1997; Hermann,
Rummel, & Spada, 2001). Although the collaborative
simulation provided a joint space for the students to
manipulate the simulation, students using the collabora-
tive simulation could not conduct individual learning as
individual exploration may have been obstructed by the
group manipulation in the collaborative simulation.
Such a limitation may partially explain the reason
why the CS groups did not improve their scores in
the application test after the CPS activity. This finding
echoes the principle of the balance of individual learn-
ing and collaborative learning asserting that a well-
integrated proportion and order of individual and col-
laborative learning phases is vital for successful collab-
oration (Rummel & Spada, 2005).
Conclusion and future work
The results of this study revealed different affordances of
the individual-based and collaborative simulations on
CPS process, learning performance and TWQ.
Regarding the collaborative simulation, CS groups dem-
onstrated a close coordination and applied a discussion-
centred strategy to solve the problem. However, they
had difficulty in transferring their discussion into execut-
able problem-solving step, and thus their learning perfor-
mance on the target problem was not improved after the
activity. On the contrary, the IS groups did not demon-
strate extensive coordination on the problem-solving
activity. They adopted a manipulation-centred strategy
to solve the problem. Although they did not frequently
coordinate to execute the simulation, they centred their
discussion on the simulation results to better solve the
problem. Thus, their discussion was reflective and thus
helped them to achieve a better understanding of the
problem.
The findings of this study provide several design
guidelines for facilitating CPS. Firstly, the CPS activities
supported by collaborative simulation can strengthen
interdependence among group members and thus
promote student participation in collaborative learning.
Secondly, the design of the CPS activities should care-
fully leverage both individual and collaborative learning.
On the one hand, the design should allow students to
conduct individual learning, avoiding interference from
peers in the collaborative simulation. On the other hand,
the result of the individual learning can become the basis
of group discussion to augment the collaborative learn-
ing effect. Last but not least, it was found that the
students using the collaborative simulation often adopted
a discussion-centred strategy to solve the problem; how-
ever, they could not transform their discussion into
executable problem-solving steps. Further studies are
necessary to develop computer-mediated microscripts
to guide students closely relate their discussion to both
the simulation and peers, and thus consolidate their
discussion to form executable problem-solving steps.
The results of this study contribute to a more compre-
hensive understanding of the affordances and limitations
of the collaborative and individual-based simulations.
12 C.-J. Chang et al.
© 2017 John Wiley & Sons Ltd

13. Owing to the time limitation of this study, the data
included only the activity logs and discussions in the
students’ first encounter with a CPS activity. It is not
clear whether students’ CPS patterns will evolve if they
participate in such CPS activities multiple times. Future
research is required to investigate the temporal CPS
patterns that can reveal the effect of long-term implemen-
tation of CPS activities. Furthermore, further studies may
be required to obtain an effective model to integrate indi-
vidual and collaborative learning so as to achieve a better
collaborative learning effect. It should be noted that the
CPS activity involved in this study focused on the appli-
cation of scientific knowledge to solve a problem. It
would be interesting to understand how individual and
collaborative simulations can be used to support scien-
tific inquiry in which students work together to under-
stand a science phenomenon. Gathering information on
these issues through further studies can help obtain a
thorough understanding of student learning patterns
during CPS activities, which would inform educators
and teachers in their design and pedagogical decisions
on the use of simulations in science learning.
Acknowledgements
This research was partially funded by the Ministry of
Science and Technology under contract numbers
104-2511-S-008-014-MY3 and 106-2811-H-008-003.
References
Bakeman, R., & Gottman, J. M. (1997). Observing interaction:
An introduction to sequential analysis (2nd ed.). New York:
Cambridge university press.
Barron, B. (2000). Achieving coordination in collaborative
problem-solving groups. The Journal of the Learning
Sciences, 9, 403–436.
Barron, B. (2003). When smart groups fail. The Journal of the
Learning Sciences, 12, 307–359.
Bell, T., Urhahne, D., Schanze, S., & Ploetzner, R. (2010). Col-
laborative inquiry learning: Models, tools, and challenges.
International Journal of Science Education, 32, 349–377.
Brush, T. A. (1998). Embedding cooperative learning into the
design of integrated learning systems: Rationale and guide-
lines. Educational Technology Research and Development,
46, 5–18.
Çakır, M. P., Zemel, A., & Stahl, G. (2009). The joint organiza-
tion of interaction within a multimodal CSCL medium.
International Journal of Computer-Supported Collaborative
Learning, 4, 115–149.
Cakir, M., Xhafa, F., Zhou, N., & Stahl, G. (2005). Thread-
based analysis of patterns of collaborative interaction in chat.
Proceedings of the 12th international conference on
Artificial Intelligence in Education (AI-Ed 2005),
Amsterdam, The Netherlands, pp. 120–127.
Care, E., Scoular, C., & Griffin, P. (2016). Assessment of
collaborative problem solving in education environments.
Applied Measurement in Education, 29, 250–264.
Chang, C. J., Liu, C. C., & Shen, Y. J. (2012). Are one-to-one
computers necessary? An analysis of collaborative web
exploration activities supported by shared displays.
Educational Technology & Society, 15, 3–13.
Chung, C. W., Lee, C. C., & Liu, C. C. (2013). Investigating
face-to-face peer interaction patterns in a collaborative Web
discovery task: The benefits of a shared display. Journal of
Computer Assisted Learning, 29, 188–206.
de la Torre, L., Heradio, R., & Dormido, S. (2013). Real-time
collaboration for web-based labs. Proceeding of the 10th
International Conference on Computer Supported Collabo-
rative Learning (CSCL 2013), Madison, USA, pp. 161–168.
Dillenbourg, P., & Traum, D. (2006). Sharing solutions: Per-
sistence and grounding in multimodal collaborative prob-
lem solving. The Journal of the Learning Sciences, 15,
121–151.
Gijlers, H., Saab, N., Van Joolingen, W. R., De Jong, T., &
Van Hout-Wolters, B. H. A. M. (2009). Interaction between
tool and talk: How instruction and tools support consensus
building in collaborative inquiry-learning environments.
Journal of Computer Assisted Learning, 25, 252–267.
Hao, J., Liu, L., von Davier, A., & Kyllonen, P. (2015).
Assessing collaborative problem solving with simulation
based task. In Proceedings of the 11th international
conference on computer supported collaborative learning,
Gothenburg, Sweden, 7–11 June.
Hermann, F., Rummel, N., & Spada, H. (2001). Solving the
case together: The challenge of net-based interdisciplinary
collaboration. In P. Dillenbourg, A. Eurelings & K.
Hakkarainnen (Eds.), Proceeding of the First European
Conference on Computer-Supported Collaborative Learning
(pp. 293–300). Maastricht: McLuhan.
Hmelo-Silver, C. E., Jordan, R., Liu, L., & Chernobilsky, E.
(2011). Representational tools for understanding complex
computer-supported collaborative learning environments. In
S. Puntambekar, G. Erkens & C. Hmelo-Silver (Eds.),
Analyzing interactions in CSCL: Methods, approaches and
issues (pp. 83–106). New York, NY: Springer.
Hoegl, M., & Gemuenden, H. G. (2001). Teamwork quality
and the success of innovative projects: A theoretical con-
cept and empirical evidence. Organization Science, 12,
435–449.
Collaborative problem solving 13
© 2017 John Wiley & Sons Ltd

14. Jara, C. A., Candelas, F. A., Torres, F., Dormido, S., &
Esquembre, F. (2012). Synchronous collaboration of virtual
and remote laboratories. Computer Applications in Engineer-
ing Education, 20, 124–136.
Jeong, H., & Chi, M. T. H. (1997). Construction of shared
knowledge during collaborative learning. In R. Hall, N.
Miyake & N. Enyedy (Eds.), Proceedings of the Second In-
ternational Conference on Computer-Support for Collabora-
tive Learning (pp. 124–128). Canada: Toronto.
Jermann, P., & Dillenbourg, P. (2008). Group mirrors to
support interaction regulation in collaborative problem solv-
ing. Computers & Education, 51, 279–296.
Lin, P. C., Hou, H. T., Wu, S. Y., & Chang, K. E. (2014).
Exploring college students’ cognitive processing patterns
during a collaborative problem-solving teaching activity in-
tegrating Facebook discussion and simulation tools. The
Internet and Higher Education, 22, 51–56.
Lin, T. J., Duh, H. B. L., Li, N., Wang, H. Y., & Tsai, C. C.
(2013). An investigation of learners’ collaborative knowl-
edge construction performances and behavior patterns in an
augmented reality simulation system. Computers &
Education, 68, 314–321.
Liu, C. C., Chung, C. W., Chen, N. S., & Liu, B. J. (2009).
Analysis of peer interaction in learning activities with
personal handhelds and shared displays. Educational
Technology & Society, 12, 127–142.
Liu, C. C., Fan Chiang, S. H., Chou, C. Y., & Chen, S. Y.
(2010). Knowledge exploration with concept association
techniques. Online Information Review, 34, 786–805.
Liu, C. C., Tao, S. Y., Chen, W. H., Chen, S. Y., & Liu, B. J.
(2013). The effects of a Creative Commons approach on col-
laborative learning. Behaviour & Information Technology,
32, 37–51.
Meier, A., Spada, H., & Rummel, N. (2007). A rating scheme
for assessing the quality of computer-supported collaboration
processes. International Journal of Computer-Supported
Collaborative Learning, 2, 63–86.
Nussbaum, M., Alvarez, C., McFarlane, A., Gomez, F., Claro,
S., & Radovic, D. (2009). Technology as small group face-
to-face collaborative scaffolding. Computers & Education,
52, 147–153.
Organisation for Economic Co-operation and Development.
(2013). PISA 2015: Draft collaborative problem solving
framework. https://www.oecd.org/pisa/pisaproducts/Draft%
20PISA%202015%20Collaborative%20Problem%20Solving
%20Framework%20.pdf. Accessed 7 October 2016.
Rummel, N., & Spada, H. (2005). Learning to collaborate: An
instructional approach to promoting collaborative problem
solving in computer-mediated settings. The Journal of the
Learning Sciences, 14, 201–241.
Rutten, N., van Joolingen, W. R., & van der Veen, J. T. (2012).
The learning effects of computer simulations in science edu-
cation. Computers & Education, 58, 136–153.
Saab, N., van Joolingen, W., & van Hout-Wolters, B. (2012).
Support of the collaborative inquiry learning process: Influ-
ence of support on task and team regulation. Metacognition
and learning, 7, 7–23.
Stahl, G. (2006). Supporting group cognition in an online math
community: A cognitive tool for small-group referencing in
text chat. Journal of Educational Computing Research, 35,
103–122.
Sun, D., & Looi, C. K. (2013). Designing a web-based sci-
ence learning environment for model-based collaborative
inquiry. Journal of Science Education and Technology,
22, 73–89.
Swaray, R. (2012). An evaluation of a group project designed to
reduce free-riding and promote active learning. Assessment
& Evaluation in Higher Education, 37, 285–292.
Tan, J. P. L., Caleon, I. S., Jonathan, C. R., & Koh, E. (2014). A
dialogic framework for assessing collective creativity in
computer-supported collaborative problem-solving tasks.
Research & Practice in Technology Enhanced Learning, 9,
411–437.
Vogel, F., Wecker, C., Kollar, I., & Fischer, F. (2016). Socio-
cognitive scaffolding with computer-supported collaboration
scripts: A meta-analysis. Educational Psychology Review,
1–35.
Weinberger, A. (2011). Principles of transactive computer-
supported collaboration scripts. Nordic Journal of Digital
Literacy, 6, 189–202.
Yamauchi, Y., Yokozawa, M., Shinohara, T., & Ishida, T.
(2000). Collaboration with lean media: How open-source
software succeeds. Proceedings of the 2000 ACM
Conference on Computer Supported Cooperative Work,
Philadelphia, Pennsylvania, USA, pp. 329–338.
Zurita, G., & Nussbaum, M. (2007). A conceptual framework
based on activity theory for mobile CSCL. British Journal
of Educational Technology, 38, 211–235.
14 C.-J. Chang et al.
© 2017 John Wiley & Sons Ltd