Evaluation socio-technical environments integrated frameworks

Evaluation
socio-technical environments
integrated frameworks
Chronis Kynigos, CTI
MCSquared FP7 Project 610467, 1st Review Meeting, 06 November, 2014, Utrecht, Netherlands 1

‘Evaluation’ in mc2 project
Focus Approach
•Social creativity in the
design of educational
resources for CMT
•the two ‘creativities’
•social creativity in design
•Creative mathematical
thinking
•Design-based research paradigm
•Creativity as
•process
•product
•context
•Bottom-up / theory-driven
•Formative/ summative
•Integrative
•3 cycles of work
•4 (x3) empirical studies
•meta-analysis & synthesis

Socio-technical environments
consist of a community of individuals engaged
in a joint enterprise, whose creative
performance is mediated through
interactions with a particular ‘technical’
environment designed to amplify the
outcome of their collaborative efforts

Integrated Frameworks
• The medium integrates
• narrative/reading with
• expressivity/constructionism
• The evaluation integrates
• DG and BC to understand SC and
CMT affordances
• Diverse operationalizations of
DG/BC integrations

Communities of Interest
• heterogeneity, diverse expertise in
their members
• Bringing representations from
diverse CoP
• fueled by a culture of participation
• exemplified in social processes,
practices and mediations
• leading to novel/new, useful and
desirable outputs

CoI as SC enhancers
• Heterogenous (members are representatives of CoPs)
• to solve a design problem of common concern (joint “interest”) that
requires the combination of multiple knowledge systems
• a rich environment for interconnectedness, different perspective-taking,
knowledge exchange and integration between diverse
domains and more opportunities for creative thinking and learning
• CoI members may be participants in more than one CoPs
• One community can exhibit characteristics of both a CoI and a
CoP
• can integrate aspects of both a CoI and a CoP its type may shift
over time due to external factors

Collaborative design:
DG and BO
Documentational genesis
• Living documents
• Instrumentation and
instrumentalization
Digital artifacts as boundary
objects
• Boundary crossings
• Improvable objects

Integrations
CoI Diversity in operationalization of the integration between DG
and BC

Starting points for S-C and CMT
• Narrative activity model (Vygotsky)
Exploration, inspiration, production, sharing
• Creative thinking spiral (Resnick)
Imagination Creation, Play, Sharing, Reflection, imagination
• Lateral, out of the box, playfulness, humor, imagination (Ackerman,
DeCortis et al)
• Model of Collaborative Creativity (Aragon, Williams)
Focus, Frame, Create, Complete
• Product-process, general-specific, solving-posing problems interplay, ill
structured open problems (Shriki, Jamir-Leikin)
• Creativity in techno-mathematical literacies and the importance of
modeling and boundary crossings and artifacts as boundary objects
(Hoyles-Noss)

‘Evaluation’ in MC2 project: progress in Year 1
Theory grounding
and integration
Research design
and analysis
‘Integration’ tools
(templates)
• lit review on s-c/ CMT
• ‘landscape’ of related
concepts and approaches
• draft versions of D2.2 / D2.3
• operational definition
• research questions
• units of analysis
• levels of analysis
• methodological tools
• for sharing, reporting,
reflecting, meta-analysing
• for contextualising
evaluation
• for theory integration

Evaluating social creativity in Cycle 1:
theory grounding and integration
Social creativity in
design
(Fischer, 1997, 2007)
Boundary
crossing
approach
(Akkerman &
Bakker, 2011)
Documentational
approach
(Gueudet &
Trouche, 2009)
The 3 theory strands
A landscape of related concepts

Operational definition of ‘social creativity’
 a design activity:
• structured, intentional (create edu resources for CMT)
• collective, participative
 embedded and nurtured within a socio-technical environment
• CoI (collective of professionals involved in the design of math edu
resources)
• C-book technology (authoring tool, widget factories, data-analytics
engine, CoICode)
 involving a set of processes and interactions:
• socio-cognitive / group processes (within the CoI)
• and the CoI with the C-book technology
 aiming at/ resulting in the production of ‘creative’ outputs:
• ideas born/ shaped in the interactions, and
• c-book units, which are:
 novel/new
 useful
 elaborate
 appropriate, w/r to the joint goal, and for the end-users

Evaluation studies of social creativity in
Cycle 1: research design and analysis
Common research
questions
• CRQ.1: How is s-c
stimulated/ enabled in
mc2 S-T env?
• CRQ.1: What kinds of s-c
emerge from the
interactions between CoI
and C-book tech?
Categories of data
collected
• Asynchronous discussion
transcripts from CoiCode
• Recorded conversations
from f-2-f / skype meetings
• CoiCode graphs, system
logs, user entries
• The c-book units

Evaluation studies of social creativity in
Cycle 1: focus of analysis
Focus of analysis
Selected segments of an
activity with a single theme
as a focus, highlighting an
aspect of social creativity
‘critical episodes’ of social creativity
 a first lens to filter collected data
 a mechanism to bring to the fore the
different approaches by the research teams
 a contribution to our understanding of s-c

Evaluating social creativity in Cycle 1: analysis
of critical episodes
foci of critical episodes
• the ‘idea’ concept
• stages in the creative
process
• interactions within S-T
environments
• the c-book unit’s
function
main and recurrent themes
identified
• prevalence of an idea if linked to
a widget instance
• enablers of s-c:
• CoI synthesis (knowledge and
expertise complementarity)
• role of ‘boundary brokers’
• the c-book unit’s unified structure
• the c-book unit serving as a
‘boundary object’, encompassing
various considerations (math, ped,
tech, context)

Refining evaluation design for Cycles 2, 3:
units – levels – tools of analysis
•The creative
path of an
idea
•CMT
affordances
of the c-book
unit
•The identity
of a
creative
idea
•Modes of
interactions
with the C-book
tech
the idea the idea
the c-book
the c-book

“The c-idea ID card”
The ID card of an idea
Idea “theme” What the idea refers to: e.g. a mathematical
idea, negotiation of the age of the students,
the nature of the narrative of the c-book unit,
e.t.c.
Idea initiator
Number of idea
commentators
Duration of discussion

“The creative idea indicators checklist”
Creative idea indicators YES NO Comments
The idea is supported by a widget.
If yes, when (at what point in the discussion, for example: third post of the initiator) does the widget

appear?
The idea is “native” to the CoI (an emerging product of CoI discussion, not previously used in
another resource system).

The idea is “immigrant” to the CoI (has been previously used by the initiator in another context / comes
from an alien resource system).

The idea is integrated in the CoI resource system, leading to some reorganisation.
(If no), when is the idea abandoned?
(If yes), number of consequent evolving versions of the same idea.
(If yes), how many CoI members were involved in the elaboration of the idea?
(If yes), time elapsed from the introduction of the initial idea till the common approval of its last version
Socio-cognitive processes processes employed by the CoI members in the deployment of the idea from

its first wording to its final development
One or more CoI members have explicictly characterised the idea as novel, unusual or new, at some
point in the discussion.

One or more CoI members have explicitly characterised the idea as appropriate (conforming to the
users’ requirements), at some point in the discussion.

The discussion of the idea is continuous (discussion flows in non-interrupted threads).
The discussion of the idea includes the element of questioning / challenging / disagreement.
The discussion of the idea indicates elements of one or more processes of boundary crossing.


“The CMT affordances evaluation grid”
The CMT affordances evaluation grid
Does the c-book unit engage the ‘reader’ in enough learning situations
involving…
YES NO JUSTIFY (when
necessary)
• open problems/open questions ?
• problem-posing ?
• simulation ?
• the application of other (non-mathematical) subject domains ?
• the combination of more than one mathematical areas(or concepts) ?
• alternative uses of familiar digital tools ?
• real-life mathematics ?
• the combination of more than one available widgets ?
• multiple representations ?
• constructionist activities ?
Does the c-book unit trigger in the ‘reader’ a sense of …
• pleasure/ amusement/ enthusiasm ?
• being motivated / challenged ?
• gamification/ playfulness ?
• beauty in mathematics ?
• personal engagement ?
Does the c-book unit give the ‘reader’ enough opportunities for…
• collaboration/ communication/ Interaction with others ?
• discussion ?
• free-choice learning ?
• competition ?
• exploration ?
• creating a “story”?
• experimentation ?
Does the c-book unit allow / engage/ faciltate the ‘reader’ to…
• formulate her own mathematical assumptions ?
• take initiatives ?
• to find multiple solutions to the same problem ?
• to use multiple paths/ strategies to get to the solution ?
• to apply various forms of reasoning ?
• to get to unexpected solutions ?
• to create new products/ original ideas ?
• to apply various math techniques ?
• to reflect on the solution ?

Evaluating CMT in the 3 cycles of CoI work
Cycle 1
•Identifying the
CMT initial
representations
and criteria of
the CoI
members/
designers
Cycle 2
•CoI members
self-evaluate
and cross-evaluate
the
CMT
affordances of
their c-book
units
Cycle 3
•CMT
affordances of
c-book units
are evaluated
in the small
case-studies
with students
The CMT
affordances
evaluation grid
The revised CMT
affordances
evaluation grid

The ‘templates’: tools for supporting evaluation,
for meta-evaluation and theory integration
The 7 ‘templates’: 5 (wp6), 1 (wp7) and 1 (wp5)
Purpose and intended use
• Description tools
• Support partners to
• design and report work
• externalise, exchange,
and reflect on their
approaches
• ‘Living documents’
• ‘Boundary objects’
• Analysis and integration tools
• Completed templates are
used as raw material
• for meta-analysing and
contextualising the
evaluation studies (across
the 4 CoIs/ research teams)
• for explicitation of research
perspectives and synthesis
of evaluation findings

The ‘templates’: tools for supporting evaluation,
meta-evaluation and theory integration
CoI
constitution/
function
The CoI profile
template
The CoI
moderation
template
CoI activity
report
The c-book unit
description
template
The cross-case
analyses
template
Evaluation of
the two
‘creativities’
The S-C
evaluation
template
The CMT
evaluation
template
Evaluation of
the C-book
affordances
The c-book
analytics
requirements
template

The “CoI profile” template:
Structure and rationale:
• The CoI identity
• membership, diversity and
size of the CoI
• configuration of the CoI
• CoI 'interest' and
sustainability
• The CoI role and workplan
in Cycle 1 (or 2, 3 )
• intended role of the CoI
• aspired work plan of the
CoI

Excerpts: membership & configuration of the CoIs
Communities of Practice Number of
representatives
1. Researchers in Education & Mathematics Education 5
2. School teachers & Teacher trainers 4
3. Technology developers and users in Education 3
4. ‘Experts’ on the dissemination of knowledge 2
5. Creative Industry 2
6. ‘Experts’ outside Maths Education 2
Spanish CoI
The Community of Interest (CoI) is
composed, on the one hand, of several
communities of practice (CoPs) involved in
the teaching and learning of Mathematics
and, on the other hand, of external groups
outside Maths Education. This last CoP
gives rise to different contexts link to other
fields of knowledge, promoting an
interdisciplinary and creative interaction.
Communities of Practice Number of
representatives
1. Researchers in Education & Mathematics Education 6
2. Teacher educators 6
3. Game developers 1
4. Teachers 6
5. Mathematicians 1
6. Computer Scientists 2
UK CoI
The final CoI represents various groups
involved and interested in themes involving
maths education, teaching, research,
design and creative industries. There is
some overlap in the various roles. Each
CoP will eventually be represented by at
least two, CoP active, unique people.

The “Social creativity evaluation” template
Structure and rationale:
1. Operational definition for
Social Creativity
2. Common Research Questions
3. Specific Research Questions
4. Theoretical grounding of the
study
5. Research approach towards
Social Creativity
6. Context of each CoI-specific
study
7. Categories of data and data-collection
techniques
8. Data analysis techniques

Excerpts from the completed templates - 1
Research Approach to Social Creativity
The UK CoI will use c-books as
‘improvable boundary objects’.
Therefore, one could first typify it as a
product approach. However, because
every actor involved in the evolution of
every c-book, from embryonic stage to
first version to later versions, records
his/her activities every ‘product’ is
accompanied by a ‘process’
describing exactly how the c-book
evolved. This product/ process
combination is documented by a rich
multimodal description.
To analyze social creativity we propose to
use the “componential theory of
creativity” from T.M. Amabile, 2012: “The
componential theory of creativity is a
comprehensive model of the social and
psychological components necessary for
an individual to produce creative work. […].
In this theory, four components are
necessary for any creative response: three
components within the individual -
domain-relevant skills, creativity-relevant
processes, and intrinsic task motivation -
and one component outside the individual -
the social environment in which the
UK research team individual is working.”
French research team

Excerpts from the completed templates - 2
Common Research Question 1: How is social creativity stimulated and/or
enabled within the mc2 socio-technical environment, as this is constituted by
each of the four CoIs interacting with the c-book technology with the aim to
design a series of c-book units?
Specific Research Questions
Greek
Research team
S.R.Q1: How do (boundary crossing)
interactions between CoI members
with a different knowledge domain
(i.e. mathematics and
environmental education or
engineering education), affect the
CoI's performance in terms of social
creativity?
French
Research team
S.R.Q.1: What elements/features of
the c-book infrastructure are
critical to foster social creativity
among the CoI members?

Evaluation socio-technical environments integrated frameworks

More Related Content

Similar to Evaluation socio-technical environments integrated frameworks

More from MCSquaredProject

Recently uploaded

Evaluation socio-technical environments integrated frameworks

Editor's Notes