The impact of Science Literacy delivery methods - what works?
Single mechanism analysis Working Paper
Festivals | Group 4. Activities and services
V1.0 | 10 December 2018
2. ii
Executive Summary
1. Introduction
1.1 This report presents a synthesis of the proven impact, strengths and weaknesses of
makerspaces in delivering science literacy.
1.2 This individual analysis is situated within the framework of a broad study of science literacy
aimed to establish what has been proven successful in the field; with the objective to
promote and adapt good practices and fill gaps in knowledge about ‘what works’.
1.3 The full study identified 42 single-mechanism approaches, 2 composite approaches and 1
related approach. ‘Makerspaces’ is categorized within Group 4 relating to ‘Activities and
Services’.
2. Methodology for resource discovery and analysis
2.1 From October 2017 to May 2018, the research team surveyed existing resources through
retrieval via research databases, subject databases, open access repositories and through
contact with interested organisations, institutions and individuals.
2.2 The resources were divided into impact assessments (IAs) and descriptive resources. For the
purposes of analysis, only those published during the years 2013 -2018 were utilised. Each
resource was read in detail, significant data was extracted and entered into a specifically
developed database. An example of the database mask is included in Appendix A.
2.3 Although the total number of resources located was not designed to be exhaustive or
definitive, the resources captured in this research are limited to those available in the
English language and to translations that had already been made from other languages into
English.
3. Overview of results
3.1 Over 2,100 IA studies and descriptive resources were identified in the full research process,
of which 27 relate specifically to ‘makerspaces’; of which 21 were published between 2013-
2018 (May). In view of the potential importance of makerspaces in delivery of science
literacy, consideration is now being given to an updated search to cover March to December
2018.
3.2 The subject coverage included science, technology, engineering and mathematics (STEM),
mathematics and broadly science education. The countries included in the studies were
United States of America (16), but also with examples from Australia (4), Canada (1), Estonia
(1), Indonesia (1), Mexico (1) and New Zealand (1).
3.3 The delivery models involved were implemented for two-thirds in formal education (66.7%)
and or one-third in non-formal education (33.3%). The target audiences were 66.7%
education and training, 29.6% population groups and 3.7% workers. The audiences were
reached through educational institutions (85.7%), ‘others’ (9.5%) and ‘various’ (4.8%).
3.4 The approaches to conducting assessment within the resources were found to be equally
mixed-method and qualitative, followed by quantitative. The most common data collection
approaches involved written or online surveys.
3. iii
4. Discussion
4.1 The makerspace phenomenon has morphed into three readily identifiable types
characterised by accessibility: dedicated, distributed, and mobile. Initially emerging from
universities, makerspaces are now found in locations ranging from industrial estates to high
streets, schools, museums and libraries.
4.2 Despite increased interest by educators of the potential of makerspaces in engaging
students in active learning, there is a lack of empirical evidence to demonstrate the
effectiveness, content and processes of learning in makerspaces.
4.3 The resources presented in this analysis offer a selection of current studies and impact
assessments involving makerspaces based on resources published from January 2013 to
March 2018. This report will be subsequently updated to include studies published between
April to December 2018.
4.4 The apparent dearth of recent impact assessments may indicate an urgent need for further
research on approaches to assess learning in makerspaces.
4.5 The reviewed studies range from primary school STEM-related studies (in Indonesia and
Australia), contextualized mathematics education for African American elementary and
middle-school students in the United States, through to university settings, most frequently
for undergraduate engineering students, but also pre-nursing and pre-professional health
contexts. Makerspaces were also used in professional development activities in STEM for
teachers and campus administrators in the United States.
4.6 Studies related to library makerspaces included experiences in the United States, New
Zealand and Estonia, as well as studies of makerspaces in museums, and mobile
makerspaces in the United States and rural Australia.
4.7 Studies have also been conducted surveying the makerspace landscape (specifically in the
United States), as well as assessments of makerspaces and FabLabs across multiple
countries, and a wide and current literature on a variety of facets of makerspaces.
4.8 The majority of studies reported positive impacts of makerspaces on awareness, knowledge
or understanding, which included knowledge on niche topics (for engineering students),
increased creativity and skills acquisition, competence in the use and application of new
technologies, and an overall gain in mathematical test scores.
4.9 Increased interest and engagement was reported in studies using makerspaces among
participants in a professional development program (in the United States), among
Indonesian primary students in STEM related activities, and for participants with mobile
makerspaces.
4.10 Positive attitude changes were reported by makerspace participants, ranging from medical
and biomedical engineering undergraduate students, to minority elementary and middle
school students attitudes towards STEM. Studies reported positive attitudes among STEM
teachers in professional development programs, as well as multiple reports of increased
confidence, self-confidence and motivation in makerspace related tasks, technology use,
proficiency and application.
4.11 Studies further reported positive behavioural changes, with increased willingness to
participate in more makerspace activities, increased motivation and reduced anxiety in
engineering and design related tasks.
4. iv
4.12 Skill development in makerspace related activities (particularly related to design,
manufacturing and teamwork) were reported, as well as improved problem solving and
group communication skills.
4.13 A range of other multifaceted impacts were reported that included enjoyment, personal and
professional development, and socialisation.
4.14 Makerspaces have a range of reported strengths, particularly related to increased
engagement with STEM knowledge, and the development and demonstration of 21st-
century skills such as problem-solving, critical and creative thinking, collaboration and
communication.
4.15 Studies further highlighted the potential for makerspaces in advancing interest in STEM
careers, in particular for underrepresented populations (namely people of colour and
women) in STEM.
4.16 Makerspaces also have the reported potential to cultivate creativity and innovation in
universities, as well as recasting the role of libraries and the impact they can have on local
communities.
4.17 Makerspaces provide an opportunity for meaningful community engagement: acting as
social spaces; supporting wellbeing; serving the needs of the communities in which they are
located; and providing outreach centers for excluded groups.
4.18 The reported weaknesses of makerspaces primarily relate to the lack of teacher preparation,
skill sets, expertise regarding how to use technology, pedagogical knowledge and limited
access to technology and resources, that can limit students’ potential to be positively
impacted by the experience.
4.19 Student anxiety in participating in makerspaces was further highlighted as a significant
barrier for students.
4.20 Despite the open nature of makerspaces, the fact that most early adopters of makerspaces
were affluent males, the benefits available through these facilities might not be evenly
available.
4.21 Improving STEM education through makerspaces in developed and developing countries
remains a challenge due to resource constraints.
4.22 The process of learning through makerspace require the development of appropriate tools
of assessment and analysis, in line with the challenges that still exist in measuring the impact
of informal learning environments. Mixed method approaches may help in this regard.
5. Conclusions
5.1 Makerspaces are having a transformative impact on STEM education, and have grown
rapidly in universities, schools, libraries and museums with the aim to encourage deep
engagement with STEM-integrated content, critical thinking, problem-solving and
collaboration.
5.2 Makerspaces reveal a huge potential to benefit individuals but also entire communities,
acting as a community hub, where people come to work together, learn from each other or
simply socialise, imparting value in a range of ways.
5.3 There remains a paucity of empirical research evaluating makerspaces and making, and
makerspaces and learning. The form of complex interdisciplinary learning taking place in
makerspaces demands new forms of assessments.
6. vi
CONTENTS
Executive Summary ......................................................................................................................... ii
Acronyms ...................................................................................................................................... vii
Mechanisms, groups and approaches ........................................................................................ 4 1.
Methodology for resource discovery and analysis ..................................................................... 5 2.
Search method ....................................................................................................................... 5 2.1.
Data extraction for the analysis ............................................................................................. 6 2.2.
Limitations of the resource discovery .................................................................................... 6 2.3.
Overview of results ................................................................................................................... 6 3.
Total number of resources discovered .................................................................................. 6 3.1.
Scientific subjects .................................................................................................................. 6 3.2.
Countries involved in the studies ........................................................................................... 7 3.3.
Educational delivery models .................................................................................................. 8 3.4.
Target sectors ........................................................................................................................ 8 3.5.
Delivery institutions ............................................................................................................... 9 3.6.
Approach to data collection ................................................................................................... 9 3.7.
Sampling technique and sample size ................................................................................... 10 3.8.
Discussion ................................................................................................................................ 10 4.
Contexts of use .................................................................................................................... 10 4.1.
Impacts ................................................................................................................................ 17 4.2.
4.2.1. Awareness, knowledge or understanding ............................................................................ 17
4.2.2. Engagement or interest ........................................................................................................ 17
4.2.3. Attitude ................................................................................................................................ 18
4.2.4. Behaviour ............................................................................................................................. 18
4.2.5. Skills ...................................................................................................................................... 19
4.2.6. Others ................................................................................................................................... 19
Strengths .............................................................................................................................. 20 4.3.
Weaknesses ......................................................................................................................... 21 4.4.
Costs and feasibility ............................................................................................................. 22 4.5.
Suggestions for improved methodologies and for future studies ....................................... 22 4.6.
Conclusions and overview ........................................................................................................ 25 5.
APPENDIX A: Example of data input mask ..................................................................................... 26
APPENDIX B: Selected bibliography ............................................................................................... 28
7. vii
Acronyms
3D three-dimensional
AM additive manufacturing
DIY do it yourself
EFT exploration and fabrication technologies
GPA grade point average
ME multidisciplinary education
STEAM science, technology, engineering, art and mathematics
STEM science, technology, engineering and mathematics
11. 7
Main subject area Detailed subject References
Applied / STEM
4
General
Forest et al. 2014; Foth,
Lankester, and Hughes
2016; Morocz et al. 2016;
Lagoudas et al. 2016;
Blackley et al. 2017;
Blikstein et al. 2017;
Galaleldin et al. 2017;
Ludwig, Nagel, and Lewis
2017; Sheffield et al. 2017;
Sinha et al. 2017; Tomko et
al. 2017; Blackley et al. 2018
Formal Mathematics Tillman et al. 2014
Social Science education
5
Sheridan et al. 2014; Litts 2015;
Gahagan 2016; Lille 2016;
Marshall 2016; McCubbins
2016; Miller 2016; Miller,
Christensen, and Knezek 2017
Table 4. Main scientific subjects of the resources analysed.
Countries involved in the studies 3.3.
The countries where the studies have taken place are listed in Table 5 and can be visualized on the
world map in Figure 1.
Countries No. of studies for each country
United States of America (US) 16
Australia 4
Canada, Estonia, Indonesia, Mexico, New Zealand 1
Table 5. Number of impact assessment studies for each country.
4
Science, technology, engineering mathematics (STEM).
5
Including assessment methodologies in makerspaces.
13. 9
Main target sector Sub-divided target sector References
Education & Training
(18) 66.7 %
Primary education
Tillman et al. 2014; Blackley et al. 2017;
Blikstein et al. 2017; Sheffield et al. 2017;
Blackley et al. 2018
Secondary education Marshall 2016, Blikstein et al. 2017
Bachelor’s or equivalent level
Forest et al. 2014; Lagoudas et al. 2016;
Morocz et al. 2016; Blackley et al. 2017;
Galaleldin et al. 2017; Ludwig, Nagel, and
Lewis 2017; Sinha et al. 2017; Tomko et al.
2017
Not elsewhere classified Miller 2016
(Not specified) Gahagan 2016
Population groups
(8) 29.6 %
Adults Sheridan et al. 2014
Children Sheridan et al. 2014; McCubbins 2016
Youth
Sheridan et al. 2014; Litts 2015; Sheffield
et al. 2017
(Not specified)
Foth, Lankester, and Hughes 2016;
Galaleldin et al. 2017
Work
(1) 3.7 %
Lille 2016
Table 7. Target sectors and relative percentage over the total number of instances.
Delivery institutions 3.6.
The delivery institutions promoting ‘Makerspaces’ are presented in Table 8 as identified within a
wide categorisation identified by the research team.
Approach to data collection 3.7.
Of the 21 impact assessment studies, 8 used a mixed-method approach, 8 were primarily qualitative
and 5 quantitative.
For these studies, data collection approaches involved written or online surveys (17 studies),
interviews (9), observations (8), experiment (4), case studies (3), focus groups/discussions groups
(2),
documentation review/archival (2) and email (1).
Among the data collection tools or scales employed there were Likert scales (9 studies),
questionnaires (7) and tests (1).
The statistical approaches used involved hypothesis testing such as t-tests (4 studies), ANOVA (3)
and chi-squared (1). Some studies specified the software tool or app used for analysis: Microsoft
Excel (1), SPSS (1), SPSS 16 (1), SPSS 24 (1) or others (3).
14. 10
Delivery institution References
Educational institution
(18) 85,7 %
School Blackley et al. 2018
University
Forest et al. 2014; Tillman et al. 2014;
Litts 2015; Lagoudas et al. 2016; Gahagan
2016; Marshall 2016; McCubbins 2016;
Miller 2016; Morocz et al. 2016; Blackley
et al. 2017; Blikstein et al. 2017;
Galaleldin et al. 2017; Miller, Christensen,
and Knezek 2017; Ludwig, Nagel, and
Lewis 2017; Sheffield et al. 2017; Sinha et
al. 2017; Tomko et al. 2017
Various
(1) 4,8 %
Foth, Lankester, and Hughes 2016
Others
(2) 9,5 %
Sheridan et al. 2014; Lille 2016
Table 8. Delivery institutions and relative percentage over the total number of instances.
Sampling technique and sample size 3.8.
Amongst the sampling techniques employed in the studies under consideration, there were
convenience sampling (17 studies), random (1) and systematic sampling (3).
Sample sizes ranged from 5 to 105 (147 mean; 52 median), 15 studies have a sample size under
100 participants and 2 do not specify the sample size.
Discussion 4.
Contexts of use 4.1.
Makerspaces, by any name, are fundamentally places to design, explore and create. Despite its
continued growth as a search term, ‘makerspace’ has only recently officially entered the dictionary
lexicon7
, in comparison to the related term ‘hackerspace’ has (Davee, Regalla, and Chang 2015).
The first use of the term ‘makerspace’ dates back to the publication of ‘Make magazine’ in 2005
and by the subsequent launch of Maker Faire, an event that demonstrated the popularity of making
and showcasing of new technologies (Wong and Partridge 2016).
Started as a grassroots movement, makerspace is particularly challenging to cohesively define
(Litts 2015) and, as a more generic and inclusive term, it has increasingly represented an extremely
wide variety of creative endeavours, tools, demographics, specializations and types of places where
making occurs (Davee, Regalla, and Chang 2015).
7
“Makerspace”, Oxford Dictionary, Accessed 28 December 2018,
https://en.oxforddictionaries.com/definition/makerspace; “Makerspace”, Cambridge Dictionary, Accessed 28 December
2018, https://dictionary.cambridge.org/dictionary/english/makerspace