Essay Writing Service http://HelpWriting.net/A-Quasi-Experimental-Analysis-Of-The-Ad 👈

1. https://doi.org/10.1177/0741713619861073
Adult Education Quarterly
1–20
© The Author(s) 2019
Article reuse guidelines:
sagepub.com/journals-permissions
DOI: 10.1177/0741713619861073
journals.sagepub.com/home/aeq
Article
A Quasi-Experimental
Analysis of the Adult
Learning Effect on
Problem-Solving Skills
Suehye Kim1
Abstract
This study builds an empirical research model that explores societal and individual
antecedents of adult learning and examines the adult learning effect on problem-solving
skills in the Organization for Economic Cooperation and Development (OECD)
countries. Considering national differences, it uses multilevel data sources available
from the Programme for the International Assessment of Adult Competencies
(PIAAC) 2008-2013 and the OECD data lab. The results from this study suggest
that young adults (25- to 34-year-olds) with more favorable backgrounds benefit
from their social origin and occupational environment in terms of nonformal learning
participation. It was also found that the participants outperform nonparticipants in the
PIAAC problem-solving skill assessment. The key findings of this study advocate for
the importance of policy interventions to combat the cumulative effects of multiple
disadvantages in the educational trajectory from initial education to adult learning as
well as to reduce the problem-solving skill gaps of young disadvantaged adults.
Keywords
adult learning and education, problem-solving skills, quasi-experimental analysis,
cross-national assessment data
Introduction
Continued economic, social, and technological developments require acquiring new
skills through education and training. As a new skill necessary for the 21st century,
well-documented research has noted the significance of problem-solving skills (Jacobs
1Korea University, Seongbuk-gu, Seoul, Korea
Corresponding Author:
Suehye Kim, Center for Social Cohesion Education, Korea University, 145, Anam-ro, Seongbuk-gu, Seoul
02841, Korea.
Email: lemontree845@gmail.com
861073AEQXXX10.1177/0741713619861073Adult Education QuarterlyKim
research-article2019

2. 2 Adult Education Quarterly 00(0)
& Castek, 2018; Rosen & Vanek, 2017; van Laar, van Deursen, van Dijk, & de Haan,
2017). Problem-solving skills in technology-rich environments refer to how well peo-
ple deal with specific types of problems using information and communication tech-
nology (ICT) tools. It is a competency domain at the intersection of computer literacy
skills (i.e., the capacity to use ICT tools and applications) and the cognitive skills
required to solve problems (Organization for Economic Cooperation and Development
[OECD], 2013a). The ability to use these tools to access, process, and evaluate infor-
mation intelligently is essential for the maintenance and upgrading of workforce skills.
Moreover, the production, distribution, and use of new knowledge and information
have created high-paying jobs.
As a result, high proficiency in these information-managing skills provides adults
with more access to rewarding occupations and to participation in further learning
and training over their career span. In contrast, low-skilled adults are likely to be left
behind, which may increase inequality in the various socioeconomic outcome dimen-
sions such as the distribution of income, nutrition, civic engagements, citizenship,
and social trust. Consequently, academic and policy discussions stress the importance
of digital readiness and the need for adults to continue learning throughout their
entire lives (Horrigan, 2016; OECD, 2015; UNESCO, 2016). Learning occurs during
every life stage in different contexts such as in the home, school, work, and commu-
nity. From a lifelong-learning perspective, all citizens as self-directed individuals
must have open, flexible, and personally relevant opportunities to develop the knowl-
edge and competences necessary at all stages of their lives. Through a broader lens, a
key skill is seen not just as a basic skill such as reading and writing but as a set of
competencies applied to tasks in technology-rich environments (Hickling-Hudson,
2007; Wilson, Scalise, & Gochyyev, 2015).
This study aims to explore the nonformal learning effect on problem-solving
skill acquisition in technology-rich environments focusing on the young adult pop-
ulation (25- to 34-year-old adults) in OECD countries. As young adults do develop
their skills through ongoing learning activities, this study adopts the conceptual
model that explores societal and individual antecedents of adult learning and exam-
ines the consequences of adult learning on problem-solving skill acquisition in
technology-rich environments (see Figure 1). Among the countries at comparable
levels of economic development, variation is noticeable in terms of national institu-
tions. As Blossfeld, Kilpi-Jakonen, de Vilhena, and Buchholz (2014) have acknowl-
edged the importance of national institutions, adult learning and skills at the
individual level might vary across countries. However, relatively few studies have
explored such a cross-national difference in terms of structural dimensions and
education systems. Considering the lack of international evidence on the impact of
the national antecedent, this study attempts to identify the specific features of
national institutions affecting the relationship between adult learning and skills.
One particular focus of this study is in examining the influence of national institu-
tions on the relationship between nonformal learning and adult problem-solving
skill acquisition.

3. Kim 3
Participation in Adult Learning at the Individual and
Societal Levels
As a way of promoting the transition from school to work, adult learning in the work-
place may improve the adaptability of workers to technological and structural changes
in the economy. Indeed, the benefits of adult learning are expected to offset disadvan-
tages from the initial schooling stage (Rubenson, 2006a, 2006b; Tsatsaroni & Evans,
2014; UNESCO, 2013). From a lifelong-learning perspective, adults can enhance key
skills via informal means in addition to formal learning in school or nonformal learn-
ing outside the formal education system (Tsatsaroni & Evans, 2014; UNESCO, 2013).
As an alternative to traditional modes of learning, ICT plays an important role. The
rapid growth in access to mobile phones and computers has enabled illiterate adults to
use text messages that can enhance their literacy (Molnár & Benedek, 2014; UNESCO,
2016). Specifically, the technology helps adults acquire literacy skills in two primary
ways (UNESCO, 2016; Wagner & Kozma, 2005). First, ICT-based instruction can
develop the cognitive processes and basic skills related to literacy (Greene, Seung, &
Copeland, 2014; Leu, Kinzer, Coiro, Castek, & Henry, 2017). Second, the develop-
ment of literacy skills can be facilitated by technology that has created new opportuni-
ties for learning at a distance (Mohammadyari & Singh, 2015). Acquiring the ability
to learn increases the interaction between education, jobs, and other contexts in adult
life. The life-wide and lifelong perspective defines learning as a continuous activity
independent of age. Adults continue to acquire and develop skills before, during, and
after compulsory schooling, in and out of school, and through formal, nonformal, and
informal learning (UNESCO, 2006, 2013).
However, participation in adult education and training is dominated by more edu-
cated populations (Desjardins & Rubenson, 2013; Fouarge, Schils, & De Grip, 2013;
UNESCO, 2013). Marginalized groups risk exclusion from the further learning
opportunities offered by social media and ICTs. In terms of inequality, Tuijnman and
Boudard (2001) pointed out that adults with little initial education might have a
lower probability of continuing to learn than their counterparts with high skills and
Figure 1. Adult learning and the role of various national institutions.
Source. Blossfeld et al. (2014.)

4. 4 Adult Education Quarterly 00(0)
a long period of initial education do. Indeed, the social theory of lifelong learning
suggests that an individual’s capacity to obtain learning opportunities depends on his
or her social environment. A theoretical lens of social interaction (Strauss, 1962)
explains a working-class male’s experience of initial education and his subsequent
learning and training differs systematically from that of middle-class females. Their
access to learning opportunities is affected by their social status. Indeed, emergence
of life course paradigm has illuminated cumulative process of socioeconomic status
into the later years of adult (Elder & Giele, 2009). Regarding this issue, Colley,
James, Diment, and Tedder (2003) developed a concept of “vocational habitus” to
explain how learners are oriented to a particular set of dispositions and are prepared
to enter occupations. They suggest that as a process of becoming, learning is socially
constructed in terms of classed relations of power within the educational field and
society at large.
The relevant literature notes that adult learning participation has been conditioned
by national institutions (Blossfeld et al., 2014; Rubenson & Desjardins, 2009;
UNESCO, 2016). At the societal level, educational systems and social policies have
been defined as influencing factors for adult learning. Within the rigid system, track-
ing in upper secondary schooling might lead cumulative advantage or disadvantage of
education over the life course (Blossfeld, Buchholz, Skopek, & Triventi, 2016;
Blossfeld et al., 2014;). That is because such an inflexible system places people on a
specific track within the educational hierarchy. For example, individual adults with a
general education are somewhat more likely to have advanced education compared
with those with a vocational education (Hanushek, Schwerdt, Woessmann, & Zhang,
2017). Those who are left behind during formal initial schooling may overcome their
disadvantage through nonformal learning (Roosmaa & Saar, 2012; UNESCO, 2013;
Werquin, 2010). As a result, the acknowledgement of a learning continuum between
formal and nonformal education is an important issue at the system level (UNESCO,
2013). In terms of a linkage between formal and nonformal education, the relative
degree of openness of formal education institutions to nontraditional students indi-
cates a development of the adult education system in different countries (Desjardins &
Rubenson, 2013). This suggests that countries that establish a system for recognizing
nonformal learning outcomes may have an increase in participation rates in adult
learning and skills.
Comparative evidence suggests that the social welfare policy is related to varia-
tions in adult education in different countries (Desjardins, 2013; Desjardins &
Rubenson, 2013; Rubenson & Desjardins, 2009). For example, Nordic countries
with a social democratic regime focus on dealing with structural barriers, whereas
Anglo-Saxon countries follow a liberal welfare-state regime underscoring means-
tested assistance, modest universal transfers, and social insurance plans. With the
recognition of the public good aspects of adult education, the Nordic countries spend
more money on adult education targeting disadvantaged groups compared with the
Anglo-Saxon countries.
Since the global financial crisis in 2008 to 2009, however, most OECD countries
have had increased demands on social protection systems providing institutional

5. Kim 5
support for adult learning at the societal level. Emphasizing the importance of an
effective labor supply, socioeconomically developed countries have activated their
welfare states to increase employability (OECD, 2013b). As a way to improve labor
market performance, OECD policy makers encourage member countries to implement
activation reforms such as Active Labor Market Programs that provide vocational
training and hiring subsides to support disadvantaged workers. These active labor
market policies have indicated that the level of social policy development is important
(Powell & Barrientos, 2004). As a type of active labor market program, public support
for adult learning and training is essential to boost adults’ motivation to participate in
further learning and to enhance their employability.
Method
Data Sources
In considering institutional differences, this study uses the data sources available from
the recent international assessment, the Programme for the International Assessment
of Adult Competencies (PIAAC) survey data from 2008 to 2013, and the OECD data
lab. The PIAAC assessment concentrates on cognitive and workplace skills necessary
for successful participation in 21st-century society and the global economy. To rule
out the possibility that aging is closely associated with the level of performance in
problem-solving skills and adult learning, this study targets young adults aged from 25
to 34 years for two reasons. First, a life span perspective defines the age of young
adults as those individuals up to their mid-30s who have started taking on key respon-
sibilities such as establishing a family or circle of friends, and getting a good job
(Armstrong, 2007). Second, in this research, adult education refers to learning activi-
ties after the initial schooling stage, thus excluding those still in their first formal cycle
of schooling. As a result, the final sample for this study consists of 23,615 adults from
19 countries including Austria, Belgium (Flanders), Canada, the Czech Republic,
Denmark, Estonia, Finland, Germany, Ireland, Japan, Korea, the Netherlands, Norway,
Poland, the Russian Federation, the Slovak Republic, Sweden, the United Kingdom
(England and Northern Ireland), and the United States.
Measurements
This study uses an outcome variable measured by the PIAAC assessment framework
that defines key adult skills in three domains (literacy, numeracy, and problem-solving
skills) in technology-rich environments. By reflecting the changing nature of informa-
tion, the assessment instruments used in PIAAC examine literacy in digital environ-
ments, the ability to access, use, and communicate mathematical information and
ideas, and the ability to analyze computer-based simulation tasks, to define goals, and
to monitor their progress (OECD, 2013a). Distinguished from previous adult assess-
ments, the PIAAC underscores these adult competencies to access, manage, integrate,
and construct information using the technologies. In what follows, operationalizing

6. 6 Adult Education Quarterly 00(0)
problem-solving skills in technology-rich environments is described as a dependent
variable.
The problem-solving assessment mainly examines the knowledge about how to
handle digital tools and to structure the problem, to set goals, to measure progress
toward those goals, and to practice metacognition (Levy, 2010). As a new skill, prob-
lem solving is distinguished from the other two cognitive domains—literacy and
numeracy—in several particular tasks, focusing on the processes of problem solving
in various environments, using pragmatic evaluation sources and the integration of
information across sources. The PIAAC assessment estimates plausible values (PVs)
from multiple imputations which combine item response theory scaling of the cogni-
tive assessments with an unobservable latent regression model using information
available in the background questionnaire in a population (OECD, 2013a). This study
uses all 10 PVs as a dependent variable granted for an equivalent estimate and ana-
lyzed using an R package for complex surveys including PVs (svyPVpack).
Key factors affecting problem-solving skills come from the individual and coun-
try levels. In line with the systematic overview suggested by Blossfeld et al. (2014),
this study takes account of characteristics of participation in adult learning at the
micro and macro levels (see Figure 2). The individual-level determinants are mea-
sured by background questionnaire items from the PIAAC survey. The PIAAC col-
lected data on the characteristics and backgrounds of respondents in five main areas:
demographics, educational attainment and participation, labor-force status and
employment, social outcomes, and literacy and numeracy practices and the use of
skills. Using the PIAAC background questionnaire, the independent variables were
operationalized as follows.
Thisstudyincludesnonformaladultlearningparticipationasasecondsetofdependent
variables as well as an independent variable. As discussed in the relevant literature, non-
formal education is categorized into two types: compensatory and complementary
(Desjardins & Rubenson, 2013). Compensatory types pertain to basic education, literacy
programs, and second chances related to formal qualifications, whereas complementary
types are on-the-job training, continuing vocational or professional training, and adult
Figure 2. Characteristics of participation in adult learning at the micro and macro levels.
Source. Blossfeld et al. (2014.)

7. Kim 7
higher education. Given that a key dependent variable—problem-solving skills in tech-
nology-rich environments—is closely associated with the complementary types, “par-
ticipation in nonformal education for job-related reasons in the 12 months preceding the
survey on nonformal learning” is defined as a measure of nonformal adult learning.
To examine the adult learning and training effect on the problem-solving skill, edu-
cational attainment initially needs to be considered as substantially discussed by the
relevant research (Hanushek, Schwerdt, Wiederhold, & Woessmann, 2013; Lam &
Warriner, 2012; UNESCO, 2013). As an indicator of initial educational attainment, the
highest level of education completed is used. At the individual level, the other key
variables are measured as follows. First, gender is measured as a dummy variable hav-
ing the value 1 for the male group. Bearing in mind that cognitive decline depends on
biological age, how old the respondent is was controlled for empirically as a continu-
ous variable. In terms of social class, two indicators are measured: the highest level of
education that either the mother or father guardian achieved and a derived variable of
the yearly income percentile rank from the lowest group (less than 10%) to the highest
group (90% and above).
The important covariates are variables that are also related to workplace environ-
ments (OECD, 2013a; Tuijnman & Boudard, 2001). To clarify the nature of the job’s
and organization’s characteristics, three measurements are used to consider the rela-
tionship between the adult’s skills profiles in terms of occupational status, learning
at work, and the use of ICT skills. First, occupational status is considered via occu-
pation and status in employment coded by the International Standard Classification
of Occupations. Also, a derived variable is employed from three items measuring the
frequency of learning opportunities from supervisors or coworkers, learning by
doing, and keeping up to date with new products or services. To examine the response
to new technology in the workplace, an index is employed indicating how ICT skills
at work are used, which is available from the PIAAC survey data.
Focusing on the importance of societal antecedents, the education system and
labor market policies are considered in the research model. First, this study defines
participation rates in adult formal education as a key country-level variable related
to the education system since countries with well-developed adult education systems
show high-participation rates for nonformal learning. Thus, the relative degree of
openness of formal educational institutions to nontraditional students is measured as
an indicator of highly advanced adult learning systems (Desjardins, 2013; Desjardins
& Rubenson, 2013).
As a measure of the “supply-side” policies of national governments, an indicator of
the labor market policy is included. To help unemployed adults back to work, OECD
countries have implemented policies including job-placement services, benefit admin-
istration, and labor market programs such as training and job creation (OECD, 2013b).
To cover this aspect, a scaled variable is used that measures the protection of perma-
nent workers against individual dismissal. To avoid a multicollinearity issue among
the independent variables, a correlation matrix was screened prior to research model-
ing (see the appendix).

8. 8 Adult Education Quarterly 00(0)
Analytical Strategies
In this study, four specific analytical techniques are used. First, a preliminary analy-
sis builds a specific picture of national variations in problem-solving skill gaps
through the lens of nonformal adult learning across 19 countries. For this, the R
package (instvy) is used to estimate an average difference in the problem-solving
skill assessment in the PIAAC between participants and nonparticipants of nonfor-
mal adult learning. It calculates a correct estimate of the means and associated stan-
dard errors of problem-solving skill-achievement variables measured by 10 PVs.
Second, a multilevel analysis considers the institutional difference in the partici-
pation in nonformal learning, assuming individual adults are nested within coun-
tries. As a common way to analyze data from nested samples, the multilevel
approach takes account of variations at two levels and decreases aggregation bias
(Raudenbush & Bryk, 2002). The multilevel model analysis predicts an adult’s pro-
pensity for post-initial learning activities. Through a propensity score analysis, this
study aims to examine unbiased estimates of the nonformal adult learning effects.
By replacing the confounding covariates with one scalar function of these covari-
ates, it reduces unobserved heterogeneity. As a primary data source, the PIAAC
provides a variety of variables used as covariates to characterize adults and their
ecological backgrounds such as households and workplaces. To adjust for such dif-
ferences in the background variables, the propensity score is estimated as a balanc-
ing score.1
This propensity score needs to include unobserved covariates in multilevel
structured populations (Arpino & Mealli, 2011; Hong & Raudenbush, 2005; Kim
& Seltzer, 2007). Since the average probability of participating in adult learning
programs may vary across countries, a multilevel technique specifies the propen-
sity score for two different levels, assuming that the variation may depend on
measured or unmeasured country-level characteristics. Without considering the
national contexts, it might be problematic to compare the educational trajectory of
adults among the different countries (Arpino & Mealli, 2011). In considering such
institutional variations, this study uses a multilevel logistic regression model to
estimate the propensity score that predicts adult learning and training. With the
covariates used in initial generalized linear models, this study examines the treat-
ment effects after controlling not only for individual-level characteristics but also
for country-level variables.
Propensities derived from the multilevel model are used to match participants and
nonparticipants in nonformal adult learning. To find the best match for the participa-
tion in nonformal learning, the covariate balancing propensity score (CBPS) method-
ology is implemented, resulting in an optimized balance (Imai & Ratkovic, 2014)
using the CBPS package in R. A propensity score weight analysis is also carried out
for the complex sample design of PIAAC via the toolkit for weighting and analysis of
nonequivalent groups (twang) package and a package for complex surveys including
PVs (svyPV) in R. This tailored approach is essential when using complex population-
based sample survey data such as that found in PIAAC.

9. Kim 9
Last, this study employs multiple imputation methods to handle missing values.
When observations are missing, the available options are to delete the data or to
replace the missing value with an imputed value. Given that listwise deletion reduces
the analytic sample size, which can be problematic if missing observations occur for
many subjects (Kline, 1998), this study adopts multiple imputations using the Markov
chain Monte Carlo algorithm from the mice package in R. Using multiple imputation
analysis, the final sample consists of 23,615 young adults from 19 countries for which
individual backgrounds and national characteristics were collected.
Results
Countries vary in terms of adult education participation and have different patterns for
the relationship between nonformal learning participation and the problem-solving
skills of young adults. The purpose of this study is to examine variations across coun-
tries and to determine whether individual and societal antecedents are related to par-
ticipation in nonformal adult education. The ultimate goal is to estimate the effect of
nonformal education on problem-solving skills in technology-rich environments
across and within countries.
National Variations in the Nonformal Learning Effect on Problem-
Solving Skills
A preliminary analysis of international assessment data finds an explicit tendency
for problem-solving skill gaps linked to participation in nonformal adult learning
among different OECD countries. It shows cross-country variations in the nonfor-
mal learning effect (see Table 1 and Figure 3). When controlling the covariate vari-
ables, three countries demonstrate a significant association between nonformal
learning and problem-solving skills (Canada, the Russian Federation, the Slovak
Republic). This covariate analysis reveals a prominent pattern of educational attain-
ment that is strongly associated with problem-solving skills in all OECD countries.
Considering the close linkage between two factors—educational attainment and
adult learning participation (OECD, 2015; UNESCO, 2016)—, a further examina-
tion is required to determine which factors predict nonformal education participa-
tion in addition to educational attainment. Consequently, the association between
individual and societal antecedents and participation in nonformal adult education
is estimated with a series of fixed effects multilevel models. The results are sum-
marized in the following section.
Individual and Societal Antecedents of Nonformal Learning Participation
Using multilevel logistic regression sets, a young adult’s propensity score to receive
nonformal education is predicted with important covariates including both their indi-
vidual-level background and national context. The first model includes individual-
level variables that are fixed so that they have the same effect for countries. Table 2

10. 10 Adult Education Quarterly 00(0)
shows that participation in nonformal education is significantly associated with a vari-
ety of individual characteristics. First, the gender effect shows that males are 31%
more likely to participate in nonformal education relative to females. Among 25- to
34-year-olds, age has statistical significance, but this is not practically meaningful due
to the limit of sample size. The statistically significant positive estimated effect of
social class holds true. The odds of taking nonformal education among adults with
highly educated parents are 1.13 times higher than those with low-educated parents.
Education has a noticeable effect. Adults with a higher level of educational attainment
are more likely to take nonformal education, with an increase in odds of 14% per stan-
dard deviation increase in the level of educational attainment. The odds of participat-
ing in nonformal education among workers with a higher yearly salary are 1.21 times
higher than for low-paid workers.
After controlling for individual and family background characteristics, work-
place characteristics are consistently found to predict the likelihood of being a par-
ticipant in nonformal education. The coefficients indicate that the more highly
skilled occupational groups are 43% more likely to undertake nonformal education
than the less-skilled occupational groups are. Learning opportunities at work have
Table 1. National Variations in Problem-Solving Skill Gaps.
OLS regression
slopes
NFE EDU
B SE B SE
Austria 3.53 2.55 4.61*** 1.07
Belgium 4.98 3.31 4.74*** 1.01
Canada 9.86*** 2.56 5.6*** 0.8
Czech Republic 0.88 3.84 5.66*** 1.28
Denmark 1.02 3.11 4.68*** 0.95
Estonia −0.59 2.44 4.49*** 0.86
Finland −0.42 2.91 4.59*** 1.24
Germany 0.8 3.19 7.9*** 1.02
Ireland 1.8 2.54 5.48*** 0.99
Japan 0.15 0.03 4*** 1.29
Netherlands 2.67 3.67 6.99*** 1.25
Norway 1.43 2.74 4.08*** 0.94
Poland 1.93 3.83 2.78*** 0.96
Russian Federation 10.06** 4.36 4.55*** 1.43
Slovak Republic 13.14*** 3.82 3.2** 1.08
South Korea 1.44 2.03 5.4*** 1.02
Sweden 1.96 2.86 4.08*** 1.14
The United Kingdom 5.77 3.68 1.66** 0.65
The United States 4.46 3.13 6.75*** 1.31
Note. OLS = ordinary least squares; NFE = nonformal education; EDU = educational attainment;
B = coefficient; SE = standard error.
***p < .001. **p < .01.

11. Kim 11
a positive effect as well. The coefficients indicate that adults who more proactively
learn at work are 27% more likely to participate in nonformal education. A positive
effect of ICT skills is prominent in the workplace. Adults with proficient ICT skills
at work have 1.16 times the odds of those with limited ICT skills of participating in
nonformal education.
By adding a variety of country-level predictors in Model 2, the probability of par-
ticipating in nonformal education is predicted by national characteristics. After con-
trolling for individual background variables, the adult education system at the country
level is significantly related to nonformal learning participation. Specifically, the odds
of taking nonformal education among adults in a country with a more flexible educa-
tion system are 1.05 times higher than for those in a country providing limited educa-
tional access for nontraditional learners. In addition, adults in a country with a labor
market policy that protects permanent workers against dismissal are more likely to
participate in nonformal learning, with an increase in odds of 9%, but this is not statis-
tically significant.
In Model 3, which adds interaction terms between country-level variables and
individual variables, a net effect of the national context on nonformal learning par-
ticipation is found. The multilevel logistic regression detects a significant cross-level
interaction effect between institutional and individual characteristics. First, nonfor-
mal learning participation increases with relatively low educated adults in countries
with well-developed adult learning systems. The odds of young adults undertaking
Figure 3. National variations in problem-solving skill gaps.
Note. OLS = ordinary least squares; EDCAT7 = OLS regression slopes of educational attainment;
NFE12JR = OLS regression slopes of nonformal education.

12. 12 Adult Education Quarterly 00(0)
nonformal education with a lower level of educational attainment in countries with
rigid educational systems are 0.99 lower than for those in countries with more flexi-
ble and open educational systems. Although this odds difference implies a well-
developed education system might mitigate against the impact of initial education
inequality on adult learning participation, it cannot be practically meaningful due to
the sample size.
To test differences in the overall fit of the sets of nested models, the model fit sta-
tistics were examined. The overall goodness of fit indicates that the association
between predictors and the outcome varies across countries, and individual and soci-
etal antecedents condition participation in nonformal learning. From between the two
types of multilevel models, the Model 3 multilevel model was selected as it takes the
country-level variation as arising from the interaction between institutional and indi-
vidual characteristics based on the overall deviance test. Using Model 3, which pre-
dicts the probability of nonformal learning participation, a propensity score was
obtained for each individual adult. The sections below specify how the multilevel
propensity score model was run and analyze the causal effect of nonformal education
on problem-solving skills in technology-rich environments.
Table 2. Multilevel Propensity Models for Nonformal Educational Participation: Fixed
Effects.
Fixed effects
Model 1 Model 2 Model 3
Exp. (%) Exp. (%) Exp. (%)
National characteristics
Labor market policy (γ01) 1.09 1.08
Adult education system (γ02) 1.05* 1.09***
Individual characteristics
Gender (γ10) 1.31*** 1.31*** 1.31***
Age (γ20) 1.01** 1.01** 1.01***
Parental education level (γ30) 1.13*** 1.13*** 1.13***
Educational attainment (γ40) 1.14*** 1.14*** 1.22***
Yearly income (γ50) 1.21*** 1.21*** 1.21***
Workplace characteristics
Occupational status (γ60) 1.43*** 1.43*** 1.43***
Learning at work (γ70) 1.27*** 1.27*** 1.27***
ICT at work (γ80) 1.16*** 1.16*** 1.16***
Interaction effects
Educational attainment ×
adult education system (γ11)
0.99**
Model fit statistics
Deviance (chi-square test) 26425 26420† 26412**
AIC 26445 26444 26438
Note. ICT = information and communication technology; AIC = Akaike information criterion.
***p < .001. **p < .01. *p < .05. †p <.10.

13. Kim 13
Multilevel Propensity Score for Nonformal Learning Participation
Multilevel propensity score modeling was used to estimate the effect of nonformal
learning participation on problem-solving skills in technology-rich environments.
Three analytical packages (CBPS, twang, and svyPV) were used for this purpose. This
allows for the maximization of the resulting covariate balance and the prediction of
treatment assignments using the PIAAC data with a complex survey design and PVs.
The propensity score predicting adult education participation includes a number of
significant covariates at both the individual and country levels. Propensity score
weighting was applied with multilevel data to estimate the average treatment effect of
nonformal learning participation on problem-solving skills. Table 3 below summarizes
the covariate balance evaluation to ensure an equivalence in the distribution of each
covariate for the treated (participation) and control (nonparticipation) adults. The
results of the statistical test in the table (p values) demonstrate that two groups are bal-
anced in terms of propensity scores and that selection bias has been removed.
Estimated Effects of Adult Learning on Problem-Solving Skills
Table 4 displays the estimated effects of nonformal learning on problem-solving
skills in technology-rich environments. The participants generally outscore nonpar-
ticipants for OECD countries. The average achievement of the participant group
(296.66) is significantly higher than that of the nonparticipant group (292.02). This
result suggests that nonformal learning might make a difference in problem-solving
skill acquisition after accounting for individual and country characteristics on aver-
age across OECD countries.
Table 3. Covariate Balance.
Variables tx.mn tx.sd ct.mn ct.sd stat p
Gender 0.48 0.50 0.47 0.50 1.12 0.26
Age 29.50 2.93 29.56 2.93 −1.28 0.20
Parental education level 2.24 0.68 2.23 0.69 1.01 0.31
Educational attainment 4.56 1.76 4.57 1.80 −0.33 0.74
Occupational status 3.13 0.98 3.16 0.94 −1.33 0.18
Yearly income 3.48 1.38 3.49 1.39 −0.43 0.67
Learning at work 2.12 0.95 2.10 1.01 1.60 0.11
ICT at work 1.97 1.00 1.95 1.05 1.23 0.22
Labor market policy 1.84 0.69 1.82 0.70 1.27 0.20
Adult education system 10.23 3.81 10.17 3.81 0.95 0.34
Note. ICT = information and communication technology; tx.mn/ct.mn = treatment (participation)
means and the control (nonparticipation) means; tx.sd/ct.sd = the propensity score weighted treatment
(participation) and the control (nonparticipation) group’s standard deviations; stat, p = depending on the
types of variables (i.e., continuous or categorical), stat is a t statistic or chi-square statistic and p is the
associated p value. Insignificance results in covariate balance, statistically rejecting a significant difference
in backgrounds between treated and controlled groups.

14. 14 Adult Education Quarterly 00(0)
Discussions
The key findings of this study have called attention to three important issues. First, the
influence of adult learning on skill acquisition provokes a concern due to the close
relationship between education and skills. A number of intergovernmental reports
have emphasized the importance of learning and training as an investment in human
resources (Guile, 2006). To enhance adult skills, developed countries have a wide
variety of approaches and provisions for vocational training in different types of insti-
tutions (Dolton, 2004). In terms of the provision of further education, for example, the
United Kingdom shows a strict dichotomy between work-based training and academic
training, while many other countries have a much more united integration of the work-
based training system. Despite marked variations in adult learning systems among
different countries, it is commonly known, however, that access to learning opportuni-
ties substantially depends on one’s family background, especially for young adults.
From a sociological perspective, adult skills are learned through interaction with
a sociocultural environment and skill acquisition should be understood as a learn-
ing metric (Steinberg, 1990; UNESCO, 2015b). In this context, skill gaps relate to
an unequal chance of learning, and it is crucial to promote lifelong learning for all
to increase sustainable development. Regarding social reproduction, this study
addresses the question of unequal access to educational experiences from a life-
course perspective. Adults from low SES families are less likely to have good lev-
els of initial education or to obtain more access to further learning opportunities.
Those from educationally disadvantaged backgrounds might find it hard to get
decent jobs that could provide more opportunities for them to improve their profes-
sional development via workplace learning. This might trigger a skill gap where the
low-skilled jobs are overrepresented by a low-educated workforce. A vulnerable
adult with a low education level might encounter the double jeopardy scenario of a
lack of work-based learning and skill development. In a lifelong-learning society,
skill can beget skill when mediated by workforce learning, which might repeat a
vicious cycle. Herein, the hypothesis that high-skilled adults are more likely to
participate in further learning is not examined, although further examination is
needed of the other causes linked to skills and adult learning in the near future.
In general, more systematic and structured approaches are necessary for the disad-
vantaged groups. To meet their different learning needs and demands, there is strong
Table 4. Nonformal Learning Effect on Problem-Solving Skills Within the Strata.
Participant Nonparticipant Mean difference
Mean score for problem-
solving skill assessment
296.66 292.02 4.64***
Number of cases 9733 11899
Sum of weights 21006.19 21537.5
Note. Average treatment effect: 4.64*** (t = 7.19, p = .000).

15. Kim 15
support for developing a system for recognizing nonformal learning outcomes
(UNESCO, 2013; Werquin, 2010). Consequently, some countries now employ two
routes toward achieving this goal: law and negotiation for social consensus (Werquin,
2010). As for the state-led model, the law and policies are globally observed in the
legislative framework of recognition, validation, and accreditation of nonformal and
informal learning (UNESCO, 2015a, 2015b). In Finland, for example, the national
policy has legally validated learning outside the formal system in the various sectors
of education from comprehensive schooling to adult vocational education through the
Competence-Based Qualification system (Damesin, Fayolle, Fleury, Malaquin, &
Rode, 2014). The recognition of nonformal learning has proceeded with agreement
among the social partners as well. In particular, the Nordic countries and Germany
have demonstrated a well-developed institutional system driven by shared responsibil-
ity and a social partnership among the stakeholders such as the government, employ-
ees, employer organizations, and trade unions based on the coordinated market system
(Rubenson, 2006a, 2006b; UNESCO, 2015a).
Since inequality in access to adult learning cannot be reduced without institu-
tional and public policy frameworks, nation-specific institutions should be consid-
ered (Desjardins & Rubenson, 2013; Saar, Ure, & Desjardins, 2013). This study also
identifies those who are excluded from access to post-initial learning and the most
at risk of demonstrating poor skills in terms of social equity. To promote more equi-
table access to participation in adult learning and education, combating the cumula-
tive effects of multiple disadvantages is of particular importance. The results of this
study suggest that women, those from low SES backgrounds, low-educated adults,
and adults with less experience of ICT in low-skilled jobs are likely to be marginal-
ized in adult learning. To implement effective educational strategies for the disad-
vantaged groups, the system needs to respond to identifiable groups entering into
trajectories of multiple disadvantages, in particular during the initial education
stage. Therefore, systematic support should target the disadvantaged groups via all
educational policies and interventions.
Last, the influence of societal antecedents found in this study sheds light on the
importance of structural differences in the national systems of adult learning and skill
formation. As a demand-side factor at the macro level, the level of participation in
nonformal learning is associated with the occupational structure and investment in
innovative activities (Roosmaa & Saar, 2012). In market-centered societies, a higher
proportion of low-skilled workers in the occupational structure may decrease partici-
pation in nonformal learning. Innovation characteristics such as employment in high-
tech services and manufacturing might have a positive effect on participation rates as
well. Likewise, adult skill formation is related to the different types of political econo-
mies (Estevez-Abe, Iversen, & Soskice, 2001). In a coordinated market economy,
firms place an emphasis on industry-specific skills and the promise of employment
and unemployment security. The Nordic countries including Sweden, Norway,
Finland, and Denmark encourage firms to invest in training that is based on long-term
worker loyalty, supporting the social democratic welfare state. By contrast, the liberal
market economy provides economic actors with an opportunity to acquire general

16. 16 Adult Education Quarterly 00(0)
skills to deploy their resources for higher returns. Lacking in employment protection
and adequate wages, these liberal welfare-state countries hardly create any incentives
for firms to invest in industry-specific skills. Differences in the social policies may
indicate how training opportunities are distributed for skills development (Roosmaa &
Saar, 2012). At the societal level, however, the empirical results of this study do not
strongly prove that countries with stronger employment protection show higher par-
ticipation rates in nonformal learning related to jobs, as has been suggested by the
relevant research (O’Connell & Bryne, 2012).
Indeed, a limitation of this study is not to ensure that a flexible educational system
at the country level can increase participation in adult learning at the individual level.
In spite of statistical significance on a relationship between educational system and
adult learning participation, that observation is not practically effective because of
sample size issue. Such a small magnitude of effect restricts to verify a mean differ-
ence in problem-solving skill as well. The other constraint of data analysis is to use a
dichotomous variable of adult learning participation, which does not consider various
range of learning activities and specify different impact of adult learning. To enrich the
theoretical foundations relating to adult learning, follow-up research needs to capture
a wider range of adult learning participatory modes beyond the dichotomous classifi-
cation linked to nonformal learning.
Appendix
Correlation Matrix.
1 2 3 4 5 6 7 8 9 10 11
1. Gender —
2. Age 0.016 —
3. Parental education level −0.049 0.104 —
4. Educational attainment 0.042 0.025 −0.078 —
5. Adult education system 0.007 −0.001 0.001 0.402 —
6. Occupational status 0.207 −0.05 −0.091 −0.149 −0.006 —
7. Yearly income −0.184 −0.113 0.001 −0.016 0.005 −0.054 —
8. Learning at work −0.008 0.055 0.012 0.002 −0.01 −0.068 −0.016 —
9. ICT at work −0.097 −0.038 −0.037 −0.038 0.006 −0.235 −0.127 −0.091 —
10. Labor market policy −0.005 0.019 0.007 −0.011 0.154 −0.008 0.011 0.014 0.002 —
11. Educational attainment ×
adult education system
−0.006 −0.028 −0.008 −0.924 −0.433 0.015 −0.023 0.002 −0.001 0.011 —
Note. ICT = information and communication technology.
Declaration of Conflicting Interests
The author declared no potential conflicts of interest with respect to the research, authorship,
and/or publication of this article.
Funding
The author disclosed receipt of the following financial support for the research, authorship, and/
or publication of this article: Financial assistance provided by National Research Foundation of
Korea Grant (NRF-2016S1A3A2924944) is gratefully acknowledged.

17. Kim 17
Note
1. When comparing the group of nontreated individuals with the treated subjects who are
equivalent in all relevant pretreatment characteristics, the differences in outcomes between
the control and the treatment groups indicate the average treatment effects.
ORCID iD
Suehye Kim https://orcid.org/0000-0002-5964-1406
References
Armstrong, T. (2007). The human odyssey: Navigating the twelve stages of life. New York, NY:
Sterling.
Arpino, B., & Mealli, F. (2011). The specification of the propensity score in multilevel observa-
tional studies. Computational Statistics & Data Analysis, 55, 1770-1780.
Blossfeld, H. P., Buchholz, S., Skopek, J., & Triventi, M. (Eds.). (2016). Models of second-
ary education and social inequality: An international comparison. Northhampton, MA:
Edward Elgar.
Blossfeld, H. P., Kilpi-Jakonen, E., de Vilhena, D. V., & Buchholz, S. (2014). Adult learn-
ing in modern societies: An international comparison from a life-course perspective.
Northhampton, MA: Edward Elgar.
Colley, H., James, D., Diment, K., & Tedder, M. (2003). Learning as becoming in voca-
tional education and training: Class, gender and the role of vocational habitus. Journal of
Vocational Education & Training, 55, 471-498.
Damesin, R., Fayolle, J., Fleury, N., Malaquin, M., & Rode, N. (2014). Challenges, actors
and practices of non-formal and informal learning and its validation in Europe. Brussels,
Belgium: European Trade Union Institute.
Desjardins, R. (2013). The economics of adult education. New Directions for Adult and
Continuing Education, 138, 81-90.
Desjardins, R., & Rubenson, K. (2013). Participation patterns in adult education: The role of
institutions and public policy frameworks in resolving coordination problems. European
Journal of Education, 48, 262-280.
Dolton, P. (2004). What do policy makers need to know about the skills of young people
and the school to work transition. Retrieved from http://www.oecd.org/employment/
emp/34474626.pdf
Elder, G. H., & Giele, J. Z. (Eds.). (2009). The craft of life course research. New York, NY:
Guilford Press.
Estevez-Abe, M., Iversen, T., & Soskice, D. (2001). Social protection and the formation of
skills: A reinterpretation of the welfare state. Retrieved from https://pdfs.semanticscholar.
org/6c7a/294833375219875442e4a9365932a49a7184.pdf
Fouarge, D., Schils, T., & De Grip, A. (2013). Why do low-educated workers invest less in
further training? Applied Economics, 45, 2587-2601.
Greene, J. A., Seung, B. Y., & Copeland, D. Z. (2014). Measuring critical components of digital
literacy and their relationships with learning. Computers & Education, 76, 55-69.
Guile, D. (2006). Access, learning and development in the creative and cultural sectors:
From “creative apprenticeship”to “being apprenticed.” Journal of Education and Work,
19, 433-453.

18. 18 Adult Education Quarterly 00(0)
Hanushek, E. A., Schwerdt, G., Wiederhold, S., & Woessmann, L. (2013). Returns to skills
around the world: Evidence from PIAAC. Cambridge, MA: National Bureau of Economic
Research.
Hanushek, E. A., Schwerdt, G., Woessmann, L., & Zhang, L. (2017). General education,
vocational education, and labor-market outcomes over the lifecycle. Journal of Human
Resources, 52, 48-87.
Hickling-Hudson, A. R. (2007). Beyond schooling: The role of adult and community education
in postcolonial change. In R. Arnove & C. A. Torres (Eds.), Comparative education: The
dialectic of the global and the local (pp. 197-216). Lanham, MD: Rowman & Littlefield.
Hong, G., & Raudenbush, S. W. (2005). Effects of kindergarten retention policy on children’s
cognitive growth in reading and mathematics. Educational Evaluation and Policy Analysis,
27, 205-224.
Horrigan, J. B. (2016). Digital readiness gaps. Washington, DC: Pew Research Center.
Retrieved from http://www.pewinternet.org/2016/09/20/digital-readiness-gaps/
Imai, K., & Ratkovic, M. (2014). Covariate balancing propensity score. Journal of the Royal
Statistical Society: Series B (Statistical Methodology), 76, 243-263.
Jacobs, G. E., & Castek, J. (2018). Digital problem solving: The literacies of navigating life in
the digital age. Journal of Adolescent & Adult Literacy, 61, 681-685.
Kim, J., & Seltzer, M. (2007). Causal inference in multilevel settings in which selection pro-
cesses vary across schools (CSE technical report 708). Los Angeles, CA: National Center
for Research on Evaluation, Standards, and Student Testing.
Kline, R. B. (1998). Principles and practice of structural equation modeling. New York, NY:
Guilford Press.
Lam, W. S. E., & Warriner, D. S. (2012). Transnationalism and literacy: Investigating the mobil-
ity of people, languages, texts, and practices in contexts of migration. Reading Research
Quarterly, 47, 191-215.
Leu, D. J., Kinzer, C. K., Coiro, J., Castek, J., & Henry, L. A. (2017). New literacies: A dual-
level theory of the changing nature of literacy, instruction, and assessment. Journal of
Education, 197(2), 1-18.
Levy, F. (2010). How technology changes demands for human skills. Paris, France: OECD.
Mohammadyari, S., & Singh, H. (2015). Understanding the effect of e-learning on individual
performance: The role of digital literacy. Computers & Education, 82, 11-25.
Molnár, G., & Benedek, A. (2014, June). ICT in education: A new paradigm and old obsta-
cle. Paper presented at the ICCGI 2014, The Ninth International Multi-Conference on
Computing in the Global Information Technology, Seville, Spain.
O’Connell, P. J., & Byrne, D. (2012). The determinants and effects of training at work: Bringing
the workplace back in. European Sociological Review, 28(3), 283-300.
Organization for Economic Cooperation and Development. (2013a). OECD skills outlook 2013:
First results from the survey of adult skills. Paris, France: Author.
Organization for Economic Cooperation and Development. (2013b). Public expenditure
on active labour market policies 2013/1. Paris, France: Author. Retrieved from https://
www.oecd-ilibrary.org/employment/public-expenditure-on-active-labour-market-poli
cies-2013-1_lmpxp-table-2013-1-en
Organization for Economic Cooperation and Development. (2015). Adults, computers and
problem solving: What’s the problem? Paris, France: Author.
Powell, M., & Barrientos, A. (2004). Welfare regimes and the welfare mix. European Journal
of Political Research, 43, 83-105.

19. Kim 19
Raudenbush, S. W., & Bryk, A. S. (2002). Hierarchical linear models: Applications and data
analysis methods (Vol. 1). Thousand Oaks, CA: Sage.
Roosmaa, E. L., & Saar, E. (2012). Participation in non-formal learning in EU-15 and EU-8
countries: Demand and supply side factors. International Journal of Lifelong Education,
31, 477-501.
Rosen, D. J., & Vanek, J. B. (2017). Technology for innovation and change in adult basic skills
education. New Directions for Adult and Continuing Education, 155, 51-60.
Rubenson, K. (2006a). Adult education and cohesion. In H. Lauder, P. Brown, J.-A. Dillabough,
& A. H. Halsey (Eds.), Education, globalization, and social change (pp. 936-948). Oxford,
England: Oxford University Press.
Rubenson, K. (2006b). The Nordic model of lifelong learning. Compare, 36, 327-341.
Rubenson, K., & Desjardins, R. (2009). The impact of welfare state regimes on barriers to
participation in adult education a bounded agency model. Adult Education Quarterly, 59,
187-207.
Saar, E., Ure, O. B., & Desjardins, R. (2013). The role of diverse institutions in framing adult
learning systems. European Journal of Education, 48, 213-232.
Steinberg, R. J. (1990). Social construction of skill gender, power, and comparable worth. Work
and occupations, 17, 449-482.
Strauss, A. (1962) Transformations of identity. In A. Rose (Ed.), Human behaviour and
social processes: An interactionist approach (pp. 63-85). London, England: Routledge.
Tsatsaroni, A., & Evans, J. (2014). Adult numeracy and the totally pedagogised society: PIAAC
and other international surveys in the context of global educational policy on lifelong learn-
ing. Educational Studies in Mathematics, 87, 167-186.
Tuijnman, A., & Boudard, E. (2001). Adult education participation in North America:
International perspectives. Ottawa, Ontario, Canada: Statistics Canada.
UNESCO. (2006). Education for all literacy for life, UNESCO education for all global monitor-
ing report. Paris, France: Author.
UNESCO. (2013). 2nd Global report on adult learning and educaton: Rethinking literacy.
Hamburg, Germany: UNESCO Institute for Lifelong learning.
UNESCO. (2015a). Global perspectives on recognising non-formal and informal learning:
Why recognition matters. Hamburg, Germany: UNESCO Institute for Lifelong
Learning.
UNESCO. (2015b). Levelling and recognizing learning outcomes: The use of level descriptors
in the twenty-first century. Paris, France: Author.
UNESCO. (2016). Third global report on adult learning and education (GRALE 3). Hamburg,
Germany: UNESCO Institute for Lifelong Learning.
van Laar, E., van Deursen, A. J., van Dijk, J. A., & de Haan, J. (2017). The relation between
21st-century skills and digital skills: A systematic literature review. Computers in Human
Behavior, 72, 577-588.
Wagner, D. A., & Kozma, R. (2005). New technologies for literacy and adult education. Paris,
France: UNESCO.
Werquin, P. (2010). Recognition of non-formal and informal learning: Outcomes, policies, and
practices. Retrieved from http://www.eucen.eu/sites/default/files/OECD_RNFIFL2010
_Werquin.pdf
Wilson, M., Scalise, K., & Gochyyev, P. (2015). Rethinking ICT literacy: From computer skills
to social network settings. Thinking Skills and Creativity, 18, 65-80.

20. 20 Adult Education Quarterly 00(0)
Author Biography
Suehye Kim is a research professor at the Center for Social Cohesion Education in Korea
University. She worked as a Programme Specialist at the UNESCO Institute for Lifelong
Learning (UIL) in Hamburg, Germany from 2015 to 2017 after attaining a PhD from the State
University of New York in Albany. Her research has mainly addressed the interaction between
micro- and macro-level processes in the dynamics of learning outcomes and the social environ-
ment from a comparative and international perspective.