Impada o1 report_final ibe

1
CRITERIA FOR MEASURING
ADULT EFFECTIVENESS
FOR DISADVANTAGED GROUP
IMPADA PROJECT – OUTCOME 1

2
Contents
1 Introduction ........................................................................................................................ 4
2 Analytical framework.......................................................................................................... 5
2.1 Key definitions ............................................................................................................. 5
2.1.1 Adult education.................................................................................................... 5
2.1.2 Effectiveness of (adult) education ....................................................................... 5
2.1.3 Disadvantaged groups.......................................................................................... 6
2.2 Identification of disadvantaged groups....................................................................... 6
2.2.1 What do we know about PIAAC? ......................................................................... 7
2.2.2 Short typology ...................................................................................................... 9
2.2.3 Different approach to different disadvantaged group?....................................... 9
2.3 Methodology of research............................................................................................ 9
2.4 Comments on desk research..................................................................................... 10
2.5 Comments on focus group interviews ...................................................................... 11
2.6 Comments on documentation analysis..................................................................... 12
3 Different outcomes of (adult) education.......................................................................... 13
3.1 Economic outcomes of adult education.................................................................... 13
3.2 Personal and social returns ....................................................................................... 15
4 Factors influencing the effectiveness of adult education of disadvantaged groups........ 18
4.1 External vs. internal factors influencing effectiveness.............................................. 19
4.2 Participation in adult education................................................................................ 20
4.2.1 Promotion........................................................................................................... 21
4.2.2 Recruitment........................................................................................................ 23
4.3 Persistence in adult education .................................................................................. 24
4.4 Quality of provision ................................................................................................... 28
4.4.1 Length of a course.............................................................................................. 31
4.4.2 Organization of learning..................................................................................... 31
4.4.3 Teachers quality................................................................................................. 33
4.4.4 Other factors ...................................................................................................... 35
5 Methods of assessment.................................................................................................... 36
6 Criteria on measuring adult education effectiveness....................................................... 41

3
6.1 Criteria oriented directly on the expected outcomes (individual perspective):....... 41
6.1.1 Objective measures............................................................................................ 41
7 Key conclusions....................................................................Error! Bookmark not defined.
8 Bibliography ...................................................................................................................... 47

4
1 Introduction
This report presents a review of findings from studies and publications about the
effectiveness of adult education for disadvantaged groups. The main goal of the study was to
recognize key criteria for measuring the effectiveness of adult education. Therefore, we
focused the review around two questions: (i) what does it mean that education is effective
and (ii) what are the ways to measure it. Answering those questions will help to understand
the challenges of adult education for disadvantaged groups and – in consequence – to
develop proper solutions of improving effectiveness of this education.
This report is first outcome of IMPADA project - Improving Adult Education of Disadvantaged
Groups. The project aims to develop a Common Assessment Framework to evaluate the
effectiveness of adult education for disadvantaged. Such Framework will help adult
education providers in self-assessment, self-evaluation and – in consequence – improvement
of provision of adult learning services for disadvantaged groups.
The report summarises two activities within the project:
A1 Desk research on factors that affect the effectiveness of adult education for
disadvantaged groups
A2 Focus groups with adult education stakeholders aimed at identification of criteria
influencing the effectiveness of AE for disadvantaged groups
First part of the report presents the definitions of key terms and methodology of the
research. Second part presents considerations about how to understand the statement that
the education is “effective” and general findings from the research on effectiveness of adult
education. Third part presents key factors affecting the effectiveness of adult education
disadvantaged groups, grouped in three areas: participation in adult education, persistence
and outcomes of learning.
As seen in chapter 2.1.1, in the project adult education is defined as a non-vocational
education. Linking this assumption with the focus on disadvantaged groups led us to pay
particular attention to the “basic” or “foundation skills”(see Looney, 2008). Among them
most important are literacy (e.g. reading and writing) and numeracy skills as well as soft and
digital skills.

5
2 Analytical framework
This chapter presents the approach, methods, sources and definitions of terms used in the
report.
2.1 Key definitions
The aim of the report is to find and analyse the criteria for measuring the effectiveness of
adult education for disadvantaged groups. Therefore, the first step in our work was to define
key terms: “adult education”, “effectiveness of (adult) education” and “disadvantaged
groups”. This sub-chapter presents agreed definitions and comments on those three terms.
2.1.1 Adult education
The term “adult education” is sometimes understood as any type of education that occurs
after the theoretical age of completion formal education. In IMPADA project we use this
term in a narrower sense and follow the definition of Erasmus+ guidelines, which states that
“adult education is all forms of non-vocational adult education, whether of a formal, non-
formal or informal nature” (European Commission, 2016).
In that sense learning of adult persons is divided into three types: higher education,
vocational education and adult education. One can argue (and – as will be shown later in this
report – there are many arguments for this hypothesis) that any type of education can have
an impact on labour market status of a learner. We thus assume that “vocational education”
is the one with direct impact on labour market status and the one where its improvement is
a primary goal.
As a consequence, the adult education in this report usually (but not explicitly) refers to
basic skills courses. Among them the most popular are those aimed at providing literacy and
numeracy education, followed by soft skills, ICT and motor skills education. Those are also
primary themes in the literature on adult education (see e.g. Vorhaus, Litster, Frearson, &
Johnson, 2011; Windisch, 2015). Important characteristic of such education is that – contrary
to vocational education – it does not always lead to a qualification, e.g. in “community
learning” or “family literacy courses”.
2.1.2 Effectiveness of (adult) education
The term “effectiveness of education” is usually narrowly understood in economic terms.
Indeed, the impact of education on earnings and employability is well proven, and appears

6
to be one of the most important characteristics of education. But one must conclude that
economic outcomes are not solely results of education. Among others, one can mention
better health, citizenship, self-confidence, stronger relationships etc.
Therefore, in this report we understand the effectiveness of adult education as all outcomes
of education that improve learner’s quality of life.
This statement is further discussed in chapter Error! Reference source not found..
2.1.3 Disadvantaged groups
By the term “disadvantaged groups” we understand the adults, who lack the basic skills
(literacy, numeracy, soft skills) in a way that disturbs their functioning in society. Such group
is usually link with some “physical” characteristics as age, gender, race, state of health, or
economic status. We therefore treat this term broadly, with no special focus on any of the
more specific groups.
Such definition stays in accordance with other known from literature. E.g. (Jarvis, 2005)
defines disadvantaged as “persons who, by reason of physical or social handicap, have not
had equality of opportunity to achieve their potential through educational means”
(Anderson & Niemi, 1969) noticed that disadvantaged status is not only a consequence of
economic (or: objective) factors, but also subjective determinants that shapes person’s
opinion about herself. They also list a number of synonyms of this term: “hard-core poor”,
“the lower socio-economic citizenry”, “low income people”, “the culturally deprived”, “the
functionally illiterate”, “the educationally deficient”, “the hard-core unemployed”.
They assume, that disadvantaged learner has two characteristics: living in a poverty and gaps
and limitations in the “cognitive environment”, what he understands as specific family
status.
Next chapter shows short findings on the identification of disadvantaged adult learners.
2.2 Identification of disadvantaged groups
There are many definitions of disadvantaged groups. In the report we focus on low-skilled
adults, since the low level of skills is usually correlated with other aspects of being
disadvantaged. Windisch(2015) finds three main characteristics of low-skilled adults:
- Low skills are strongly correlated with low level of education (ISCED 0-2). A group of
particularly high risk are drop-outs;

7
- Such correlation is the strongest in the group of young adults. Among older adults
other factors start playing crucial role. These include for example: working in an
environment that requires the use of literacy and numeracy skills as well asusing the
skills in family or social context (e.g. helping children with homework);
- Literacy and numeracy skills tend to deteriorate because of non-use.
German Ministry of Education advises to providers to use some hints in recognising the low-
skilled adults. The most often factors appearing in CVs are: low level of formal education;
frequent changes in education (different schools), unexpected breaks in education or
employment (BMBF, 2012).
National strategies such as England’s Skills for Life Strategy, Australia’s National Foundation
Skills Strategy for Adults, Indonesia’s AKRAB programme, France’s National Agency for the
Fight against Illiteracy, Germany’s National Strategy for Literacy and Basic Education of
Adults are often designed as to improve the situation of adult disadvantaged learners, thus
they are an important source of knowledge.
2.2.1 What do we know about adults with low skills?
Following the assumption that disadvantaged learners are those who lack basic skills and
qualifications, we reviewed the results from the Programme of International Assessment of
Adult Competencies (PIAAC). PIAAC is an international survey hold by OECD in order to
recognise the structural factors influencing adult competencies and to diagnose main
deficits. The survey was held among representative adults from different OECD countries,
with the same questionnaire, including measurement of numeracy and literacy,, so it is
possible to make the international comparisons based on a results of PIAAC. More than 160
thousand adults from 24 countries took part in the first edition of the survey (OECD, 2013a).
The unique feature of PIAAC is that it measured skills levels among adult population using
computer-based or paper tests, identifying those with lowest level of skills, i.e. those whose
skills are at level 1 or below.
We used the PIAAC data to identify the vulnerable groups in four countries that are taking
part in IMPADA project: Poland, UK, Estonia and Italy and participated in PIAAC (Greece was
not included, as Greece was not participating in the PIAAC survey).
The low-skilled adults were there defined as those who performed less than 226 points in
numeracy or literacy tests1. Such an absolute measure caused that the share of people in the
total adult population (in age group 15-65) classified as low-skilled was different in different
countries: Poland – 29,5%, UK – 25,8%, Italy – 37,8%, Estonia – 18,5%.
1
PIAACscoresare normalised,sothe OECD-average is500. Thus the score 226 may be interpretedas
“no more than a half of an OECD-average score”.

8
The analysis allowed identifying from 3 to 11 groups (clusters) of adults2, that are particularly
at risk of being low-skilled. Those clusters are presented in the Table 1.
Table 1 Identification of vulnerable groups on PIAAC data
Country Group Group as
percentage
of total
population
Percentage of
a group
identified as
low-skilled
Italy ISCED 1 or lower education; retired or working;
disabled
6,3% 69,9%
Italy ISCED 1 or lower education; unemployed or
housekeeping
4,5% 83,1%
Italy ISCED 2 education; possesses less than 10
books; unemployed for more than 5 years
6,6% 68,2%
Poland ISCED 2 or lower education; more than 47 years
old; possesses less than 10 books, single
1,5% 84,2%
Poland ISCED 1 education; possesses less than 100
books
2,4% 68,2%
Poland ISCED 2 or lower education, low ISEI index3
(<22.16)
2,5% 64%
UK ISCED 2 or lower education; housekeeping or
unemployed; possesses less than 25 books
2,9% 83,2%
UK ISCED 2 or lower education; low ISEI index
(<28,5), low monthly earnings
1,9% 78,2%
UK ISCED 4 or lower education, age above 40 years;
not working for 5 years or more
1,6% 78,6%
Estonia Unemployed or housekeeping; possessing less
than 10 books
1,8% 58,2%
Source: own estimates based on PIAAC database (OECD, 2013a)
The table shows those groups, in which the share of low-skilled is above 50% and such
statistics is found to be statistically significant. As one can notice, in almost all cases the key
distinctive characteristic is low educational attainment (at ISCED level 2 or lower), but it is
not always the case. Particularly interesting seems the Estonian case, where the
unemployment and social environment (as one can understand the number of books) are
sufficient factors leading to low level of skills, regardless the education.
2
Please note,thatgroupsmayoverlap.
3
International Socio-EconomicIndex of OccupationalStatus

9
2.2.2 Short typology
The existing literature allows identifying some attributes of disadvantaged groups that are
most likely to be identified as low-skilled. Such characteristics are:
- Age (older workers to be disadvantaged)
- Level of educational attainment (those with lower formal qualifications to be
disadvantaged)
- Labour market status (unemployed or low performers)
- Social status (e.g. past criminals)
- Geographical factors (rural or post-industrial areas)
- Demographic status (minorities or migrants)
- Physical characteristics (mental or physical disabilities).
2.2.3 Different approach to different disadvantaged group?
There is no consensus in literature if the disadvantaged adults should be treated as one
group. Some sources suggest that adult education programmes are more effective when
persons from different subgroups are treated differently. If this were true, educational
providers would have to design their programmes specifically for given groups (migrants,
single mothers etc.). However, such evidence is limited. (Martin & Grubb, 2001) found that
some measures are more effective for women re-entrants, while less effective for prime-age
men with low initial education (formal classroom training and on-the-job training). Contrary,
they see “special youth measures” such as training and employment subsidies appear not to
be effective for disadvantaged youths, who seem to be main recipient (see also Carneiro &
Heckman, 2003).
Since there is no clear evidence supporting this view, we address this report to all the adults
classified as low-skilled, with an adequate commentary where the evidence suggesting
different approach for a given group is available.
2.3 Methodology of research
There were three sources of information in this report. First of all, we did an extensive desk
research of key reports and journal articles dealing with issues of adult education,
effectiveness of education or disadvantaged groups. Secondly, the focus group interviews
(FGIs) were conducted by all five partner institutions. Finally project partners: UPTER, DACES
and ENAEA, who serve as adult education providers, shared their internal documentation
that was analysed from the perspective of the main questions in the project. Figure 1
presents the methodology of the research. Desk research and focus group interviews are
fully described in chapters 2.4 and 2.5, respectively.

10
Figure 1 Analytical framework of a research
Source: Authors
2.4 Comments on desk research
Since there are few publications directly focusing on the issue of measuring the effectiveness
of adult education for disadvantaged groups, we propose the analytical framework that
allows using relevant information from literature that is related to this topic. We start with
the general topic of effectiveness of education, in order to find some general findings, and
then we checked how it is applicable for adult education and education for disadvantaged
groups. Figure 2 shows the analytical framework.
Figure 2 Analytical framework of desk research
Source: Authors
Effectiveness of education
Effectiveness of adult education
Effectiveness of adult education on

11
There were numerous sources used in the research. Journal articles were obtained via
databases such as: Google Scholar, Jstor, Science Direct EBSCO, with particular focus on such
journals as: Adult Education Quarterly, Oxford Review of Education, Economics of Education
Review, Studies in the Education of Adults, Journal of Adolescent & Adult Literacy. The
keywords used in the analysis were e.g.: “Effectiveness of adult education”, “Effectiveness of
literacy courses”, “Effectiveness of numeracy courses”, “Effectiveness of basic skills”,
“disadvantaged adult education”, “quality of provision in adult education”, “quality of
teachers in adult education” etc.
Additionally, we reviewed reports prepared by following institutions: OECD, UNESCO,
National Institute of Adult Continuing Education, DeutschesInstitut für Erwachsenenbildung,
Institutul Român Educație a Adulților, UKCES, National Research and Development Centre
for Adult Literacy and Numeracy (NRDC), Bundesinstitut für Erwachsenenbildung.
The bibliography is presented in the last chapter of the report.
2.5 Comments on focus group interviews
Following the work plan of the IMPADA project, agreed methodology instructions and
methodological tools, the partnership conducted 5 focus groups (of 1.30-2.30 h duration), in
each partnership country. The participants of the focus groups where: a) adult education
experts, and b) decision makers in the field of adult education, ideal for providing exclusive
insight on the elaboration of criteria to evaluate the effectiveness of adult education for
disadvantaged groups. Key individuals from the field of Adult Education were nominated and
invited by the partnership organizations, taking into account their high level of expertise and
familiarity with the topic.
All focus groups were conducted according to the agreed semi-structured scenario, which
was necessary to gather coherent and comparable data among all project partners. The
synthesized data analysis of all focus groups followed the content analysis approach and
involved coding participants’ open-ended answers to main questions into closed categories.
It allowed to identify issues that seem to be most important and relevant for the study as
well as organize the data into various typologies (by comparing and contrasting collected
information) (PROMEA, 2016). The conclusions from the analysis appear in the report.

12
2.6 Comments on documentation analysis
Project partners who practice adult education: DACES, UPTER and ENAEA made available
their own assessment reports. That material was further analysed by IBE research team. The
conclusions from the assessment reports are also a part of findings in this report.

13
3 Outcomes of (adult) education
The term “effectiveness of education” is usually associated with its impact on labour market
status of learner. Since late 60’, there is a standard method in economics of education to
measure the effectiveness of education with so called “Mincerian wage equations”, named
after Jacob Mincer (see (Heckman, Lochner, & Todd, 2003). Those equations allow
measuring the rates of returns to education, e.g. how much will the wage increase with an
additional year of education. There is a vast amount of empirical literature proving, that
there is strong, positive and statistically significant correlation between education and wages
in virtually all the countries in the world4 (see e.g. (Card, 1999; Psacharopoulos & Patrinos,
2002).
However, the economic returns to education are not the only effects of education.
McMahon (1999) basing on macroeconomic perspective, shows 7 “sectors” influenced by
education: economic growth, population and health (net population growth, fertility,
longevity, infant mortality), democratization, human rights and political stability, reduction
of poverty and inequality, environment (forest and wildlife preservation, air pollution, water
pollution), crime and drug use, labour force participation.
3.1 Economic outcomes of adulteducation
The outcomes of adult education are similar to those described in case of education system
in general, but evidences are often less clear. Vorhaus et al. (2011) notice that – despite the
fact that literacy and numeracy skills are strongly correlated with labour market status
(employment status and earnings) – it is not obvious, if such relation is also true for literacy
and numeracy qualifications. It appears that in case of adult learning signalling effect of
actual literacy and numeracy skills5 is much less important than in case of formal education.
Similarly, (Champagne, 1987) examined the effects of adult education such as: educational
participation, obtaining qualifications, lower dependence on public assistance (e.g.
unemployment benefits), employability, quality of employment (temporality of contract,
income, job-satisfaction), confidence. The author focused on the effectiveness of career
counselling within adult education to prove, that there exists some moderate effect on all of
those characteristics.
4
Even though, it is unclear if it is an effect of “signalling” or improvement in human capital.
5
Signallingineconomicliteratureisunderstoodasa situation, inwhich havingcertificate of diploma
is a “signal” for a potential employer that a person has a given set of skills and capabilities. In such
understanding of education signals happen to be more important than an actual increase of
knowledge, skills or competencies (Spence, 1973).

14
Broader criteria of effectiveness are also an important element of evaluation and
assessment frameworks. American National Reporting System for Adult Education (NRS) lists
three key measures: “educational gain”, entered or retained job and further education (e.g.
entering post-secondary education). Educational gain is here defined as the difference
between competencies of a learner at the beginning and at the end of a programme
(National Reporting System for Adult Education, 2015).
NRS lists also four follow-up measures, based on formerly described criteria:
- Entering employment
- Retaining employment within two years
- Completing secondary education (obtaining GED certificate)
- Entering post-secondary education.
(Vorhaus et al., 2011) identified several outcomes of adult education. First, they concluded
that economic outcomes are important, but not crucial in adult education. There is some
moderate influence on earnings, though it is smaller than expected by comparisons with
“regular” types of education. There is also an influence on employment rates: adults who
participate in adult education tend to have higher employment rates. That effect is not
equal: for men the bigger effectiveness is obtained through literacy courses, while for
women bigger impact have numeracy programmes.
However, the most important outcome of adult education recognised by Vorhaus et al.
(2011) is the improvement of relative labour market status of a person via improvement of
her job and job-seeking skills, such as:.
Job-seeking skills:
- Increased confidence in applying for a job
- Confidence in getting a job
- Increased aspirations (applying for a better job)
- Increased motivation
- Enhanced job search skills;
Job skills:
- Team work
- Appropriate behaviour
- Willingness in taking responsibility
- Timekeeping
- Better stocktaking skills.
Vorhaus et al. (2011) underline that
“However, over the course of the programmes, the participants’ optimism about the
employment impacts of the programmes tended to decrease. Individuals reduced

15
their expectations about their future job prospects, and often became frustrated and
discouraged.”(p. 36)
Some interesting and non-obvious findings are the element of Speckesser & Bewley (2006)
report. They analysed three types of work-based learning for adults: “Basic Employability
Trainings”, “Short Job-Focused Trainings” and “Longer Occupational Trainings” to examine
their long-term (after 40 months) effects on employability and dependency on social
benefits. Short Job-Focused Trainings are defined as a short-term (up to 6 weeks)
programmes aimed at increasing the job match, e.g. training to obtain the health certificates
required in applying for a job. Longer occupational trainings are the programmes up to 14
weeks designed to obtain some particular qualification6. Finally, Basic Employability
Trainings are longer (up to 26 weeks) programmes addressed to adults who have a poor
level of basic skills and focus on general literacy and numeracy skills.
According to their research, Short Job-Focused Training had the most significant impact: the
employment rate among participants was increased by 5 percentage points and that effect
seemed to be sustainable. Effect of Longer Occupational Trainings was similar, about 4
percentage points, but delayed: it is therefore important not to measure the impact
immediately after the completion of the course, but also later in time. Finally, the Basic
Employability Trainings also proved to have a positive impact on employment rates, however
no impact was found on benefits dependency. That may be considered as a field that has to
be better examined in order to find solutions and policies that would lead to reduced benefit
dependency.
3.2 Personaland social returns
It is important to notice, as researchers suggest, that some of the literacy and numeracy
programmes outcomes cannot be examined by a typical quantitative comparison of skills
proficiency at the beginning and after the completion of the programme. There are some
effects, such as improvements in confidence, teamwork, employee initiative and problem
solving that cannot be measured in a typical way (Alkema & Rean, 2014).
This way of understanding the role of adult education was strongly visible among IMPADA
focus groups' participants who – when asked what type of content should be included in
adult education training programmes for disadvantaged groups – indicated priority focus on
‘soft’ skills and general well-being rather than just outcome qualifications (5/5 indications).
Soft skills (e.g. confidence, social skills, self-belief, better routine) were often mentioned on
the occasion of answering the question about what are the areas that could provide input
for the assessment of the education provided (3/5 indications). Other often mentioned areas
6
Consequently,theyfall apartthe definitionof adulteducationpresentedinChapter 2.1.1.

16
were: “progression and destinations”7, including ‘continuing further adult education
programmes’ and ‘improvement and use of ICT skills’ (3/5) and “employability – life
transformation” (3/5).
(Vorhaus et al., 2011) in their summary of research and publications list the following
outcomes of adult education that are linked with psychical wellbeing of a person:
- Increased self-confidence
- Confidence to try new things
- Self-esteem
- Belief in own ability
- Belief in personal achievement
- Reduced embarrassment or stigma
- Improved health outcomes (physical and psychical)
- Positive attitude.
Second field of non-economic outcomes found in the same report are improvements in
family and everyday life. Those include:
- Increased ability to help children with their homework
- Increased confidence to help children
- Increased confidence to undertake the activities that involve literacy and numeracy
skills, such as signing the agreements
- Ability to contribute to family life.
The adult education has also a positive impact on participation in further education.
Furthermore, the effect of increased confidence and motivation to engage or re-engage to
learning appears to be durable (Evans & Waite, 2008).
Ofsted’s report on literacy and numeracy of foreigners suggested that the programmes had
positive impact on soft skills, ICT skills, self-confidence and self-efficacy. Some learners
benefited also in their level of oral communication (Ofsted, 2003).
Other literature by (Windisch, 2015)or by (Mcintosh, 2004) also confirms the following
(mentioned before) outcomes:
- Gains to self-confidence
- Encouragement to get further education
- Better family background, via helping children.
Centre for Literacy of Quebec (2010, p. 8) emphasized the impact of adult education on
social capital:
7Broadly understood as learners’participation in life-longlearning(LLL).

17
“These studies, as well as related discussions (Sticht 1999, 2000, 2006, 2007), and
literature surveys such as those produced by Benseman and Tobias (2003), Westell
(2005), Gray (2006), Hartley and Horne (2006), and Salomon (2009), point
overwhelmingly to social capital as an almost ubiquitous outcome of adult learning
and literacy interventions internationally in the past twenty years’ (…). Social capital
outcomes include communicating and becoming involved with others, whether at
work or home, with the family, or in the community.” (p. 8)
The fact that non-economic outcomes of adult education are sometimes equally or more
important than economic outcomes is also visible among motivations of learners. As
(Vorhaus et al., 2011) noticed,
"Learners are often motivated by a far wider range of factors than the wish to improve
job performance, and managers by factors other than the desire to fill skill gaps or
improve productivity" (p. 62).
That fact also should serve as a basis while discussing learners motivation on participation
and persistence in programmes, issues that will be further discussed in chapters 4.2 and 4.3.

18
4 Factors influencing the effectiveness of adult education of
Even though some solutions are more effective than others, there is much evidence showing
that basic skills courses are generally effective in terms of improving adult learners skills
(Myers & Myles, 2005; Vorhaus et al., 2011; Windisch, 2015). Myers & Myles (2005) found
that the least educated learners (or disadvantaged) are most likely to gain from the adult
education. This effect is not equal for all disadvantaged groups, e.g. evidence suggests that
adult women gain more than men (Heckman, 2003). The problem is that, at the same time,
disadvantaged adults are least likely to participate in any type of education (Condelli,
Kirshtein, Silver-Pacuilla, Reder, & Wrihley, 2010).
Vorhaus et al. (2011) states that
“Improved practice often precedes improved achievement: participation in ALN
programmes leads to increased engagement in literacy and numeracy practices, and
over time this can lead to gains in proficiency and achievement”. (p. 62)
To better understand the character and subtleties of adult education effectiveness, we
started with the analysis of factors influencing adult education for disadvantaged groups.
Factors from the first group (external factors) – generally – cannot be improved by the
educational providers, so they seem to be outside the scope of this study. However, each
educational provider has to be conscious of those factors. Such external factors sometimes
serve as an obstacle, so educational provider has to adjust its programme or organization in
order to fully utilize its potential and benefits from its own work. The second group are the
institutional factors (Ofsted, 2015).
In general terms most of factors (either external or institutional) influence also other types
of education (e.g. shape of curricula). However, there are two crucial differences:
1) adult learners from disadvantaged groups belong also to “hard-to-reach” groups, so
the participation rates are much lower;
2) rates of achievement in adult education are non-linear, e.g. learners who completed
the programme benefit (on average) more than twice more than those who dropped
out in the middle of a programme (see e.g. Vorhaus et al., 2011).
Given the above, the third part of this chapter, focused on the specificity of adult education
for disadvantaged groups (especially basic skills teaching), is divided into 3 sections, and
each of them describe one of the issues particularly important in case of this type of
education. It should be noted that identified areas are complex to assess, as they are
strongly influenced by both external and institutional factors. These are:

19
- Participation in adult education that – apart from other impacts – depends on
adequate and targeted promotion and on proper recruitment of adults from
disadvantaged groups to adult learning;
- Persistence in adult learning that, according to subject literature – apart from other
impacts – depends on actions aimed at enhancing adult learners to finish courses;
- Quality of provision that means solutions of improvement outcomes of the students
who finish the course.
This approach will be followed in consecutive chapters. Each of sub-chapters follows the
same structure: in the first part we list hypothetical factors that influence the effectiveness
of adult education of disadvantaged groups. Second, we check the evidence in a literature
and exclude those that have no proven impact – since they seem unimportant or
insignificant. Finally, we find the criteria by which such factors may be measured.
This approach is graphically shown in the Figure 3.
Figure 3 Key aspects of effective adult education
Source: Authors
4.1 External vs. internalfactorsinfluencing effectiveness of adulteducation
Factors having positive or negative impact on effectiveness of adult education for
disadvantaged groups can be generally divided into two types: external and internal. The
latter can be improved by adult education providers on their own. The former cannot be
controlled, but it is also important two analyse them. Full knowledge on external factors
shaping adult education helps providers, how to adjust their programme in order to
overcome the given obstacles.
There are three main types of external factors affecting adult education:
1. Public policies on education; this issue is not limited to policies on adult education,
since it can also be affected by – say – formal elementary education;

20
2. Social policy – vast majority of programmes for adults from disadvantaged groups is
addressed to those, who are also the beneficiaries of social policies. It is particularly
crucial in case of activisation policies or different types of mandatory programmes.
The internal factors are then introduced in further sub-chapters.
4.2 Participation in adult education
Participation barriers – e.g. lack of awareness of learning opportunities as well as low
motivation to learn – are one of the most important factors influencing the effectiveness of
adult education (Windisch, 2015). Various types of such barriers apply both to unemployed
and those whose workplaces don’t play the role of learning environments by engaging their
employees in self-development and providing them basic education and training (Porras-
Hernández & Salinas-Amescua, 2012).
The most common types of barriers to lifelong learning, identified in a lot of previous
research and applicable also to adult education (Desjardins & Rubenson, 2013; Litster, 2007;
Windisch, 2015), are:
i) Situational barriers, created by learner’s personal situation. Usually it is linked
with lack of time, caused by family and job obligations. It may also refer to other
physical barriers as inability to travel.
ii) Institutional barriers that occur when learner is unable to fill the requirements to
participate in a programme (financial or formal, e.g. lack of prerequisite
qualifications);
iii) Dispositional/psychological barriers when a learner has negative attitude to
participate in a programme. In case of disadvantaged learners it is very often
linked with bad formal education experiences;
iv) Informational barriers when a learner is unable to receive a proper information
about the actual offer and potential benefits of a programme;
v) Financial constraints that are strictly linked with situational and institutional
barriers.
(Desjardins & Rubenson, 2013) conclude that there are two key aspects of barriers to adult
education: individual and structural barriers. Educational providers have to, therefore, apply
different strategies to different sets of problems. It is particularly addressed to institutional
barriers at the provider level.
From the OECD’s review based on data from years 1999-2004 (OECD, 2006) one can
conclude that – in order to improve participation in LLL, including participation in adult
education – educational authorities and policy makers should create and implement public

21
policy focused on: (i) creating the structural preconditions for raising the benefits of basic
skills learning; (ii) sharing information on learning opportunities and benefits; (iii) promoting
well-designed co-financing arrangements; (iv) improving delivery and quality control; and (v)
ensuring policy co-ordination and coherence by involving stakeholders such as social
partners, employers, labour unions, and education providers.
Apart from the efforts that should be made by authorities, one can indicate activities
necessary to implement by education providers from that promotion and recruitment seem
to be the most important in the context of the need to increase disadvantaged adults’
participation in education.
4.2.1 Promotion
First aspect of an effective engagement into adult education programmes is its proper
promotion. Disadvantaged adults often lack the information on the possibilities of taking
part in adult education. One of the most important reasons is that they use another media
than those in which such programmes are promoted. Furthermore, even those who have
information about the possibility to apply to a programme often lack the knowledge about
the potential benefits of such programmes and, consequently, are not encouraged to apply.
Finally, disadvantaged (or hard-to-reach) adults very often have bad formal school
experiences, so they are discouraged from taking part in any other kind of education.
The latter issue is also an important factor in discussion the mandatory programmes (e.g.
when participation in adult education is requirement for receiving unemployment benefits).
People who are discouraged from any kind of education are very unlikely to achieve any
significant progress, while participating in adult education. Furthermore, lack of progress
only deepens the discouragement, creating a harsh vicious circle. Thus, the promotion of
positive outcomes of adult education should also be directed to mandatory students.
All of those factors cause that the effective adult educational provider should implement a
number of strategies to adequately promote its services. One can find a growing number of
the literature on which strategies happens to be effective and why.
(Windisch, 2015) identifies some characteristics of promotion the hard-to-reach groups. First
of all, they are far more easily reached in community context. Thus, one way to improve the
participation rates is to work in community (Hamilton & Wilson, 2005). Similarly, (Mcintosh,
2004) suggests that the one-to-one approach is the most effective way to engage learners.
Such approach may be best achieved in community context.
Communities are also good places to provide education programmes. Places such as
community centres or workplaces are more accessible for potential learners, so it is much
easier to convince them to try applying to the programme.

22
There are also another benefits of communities. If a person live in a community where
education is believed to be profitable or valuable he/she is more likely to engage in any kind
of formal or non-formal education. Thus, educational providers may try to change attitude of
whole community.
There is also growing body of evidence on how the programmes are perceived by potential
learners. Some researchers suggest that a name of a course is one of the factors increasing
or decreasing participation rates. Programmes which names directly suggest that they are
addressed to the low-skilled or disadvantaged learners (e.g. “Reading and writing”) are
perceived by learners as a kind of stigma. To avoid such risks, it is recommended to use
indirect names, such as “communication skills”.
Another aspect that must be analysed by an education provider is how a certain programme
fits into specific type of disadvantaged learners. For instance, it is shown that adults who
have higher numeracy levels are more likely to participate in on-the-job trainings (Parsons &
Bynner, 2007). Such observations have to be taken into account while designing promotion
and recruitment activities.
Even though adult education is understood as non-vocational type of education, one should
not pass over the occupational context. As Vorhaus et al. (2011, p. 62) noticed:
“Workplace basic skills courses can reach people who are not normally involved in
continuous education or training”. (p. 62)
This aspect was noticed by NAO (2008), who claims that the first crucial point of a successful
educational offer addressed to adults from disadvantaged groups is creation of an offer
integrated with a specific place of work (specific job). This can be explained by the fact that
people who are away from education are likely to be interested in returning to learning, only
if they believe that the knowledge they can gain will help them to take a chosen job. On the
basis of positive examples one may identify specific factors that play a role in this process.
These include enhanced working environment, professional support provided by a qualified
tutor experienced in working with people with difficulties in learning and the possibility of
validation and certification of informal and non-formal learning (e.g. skills acquired at work),
particularly important because otherwise skills acquired in such ways would not be
recognized by other (future) employers. Ambos stresses also that the low skilled prefer to
acquire professional skills informally or in non-formal way, in particular through observation
and practice as well as through training or apprenticeships in their workplace. At the same
time, they demonstrate distance to classic education opportunities (such as courses
organized outside their working environment) and to traditional methods of learning (such
as reading professional literature or learning via Internet) (Steiner, Voglhofer, Schneeweiß,
Tamara Baca, & Fellinger-Fritz, 2012).
Individual counseling is one of the most important factors that contribute to arousing
interest in further education among members of this group. With respect to people who are

23
"away from education" jet still professionally active, it is recommended to organize such
counseling at their place of work, because it allows to clearer understand their specific
professional situation. Furthermore, in case of people that wish to learn and return to the
labor market, individual counseling allows facing their fear for failure, which is often
indicated as the main barrier to making decisions about further education. It applies
especially to elder people who are often afraid that they will not be able to work with much
younger coworkers (ibidem).
In order to reach this group, it is important that educational offer was easily accessible.
Three aspects play here a key role: 1) easy access to education programme, without
bureaucratic procedures, 2) lack of entrance exam that would generate high risk of failure, 3)
easy access to the place where a process of education is being realized. Accessibility of
educational offer is also connected with its costs. It is obvious that free offers are particularly
attractive and thus easily accessible. Since people belonging to this group are usually low
paid and pay little attention to education, the need to cover costs of education is one of the
main barriers to learning. At the same time, these people often lack the ability to find a
suitable offer via Internet. For these reasons, it is important to create opportunities to
provide information about learning programmes with the use of methods alternative to
Internet. One of such methods is posting ads in regional and local press (Steiner et al., 2012).
4.2.2 Recruitment
Windisch (2015) presents the approach designed by German Ministry of Education and
Research (BMBF). According to this approach, recruitment of adults with low basic skills
involves two key elements: recognition of such persons and proper talking to them. Apart
from demographic or social characteristics (see chapter 2.2), it is advised to analyse certain
behavioural characteristics and – on this basis – to recognise needs of adult learners. Typical
recognised behaviours are:
- Refusals of reading or writing, using hardly possible excuses (such as the lack of
glasses)
- Refusals of educational offers
- Signing unread documents
- Ignoring written materials, e.g. flyers
- Inability to read or write a text longer than a single sentences.
(Windisch, 2015) finds also that screening and initial assessment are crucial. However, it has
to be conducted sensitively and carefully in order to better meet the learners’ needs and not
to demoralise them.
Good recognition of adults with low skills is crucial for successful recruitment. First of all, it
allows a provider to place a learner in a proper programme – if the person is wrongly placed
in the programme (e.g. he/she finds the programme as a throwing into deep water), he/she

24
can be easily discouraged, drop out from the programme (see chapter Error! Reference source
not found.) and, finally, refuse to participate in another programmes. Secondly, proper
recognition of a disadvantaged person allows provider to adapt the recruitment process in
order to increase the probability of convincing him/her to participate in a programme. BMBF
lists five key aspects of adequate communication with low-skilled adults: sympathy,
empathy, openness, patience and individual approach. Sensitiveness and respect to all
learners may be crucial in encouraging them to participate in adult learning and in further
successful completion of the program.
Next aspect identified by (Windisch, 2015) are partnerships of employers and trade unions.
They are considered as an important and effective drivers of workplace based basic
education programmes that help to support learners’ motivation and persistence.
In addition, it is typical for this group of people that they generally are not able to express
concrete expectations towards educational offer, which is of their interest. It is often due to
their lack of knowledge and awareness of the available educational opportunities. For this
reason, institutions involved in adult education have to identify and meet interests and
needs of these people and create offer, which will be most adapted to their needs. One of
the possibilities of reaching this group is to place educational offer in group’s cultural and
social environment (community context - see above). (ibidem)
The importance of proper recruitment was also stressed by the participants of IMPADA focus
groups, especially when they answered the question about what could be improved in the
adult education provided to disadvantaged groups in order to deal with the issue of group
heterogeneity. In 4 out of 5 groups "targeted appropriate support and differentiation"
(including - among others - 'recognition of prior knowledge') was pointed, in 3 out of 5 -
"ethnographic approach" (including - among others - 'using of methods such as interviews,
observation, cultural mediations etc., in order to understand each group of learners needs'
as well as 'challenging stereotypes').
4.3 Persistence in adult education
As was mentioned in the introduction to this chapter, the key to successful adult education
programme is persistence. Learners benefit the most when they complete whole
programme as well as such impact is correlated with attendance to particular lessons
(Windisch, 2015).
However, adults from disadvantaged groups are not only less likely to participate in adult
education, but also are more likely to drop out during the programme. (Coben et al., 2007)
mention that average attendance in 150-hours courses was just 39 hours – such pattern is
also a case in many other programmes. Therefore, keeping high motivation among learners

25
is one of the key tasks for educational provider. There is no unequivocal evidence suggesting
who is most likely to drop out during the course, mainly because hardly any provider
monitors this issue which results in lack of proper data (Vorhaus et al., 2011).
Numerous sources (Alkema & Rean, 2014; European Commission, EACEA, & Eurydice, 2015;
Vorhaus et al., 2011; Windisch, 2015) cite guidance services for disadvantaged adults as an
important way to improve their motivation to participate in adult education. Though this
factor seems to be crucial, there are hardly any European countries that provide guidance
services for adults who do not participate in public employment services, and this group is
limited and do not cover a large fraction of disadvantaged adults (European Commission et
al., 2015). Windisch (2015), therefore, concludes that a structural guidance services aimed to
support disadvantaged learners are not popular among European countries. In a report of
(European Commission et al., 2015) there are just a few examples of such strategies on a
national level: Austrian “Zentrale Beratgungsstelle für Basisbildung und Alphabetisierung”),
German telephone guidance services, Luxembourg’s “Maison de l’orientation” and
Portugal’s “Centros para a Qualificação e o Ensino Profissional”. In a majority of countries
where such strategies are absent it is advised to prepare (out of necessity in a smaller scale)
similar solutions by adult education providers.
(Windisch, 2015) finds appropriate information, advice and guidance as crucial factors
increasing persistence rates in adult education of disadvantaged groups. Many
disadvantaged students are unaware of potential benefits of education and therefore do not
take advantage of them. That often contributes to the decision of dropping out the
programme. American researchers suggest that the fraction of students actually benefiting
from the different forms of guidance is very slight and even smaller among the
disadvantaged adults. They therefore suggest implementation of Crockett’s idea of
“intrusive advising” (Mcdonnell & Soricone, 2014).
It is important to note that lack of persistence does not always prove that the programme
failed. Sometimes the changing circumstances in personal situation of a learner (such as
sicknesses or personal problems) force him/her to give up from the programme. In such
cases an educational provider should recognise such persons and provide them support in
retaining to the programme, when the situation gets back to the normal. Such support
should principally involve strengthening learner’s motivation and additional measures as
distance and blended learning that helps learners to minimize losses (Vorhaus et al., 2011).
Supporting learner persistence means assisting learning during and beyond formal
education. One interesting conclusion of Vorhaus et al. (2011, p. 14) is that
“breaking off from programmes is not always a programme failure but may be a
more 'rational and positive response to changing circumstances' (...)”.
Taking it into account, adult education providers must ensure that - when those external
circumstances change - a learner will not be discouraged from coming back into programme.

26
Thus, the guidance and assistance for learners must occur not solely during the programme,
but also beyond it.
Vorhaus et al. (2011) find that the key to learners’ persistence is a sensitive monitoring of
progress. The lack of (self-reported) outcomes of adult education is often a reason why
learners are discouraged from programmes. Such monitoring should also involve the “soft
outcomes, such as improvements in self-confidence”.
Litster (2007) summarizes the number of surveys and British experiences in strategies aiming
at improvement of learners’ persistence. She groups it into five groups: safe learning
environment, community in the classroom, individualization of learning, goal-oriented
education and supporting self-improvement.
The first group, a safe learning environment, is mainly addressed to teachers. It is crucial to
create the relationship between teacher and learner that makes the latter more confident
and willing to learn. She lists the examples such as:
- Finding common interests with learners
- Knowing their names
- Communication in a language of students (at least learning few words)
- Understanding the learners’ personal situation and their problems with motivation
and persistence
- Chatting with learners and giving informal advises
- Cooperation with other teachers and staff who also deal with a given student.
Community in a classroom focuses on creating the cooperation among students that help
them benefit from each other. She mentions following strategies:
- Controlling learners attendance and finding out the reason, with offering the help to
solve problems that limit their participation
- Team working, working in pairs
- Setting up a “buddy system”, in which one student is a support for the other
- Creating possibilities to make up for those who miss a class
- Engaging former students to talk about their motivation and successes
- Engaging students to talk about their “goals, motivations, hopes and fears”.
Third element, individual approach to the learner, is complementary to former two. It helps
a learner to understand that he/she can personally benefit from a programme and –
therefore – improve his/her motivation to actively participate. There are three proposals of
obtaining this goal:
- Establishing clear learning routes
- Finding links to learners’ personal experiences and redesigning a programme as to
make it relevant to their lives

27
- Creating activities that involve personal impact in the learning, such as writing the
diaries or personal histories.
Goal-oriented education is a strategy that is common for adult and regular education. In
case of the former the proposed strategies involves:
- Explaining to students what does the goal-setting real mean
- Finding out learners’ real goals
- Asking students about the factors that might affect their performance and –
therefore – obtaining their goals
- Revisiting goals on a regular basis
- Using formative assessment (see later).
One of the most important goals of adult education is to encourage learners to continue
their education and self-development also after the completion of one programme. The
strategies that support self-learning and self-development are:
- Encouraging learners to take responsibility for their education by working
independently, fulfilling homework etc.
- Solid assessment of homework
- Giving the comments and advises to the homework, both verbal and written.
According to Porter, Cuban, & Comings (2005), education providers are challenged to value
learning outcomes in a way that will allow to be sure of their quality and comparability for
each learning pathway.
Another factor recognised by Windisch (2015) is supporting self-study. It helps to achieve
additional goals as well as is one of the most efficient ways to support persistence: when a
learner is prepared to self-study, it is much easier to make him/her to survive the period
outside of education and henceforth come back to the programme.
Some strategies, especially those involving advisory, are common for all types of education
and experiences in another schools may be also useful for adult education for disadvantaged
groups. For instance, Bailey & Alfonso (2005) analysed the practices in (2-year) community
colleges in United States and found 4 strategies that were proven to have a positive impact
on the effectiveness of provision in those institutions. Those are:
1) Supporting learning with advisory, mentoring and counselling;
2) Creating learning communities that help learners to better benefit from teachers’
experience (it is also shown that learning communities are far more effective, when
the involved staff is employed full-time);
3) Creating special programmes or services for underprepared learners (this point is
crucial from a point of view of analysing the adult education for disadvantaged
groups);

28
4) Having a “college-wide” perspective on reforming and improving services: since the
institutional elements of education programme are highly complementary,
educational provider must not analyse learners' performances on a basis of one
single aspect.
Very similarly, Siebert (2003) proposed five recommendations to maximise persistence in
programmes:
- Counselling based on full, realistic and objective information
- Recognition of learners’ interests and goals in the beginning of the programme
- Ongoing assessment
- Recognition of the causes on why some students do not persist, based on interviews
with both those who dropped out and those who completed a course
- Enabling the flexible learning modes by modularisation of the programme.
Casey et al. (2006) found the embedded programmes to have a positive impact on learners’
persistence. They understand embedded as the programmes that integrate vocational and
basic skills provision. Particularly, most effective embedding programmes were
characterised by a strict collaboration between literacy, numeracy and vocational teachers.
In order to adjust an educational offer to the possibilities and needs of this group, it is also
important that the form of learning and thus developing new professional skills does not
refer to their past school failures. Regular feedback focused on made progress and
implemented in the form of coaching may be here of help. Furthermore, one has to keep in
mind that motivation is the key aspect not only before the start of learning, but it should be
the integral part of the whole process (Steiner et al., 2012).
Depending on the kind of disadvantaged people (in terms of specific sub-groups), one should
pay particular attention to the kind of conditions and support structures that should be
created and used in case of this group. For those with the care responsibility it is important
to provide childcare possibilities or to create family friendly places of learning. The most
important aspect in this case is flexibility in terms of time of learning. Learning is problematic
especially for women with the care responsibility. Financial issues should also be taken into
account, since high tuition could be and often is a major barrier to making decisions about
starting learning among members of this group. Moreover, it is strongly recommended to
create separate educational offers aimed at various sub-groups, such as migrants and older
people (ibidem).
4.4 Quality of provision

29
Quality of provision in the case of learning programmes for disadvantaged adults is also
important. There are five groups of most important factors that contribute to high quality of
educational provision: organisation of the course, curricula design, organisation of learning,
infrastructure and quality assurance.
The first group covers strictly organisational issues including: average length of a course,
service intensity (e.g. average number of learning hours per week/month, total number of
teaching hours, number of hours of instruction received by an average individual learner),
and number of learners per course. The last issue was raised by some of IMPADA focus
groups' participants who indicated the number of 8 to 12 participants - depending on
characteristics of learners - as the optimal size of an average adult disadvantaged learners'
class as 'accessible and attractive to such learners and providing enough support and time
for those in most need').
In the case of curricula design issues that are visible in the literature include: curricula
diversity, its flexibility (e.g. short optional courses as a part of a larger education
programme), adequacy and responsiveness to the actual labour market’s needs (Swain et al.,
2009).
Curricula design was also a topic that was raised directly during IMPADA focus groups.
Participants of FGIs, asked about factors that should be included in the AE for the
disadvantaged training programmes, indicated - among others:
- "improving learners' progression routes" (4/5 indications), including 'focusing on the
big picture of each learner's journey', 'individualized approach' and 'extending the
use of optional courses';
- "programming - flexibility: creating individual paths for learners" (3/5);
- "focus on the labour market needs and Increased Joint Practice Development
between practitioners" (3/5).
The third factor of quality provision is the organisation of learning - use of certain practices,
tools, settings, methods and approaches aimed at increasing learners’ engagement and
motivation to learn as well as adjusting educational process to diagnosed learners’ needs,
including:
- focus on identification of learners' needs;
- additional support for learners - if needed (e.g. childcare, financial support, adaptive
equipment);
- establishing partnerships and working closely with different stakeholders in order to
link formal and informal learning and learning in different environments as well as to
identify local needs in the field of adult education.
Factors connected with the organisational site of learning were often mentioned during
focus groups, especially :

30
- when discussing what could be improved in adult education programmes for
disadvantaged groups in order to improve its effectiveness (5/5 indications in case of
"identification of learners' needs", 3 - in cases of "additional support" and
"partnerships");
- when discussing factors that should be included in the AE for the disadvantaged
training programmes (3/5 indications in case of "using social pedagogy - supporting
the process of learning, e.g. 'searching for <knowledge distributors> also within
learners' social environment', 'working with members of families' etc.).
Learning infrastructure (including accommodation, resources, access to ICT, settings
adjusted to programme’s specificity) also contributes to quality of educational provision.
This factor was mentioned by the participants of almost all IMPADA focus groups (4/5
indications) in the context of naming characteristics crucial for training institutions that
provide AE for the disadvantaged. During the discussions on this topic the following factors
were mentioned: 'infrastructure adapted to the specifics of implemented actions and to the
needs of their participants (trainers, teachers and learners)', 'small, accessible venues in the
communities with the most disadvantaged learners', 'learning that is based where the
learners are (not necessitating travel to a campus)', 'small scale and cozy buildings, less
intimidating to disadvantaged than big institutions'.
Last but not least, quality assurance (internal QA systems implemented by training
providers, external QA – depending on existing regulations, respected adequate QA
procedures etc.) is crucial for the quality of outcomes.
Summarising, quality of provision depends on various factors, which can be shaped by a
training provider. Knowing this, we asked IMPADA focus groups’ participants about
characteristics that training institutions should have in order to provide effective adult
education for the disadvantaged. Apart from infrastructure, the most often received answers
to this question were:
- “good organisation qualities and characteristics” (5/5 indications), including ‘listening
to learner's voice’, 'taking actions to make improvements based on feedback’, ‘using
feedback mechanisms after end of programme and after ½ a year’, ‘using effective
Quality Assurance Procedures’, ‘being characterised by respect and responsibility
toward individuals’, ‘being characterised by innovation and reliability (ability to
demonstrate experience in effective management of educational projects)',
‘employing varied team (trainers and teachers) in terms of age, experiences and
skills’, ‘being characterised by flexible approach in learning activities (programmes)’,
‘offering continuing education courses for teachers’;
- “experienced, dedicated and looked-after staff” (5/5 indications);
- “successful training networks and partnerships”.
The following sub-chapters extend some of the abovementioned issues.

31
4.4.1 Length of a course
As mentioned in the previous chapter, the key to positive outcome of adult education are
attendance and completion of a course. One may ask, though, how long should be a course
in order to have an impact on learner’s competencies. (Alkema & Rean, 2014) cite numbers
of evidence to show that the answer to this question is unclear and hugely depends on the
way of defining the word “improvement”. In USA, e.g., there is a term “measurable gain”
which is understood as the improvement by a level (or gaining a GED certificate). In that case
the evidence shows that there is the need of 100 hours of teaching. Contrary, studies in New
Zealand and United Kingdom focused to measure “a significant gain” (understood as
statistically significant improvement in competencies) and showed that even shorter
courses, lasting 35-39 hours, may improve literacy and numeracy of learners.
Based on the analysed evidence, (Alkema & Rean, 2014) propose the useful distinction
between analyses focusing on the issue of programme’s length and divide them into two
categories: (1) courses that are focused on gaining new skills and (2) courses that focus on
“brushing-up” existing skills, which is particularly often the case regarding to literacy and
numeracy programmes. The latter appear to be shorter ones.
The report (European Commission et al., 2015) shows that there is a variety of approaches
among European countries in designing the length of courses. In majority cases “basic
education programmes” last between 100 and 300 learning hours or one year of learning. In
a number of countries such programmes are divided into smaller unites as in case of
Portugal where each “module” lasts 50 hours, or in Denmark where each “step” lasts
between 30-60 hours (see also Windisch, 2015). Similarly, Coben et al. (2007) found that to
achieve a progress of one level of qualifications framework, it is required to learn for 100-
150 hours, but an observable progress is achieved by those who participate just for an
average 39 hours.
To sum up, the length of a course is an important factor influencing the outcomes of adult
education. In order to optimize its impact, a provider has to take into account following
criteria:
- The needs of learners;
- Character of a course (gaining new skills or brushing-up existing ones);
- Persistence of students.
4.4.2 Organisation of learning
One of the most important characteristics of adult education of disadvantaged groups is that
learners often have bad experiences with formal class-based education and, therefore,
negative attitude to learning. Such attitude affects not only participation and persistence of

32
learners, but also their performance – discouraged learners are more passive during a
programme and, consequently, less likely to succeed. Windisch (2015, p. 58) puts it simply:
“the learning context matters for success”.
National Research and Development Centre for Adult Literacy and Numeracy prepared series
of reports analysing the effectiveness of particular types of basic skills education: writing
(Grief, Meyer, & Burgess, 2007), reading (Brooks, Burton, Cole, & Szczerbiński, 2007) and
numeracy (Coben et al., 2007).
According to the report on writing (Grief et al., 2007), the following strategies were proven
to be effective:
- Using texts of different kinds, e.g. applying writing short stories, letters or e-mails,
complaints or acknowledgments etc.;
- Using texts with meaningful contexts in that learner can find the links with real life,
both in substance (e.g. stories on his own autobiography or family) and form (e.g.
complaints on real products, as mentioned above);
- Discussions on writing, both with a teacher and with a group;
- Individual feedback of teacher – marking of writing should involve not only grades,
but also teacher's comments on what should be improved.
Report of (Benseman, Sutton, & Lander, 2005) on reading focuses on slightly different
approaches. According to them, the key to effective reading provision are skilled teachers
who apply a variety of teaching strategies. The case of quality of teachers is discussed in one
of the next chapters of this report. Second factor identified by Benseman, Sutton & Lander
(2005) is – similarly to the case of writing – creating curricula linked to life. Provision is much
more effective when the examples of texts that are read during the course are somewhat
familiar for students, e.g. have reference to their professional experiences or living
environments. Finally, the strategies that is proven to be very effective is implementing the
reciprocal reading, involving the cooperation among students.
Additional factor identified by (Vorhaus et al., 2011) is the assistance for a teacher. As was
stressed above, the effective provision of adult education must combine two elements: good
subject knowledge and pedagogical preparation. Since subject teachers often lack the
competencies adjusted to the needs of disadvantaged adults, a teaching assistant may serve
as an important factor of improving adult education effectiveness.
(Ofsted, 2007) points on the desirable character of such assistance. Competencies of a
subject teacher and a support tutor should be highly complementary, and their work is most
effective when they tightly cooperate, including the preparation of a programme and the
design of curriculum. Situation in which a teaching assistant limits his/her engagement to in-
class activities is, however, an often situation, and lowers overall programme effectiveness.

33
(Brooks et al., 2007) see the time spent in a classroom as a key to effective literacy provision.
Their findings suggest that self-study may be an appropriate strategy to increase persistence
(see chapter 4.3), but still it is less effective in improving literacy than actual working with a
teacher or trainer. They therefore formulate 4 recommendations for adult education
providers that seem to be important in effective provision:
- Programmes should be more intensive, so that shorter courses involve more hours of
attendance;
- Providers should aim in decreasing learner:staff ratio;
- Time of working alone appears to be too long in many cases;
- Most practical ways to decrease the time of working alone is to enhance working in
groups or pairs (e.g. buddy system).
Third crucial type of basic skills education are the numeracy programmes. (Coben et al.,
2007) examined their provision in UK to conclude that the most important factor is
flexibility. That should involve: the possibilty to adapt a programme to diverse group of
learners and changing during the course expectations on final outcomes. However, they
noticed that among British providers such diversity was mediocre: they identified very
limited use of practial tools and ICT. The organisation of learning was dominated by whole-
class approach to learning and limited use of group or collaborative work.
4.4.3 Teachers quality
The issue that is analysed in virtually all the publications on effectiveness of adult education
is the quality of a teachers and trainers. There are two approaches to this case: one focuses
on the (formal) qualifications of teachers, the second on soft skills.
The key conclusion of (Vorhaus et al., 2011) is that qualifications matter. There is an
enormous evidence suggesting, that – contrary to some popular beliefs – provision is more
effective when staff is trained and has pedagogical background.
(Benseman et al., 2005) stress that teaching is far more effective when staff is employed in
full-time. It is an important conclusion, since the majority of staff of European AE training
providers appears to be part-time employees or voluntary. (Ofsted, 2011) points that the
most successful providers are those with at least 1 senior member of staff with an excellent
knowledge of literacy and its pedagogy.
Similarly, (Ofsted, 2011) in his report on literacy also underlines the role of teacher.
According to his findings, the most effective providers were hiring at least one member of
staff “with an excellent knowledge of literacy and its pedagogy”.
There is one exception from the following rule: overqualified teacher. In some cases
teachers whose qualifications are above average appear unable to communicate with
learners, which affects the effectiveness of provision and learners’ motivation. However, one

34
may argue that such situation is an effect of lack of soft skills that help to overcome the
barriers in teacher-learner relationship.
Given the above, the second characteristic of effective adult education teachers and trainers
are soft skills. While dealing with disadvantaged adults, who often lack of motivation to learn
(as shown in chapter Error! Reference source not found.) and have bad experiences from
formal education, adult education teachers have to implement the special approach to
teaching.
(Ofsted, 2011) lists the following strategies that appear to be effective:
- teachers having high expectations about learners’ achievements;
- adapting curricula to meet the needs of particular group of disadvantaged adult
learners;
- intervention to a curricula or a programme, when learners’ expectations change or
previously assumed strategies appeared to be ineffective;
- understanding barriers of disadvantaged learners, that are often different than those
of “regular” learners.
One conclusion is particularly important for providers of adult education for disadvantaged
groups: some teachers tend to limit their expectations about their students and measure
their progress by a comparison with a whole group, rather than with national average, which
is presumably higher. Such strategy is found to be ineffective and makes the gap between an
average and disadvantaged student to persist (Ofsted, 2011).
(MacLeod & Straw, 2010) underline the need of providing strong teacher-learner
relationships. Informal contacts between teacher and learner help the latter to improve
their provision and adapt the solutions described above.
Report of American Institutes for Research within the project “Promoting Teacher
Effectiveness” proposed the useful framework to assess and identify teachers’ skills and
competencies. The framework is based on 4 aspects: domains, competencies, performance
indicators and sample illustrations. Table 2 presents their approach.
Table 2 AdultEducation TeachersCompetencies framework
Domains There are four domains that represent broad areas of activity
for an adult education teacher:
1. Monitors and manages student learning and performance
through data 2. Plans and delivers high-quality, evidence-based
instruction 3. Effectively communicates to motivate and engage
learners 4. Pursues professionalism and continually builds
knowledge and skills.

35
Competencies Within those 4 domains of activity 17 individual observable
competencies represent the knowledge, skills, and abilities that
an adult education instructor should possess to be effective
within that domains. Each domain has four to five
competencies.
Performance Indicators Each competency has a set of indicators that articulate what the
performance of this competency looks like in an adult education
context.
Sample Illustrations Each performance indicator is accompanied by a sample
illustration that provides examples of the practices in different
adult education settings (such as a multilevel English as a second
language classroom, a basic literacy class for native).
Source: (American Institutes for Research, 2015)
(Yin, Condelli, Ogut, & Cronen, 2013) analysed the variety of aspects of adult education
teachers’ performance. Some of them appeared to be less significant than it was assumed at
the beginning of the survey, whereas others proved to be crucial factors of effective adult
education provision. The following aspects were examined:
- Demography of teachers proved to have no impact on students’ performance. There
were some suggestions that some demographical features may be important (e.g.
gender or age of teachers), but there were no statistically significant results, or
results found in one states were not proven in another;
- Teachers' professional development programmes hugely vary among different states,
so it is impossible to give one simple answer on their effectiveness. However, there is
a small, but significantly positive impact of the crude fact that one has participated in
such programmes;
- Teacher's experience proved to be one of the most important factors. It has
significantly positive impact on effectiveness in all analysed aspects: overall, language
and mathematics post-tests.
4.4.4 Other factors
IMPADA focus groups' participants – asked about what kind of knowledge, skills,
competences, qualifications and support do trainers for disadvantaged adult learners need
in order to provide effective education - gave answers such as:

36
- "empathy and understanding" (5/5 indications), including 'building relationships
with learners', 'awareness of learners diversity', 'high levels of tolerance and respect',
'communication and listening skills', 'stimulating adult learners to have faith in their
ability to learn and change', 'creating relationships of trust with learners, as a pre-
condition to the effectiveness of the learning process', 'reducing distance between
trainer/teacher and projects' participants by using the appropriate appearance and
behaviour', 'personal skills such as psychological resistance, inner-direction, sense of
mission, ability to separate work from personal life, maturity, stability, high level of
energy, mindfulness etc.';
- "teaching skills" (5/5), including 'ability to teach through active methodologies and
methods adapted to adult learners', 'knowledge of different learning styles and
learners' differences', 'knowledge of motivation factors', 'ability to understand and
influence the group process', 'ability to create training situations conducive for the
group engagement', 'ability to create alternative learning situations', 'coaching skills –
ability to ask the right questions';
- "equality, diversity and inclusion skills" (3/5), including 'experience of working with
people from a range of disadvantaged groups, e.g. autism, schizophrenia, bipolar,
depression, anxiety etc.', 'compulsory training on Prevent, CSE, safeguarding';
- "organisational, fundraising and community skills" (3/5);
- "support from colleagues and employers" (3/5), including 'work shadowing and
sharing other good practices', 'coaching support, counselling, weekly meetings to
discuss current personal and training issues', 'encouraging to generate demand for
and interest in initiatives to nurture staff creativity', 'providing to novice
trainers/teachers many opportunities to participate in internships and perform real
professional tasks with the support of experienced professionals'.
(Condelli et al., 2010) proposed the following strategies to be effective:
- Contextualization of basic skills instructions and strategies;
- Blended learning;
- Flexible teaching content that allows to be rearranged due to changing
circumstances.
5 Methods of assessment
Evaluating the effectiveness of basic skills programmes is challenging and needs to be
conducted over an extended period of time, taking into consideration that learners facing
difficulties with basic skills do not necessarily follow a direct or uninterrupted learning path
(European Commission/EACEA/Eurydice, 2015a).

37
Measurement the effectiveness of education for disadvantaged is more difficult and
nuanced than among average students. It is a very important note: if a disadvantaged person
doesn't find a job after adult education, it does not always mean that the course was
ineffective. Employability is just one of the aspects, but it may serve as a good illustration.
Therefore, it is advised to measure "net" effect rather than "gross" – relative improvement
since joining a programme. For instance, in Polish formal education system there is being
implemented a method called “Educational value added” to measure, if the absolute
improvement of a student is a merit of school or some external factors (such as innate
capabilities of a student). In adult education there are no such tools, so it may be found as a
niche to fill.
Education providers should also focus on measuring the effectiveness of awareness-raising
and outreach activities, which is found to be particularly difficult (Windisch, 2015).
As noticed in (OECD, 2013b), the field of educational evaluation is constantly growing in all
levels of education and both in formal and non-formal education. This is even more
important, since there is a huge demand for more effective and equitable education.
Furthermore,
“Although educational evaluation within school systems is not a recent concern, it
has traditionally focussed mostly on the assessment of students. In recent years,
countries are increasingly developing more comprehensive evaluation and
assessment frameworks.” (p. 18).
Apart from the disputes on what does the effectiveness of education mean (as well as the
quality or equity), the biggest problem is how to measure it. In general, there are two main
sources for assessing the direct outcomes of education:
- Self-reported returns;
- Objective measures.
Both have major limitations. Self-reported returns are biased by subjective feeling, especially
among disadvantaged learners who usually have bad experiences from formal education
(what can lead both to under or overestimation of real effects). On the other hand, objective
measures often require costly monitoring methods, which may be not efficient way to assess
the impact. Furthermore, they often hide some indirect effects.
In case of adult education for disadvantaged groups, it is also important to note that lack of
measurable economic effect does not always mean that the course was ineffective. For
example, a disadvantaged person may have problems with finding a job after graduation
despite the fact that his/her competencies and relative labour market value have been
improved.

38
Table 3 Assessment methods of learning outcomes
Learning outcome Level of understanding Assessment task
Basic facts, definition of
terms
(example: state the cost of
materials needed to make a
brick wall)
Recall and recognition, some
relational and
multistructural
Multiple choice, close, short
answer questions
Distinguish between
different
concepts, relate facts
(relational and multi-
structural)
Differentiate between fair
and unfair dismissal
Assignment, presentation,
comprehension test, case
study
Demonstration of skill
(search a database by fields)
Procedural knowledge Presentation, simulation
exercise, audio/videotaping
Converse fluently in foreign
language
(conduct a one-to-one
conversation about the
Weather)
Relational, functional
knowledge
Role play, audio/videotaping
Develop an argument and
defend own views (specify
how humour is used to
portray stereotypes in
‘Jeeves’)
Evaluation, relational and
extended abstract
Discussion, long essay,
seminar presentation
Reflect on own learning
(monitor own progress)
Extended abstract,
relational
Log/diary
Source: (Hillier, 2005)
The report (Hanover Research, 2015) on assessment of adult education programmes in USA
presented some of the most common measures. All the states follow the National Reporting
System framework, which focuses on five core indicators: skill level gains, educational
attainment (secondary or post-secondary), employment outcomes (unsubsidised
employment and retention in employment).
Table 4 Types of assessment
Formative Summative
Informal Where are we so far? What What do we learn?

39
do we want to do next?
Example: a fitness class for
people with arthritis
Example: an introduction to
art history class
Formal How much assessment
specification/learning
outcomes have we
achieved?
Example: an NVQ portfolio
development workshop
Did we achieve all the
learning outcomes? What
grade/level have he
attained?
Example: AVCE/A level
classes
Source: (Hillier, 2005)
(Looney, 2011) proposes the distinction on two types of assessment: “formative” and
“summative”. Summative assessment is more regular and common one: it involves the
assessment of learner’s performance based on her achievements, based on some tests or
examinations taking place by the end of the programme. Summative assessment is also a
part of a validation process, since it is a way to prove, that the student obtained the learning
outcomes required for some qualification (certificate or diploma) and, therefore, it may be
used to get access to higher levels of education or some specific occupation.
On the other hand, formative assessment is also based on student’s performance, but it has
more ongoing character and another goal: the improvement of learning process. Thus, the
formative assessment may serve as a way to identify learning needs and to adjust a teaching
processes.
Looney sums up the difference, noticing that, “summative assessment is sometimes referred
to as assessment of learning, and formative assessment, as assessment for learning”.
5.1 Formative assessment
Derrick & Ecclestone (2008) analysed the issue of formative assessment in detail. As was
mentioned in previous chapter, formative assessment is believed to be a most effective
strategy to improve learners’ persistence. It also has a positive impact on outcomes of adult
learning, since it helps to adjust the curricula to the changing motives of learners and their
progress in learning.
They identified two aspects of adult learning:
“The first of these consists of practices and activities in which the purpose is to
produce evidence for the planning of future learning and/or for constructive feedback
and review: these might include activities such as assignments, tests, role-play,

40
performances, observations, questioning, etc. It is important to note that these
activities can also be used primarily for purposes that are not essentially formative or
to support learning, for example to fulfil bureaucratic requirements or to serve the
purposes of summative assessment. Secondly, formative assessment can take the
form of learning activities which aim to develop the autonomy of the learner(s) by
giving them practice in the use of relevant skills and knowledge in real-life contexts,
and particularly in the assessment and evaluation of their performance and those of
others, in those contexts.” (p. 71).
They also identified the following aspects of formative assessment that are crucial for an
effective provision:
- Formative assessment cannot be used instead of summative assessment. The latter
still exists as an essential strategy in adult education. However, its tools shall be
“used formatively”. (Derrick & Ecclestone, 2008, p. 73) puts it:
- “this involves finding ways to get students to <<get beneath>> and <<go beyond>>
the bald results of the summative assessment processes and try to understand how
they work and reflect on what they mean.”
- Formative assessment may be used in planning activities, as to adjust the plans to
actual needs.
- There should be an on-going evaluation of teachers’ communication skills,
particularly those involving listening, empathy and understanding;
- Feedback should be constructive and practical. It also should not be taken by a
learner personally, so it has to address the task;
- The assessment should aim at “improving motivation and confidence, autonomy, and
citizenship”;
- Good cooperation between teachers should lead to an exchange of questioning
techniques. The authors state that open questions are more effective than those as:
double questions, leading questions, rethorical questions, closed questions.
- The assessment during all the stage of adult education programmes should involve
peer assessment and self-assessment;
- The assessment techniques are often not understood by learners. One of the most
popular reasons for that fact is that adult learners from disadvantaged groups use
different language than teacher. In practice formative assessment should involve:
“Encouraging learners to develop, discuss and evaluate their own assessment criteria
and assessment materials, as well as collectively designing „perfect‟ answers, will at
the same time help them understand and critique the language of official assessment
criteria. “ (Derrick & Ecclestone, 2008, p. 75)
- There is sometimes a tension between “letter” and “spirit” of formative assessment.
Formative assessment is often understood in very narrow way, that is limited to
defining the learning goals. Spirit of formative assessment means though, that it is “a
way of engaging learners deeply with their learning in order to develop critical and
cognitive autonomy.” (ibidem)

41
- Formative assessment is useful in all types of adult education.

42
6 Criteria on measuring adult education effectiveness
Evaluating the effectiveness of education is critical to education programmes accountability
and, of course, it applies to adult education (including education for disadvantaged groups)
in the same way as to general and higher education. Basing on intuition and common sense,
we can tell about the strong positive influence that literacy programmes have on adults’
employment, family relations or other areas of life – referring to the fact that such influence
is indicated in the case of literacy proficiency itself. At the same time, a clear indication of
reliable ways to measure the actual results of education (specific education programmes),
based on properly selected criteria and adequate methods of measurement, remains among
common research challenges, especially when we talk about the long-term outcomes and
focus on the specific group of disadvantaged adults. It is due to the lack of surveys based on
scientific approach (e.g. with the use of comparison groups and conducted within a long
period of time) as well as to mutually exclusive results obtained in the completed studies
focused on this issue (Moore & Silva, 1999).
In this chapter we present the typology of the criteria on measuring the effectiveness of
adult education for disadvantaged groups, based – on one hand – on our research on the
specific characteristics and needs of this group, and on the other hand – on findings
regarding various outcomes of (adult) education, factors influencing the effectiveness of
such education, as well as methods of its assessment.
6.1 Criteria oriented directly on the expected outcomes (learner’sapproach)
The first broad group of criteria focuses on direct, short- and long-term outcomes of
learning, while the second one is based on inferencing about the effectiveness of learning
process (education programme) from indicators characterising AE providers (and depending
on their educational activities and achievements).
6.1.1 Objective measures (improving in/acquisition of new skills)
In case of this group the word “objective” refers to the fact that the process of measuring
the effectiveness of education based on these criteria requires using assessment methods,
which are aimed at the observation of certain objective indicators, and not obtaining
subjective opinions. Based on our desk research, we distinguished two types of criteria that
belong to this group, from which those that we assigned to the first type can be used to
measure the effects of learning observable in a short term, while the criteria assigned to the
second subgroup are useful in the assessment of long-term results of (adult) education.

43
Measuring during validation process (formative and summative assessment)
This group of criteria refers to measuring the improvement in/acquisition of new selected
skills by learners, whose performances are compared to those that were observable at the
time before starting a learning process. This approach focuses on the outcomes that are
“visible” shortly after completing a certain education programme, and thus can be assessed
during the validation process (which can be directed towards a certification of specific skills).
As it was highlighted in the previous chapters, in case of the adults from disadvantaged
groups skills that should be learned (and thus taught) in the first place are:
i) Literacy and numeracy;
ii) Digital skills;
iii) Soft skills, which can be grouped as in Table 5.
Table 5 Different types of soft skills and examples of evaluation criteria
Skills Learning outcomes (assessment criteria)
awareness, self-care is able to be aware of personal attitudes, skills, vocations
and deep values
is capable of developing self-esteemand self confidence
is capable of dealing with own emotions, physical sphere
and interior life
acting autonomously is able to act within the big picture, developing critical
thinking
is able to form and conduct life plans and personal projects
(to demonstrate self-direction, self-discipline etc.)
is able to assert own rights, interests, limits and needs
is capable of learning
is able to adapt and take initiative
using tools interactively and
in a creative way
is able to use language, symbols and text interactively
is able to use knowledge and information interactively
is able to use technology interactively
is able to create arts and enjoy creativity
is capable of creating (handcrafts, cooking etc.)
interacting with others, also
in heterogeneous groups
is able to build relations with others
is able to interact positively with people from various
backgrounds
is capable of taking care of others
social and civic skills is able to manage and resolve conflicts and create
coexistence
is able to demonstrate global awareness, multicultural
literacy, humanitarianism

44
Skills Learning outcomes (assessment criteria)
is able to cooperate with others and contribute to the
community
is able to demonstrate civic, ethical, and social-justice
literacy
Source: UPTER’s own documentation
Measuring of long-term outcomes
The second group of objective measures oriented directly on the expected outcomes of
adult education for disadvantaged groups are measures possible to obtain by comparing
group’s of learners (social, vocational, educational, financial) position/situation before
starting a learning process (a certain educational activity) and after some time from
completing it. These criteria can relate to areas such as:
- Social wellbeing/social exclusion, measured by various indicators and scales (e.g.
learners’ better existence in local communities, better participation in civic society –
e.g. by voting – better access to basic needs services such as good quality water and
food, appropriate shelter, public transport, social and health services, greater
participation in culture etc.); (Moore & Silva, 1999) notice that there aren’t a lot of
credible research studies regarding the measurement of these outcomes;
- Family literacy, mentioned by Vorhaus et al. (2011) as one of the outcomes linked
with psychical wellbeing of a person (see also p. 16 of this report);
- Employability (e.g. percentage of graduates who were employed for at least 6
months by the end of the first year after graduation), which – as was stressed in the
previous chapters – is one of the most often quoted outcomes, even in the case of
basic skills programmes, which – as a rule – are not directly aimed at improving
learners’ situation on the labour market;
- Level of professional/vocational sufficiency (e.g. increased sufficiency measured by
performance indicators), which – in case of employed learners – contributes to
productivity at the company level; when it comes to this kind of outcomes (Moore &
Silva, 1999) stress that
“(…) companies often lack the necessary data, or will not share these data beyond
corporate management, to empirically establish the programs’ effect on indicators
such as reductions in safety violations, reduced employee turnover, or increased
employee attendance.” (p. 10),
so this issues require further investigation;
- Participation in LLL – continuous/further education (e.g. percentage of graduates
who successfully participated in educational activities within the first 3 years after

Impada o1 report_final ibe

Recommended

Recommended

More Related Content

Viewers also liked

Viewers also liked (11)

Similar to Impada o1 report_final ibe

Similar to Impada o1 report_final ibe (20)

More from Andrea Ciantar

More from Andrea Ciantar (20)

Recently uploaded

Recently uploaded (20)

Impada o1 report_final ibe