thesis in English

This research is welcome for any clarity /supplementation>> copyright 2013
By:Laguna State Polytechnic University-San Pablo City Campus
With Marilyn Atienza Mendoza & Carie Justine P. Estrellado
EFFECTIVENESS OF CLOZE – TEST AS INTEGRATIVE
ASSESSMENT
IN READING COMPREHENSION AT
LAGUNA STATE POLYTECHNIC UNIVERSITY
SAN PABLO CITY CAMPUS
ACADEMIC YEAR 2012-2013

For more details call the authors: 09103240206 &
09297447542

A Research Paper
Presented to the
Faculty of the College of Teacher Education
Laguna State Polytechnic University
San Pablo City Campus
San Pablo City

In Partial of Fulfillment
Of the Requirements in
Methods of Research for the Degree
Bachelor of Secondary Education
Major in English

CARIE JUSTINE P. ESTRELLADO
&
MARILYN ATIENZA MENDOZA

No part of this research may be altered or modified without the written permission from
the publisher. TechAd Group of Publications, 4000 Laguna, PH.

October 2012

Republic of the Philippines
LAGUNA STATE POLYTECHNIC UNIVERSITY
San Pablo City Campus
San Pablo City

COLLEGE OF TEACHER EDUCATION

Approval Sheet
The Thesis entitled, ―EFFECTIVENESS OF CLOZE TEST AS INTEGRATIVE
ASSESSMENT IN READING COMPREHENSION AT LAGUNA STATE
POLYTECHNIC UNIVERSITY ACADEMIC YEAR 2012 - 2013‖ is prepared and
submitted by Marilyn A. Mendoza and CJPE in partial of fulfillment of the requirements
for the degree of Bachelor of Secondary Education, Major in English, has been examined
and is recommended for acceptance and approval for oral examination.

NELIA T. SALVADOR, Ed.D.
Adviser

PANEL OF EXAMINERS

Approved by the Committee on Oral Examination with a rating of __1.77__.

NELIA T. SALVADOR, Ed.D.
Chairman

ARNOLD ABAD C. TENORIO, M.D. ANGELA L. REGINALDO
Subject Specialist Statistician

RUBY B. BRION, Ph.D. NERISSA Y. BILO, M.A.T.
Technical Editor English Critic

ELSA C. CALLO, Ed.D.
Dean, College of Teacher Education



Date:___________
Research Contribution no.__________

Acknowledgment

The researcher would like to extend her deepest appreciation to those who have

helped and supported her throughout the conduct of the study:

Daniel, her son who gave her the inspiration to continue despite life‘s difficulty;

Prof. Nelia T. Salvador, Ed.D., her adviser for the guidance, fondness and

concern, without her this study would not be possible;

Ms. Angela Reginaldo, Ms. Ruby Brion, Ms. Nerissa Bilo and Mr. Arnold Abad

Tenorio who gave her support in finishing this research;

To all the respondents who were very cooperative in giving information needed in

the conduct of this research;

Carrie Justine ‗ISPOY‘ Estrellado for his effort and unsolicited ideas that he

shared;

Friends, for providing support and concern in the best way they can; and

Almighty God who gave her strength and knowledge to be able to accomplish this

requirement.



Dedication

To her:
Parents;
Sister, Myla;
Husband, Dennis &
Affectionate son, Daniel

Thank you for the Love,
Memories,
Time
and
Values
- M.A. Mendoza

ABSTRACT



This study aims to know the effectiveness of cloze – test as integrative

assessment in reading comprehension at Laguna State Polytechnic University, San Pablo

City Campus, Academic Year, 2012-2013.

1. What is the profile of the respondents in terms of:

1.1 Age;

1.2 Gender; and

1.3 Grade point in english subject?

2. What is the level of perception of the following assessment – related variables as

perceive by the respondents:

2.1 Controllability;

2.2 Reliability and validity; and

2.3 Usability?

3. What is the mean score of the respondents in Cloze – test?

4. Is there a significant relationship between the assessment related factors and the

mean score in Cloze – test?

5. Is the grade point average in English is significantly related to the result of Cloze

– test mean score?

This study used the descriptive type of research in determining the effectiveness

of Cloze Test in measuring reading comprehension as a subject in secondary level. The

possible solutions obtained through a questionnaire as well as interviews and observation


made by the researcher. This study used of the third year students in Laguna State

Polytechnic University as its respondents, the main source of data. The instrument used

was a collaboration of information based on Crystal, David (2004). Contextual

Constraints in Cloze Test. Cambridge: Cambridge University Press. ISBN 0-521-53032-6.

Some of its modifications were undertaken to adapt on the research problem and scope of

the study.

Findings

Majority of the respondents are in age 15y/o with a frequency and percentage of

twenty (20) or 69%; seventeen (17) or 56.62% are female students; ten (10) or 34% of the

grade point average of 3rd year students lie between 87 – 89.

All of the assessment – related variables such as Controllability (mean = 4.32),

Reliability & Validity (mean = 4.53), and Usability (mean = 4.32) and interpreted as

‗Strongly Agree‘.

The Cloze test result has a frequency of two – hundred twenty one (221) and has a

mean of 0.28 which is overall interpreted as ‗Fair‘.

The assessment related variables such as Controllability, Reliability and Validity

and Usability are all ‗Significant‘ to result of Cloze test.

Grade Point Average of the 3rd year students are ‗Not significant‘ to the result of

Cloze test having a p value of 0.0676.



Conclusion

Based on the findings posited in the study the following are hereby concluded:

1. The hypothesis that there is no significant relationship between the assessment

related factors and their mean score in Cloze – test is ‗Rejected‘.

2. The hypothesis that there is no significant relationship between grade point

average in English and their result of Cloze – test is ‗Accepted‘.

Recommendations

Based on the findings and conclusions made in the study, the following

recommendations are hereby presented:

1. Institutions may provide proper and concrete assessment procedure since the

manner of determining the complex ability of a child starts from the institution

strongly claimed by Greene (2001) may endow materials such as Audio media

since the respondents has low frequency rate in cloze test and specially provide

good environmental atmosphere for learners.

2. For the concern of students‘ improvement, teachers may suit the interest

particularly those specializing in Communication arts so that they will encourage

students to study more.

3. Since the assessment – related variables are significant to the result of cloze test it

is associated that parents are vital provocative when it comes to their children in

assessing academics according to colloquium experts (International Cognitive


studies Association, 2001) have been mentioned as one of the recommendations,

to wit:

―Parents are direct response indicator for their young and

influentially provide such learners‘ needs to gave interest; help

them (children) to develop journal or diary and reading factual

references in English; schedule periodic parent – teacher meetings

so that parents can discern their progress; enforce rules frequently

to limit them to play and provide a pleasant climate so that when

they go to school predicaments is partially reduced in view of the

fact also in assessment in academics‖.

4. Researchers must supplement more various factors that will help contribute to the

proper assessment in English subject.

5. Finally, there is a need for more researches and case studies about effectiveness of

cloze test so that implications can be drawn and adopted on a wider basis for the

learners‘ needs.



TABLE OF CONTENTS

Page

TITLE PAGE ................................................................................................................i

APPROVAL SHEET ....................................................................................................ii

ACKNOWLEDGMENT...............................................................................................iii

DEDICATION ..............................................................................................................iv

ABSTRACT ..................................................................................................................v

TABLE OF CONTENTS ..............................................................................................ix

LIST OF TABLES ........................................................................................................xi

LIST OF FIGURE.........................................................................................................xii

Chapter

I. THE PROBLEM AND ITS BACKGROUND

Introduction ...........................................................................................1

Background of the Study ......................................................................3

Statement of the Problem ......................................................................6

Significance of the Study ......................................................................7

Scope and Limitation of the Study........................................................7

Theoretical Framework .........................................................................8

Conceptual Framework .........................................................................10

Hypotheses ............................................................................................11

Definition of Terms...............................................................................12

II. REVIEW OF RELATED LITERATURE AND STUDIES

Related Literature..................................................................................13

Related Studies......................................................................................19

III. RESEARCH METHODOLOGY

Research Design....................................................................................24

Respondents of the study ......................................................................24

Research Instrument..............................................................................25


Statistical Treatment of Data ................................................................25

IV . PRESENTATION, INTERPRETATION

AND ANALYSIS OF DATA ...........................................................28 – 38

V. SUMMARY OF FINDINGS, CONCLUSION AND

RECOMMENDATIONS

Summary of Findings ............................................................................39

Conclusion ............................................................................................42

Recommendations .................................................................................43

BIBLIOGRAPHY .........................................................................................................44

APPENDICES ..............................................................................................................46

CURRICULUM VITAE ...............................................................................................56

List of Tables

Table Title Page

1. Distribution of Respondents According to Age ...................................................28

2. Distribution of Respondents According to Gender ..............................................29


3. Distribution of Respondents According to Grade Point Average ........................29

4. Respondents‘ Perception on Controllability

As a Factor of Integrative Assessment .................................................................30

5. Respondents‘ Perception on Reliability and Validity

As a Factor of Integrative Assessment.................................................................31

6. Respondents‘ Perception on Usability

As a Factor of Integrative Assessment.................................................................32

7. Distribution of Results in Cloze Test ...................................................................34

8. Correlation between Assessment – related Variables

And Cloze Test .....................................................................................................36

9. Correlation between Grade Point Average in English

And Cloze Test ....................................................................................................37

List of Figure

Figure Title Page

1. Research Paradigm ........................................................................................11



Chapter I

The Problem and Its Background

Introduction


The cloze procedure is considered by many as an 'integrative' method of

assessment, in contrast with 'discrete point' methods, The Cloze procedure deals with

several linguistic components at once, focusing more on language use, and typically

requiring the examinees to read and comprehend a substantial amount of discourse. Much

research has been devoted to the validity and the reliability of Cloze tests. Cloze tests are

reported to have moderate to high correlations with standardized tests and their subtests

such as listening comprehension and reading comprehension. The report on high

correlations between Cloze tests and International Journal of Human and Social Sciences

tests of listening comprehension, writing, reading comprehension, and oral interview. The

findings of these researchers reveal a relation between the scores on Cloze and global

language ability tests.

As for C-test, since its inception in 1981, its principles have been applied to more

than 20 languages. C-tests have been used in numerous contexts and for various purposes

.C-tests, then, figure prominently in the fields of language testing and assessment. C-

tests, like the classic Cloze test, are an operationalization principle of reduced test.

Although there are weaknesses in the reliability of cloze procedure, it is widely used in

English teaching. As Legenza and Elijah (2001) point out, many studies have been

conducted on cloze procedure concerning the construction of cloze tests, the relationship

between cloze and reading comprehension, and the usefulness of cloze as a teaching

technique. Cloze procedure has been used in English teaching in accordance to

communication subject for decades since 1953. It was firstly used as an instrument for

assessing the readability of written materials for school children in the United States

(Brown, 2002). Then it was used in teaching for different purposes.

Cloze procedure is not only valuable in providing information as to how readable

the particular texts are for the students who are going to have to use them, but also

invaluable as a means of comparing different text. Cloze procedure is also used for

testing purposes. It seems that a wide range of skills like vocabulary, grammar, structure,

and reading skills are involved in the process of completing a cloze procedure. Many

researchers indicate that cloze procedure is a good test of overall English language

proficiency and in History. For example, Ahluwalia (2005) claims that cloze procedure is

an integrative, global measure of language competence. She explains that cloze tests

measure the grammar of expectancy underlying the skills of thinking, understanding,

speaking, reading, understanding and writing. For Cohen (2001 cited in Ahluwalia, 2005)

cloze procedure measures global language competence consisting of linguistic

knowledge, textual knowledge, and knowledge of the world. As it calls on examinee to

use knowledge such as vocabulary, grammar, sentence construction, text structure,

cohesion and the reader‘s prior background knowledge.

Askes (2002) regards cloze procedure as one of the integrative tests (global tests)

that integrate language components into a total language event, which requires an

integrated performance from the learner in a meaningful context. Thus for Steinman

(2002), a cloze can replace the sections on test structure, written expression, vocabulary,

and reading comprehension not only in English subject as well as different areas of

learning.

The reason behind giving a Cloze test and obtaining a test score is interpreting

that score as an indicator of what a test taker knows or what he/she can do with that


knowledge. Furthermore, our interpretation of that test score forms the basis for decision

making. As such, when using a test score, we make an implicit link between test

performance and a domain of language knowledge the test taker has or something the test

taker can do with language in some language use domain beyond the test itself. In other

words, when we use test scores, we are essentially reasoning from evidence, using the

test score as the evidence for inferences or interpretations and decisions we want to make.

Yet, we cannot simply draw on test score to make inferences and decisions without

efficient justification. If we want to use a test score for a particular purpose, we must

justify it through a rationale and supporting evidence. As Bachman (2005) puts it, "We

need to demonstrate, with logical argumentation and empirical evidence, that the

intended interpretations and uses are valid." Validity in testing and assessment has

traditionally been understood to mean "discovering whether a test measures accurately

what it is intended to measure", or uncovering the "appropriateness of a given test or any

of its component parts as a measure of what it is purposed to measure".

Background of the Study

Cheng (1999) claims that the term ‗cloze procedure‘ was first developed by

Wilson Taylor in 1953. He explains that the term cloze derived from the Gestalt

psychology concept of ‗closure‘. It describes a tendency that humans have to complete a

familiar but not-quite-finished pattern. Ellis (2004) further explain that it refers to the

tendency of individuals to complete a pattern once they have grasped its overall

significance. Rye (1982) explains that Cloze procedure is essentially a cognitive task. The

reader has to reason and construct suggestions to fill the gap on the basis of the evidence


derived from the context the completion of meaning, based on understanding and

reasoning is a cognitive task. Brown (2002) claims that it is not difficult to get people to

take a cloze test because of the compulsive human need to fill gaps.

Clarke (2001) was the first to study cloze procedure for its effectiveness as an

instrument for determining the readability of materials in the reader‘s native language in

1953. After that initial study, it was investigated for its appropriateness as a measure of

readability of L1 and L2 materials. In the 1960s, studies focused on cloze tests as a

measurement of reading comprehension in L1 and L2. During the 1970s, cloze tests

began to be used as a measurement of overall L2 proficiency (Ahluwalia, 1992:82).

Today, cloze tests are widely used in some places (such as China) and as part of some

large-scale language. What is a cloze test? A standard cloze test is a passage with blanks

of standard length replacing certain deleted words which students are required to

complete by filling in the correct words or their equivalents. During traditional cloze

testing, every fifth word is removed from a 250-500 word reading passage, and is

replaced by a standard-length blank space (Helfeldt et al, 1986:216). Usually, no word is

omitted either in the first or the last sentence of the passage. Students are required to

supply either the original word of the author or an appropriate equivalent in the blank

space. Many studies show that the reliability and the validity of cloze tests are affected by

factors like the rate of deletion, nature of the text and scoring systems, etc. ‗Deletion rate‘

refers to the frequency of deleting words. As Steinman (2002:293-294) explains, there are

two options in designing a cloze test according to its deletion rate: a random cloze or a

rational cloze. A random cloze deletes every nth word consistently, so that all classes and

types of words have an equal chance of being deleted. A rational cloze is the one in

which a specific type of word is deleted according to a linguistic principle, such as nouns,

verbs, adjectives, etc.

Alderson (2000) cited by Yamashita (2003:269) clearly differentiates between

these two types of format by calling the rational cloze ‗gap-filling tests‘ and confining the

term ‗cloze‘ only to the random cloze. Ahluwalia (1992:83-84) states that different

deletion rates would result in either increasing the difficulty of the text or in

unpredictable differences in the difficulty. Another factor that would affect the reliability

and the validity of the cloze is the nature of the text. The nature of the text such as its

familiarity level and difficulty level would improve or hamper cloze performance

(Ahluwalia, 1992:86-88).

However, Alderson (1980, 1983 cited in Ahluwalia, 1992:86) and Yamashita

(2003:286-287) argue that cloze test performance is not directly related to the difficulty

level of the text; it involves other factors such as scoring procedures, and content

familiarity for the readers. Different scoring methods include: exact word method,

multiple-choice scoring method and contextually acceptable word method. 16‗Exact

word method‘ requires the examinee to provide the original word deleted from the text

while ‗contextually acceptable word method‘ allows for the words that fit the gap either

to be synonyms of the deleted.

Statement of the Problem






6.1 Age;

6.2 Gender; and






7.3 Usability?


9. Is there a significant relationship between the assessment - related factors and

their mean score in Cloze – test mean score?



Scope and Limitations

This study focused on the effectiveness of cloze test in measuring performance

related to English subject.


It is limited only to twenty - nine (29) respondents of the third year students who

are enrolled in Academic Year 2012-2013 at Laguna State Polytechnic University – San

Pablo City Campus, they were purposively selected.

In this study, the researcher used instruments such as survey questionnaire

intended for students and 25 items for measuring their performance using Cloze type of

test which served as the main instrument of the study.

Significance of the Study

The study will be significant to the following:

Teachers and instructors especially for those specializing English, for it will

benefit them in a way of identifying the factors underlying proper assessment;

Students, for it will help them to improve and be competent in studying English

and provoke their interest.

Parents, to furnish and give aid for their children an interest to learn the

complexities of motivating a child in reading;

And finally, for researchers, for it will help them gain satisfaction for knowing

some insights about the given problem and develop research skills to be more productive

in the field that they had chosen to belong in the future.

Theoretical framework

Cloze technique, developed by Taylor (1952), is firstly designed as a tool for

measuring readability of texts, and now it has been used widely as an exercise or test


method in foreign language teaching and testing. ―It works on the principle of closure and

anticipation, in that the reader is required to reconstruct systematically mutilated passage‖

(Hofman, J. E. & Habib-Allah, M., 1982:276). Actually, as early as 1971, John Oller

advocated that cloze testing was an excellent integrative test. Joseph Boyle and Peter

Falvey (1994) claims that cloze test is one variant of integrative tests, too. And they

believe that ―for cloze test, it is pragmatic expectancy grammar that constitutes the trait

that reflects linguistic competence‖. In addition, many researches from 1970s to 1980s on

cloze test supply people with an objective viewpoint that cloze test is a reliable and valid

test method to investigate the difficulty level of the text and the test-takers‘ capability in

reading comprehension. It can be used as a part of standard integrative language

proficiency testing and a subsidiary tool for language teaching.

Cloze is originally called ―cloze procedure‖, whose theoretical foundation is Gestalt

psychology. Those psychologists believe that when people observe the shape of an object,

they will unconsciously fill in the broken part. Such is the case in reading process. A

sentence or a passage, deleted some words, is just like a broken picture. People will

complete it unconsciously according to their comprehensive language knowledge. The

Gestalt theory also claims that the more familiar with the picture the easier for people to

recognize it. Expounded in language learning, the case is that the higher language

proficiency the higher ability to complete the broken passage.

Besides the Gestalt theory, researchers try to use Redundancy Information theory

and Expectancy Grammar theory to explain the cloze test. In Longman Dictionary of

Language Teaching and Applied Linguistics, redundancy is defined as ―the degree to


which a message contains more information than is needed for it to be understood.

Languages have built-in redundancy, which means that utterances harbor more

information than is necessary for comprehension. And 50% of normal language is said to

be redundant‖. Li Xiaoju (2001) also approves that in cloze test, people use the

redundancy information of natural language to complete the broken language. Actually,

this theory is crucial for Chinese learners when learning English. The process of

accumulating language knowledge is the process of recognizing the redundancy

information in English language. However, the information is necessary in that it can

make the meaning of the text much more accurate, although it is regarded as more than is

needed. For Chinese students, the recognition of the redundancy information is helpful

for them to improve their sensitivity to the English language environment and their

ability to understand the inner relationship in language itself and the consistency of its

usage, which enhances the students‘ language proficiency. Therefore, the cloze test

removes some redundancy information from the completed text to examine whether test-

takers obtain the capability to comprehend the broken passage and to fill in the blanks or

not.

Expectancy Grammar was firstly advocated by Oller in 1976. Later Carter (1998)

believes that it refers to language user‘s comprehensive language ability to use syntax,

semantic and discourse knowledge. In reading process, the reader can use the ability to

predict in what way the sentence will end and to what trend the text will develop. Cloze

test investigates the very ability acquired by language learners through long-term learning

and perception, and proficient mastering of language structure as well as basic use


principle of the target language. In a broken passage, according to the context and their

language ability, students can predict the syntactic and semantic function of the deleted

word, along with the relationship between the word and the other part within the

sentence.

Conceptual Framework

The conceptual framework of this study is presented to give view of the work to

be done. The inputs or the independent variables are the profile of the respondent mainly

3rd year high school students and their perception regarding the effectiveness of a test

which composed of Controllability, Reliability and Validity and Usability.

The process is the strategy to be used such as generating data through the use of

questionnaire. The possible output or the dependent variable is the result or their

performance in English by means of using Cloze Test. It is assumed to be affected by the

independent variable.

Paradigm


Independent Variables Dependent Variable

I. Profile of the respondents
Age
Gender
Cloze – Test
Grade in English

II. Assessment – related factors
Controllability
Reliability & Validity
Usability

Figure1. Shows the relationship between the independent variables and the

dependent variable.

Hypotheses

3. There is no significant relationship between the assessment related factors and the

mean score in Cloze – test.

4. There is no significant relationship between grade point average in English and

their result of Cloze – test.

Definition of Terms


For a better understanding of this study, the following terms are

operationally defined.

Age. Refers to how old or young a particular person was.

Cloze Test. Refers to integrative assessment in measuring higher cognitive task, deals

with several knowledge components at once, focusing more on cognitive language used,

and typically requiring the examinees to read and comprehend having a substantial

discourse.

Controllability. It refers to the level of manipulation of effectiveness of Cloze type of

test.

Gender. It refers to a particular person whether s/he is a male or a female.

Grade Point Average. Refers to the recent average grade of 3rd year high school students

during 2012 – 2013.

Reading Comprehension. Refers to intellectual capacity of the respondents regarding

the passage of History of Cat.

Reliability. Refers to the consistency of Cloze test in assessing English subject.

Validity. Refers to what purports to measure in a given context.



Chapter II

Review of Related Literature and Studies

The chapters presents a selection of literature and studies that has a

bearing on present study. Most of the literature gathered comprises Cloze Test

procedure.

Related Literature

Cloze test has been changing and developing. According to different deletion and

fill-in-blank methods, it can be mainly classified to fixed-ratio cloze, rational cloze,

multiple choice cloze, C-test and banked cloze.


Fixed-ratio cloze was proposed by Oller in 2005. It is constructed by deleting words

according to a fixed pattern, either at even numbers (4, 6, 8, 10, etc.) or at odd numbers

(5, 7, 9, 11, etc.). ―This procedure is intended to sample regularly various variants of

words, some of which are governed by local grammatical constraints and wide discipline

vocabularies in sorted areas of History, Mathematics, Statistics, others of which are

governed by long-range textual constraints‖. The advantages of this cloze variant are

convenience and high consistency, while there are also a lot of problems such as some of

the blanks being too easy or too difficult, the broken passage sometimes arousing test-

takers‘ dislike and resulting in a dilemma for testers to choose subjective or objective

scoring method, and being lack of control which possibly leads to examine the language

points the designer doesn‘t want to test actually. Rational cloze can also be called open

cloze or gap-filling. The test developer control over the types of the words deleted such

as functional words, verbs or nouns, and thus the language traits measured. This selected

deletion changes the random of cloze testing into objectivity and practicality. Just like in

fixed-ratio cloze, test-takers have to fill in the blanks in rational cloze according to their

comprehension capability. ―Rational cloze research and practice rests on the assumption

that different cloze items can be explicitly chosen to measure different language traits‖

(Chapelle & Roberta, 1990:122). The understanding advantage of rational cloze is that

the test designer can select particular item to determine test-point. However, it is

confronted with the problem that subjective or objective scoring method, either.

In order to solve the scoring problem of fixed-ratio and rational cloze, test

developers design multiple-choice cloze. This cloze test format requires test-takers to


select the correct answer from the provided options. It offers four or five options to each

blank, while only one is the correct answer and the others are distracters. There are no

accepted answers which get rid of scoring controversies. Generally speaking, high

reliability is the specialty of multiple-choice cloze. However, the inappropriate options or

explanation of new words will give test-takers some hints, which possibly reduce the

difficulty level and the validity of test. Therefore, multiple-choice cloze demands test

developers‘ enough attention and obeying some strict rules.

A cloze test (also cloze deletion test) is an exercise, test, or assessment consisting

of a portion of text with certain words removed (cloze text), where the participant is

asked to replace the missing words. Cloze tests require the ability to understand context

and vocabulary in order to identify the correct words or type of words that belong in the

deleted passages of a text. This exercise is commonly administered for the assessment of

native and second language learning and instruction.

The word cloze is derived from closure in Gestalt theory . The exercise was first

described by W.L. Taylor in 1953.

Words may be deleted from the text in question either mechanically (every nth

word) or selectively, depending on exactly what aspect it is intended to test for. The

methodology is the subject of an extensive academic literature; nonetheless, teachers

commonly devise ad hoc tests.

A Cloze test removes certain words from a sample of your text and asks users to

fill in the missing words. Your test participants must rely on the context as well as their


prior knowledge of the subject to identify the deleted words. It‘s based on the Gestalt

theory of closure—where the brain tries to fill in missing pieces—and applies it to written

text.

It looks something like this:

If you want to __________ out whether your site __________ understand your content,

you __________ test it with them.

It looks a lot like a Mad Lib, doesn‘t it? Instead of coming up with a sentence that

sounds funny or strange or interesting, participants must guess the exact word the author

used. While Cloze tests are uncommon in the user experience field, educators have used

them for decades to assess whether a text is appropriate for their students, particularly in

English-as-an-additional-language instruction.

Here‘s how to do it:

Take a sample of text—about 125-250 words or so.

Remove every fifth word, replacing it with a blank space.

Ask participants to fill in each space with the word they think was removed.

Score the answers by counting the number of correct answers and dividing that by the

total number of blanks.

A score of 60% or better indicates the text is appropriate for the audience.

Participants who score 40-60%, will have some difficulty understanding the original text.

It‘s not a deal breaker, but it does mean that the audience may need some additional help


to understand your content. A score of less than 40% means that the text will frustrate

readers and should be rewritten.

It might sound farfetched, but give this method a try before you dismiss it. In a

government study on healthcare information readability, an expert panel categorized

health articles as either easy or difficult. We ran a Cloze test using those articles with

participants—who had low to average literacy skills—and found that the results reflected

the expert panel‘s findings. The average score for the ―easy‖ version was 60, indicating

the article was written at an appropriate level for these readers. The average score for the

―difficult‖ version was 39: too hard for this audience.

Cloze tests are simple to create, administer, and score. They give you a good idea

as to whether the content is right for the intended audience. If you use Cloze tests—either

on their own or with more traditional usability testing methods—know that it takes a lot

of cognitive effort to figure out those missing words. Aim for at least 25 blanks to get

good feedback on your text; more than 50 can be very tiring.

When to test

Test your content at any point in your site development process. As long as you

have content to test, you can test it. Need to convince your boss to budget for content

testing? Run it through a readability formula. Got content but no wireframes or visual

design? Run a Cloze test to evaluate content appropriateness. Understands the content

key to a task or workflow? Display it in context during usability testing.


What to test

You can‘t test every sentence on your site, nor do you need to. Focus on tasks that

are critical to your users and your business. For example, does your help desk get calls

about things the site should communicate? Test the content to find out if and where the

site falls short.

So get to it

While usability testing watches what users do, not what they say they do, content

testing determines what users understand, not what they say they understand.

Whatever your budget, timeline, and access to users, there‘s a method to test

whether your content is appropriate for the people reading it. So test! And then, either

rest assured that your content works, or get cracking on that rewrite.

A language teacher may give the following passage to students:

“ Today, I went to the ________ and bought some milk and eggs. I knew it was

going to rain, but I forgot to take my ________, and ended up getting wet on the

way ________. ”

Students would then be required to fill in the blanks with words that would best

complete the passage. Context in language and content terms is essential in most, if not

all, cloze tests. The first blank is preceded by "the"; therefore, a noun, an adjective or

an adverb must follow. However, a conjunction follows the blank; the sentence would not


be grammatically correct if anything other than a noun were in the blank. The words

"milk and eggs" are important for deciding which noun to put in the blank; "supermarket"

is a possible answer; depending on the student, however, the first blank could either

be store, supermarket, shop or market while umbrella or raincoat fit the second.

The definition of success in a given cloze test varies, depending on the broader

goals behind the exercise. Assessment may depend on whether the exercise is objective

(i.e. students are given a list of words to use in a cloze) or subjective (i.e. students are to

fill in a cloze with words that would make a given sentence grammatically correct).

“ I saw a man lay his jacket on a puddle for a woman crossing the street. I thought

that was very ______. ”

Given the above passage, students' answers may then vary depending on their

vocabulary skills and their personal opinions. However, the placement of the blank at the

end of the sentence restricts the possible words that may complete the sentence;

following an adverb and finishing the sentence, the word is most likely an adjective.

Romantic, chivalrous or gallant may, for example, occupy the blank, as well

as foolish or cheesy. Using those answers, a teacher may ask students to reflect on the

opinions drawn from the given cloze.

In addition to use in testing, cloze deletion can be used in learning, particularly

language learning, but also learning facts. This may be done manually – for example, by

covering sections of a text with paper, or highlighting sections of text with a highlighter,


than covering the line with a colored ruler in the complementary color (say, red ruler for

green highlighter), so the highlighted text disappears; this is popular in Japan, for

instance. Cloze deletion can also be used as part of spaced repetition software, and

the Super Memo application features semi-automated creation of cloze tests, particularly

as part of its incremental reading feature.

Related Studies

The Cloze test is used primarily to determine a student‘s level of reading ability.

A student is first tested for reading level; in this case the Slosson oral has previously been

administered. After a student‘s reading level has been determined, a reading selection is

taken from a book of like grade level. As is standard, every fifth word is removed and

replaced by a blank space, in this assignment there were 20 blanks in the selection. In this

assignment, as was suggested, a paragraph of text preceded the actual test material.

Evaluation of the test determines a student‘s reading level; an independent reading level

indicates that the student could read the selected book on their own, instructional level

means that the student can read the book with assistance and frustration level means that

the book is difficult for the student to read and is likely above the student‘s reading grade

level. For the purposes of this assignment, the following scale was assigned for

evaluation: the independent reading level is based as 50% or more of the blanks filled in

correctly, instructional level is 30% - 50% and frustration level.

On Friday, October 6, 1995, I administered a Cloze test to a student named

Natalie at Stivers Middle School in Dayton, who I had administered a Slosson oral test to


the previous week. On the Slosson oral, Natalie scored in the 8th grade reading level

(8.95 to be exact). Taking her Slosson score, I set out in search of a text book involving

African American studies, an area she is very interested in. After much searching, I

secured the book African American History at the Wright State ERC and submitted it to

Fry Graph analysis (attachment 1). Taking three 100 word samples randomly from the

book I computed the approximate reading grade level of the book. Sample 1 consisted of

7.3 sentences and 149 syllables, sample 2 had 9.2 sentences and 155 syllables and sample

3 had 8.5 sentences and 158 syllables. The average of the samples was 8.33 sentences and

154 syllables. I plotted this point on the graph, and fortunately it fell within the 8th grade

reading level. I then constructed a Cloze test to be administered.

The test was administered to Natalie during the 4th period in the same conference

room Mrs. King had selected for me to give Natalie the Slosson oral test the previous

week. I explained the test to Natalie and gave her the test to fill out (attachment 2). While

Natalie was taking the test, I either sat down at a nearby table or walked to a nearby

window. Natalie finished the test in about eight minutes, and I quickly compared it to my

master copy (attachment 3). I then sat down with Natalie and asked her why she had

chosen each of her selections, the correct and incorrect ones alike. I recorded her

responses, and thanked her for helping me and she returned to class.

Natalie correctly filled in 11 of the 20 blanks (55%) and placed in the independent

reading level. Below is a chart of error, indicating the correct response, the response

given, the reason the answer was given, if there was a change in grammar and if there

was a change in the meaning of the statement. Following the chart will be a more detailed


account of why each incorrect answer was given for different items.

I asked Natalie why she chose each of her selections, and asked her if other

words would make sense in the context of the sentence. I did this for all 20 selections;

including the ones she got right as well as the ones she got wrong. For example on

selection 1, which she filled in correctly, I asked her if ‗is‘ would have worked. She

promptly told me that everything was in the past, so ‗is‘ would be wrong.

In her first mistake on the test, Natalie substituted ‗food‘ for the word

‗literature‘. I asked Natalie if the word ‗literature‘ would fit better, and Natalie told me

that most likely books would be together with art. On the second mistake, Natalie used

the phrase ‗brought on‘ instead of ‗stimulated‘. I asked her about using ‗stimulated‘ and I

was promptly asked what the word meant. The third mistake made was the substitution of

the phrase ‗in many‘ instead of ‗of‘. When I asked Natalie about this, she read the

sentence back with ‗of‘ in its place and said it didn‘t sound right. On this I really had to

agree.

The fourth mistake made was the usage of ‗racism‘ instead of ‗oppression‘.

When I asked Natalie about this I was met with the same response I got for ‗stimulated‘:

what does it mean? Natalie‘s next mistake came when she used ‗art‘ instead of ‗society‘

in the selection. Again I asked her about using society in the sentence, and Natalie said it

seemed right to her since art had been used before in the paragraph.

Natalie‘s sixth error was in putting ‗something‘ in the selection instead of

‗things‘. When I asked her about the possibility of using ‗things‘, she simply told me that


nobody used that word that way. Again, I would have to agree with her since I have heard

few people ever say ‗things new‘. The seventh mistake was the use of ‗movement‘

instead of ‗Renaissance‘. As I had done several times before, I asked her about her choice

and asked if ‗Renaissance‘ would have worked. Natalie‘s response was that it wouldn‘t

work because a renaissance is something 'really old'. I could understand this reasoning,

since the only time students really hear this term is in early European history.

The eighth mistake made on the Cloze test was the use of the word ‗racist‘

instead of ‗this‘. I really didn‘t understand this substitution, and when I asked about it

Natalie said that she couldn‘t think of anything else, that she had used ‗racist‘ before in

the selection and ‗racist‘ was the only thing she could associate with New York in the

context of the sentence. The ninth, and final, mistake Natalie made was the use of the

phrase ‗period for‘ instead of ‗by‘. By this time Natalie seemed tired of all this and

simply said she couldn‘t think of any other word to use. She wanted to return to her class,

so I thanked her for putting up with me and my tests and let her return to the room.

From my perspective, most of Natalie‘s errors on the Cloze test could be

attributed to two things: she was unfamiliar with some of the vocabulary used and the

text in some places was written in a way in which few students (or adults for that matter)

speak. So, what can be done about it? If this had been an actual textbook assignment, one

thing that could be done is to provide the students with a list of vocabulary words they do

not understand completely strongly affirmed According to most researchers in Europe.



Chapter III

Research Methodology

This chapter describes how the study was conducted and the researcher used in

gathering the necessary data for the completion of the study. This includes the research

design, the population and sampling, sampling procedure, data gathering procedure,

research instrument and the statistical treatment of data.

Research Design


of Cloze Test in measuring reading comprehension as a subject in secondary level. The

possible solutions obtained through a questionnaire as well as interviews and observation

made by the researcher. This study used of the third year students in Laguna State


Polytechnic University as its respondents, the main source of data. The instrument used

was a collaboration of information based on Crystal, David (2004). Contextual

Constraints in Cloze Test. Cambridge: Cambridge University Press. ISBN 0-521-53032-6.

Some of its modifications were undertaken to adapt on the research problem and scope of

the study.

Respondents of the Study

This study was conducted at Laguna State Polytechnic University, San Pablo City

Campus Academic Year 2012-2013. The researcher gets a representative for 3rd year high

school department to suit the twenty - nine (29) respondents needed for the study.

A purposive sampling as a sampling technique was utilized.

Research Instrument

The main instrument used in the study is a questionnaire which was used to know

person related factors such as age, gender, and grade point in English during second

semester A.Y. 2012- 2013. And an integrative assessment in reading comprehension

comprises of twenty – five (25) items using Rational Cloze Test procedure. After reading

and studying samples of questionnaire from related studies, the researcher adopted a

rational cloze test which based on Crystal David’s educational research: Cambridge

year 2004. She also consulted some knowledgeable people and sees to it that there were

enough items to collect data to cover all aspects of the problem and to answer all the

specific questions under the statement of the problem.


Statistical Treatment of Data

The data were gathered from the respondents treated and interpreted using the

statistical tools. The following statistical analysis was coded in the study:

Survey Questionnaire

Age

Legend:

14y/o below – 1

15y/o – 2

16y/o and above – 3

Gender

Legend:

Male – 1

Female – 2

Current Grade Point in English

Legend:

93 and above – 1

90 – 92 – 2

87 – 89 – 3

84 – 86 – 4

81 – 83 – 5

80 and below – 6


Cloze test

Legend:

correct – 1

incorrect - 0

Percentage distribution was utilized to describe the person related variables in

terms of, age, gender, and grade point in English.

For the assessment – related factors, the researcher utilized the mean and standard

deviation using a Likert scale, 5 being the highest, denotes ‗strongly agree‘ and 1 being

the lowest, denotes ‗strongly disagree‘.

To find that if there is a relationship existing between the independent and

dependent variables, Pearson Product Moment of Correlation Coefficient was used to tell

how well two sets of continuous data correlate to each other at 0.05 level of significance.



Chapter IV

Presentation, Analysis, and Interpretation of Data

This chapter presents the findings of the study in illustrative tables and analysis as

well as the interpretation based from the treatment of the data.

Table 1
Distribution of Respondents According to Age
Age Frequency Percent
14 y/o below 8 28
15y/o 20 69
16y/o and above 1 3
TOTAL 29 100

Table 1 shows the distribution of respondents according to their age, the majority of the

respondents are 15y/o having twenty (20) or 69% of the total respondents.


This implies though the inability to randomize the selection because the research has

lack of features of the true experiment and based on the experimental values the respondents

are equipped subject for experimental condition since they undergone English as subject. As

Datta (2002) mentioned Cloze tests require the ability to understand context and vocabulary in

order to identify the correct words or type of words that belong in the deleted passages of a

text most of the studies underlies puberty period. This exercise is commonly administered for

the assessment of knowledge and second language complex learning and instruction.

Table 2
Distribution of Respondents According to Gender
Gender Frequency Percent
Male 12 41.38
Female 17 56.62
TOTAL 29 100

Table 2 clearly shows that female outnumbered the male having a frequency of 17 or

56.62% and the male has 12 frequency or 41.38%. This implies that more female are subjected

in the study.

Table 3
Distribution of Respondents According to Grade Point in English
Grade point Frequency Percent
93 and above 0 0
90 – 92 6 21
87 – 89 10 34
84 – 86 8 28
81 – 83 5 17
80 and below 0 0


TOTAL 29 100

Table 3 shows the grade point average in English of the respondents. The highest grade

point average that the respondents attained ranged from 87 – 89 having a frequency of ten (10)

or 34% second ranging from 84 – 86 having a frequency of 8 (eight) or 28% followed by 90 – 92

having six (6) or 21% subsequent to that is ranged from 81 – 83 and the grade point average of

93 and above and 80 below has zero frequency.

Table 4
Respondents’ Perception on Controllability as a Factor of Integrative Assessment
Standard
Indicators Mean Remarks
Deviation
1. The Cloze test procedure can easily adjust Strongly
4.59 0.56
the instructional planning. Agree
2. It can limit the learners the extension of Strongly
4.72 0.58
different vocabularies. Agree
3. Can easily manipulate by the teachers and Strongly
3.72 0.83
learners as well. Agree
4. Placement of the blank at the end of the
Strongly
sentence in the passage restricts the possible 4.59 0.56
Agree
words that may complete the sentence.
5. The instructors can readily assume the
4.0 0.74 Agree
fairness in the interpretation results.
Strongly
TOTAL 4.32 0.65
Agree
Legend:
4.21 – 5.0 : Strongly Agree
3.41 – 4.20 : Agree
2.61 – 3.40 : Fair
1.81 – 2.60 : Disagree
1.0 – 1.80 : Strongly Disagree


Table 4 shows the respondents’ perception on Controllability as a variable of

assessment related factors. As we observed item number four reigns the other indicators having

a mean of 4.49 and interpreted as Strongly agree by the respondents on cloze test. Along with

the total average resulted with 4.32 and interpreted as Strongly agree over all. This implies a

strong affirmation on the effectiveness of cloze test in assessing reading comprehension.

However, Alderson (1980, 1983 cited in Ahluwalia, 1992:86) and Yamashita (2003:286-

287) supported that cloze test performance is not directly related to the difficulty level of the

text; it involves other factors such as scoring procedures, and content familiarity for the readers

to be able them to manipulate their cognitive concerns.

Table 5
Respondents’ Perception on Reliability and Validity as a Factor of Integrative Assessment
Standard
Deviation
1. Cloze test have adequacy and Strongly
4.55 0.56
representativeness of learning outcomes. Agree
2. The passage measures what it purports to
4.14 0.59 Agree
measure.
3. Cloze test must first consider the errors of Strongly
4.31 0.70
measurement. Agree
4. It can easily interpret the test results to Strongly
4.76 0.73
improve teaching and learning. Agree
5. It has a value of testing as a tool to improve Strongly
4.90 0.55
instruction and learning performance. Agree
Strongly
TOTAL 4.53 0.63
Agree
Legend:
3.41 – 4.20 : Agree
2.61 – 3.40 : Fair
1.81 – 2.60 : Disagree


Table 5 shows the respondents’ perception on the reliability and validity of cloze test in

assessing knowledge in english. As the total mean yields 4.53 and is interpreted as Strongly

agree, which implies the elevating consistency and efficiency in english assessment treatment.

. Much research has been devoted to the validity and the reliability of Cloze tests.

Cloze tests are reported to have moderate to high correlations with standardized tests and

their subtests such as listening comprehension and reading comprehension. The report on

high correlations between Cloze tests and International Journal of Human and Social

Sciences tests of listening comprehension, writing, reading comprehension, and oral

interview. The findings of these researchers reveal a relation between the scores on Cloze

and global cognitive ability tests (Greene, 2001).

Table 6
Respondents’ Perception on Usability as a Factor of Integrative Assessment
Standard
Deviation
1. It manifests strong retention constructed –
3.48 1.04 Agree
response type.
2. Creates critical reasoning for learners on Strongly
4.62 0.89
how to arrive on precise conclusions. Agree
3. Learners can employ to assess vocabulary,
understanding facts, seeing, relationships,
Strongly
drawing inferences, detecting author‘s style, 4.48 0.56
Agree
and approximating the patterns of language
structure.
4. The test provides communicative skills in Strongly
4.55 0.97
understanding such areas of learning. Agree
5. Contributes logical argumentation and
empirical evidence for learners such as Strongly
4.46 0.62
resourceful in resolving authentic problems Agree
in studying english.
Strongly
TOTAL 4.32 0.82
Agree


Legend:
3.41 – 4.20 : Agree
2.61 – 3.40 : Fair
1.81 – 2.60 : Disagree

Table 6 shows the level of perception of 3rd year students with regards to the usability of

cloze test in assessing reading comprehension. Based on the statistical treatment the overall

mean is 4.32 and the standard deviation is 0.82 and interpreted as ‘Strongly agree’ which also

implies cloze test useful in measuring complex cognitive ability of the respondents. As Greene

(2001) claimed that sentence or a passage, deleted some words, is just like a broken picture.

People will complete it unconsciously according to their comprehensive knowledge. The Gestalt

theory also claims that it is very useful in recognizing evaluative range of knowledge since the

more familiar with the picture the easier for people to recognize it. Expounded in language

learning, the case is that the higher language proficiency the higher ability to complete the

broken passage and promotes higher order thinking skills but also more on life – relating

tendencies which could be applicable to further level of a child.



Table 7
Distribution of Result in Rational Cloze Test
Item no. Frequency Percent Mean Interpretation
1 13 52 0.45 Satisfactory
3 8 32 0.27 Fair
4 9 36 0.31 Fair
6 12 48 0.44 Fair
7 2 8 0.06 Needs improvement
8 12 48 0.17 Fair
12 11 44 0.06 Fair
14 10 40 0.04 Fair
15 12 48 0.27 Fair
17 12 48 0.17 Fair
18 0 0 0 Needs improvement
20 0 0 0 Needs improvement
21 8 32 0.27 Fair
22 11 44 0.06 Fair
TOTAL 221 100 0.28 Fair

Legend:
24 – 25 : Outstanding
20 – 23 : Very Satisfactory
13 – 19 : Satisfactory
8 – 12 :Fair
0 – 7 : Needs Improvement

This table shows the result of the cloze test of 3rd year high school, as we observed both

item number 10 and 11 has a high frequency which interpreted as ‘Satisfactory’ based on the

scale values above having a 60% of the total correct answer. As we look generally there are

more results ranged from 8 – 12, thus the mean score of the respondents based on the

experiment is interpreted as ‘Fair’.

This implies that the respondents performed the test even-handedly since the total

interpretation is said to be ‘Fair’ with regards to this Legenza (2001) reasoned behind giving a

Cloze test and obtaining a test score is interpreting that score as an indicator of what a test

taker knows or what he/she can do with that knowledge. Furthermore, our interpretation of

that test score forms the basis for decision making. As such, when using a test score, we make

an implicit link between test performance and a domain of knowledge the test taker has or

something the test taker can do with schema in some area use domain beyond the test itself.

The advantages of this cloze variant are convenience and high consistency, while there

are also a lot of problems such as some of the blanks being too easy or too difficult, the broken

passage sometimes arousing test-takers’ dislike and resulting in a dilemma for testers to choose

subjective or objective scoring method, and being lack of control which possibly leads to


examine the language points the designer doesn’t want to test actually. Rational cloze can also

be called open cloze or gap-filling.

Table 8
Correlation between Assessment – related variables and Cloze Test
Variables r - value p - value Interpretation

Controllability .789 .0376 Significant
Reliability and
.076 .0167 Significant
Validity
Usability .098 .0092 Significant
Legend:
p >.05 : not significant
p <.05 : significant

Table 8 clearly shows that all of the assessment related variables are interpreted as

‘Significant’ since their p – values are less than 0.05 level of significance.

This procedure is intended to sample regularly various variants of words, some of

which are governed by cognitive constraints and wide discipline vocabularies in sorted

areas of History, Mathematics, Statistics, others of which are governed by long-range


textual constraints (Datta, 2002). Also supported by Greene (2001) Much research has

been devoted to the validity and the reliability of Cloze tests. Cloze tests are reported to

have moderate to high correlations with standardized tests and their subtests such as

listening comprehension and reading comprehension.

Table 9
Correlation between Grade Point Average in English and Cloze Test
Variable r - value p - value Interpretation

Grade Point Ave. -0.565 .0676 Not Significant

Legend:
p >.05 : not significant
p <.05 : significant

Table 9 shows the correlation between the grade point average of third year students

and their results in cloze test. Clearly observed that the grade point average in english is

interpreted as Not Significant to the result in cloze test since the p – value is greater than 0.05

level of significance.

Askes (2002) regards cloze procedure as one of the integrative tests (global tests)

that integrate language components into a total language event, which requires an

integrated performance from the learner in a meaningful context. Thus for Steinman


(2002) a cloze can replace the sections on test structure, written expression, vocabulary,

and reading comprehension not only in English subject as well as different areas of

learning.

The reason behind giving a Cloze test and obtaining a test score is interpreting

that score as an indicator of what a test taker knows or what he/she can do with that

knowledge. Furthermore, our interpretation of that test score forms the basis for decision

making. As such, when using a test score, we make an implicit link between test

performance and a domain of language knowledge the test taker has or something the test

taker can do with language in some language use domain beyond the test itself. In other

words, when we use test scores, we are essentially reasoning from evidence, using the

test score as the evidence for inferences or interpretations and decisions we want to make.

Yet, we cannot simply draw on test score to make inferences and decisions without

efficient justification. If we want to use a test score for a particular purpose, we must

justify it through a rationale and supporting evidence.

Chapter V


Summary, Conclusions, And Recommendations

This chapter presents the summary of findings, conclusions, and

recommendations related to the effectiveness of cloze test in assessing reading

comprehension.

Summary





1.1 Age;

1.2 Gender; and






2.3 Usability?


4. Is there a significant relationship between the assessment related factors and the

mean score in Cloze – test?





of Cloze Test in measuring English as a subject in secondary level. The possible solutions

obtained through a questionnaire as well as interviews and observation made by the

researcher. This study used of the third year students in Laguna State Polytechnic

University as its respondents, the main source of data. The instrument used was a

collaboration of information based on Crystal, David (2004). Contextual Constraints in

Cloze Test. Cambridge: Cambridge University Press. ISBN 0-521-53032-6. Some of its

modifications were undertaken to adapt on the research problem and scope of the study.

Findings

Majority of the respondents are in age 15y/o with a frequency and percentage of

twenty (20) or 69%; seventeen (17) or 56.62% are female students; ten (10) or 34% of the

grade point average of 3rd year students lie between 87 – 89.

All of the assessment – related variables such as Controllability (mean = 4.32),

Reliability & Validity (mean = 4.53), and Usability (mean = 4.32) and interpreted as

‗Strongly Agree‘.

The Cloze test result has a frequency of two – hundred twenty one (221) and has a

mean of 0.28 which is overall interpreted as ‗Fair‘.


thesis in English

Recommended

Recommended

More Related Content

What's hot

What's hot (20)

Similar to thesis in English

Similar to thesis in English (20)

More from Carie Justine Estrellado

More from Carie Justine Estrellado (20)

Recently uploaded

Recently uploaded (20)

thesis in English