Documentary Research: Refers to research that is based on documentary sources or factual records and reports.
Text, in this sense, refers to all forms of recorded material.As a research technique, it is required that all studies be replicable. In order to meet this requirement, all forms of communication need to be stable and durable. (Holsti, 1969 p. 2). Documents include any text, audio recordings, music, videotapes, drawings, television, or images. In the digital age, content analysis includes the analysis of stable email, posts, blogs, podcasts, webinars, and all forms of digital communication. Lasswell, a communications scholar, used content analysis to study propaganda. .This well-known question combines quantitative procedures or enumeration and qualitative procedures or context and proximity.
Categories were formed that encompassed a concept that many words could convey. The notion of the connotation and ambiguity of words and their meanings were recognized.
The process of determining authorship involves compiling a list of probable authors and comparing their, language, use of language, themes, and all stylistic features that characterize each author. That author whose textual content is most similar in all defined categories to the text in question would be designed the rightful author. One of the most recent, successful examples of this use was in identifying the anonymous author of the 1990’s book,” Primary Colors.” The book dealt with Clinton’s 1992 Presidential campaign, but the authorship had not been disclosed. Using the method of content analysis, the literary techniques of rhetorical analysis and narrative analysis, Donald Foster, a Vassar professor, identified Joe Klein, a Times magazine writer, as the author of the book.
Program effectiveness in education would look at mission statements and program objectives, then evaluate whether the criteria used to measure program effectiveness (measures of achievement—test scores, performance, behavior) reflects the objectives. Or, how do criteria used to assess learning reflect objectives established for a course. Objectives and criteria should be established at the start of a program or course—not during.
Strictly speaking, qualitative studies require a priori categories; qualitative studies are characterized by their emergent and inductive coding of categories.
(Krippendorf, 2003)Stemple (2001) cites three problems associated with compiling documents for content analysis 1. a significant number of documents from the population are missing 2. Records or Documents that do not meet the definition of the document should be discarded and a notes citing the reason kept. 3. Documents meet the definition, but can not be coded because they are missing passages or have ambiguous content.
Paraphrased from Holsti (1969, p. 4).
Objectivity, System, and Generality are not exclusive to content analysis. They are required conditions of all scientific or reasoned inquiry.
(Holsti, 1969 p. 10)By interact, I mean that each methodology can inform the other.
Max Weber formally emphasized the importance of applying quantitative and qualitative approaches in analyzing how the contents of newspapers effect public opinion. Weber referred to this study as the sociology of the press.
This is a seriously rough means of testing reliability. However, it may reveal some of the difficulties involved with determining features and consistently coding them.
Methodology & Content analysis
Content Analysis Multiple Definitions Background Applications and Procedure An Illustration Consistency within Diversity Growth Reliability Validity
General Definitions “...a research technique that uses a set of procedures to make valid inferences from text” (Weber, 1990, p. 9) A summarizing, quantitative analysis of messages relying on scientific method and not limited to types of variables measured or context of messages (Neuendorf, 2002) "Any technique for making inferences by objectively and systematically identifying specified characteristics of messages” (Holsti, 1969 p. 14)
Definitions (continued) “...technique for objective, systematic, & quantitative description of manifest content communication” ( Berelson, 1952, p. 18) “ …systematic, replicable data reduction technique, compressing many words of text into content categories based on explicit rules of coding” (Stemler, 2001) “Close analysis of explicit and implicit messages of a text through classification and evaluation of key concepts, symbols, and themes to determine meaning and explain its effect it on the audience” (Reitz, 2004) “...systematic reading of texts and symbolic matter not necessarily from an author’s or users perspective” (Krippendorf, 2004, p. 3)
Essential PurposeWho (says) What (to) Whom (in) what Channel (with) What Effect? (Lasswell, 1948)
Goal of Content Analysis To draw inferences about: A text To obtain documentary evidence about: A sender A message A receiver (audience)
Procedures Communication transmission model (manifest content) Semiotic model (latent content) of the signifier and the signified
Analysis of Texts Researchers have analyzed texts since the start of writing Content analysis draws on Aristotelian principles of textual analysis as described in “Poetics” Subject of text Creator Audience Purpose Context Ethos Pathos
Analysis of Texts Medieval Scholastics recognized latent meanings of words, leading to categories First documented content analysis: 18th century Sweden Collection of religious hymns blamed for subverting orthodox state church and inciting dissent Words and symbols in songs were same as orthodox hymnal, but occurred in different contexts (Dovring, 1951; Krippendorf, 2009)
Analysis of Texts During early 20th century Weber studied content of newspapers raising fundamental questions: How do newspapers affect readers? How do the contents of newspapers compare quantitatively over a generation? What is carried as news What is carried as editorial What is omitted Weber is regarded as a pioneer in the sociology of the press
Psychological Analysis Gordon Allport Pioneer in personality theory Analyzed “Letters to Jenny” (Diary) Aimed to understand her experienced world by identifying psychological traits as expressed in her letters and diaries (Krippendorf& Bock, 2009)
Content Analysis and Data The Effect of Semantics on Public Opinion and Propaganda Program Effectiveness Readability of Text Television Violence Library Transactions Characteristics of Reference Questions Personality Traits Determination of Authorship
Questions for Content Analysis “How are minority ethnic groups treated in short stories in popular magazines?” (Berelson, 1952) “How can communications be tested for propaganda elements?” “What ways do motion pictures reflect popular feelings and desires?” “What are the dominant images in Shakespeare’s plays?” “What happens to a good book when it becomes a movie?” “How do criteria used for program effectiveness reflect program objectives?” How do attitudes reflect personality structure? What domains are currently being studied in library and information science?
A Quantitative IllustrationResearch Problem: What role does popular magazine fiction play in affecting attitudes toward ethnic minorities? (Berelson and Salter, 1946)
Hypotheses Asserted Sympathetic characters are from the majority Unsympathetic characters are from minorities and usually unsympathetic and secondary Social interaction between majority and minority groups seldom occurs. When interaction occurs, minority group in a subordinate position. The problems of majority and minority groups differ The majority has individual (personal) problems The minority has social and economic problems The goals of the majority (love, comfort) more approved than goals of the minority (material gain).
Hypothesis and Categories Hypotheses were translated into categories – a priori assignation The first hypothesis yielded 3 categories for analysis: Ethnic identification of character Role in the story Level of approval (sympathy for character) Other hypotheses provide more categories: Socio-economic status of characters Kind of interaction Problems Goals After categories for the hypotheses are exhausted, they can be addressed by potential data
Final Step in Analysis Establish Categories Determine Indicators (a priori) First general category was ethnicity Indicators of ethnicity rarely explicit (i.e. “He was Italian-American”) Indirect indicators of ethnicity conceived – indicators involved names (i.e. Antonio), language, appearance, memberships, etc. Second category was character’s role in story Was the role major or minor? Indicator was the space allotted to specific character Third category was socio-economic level Indicator included characteristics as occupation, education, income, possessions, standard of living, etc. After examining the stories other categories and indicators were noted. These emergent categories extended the hypothesis (a posteriori)
The Basic Steps Which data are analyzed? (Unit of Analysis) How are data defined? (Definition of Terms--these are the variables to measure. Both internal and external validity come into play here.) What is the population or unit from which the sampling data are drawn? (Sampling Frame or Unit of Sampling) What is the context relative to which the data are analyzed? What are the boundaries of the analysis? What is the target of inference? Code Book - Coding Form
Comparison: Berelson’s Study and Krippendorf’s 6 Questions Krippendorf Berelson1. What data are analyzed? 1. Units of analysis are words.2. How are the data defined? 2. Data defined by categories and their indicators.3. What is the population or unit from which the sampling data are 3. Frame is fictional story in drawn? popular magazine.4. What is the context relative to 4. Context involves attitudes which the data are analyzed? toward ethnicities.5. What are the boundaries of the 5. Boundary is single story in analysis? selection of magazines.6. What is the target of inference? 6. Do indicators support hypothesis?
Two MethodsQualitative and Quantitative Procedures
Content, System, Goals The objective of content analysis is determining answers to questions that the text implies – explicitly and implicitly It is a systematic technique of “reading between the lines” Can this goal be attained most effectively with quantitative or qualitative techniques? Distinguishing feature between 2 approaches is starting point: hypothesis or grounded induction methods of deriving categories what depth of communication accepted as an indicator
Quantitative Procedures Content analysis starts with the hypothesis Categories derived from possibilities of hypothesis Categories established a priori and indicators based on manifest (surface) meaning
Qualitative Procedures Qualitative approach derives categories after examining document – emergent coding Factors in latent meanings of words or messages Context and proximity suggest the intended meaning of a message
Media: Qualitative Analysis Narrative Analysis (focus on literary composition and structure) Rhetorical Analysis examines properties of text; emphasis not on meaning, but how meaning is conveyed) Discourse Analysis – examines manifest language in text – language and word usage Semiotic Analysis – focus is on latent meanings and deep structures Interpretive Analysis – formation of theory from observation Conversation Analysis – used by social scientists in psychology, communications and sociology Critical Analysis – analyzes media messages (how are groups represented in media of all forms)
Consistency within Diversity The definitions, uses, and techniques vary according to whether the researcher conducts a quantitative study or a qualitative study Content analysis stipulates three core conditions for qualitative or quantitative character of study These central conditions include: Objectivity System Generality (Holsti, 1969)
Objectivity (Holsti, 1969) The test of objectivity is whether another researcher, following identical procedures with the same data, can arrive at a similar conclusion Objectivity, in content analysis, means that explicit rules and procedures guide the research process For example, what criteria will be applied in formulating a category? What criteria determines content unit (word, theme, webpage, animation, imagery, kinds of imagery, narrative, dialogue, space, advertisements, maps, transactions)? Objectivity requires researcher explicitly state ground a category is selected and what rationale prompts one inference rather than an alternative
System Systematic procedures facilitate objective, unbiased selection of content and categories (Holsti, 1969) Assigning firm rules to regulate inclusion & exclusion of content and categories prevents researcher from biased, self-serving selection Rules preclude an analyst from intentional or unintentional bias in selecting only content or category supporting hypothesis, proposition, or self-interest.
Generality Select content based on clear definitions for determining content and categories – fosters objectivity Establish procedures for inclusion or exclusion of data -- contributes towards consistency and systematic, impartial selection Promote principled, legitimate descriptive analysis with systematic procedures – provides a basis for generalizing about the data collection Relate data to an established theory or reasoned proposition for data to be conclusive or meaningful
Arguments Refined Quantitative and qualitative methods are different approaches to investigating same problem Must content analysis be defined by quantitative or qualitative methodologies? Holsti (1969) holds that other measures besides quantitative frequency counts & enumeration valid Alternative method of quantifying data with greater refinement is “contingency analysis” Contingency analysis codes presence or absence of a trait within document or unit of sampling. Inferences derived from proximity of two or more units of analysis in sampling unit The use of quantitative procedures or qualitative is not an either/or issue -- they can interact Restriction to quantitative analysis has been criticized on several grounds Such restriction leads to bias in the problems to consider It is also stated that qualitative methods can result in more meaningful or descriptive inferences
Manifest and Latent ContentConsiderations: Must content analysis be restricted to manifest content – or surface meanings; elements “physically present & countable” (Neuendorf, 2002 p. 23) Holsti offers compelling example of limits of manifest content. Limit to manifest content would not yield this notion Describes patient in mental hospital who states he is in Switzerland. In this case, the patient’s meaning of Switzerland is “freedom.” By stating he is in Switzerland, he is stating that he wants freedom. Latent content & meaning would provide the context whereby the receiver of the message would understand the idiosyncratic meaning. Latent content need not be dismissed. However, as Holsti emphasizes, it should be verified by “independent evidence.”
A Growing Trend During early 20th century Max Weber applied techniques of content analysis to journalism when he introduced the notion of studying the sociology of the press (Weber, 1924) Over 24 years -- 1971-1995, Riffe&Freitag (1997) noted six-fold increase in number of content analyses appearing in journal, J&MC Quarterly Software applications and the WWW enable text & data analysis Neuendorf (2002) reported more than 24 text content analysis programs are available (THAT WAS 9 YEARS AGO) Many electronic content analysis applications now available – text, audio, video
Reliability Reliability measured basically in two ways (Stemler, 2001) Stability – (intra-rater reliability Does same coder get same results again and again? Reproducibility – (inter-rater reliability) Do different coders, rating the same text, get the same results?
Internal Validity Internal validity is degree a procedure of measurement represents the defined concept Does the procedure measure what one intends to measure? Research question determines what one intends to measure In previous example, the concern was how individuals in majority population and minority population were characterized in fictional stories carried by popular magazines Study valid if results of inquiry yield data related to this concern and research question
External Validity External validity or the ability to generalize to the wider population, depends largely on the soundness of one’s sampling technique and sample Need experimental controls or validity becomes serious concern Triangulation or independent corroboration is one means of establishing validity Another involves conducting same inquiry using another method of inquiry – formal experimental design employing surveys or qualitative procedure with focus group/interviews
BibliographyAllport. G. (1965). Letters from Jenny. In K. Krippendorff& M. Bock (Eds.), The contentanalysis reader (p. 28-37). London: Sage. Print.Holsti, O. R. (1969). Content analysis for the social sciences and humanities. Reading, MA: Addision-Wesley Pub. Co. Retrieved from http://www.questia.comKrippendorf, K. (2003). Introduction to Content Analysis. CA: Sage. PrintKrippendorff, K. & Bock, M. A. (2009). The Content Analysis Reader. CA: Sage. Print.Neuendorf, K. A. (2002). The content analysis guidebook. London: Sage. Online resource: http://academic.csuohio.edu/kneuendorf/contentReitz, J. M. (2004). Dictionary for library and information science. CN: Libraries Unlimited.(Print).Riffe, D. &Freitag, A. (1997). A content analysis of content analyses: Twenty-five years of Journalism Quarterly. J&MC, 74, 4, 873-882.
Bibliography (continued)Stemler, S. (2001). An overview of content analysis. Practical Assessment, Research and Evaluation, 7 (17). Retrieved from http://PAREonline.netWeber, M. (1924). Toward a sociology of the press: An early proposal for content analysis. In K. Krippendorff& M. Bock (Eds.), The content analysis reader (p. 9-11). London: Sage. Print.Weber, R. (1990). Basic Content Analysis. London: Sage. Print.
Thanks very much!Florence M. Paisey, April 2011