Metcalfe critique2


Published on

Published in: Education, Technology
  • Be the first to comment

  • Be the first to like this

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide

Metcalfe critique2

  1. 1. CRITIQUING ARTICLES by Mike Metcalfe ©June 2002“New Perspectives provides new knowledge…” [Churchman 1979] 1
  2. 2. CONTENTS• Preface: Why this book came about, and how to critique it.• Why Critique? Why Articles?• Simple Argument Perspective• Problem Solving Systems Perspective• Picturing Perspective• Things, Cultures and People Perspective• Perspectival Thinking• Empirics as Action Perspective• Metaphoric Perspective• Dialectic Perspective• Evolutionary Perspective• Power Perspective• Fuller Argumentation Perspective• Conclusion 2
  3. 3. PrefaceGood critiquing requires insight, asking thoughtful questions and seeingbehind the text. How can someone practice this art? This book aims toprovide some pointers. Having been involved with critiquing articles withsystems thinking PhD candidates for a number of years, I have attempted toexpand on the ancient, yet wonderful, “What is their argument, and is itjustified?” as a critiquing method using perspectives such as systemsthinking, evolution theory, picturing, dialectic and the perspectival thinking.My perspective on critiquing articles comes from trying to develop ways toappreciate, and if possible solve, human activity problems. Learningcritiquing skills is one approach. The way to judge this book is in terms ofyour own thinking; critique your own learning. The book aligns with C. WestChurchman (ex editor of the Journal of Philosophy) that learning is aboutfinding new perspectives, to give you more choices. I hope this book givesyou new perspectives and new choices.The book is laid out in 11 main chapters, each with a page or two explainingthe perspective followed by a set of sample questions that may help yourthinking about whatever article you wish to critique. Regards, Mike Metcalfe, June 2002Further advice can be found at the web site of “The Writing Centre” at Harvard University.They, understandably, focus on extracting a justified argument1.“The Writing Centre” atHarvard University, The word argument is being used here in the sense of a one line conclusion, proposition, conjecture orclaim that needs to be justified by supporting evidence. It is not a quarrel, but well reasoned, nor purelogic but structured conversation. 3
  4. 4. WHY CRITIQUE?Why do I use the word ‘critique?’I am not assuming the word ‘critique’ has an immoveable definition. I amusing it in the sense of ‘constructive evaluation’ or ‘literary review’, hoping toinclude concepts like, ‘lessons learnt’ and ‘how it might be done differentlynext time’. I have chosen it over words like ‘review’, ‘analyse’, ‘evaluate’ and‘discuss’, because I find them too vague or too mathematical. This booklet ismy explanation of ‘critique’ by example.In terms of process, I am assuming undertaking a critique involves reading anarticle, using the questions listed in this book to think of an overall argumentyou wish to make and writing it into a ‘critique essay’ which itself has a veryexplicit, well justified, conclusion (the argument).Why undertake a critique?The intent of a critique is for learning to take place, yours and, wherefeedback is possible, the author of article. So, the way you go about critiquingan article needs to include methodically opening up the way you think. Thismeans two ‘learnings’ need to take place. One is about the article beingcritiqued and the other is how best to undertake the critiquing process. Thisbooklet provides examples for beginners but you should develop your owncritique perspective and resulting set of questions. Alternatively you maythink about how the questions can be used. By improving the critique processyou will increase or get a better understanding of what you learn from thearticles. WHY ‘ARTICLES?’The critique methods mentioned in this booklet can be applied also to books,chapters, lectures, courses, projects, human inquiry, human activity problems,and policy. The term ‘article’ was used only for brevity. 4
  5. 5. THE SIMPLE-ARGUMENT PERSPECTVEJustificationThis style of critique draws on the ‘argument as inquiry’ perspective, which inmodern times is attributed to Popper’s [1963] book ‘Conjecture andRefutations’, Perelman’s [1989], ‘The New Rhetoric’, Walton’s [1998] ‘TheNew Dialectic’ and to van Eemeren’s [1987] the ‘Handbook ofArgumentation’. By argument it not meant formal logic nor quarrels butrather reasoned debate where an article is required to have an explicit upfrontconclusion (the argument line) that needs to be justified with supportingevidence. The article should lay out this evidence in the form of reasoningand/or empirics (descriptive or measurement). Counter arguments need tobe anticipated and satisfactorily dealt with otherwise the conclusion will notbe convincing to the critiquer. Critiquing an article using this perspectiveinvolves evaluating the argument and supporting evidence.Start by looking for the one sentence argument (conclusion, point) of anarticle – it should be presented in the abstract, the introduction and theconclusion. The conclusion is often the safest place to look. Care needs to betaken to compare the explicit and implicit argument. The authors may clearlystate an argument but you may finish reading the article under the impressionthere was an alternative implicit argument. The argument also needs to havesome surprise value, be a little insightful or, as Popper argues, be risky,falsifiable. Arguing the sky is blue would not be very insightful, arguing thatit was red in the early stages of the earth’s development might be. Theinnovation may be in the evidence. If you had novel evidence that the skywas blue, that may be convincing. An article should not fail to convince you that its conclusion is justifiedmerely because of poor definition of some key words. Technology, sociologyand medicine are disciplines that have developed their own extensivevocabulary. History has not. It is the job of the author to communicateclearly with the intended audience, so an article can be criticised if it uses ill-defined terms. The greatest danger occurs when a word has several meaningsand the reader is not alerted to that which the author is using. For example,the word ‘critical’ means negative, exact, nuclear, urgent and emancipatory.Scientists have spent many centuries defining their ‘technical ‘ terms but insocial inquiry an author may need to spend some space defining, boundingand contrasting terms and concepts.A reasoned argument needs at least two people. With article critique, thiswill be the author(s) presenting his or her argument to you. The arguers needto introduce themselves, their expertise in this area, their motivation for 5
  6. 6. writing the paper, the motivation for why you might want to bother readingthe article and to acknowledge they are presenting a justified conclusion. Thisbackground may assist in your acceptance of their evidence. It should alsoassist you in anticipating where their evidence may be weak.There are two basic types of research articles, the discovery story and thejustified conclusion. People who watch the New York TV series “Law andOrder” will appreciate the difference. The cops discover who did it, thelawyers convince a jury. Analogous to the lawyers, in the justified conclusionstyle of article, the argument is presented upfront so you (or the jury) candecide if the evidence is convincing or not. Analogous to the ‘cops’, in thediscovery style, very common in scientific writing, the conclusion may bepresented as a discovery only at the end of the article. This seems a little bitmisleading if the author knew this ending before he or she started writing thearticle! It is really just a style thing, as the cop story, like the scienceexperiment, like the ‘who-done-it’ novel, is still trying to convince you whodid it. But be careful of the lure of the detective novel. Weick (ref or who heis) warns that most academic topics are so boring that the audience leavesbefore the ending. A good critique will discuss argument style – the lawyerstyle.Having located the argument (conclusion), the next part of the argumentcritique method is to evaluate the supporting evidence. There are also two (Ithink) types of evidence; reasoning and empirics. The classic use of reasoningevidence is Einstein’s [1939?] ‘Theory of Relativity’. He argues the presenceand importance of relativity by using only reasoning. It is in the form ofasking you to imagine you are in a moving train and throw an object out ofthe window. The path it appears to take is compared to the path it wouldappear to take if you were standing watching the train pass-by. The othersort of evidence, empirics, involves perceptions that have passed through thesenses of the author, typically his or her eyes, sometimes as measurement,sometime as years of experience. Of course, an author can call on theexperiences of other writers and their reasoning through the use of ‘theliterature review’. The quality of evidence is not a simple or absolute thing.Science likes very exact measurement, social inquiry likes real insight and totreat the collection of empirics (experience through the senses) as a learning-by-doing action that assist the brain to find insights. The only suggestion Ihave to help you decide whether or not the evidence is adequate is to simplyask if it would be convincing to a knowledgeable audience. 6
  7. 7. Argument QuestionsAsk yourself the following questions about the article.Argument • What is the explicit or implicit argument (conclusion) of the paper? Was it stated upfront? • What was their insight, ie was the argument novel, risky, open to falsification?Definitions • Are all key words well defined (described)?Arguers • Who are the authors? • Have they established their expertise? • Why have they selected this particular argument?Evidence • What evidence is brought to support the argument (conclusion)? • Was this evidence convincing, novel, insightful? • Was the counter argument fully considered? • Were there any empirics? If so, why? Should there be?Audience • Who is the intended audience? • Is the paper explicitly persuasive to this audience?Motivation • Is the importance of the argument fully explained? • What was the problem? • Is it an important problem? • What did you learn from the article? • How could you use it to improve people’s lives? 7
  8. 8. THE PROBLEM-SOLVING-SYSTEMS PERSPECTIVEThere are numerous species to this now very diverse genus, systems thinking.The species I am interested in seems to be traceable in the US to Boulding’s‘General Systems Theory: the Skeleton of Science’ [1954], Churchman’s ’TheDesign of Inquiry Systems’ [1971] and his students Mason, Mitroff, Ackoffand Ulrich who have produced work similar to Weick and Argyris. On theother side of the Atlantic, Checkland [2000] strengthen the meme with his softsystems. The important point being that these writers see systems thinking asa broad scope, fairly generic approach, to appreciating human activityproblems. An article can be seen as a human activity problem and its critiqueas a problem solving exercise. If the article is about research into somephysical problem, like the overall mass of the universe, rather than a socialone, like saving the environment, the systems thinking critique method maybe less appropriate.While still a developing concept, my interpretation of systems thinking forsolving social problems is that it encourages these types of problems to beseen using five constructs. These include purpose, interconnectivity,connectivity, boundary, seeking new perspectives and learning from doing.These problem solving constructs can be used to critique an article.PurposeOne distinguishing factor of a human activity is that it is purposeful. All thestakeholders are going to give it a purpose. Students may give a Universitythe purpose of getting an education, the lecturers of being able to learn moreabout their discipline and the administrators to processing student numbers.The construct ‘purpose’ is used in systems thinking in place of the morecommon term ‘outcomes’ as it the there can be a range of acceptable outcomesthat are still compatible with a single overall purpose. For example, studentscan select different courses as they learn from doing courses but still achievetheir purpose of getting an education.In a critique, the construct, purpose, can be used to ask questions like, whatpurpose did the authors have in writing the article? What purpose will thecritiquer give to the paper? Will it merely be a stepping stone to aqualification or will it inform their inquires. If the authors are participating inthe article, then what was their purpose in participating and what purposewill they give to the article? Moving on to the flexibility assumption inpurpose, how else might the authors’ purpose have been achieved?ConnectivityA human activity system is made up of interrelated sub-systems and is itself asub-system of something else. One of the core constructs of systems thinkingis this appreciation of hugely complex, impossible to predict, inter-connectivity of systems- this is the context of the system under consideration. 8
  9. 9. The article has been written with the wider international academic articlepublishing system, including the subject discipline system. Adding one morearticle to this complex of inter-related systems, or taking one, out will havehard to predict ripple-effects throughout this wider system.The connectivity construct encourages questioning about what the article isconnected to, for example, its place in the literature. “What else has theauthor done?” “Is the conclusion unique?” “What does the paper remindyou of, how does it sit with what else you know?” “What other evidence isavailable?” “What ripple effects will it have on wider systems?” “What doyou see as the wider system on which it will have the largest impact?” Putanother way, “What genre is it from, what inquiry tradition, what school ofthought?”BoundariesWhile systems thinking encourages a more holistic approach to problems,what separates it from the universality of science is that lets people putarbitrary boundaries around problem domains. In this case it may be thearticle, the discipline literature or the research methodology. Boundaryquestions may include, “Iis the article complete?” “Are all the issues andconcepts raised well defined and scoped?” “Does it present a self-containedstory?”Learning from ActionThinking of something as a system rather than as a thing encourages thinkingabout changes over time. What comes into the system, is processed, and whatgoes out? This is sometimes called the transformation construct in systemsthinking. It should encourage temporal (time) thinking such as, “Does thedate of the article matter and what is the pattern of events that lead to thisarticle?” “What does this article change in my thinking (lessons learnt)?”Internal to the article the processes might be questioned. “What were theinputs to the article? What skill level was required of the reader? What data(empirical and previous literature) was the article founded upon? Does thearticle process these inputs well? Then what are the outputs of the article, theconclusion and recommendations? Are they fully justified?”Stakeholders’ PerspectiveThe last construct of systems thinking for solving human activity problemscan be very challenging to some people’s definition of knowledge. Newknowledge can come from new perspectives. Put another way, humanproblems are often the result of different stakeholders having very differentperspectives on the truth of the situation. Neighbours can disagree over thesuitability of a fence. Both opinions are fully justified, errors of fact can besorted out but differences of opinion cannot be sorted out by measurement,no matter how exact. This leads to the suggestion that problem solvinginvolves seeking the perspective of all stakeholders. “Does the article give all 9
  10. 10. stakeholders a voice; or rather does the author talk for his or her subjects?”Historical articles about warfare are a good example here. “Are theperspectives of both sides, the leaders and the lead given?” To avoidrelativism, these perspectives need to investigated and fully justified. 10
  11. 11. Systems QuestionsAsk yourself the following questions about the article.Purpose • What purpose has the author had in writing the article? • How else might the author’s purpose have been achieved? • What purpose will the critiquer give to the paper? • What purpose do you think those participating had? • What purpose will they give to the article?Inter-Connectivity • What is the article connected to? • What is its place in the literature, disciple or topic? • What else has the author done? • Is the conclusion unique? • What does the paper remind you of, how does it sit with what else you know? • What other evidence is available? • What ripple effects will it have on wider systems? • What do you see as the wider system on which it will have the largest impact? • What genre, inquiry tradition, and school of thought is it from?Boundary • Is the article complete, does it present a self-contained story? • Are all the issues and concepts raised well defined and scoped?Transformation (Action and Learning) • Does the date of the article matter? • What is the pattern of events that lead to this article? • What does this article change in my thinking (lessons learnt)? • What were the inputs to the article? • What skill level was required of the reader? • What data (empirical and previous literature) was the article founded upon? Does the article process these inputs well? • Are the outputs of the article, the conclusion and recommendations fully justified?Stakeholders’ Perspectives • Does the article give all stakeholders a voice, does it seek the perspective of all involved? • Are the stakeholders’ perspectives critiqued and/or justified? 11
  12. 12. PICTURING PERSPECTIVEThis approach to critiquing an article involves drawing a picture of theproblem represented by the article. You may want to explore different waysof visually representing articles; one possibility is presented. It is based onCheckland’s [2000] ‘rich pictures’. The usefulness of picturing is usually inthe process which encourages appreciation of the problem in the article ratherthan the finished picture being a useful means of communicating with others.However, picturing done as a group, say using a white board, ‘… improve(s)communication amongst people in the problem situation and build a deeper,more shared understanding’ Bronte-Stewart [1999].An ExampleThe example of the “quarrelling bus conductor” will be used. This is aproblem described by Ackoff (2000; where a dispute hasarisen between bus conductors and bus drivers as each tries to maximise theirconflicting bonus scheme.1) Identify the problem the article is addressing; this may involve restating the conclusion of the article. A problem is the gap between what is happening and what is sought. The problem makes up the picture’s title, in this case, the quarrel between the conductors and drivers.2) Draw a large circle to denote what you identify as the system that bounds the problem, eg. the bus transport system. Write in the purpose of that system, eg. cheap, reliable transport around town. Bus transport system PROBLEM: The Conductor and Driver Cheap are reliable quarrelling. transport3) Draw as nodes around the circle, the stakeholders, e.g. bus driver, bus conductor, passengers, inspector and so on…4) Put in the various stakeholders’ main concerns as text in their thought bubbles. 12
  13. 13. Trip Driver Clock check conducto Bus transport system home, r work Passengers Cheap reliable transport Inspector Conductor Collect 5) Draw as nodes on the circle the objects or things involved in the problem, e.g. bus stops, the bus, the depot clock, and so on… 6) Draw as nodes the organisations (cultures?) involved, eg the Unions, the Bus Company and the Local Government. 7) Ask yourself what tensions can you identify from the picture? Draw as a line between the nodes any tensions that you feel are relevant between the nodes, eg the inspector is checking up on the conductor; the driver wants to get back to the depot on time; the conductor wants more time to issue tickets; passengers don’t want to stop between their stops and the conductor wants fares from the passengers. In an academic article, containing an experiment, the tension lines may represent validity or reliability issues. 8) Ask yourself how these tensions might be reduced? driver I’m on the clock mu not st mis council tick s 13 ets Bus transport systemunionsconductor Conductor Buscustomers Co.
  14. 14. home 14Trip Clock Purpose = cheap reliable … tor conduc watch
  15. 15. THINGS, ORGANISATIONS AND PEOPLE PERSPECTIVEThere has been a long tradition in dividing the world of human activities intothree, the things (T), organisations (O) and people (P). Linstone, [1984, 1993,1999] sometimes with Mitroff [1993], has expanded on this TOP approach,providing a useful way to critique articles. The complication is that TOP canbe perceived a few different ways.One perspective of T O P is that it represents different ways of knowing(epistemology). T represents the rational scientific way of definingknowledge as seeking objective facts, now associated with positivism andobjectivism, which typically means quantification, analysis, reduction ofproblem into parts and calls for repeatability. O and P can be combined torepresent a way of knowing that defines new knowledge as seeking newperspectives. It is an approach identified by Kant, Popper, Kuhn, Churchmanand many other theory of knowledge writers. When critiquing, this can beused to ask if the article assumes new knowledge to be found by seekingobjective, usually measured facts. Alternatively it may be seeking newperspectives.The TOP division can be used to appreciate that people (P) have differentperspectives on problems depending on their values (ethics) and experiences,each possibly available to learn from. National, departmental andorganisational cultures (O) can over-ride these personal perspectives assituational responses overwhelm personality. It is assumed that things (T)don’t have a perspective; but some writers, especially in history pieces, cantry to pretend there is no perspective being taken but merely THE truth isbeing presented. This may be labelled a T perspective. For critiquing, thismeans that you can reflect on which perspective the article has been writtenfrom, and how it would be different if written from another perspective. Forexample, the space shuttle Challenger disaster can be presented as amechanical (T) failure, or as a failure of a few individuals (P) making baddecisions or as an organisational culture (O) issue with managerialismoverriding the engineering culture. This example brings out the importantdifference between an organisational perspective and the personal one. Thedecision to launch is made by a spokesperson representing the organisation,so can be seen as a personal. However, the organisation as a level of system ismore than a collection of individuals if you believe as a group certainemergent properties exist. Look for group effects like synergy and emergentproperties in your article. 15
  16. 16. T.O.P QuestionsAsk yourself the following questions about the article. • Is the article about people, organisational cultures or things? • How would it have been different if it were from a different perspective? • Have the different perspectives of the people and/or cultures being provided? • Have the people or cultural norms been treated as ‘things?’ or as intelligent and experienced people who can inform the author? • Have all stakeholders had a voice? • Has the author treated the problem addressed in the article as one that can usefully be solved by using scientific methods, ie taken a technical perspective? If so: o Is the problem usefully subdivided into parts that can be measured? o Is the situation repeatable so the measurements can be confirmed? o Is it realistic to exclude any variables that have been excluded? o Which stakeholder is to judge the solution to be valid? o Is the evidence provided direct empirical or experiential? • Has the author treated the problem addressed in the article as best being solved by appreciating the perspectives of the stakeholders (P)? If so: o Do you get to hear their perspectives in their own words? o How were opinions justified? o Was the author cynical of whether the stakeholders understood their own minds or actions? o Was any confirming evidence sought? o What did stakeholders think of other stakeholders’ perspectives? • If a cultural norms perspective is being taken, is there a clear distinction between cultural and personality? For example, o Does the author assume the culture is more than the personality of a few dominant leaders? o Are the emergent properties of the culture identified and relevant? o How universal is the culture, are there contradictory sub cultures? 16
  17. 17. PERSPECTIVAL PERSPECTIVESGeneralising from the TOP approach, the more generic perspective draws onthe work of Kuhn’s [1970] ideas of theory laden observation, Popper’sexpectations [1963], Churchman [1972] multiple inquiry methods, Linstone’s[1999] multiple perspectives concepts and Haynes’ [2000] PerspectivalThinking which draws on Polaymi, Hegel and Heiddegger. The underlyingtheory of knowledge is that it is useful to identify two types of knowledge.The scientific type that produces ‘objective facts’ is well enough known. Thealternative is that knowledge is perspectival. New knowledge means findinga new perspective. Kuhn uses the word ‘paradigm’ to describe this.This dual knowledge approach believes that it is informative to separateinquiry into the ‘thing’ being inquired about from how it is being perceived.To take a simple example, I might study an organisation (the object) from amanagerial efficiency perspective, a learning perspective or as a source ofreliable income (3 possible perspectives). Critique involves attempting toseparate these two types of knowledge, the thing being studied from theperspective. Perspectives Object Mode of InquiryTwo broad types of perspective have been identified. One is like that used byLinestone [1999], Kuhn [1970] and the Critical Social Theorist. It is externallyprovided. For example, Linstone suggests engineers use his TOP to remindthemselves to not only think of their normal technical (T) perspective, butalso to think of the social (O or P) issues related to whatever project they areworking on. Kuhn, and many others, argue that the dominant scientificperspective is imposed without discussion. At one time, it was a clockworkuniverse, then it was one full of attractive forces, now it is a relativistic one.Another example, one from the social inquiry, is Critical Social Theory whichadvocates seeing human activity through an emanicipatory perspective. Sowhen critiquing, you can ask if the author is trying to suggest a newperspective.The other sort of perspective is more implicit and personal. Unprompted,people perceive new problem situations in different ways depending on theirpast experiences and their values. Terms like real interests, worries, theoriesof action and people’s concerns align with this concept of an implicitperspective formation that make us pre-judge, or appreciate situations in 17
  18. 18. certain ways. A critique might try to identify the concerns or primaryperspective of the author (or maybe some participants in the article). 18
  19. 19. Perspectival QuestionsAsk yourself the following questions about the article. • What is the article about, what is the thing, the object under consideration? Think of this thing as being a system and ask yourself the system critique questions: o Are its boundaries well defined, what can it be contrasted with? o What is its purpose? o What does it change over time? o What is it connected to? o Who are the stakeholders? • What perspective is being taken of the object under study? o What are the origins of this perspective? o Is it an external perspective or a concern? o What are the limits of this perspective? o Is it part of a wider perspective? o What other perspectives might have been taken of the object under study? 19
  20. 20. EMPIRICS-AS-ACTION PERSPECTIVEThere appears to be some very separate understandings about the role ofempirics in research. Some philosophers and critical theorists seem to thinkempirics are more trouble than they are worth, rarely using them in research.Reasoning and thought experiments are considered sufficient to create usefulknowledge. Measurement problems, the lack of reliability of the humansenses and a lack of explanation of why the observation occurred contributesto this attitude. Moreover, Marx’s, who has influenced many social theorists,has a particular position on empirics [Sowell, 1985, chp.2]. While Marx calledhis work empirical, he was not interested in the mere appearance ofsomething. Rather first it is important to understand the underlying processesin tension going on behind the appearance of something. For example, amarried couple can be living, eating, sleeping together but not be happy withtheir relationship, to understand the marriage you need to understand theunderlying tensions. The same is true of a caterpillar; you cannot explain theappearance of a caterpillar unless you understand it is about to change into abutterfly. Put another way we have to learn to be able to see. A baby has tolearn which sets of colors and shapes makes up a tree and a surgeon has tolearn he or she is seeing when he/she opens up a body. Social researchershave to reason what is going on behind an observation. This can be used tocritique empirics by asking what underlying tensions has caused the thingyou are looking at to be there as it is.This lack of centrality of empirics to research can also be found in some areasof science. Einstein’s work was mainly mathematical and mind experiments(anologies). His 1939 book only draws on analogies as aid to reasoning. Howthe author perceives or uses empirics provides a perspective with which tocritique an article. You can ask whether the author explicitly uses reasoningor empirics as evidence to convince you of the conclusion.However, in the positivist tradition, empirics are central, especially theexperiment. So you can ask if precision of measurement is consideredinsightful observation. Does the author display observation or measurementskills greater than a lay person or common sense? If so, then these are thehall-marks of the positivist scientist. Further, does the style of the authorsuggest that they are reporting the truth not a perspective, and that there isonly one correct perspective? This is usually associated with the assumptionthat the observations can be repeated to get the same result and the observeris independent and unbiased.The American Pragmatism approach to empirics seems to be slightlydifferent. It sees empirics as problem solving by action, an experience, whichenables the brain to draw on more analogies. The emphasis is not on seekingindependent objective observations but to improve reasoning from action,including think of different ways to perceive the problem. This emphasis has 20
  21. 21. been taken up in the action science, action research and learning from doingliterature. In particular, Weick [19??] Argyris and Schon [1996], Ackoff [2000]and Checkland [2000] have taken it up as a problem appreciation methodcharacterised by the learning or reflective loops approach. It advocateslearning from a series of small trial and error learning loops rather than the‘one big plan’ approach. Look for these assumptions in the empirics in anarticle.Moreover, if root cause or double loop learning is to take place, then theeffects of the action (eg collecting empirics) need to be reflected upon both interms of the object being studied and on the perspective being taken of thatobject (see perspectival thinking). So, if an organisation is the object, theperspective is adaptability, and the action (node of inquiry, the empirics) isthe introduction of a new information system, then the empirics need to beused to revise our understanding of both organisations and adaptability. Thisintroduction might be undertaken in small stages so that reflective loops onboth of these factors can take place. When critiquing an article look forempirics being used to provide actions to reflect on an upfront expectation. 21
  22. 22. Empirics QuestionsAsk yourself the following questions about the article.Why Empirics? • Why does the article have empirics, why does it not? • Are the empirics intended to provide objective knowledge or merely to assist thinking? • Could any empirics be replaced with an analogy, reasoning or mind experiment? • Do the empirics convince you of the articles conclusion any more than the reasoning evidence did?Underlying Tensions Empirics • What are the underlying tensions or forces that produced the ‘thing’ you are looking at as it now appears before you?Empirics for objective knowledge • Are the authors seeking the one truth or do they acknowledge alternative interpretations of the empirics? • Do the empirics produce convincing, objective knowledge? • Are the empirics repeatable? • Are the empirics generalisable to many other situations, will they remain valid across time and universally around the world? • Are the empirics all the evidence that is provided towards the conclusion? • Is there any acknowledgement that empirics are ‘theory laden’?Empirics for learning from action • Were the actions that produced the empirics seen as an exercise in learning from action? • Was there any attempt to undertake a series of small actions and reflect on each so as to redirect future actions to collect empirics? • Is there any evidence of the author or others changing their perspective as a result of the empirics? • Have all stakeholders been given an opportunity to provide their perspective on the actions that created the empirics? • What other modes of inquiry (action) might have provided other learning? • Was there any reflection as a result of the actions against the perspective (intellectual frame, theoretical construct) informing the empirics? 22
  23. 23. 23
  24. 24. ROOT-METAPHORIC PERSPECTIVEMorgan [1986], in his ‘Images of Organisations’, classified the literature onorganisational theory by root metaphor, a concept discussed earlier by Pepper[1942]. His book suggests that, when people wrote about organisations, itwas possible to identify certain common root metaphors. For example,Taylorism assumed a machine root metaphor making people like cogs in thatmachine. Another example, those in education often complain about, is theassumed root metaphor by some education administrators that knowledge isan object that lecturers pass over to students for money. He identifies anumber of these assumed root metaphors, suggesting it is a useful critiquingexercise to see if you can spot the root metaphor in an article. Social scientistsoften use a root metaphor that the subjects of their research are less intelligentthan themselves. Johnson and Lakoff [] provided a popular discussion onmetaphors while Ortney [] discusses how metaphors affect our thinking.There was a flood of papers that appeared in the 1980’s and early 1990’strying to identify root metaphors in various human activities. It is hard toprovide exact direction on how to spot root metaphors. All I can think tosuggest is that you read through the article and write down any metaphorsyou can identify. From this list ask yourself if there is a collective name, themeor root metaphor that could be used to sum up the list. For example, I recentlyasked some fellow academics for metaphors of a literature review. Theirsuggestions are listed below. I have separated them into two as I felt therewere two root metaphors.Group 1These metaphors of a literature review have been allocated to the rootmetaphor of ‘investigation’ (seeking out, discovery, detection, journey).Undertaking a literature review is a search for (in) the unknown, going out tofind out something you don’t know. • map ("go where you like") • detective (seeking clues for new research directions) • beachcomber (sifting through others castoffs) • prospector (seeking gems or nuggets). • lens (to focus readers), • pathfinders (layout an area and point to key sources) • sifting and winnowing • funnel (funnelling in) • signpost (to find your empirics) • concertina (narrowing and enlarging your search, like a concertina windbag) • focus mechanism (you are talking about this, not that) • releasing the imaginationGroup 2 24
  25. 25. These metaphors of a literature review have been allocated to the rootmetaphor of ‘prosecution’ (justification, evidence, proof, argumentation). Theliterature is used to support a reasoned argument, justify a conjecture, orvalidate a hunch. • expert witnesses (courtroom evidence... ). • currency (to buy credibility) • building blocks, • concrete foundation, • the history, • mirror (to see oneself in context) • requirements document • credibility filter (whose work do you draw on, what is the source of your ideas?) • due diligence (have you acknowledged others whose ideas you draw on?) • social courtesy (can you be relied on not to steal ideas?) • situational context (what conversation is this paper a part of?) • family tree • foundation (you are educated and building on a tradition) • puzzle (your idea fits into this larger puzzle) 25
  26. 26. Metaphor Questions:Having listed the metaphors in the article ask yourself the followingquestions about the article in order to find a root metaphor that suits you.People: • Are the people in the article treated like ‘things’ that need to be organised or as sources of knowledge? • Is any human activity treated as though it occurred in a linear sequence (from A to B to C) rather than in a recursive, or interactive one? • Are the suppliers of capital assumed to be superior to people investing their time (labour)? • Is gender considered relevant? • Does the article take a personal perspective or organisational one? • Is technology assumed superior to humans?Organisations • Are any organisations presented as machine like? • Are any organisations like adapting, evolving organisms interacting with their environment? • Are any organisations liberating or oppressive to stakeholders? • Is there any indication of the dependency or organisations on the broader community? • Are any organisations depicted as flows of information rather than people interacting? • Are the interpersonal and/or interdepartmental power issues made explicit? • Is efficiency and effectiveness a communal problem, not related to the desires of particular stakeholders? • Is Government treated as more powerful than the market place?Things • Do any machines have gender, or are they analogous to any body parts? • Do problems exist separately from the people who have the problem? • Is information and/or education treated as a commodity? • Is equal access to information assumed? 26
  27. 27. DIALECTIC PERSPECTIVEThe word ‘dialectic’ is being used here in the same way it is used in theorganisational change literature [Nielsen, 1996; Mason 1996; Morgan 1986] torefer to “change that emerges from the interplay of conflict and amongdifferences and affirmation of areas of agreement”. Nielsen [1996] identifiesfive types of dialectic processes which he explains using terms like ‘iteration’,’spiralling’, ’up building’, and ‘transformation’ which can be involved whenthere is a social reconstruction of human activities. He argues (in a dialecticwith the reader?) that these change processes differ in terms of emotionality,they do not have to be quarrelsome, aggressive or conflictive but they can be.Identifying these types of dialectic can form a further method of critiquingarticles.In ‘interaction dialectic’, an idea (conjecture) is constructively refuted with theintent of developing an improved idea. The action science dialectic isanalogous to Argyris and Schon’s [1983] idea of experiment, reflect, andexperiment learning loops. The up-building dialectic is similar to the ‘notinvented here syndrome’ where one human group develops a culture of howchange should occur that an outsider does not respect. There is then tensionbetween these two groups. For strategic dialectic, Nielsin draws on the workof Mason [1969] who suggested that alternative plans of the future can beused to develop a synthesis or improved third plan.The transformation dialectic is about the underlying social and political forcesbetween group vying for resources. It is hard to mention this version of‘dialectic’ process without mentioning the hugely influential work of KarlMarx. Sowell [1985], a Marxist economist, argues that their inquiry approachwas to consider the ‘underlying dialectic forces’. In their case, this wasmainly the underlying political and social forces associated with the classstruggle. Nielsen identifies more ‘localised’ struggles that go betweenparticipants of human groups engaged in a common activity. For articlecritique, this can be translated to suggest you think about the underlyingtensions inherent in the situation outlined by the article. 27
  28. 28. Dialectic QuestionsAsk yourself the following questions about the article.Iterative Dialectic • Does the author systematically question the article’s conclusion? • Is the author suitably suspicious of his or her own conclusion? • Is there a specific perspective guiding this suspicion? • Are improvements to the ideas under suspicion being suggested?Action-Learning • Was there any action undertaken simply as a lets “try it and see what happens”? • Were any empirics (research) seen as a learning-from-doing exercise? • Was there a full and systematically post-mortem of all actions? • Was there any response to unanticipated results? If so, did it lead to learning?Up-building • Is there any suggestion of tradition or culture about how the article was developed? • Was there any consideration of how ‘radicals’ or ‘a devil’s advocate’ would have responded to the activities depicted in the article? • Have all the possible responses to the article been given a voice?Strategic • Does the article set up any sort of dialectic between two alternatives? • Do they get equal consideration? • Can you think of an alternative that might (also) be set to what is used in the article? • Is there any evidence of the article trying to learn by setting up an interaction between two alternatives?Transformation • Can you identify any underlying tensions between human groups vying for resources? 28
  29. 29. EVOLUTION-METAPHOR PERSPECTIVEThis perspective draws on the metaphor of an article being a species, abiological entity. More specifically, an article is to be thought of as thereproductive young of some species; something’s baby. The perspective isinspired by the work of Darwin, Dawkins [1989] and Dennett [1996]. Thebasic tenet of evolution is that the young are a random genetic variation onthe genetic possibilities of the parents. The environment at the time of itsgrowing to maturity determines whether the variation is able to reproduce itsparticular attributes to another generation. In most species, many more areborn than reproduce. The appropriate level of analysis is the gene pool or thesurvival of the species not the individual. Birth and death is seen as beingessential to improving adaptability to the environment.The article can be thought of as the newborn. The parents being the majorinfluences on the author, the genus being the school of thought the articlecomes from. The degree of variation is how different the article is from theparents, who may or may not admire their offspring’s features, but theenvironment will decide success, whether later authors are influenced (cite)by it. Publication may be analogous to birth; working papers and drafts canbe seen as failures to reach birth and the environment the audience. Somespecies can alter their environment in order to survive, others cannot. So,when critiquing an article, examine the references etc 29
  30. 30. Evolution QuestionsAsk yourself the following questions about the article. • From the references can you identify the genus of the paper? • From the references can you identify the parents? • Would the parents have admired this article or not? • Checks the citations index, do later authors use this article? • Were the authors influential enough to alter their environment? • Would the environment in which this article was published be hostile, friendly or manipulated by the article? • What environment might have treated the article very differently? 30
  31. 31. POWER PERSPECTIVEBoulding [19??] talks of the three faces of interpersonal power; economic,destructive and integrative. Economic power comes from being able toallocate resources, destructive power comes from being willing to notparticipate in some activity, and integrative power comes from manipulatingthe human need to belong to social groups. Critical Social Theory [Alvessonand Skodberg, 2000] focuses on institutional power. Members of institutionshave power from their formal positions. Journal editors accept and rejectcertain research methods, politicians have the power to influence legislation,the police from having to process law-breakers, doctors by having patientsseen as victims, scientists by determining what is good knowledge andteachers by claiming to be more knowledgeable than their student. Markus[19??] identifies power from technical knowledge that can be extended toknowledge over regulations and processes (bureaucratic power). Thefeminist movement has been very effective in increasing awareness of theimplicit power given to certain groups through language and Kuhn talks ofthe power of paradigms inside which people act while not appreciating manyof their own assumptions.These perspectives can be used to critique articles, look for the implicit powerand persuasion in an article both between the author and the reader but alsobetween any participants in the article. Be emancipatory, help thedisadvantaged to see the causes of their stress, look behind the curtain ofinstitutionalised power, assume there are no tall poppies, but rather somegroups of people exploiting another group of people. 31
  32. 32. Power QuestionsAsk yourself the following questions about the article. • Who determined that the topic of the article is important? • Do the authors use their position in an organisation, or superior insight, to convince rather than using reasoning or empirics? • Is the journal pushing any particular topics of research or research methods? • Is science assumed to be the only reliable source of knowledge? • Are all contributors to the article’s conclusion acknowledged? • Does the field of study have a small group of ‘experts’ who gatekeep on journal article acceptances? • Are there any rival journals that intentionally take an alternative approach?Within the Article • Does the article reveal any struggle for resources from the point of view of all those involved? • Does every point of view get a voice? • Is the language exclusive, acting as a barrier to outsiders and the application of common experience? • Are there any appeals to authority? • Are references justified in terms of what the author’s studies or on the basis of quality of journal? • Are all the counter arguments fully explored? • How would low-income people view the article and its contents? 32
  33. 33. FULLER ARGUMENTATIVE PERSPECTIVEThe author has a dominant perspective that argument is the basis of humancommunication, the creation and testing of knowledge. This perspective hasbeen well argued by Pereleman, van Emmeren, Walton and Rehg. Moreover,I need a way to wrap up all the diverse critique perspectives presented in thisbooklet! Therefore, this last section will attempt to combine all theperspectives presented. The umbrella perspective will be simple argument,with the others providing detail to expand this critique perspective. 33
  34. 34. Argumentation QuestionsArguers • Who are the authors? • What is the basis of their expertise? • What else have they published?Audience • What is the intended audience? • What audience would it not work for? • Is the paper explicitly persuasive to this audience?Object or Thing Under Study • What is the object or ‘thing’ the article is studying? • Did they discuss, define this ‘thing’ in system thinking terms; its boundaries, relationships (context), how it changes over time, and its purpose?Concern • What is the authors’ concern (perspective, lens, frame) on the objects under study? • Did they scope this concern or perspective? • What is the problem they seek to address?Argument • What is the explicit or implicit argument (conclusion) of the article, was it stated upfront? • What was their fresh insight, i.e. was the argument novel? • Can you draw a picture of the article?Definitions • Are all key words well defined (described, bounded, scoped)?Motivation • Is the importance of the study fully explained? • What is the purpose of the authors writing the article?Evidence • How were they (implicitly) defining knowledge? • How did they know things? • What evidence is brought to support the argument/conclusion? • Was this evidence convincing, novel, insightful? • Was the counter argument fully considered? • Where there any empirics if so why, if not why not? 34
  35. 35. Empirics as Learning • Were they treating empirics as an exercise in precise measurement to produce objective knowledge or as action to learn? • Did they build and test something, what and how? • Was the testing integrated in the building and evaluated using learning loops and multiple stakeholders’ perspectives? • Was the evaluation convincing? • Were all stakeholders given a voice? • Were they searching for ‘the truth’ or a range of perspectives?Implications • Were the implications (the so-what) of accepting their argument/conclusion fully explained? • What good knowledge was created? • What actionable knowledge (rules of thumb, recommendations) was created to aid future decision-making?Lessons Learnt • What did you learn from the article? • What purpose do you give to the article, how could you use it in your own studies?© Mike Metcalfe, June 2002 35
  36. 36. REFERENCESArgyris C. and Schon D.A. (1996) Organisational Learning II, Mass: AddisonWesley.Arygris C. (1996) Actionable Knowledge, Journal of Applied Behavioural Science,32(4), 390-BouldingBronte-Stewart, M. (1999), Regarding Rich Pictures as Tools forCommunication in Information Systems Development, Computing andInformation Systems, Vol 6, pp83-102.Chalmers AF 1982, What is this thing called science, U of Queensland PressCheckland, P., (2000), “Soft Systems Methodology: A Thirty YearRetrospective”, Systems Research and Behavioural Science, Vol 17 Number 1, ppS11-S58,Churchman, C. W., (1971) The Design of Inquiring Systems, Wiley, NewYork.Churchman C.W. (1979) The Systems Approach and Its Enemies, Basic BooksCohen H.F. (1994) The Scientific Revolution, University of Chicago Press.Dawkins, R. (1989), The Selfish Gene, Oxford: Oxford university pressDennett D.C, (1996), Darwin’s Dangerous Idea, NY: TouchstoneDillon, J., (1988), “Questioning in Science”, in Meyer (ed) Questions andquestioning, Walter de Gruyter & Co, BerlinEemeren, van F.H., Grootendorst, R. and Kruiger, T., (1987), Handbook ofArgumentation Theory, Dordrecht: Foris Publications.Haynes J, 2000, Perspectival Thinking, NZ: OneIdea CompanyJohnson and Lakoff …Kuhn T.S., (1970), The Structure of Scientific Revolutions, 2nd Ed., University ofChicago Press.Linstone, H. (1984) Multiple Perspectives for Decision Making: Bridging the GapBetween Analysis and Action, Elsevier Science Pub Co, NY.Linstone H A. [1999] Decision Making for Technology Executives : UsingMultiple Perspectives, Artech House Publishing: BostonMarkus L. (19??) … powerMason, R O [1969],“A Dialectical Approach to Strategic Planning,”Management Science, Vol 15, B-403 – B-414,Mason, R.O. and Mitroff, I.I. Challenging Strategic Planning Assumption, JohnWiley and Sons, New York, NY, 1981.Mason, R O [1969],“A Dialectical Approach to Strategic Planning,”Management Science, VolMitroff, I and Linstone H, The Unbounded Mind, Oxford University Press, 1993.Morgan G., (1986), Images of Organisations, Calif.: Sage Publications.Ortney… 36
  37. 37. Perelman Ch. and Olbrechts -Tyteca L., (1969), The New Rhetoric: A treatise onargumentation, University of Notre Dame.Popper K.R. (1963) Conjectures and Refutations, London: RoutledgeRehg W., (1999) Argumentation Theory and the Philosophy of Science SinceKuhn, Inquiry, SummerSchon, D. (1983), The Reflective Practioner, NY Basic Books.Sowell T (1985), Marxism, London: UnwinUlrich, W. (1983) Critical Heuristics of Social Planning, Wiley, New York.Walton D., (1998), The New Dialectic, Toronto: Tronton University Press. 37