Published on

Computer scientists tend to reach first for usability testing and clinical evaluation when assessing hypertexts and hypertext systems. Other methods and approaches may yield better judgments

Published in: Technology, Education
1 Like
  • Be the first to comment

No Downloads
Total views
On SlideShare
From Embeds
Number of Embeds
Embeds 0
No embeds

No notes for slide
  • My name is Mark Bernstein. I build hypertext systems, and I’m a hypertext publisher. This paper reviews approaches to hypertext criticism. The talk is challenging; the paper is easier.
  • The paper takes a sincere shot at broadly surveying hypertext criticism and the many approaches people have taken to figuring out what is good and what is not. I believe this to be the first review of the area.
  • A big part of what we do in hypertext research is, simply, figuring out whether the stuff we’ve done is any good.

  • Those 93 references aren’t padding. Lots of people have reviewed hypertext. I disagree with some, but read and form your own opinions. The strength of this conference had been its scholarship, and that scholarship is not what it was. You need to know the literature.

    If the idea of literary hypertext seems strange to you, Diane Greco and I recently pulled together a volume of readings about Reading Hypertext.
  • We make judgments when building our systems, and when writing hypertexts, and when designing Web sites, and when submitting papers to this conference.
  • Literary machines inhabit a literary economy. I outline the shape of that economy in the paper. You know the essence: Writing, publishing, bookselling.
  • Everyone thinks their own work deserves to be rewarded. There is nothing outside the economy; open source is not a stairway to heaven. And we claim to be a science; we want evidence and we’d like proof that we know what we know.
  • I won’t recapitulate the whole discussion of the economy. The most salient fact about books is that books are numerous. And their plenitude is precisely what we want to preserve and expand in hypertext; we want to build things that are better than books.
  • How do we know what we know? Most of you are computer scientists, and computer science has an answer. Some of you may be writers, or scientists (who after all are writers) and Literature has its own answers.
  • I want to suggest today that it’s all wrong. We’re rejecting work that might be good, and accepting work that might be wrong.
  • Choosing unwisely is an expensive proposition.
  • All of us have sometimes reviewed conference papers that were, frankly, hopeless from the outset. Sometimes, we’ve heard them presented. Maybe this is one of them.

    A bad conference paper costs taxpayers and benefactors fifteen or twenty thousand dollars — and that assumes it’s not pernicious or misleading.
  • Of course, the costs can be much greater. We take a paper that’s not QUITE right, and people rely on it, and cite it, and pretty soon we’ve got entire research projects that are motivated by a mistake.

    This has happened several times at this conference, where (after all) we don’t build bridges.
  • When we misjudge research and writing — in either direction — we tempt practitioners to despise all research. Readers assume that we can’t advise them. Worse: that we don’t want to. The trade press fills up with corrupt pseudo-analysis. And students stop doing the work. THIS SHOULD SOUND FAMILIAR TO YOU.
  • We need to know what it to read, and we need to know how to write hypertexts worth reading. How can we tell? The favorite tool of computer science has been empirical evaluation.

  • We implement a system of some sort, and then we test that system on actual test subjects.
  • We can, for example, take a print story and turn it into a hypertext. We ask a bunch of our students whether they like the jumbled story or the real story. They give us the answer we want; and we get a publication.
  • Drag races between media are almost always a bad idea. You can’t measure the average productivity difference between an opera and a bronze statue. WE HAVE UNDERSTOOD THIS SINCE 1991, and yet we see these papers every year.
  • Evaluation is sometimes useful, but often leaves us adrift. First, it privileges first and casual encounters, which are not really our first interest. Second, the really important outcomes —changing the world — are by definition rare, especially with undergraduates! Finally, we keep defining “work” as “finding facts” or “filling out forms”; this is easy to measure but not in fact what knowledge workers do.
  • We ourselves don’t do evaluation very well. I review a paper with this sentence almost every year. So do you. I’ve always thought this was self-evident grounds for instantly rejecting the work. Not all of you agree, though I’ve never heard a convincing defense.
  • Let’s face it: we insist the papers have evaluation sections, but we don’t love reading them. When we meet, we don’t rush to talk about our latest tricks for recruiting test subjects. It shows in the papers: the statistical sections are often ill-written, stilted, and artificial. And it’s equally hard to get referees to pay close attention.
    We insist that it be done, but it’s obviously not what we want to do. This is not determinative, but it is (I think) a hint.
  • Summarizing quickly and oversimplifying: evaluation keeps us from deceiving ourselves, but it threatens to turn us into a conclave of pettifoggers, refining details that nobody cares for.
  • A variation that designers sometimes hold out: we’ll ask the users what they want.

    Readers dont know what they want. They haven’t found it yet; that’s why they’re reading!
  • So, perhaps there might be an alternative to applying Student’s t-test to a room full of our students.

    Others, instead, have turned to careful examination of actual hypertexts.
  • There’s quite a lot of excellent work along these lines, examining actual hypertexts and exploring what they do and how they do it.
  • Notable among this group was a cadre of young European and Oceanic scholars, many of whom, interestingly, now write much less than they did ten years ago. This is a problem.
  • Criticism can potentially answer any question we care to ask it. We are inclined to distrust it, but often find it gives the right answers. But our distrust is not unsound; partiality can be masked, and induction from a single sample is hard to trust.
  • Another approach — one that has been very popular amongst our literary colleagues — has been to explore the essential properties of digital media.
  • Discussing specific hypertexts, one by one, is hard work. Can we draw any lessons, beyond respect for isolated genius? It seems more attractive to identify general tendencies in the nature of hypertext.
  • Often, these take the form I’ve called “Kids Today”, deploring the ever-declining abilities of the latest technology-addled teens.
  • This never changes and has no content: its essence is that it is – precisely – essentially irrelevant.

    There is better work essentialism, but it has proven surprisingly difficult.
  • In his “Golden Age” speech. for example, Coover retreats from a modernist examination of actual hypertexts to argue from the Web’s apparent propensity for superficial imagery. He has no particular evidence for this, and I think it was wrong, but since Coover says it can’t be done, a generation hasn’t tried very hard to do it.
  • Or take Hamlet On The Holodeck. Murray says digital environments are procedural, participatory, spatial, and encyclopedic. But this isn’t an empirical observation; it’s an argument from the supposed essence of computation. This is fine when the essence is necessarily true, as is is for Joyce in “Nonce Upon Some Times”. But here we can easily quibble: is “Lust” encyclopedic? Is “afternoon” procedural? Is Prolog?
  • Finally, an important thread of essentialism begins from the conclusion that new media are pernicious. Since hypertext is harmful (especially to kids), these critics tend to avoid reading much of it.
  • I’m being hard on essentialism here. In fact some of my essentialist colleagues have done some hard work, and we have learned something. But I think essentialism can take a vacation.
  • Essentialism is popular in those parts of the digital humanities that like to talk about the essence of the digital. It’s largely discredited, and computer science it is especially unpopular.
  • The reaction against essentialist modernism in the arts is called postmodernism, and it is with this reaction that hypertext is chiefly identified in the humanities.
  • It was Kael's therapeutic advice to the overcultivated that if they just concentrated on responding to the stimulus, the aesthetics would take care of themselves. What good is form if the content leaves you cold?
    The academic term for the kind of antiformalism Kael promoted is "postmodernism."
  • Formality considered harmful. Spatial Hypertext. LITERARY MACHINES. All postmodern arguments.

    But, absent form and power, it hasn’t been clear how Theory (with a capital T) can be wrong on the merits, not just the politics.
  • A number of critics have embarked on a reflective practice, essentially auto-ethnography. They read and, reflecting on their reading, try to figure out what’s happening in front of them.
  • We can record our reading of a hypertext, our reaction to it, our theories about what is happening. Jane Douglas pioneered this method, Jill Walker won the first Nelson award for it; we don’t do it anymore.
  • The same thread runs through much of the best new media criticism: looking both at the work and at its experience. Think for example of Marshall’s thoughtful studies of annotation, Efimova’s reflections on blogging about writing a dissertation on weblogs.
  • Autoethnography — which is to say, intelligent and critical reflection on your own system — can yield intelligent reflection that would probably be beyond the capacity of a bunch of undergraduates working for a pizza. The potential for bias is clear. More important, I think we can sometimes find self-criticism too convincing. Randy Trigg talked us out of link types in 1989, and we believed him
  • Though I’ve been hard today on the Essentialists and the Evaluators, all four threads contribute to help us know what hypertext to read and to explain how it works. We can practice all four to good effect.
  • Let me conclude with some guidelines about how this new economy of judgment could be most helpful.
  • I return again to artistic and political practices of the late 19th and early 20th century radicals from Ruskin to the early Bauhaus, practices I call (without regard to nationality) NeoVictorian. The key concept that runs throughout these disparate movements is the integrity of the artist and the work. As a field, we have affected to honor these ideas and have not always succeeded as well as we ought.
  • First, we need to get our hands dirty. We need to read hypertexts and we need to read about them. We need to use hypertext tools in our daily work. Here, for example, are some notes on my recent reading from Tinderbox, a hypertextual tool I designed for making and analyzing notes. We should writing new media. We should demand excellent new media from students. You can (and must) understand computers now.
  • In my view, a great fault of digital humanities today is that there are few consequences (and many rewards) for being wrong. Indeed, being spectacularly wrong gets you press coverage and party invitations. Further, it is one thing to be mistaken and another, entirely, to be wrong and know it. In the sciences, we call this fraud; until you get this right, many will consider your discipline contemptible.
  • We do not wish to return Art to the patron, the prince, and the priest. It is easier for a critic to pin a fault on a student than to censure a colleague. It is also wrong. It is easy to assign nefarious motives to corporations and to sneer at tradesmen. Absent evidence, it is also wrong. Capitalism may be foul, but it is not our fault.
  • Convincing critical appreciations approach the hypertext in detail, yet draw on muriad sources and ideas. Sympathetic reporting of what was missed — what, as Harpold suggests, the hypertext turns away from — helps establish the critic’s position, and candor.
  • Failure can be as interesting as success. Many of the best readings of hypertexts begin from incomprehension and confusion. I hated afternoon the first time I read it; I thought hypertext fiction preoposterous.
  • The critic may be wrong. Take Lust: I edited the story, and I know it fairly well. I’ve read it dozens of times. Yet one of Rich Higasson’s students’ comments, reported in his dissertation, reveals my reading was wrong.
  • We should expect to write seriously about actual new media throughout our careers, not only as as the outset.
  • Special thanks to Eastgate editor Stacey Mason, whose survey of criticism formed the core of this review, and of course to all of you who build hypertext systems and write hypertexts. Where, by the way, are the hypertexts at this conference? Thank you.
  • Criticism

    1. 1. Tis hard to say, if greater want of skill Appear in writing or in judging ill. But, of the two, less dang’rous is th’ offence, To tire our patience, than mislead our sense. Criticism Mark Bernstein ❧ Eastgate Systems Inc. Hypertext 2010
    2. 2. The Paper “Our methods for accumulating and testing • 93 references evidence of a hypertext’s successes and shortcomings • [Aarseth 97]-[Zaid 03] are numerous but poorly understood. This paper • From 1711 to today surveys the most influential approaches to evaluating • From science to art hypertexts and considers their impact on crafting a • We should read more hypertext criticism new literary economy.”
    3. 3. Text Evaluation? ?noitaulavE
    4. 4. William McGregor Paxton, The Housemaid
    5. 5. Poets are partial to their wit, ’tis true, But are not critics to their judgment too?
    6. 6. Lewis W. Hine. Group of newsboys on a stoop at 4th & Market Sts, Wilmington Delaware. "Take our mugs, mister?"
    7. 7. Photo: Lee Russell An Economy of Judgment The disappearance of newspaper and magazine review Mark Bernstein And Diane Greco, “Designing A New Media Economy” Genre XLI 3/4, 2010
    8. 8. Lewis Wickes Hine. A "Reader" in cigar factory, Tampa, Fla. He reads books and newspapers at top of his voice all day long. This is all the education many of these workers receive. He is paid by them and they select what he shall read. There is a lot to read. The most salient economic fact about books: books are numerous.
    9. 9. How do we discover what we need to read? ✤ teachers, friends, colleagues ✤ booksellers ✤ reviews in newspapers, magazines, journals ✤ weblogs Roosevelt reading in front of his tent in hunting camp, 1910.
    10. 10. But there are lots of problems For example, small literary magazines receive thousands of submissions, but have hundreds of readers. Lewis Hicks Wine, Newsgirl, Park Row, New York.
    11. 11. Whoever thinks a faultless piece to see Thinks what ne’re was, nor is, nor ne’er shall be.
    12. 12. What Is Writing the paper: 80 Professor hours @ 100 $8,000 Presenting: 50 professors x 1 hour $5,000 Travel: $1000/ conference ÷ 18 papers/ conf x 50 $2,700
    13. 13. Bad research can be very expensive Tacoma Narrows
    14. 14. William McGregor Paxton, Leaving The More Costs Practitioners learn to despise all research Readers assume our work is “academic” and uninteresting The trade press fills with corrupt pseudo-research. Students leave the field
    15. 15. Evaluation Trust not yourself; but your defects to know, Make use of ev’ry friend — and ev’ry foe
    16. 16. Trust not yourself; but your defects to know, Make use of ev’ry friend – and ev’ry foe.
    17. 17. Miall & Dobson 2006 ✤ Take “The Demon Lover” ✤ Make it into a hypertext (how?) ✤ Ask a bunch of undergraduates to read it ✤ Run the statistics ✤ Write the paper
    18. 18. Moulthrop 1991 To require hypertext to function like a book is a bit like expecting a jetliner to behave like a locomotive: yes, it’s very fast but the blasted thing won’t stay on the rails.
    19. 19. Evaluation and its discontents Usability privileges the first encounter Statistics wash out the exceptional case (when not discarded as an outlier) Unrealistic definition of “work”
    20. 20. “While the results were not statistically significant, we note nontheless that the trend was strongly…”
    21. 21. We don’t Like It
    22. 22. Evaluation ✤ Strength ✤ Not easily swayed by our hopes and dreams ✤ Weakness ✤ Often silent on the matters that matter most
    23. 23. VARIANT: USER CENTERED DESIGN Lewis Wickes Hine, Current Education, 1912.
    24. 24. Who to a friend his faults can freely show, And gladly praise the merit of a foe. Real riticism criti Re al c of cism of r eal real xts! hype hyp erte rtex ts!
    25. 25. Who to a friend his faults can freely show, And gladly praise the merit of a foe
    26. 26. Who to a friend his faults can freely show, And gladly praise the merit of a foe. ✤ Strength ✤ Flexible, concrete ✤ Weakness ✤ Partiality can be masked ✤ Small samples, hard work
    27. 27. William McGregor Paxton, The Breakfast, Metropolitan n Museum Most critics, fond of some subservient art, Still make the whole depend upon a part Essentialism
    28. 28. Most critics, fond of some subservient art, Still make the whole depend upon a part.
    29. 29. It never changes slide 25 from my Hypertext 99 opening keynote
    30. 30. Coover & The Golden Age Hypertext is over; you missed it.
    31. 31. Murray “When we stop thinking of the computer as a multimedia telephone link, we can identify its four principal properties, which separately and collectively make it a powerful vehicle for literary creation. Digital environments are procedural, participatory, spatial, and encyclopedic.”
    32. 32. claptrap ✤ Essentialism run amok: the essence of computational media is that they are pernicious (and so we need not — should not — read them) ✤ Believing hypertext to be pernicious or harmful, these critics read little (Miller, Kakutani) or almost none (Birkerts)
    33. 33. William McGregor Paxton, The Breakfast, Metropolitan n Museum Strengths you don’t need to work very hard Weaknesses you don’t learn much Essentialism
    35. 35. Our sons their fathers’ failing language see, And such that Chaucer is, shall Dryden be.
    36. 36. It was Kael's therapeutic advice to the overcultivated that if they just concentrated on responding to the stimulus, the aesthetics would take care of themselves. What good is form if the content leaves you cold? The academic term for the kind of antiformalism Kael promoted is “postmodernism.” Louis Menand, 1995. Finding It At The Movies. The New Yorker. 42 (5)
    39. 39. Are we reading
    40. 40. Although people sometimes have a hard time deciding whether or not something is art, they are
    41. 41. STRENGTHS situated in intelligent the individual reflection WEAKNESSES
    42. 42. The NeoVictorian Critic Evaluation • Modernism • Essentialism • Postmodernism
    43. 43. In ev’ry work regard the writer’s end, Since none can compass more then they intend; And if the means be just, the conduct true, Applause, in spite of trivial faults, is due.
    44. 44. respect artist work integrity
    45. 45. Dirty hands
    46. 46. Consequences for being wrong
    47. 47. Indifference to persons and
    48. 48. o r i g R a i l et D
    49. 49. t r e rag a v i t n te s b n ain e s o io st
    50. 50. humilit y
    51. 51. doing the work: a commitme to criticism
    52. 52. thank you ✤ Pope On Criticism (1711) ✤ Stacey Mason, Samantha Panepinto ✤ Fonts: Neutraface 2, Epic, Kane, Tungsten ✤ Images: Library of Congress, iStockPhoto, MFA, Metropolitan Museum serious of Art. hypertext since 1982