UPDATE 30 September 2015: This blog post has been expanded and developed in Shaowen Bardzell and my book, Humanistic HCI (Morgan & Claypool, 2015), in “Chapter 4: Enacting the Struggle for Truth in Full View: Writing and Reviewing Humanistic HCI Research.”

* * * * * *

Here is my thesis, so if you skip the rest of this post, you at least know what I want to say: if you peer review an “argument” paper for HCI this year (also known as an essay), whether or not you agree with it doesn’t matter very much. The ability of an essay to compel agreement is far lower than, say, that of a scientific report. And thus the contribution of an essay can’t inhere in its ability to compel agreement; it must be elsewhere. In this post, I define what an essay is and does; I argue that constructive disagreement is at the heart of the essay and humanistic dialogic methodologies in general; and finally I reject and then propose some norms for evaluating argument/essay paper submitted to HCI venues.

I begin with the essay itself. Essays are a common form of discourse in which the author(s) attempt or try (hence the etymology of “essay”) to think deeply and reflectively about a problem, theory, concept, etc. That is, essays do not present themselves as offering the truth or of being correct, but rather are the product of a skilled and personal struggle with a difficult issue of interest. A significant portion of original scholarly work in the humanities takes an essay form, including philosophy, literary and other forms of criticism, and ethnography, among others.

And even a very passing familiarity with essays in any field reveals something fundamental to them: almost no one ever agrees with them (including their authors themselves some years later). I’ll share an example or two just to show what I mean.

The great 20th century aesthetic philosopher Monroe C. Beardsley wrote a book, Aesthetics: Problems in the Philosophy of Criticism in 1958, which for the next several decades, was a primary source of intellectual target practice in philosophical aesthetics. For example, philosopher Noël Carroll begins his own theorizing of aesthetics by summarizing Beardsley in order to refute him, a habitual strategy not just for Carroll but for his mentor, George Dickie:

my use of Monroe Beardsley, in this essay and others, as my leading foil also shows the influence of George Dickie, since it was Dickie who taught me always to consult Beardsley’s work for the most worked-out and authoritative position on any subject in aesthetics, even if, in the end, I wound up criticizing it.” [Carroll, Beyond Aesthetics, p.2]

The benefit of Beardsley’s work is not its ability to compel agreement, but rather because in Beardsley one finds “positions” that are “worked-out and authoritative.” To say Beardsley doesn’t compel agreement is an understatement, because Carroll and Dickie are both quite clear in disagreeing with Beardsley, who rather plays their “foil.”

Beardsley himself also exemplifies this form of constructive disagreement. In a lengthy 1981 postscript to his earlier book, Beardsley confronts the decades of work that critiqued various aspects of his 1958 book. He makes no effort to defend the earlier book as written, but rather acknowledges specific arguments against his positions that have made him go back and revisit his earlier ideas, and he also critiques these intervening works on their own terms. A typical (in the sense that one can find literally dozens of analogues in the book) rhetorical positioning is this one (note that I’ve redacted the intellectual substance to foreground Beardsley’s rhetoric):

In his complex and idea-rich book, The Mind in Art, Guy Sircello has expounded and defended an original form of [Aesthetic Concept X]…. The key to his theory is the concept of [Aesthetic Concept X(1), which Beardsley explains and illustrates with an example]. Thus Sircello has called attention to an important and neglected feature of artworks, and with numerous and telling examples. The doubt that lingers with me, however, has to do with [Beardsley offers his own criticism of Aesthetic Concept X(1)]. [Beardsley, Aesthetics: Problems in the Philosophy of Criticism, 1981 ed., p. xlii]

For Beardsley, Sircello’s probing interrogation of the concept, and its application to numerous examples, is a significant philosophical contribution, even though he doesn’t agree with it.

I could illustrate this point ad nauseam for another hundred paragraphs, but I hope I have been sufficiently persuasive that I don’t need to. Philosophy (and other humanistic fields) proceeds dialogically, that is, through a living constructive dialogue, not through incremental accretions on a broadly accepted core, as “non-revolutionary” science is generally understood to do. The result is the development of a library of positions on a collection of issues of interest, a library whose quality is measured in credible diversity (it’s not pure relativism!). This is somewhat different from a scientific paradigm in which there is broad consensus on the types of research questions, methods, data, and findings that are considered legitimate and useful at a given time.

I’d like to return now to reviewing argument/essay paper for CHI and other HCI venues now. I am arguing that we should not use the following normative criteria to judge argument/essay papers:

• Do I agree with the essay (or argument paper)?
• Is this argument convincing?
• Is the author fully justified in making that argument?
• Did the author cite every author or theory that I would have, were I making this argument?

Instead, I would propose that we consider some of the following alternative questions, the affirmation of each of which in some way and degree contribute towards a positive reviewer judgment of the paper:

• Is a coherent (if not ultimately “correct”) position put forward, that is, a clear and non-universal point of view, perspective, or authorial voice?
• Is the position put forward interesting to [some relevant portion of] the HCI community? Is there any practical benefit for researchers or designers in HCI to have this sort of position available in our collective library?
• Is the treatment of the position authoritative, that is, well informed, substantially developed, and insightful?
• Does the position make a controversial concept, theory, or perspective more plausible or at least better worked out than it had been before (even if it ultimately fails to persuade)?
• Does the position challenge your own thinking productively? Or how people commonly think in [some relevant portion of] HCI?
• If you are inclined to formulate a response (e.g., by expressing how you disagree with this position), does your response push you to clarify your own positions and thinking in original ways (even if you disagreed with the position)?
• Does the position offer the possibility of a new way of thinking about a relevant problem?
• Does the position provide new resources, such as new concepts, technical definitions, canonical examples, categorization schemes, and/or distinctions that make our work as researchers and designers more productive or valuable?
• Does the position shed new light on familiar examples?
• Does the position reveal the importance of, or give us new reason to look at, hitherto overlooked examples?
• Does the position propose a new way of categorizing or classifying examples, and is the categorization itself interesting or important independent of what the position specifically does with it?
• Does the position help you or [some portion of] the HCI audience appreciate something of interest to the community as more complex, more successful, more problematic, etc., than it is normally considered to be?

When I think of the best humanities scholarship, it is far more likely to have affirmative answers in the second list than in the first. So too with the best HCI essays: including many (though not all) of the papers of the “X Considered Harmful” variety, Dourish’s “Implications for Design,” McCarthy & Wright’s book length essay Technology As Experience, Grudin’s “The Computer Reaches Out,” Bødker’s “When Second Wave HCI Meets Third Wave Challenges” to name just a few.

If we use the first list as our primary reviewing criteria, we are likely to stifle this sort of position-taking. Going back to the Beardsley and Carroll examples, it is quite clear neither thinker ever would have gotten off the ground were it not for constructive intellectual disagreements with other works that they had read: their disagreements were generative of new thoughts, not indicators of the unpublishability of the earlier writings!

Let us avoid a circular firing squad and evaluate the distinctiveness of the essay as offering a position, featuring depth and quality of discriminations, authoritativeness, insights, and speculative reasoning, rather than the less interesting and less significant question of whether or not you agree with it.

UPDATE: I forgot to make clear the obvious corollary: just because you agree with the point an essay is trying to make is not grounds for accepting it for publication. Think about it this way: do you want your own position to be represented poorly in the community and for your hard work to be condemned by association with it? It’s better to help the authors develop the position you all agree on so that it is as strong as it can be.

ANOTHER UPDATE: It is possible to read this post as defensive or complaining about how the HCI community “just doesn’t understand us.” That was not my intent. First of all, CHI now explicitly embraces the argument (so yayy!–no cause for complaint there), but we don’t, as a community, have much experience evaluating this type of contribution, so I offered this post in service of that goal. Second, there are already lots of beautiful and influential essays at CHI (again, yayy!). Third, many early-career HCIers have expressed a desire for practical reviewing guidance, which I view as a good thing, so this post is meant to offer just that. Finally, and perhaps most importantly, I love good essays (even ones that I really disagree with), and I want to share and cultivate that passion within my own community. So the spirit of this post was positive and passionate, not whiny or cranky. Sorry if my high spirits didn’t come through, but they were there.

FURTHER READING: This is a continuation (and application) of some of the thoughts I developed in this position paper on the role of the essay in HCI, which in turn was based on this blog rant.