This post is continued from two earlier parts:
- In Part 1 I offer a broad rationale for my position
- In Part 2 I offer specific recommendations for ACs and reviewers
In this part, I offer my own recommendations for the research community moving forward. I stress from the outset that this is a position. I do not attempt to speak from the Voice of God, but mainly from my own experience. I just want to stimulate a conversation; I have no ambition here of offering the final word.
My recommendations begin after the fold.
Recommendation 1. Treat ACing as explicit, rather than tacit, knowledge that is distinct from peer reviewing.
This recommendation is necessary because the community is growing and the number of submissions keeps increasing. We have to bring in new people, sometimes in large numbers. Rather than leaving it up to them to figure it out, or to rely on SCs who are both overworked and unlikely to spontaneously handle this in a consistent way across the board, let us make an effort as a community to articulate what we want from ACs so junior researchers have at least a model to hold themselves to. We teach our Ph.D. students how to do peer reviews. We need to teach junior scholars how to be ACs. I put my money where my mouth is in the previous part.
Such an approach could also help undermine the “Old Boy Network” that the CHI community (like any other community) sometimes slips into, because it would offer mechanisms to support the infusion of new blood.
Recommendation 2. Each major HCI subcommunity should consider explicitly articulating (and maintaining over time) its primary peer reviewing criteria.
What makes a good contribution to the Design subcommunity is likely to differ from a good contribution in the UIST or CSCW subcommunities. Yet reviewers and ACs (for good and desirable reasons) often cross these lines–but do they bring tacit norms with them?
Such an approach would also be effective for handling emergent scholarly practices. For example, for years a number of us have been vocally complaining about “opinion papers” as a category in HCI, where “essay” is what (presumably) was meant. Good news: the language surrounding the submission process for essays is greatly improved. Bad news: it is less clear how well the HCI community understands the epistemologies and norms of effective essays.
Recommendation 3. Ensure that adequate opportunities for scholarly dialogue exist surrounding our paper decisions.
I feel very strongly about this one.
Beyond reviewer-AC dialogue, this recommendation also advocates for author-AC dialogue and ideally author-reviewer dialogue. Because papers don’t just “speak for themselves,” and given the stakes involved in the decisions we make, if authors feel confused or wronged during the process, there should be some mechanism to express that and work it out in a reasonable way.
This one is extremely difficult to deal with, given all the constraints involved. But I can still make some recommendations here:
Recommendation 3a. Allow authors to respond to as many points (criticisms, suggestions for revision) as they feel they need to during rebuttals.
I know people disagree with me on this, and in some subcommittees it may be appropriate if all that is at stake are questions of fact. However, in my subcommittee, at stake are often judgments and interpretations and the reasoning and assumptions surrounding them. The 5,000 character rebuttal is insufficient to support the intellectual role it was intended to play. It’s also not clear to me that the persent form is better than nothing.
Journals in general and CSCW 12 have a process where authors typically submit a revision with a statement in a two-column format. In the left column is a quote or paraphrase of a particular reviewer concern, criticism, or suggestion. In the right column, the author responds to it and explains what will be/has been done about it. Such documents are arguably easier to read than rebuttals, and they structurally encourage people to stick to the issues, rather than expressing anger or frustration in a general sense. They also support quality revision, which is good for the long-term health of the discipline.
Recommendation 3b. Keep trying and iterating on the CSCW/UIST reviewing process from 2011.
In particular, the Revise and Resubmit process at CSCW was effective at supporting reasonable and, when necessary, lengthy dialogue between ACs, reviewers, and authors. And yet it was less work overall for ACs (at least, in my experience). Also: better decisions were made, and accepted papers had been much more thoughtfully revised than normal (at least in my experience). Now CSCW is having a banner year with tons of great papers that were substantially, not cosmetically, improved.
Recommendation 3c. Allow authors to communicate directly with their ACs.
Allowing this backchannel communication can alleviate many of the pressures on the rebuttal process, support productive scholarly dialogue, and lead to better decisions for now, and better papers down the line.
Recommendation 4. Find or develop mechanisms to enhance both reviewer and AC accountability–and make it part of institutional memory.
We could use both crude quantitative measures and also simple qualitative measures to evaluate reviewers and ACs and encourage high quality reviewing decisions.
Some crude quantitative measures include number of reviews done in the PC system, average word length of reviews, scoring patterns (reviewer average compared to paper average, which would help identify the Negative Nellys and the Pollyannas), number of times accept/reject recommendation agreed with final decision, timeliness of reviews, and I’m sure many others.
Some qualitative measures might include ratings of ACs by authors and SCs and ratings on reviewers by ACs and authors. These might include simple Likert scales that ask whether the reviewer understood the paper, whether actionable revision suggestions were offered, whether the rationale for the decision was explained, and so on.
Another idea is some sort of post-mortem, where SCs and ACs talk about their experiences–not just in a general way, but in an individual way. If an AC is a Negative Nelly or a Pollyanna, he or she might not even realize it, and yet I bet the SC generally knows.
As I get feedback and pushback, I may revise and/or expand this. Regardless, it is meant to be constructive and to start, not to finish, a conversation.