The New York Sophist

Or We Need Actual Peer Review, not this Monstrosity

In Jacques Ranciere’s incredible book The Ignorant Schoolmaster, he argues that the time of the Old Master is behind us. Instead, true education comes from equally exposed minds on a subject in agreement on mutual respect for the working of the mind. Through this matrix of mutual respect for minds able to figure out the world and offer reasonable and thoughtful answers, any subject can be learned. What strikes me about this model is the total lack of a bank or reservoir of information that must be conveyed across dangerous, difficult, or sometimes impassible terrain to get to the student. Although this is a powerful way to think about teaching, I’d like to start with Ranciere’s characterization of the Old Master as the problem with peer review. I am leaving out of this critique any peer review that is focused on experimental or methodological review – I think that has important reasons to exist. What I want to focus on is how throttling and destructive peer review is for things like writing about theory, doing criticism, or other sorts of scholarship based on interpreting not data but other texts. As Hans Georg-Gadamer asks, “Why must we always be clamoring after the social sciences to bulk up our research?” Peer review is a failed model, and it needs to be addressed in order to save the humanities and provide good and compelling reasons for the research university to exist.

What Peer Review Really Is

In the process of peer review, editors seek out an expert, or “the” expert on the thing that the essay is about. This expert will evaluate the manuscript to see if it conforms to the highest levels of scholarship in that niche area of the field that the person is writing about. However, this is decidedly not “peer” review. It’s “expert” review, coming from a completely different philosophy. The expert will decide if the manuscript “passes the test,” “conforms to the rules,” etc. in accordance to what this expert thinks the current state, rules, and scholarship is. This is an exam.

Furthermore, this is not a “review” of any kind but a challenge. There are numerous social media groups in the humanities that poke fun at “Reviewer Number 2,” an archetypical figure that is simply negative and not satisfied that the essay is as good as it would have been if they had written it themselves. Reviewer Number Two’s demands are extreme, ridiculous, and often pointing to their own insecurities about the field, offering a list of necessary citations that have to be included that is often longer than the piece itself.

This points to what is really happening – it’s not a review, but a challenge. It is an invitation to an antagonistic argument. The person chosen to review is not there to help the paper become better, but to point out its faults and flaws. The attitude of the standard peer review process is oriented around the question of expert quality. If there are gaps, flaws, or places where the argument is bad in the view of the so-called ‘reviewer,’ the paper should not be published.

As a moment of challenge, the reviewer becomes oppositional in a way to challenge the author(s) to make something stronger in their opinion, without any justification as to why the review is the correct, or right argument against it. Often reviews are written with a lot of questions in them in a spirit of trying to collaborate with the author(s), but this always gets read by editors and authors as questions that must be answered in order to get the paper published. The reviewer takes on the role or the attitude of being the most difficult, hostile, and thorough audience you could imagine. This sort of audience is one that would be identified by Chaim Perelman & Lucie Olbrects-Tyteca as a “vanguard” audience; one that stands in for ‘real’ audiences by being smarter, better, and more invested than any actual audience would be.

An actual Peer Review System

Opposed to this would be actual peer review, where someone at the same level as the author(s), not an expert but interested; not a master of the subject matter but someone familiar with it, examines it to see if it is something that they think other people would benefit from reading. This benefit would not be based on the manuscript meeting a “stable-state” metric, but something controversial. Instead of meeting extant metrics, the peer notices that the conversation opens up, becomes more foggy, more messy, and more of an invitation to engage rather than an attempt to shut down. Actual peer review (as opposed to ‘Peer Review’ in the received view of it) opens up conversation rather than trying to shut it down. It makes you want to engage with the work rather than be the defining moment of that field and that work.

Instead of an adversarial challenge from an expert, the peer reviewer takes on the role of an interested collaborator, reading the paper as an invitation to join in on the argument. Instead of being a hostile audience who is the most rigorously demanding sort of opponent you could muster (a vanguard guarding the gates), the attitude would be one of someone who is a potential collaborator, a fellow organizer, someone who identifies some common ground and wants to help you build something meaningful.

What I would want out of a peer review process could be summed up in the attitude of the co-conspirator. There’s a loose plan, it has a good aim, and I have figured out some of, or even most of, what needs to be done. In a spirit of trust I show the plan to someone who I think has similar political views as me; they are at least at the same parties I go to. Through that sharing, I get a shift in attitude: The person I approach identifies with my plan and wants to help construct it in various ways. They offer things that help it come to be. At some point they are not a critic, but still are offering criticism in the way a partner would.

This attitude can be invoked by communicating (!) to reviewers the attitude they should invoke as a reader of a submitted text. Instead of the current attitude – “this has flaws; as the expert I must point out the flaws; where there are no detectable flaws I must make the strongest argument imaginable that I could make against this paper,” etc. – an alternative statement could be offered. “Would this be something you’d like to co-author? Would this invoke more submissions to the journal? Would this circulate in conversation in a seminar/faculty group/among you and your friends?” I think what I’m going for is that peer review would accept pieces on their ability to stimulate generative text rather than the status quo, which is only accepting papers that are so airtight that one cannot find any air with which to utter anything about. That kind of stale perfection is death; we want to end conversations with our current attitudes about peer review.

Another missing element here is: Would this essay spark necessary question and conversation for graduate seminar pedagogy? I think graduate pedagogy is a vast, depressing wasteland for most. Maybe that’s my own experience talking, but imagine attending a mountain-climbing course where all you are presented with is video of people doing what looks like the impossible. There’s no discussion of their failures, their previous attempts, or even how they got access to the equipment and practice sessions that helped them discover how to summit a massive, hostile peak. None of that – just the images of success. Then the instructors say, “Ok this is what you are to do.” A lot of graduate instructors also are dishonest, and don’t talk about the luck and situational moments that allowed them to get a piece into a good spot to set everyone’s mind on fire. These pieces are the exception not the rule. What if we offered imperfect pieces simply because they would stoke responses? What if peers really did review the pieces, stirring the experts to respond? Articulation is a necessary force for argument: If we never have to re-articulate our positions when facing new contexts, new conversation, new proposals, we run the risk of our ideas becoming stale or worse – dicta.

One way around the problems created by the Old Master of peer review is to find another model – maybe Ralph Ellison’s “little man behind the stove” is instructive. The peer reviewer can keep out a weather eye for the authors in the terms of the question “are you treating the audience ethically, or are you speaking down to them?” An expert would never ask such a question; their loyalty, even in rhetoric surprisingly, would be to the purity of the thoughts, the theories, the ideas. Are these being appropriately cited and used? The little man asks the question “for whom?” on the end of that. And the reception of those ideas does not come through the predictable channels, as Ellison expertly argues in that essay of the same name.

There are a number of other areas to look for models for a review that brings up instead of shuts down pieces that are generative. It’s difficult to find a journal worth reading these days because in academic journals there are no conversations. The standards for peer review are Expert Opponent standards. There are no peers to be found and no review. There’s only a battlefield. Instead, let’s think about how to write an invitation rather than an indictment. The attitude should be one of welcoming in rather than keeping out. Conversation cannot exist where there’s no air. The cold lack of atmosphere in our journals shows this as a direct result of very solid, very good vanguard activity at the gate.

Actual peer review should draw upon the professions teacher side as well and establish a rubric rather than a gauntlet. If questions are proposed by journal editors for reviewers to answer that are specific and have different descriptions in rankings (just like a content analysis coding rubric, sorry Gadamer!) we might just be able to leverage the expertise of our field in more than just a hyperbolic, antagonistic way. We might be able to access the agonistic, the contest which, ironically, requires a lot of cooperation for the sport to be good. The contest is: Can I see this essay as having a hand in producing more valuable discourse about this topic? What could I suggest to add in order to ensure that this catches fire? These questions are in the spirit not the specificity of what would be needed but if you think that our journals are more important than our teaching (and most speech communication profs would find that a no-brainer) then where’s the peer review rubric?

A Call for Conversation

Let’s develop the attitude that the pages of the journal are community space, and we all benefit from having good and thoughtful engaging work there even if it is imperfect. In speech communication rhetoric (NCA focused mostly), it seems we are either building monuments or expensive condos, and not everyone can build on the extremely valuable real-estate of our ‘flagship’ journals. We want something that is sexy, lasts, and looks expensive. I think the aim is for a gated community, if there’s any aim at a community whatsoever. Where’s the public space? Where’s the place that someone can wander in and examine what we make?

The Old Master is alive and well in the world of humanities journal peer review. Instead perhaps we can all take a lesson from Ralph Ellison, and simply serve as the warning that the “little man behind the stove” is out there. We can help one another, as Ellison did, in addressing, as co-authors those who would not want our art in their communal spaces. We don’t have to take on the role of opposition, we can use opposition as a creative force to help one another say something in a way that encourages others to speak and write.

Vibrant conversation about importance that is interesting and accessible – and has a lot of interesting angles – is a great way to defend what it is the university does outside of “barely provide functional job skills at too high a price.” This rhetoric of the university and higher ed as a failing place is something we can directly address through the way we perform scholarship: An engaging argument that is valuable to have and performs that value in myriad ways. Waiting for an article to reach the standards of some Old Master’s view of perfection is not going to help. Who does that help? What does that system benefit? Vibrancy and recency, although not totalizing values, never appear. An actual peer review standard gets more flowing and produces more, calling attention to how experts disagree for audiences. It’s what the university should be about – a place of active and interesting inquiry, living and breathing, not a city of stone monuments that reference one another.

This September is finally starting to feel like fall and I’m experiencing quite a bit of nostalgia for my old debating life. The smell in the air and the quality of the light on campus make me anxious that have not booked a bus or hotel yet; that I made plans on a Saturday to do something in the city, and so on.

Habits are strange things that we often think of as bad, of breaking them before they break us. But there are habits of thought and feeling that are triggered by conditions that have little to do with our cognitive state or our intellect.

For example, I thought today how strange it would be to work on this campus and never interact with a student. How odd would it be to work at a college or university and not interact with students. Not out of choice, but because your job does not require any student interaction or meeting at all. Perhaps your job is one where you are not supposed to interact with students?

I have thought about leaving my university job more than many times. To do what, I am not sure. I really think the barrier is the incapacity I have to imagine doing anything in September other than standing before nervous students talking about the importance of speech. I would feel so lost and so confused by a September that lacked those things.

My immediate reaction to this thought is self-accusation: Why do you lack the capacity of imagination here? But perhaps that’s not the end point of where that question leads. Perhaps the self-accusation is realization that being in this position in discourse with students about, well, discourse, is not a lack of capacity but the root of capacity in imagination. Perhaps the classroom is my commonplace book.

“Several years ago you had my sister,” said the new student – but in a college classroom not a high school one. This is a commonplace for most secondary teachers: the announcement of legacy status. But in university this is unheard of. Perhaps I’m wrong about this, but it took me (a)back. “How is your sister doing?” A full report was filed and I wondered and resented the lack of change in my position in this dusty classroom hearing about all the change in the years that have passed between her and me and now.

I assign textbooks in September; I regret textbooks in September. Maybe the ultimate capacity of the self-accusation of lack is to embrace it fully and just go. There is plenty to talk about; what’s on your mind? What’s on their minds? If rhetoric is a thing as powerful as you write about, think about, and imagine it to be, then what are you scared of? What holds you back from just going in? Why all the accoutrement? Why the bumpers, why the railings, why the handholds?

The real trouble is thinking of an incapacity as the root of all capacity. That’s a tough order. Imagining an incapacity as inventive is really a stretch. But it isn’t really – what else is the point of a commonplace book in rhetoric other than to recognize – and admire – your incapacity to “say it just like that.” Collecting bits of information, quotes, and statements – images from others – is a good habit that seems as if it is there to remind us of what we cannot do. But the contemporary mode of the commonplace book – the “vision board” of the younger generation – is an inventional device that is created to help you imagine and work toward creating something that is within your capacity. The old commonplace book should be seen this way as well I think.

But habits are hard to break. I am grateful for that, particularly in September when the light has this certain motivating quality even though I really don’t have anything to prepare for it. The feeling of anxiety, then relief is good but nostalgic.

Attitude and Engagement at the start of the term

The start of the term means figuring out the archaeology of the class. This requires some digging, brushing and poking around to determine what substrate the students are in. Where did they come from?

The values of the students, their tools, their approach to learning is a collection of practices that developed over time in relation to an environment or environments. We would be remiss as teachers not to investigate that sediment for the things that worked for students in previous classrooms.

The metaphor opens up the idea that when students are acting in ways that frustrate you, or make you launch into a narrative of “how terrible students are today,” you can change that attitude and ask after what previous classroom experiences they had that made them act in these ways. It takes the blame off of your students and puts it on what we know to be an underfunded and misguided system of state requirements for moving between somewhat arbitrary grade levels.

Also shame on you if you blame your students as individuals for their classroom behavior and don’t consider what experiences they had in classrooms that might have led them to this behavior.

Approaching student behavior or questions or engagement with the “nuts and bolts” of the class (as a favorite history professor of mine used to call it) should be done with care. They are showing you valuable information as to what “survival strategies” and “technologies” of learning worked for them in the past. The blame is not with them, unless you want to blame them for making it this far.

The first few days of teaching are always very insightful as you can see the students trying to figure you out using the paradigms they have on file for every previous teacher. You can use this to your advantage to raise some incredible positions, questions, possibilities with the class as to what it means to sit in mostly forward-facing, plastic seats and face a wall for 90 minutes and listen to someone talk.

Today is day 2 of my classes for the most part, except for my terrible hybrid section (I despise hybrid courses but they keep giving them to me) which because of University scheduling will not meet in person till the 12th. I’m interested to see what bubbles up in the conversation today. What fragments of pedagogical engagement will come to the surface?

A great chat I had impromptu this morning with my friend and colleague Matt about one of my rhetoric lectures that I have given over the years at Cornell University. The version we are discussing is this one from about five years ago:

Even though I have several iterations of this talk available on YouTube, this one has the most views by a wide margin. Not sure why. Initially I thought it was the shirt. This doesn’t seem to be supported by any reliable data.

In the video above this one Matt and I talk about his recent viewing of the video and how he thought about it from the “meta” perspective: What is it a student can do with this lecture and what impact does it have (or should have) in the context of teaching. It’s a good conversation between us about the aim and purpose of a class lecture and what might actually occur. I am glad I recorded it because we have a very nice candid and rhetorically-oriented conversation about purpose, audience, attitude, and things of that nature. I’m very happy we had the chat and it was a nice stimulation to get refocused on teaching as the summer enters its final month.

Also spoiler warning: At the end we talk about He-Man: Revalations on Netflix so if you don’t want to learn about some of the more interesting elements of that show, stop the video when Matt mentions it.

This summer has been one of great travel and great ideas. I haven’t really been meeting my goals of doing regular videos or even regular writing, but June was a very concentrated moment for me in working through some nascent thoughts.

Here’s a talk I gave with Korey Stegared-Pace from Stockholm’s Microsoft Reactor. They focus on offering a lot of different helpful content for developers of all kinds. Korey covers the technical aspect while I consider some of the other reasons why we might want to do this and what it might mean.

The biggest question of course is the unanswerable one – what’s a debate? How do you know? There are endless definitions of a debate and most of them are good. I have very particular pedagogical preferences when it comes to what a debate is. I’m willing to change it based on the situation at hand. But when you say “I’m going to teach an AI how to debate,” the bigger question for me is “What’s a debate,” not “how are you doing to teach it?” If you don’t have some specifics there, you might not be teaching what you think you are.

Another question is the relationship between argument and debate. This is rarely thought about outside of the rather obvious “arguments appear in debates” or the somewhat naïve “debate is a kind of argument.” This also has to be grappled with a bit I think and we have to consider what this relationship would be. Again, no objectively right answer here, but you need a working definition in order to teach this. I’d say the way to think about it is history – you have to have a definition of history to teach it, and that definition has to be serviceable, not universally right – historiography is fascinating because of this continuously appearing feature of history. Same with debate and argument.

Something that didn’t make it into the talk but I’ve thought about since then is the necessity of having a temporary certainty to teach about something that circulates around uncertainty. Science pedagogy is quite comfortable with this – the good way is through the scientific method, a navigation process to teach for the swirling madness. The bad science pedagogy is the one of truth-discovering or “we are right” discourse that even good scientists (as apart from good science teachers) will sometimes say. To teach proof of concept of AI we need some clear understanding of a debate, what it is and how to look at it, a notion of what arguments are in that context, and then we can move forward and say an AI can debate or can’t debate – and what that might mean.