The “general-interest” label

The “top5” journals are also referred to as “top general-interest” journals. In our discipline, there have traditionally been rankings of journals. Even before the top5itis era, some journals were perceived as more prestigious, largely manifested through their high rejection rates. Back in the day, when economics was not as well developed as it is now (in terms of its volume of scientific production), the more demanding journals used to publish “general-interest” papers, with the idea that such papers could attract the attention of very diverse economists.

My impression about the current state of our discipline, however, is that there are basically no generalists in the profession. That is, perhaps as a natural evolution of a mature field, we all have become specialists. In this light, it seems a bit strange to keep insisting on the “general-interest” terminology. I am not opposed to maintaining that label, but in my opinion, if economics wishes to keep insisting on it, the history of economic thought should be taught in all top programs, so that the new generations of best-trained economists are capable of evaluating contributions in their field in light of a bigger picture. History of thought has been absent for many decades from pretty much all graduate economics programs. Perhaps as a result, right now I cannot think of many economists who I would label “generalists” in terms of having a strong command of many different fields.

One way to underscore this point is that I doubt most theorists read the latest “general-interest” papers in econometrics published by some of the top5 journals, or that most macroeconomists read the “general-interest” papers in theory published in the top5 journals, and so on. The assertion can probably be strengthened to hold within areas: for instance, I doubt a theorist interested in repeated games reads carefully the latest paper in social choice published in a top5 journal, and vice versa. Thus, perhaps what those journals publish is something else; perhaps they publish papers that, in the view of the referees and editor involved, represent a significant leap forward in a given literature, even though that literature may not be one of “general-interest.”

One possible manifestation of top5 journal editors also being specialists is that often they pick referees from the list of references in the paper. This is not a bad practice, of course, but it may lead in some instances to excessive monopoly power awarded to some referees. I have heard from multiple colleagues how, in some of their submissions to these journals, they have faced the same referee in two, sometimes three of them, which in itself creates an undesirable consequence. In my view, no single person should be awarded monopoly power over the fate of a piece of research, and relying on such a small set of individuals seems particularly odd when the bar one is trying to pass is that of the “general interest.” By the way, many of these stories have come to me from junior scholars in the profession who are starting their careers. Given the perverse incentives created by top5itis, it is not surprising that these colleagues keep trying to exhaust the full list of top5 journals, perhaps under the belief that there is no life outside of that reduced set of journals. This is no doubt another negative consequence of the disease, as publishing in great outlets, such as very demanding top field journals, is definitely better than wasting several years with negative experiences at the top5. Given the current incentives, this decision is a tough one, of course, since ex ante versus ex post beliefs may be very different in terms of evaluating the publication prospects of a given paper.

Refereeing

Some colleagues defend the top5 label category by arguing that the editorial and refereeing standards applied there are far tougher and better in quality terms, justifying their special status. While this is possibly true -many editorial decisions in top5 journals are of high quality and referees require a higher bar of significance for a paper-, enough anecdotal evidence suggests that this is not the case many other times. Let me first point out a dangerous trend I have found, another consequence of top5itis. Under the belief that a journal outside of the top5 is already substandard, some colleagues have confessed to me that they do not put too much effort in refereeing for those journals. This is a serious problem, that is, the myth of the top5 journals associated with top5itis should never be an argument to justify sloppy refereeing in other journals. Sloppy refereeing is never justified, and in fact, good refereeing should be understood by all of us as an essential part of the discipline, since in the end high-quality peer-reviewing is key to evaluate our scientific output. I will refer here to ^{Thomson’s (2011)} excellent authoritative piece on suggestions for good refereeing.

In particular, what does not seem acceptable is the practice of one-line reports, written by some referees of top5 journals, in which the only thing said is that “the paper under consideration does not belong in the top5.” I am troubled that this practice has sometimes been encouraged by some top5 journals. That is, in my view, referees should not be desk-rejecting a paper because that is not their role: after the editor has decided to seek the opinions of expert referees, it should be understood that s/he has decided not to desk-reject the paper, and hence, what the editor needs at this point is a careful analysis and evaluation of the contribution in the paper made by experts that s/he trusts. And while such a report could hinge on the assertion “this paper is not of sufficient general interest,” I do not think this should be used as a blanket statement. It should be reasoned. For example, if the main result in the paper makes unduly restrictive assumptions on preferences in a matching model, or implausible assumptions on the path of monetary policy -and here of course there is always room for some subjectivity in the assessments- one can see an argument for why the “general-interest” label could be unwarranted in that case.

The crystal ball

The task of any editor is important and challenging, i.e., to decide whether a new piece of research deserves to be added to the stock of published knowledge. The task of an editor in one of the top5 journals is even more important and arguably harder, given the higher impact of their decisions. An editor of one of these journals rejected a paper that was connecting literature A with literature B. S/he was seeing not enough significance in this connection to be judged of sufficient general interest. While this editor was not known to have contributed to either literature A or B, such a judgement is fair game, as editors of these journals tend to be great scholars who are certainly entitled to decide the direction of the journal they edit. The problem is that the editor went further and wrote: “In fact, I do not envision this connection generating sufficient important papers in the next decades.” The use of crystal balls of this kind should not be part of good editorship, as it is at best a dubious practice. That is, good editors, who already have a difficult job in their hands, should not play to be fortune tellers.

Desk-rejections

Desk-rejection decisions are important, and they will always be present as part of a good editor’s tasks. Any editor has the prerogative of desk-rejecting a paper. That is, to issue a final rejection decision even without consulting with referees. When I was the editor-in-chief of Economics Letters a few years ago, I had to desk-reject a significant fraction of submissions, and I remember each of those as probably the hardest task I had to deal with in my editor job. The reason is simple: it is an important decision for the fate of that submission, and it is made without relying on anyone else’s advice. Needless to say, this meant that, before desk-rejecting a paper, I had to be quite sure of my reasons, which were always provided in my decision letter. In those, I tried to avoid as much as possible blanket statements about general-interest -Economics Letters also being a general-interest outlet, albeit not a top one, because of its nature as a quick dissemination vehicle of short pieces. I am sure I made some wrong calls in those six years of EL editorship, even though I tried hard to avoid them.

There are some practices in desk-rejections that ought to be avoided. For example, an editor of a top5 journal desk-rejected a paper because ”although I am not an expert in this area, this result must be known” without offering a concrete reference. In further correspondence between the authors and the editor, the editor asked the authors to run the paper by an expert and, if that expert would support the paper, to inform the editor so that the decision could be revisited. This decision and its subsequent course of action seems awkward and arbitrary. That is, an editor should be providing better-founded reasons to back the desk-rejection.

At the opposite extreme, an editor of a top5 journal desk-rejected a paper because the main result contradicted the editor’s intuition. Obviously, while we all typically trust our intuition at first, the process of learning should open our minds to countering it. There are many important results that, at first, seem totally surprising, yet the careful analysis of the proof reveals the nontrivial argument that leads to the result. In my opinion, if I were in the editor’s chair, when a result strongly goes against my intuition, I should stay away from the temptation of desk-rejection, as a more detailed evaluation seems warranted.

Another editor of a top5 journal rejected a paper that uses the axiomatic approach with the following sentence: “our journal is not interested in axioms.” When I heard this story from the author, I told him that, using that argument, key contributions to economics -Arrow’s impossibility theorem, the Nash bargaining solution, the Shapley value, and so on- would have also been turned down by that journal, so the author should be proud to be in such very good company. But more fundamentally, the editor’s argument was flawed because saying “our journal will not publish axioms” is in itself an axiom, and a bad axiom if you ask me. (Recall that an axiom is an undisputed principle that is formulated before we proceed to any analysis of its consequences.)

There are desk-rejections that are not really desk-rejections: some editors of top5 journals have used associate editors to ”help their desk-rejection decision”. In a couple of instances that were relayed to me, this was not fair to the AE, who was already told the direction in which the evaluation should go, nor to the author, who was not given all the relevant information, i.e., if an AE was consulted, the report should have been part of the review, which of course would underscore the fact that it was not a desk-rejection.

Same reports used in a vertical hierarchy of journals

Some of the top5 journals began a practice a few years ago of accompanying some rejections with the suggestion of a specific journal as the next step, mentioned by name, and where the same reports could be used. In principle, this could be a good idea because it could expedite the review process, and it has worked sometimes. But I have also heard multiple accounts where this ended up in a very negative experience. For example, an editor of journal X, a top5 journal, rejected a paper and suggested journal Y, asserting in the decision letter that all referees were also supporting the idea of publication in journal Y. The author liked the plan and revised the paper addressing all points in the referee reports. When the author submitted to journal Y and asked the same referee reports to be used, the editor of journal Y agreed to the course of action. But soon after that, the editor rejected the paper, with no new reports, explaining that actually the lack of enthusiasm of the referees did not allow him to move forward. Evidently, it is always possible that referees’ cover letters (not to be shared with the author) could include additional arguments not mentioned in the report, but it would be desirable not to give the author wrong or misleading information, which may lead to the belief that some criteria have been changed along the way without a proper explanation. I also suspect that some editors of very good journals, but not in the top5, like to make their own decisions in terms of how the paper should be processed, including who the best referees should be; in this light, some of them may feel that their important role, which should be exercised with full autonomy, is somewhat diminished by this practice.

Views from mathematics and applied mathematics

I have talked to colleagues in other disciplines to better understand how our review and publishing processes compare to theirs. Let me first summarize the views I have heard from some mathematicians and applied mathematicians. instead of few top journals, these disciplines have a large group of prestigious journals, often ranked differently by individuals or schools, associated with institutions, like SIAM (Society for Industrial and Applied Mathematics), AMS (American Mathematical Society), Informs, etc., but never is a journal label a substitute for reading the paper. When I described to them the top5itis disease, the reaction I got was one of perplexity: economists understand the advantages of competition, and therefore, it seems a bit bizarre to “give so much power to so few journals” - this very point was actually made by James Heckman in the 2017 panel (^{Heckman et al. (2017)}). On the other hand, these disciplines seem to have similar review processes to those in economics, including an editor, associate editor, and two or three referees to review a paper.

Views from computer science

My conversations with computer scientists were also useful. The discipline of computer science has exploded in size. As a consequence, there are many more submissions to conferences and reviewers can’t keep up. Now all submitters are requested to review other submissions. In general, this has resulted in a deterioration of reviewing quality, although some solutions have been implemented. First, the review is now open, so that reports can be viewed by others in the reviewing website. Second, a phase of rebattling has been introduced and is very helpful (in it, the different referees communicate anonymously with each other to try to iron out differences in their assessments). Most publications appear in proceedings of conferences, which are typically within a given field (machine learning, natural language processing, etc.) There is no general-interest conference, except that every four years, all the field conferences colocate. The Journal of the ACM (Association for Computer Machinery) publishes revised versions of papers that have proven to be of ”general interest” - earlier versions of such papers already appeared in some conference proceeding and have been influential in much work. The large increase in submissions has not resulted in much larger or longer conferences, as these are time- constrained. There has been some movement to increase the number of parallel sessions, although there is reluctance to do this. In general, the overall effect ends up being much higher rejection rates. There are A-level conferences and B-level conferences, although it is also somewhat unclear how those labels came to be what they are. There are some advantages to B-level, usually attended by fewer people, which facilitates deeper and more meaningful discussions. Another problem is that a given author submits a large number of papers to the same conference: since the perception by some seems to be that refereeing is random, such authors may think that at least one of their ten papers submitted will get in. Perhaps an upper bound to submissions per author should be imposed, but that has not happened yet.

Views from psychology

In psychology and cognitive sciences, there is a large number of journals, roughly ranked by their impact factors. Importantly, publications in general science outlets, like Nature or Science, are extremely highly regarded, as it is widely understood how hard it is to publish there. When the psychologists I talked to learned of the top5itis disease in economics, they point out an additional problem from their point of view: perhaps this is a reflection of economics as a very narrow discipline, truly self-centered, which leads some psychologists to accuse some economists of refusing or hindering interdisciplinary collaborations. (I am here reporting views, even if my impression is that much recent research in economics has opened up to other fields, such as psychology, but these views perhaps should not be discarded outright.) Journals like Psychological Review or Psychological Science stand out as” general-interest,” but they coexist with prestigious field journals in cognitive psychology, neuropsychology, etc. One commonly used criterion to measure” general interest” in a researcher is to see if her publications spread around a sample of diverse journals, instead of publishing always in a small number of journals.

Views from biology

As correctly pointed out by Drew Fudenberg in the ^{Heckman et al. (2017)} panel, biology seems to have a similar disease to top5itis in economics. Biologists tell me that “impact factors seem to drive everything.” In fact, the situation in biology seems to be even more serious. For example, some universities have started to peg faculty salaries to the impact factor, which is in itself a very perverse incentive. Furthermore, publishing is entirely in the hands of large commercial publishers, without the counterbalancing effect of a professional noncommercial association or scholarly society. There are many journals, in many fields and subfields, as well as more general journals, but the obsession with Impact Factors is pervasive. The large commercial publishers run the most prestigious journals, and the profit motive sometimes interferes with the scientific quality criteria (some editors have a hard time convincing the publishers of raising standards, as larger rejection rates may result in fewer pages being printed). There is a lot of dissatisfaction with the current system, but people do not seem to know how to build consensus about making a real change. Some people are suggesting using other indices, such as the H factor, or the eigen factor, which would start to deemphasize the dictatorship of impact factors.

A closing thought

Although relying on several well-chosen statistics is usually enlightening, basing the evaluation of a complex issue on a single metric, whatever that is, is an oversimplification, it creates a perverse incentive system, and it is potentially responsible for many incorrect decisions. A more flexible and multidimensional analysis seems appropriate to evaluate the scientific output in economics.