There was a recent blog post at Frontiers pointing out that journals' publicly-available rejection rates are not associated with their impact factors. Their post discusses several factors that contribute to this, but I've been thinking about how rejection rates are calculated, particularly publicly stated rejection rates. For example, the 2013 rejection rate for both JEP:LMC and JEP:HPP is 78% and JEP:General is slightly higher at 83%. These are top-tier experimental psychology journals and those rejection rates seem intuitively appropriate for selective outlets, but I think they might be inflated because many papers are rejected with an invitation to revise and resubmit.
Here's how the math works out: let's imagine a typical journal where a small percentage of the submissions are accepted immediately (let's say 5%), a substantial minority are rejected either with or without review (let's say 30%), and most papers are rejected with an invitation to revise and resubmit (on our example, 65%). Most authors would revise and resubmit in this situation (let's say 80%) and, if the editors are doing a good job, most of those resubmissions would get accepted (again, let's say 80%).
We can now calculate the key quantities based on:
Number of submissions = N*(1+(p_rej_resub*p_resub))
Number of rejections = N*(p_rej + p_rej_resub + (p_rej_resub*p_resub*(1-p_a2)))
Number of acceptances = N*(p_a1 + (p_rej_resub*p_resub*(p_a2))
If our hypothetical journal got 150 initial submissions, the total number of submissions (including resubmissions) would be 228, the number of rejections would be 158 for a 69% rejection rate, and the total number of acceptances would be 70 for a 46.6% effective acceptance rate. That is, 46.6% of the initial submissions were eventually accepted. Note that the effective acceptance rate and the rejection rate add up to more than 100% because a bunch of papers count twice -- as a rejection (with invitation to resubmit) and as an acceptance (after revision).
It's easy to see how this could be pushed even further by adding more revise-and-resubmit cycles: imagine that the same 5% of initial submissions are accepted, only 10% are rejected, the rest are rejected with invitation to resubmit. Most are resubmitted (90%) and of those 50% are accepted, 50% are rejected with an invitation to revise a second time, and those revisions are always submitted and accepted. Now those 150 initial submissions turns into 322 total submissions, 200 rejections (62% rejection rate), and 122 acceptances (81.5% effective acceptance rate). So you can nearly double the effective acceptance rate with a fairly small impact on the overall rejection rate. In other words, the publicly stated rejection rate may not tell you very much about the "selectivity" of a journal unless you also know their effective acceptance rate.
An interesting reverse case is the Proceedings of the Cognitive Science Society Conference. The primary submission format for CogSci is a 6-page paper (standard two-column publication-ready layout) that goes through a single round of standard peer-review (about 3 reviews per paper). There is only one round of review -- papers are either accepted or rejected (accepted papers can be revised before publication in the proceedings). Typically, about 30% of 6-page paper submissions are rejected, which may sound not very selective compared to the 80% rejection rate of the JEPs, but, as we've seen, the 70% acceptance rate might be quite typical once you consider the revise-and-resubmit cycle at those journals.
Here's how the math works out: let's imagine a typical journal where a small percentage of the submissions are accepted immediately (let's say 5%), a substantial minority are rejected either with or without review (let's say 30%), and most papers are rejected with an invitation to revise and resubmit (on our example, 65%). Most authors would revise and resubmit in this situation (let's say 80%) and, if the editors are doing a good job, most of those resubmissions would get accepted (again, let's say 80%).
We can now calculate the key quantities based on:
- number of initial submissions (N)
- proportion accepted on first round (p_a1)
- proportion rejected outright on first round (p_rej1)
- proportion rejected with invitation to resubmit (p_rej_resub)
- proportion resubmitted (p_resub)
- proportion of resubmissions that are accepted (p_a2)
Number of submissions = N*(1+(p_rej_resub*p_resub))
Number of rejections = N*(p_rej + p_rej_resub + (p_rej_resub*p_resub*(1-p_a2)))
Number of acceptances = N*(p_a1 + (p_rej_resub*p_resub*(p_a2))
If our hypothetical journal got 150 initial submissions, the total number of submissions (including resubmissions) would be 228, the number of rejections would be 158 for a 69% rejection rate, and the total number of acceptances would be 70 for a 46.6% effective acceptance rate. That is, 46.6% of the initial submissions were eventually accepted. Note that the effective acceptance rate and the rejection rate add up to more than 100% because a bunch of papers count twice -- as a rejection (with invitation to resubmit) and as an acceptance (after revision).
An interesting reverse case is the Proceedings of the Cognitive Science Society Conference. The primary submission format for CogSci is a 6-page paper (standard two-column publication-ready layout) that goes through a single round of standard peer-review (about 3 reviews per paper). There is only one round of review -- papers are either accepted or rejected (accepted papers can be revised before publication in the proceedings). Typically, about 30% of 6-page paper submissions are rejected, which may sound not very selective compared to the 80% rejection rate of the JEPs, but, as we've seen, the 70% acceptance rate might be quite typical once you consider the revise-and-resubmit cycle at those journals.
No comments:
Post a Comment