8-1. Open Access and Quality

8-1. Open Access and Quality
Contributors (1)
PS
Published
Mar 14, 2019

From “Open access and quality,” SPARC Open Access Newsletter, October 2, 2006.

http://dash.harvard.edu/handle/1/4552042


If an article is published in a toll-access (TA) journal and then deposited in an open-access (OA) repository, its quality does not change. And conversely, if it's first deposited in an OA repository and then published in a TA journal, its quality does not change. That's the sense in which quality and access are independent. It's obvious and it's basic. But it's not the whole story.

There are other, subtle ways in which quality and access intersect. This is an attempt to disentangle a large tangle of them.

Most of the ways in which access affects quality are very indirect. But there's one family of indirect effects that I won't cover here: the ways in which OA improves the quality of published research by improving the productivity of the researcher. That goes to the heart of the case for OA, but it's more familiar, and actually larger, than the topics I want to explore here.

*

The main factors that affect the quality of journal literature are price- and medium-independent: the quality of authors, the quality of editors, and the quality of referees. We know that these key players can be just as good at OA journals as at TA journals because they can be the very same people. An excellent TA journal can convert to OA and use the same standards and the same people that it used before. An excellent newly-launched OA journal can use the same people as an excellent TA journal. And of course two journals don't have to use the same people to use people of comparable skill and experience.

*

We have to acknowledge from the start that there are strong and weak OA journals, just as there are strong and weak TA journals.

Hence, any analysis focusing on weak OA journals and strong TA journals (as if to show the superiority of TA journals) would be as arbitrary as one focusing on weak TA journals and strong OA journals (as if to show the superiority of OA journals). Without some additional argument showing that the journals on which they focus are typical of their breeds, they would be guilty of cherry-picking and generalizing from an unrepresentative sample.

Moreover, we know that something has gone wrong if an argument for the virtues of either model implies that weak journals using that model do not exist.

*

TA publishers have often charged that OA journals compromise on peer review. The allegation is that if a journal accepts a fee for every paper it publishes, then it has an incentive to lower its standards in order to accept more papers. It sounds plausible but it doesn't stand up to scrutiny.


http://www.earlham.edu/~peters/fos/newsletter/03-02-04.htm#objreply


First, even if OA journals charging author-side fees have an incentive to accept more papers, it doesn't follow that they have an incentive to lower their standards. If they have a large number of excellent submissions, they can increase their acceptance rate without lowering standards. Unlike TA journals, they have no space limitations to hem them in.

Second, OA journals charging author-side fees often waive the fees in case of economic hardship. Not every accepted paper will create revenue, and some will have the opposite effect.

Third, OA journals charging author-side fees have editorial firewalls in place to insulate the peer-review process from business decisions about fees. For example, it's very common for editors not to know whether an author has requested a fee waiver. Many OA journals ensure that at least one voice in the editorial decision is not employed by the journal and has no financial stake in the outcome. At most of the new hybrid OA journals, authors don't even communicate their decision on the OA option until after the paper has been accepted.

Fourth, just like TA journals, OA journals know that their submissions and prestige depend on their quality. Preserving their quality will always be more valuable to them than another fee. This is especially true when we realize that the fees themselves are set at or near (and sometimes below) the subsistence level. A journal gains nothing and loses much if it lowers its quality in order to bring in a fee that does little more than pay the costs of bringing in the fee.

Fifth, the majority of OA journals charge no author-side fees at all.


http://www.earlham.edu/~peters/fos/newsletter/06-02-06.htm#facts


We're hearing less of this objection now that some of the critics (like Elsevier and the Royal Society) are offering hybrid journals and accepting author-side fees themselves. As full and hybrid OA journals spread, the objection will continue to fade away.

*

Is peer review at OA journals less rigorous than at TA journals? In October 2005 the Kaufman-Wills report (The Facts About Open Access) concluded that it was, based on a finding that TA journals used external reviewers, or reviewers outside the journal's editorial staff, more often than OA journals. However, a subsequent addendum retracted most of that conclusion as based on an erroneous interpretation of peer review practices at BioMed Central.

The Kaufman-Wills report (October 11, 2005)


http://www.alpsp.org/Ebusiness/ProductCatalog/Product.aspx?ID=47


Post-publication addendum (October 28, 2005)


http://82.45.151.109/Ebusiness/Libraries/Publication_Downloads/FAOAaddendum.sflb.ashx


For more details on the original criticism of OA journals and the post-publication retraction and clarification, see my interview with Cara Kaufman (November 2, 2005).


http://www.earlham.edu/~peters/fos/newsletter/11-02-05.htm#kaufman


On the other side, there are reasons to think that TA journals face stronger incentives to lower standards than OA journals.

First, the Kaufman-Wills report showed that more subscription journals charge author-side fees than OA journals. Author-side fees needn't cause a lowering of standards at either kind of journal. But insofar as they have that tendency, TA journals are afflicted more often than OA journals.

(Not only do a greater number of TA journals charge author-side fees, but a greater percentage of them do so as well. Of course, at a TA journal, author-side fees are laid on top of reader-side subscription fees.)

Second, TA journals often justify price increases by pointing to the growing volume of published articles. This is the incentive that critics saw in fee-based OA journals: the incentive to increase quantity in order to increase revenue. As with OA journals, this needn't result in a decrease in quality; but insofar as it has that tendency, the problem exists at both kinds of journals. Is it worse at TA journals? Consider the next factor.

Third, subscription fees at TA journals include substantial profits or surpluses, often more than 35%. An EPS report from July 2006 showed that the average profit margin at STM publishers in 2005 was 25%.


http://www.econtentmag.com/Articles/ArticleReader.aspx?ArticleID=16942&CategoryID=17

http://www.earlham.edu/~peters/fos/2006_07_09_fosblogarchive.html#115288206732578292


At fee-based OA journals, the incentive to accept more papers to generate revenue is small, because the fees barely cover their costs. At TA journals, the incentive to accept more papers to justify price increases is much larger, because subscriptions often contain significant profits or surpluses. As I've noted, when a journal has an abundance of excellent submissions, then it can increase quantity without decreasing quality. But when it doesn't have enough excellent submissions, then it can only accept more papers by lowering standards.

Not all TA journals have towering profit margins. Not all are even in the black. However—though no one has yet done the relevant studies—I would bet that subscription revenue exceeds costs more often at TA journals than fee revenue does at OA journals, and conversely, that fee revenue falls below costs more often at OA journals than subscription revenue does at TA journals. If so, then the average TA journal using increased quantity to justify price increases will get a bigger revenue bump from increasing its acceptance rate than the average OA journal will. Hence, the incentive to increase the acceptance rate by lowering standards is stronger at TA journals than at OA journals. Within the domain of TA journals, it's stronger at high-profit journals than at low-profit or non-profit journals.

TA publishers who gloated at the news—last June—that PLoS would have to supplement fee revenue with foundation grants didn't realize that the same news showed that PLoS has no incentive to lower its standards in order to bring in more insufficient fees. Or at least TA publishers with any kind of profit margin have a greater incentive to lower standards, accept more papers, and justify a price increase. Are OA journal fees high enough to corrupt peer review or too low to pay the bills? Critics can't have it both ways.

Fourth, if TA journals have a shortage of excellent submissions, they cannot publish a short issue without shortchanging subscribers. Hence, to fill an issue they must lower their standards. Because OA journals don't have subscribers, they are free to publish short issues limited to their first-rate submissions.

Fifth, TA journals with lower standards, lower submission rates, and (consequently) lower rejection rates have higher profit margins than journals with higher standards and higher submission rates. The reason is that journals with lower rejection rates perform peer review fewer times per published paper. Hence, publishers seeking higher margins have an incentive to lower standards. (This is compatible of course with the existence of other incentives pulling in the opposite direction.)

This is the conclusion of financial analysts at Credit Suisse First Boston, published on April 6, 2004. The report is not online but I wrote a summary for SOAN.


http://www.earlham.edu/~peters/fos/newsletter/05-03-04.htm#creditsuisse


The same Credit Suisse report shows that bundling can protect weak journals from cancellation and thereby insulate publishers from the market forces that would ordinarily punish declining quality.

Note that some of these incentives amplify one another. For a TA journal, lowering standards and lowering the rejection rate not only enlarges the journal and justifies price increases but also lowers the costs of peer review and increases profit margins.

*

There's evidence from TA journals that have shortened their embargoes or converted to OA that OA increases submissions. If it increases submissions, then it allows the journal to increase selectivity and improve the quality of the accepted articles.

From an interview with Elizabeth Marincola, then-Director of the American Society for Cell Biology, which publishes Molecular Biology of the Cell (Open Access Now, October 6, 2003):


http://web.archive.org/web/20031115185846/

http://www.biomedcentral.com/openaccess/archive/?page=features&issue=6


What happened after ASCB decided to provide OA to all the articles in MBC after only two months, the shortest embargo in the industry? “We have not lost subscription income, our submissions have gone up and our meeting programs have held strong. Financially we have been able to have our cake and eat it.”

From T. Scott Plutchak, Editor of the Journal of the Medical Library Association (LibLicense, February 10, 2005):


http://www.library.yale.edu/~llicense/ListArchives/0502/msg00172.html


The Journal of the Medical Library Association (JMLA) … has been open access via PubMed Central since September 2001. At present, every article, letter, editorial and feature of every issue, back to volume 1, issue 1, July 1911, is available. The online version is generally up within a day or two of the hard copy arriving on my desk. … Benefits? Although difficult to quantify, I'd say we're seeing greatly increased readership, a striking increase in manuscript submissions, particularly from overseas, and vastly increased value of the older material, since it is now so easy to get to.

From Sara Schroter (BMJ, February 14, 2006):


http://dx.doi.org/doi:10.1136/bmj.38705.490961.55


Three quarters (159/211) [of surveyed authors] said the fact that all readers would have free access to their paper on bmj.com was very important or important to their decision to submit to BMJ. Over half (111/211) said closure of free access to research articles would make them slightly less likely to submit research articles to the BMJ in the future, 14% (29/211) said they would be much less likely to submit, and 34% (71/211) said it would not influence their decision.

From D.K. Sahu and Ramesh Parmar (the Neil Jacobs anthology from Chandos Publishing, 2006):


http://openmed.nic.in/1599/


OA has certainly helped the Indian journals to reach an international audience … The number of manuscripts submitted to the journals has increased many fold (see Figure 19.1), with increases in the number of articles coming from other countries ranging from 12–44% for various journals (see Figure 19.2).

Finally, here's an argument I made in SOAN for March 2005:


http://www.earlham.edu/~peters/fos/newsletter/03-02-05.htm#coexistence


For authors, the only reason to submit work to a TA journal is its prestige. In every other way, TA journals are inferior to OA journals because they limit an author's audience and impact. OA journals will start to draw submissions away from top TA journals as soon as they approach them in prestige. And by the time they equal them in prestige, the best TA journals will have lost their one remaining competitive advantage.


There is abundant evidence that OA increases citation impact.


http://opcit.eprints.org/oacitation-biblio.html


I've often argued that citation impact is not the same thing as quality, and I haven't changed my mind. But I can justify talking about citation impact here because for many authors, funding agencies, and university promotion and tenure committees, citation impact is a crude surrogate for quality. Some acknowledge its crudity, or its divergence from a true quality measurement; but at the same time, some act as if they preferred impact to quality, insofar as the two diverge.

Some critics have argued that part of the correlation between OA and citation impact is due to a quality bias: authors preferentially self-archive their best work. Studies by Tim Brody, Chawki Hajjem, Stevan Harnad, and Gunther Eysenbach show that there is a substantial OA citation advantage even after correcting for any effects of this bias. I doubt that the debate is over, but for present purposes we needn't decide the question. Either OA articles have greater impact than TA articles even after we control for quality, or OA articles have a higher average impact because they have a higher average quality.

If self-archiving authors do preferentially deposit their best work, then the reason could be called author pride—the quality filter that costs publishers nothing. All published articles pass through this filter, of course. But published and then self-archived articles pass through it twice.

Just for the record: I'm not saying that author pride suffices; on the contrary, we still need peer review, sometimes to ratify author pride but more often to check it. Nor am I saying that the existence of a self-archiving bias arising from author pride negates the evidence for an OA impact advantage; on the contrary, I accept the evidence that a significant impact advantage remains even after subtracting the self-archiving quality bias.

I wrote about one form of quality bias in BMJ for May 2005.


http://bmj.bmjjournals.com/cgi/content/full/330/7500/1097


This is an editorial commenting on a study by Jonathan Wren, also from May 2005.


http://bmj.bmjjournals.com/cgi/content/full/330/7500/1128


Note that the OA impact advantage centers on the citation tally for individual articles, not the impact factor of whole journals. Insofar as OA increases the citation tally for articles, it will tend to increase the impact factor for journals. But other articles from the same journal might raise or lower the impact factor, muddying the water. In 2004 Thomson Scientific did two studies of the impact factors of OA journals, and both showed that OA journals had competitive numbers. Despite the relative youth of OA journals, even in 2004 there was at least one OA journal in the top cohort of impact factors in nearly every scientific discipline.

First Thomson study: James Testa and Marie E. McVeigh, The Impact of Open Access Journals: A Citation Study from Thomson ISI, April 14, 2004.


http://www.isinet.com/media/presentrep/acropdf/impact-oa-journals.pdf


Second Thomson study: Marie E. McVeigh, Open Access Journals in the ISI Citation Databases: Analysis of Impact Factors and Citation Patterns Thomson Scientific, October 2004.


http://www.isinet.com/media/presentrep/essayspdf/openaccesscitations2.pdf



Journal prices don't correlate with impact or quality. In fact, Theodore and Carl Bergstrom have shown that journal prices are either unrelated to quality or inversely related to it. In their analysis of journal prices and citation impact (Nature, May 20, 2004), they conclude that “libraries typically must pay 4 to 6 times as much per page for journals owned by commercial publishers as for journals owned by non-profit societies. These differences in price do not reflect differences in the quality of the journals. In fact the commercial journals are on average less cited than the non-profits and the average cost per citation of commercial journals ranges from 5 to 15 times as high as that of their non-profit counterparts.”


http://www.nature.com/nature/focus/accessdebate/22.html

http://www.earlham.edu/~peters/fos/2006_06_25_fosblogarchive.html#115168252782138224


Golnessa Galyani Moghaddam confirmed this conclusion in a study published this summer (Libri, June 2006). Not only do for-profit journals cost more than non-profit journals per issue or per volume, they also cost more per citation and per point of impact factor. Moreover, of the top 30 journals by usage at the Indian Institute of Science, 20 were non-profits and only 10 were for-profits.


http://eprints.rclis.org/8547/

http://legacy.earlham.edu/~peters/fos/2006/07/more-evidence-that-journal-prices-dont.html


By hugely enlarging the audience, OA makes authors more careful. If you like, consider this another effect of author pride.

In SOAN for July 2006, I argued that one reason for graduate schools to mandate OA for theses and dissertations is to improve their quality:


http://www.earlham.edu/~peters/fos/newsletter/07-02-06.htm#etds


All teachers know that students work harder and do better work when they know they are writing for a real audience—large or small—beyond the teacher. The effect is amplified if they are writing for the public. Some teachers try to harness this power by telling students to write as if their work were to appear on the front page of the New York Times. Some arrange to give students a real audience beyond the teacher. In a law course in which I conducted moot court, the quality of student preparation and argument improved dramatically after I started videotaping them. I didn't even have to put the videos online; I just put them on reserve in the library for the rest of the semester. … OA gives authors a real audience beyond the dissertation committee and real incentives to do original, impressive work. … [E]ven when grad students think it's safe and easy to fool their committee, it's risky and difficult to fool the world.

The Chronicle of Higher Education quoted a Yale professor to the same effect just last month:


http://web.archive.org/web/20080704211022/

http://chronicle.com/daily/2006/09/2006092001t.htm


Ramamurti Shankar, a professor of physics who is teaching one of [Yale's new OA] courses, said knowing that his lecture might be watched online by a wide audience keeps him on his toes. “I have to be a little more careful than I usually am,” he said.

Here's a variation on the same theme: OA keeps authors honest.

Citing OA articles makes it easy for readers to verify that authors are accurately summarizing the cited work or data. Citing TA articles makes this harder (but clearly, not impossible) and to that extent protects authors who want to blow smoke. The most detailed case I've seen for this conclusion is also the most recent: Mark Liberman, Open-access sex stereotypes, Language Log, September 10, 2006.


http://itre.cis.upenn.edu/~myl/languagelog/archives/003565.html

http://www.earlham.edu/~peters/fos/2006_09_10_fosblogarchive.html#115799535190633252


Here's another variation on the theme: OA deters plagiarism. In the early days, some authors worried that OA would increase the incentive to plagiarize their work. But this worry made no sense and has not been borne out. On the contrary. OA might make plagiarism easier to commit, for people trolling for text to cut and paste. But for the same reason, OA makes plagiarism more hazardous to commit. Insofar as OA makes plagiarism easier, it's only for plagiarism from OA sources. But plagiarism from OA sources is the easiest kind to detect. Not all plagiarists are smart, of course, but the smart ones are steering clear of OA sources.

For the same reason, they'll avoid OA dissemination for any of their own works containing plagiarized passages.

The first tendency improves the average integrity of work quoting OA literature. The second improves the average integrity of OA literature itself.

Because OA will only reduce plagiarism by smart plagiarists, the effect may be small. And today the effect is small in any case because so little of the literature is OA. But just as we can expect good things from a pest-resistant strain of wheat, even when we've just introduced it in one field, we can expect good things from this plagiarism-resistant strain of research literature.

*

The EC's Study on the Economic and Technical Evolution of the Scientific Publication Markets in Europe (dated January 2006 but apparently not released until late March or early April) recommended that we widen our concept of a journal's quality to include quality of access or quality of dissemination. This is an excellent idea.


http://europa.eu.int/rapid/pressReleasesAction.do?reference=IP/06/414&format=HTML&aged=0&language=EN&guiLanguage=en

http://www.earlham.edu/~peters/fos/newsletter/05-02-06.htm#ecreport


A critic might object that even if adopted, this wouldn't change the fact that the OA/TA status of an individual article is independent of the article's quality. That's true but beside the point. The report isn't trying redefine the quality of articles, but to recognize other kinds of quality. When academic publishers give awards to good journals, they recognize many kinds of quality, including (for example) quality of design. That's the kind of enlargement of our thinking the EC study recommends, but it focuses in particular on quality of access, which publisher award ceremonies tend to overlook. When libraries decide what to buy, renew, or cancel, they consider many kinds of quality, including quality of access. For example, they take price into account, as one criterion among many others, and for TA electronic journals they investigate whether they will still have access to subscribed issues after they cancel.


Now and then someone will suggest that OA is fine for second-rate work but not for first-rate work. This claim is more sniffed than elaborated, so it's hard to tell what the argument is.

If it's saying that high-quality, high-prestige journals will never or rarely be OA, or vice versa, then it's a prediction, not a datum. Moreover, it seems to be a false prediction. There are already high-quality, high-prestige OA journals, for example, the Beilstein Journal of Chemistry, Nucleic Acids Research, and PLoS Biology.

(I pick these three to show that OA, quality, and prestige can exist together under a variety of circumstances: two charge author-side fees and one doesn't; two are from non-profit publishers and one from a for-profit; two are converted TA journals and one was born OA.)

There are more than this list of three, of course. But one reason there aren't already more than there are is that most OA journals are new. Even when new journals are excellent from birth, it takes time for their prestige to catch up with their quality. Another reason is that most of the money to pay for peer-reviewed journals is still tied up in support of TA journals. Since neither of these explanations depends on any intrinsic limitation in the quality of OA journals, we have good reason to think that the numbers of high-quality, high-prestige OA journals will grow as we remove the barriers to their growth.

I can accept one form of the premise, namely, that so far most high-quality, high-prestige journals are TA. But of course, so far, most low-quality, low-prestige journals are also TA. Moreover, the present ratio of excellent TA journals to excellent OA journals is just a present fact about a very dynamic, rapidly changing situation, not a fact about the intrinsic quality of either kind of journal. To mistake it for more would be like arguing in 1980 that more prestigious journals used typewriters than computers, and therefore that computers must have some intrinsic limitation keeping their numbers down.

I can accept another, more important form of the premise, namely, that most authors will seek prestige before OA, if they have to choose. The mistake is to assume that they have to choose.

There are two reasons why there's no trade-off here. First, there are already high-quality, high-prestige OA journals and their existence shows that nothing intrinsic to OA blocks that path. Second, authors can publish in a prestigious TA journal and then deposit their postprint in an OA repository. About 70% of TA journals already give blanket permission for this and many of the others will give permission after an individual request.


http://www.sherpa.ac.uk/romeo/statistics.ph


There's a related argument (related because it's more sniffed than elaborated) that the internet is the proper home for crap, not scholarship. It's really more prejudice than argument, and in 2006 it's more dead than alive. Hence, you may think it no longer deserves a response. But it was common in the early days of the net, and when it was common it was also self-fulfilling. It was one (of many) early obstacles to OA, and we're still struggling to overcome its effects. Moreover, it isn't completely dead.

It's true that the crap/gold ratio was very high in the early days of the net. But even then it didn't follow that there was no gold, let alone that there shouldn't be. And today the only people who can still say that there's no high-quality, peer-reviewed work in their fields on the internet are the ones who aren't paying attention.

Even though scholars generally know that there is good scholarship online, there is still a sense in some quarters that quality work belongs elsewhere, or that good work online is harmed by its association with crap. For example, at universities giving grad students the option to submit dissertations electronically, and where OA for electronic submissions is the default, there's evidence that some professors advise their graduate students against it. They're trying to preserve their students’ chances of publishing parts of the dissertation in the future (well-intentioned but uninformed) and trying to make their students look like real scholars rather than camp-followers (ironically more camp-following than scholarly).

The attitude is often accompanied by mutterings about the dislike of reading long or difficult works online or the love of printed books—which I share, by the way, but which are compatible with taking full advantage of the benefits online dissemination. Just as often it's accompanied by mutterings that “you get what you pay for.” And of course it's still true that the internet is full of crap, though it's false that that crap/gold ratio hasn't declined steadily in the past decade and false that the tools for finding the gold haven't improved just as steadily. It's true that putting peer-reviewed scholarship online puts it in the same bin as a lot of crap, but the same is true of the bin of print. What's astonishing is that smart people can forget that the low quality of the crap online doesn't affect the high quality of the scholarship online. More critically, putting peer-reviewed scholarship online doesn't add to the crap online; it dilutes the crap online.

If the same squeamishness about online dissemination had infected print dissemination in the age of Gutenberg, on the ground that real scholarship was inscribed by hand on goatskin, then every kind of knowledge would have been held back. What's striking is that those still carrying traces of this prejudice would rather follow the (literally) hide-bound customs of their field than take advantage of new technologies to pursue their own interests.

Discussions

Labels

No Discussions on this Branch