11-2. Thoughts on First and Second-Order Scholarly Judgments

From “Thoughts on first and second-order scholarly judgments,” SPARC Open Access Newsletter, April 8, 2002.

http://dash.harvard.edu/handle/1/4727447

What do search engines, web filters, current awareness services, and peer review have in common? They all help us churn haystacks and find needles, or process noise and find signals. They help us navigate the wilderness of information. Pick your metaphor, or try this non-metaphorical way to look at it.

Let's say that first-order scientific or scholarly judgments are judgments about what is true or probably true in astrophysics, organic chemistry, French history, epistemology, or any other field of academic research. First-order judgments are what scientists and scholars primarily produce in their roles as scientists and scholars. Let's say that second-order judgments are judgments about which first-order judgments you ought to read.

Search engines, web filters, current awareness services, and peer review give us second-order judgments. They are just a few of the many sources of second-order judgments, alongside card catalogs, book catalogs, tables of contents, spam filters, and informal networks of pointers and recommendations by trusted friends and authorities. Of course there are important differences among discovery, retrieval, evaluation, recommendation, and blocking. But there are also important similarities, namely, their second-order parasitism on first-order judgments and their ability to assist or distort research.

Second-order judgments are not necessary for researchers who personally know all those likely to make contributions worth reading, or for disciplines in which literature is sparse or disagreements are shallow. But as the universe of first-order judgments relevant to our research interests becomes unmanageably large, and as the disagreements within this universe increase in number and depth, then we need help avoiding the intellectual provincialism and risk of error that arise from reading only what it is ready to hand. We need the assistance of second-order judgments. (One recipe for crank literature is devotion without this assistance.) This isn't assistance in formulating our own first-order judgments or even in assessing those of others, but in making intelligent decisions about how to allocate the finite resource of our time and attention. As time passes and science grows, the need for second-order judgments will only increase.

There are two reasons to celebrate. The first is that the need for second-order judgments is a sign of the flourishing of the sciences and scholarship producing first-order judgments. Information overload and the need to manage it may be obstacles and irritants, but they are clearly side-effects of success. […]

The second reason to celebrate is that free online scholarship is free online data for increasingly sophisticated software that generates second-order judgments. In the age of print, second-order judgments had to be produced by trained human scholars. When scholarly literature is digital but priced, then only its owners can experiment with software to help us find what is relevant, what is worthy, and what is new. Some of these owners have the means and will to code tools of this kind, and some are creative. But when scholarly literature is digital and unpriced, and even networked so that it can appear on every desktop […], then the fetters on innovation will fall away and the pace of development will accelerate.

I'm tempted to put it this way. Computers have triggered more than one revolution in scholarly literature, apart from their assistance with first-order judgments. The first revolution was simply to digitize text, which permitted flexible writing and free copying. The second revolution was to network the digitized articles, which spread them to all connected users. At first these networked articles were all free, but as the technology evolved to block access to non-paying customers, more and more of the new literature came online only behind passwords where most readers could not reach it. The third revolution will be the return to free online access as the default for scientific and scholarly research articles. This will increase the accessibility of every article, helping readers, and increase the audience and impact for every article, helping authors. This is the FOS [Free Online Scholarship] revolution and we're still fighting for it. But it will not be the end of the line. The fourth revolution will be to write increasingly sophisticated software that takes FOS as data and returns increasingly intelligent and customizable second-order judgments about what is relevant, what is worthy, and what is new. Making online scholarship free of charge makes it universally accessibly to connected human researchers, a major plateau in the progress of the sciences. But making it free and online also makes it universally accessible to software and programmers, which has the potential to create an even higher plateau further out.

Today there are several incentives for publishers to make scholarly literature freely available online: to respond to competitive pressure from other free journals, to increase their citation rate and impact factor by reaching a larger audience, to sell auxiliary services, to accede to demands by scholars, and to assist in the dissemination of knowledge. One incentive that is weak today and will become stronger over time is to provide scholarly content to the far-flung, distributed swarm of services processing FOS and turning it into second-order judgments on which scholars rely to learn what is relevant, what is worthy, and what is new.

If the flourishing of first-order science produces information overload, and if information overload increases the difficulty of discrimination, then tools to discriminate according to my own standards will be among the most essential tools in my research toolkit. As the scholarly use of these tools becomes routine, then literature will only be visible if it is made visible by these tools. Free and online won't be enough, just as ready-to-hand isn't enough if my desk is so littered with photocopies that I can't find what I want. If the best tools or the free tools take FOS as data, then publishers will have to produce FOS in order to make their articles visible. It follows that one strategy to accelerate FOS is to write good second-order judgment software that takes FOS as data.

Commercial publishers will still produce second-order software in-house and apply it to their priced content. Insofar as their tools are good, users will have an incentive to pay for them. This not a problem for FOS. First, it is compatible with the growing number and quality of free tools taking free literature as data. Second, many publishers will choose to give away their first-order literature and sell their second-order tools and services, which is entirely compatible with FOS. Third, users and research benefit when second-order tools proliferate and compete.

The beauty of second-order tools using first-order scholarship as data is that there can never be too many of them. If proliferating first-order judgments creates information overload, then proliferating second-order judgments creates competition, and this competition will be beneficial for users and self-limiting. Second-order judgments are valuable even when they conflict, because different users have different needs, interests, projects, standards, and approaches. You should have a choice among services competing to help you decide what deserves your time and attention. Of those services that know what you want, some will be faster, cheaper, or friendlier in providing it. Of those that are fast, cheap, and friendly, some will know better what you want. If putting priced paper literature online free of charge accelerates research, then a robust market of sophisticated, competing second-order tools will accelerate it again.

Part of academic freedom is to have a free market in first-order judgments. By this I only mean that scientists and scholars need the freedom to take a stand on what is true or probably true in their field, and be immune from every kind of retaliation, except disagreement and criticism, for doing so. (I know that I've returned to metaphor by calling this a free market.) As first-order science continues to flourish, and as information overload worsens, an essential part of academic life, as vital as academic freedom, will be a free market of second-order judgments. Yes, there will be neo-Nazi filters on historical literature and fundamentalist filters on biological literature, but these will merely be electronic reflections of methodological and ideological divisions that today show up in different journals or different conferences. Yes, second-order judgments will evaluate other second-order judgments. (For more on this, see FOSN for 11/16/01.) But without the discriminating power of second-order tools, we will be at the mercy of information overload. And without the choice of different discriminating standards, first-order academic freedom will be ineffectual.

http://www.earlham.edu/~peters/fos/newsletter/11-16-01.htm

Postscript

One of the sillier objections to FOS is that it will increase information overload. Either this objection is an inept way of saying that FOS will dispense with peer review (which is untrue) or a prediction that making peer-reviewed literature free and online will increase its quantity (which may be true but unobjectionable). The standard response is to point out that the growth of peer-reviewed literature is a sign of progress, even if it creates information overload. While that's true, it may not address the part of the objection that bemoans the information overload it predicts and might otherwise value. A better response is to point out that FOS will inspire the development of second-order tools that take FOS as data. These tools are not only a remedy to information overload. They are the only remedy that doesn't require reducing the output of science and scholarship.