© 2004 American Thoracic Society
Assessing the Performance of a Medical JournalOver the past five years, a number of people have asked me, "How do you compare one medical journal with another"? The inquirer usually follows by saying, "Surely, there must be objective measures other than impact factor"? (1). I recently discussed journal performance with the Society's board of directors, and several of those present recommended I share the information with readers. Below I list ten measures of performance and use AJRCCM to illustrate them. One measure is number of submitted manuscripts. Submissions to AJRCCM increase on average by 3% a year, and have almost doubled between 1985 (962) and 2003 (1,836). Some may think a steady increase in submissions is to be expected. Not true. Over an equivalent time, submissions to the Annals of Internal Medicine decreased from 2,234 in 1982 to 2,131 in 2001 (2). A second measure of performance is journal distribution and profitability. The print circulation of AJRCCM is 18,000, which compares favorably with 23,000 for Circulation despite the far higher number of cardiologists than of pulmonologists or intensivists. The online Journal is now averaging more than 930,000 hits a month (Figure 1) , as compared with 100,042 hits a month received by AJP: Lung. Number of hits is an important measure of performance, because authors submitting a manuscript want the widest possible exposure for their work.
Financial profitability is perhaps the most concrete measure of performance. Its pivotal importance is articulated by Max Frankel, former executive editor of the New York Times (3): "The Times existed to practice great journalism. It needed to make money—not to enrich the Sulzbergers (publishers) or even the stockholders but because profits were the only guarantee of the paper's honesty, independence, and survival." The emphasis on profits is recognition that sales are the best indicator that the public is pleased with a newspaper. Likewise for a medical journal to last, it must be profitable (4, 5). AJRCCM remains a major profit center for the American Thoracic Society, and the cost to members is less than 25% of annual dues. Pages published in AJRCCM climbed steadily throughout the 1990s: 3,360 in 1993, 3,952 in 1996, and 4,200 in 1999. Continuation of this trajectory would have required 4,800 pages in 2003—a severe stress on ATS resources. Apart from money considerations, a journal that grows in thickness is not a healthy sign but instead reflects an editorial team that is not acting judiciously (6). Consequent to steps implemented by the associate editors, published pages fell to 2,992 in 2002. Costs for paper, composition, printing, and binding were reduced, as were costs for addressing, consolidation, and distribution. As a result, AJRCCM came in at $880,443 below budget for those expenses in 2002. Savings were repeated in 2003, as the Journal published 2,905 pages. A third measure of performance is publication lag (time between acceptance of a manuscript and publication in a journal). Publication lag was on a trajectory for 10 months in 1999 (Figure 2) , which defeats the purpose of a journal aiming to publish current research. Greater discrimination on the part of reviewers and associate editors, together with introducing an online repository and streamlining within our production unit, has decreased the lag to 2.5 months. Many journals achieve this timeframe only for a rapid communications section, but it is routine for all manuscripts at AJRCCM.
Fourth is finding out what readers want and satisfying their needs (7). One of the loudest messages of the last membership survey was a request for more review articles in AJRCCM: 84% of the respondents strongly agreed or agreed with that sentiment as opposed to 4% disagreeing. The number of review articles has increased about fivefold since 1997 (Figure 3) , and we have introduced six new series.
The Update in Nonpulmonary Critical Care series helps clinicians learn of recent advances in nonpulmonary fields (8). Important research in those fields is typically published in subspecialty journals, which are perused by few intensivists. Updates are co-authored by a nonpulmonary subspecialist, who provides the major content, and a pulmonary physician, who defines the boundaries and depth of coverage. The series is geared in part toward physicians taking the recertification examination in critical care. In How It Really Happened, senior investigators provided a behind-the-scene account of the human dimension to scientific discovery (9). Authors wrote of many mistakes, false starts, and blind alleys, and gave much credit to serendipity. None of the major discoveries followed the plan typically presented in a grant proposal. Confessional essays of this nature are rare to nonexistent in biomedical publications, and the series will offer future researchers a unique glimpse into laboratory life of the late twentieth century. Occasional Essays were introduced to enable discussion of nonclinical, nonresearch subjects, such as health policy and social issues (10). A series of articles is currently being published in this forum to celebrate the ATS centenary, with authors relaying the history of pulmonary and critical care medicine over the last century (11, 12). Year in Review was introduced to help readers keep up with the latest research, and to enhance cross-fertilization among researchers in different disciplines. The taxonomy on the homepage makes it easy for readers to find a group of articles on any topic. Critical Care Perspectives were introduced to complement pulmonary perspectives. Rebuttals were added to pro-and-con debates to make them more rounded, and the number of debates has increased fivefold. Review articles are essential for the advancement of medical practice. General medical journals publish a higher proportion than do subspecialty journals. But because these articles are directed at general internists, they are of limited value to subspecialists who desire a more sophisticated and nuanced understanding of subjects in their field (13). A fifth measure of performance is involvement of readers, as reflected by letters to the editor. Letter writers sometimes make the most penetrating criticisms of recently published articles, noting flaws missed by the investigators, reviewers, and editors. A letter writer may offer insightful interpretations of data that differ from those of the original investigators. Before publication, AJRCCM sends such letters to authors of the original article so they have an opportunity to respond. As such, the correspondence column provides an open forum for vigorous debate (14). Letters to the editor define the soul of a periodical (15), and show that people are reading it. And that is why the more than fourfold increase in letters over the past five years is gratifying (Figure 4) . Without a lively letters section, together with editorial commentaries and review articles, a journal is acting as no more than a repository for original research reports.
Sixth is readability of the journal. It is trite to say that articles are published to be read. Yet, some manuscripts border on incomprehensibility, not because the science is abstruse but because the presentation is careless. Last year, I contrasted the clear and concise prose of Watson and Crick against the obscure style of Franklin and Wilkins in the same issue of Nature—ensuring under-appreciation of the latter articles despite their more sophisticated experimental finding (16). Over the past five years, AJRCCM has made major efforts to improve manuscript presentation: authors are now communicating more concisely, abbreviations have decreased, and editorials have pointed out simple steps for improving presentation (13, 16, 17). My major motivation in writing the Year in Review summaries was to provide readers with the key findings of every article in simple words. Seventh is ability to take advantage of the latest advances in technology. The Internet has transformed four aspects of AJRCCM. One, online peer review was introduced in August 2001. Median time to first decision on a manuscript is now 30 days as compared with 90 days in the past. More importantly, the Internet enabled AJRCCM to improve the rigor of peer review (18, 19). The Journal has more than 6,000 referees in its Internet database, and expertise of reviewers is linked to 172 descriptor categories. Descriptor numbers enable associate editors to select reviewers in a more scientific manner than the old system, which fostered tunnel vision, overwork of a small number of reviewers, and an old-boys' network. Peer review is the primary instrument for ensuring quality control at AJRCCM, and appointment to the Editorial Board is based solely on the number of reviews performed and their cogency, rigor, and timeliness. Two, introduction of an online repository made it possible to remedy problems of page budget and publication lag. With this step, authors were encouraged to increase—not decrease—details of experimental methodology. Videos with soundtracks offer unprecedented opportunities for creative presentations of research and educational material (hear and see the seven bronchoscopy videos in this issue's State of the Art) (20). Extensive repository supplements have turned the online journal into the complete version of AJRCCM, and the paper copy is simply an abridgement. Three, the Internet made it possible to introduce articles in press in July 2002. Within days of acceptance, manuscripts are posted online. And four, the online Journal is now free to readers in developing countries who have Internet access. An eighth measure of performance is steps taken by the editor to ensure the integrity of the literature. Because clinicians and researchers rely heavily on biomedical publications, they have a vested interest in their integrity (21). There is broad agreement that integrity of a journal rests jointly on the ethical behavior of authors and editors (22)—an aspect of science that should not be confused with the honest errors inevitable in vigorous research (23). An editor's greatest responsibility is to ensure that every item published in his or her journal satisfies the highest standards of scientific integrity (22). How an editor is perceived to handle this responsibility has far-reaching effects on the trust of readers in a journal (24). And without trust, there is no worthwhile journalism (25). In the past, editors protested that handling problems arising from unethical behavior was not their responsibility. Today, editors recognize that while scientific misconduct is not their fault, it is still their problem (26). To ensure the integrity of his or her journal, an editor has no choice but to spend a considerable amount of time investigating allegations of scientific misconduct (27). Allegations usually originate with reviewers or readers. The editor must assume the whistleblower is acting in good faith, and never reveal the person's identity. Accused authors are presumed innocent until misconduct is established, but that innocence must be subjected to examination (27). Editors must bend over backward to give accused authors every opportunity to clear their name (due process to protect innocent people against reckless charges) (28). The best guard against unfairness is to assume that all correspondence with the accused authors will be made public. At times, an editor needs to seek advice from ethicists, jurists, sociologists, or other editors because some problems of journal integrity are quite complex (29). Stemming from our experiences over the past five years, AJRCCM has developed policies that deal with duplicate publication (30), retractions (29), peer review (19), and conflicts of interest (31). The general public is increasingly disturbed about conflicts of interest. After the New York Times published a recent expose of the lax policy of Nature Neuroscience (32–36), all Nature journals markedly tightened their financial disclosure policies (37)—as has the Lancet (38). The policy of AJRCCM differs from many journals in three respects (31). One, criteria for a financial relationship are broad. Our instructions, for example, state: "if an author has received money from a company that manufactures a bronchodilator and the manuscript deals with any aspect of airway biology or disease, the author is required to disclose receipt of the money". Two, the Journal does not ask authors to judge if they have a conflict of interest. Instead, our instructions require authors to simply state whether they have, or do not have, "a financial relationship with a commercial entity that has an interest in the subject of this manuscript." Three, we do not exert paternalism. Some journals believe that editors should decide what information to keep secret or to disclose (39). At AJRCCM, we believe that each reader should have the necessary information to decide whether the authors have a financial conflict. The age of paternalism in medical science has passed, and to gain the trust of readers we must accept openness and accountability (40, 41). Most journals now employ online repositories, meaning that shortage of space is no longer a persuasive defense against detailed disclosure. In the past year, authors have submitted more than 12,000 disclosure forms to AJRCCM, reaching a level of transparency achieved by few other journals. In October 2002, the ATS decided it would no longer publish industry-sponsored supplements within AJRCCM. About a third of articles on asthma and airway biology in 2000 had been in the form of industry-sponsored publications. The change in policy largely resulted from a desire to ensure a unified standard for acceptance of manuscripts, enhancing journal integrity (22, 42). Ninth is editorial independence, which is considered the "sine qua non of a respectable medical journal" (43–45). Its importance was emphasized repeatedly in commentaries that followed the recent unwilling exits of two esteemed editors (4, 5, 46–49). DeAngelis, current editor of JAMA, interprets editorial independence to mean that an editor is providing a guarantee to an author "that peers will review his or her study and that the only factors that influence publication are merit and available space" (50). As such, the relationship between an editor and an author-reader is a covenant of trust—the same type of relationship as between a doctor and a patient. The relationship is between two unequal parties, and the more powerful party is concerned about the welfare of the less powerful party. Moreover, the responsibility of an editor to author-readers is greater than his or her responsibility to the owner (51). Prominent members of a sponsoring scientific society may harass an editor and demand special treatment. But as Kassirer, former editor of the New England Journal of Medicine, says, "To preserve (trust), an editor must not be beholden to any special-interest group and must be free to cover controversial subjects even if they involve the medical organization that runs it" (52). And, "If they (editors) are doing their jobs well, they should give no favors, and they should have no friends" (52). A tenth measure of performance is fairness to authors, which unfortunately cannot be quantified. Peer review involves subjective value judgments by one scientist on the originality, validity and importance of another scientist's work (18, 19). But reviewers do not have the authority to accept or reject a manuscript. Instead, they serve as advisors to the editor, who carries that responsibility. Authors find the lack of concurrence among reviewers unsettling, but editors often deliberately invite reviewers who are expected to differ. An editor keeps in mind that a reviewer may be a competitor of the authors. An editor is the authors' only protection against a capricious reviewer, and also the only guard against the suppression of innovative ideas. In making a decision on a manuscript, an editor does not tally reviewer score marks as straw votes but places greater value on the cogency of arguments in the critiques. More than 98% of manuscripts submitted to AJRCCM over the past five years were entered into peer review. Even when manuscripts were rejected, we hope the feedback given to authors will ultimately enhance the quality of pulmonary and critical care research. As such, a journal is much more than the articles published within its covers. In some fields, submissions to AJRCCM are so many that manuscripts are shared among four associate editors. It is disturbing for an editor to know that his or her selection of an associate editor markedly influences the likelihood of rejection. When the current associate editors had completed nine months of their term, rejection rates differed by as much as 30% among associate editors handling a given field (obviously unfair to authors). Through constant communication—yet maintaining the confidentiality of each associate editor—rejection rates now differ by less than 3% within and across fields of equal scientific quality. Authors occasionally write to AJRCCM because they believe their manuscript was rejected unfairly. In each instance, the associate editor reevaluates the file (editors are no more immune to faulty decisions than any other judge) and writes back to the authors. When readers find fault with the policies or judgments of the editor, the letters should be published whenever possible with a reply or apology (53–61). The above ten factors provide a yardstick—beyond impact factor—for assessing the performance of a journal. But the discerning reader will note I omitted the most important measure: does the journal help readers improve the outcome and reduce the harm done to patients under their care? We hope AJRCCM helps in this regard—if not, our other improvements are irrelevant. FOOTNOTES Conflict of Interest Statement: M.J.T. is editor of AJRCCM. He receives a fixed stipend from the American Thoracic Society. He does not receive financial support for research from pharmaceutical, biotechnology, or medical device companies. He does not serve as a consultant to or on the advisory board of any company. He receives royalties for two books on critical care published by McGraw Hill, Inc. REFERENCES
This article has been cited by other articles:
|
||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
