Making Something Out of Bupkis: The AHA’s Ad Hoc Committee on Professional Evaluation of Digital Scholarship

James Grossman and Seth Denbo, February 2014

Bupkis. That’s the value of a lot of good historical scholarship that appears in certain “nontraditional” formats. At least if we measure “value” according to what one’s work contributes to hiring, promotion, and tenure rather than intellectual development or contribution to historical scholarship and knowledge.

Like most Yiddish words that have made their way into the American lexicon, “bupkis” (sometimes given as bubkes or bupkus) somehow just “sounds right.” It implies not just “nothing,” but emphatically nothing. In New York you might hear it as “That ain’t worth bupkis.” For many historians interested in publishing in formats other than the monograph or traditional synthesis, it is not unreasonable to worry that—when it comes time to look for a job, compile a tenure file, or apply for promotion to full professor—a digital project, encyclopedia, or exhibition will be of little value despite the intellectual content and public and scholarly value of such work.

This makes no sense. It robs our discipline of the innovative energy that many historians either keep under their desk until they’ve safely published that second book or simply leave to others willing to take the risk. It marginalizes scholars who do take the risks. It impedes the development of genres that can contribute even more to scholarship, teaching, and wider public access to the best work of historians. It contributes to a culture that discourages the kinds of collaborative work that are valued—in some cases required—in nearly all other venues of creative enterprise.

Any work that “counts” toward career advancement, however, ought to have standards of evaluation that enable specialists to offer informed judgment about quality. Every work of historical scholarship has particular purposes, is aimed at particular audiences, and draws on particular frameworks of previous scholarship. Evaluation must take these and other factors into account, and must itself be framed in ways that are comprehensible and convincing to nonspecialists.

To this end the AHA has established an ad hoc committee to address the professional evaluation of digital scholarship, with a charge approved at the January meeting of the AHA Council. Our goal is not only to address a “problem” (evaluation of a growing body of scholarship), but to encourage innovation. By producing guidelines and criteria that can be used to evaluate digital projects, this committee will help the discipline to better recognize, understand, and appreciate these new forms of scholarship. These include not only what we know to exist, such as websites, e-­books, blogs, etc., but also forms of scholarship we don’t even know about yet. The nature of the digital environment is such that new formats and methods emerge quickly, and we need a way to encourage our colleagues to take advantage of new opportunities. We need guidelines for including a variety of forms of historical work in a tenure and promotion file, and for evaluating such work when it’s submitted.

This is not a simple task, and the committee will have its work cut out for it. The term digital publication encompasses a multitude of forms and diverse of types of content. Some digital projects have clear analogues in traditional scholarly forms, while others take innovation to a level that makes such analogies less clear. This in itself makes the solutions complex in a discipline that has traditionally based scholarly and professional reputation largely on a limited number of types of publication.

The Ad Hoc Committee on Professional Evaluation of Digital Scholarship by Historians

  • Edward Ayers (University of Richmond, US history, committee chair)
  • David Bell (Princeton, European history) 
  • Peter Bol (Harvard, Chinese history) 
  • Tim Burke (Swarthmore, African history)
  • James Gregory (University of Washington, US history)
  • Claire Potter (New School for Public Engagement, US history)
  • Jan Reiff (University of California, Los Angeles, US history)
  • Kathryn Tomasek (Wheaton College, US history)
  • Seth Denbo (AHA, British history, committee staff)

On top of the complicated challenge of what kinds of digital engagement should be considered, the committee will be faced with addressing specific issues, such as methods for peer review, the collaborative nature of digital scholarship, and the variety of types of contributions historians make to these projects. Alongside these considerations about projects and contributors, the committee will have to consider the institutional context. Different types of departments and institutions will be applying the criteria—just as different types of departments apply different criteria to the promotion and tenure of historians who write traditional articles and books.

The committee will produce these guidelines by fall 2014 for consideration by the AHA Council at its January 2015 meeting. The committee will work in as public a fashion as possible, soliciting input from historians and publishing early drafts to allow time for comments and contributions from the many scholars who know the landscape of digital scholarship. We also want comments from historians who are less familiar with such work, since in the end evaluation of any scholarship must be communicated to and accepted by colleagues not only working with other methods, but even in other disciplines. We will be using the AHA blog, Perspectives, and AHA Communities to keep our members informed of the committee’s work.

This process, and the outcomes that will result from it, will influence the work of historians for many years to come. The committee was approved last year, and since then we have been working hard to enlist a group of knowledgeable scholars committed to this kind of work and planning the exact nature of the project in which they would be involved. Because this is so important for the future of the discipline, it is crucial to get it right.

It equally vital that the guidelines are not merely produced and published, but that they are applied by departments and institutions to individual cases of promotion and tenure. One of the responsibilities of the committee will be to help promote the guidelines and encourage our colleagues to make use of them. Many in the discipline have expressed the need for a document of the kind the committee will produce, but it is only through widespread acceptance and use that it will have the impact necessary to effect change and make these serious scholarly endeavors mean more than “bupkis.”

—James Grossman is executive director of the AHA.

—Seth Denbo is the AHA’s director of scholarly communications and digital initiatives.

Charge to the Ad Hoc Committee on Professional Evaluation of Digital Scholarship by Historians

Historians are increasingly producing online publications and using new media for research and teaching. The American Historical Association seeks to respond in creative and responsible ways to these exciting developments in digital scholarship. This committee is an important part of that response.

The committee will explore the landscape of digital scholarship and online communication, assessing existing models for the integration of digital publications into the hiring, tenure, and promotion systems of history departments (and in other disciplines as well). The committee will not be asked to provide a formal report on its findings about the situation as it exists now. Instead the research that the committee undertakes can be summarized in a brief memo and aimed at producing a practical set of guidelines going forward. Those guidelines will describe tools and resources to help departments effectively integrate the evaluation of digital work in history into the overall assessments required for hiring, tenure, and promotion.

The document should include the following elements:

1. Criteria for evaluating digital projects and online scholarly communication for hiring, promotion, and tenure.

2. A framework for applying those criteria to help departments and promotion committees in using them for actual cases.

The committee should consider:

  • The different types of institutions and departments in which the criteria will be applied;
  • the diversity of content and form, which would include, for example, experimental work that is distributed in pre-­review form;
  • the question of peer review for digital publications;
  • the kinds of projects, publications, and engagement that should be considered;
  • the weight of different kinds of digital activity, including those related to teaching, research, and communication which are not peer-­reviewed but can be assessed to show impact of research, engagement with a wider audience, etc.;
  • how we should review collaborative projects, or those that are editorial or curatorial by nature;
  • how different types of contributions (e.g., software development, project management, consultation) to a work of digital scholarship count in assessing the work of historians.

The success of the work of the committee will require engaging with historians working within and outside of digital scholarship, and obtaining input and feedback from scholars throughout the process of developing and revising the criteria. It is also important that even after the committee has delivered the materials that it remain engaged in the project by assisting actively with promotion and advocacy to ensure serious consideration across a wide range of institutions.

The committee will be asked to produce its guidelines by November 15, 2014, for consideration by the AHA Council. The document will be reviewed, revised, and emended by Council for approval at the January 2015 meeting. It will then be published as a formal AHA document.