Excellent Science

Early on after my arrival at NIMH just over a year ago, I asserted that our first priority was to fund excellent science, and within the context of excellent science, we must fund a broad portfolio of research with a diversity of timeframes for potential public health impact. In my Sophomore Year message, I described how, in listening to and learning from my NIMH colleagues and our various constituencies, I have learned that in reality, funding decisions can be challenging, particularly when trying to compare the quality of research proposals across the full breadth of the NIMH portfolio.

With this in mind, and to best accomplish our mission, NIMH leadership has defined and adopted a set of principles for excellent science that facilitates comparisons between grant applications from across the diverse disciplines supported by the NIMH research portfolio. These principles fall into four categories—rigor, impact, innovation, and investigator—each associated with a cardinal question, and each considered below in turn. They are meant to supplement, not circumvent or supersede, the evaluation conducted by NIH peer review study sections, which do an excellent job of judging science within a particular discipline. Indeed, these principles echo the primary considerations of quality used within these study sections. But here I delineate aspects of these principles that can be employed after the peer review process, with the explicit goal of comparing the relative merits of proposals that have widely varying aims and approaches. The combined peer review and post-peer review perspectives help us look across our divisions and offices and prioritize excellent science while assuring portfolio balance across timeframes.

Four categories/four questions

Rigor: Will the study be definitive?

In the past several years, the NIH has put increasing emphasis on ensuring rigor and reproducibility in the science we support. NIMH has fully engaged in these efforts. To the NIMH, a rigorous study means several things. It means multiple, complementary approaches towards answering a question, to ensure that the answer is not dependent upon the method used. Materials and methods must be validated against established norms. Comprehensive assessments of outcomes should be relevant to the question being asked. Studies must be well-controlled and well-powered. In many cases, especially those using big data approaches or large-scale genetics platforms, the ideal study will include a replication sample. And the research team should have access to proven expert personnel and infrastructure necessary for execution and analysis of the study.

Impact: How much will the study change the field?

The mission of the NIMH is to transform the understanding and treatment of mental illnesses. We cannot accomplish this mission if we don’t ensure that the research we support is impactful. Impact can mean different things for different kinds of research. A study can be impactful by developing novel concepts or expanding the knowledge base in a given area. Some grant proposals seek to generate novel tools, resources, or approaches aimed at enhancing future research endeavors. Others might use these tools and knowledge to advance a concept along a translational pipeline, pioneer a novel treatment method, or demonstrate clinical efficacy. And of course, translational, clinical, and services research should focus on ensuring public health significance, by addressing issues of need in mental health care.

Innovation: How will the study shake things up?

Rigor and impact are crucial components of judging the quality of a proposal, but not the only factors to consider. Innovation holds the promise of enhancing the impact of future research, by introducing truly novel approaches and concepts to the mental illness research community. Projects can innovate by making new connections between different areas of science, by developing new fields of research in previously neglected areas, or by proposing truly unique ideas within an established area. Innovative proposals often describe high risk, high reward projects that have the potential to disrupt a field in a positive way. Of course, innovation for its own sake is not necessarily a good idea—innovative ideas should, therefore, be judged for rigor and impact as well.

Investigators: What do the scientists bring to the field?

Scientific excellence, in the end, relies on the investigators, each of whom has the potential to enrich the field. From this perspective, diversity in our investigator pool is crucial—the greater the breadth of different perspectives and approaches, the more likely we are to make progress. We also look for different things from different types of investigators as indications of quality science. Early stage investigators can be a hotbed of creativity, contributing to innovation, and we want to ensure their science is on the cutting edge, so they can continue contributing throughout their career. Mid-career investigators should show evidence of productivity, and demonstrate the ability to contribute notable advances. The best senior investigators have not only a great track record, but evidence of continued creativity as well as generativity—endowing their trainees with what they need to succeed.

A comprehensive process for evaluating science

The principles related to rigor, impact, innovation, and investigator can be applied universally across our portfolio. There are of course field-specific issues that are taken into account in judging quality by these principles. For example, consider impact. In basic science applications, the key goals are expanding knowledge and developing conceptual advances. In translational science, novel treatment efforts should address real clinical needs and demonstrate potential effect sizes beyond current treatments. And in services and implementation science, the proposals should address therapeutic approaches that would be feasible and represent genuine improvements in care, preferably with stakeholder engagement.

As noted above, we rely on peer review to adjudicate these field-specific considerations. Accordingly, we give great credence to the incredibly important contributions made by peer reviewers run by both the Center for Scientific Review  and our NIMH Scientific Review Branch. These reviews provide an invaluable and rigorous look at the quality of applications within a field.

The secondary, post-peer review process that takes place at NIMH needs to be just as rigorous. Program staff make key recommendations, and our National Advisory Mental Health Council provides a second level of review. In the end, I make the final funding decisions by taking into account these additional reviews. These processes take into account factors that peer reviewers usually aren’t able to consider—the full complement of funded projects and investigators, institutional priority areas, and the diversity of the NIMH portfolio.

As described here, these principles for evaluating excellent science across fields are meant to enrich our decision-making process by helping to assess quality across our diverse portfolio, ensuring we support only the best science in all our endeavors.