Bridging Science and Service: A Report by the National Advisory Mental Health Council’s Clinical Treatment and Services Research Workgroup

Table of Contents

• Preface
• Executive Summary
• Treatment and Services Research: An Alliance for Progress
  • Introduction
  • The Clinical Treatment and Services Research Workgroup
  • Definitions
  • Integration of Research Areas
  • Status of the NIMH Extramural Research Program
  • Defining the Issues
• Enhancing the Public Health Value of Mental Health Research
  • Introduction
  • Key Perspectives and Priority Setting
  • Issue: Integrating Public Health Benefit and Scientific Merit to Establish a Research Agenda
  • Issue: Research-Based Consensus Development
  • Issue: Creating an Infrastructure for Monitoring Public Mental Health and Assessing the Impact of Research Summary
• Making Connections
  • Introduction
  • Fostering the Dynamic Nature of Public Health Research
  • Issue: Building the Bridges Among Efficacy, Effectiveness, Practice, and Service Systems Research
  • Efficacy Research
  • Effectiveness Research
  • Practice Research
  • Service Systems Research
  • Summary
• Methods Development and Innovation
  • Introduction
  • Issue: Incorporating Measures of Costs Associated with Interventions
  • Issue: Tailoring Interventions to Communities
  • Issue: Tailoring Interventions to Distinct Subgroups
  • Issue: Considering Treatment Preferences and Decision Making
  • Issue: Confidence in Drawing Conclusions from Findings
  • Issue: Assessing External and Internal Validity
  • Issue: Fostering Innovative Methods
• NIMH’s Leadership Role in Program Development, Review, and Administrative Activities
  • Introduction
  • Program Development: Selecting or Creating an Appropriate Funding Mechanism
  • Issue: Mechanisms for Training
  • Issue: Mechanisms for Creating Collaborative Research at Provider Sites
  • Issue: Requirements for Meeting the Broader Scope of Public Health Research
  • Issue: Review Criteria
  • Issue: Expediting Administrative Decisions
  • Review Considerations
  • Issue: Review Structure
  • Issue: Ensuring Expertise in Peer Review
  • Issue: Conflicts of Interest
  • Issue: Role of the Scientific Review Administrator (SRA)
  • Administration
• References
• Appendix A: National Advisory Mental Health Council (NAMHC)
• Appendix B: NAMHC Clinical Treatment and Services Research Workgroup
• Appendix C: Roster of Consultants

Preface

An urgent task before the National Institute of Mental Health (NIMH) is to generate information that, properly used, will better enable people with mental illnesses to receive optimal care. Such information is needed by mental health care providers, general health care personnel, service systems administrators, policymakers, and, importantly, patients and their families. Through research we continue to develop and refine an array of safe and efficacious interventions, yet more emphasis must be devoted to translating the yield of basic and clinical research into effective treatments for patients encountered in non-research settings. More attention must be paid, also, to designing treatment delivery systems and policies that facilitate provision of optimal care. Persons with mental disorders and their care providers are entitled to the same high quality of research-based information upon which to make treatment and service decisions as persons with heart disease, cancer, or other general medical conditions.

The NIMH in recent years has targeted resources to increase substantially the breadth and depth of the mental health services research field. Our growing services research capacity is an important counterpoint to our strong basic research portfolio in neuroscience, molecular biology, and basic behavioral research. These two arenas—basic science and service systems research—bracket a third component of the NIMH research program: clinical research and, particularly, treatment trials. The pace of advance in basic and clinically inspired research, and the extent of change in service systems and the questions posed by the various types of patients, providers, and settings demand a thorough reappraisal of the manner in which we explore and evaluate clinical innovations and move them into the hands of service providers.

Recognizing this need, the NIMH leadership requested the National Advisory Mental Health Council (NAMHC) to establish a Clinical Treatment and Services Research Workgroup that would undertake an in-depth review of the pertinent issues. Under the insightful and energetic leadership of A. John Rush, M.D., a core group of the Nation’s most distinguished treatment and services researchers met over the course of a year, among themselves, with outside consultants, and with members of the NIMH and other Federal officials. The outcome of these deliberations is seen in the report that follows, and specifically in the recommendations made for modernizing and revitalizing our approaches to treatment and services research.

No less critical than the calls for action at institutional levels is the open, inviting tone of the report. Unlike laboratory research, relevant treatment and services research demand the broadest possible levels of interest and participation on the part of the clinical community, policymakers, and consumers of services. Special efforts were made by the Workgroup—and will be continued by the NIMH—to solicit the needed breadth of involvement. In conjunction with the completion of the work of the Clinical Treatment and Services Research Workgroup, for example, we were pleased to announce that NIMH is establishing a new Office of Communications and Public Liaison, one strategy for broadening the interface between our scientific programs and the many “stakeholders” in our research. We will continue to seek more innovative ways of achieving productive interactions with individuals and organizations who have contributions to make to our research planning and specific projects. A user-friendly NIMH home page on the World Wide Web offers one approach; active solicitation of committed and creative individuals to serve as scientific and public reviewers on the Institute’s advisory committees or as members of local Institutional Review Boards offers another.

We are pleased to make this document available to all interested parties and we welcome your comments, including suggestions for additional steps we might take to enhance the relevance of clinical and services research and the power of such research to put vitally needed information in the hands of those who will use it to the benefit of Americans with mental illnesses.

Steven E. Hyman, M.D.
Director, National Institute of Mental Health

Executive Summary

Historically, the vulnerability and suffering of persons who live with disabling mental illnesses too often were ignored or misunderstood by much of society. Yet over the past half century, the people of the United States, through the National Institute of Mental Health (NIMH), have supported medical, neuroscientific, and behavioral research on mental illnesses consistently and generously. With that support, and that of others (e.g. foundations, industry, and other Federal agencies), a remarkable scientific effort has demonstrated convincingly that mental illnesses affect a specific organ-the brain-and that in the vast majority of instances, mental illnesses can be treated successfully using an array of specific interventions.

Researchers, policymakers, health care providers, and most critically, individuals with mental illnesses and their families today recognize that translating the remarkable breakthroughs into procedures and policies of everyday clinical practice is an urgent, essential, and achievable task. It is a challenge that has profound implications for the quality of the lives of Americans with mental illnesses and for the health of the Nation.

The Clinical Treatment and Services Research Workgroup of the National Advisory Mental Health Council (NAMHC) was charged by the Director of the NIMH to advise the Council on strategies for increasing the relevance, speeding the development, and facilitating the utilization of research-based treatment and service interventions for mental illnesses into both routine clinical practice and policies guiding our local and national mental health service systems.

The Workgroup reviewed the NIMH research portfolio that extends from academic research settings to large, State-wide service systems, to the moving target of “front-line” clinical care. The review made vividly clear the need for mutually enriching interaction between research and both practice and service systems. In addition, the Workgroup consulted with representatives from advocacy groups, insurers, public and private purchasers, researchers, and State and Federal policymakers. Although these perspectives were not all concordant, they did highlight the fields’ capacity to enhance the delivery of treatment and services available to individuals with mental illnesses.

Toward this end, the Workgroup shaped an action plan with 49 recommendations for fulfilling the Nation’s commitment to individuals with mental illnesses. This action plan is structured by the goals of informed priority setting, using a dynamic and rapidly growing knowledge base, as well as methodological innovation, and administrative and infrastructure enhancements. The specific recommendations follow.

• Increase the usefulness of NIMH research for individuals with mental illnesses, clinicians, purchasers, and policymakers through informed priority setting.
  1. NIMH should establish an ongoing priority-setting process that integrates the perspectives of patients/consumers, providers, purchasers, researchers, and policymakers in determining long-term initiatives and responding to scientific opportunities to improve the relevance of mental health intervention research.
  2. NIMH should renew its role in providing a forum for focusing the key perspectives in the mental health research enterprise to develop clear and productive lines of bi-directional communication.
  3. NIMH should support the synthesis of available information on mental illnesses, their treatment, and service needs to enhance priority-setting meetings.
  4. NIMH should routinely organize consensus development conferences on specific mental illnesses to synthesize the research findings.
  5. NIMH should conduct these activities in conjunction with appropriate partners, such as the Center for Mental Health Services and the Agency for Health Care Policy and Research, to provide the necessary integration for the Federal effort.
  6. NIMH should collaborate with Federal, State, and private agencies to establish a mechanism to monitor the value (as defined by an equation of quality, access, and cost) of mental health care delivered across the Nation and the impact of policies on practice and service systems.
• Selectively expand the NIMH portfolio in the domains of efficacy, effectiveness, practice, and service systems research to foster integration across these fields and to expedite the implementation of research-based practices and policies.
  7. NIMH should augment its efficacy portfolio with research that assesses the generalizability of interventions across diagnostic complexities (e.g., comorbidity or chronicity), as well as individual, social, and demographic factors.
  8. NIMH should enrich its efficacy portfolio with trials that measure change in terms of both symptoms and function over a meaningful period of time.
  9. NIMH should incorporate rehabilitation into its intervention portfolio.
  10. Research should be encouraged that better characterizes promising interventions, patient/consumer populations, ongoing treatments, and service settings.
  11. NIMH should expand its effort in clinical epidemiology through workshops and through developing research addressing the issues in the epidemiology of care.
  12. NIMH should expand its effort in generating research to inform and evaluate treatment guidelines in conjunction with other Federal agencies.
  13. NIMH should review the current measures of functional status and support tests of which measures or items are reliable, valid, and have clinical utility.
  14. NIMH should expand its research portfolio to include the interface between the architecture of services (i.e., the structure, organization, and financing of services) and its effects on the quality of care and clinical outcomes (symptoms and functioning).
  15. NIMH should support targeted research on how to synthesize and incorporate existing knowledge into clinical practice better, with particular emphasis on understanding factors and mechanisms involved in changing practice and systems of care.
  16. NIMH should encourage and sponsor more research that links service systems changes with quality of care and other clinical indicators of patient/consumer status.
  17. NIMH should sponsor validation studies of assessment tools designed to measure the quality of health care across systems of care to assist policymakers and patients/consumers with decision making.
  18. NIMH should encourage the use of existing databases for understanding service systems.
• Identify research innovations in design, methods, and measurement to facilitate the translation of new information from bench to trial to practice.
  19. NIMH should improve measures and analysis of costs in intervention studies, with special attention to successful examples from general medicine.
  20. Large intervention studies should include a cost-effectiveness component that uses the best methodology available.
  21. Approaches to conceptualize and assess key characteristics of intervention settings should be developed, as should models for understanding the effects of these characteristics on outcomes.
  22. NIMH should encourage the development and evaluation of key research measures for assessing “usual care” and develop analytic methods to adjust for variation in components of usual care.
  23. NIMH should promote development of innovative sampling strategies for inclusion of underrepresented groups and sufficiently large intervention studies to incorporate representative community populations.
  24. NIMH should encourage the development of methods to study and incorporate clinician and patient/consumer decision-making processes into intervention research.
  25. NIMH should support research to identify common practices believed to be helpful and bring them under research scrutiny, that is, ascertain what is going on in the practice community and determine how much of that is beneficial.
  26. NIMH should encourage the improvement of methods for both evaluating clinician implementation and patient/consumer adherence to treatment recommendations and estimating the consequences of these variations on the effectiveness of treatment.
  27. NIMH should explore new methods for analysis of data from studies that incorporate innovative combinations of research designs.
  28. NIMH should encourage development of methods to explicitly evaluate trade-offs in alternative design features that differ in their implications for internal and external validity.
  29. NIMH should convene a methods workshop to identify options for advancing intervention and service systems research. The results of this workshop should be debated broadly and options tested in appropriate follow-up activities.
• Strengthen NIMH’s leadership and administrative activities to provide the infrastructure to achieve the goals stated above in a timely manner.
  30. NIMH should develop additional training and career development programs that offer hands-on experience in diverse research settings in order to provide researchers with an enriched training experience.
  31. NIMH should revise and renew program announcements (PAs) in the spirit of the Public-Academic Liaison (PAL) Program to maintain and promote existing partnerships between academic researchers and public care systems, health plans, both carve-outs and health maintenance organizations (HMOs), and employers providing health benefits and their representative groups.
  32. NIMH research centers, which demonstrate great potential to secure partnerships with service delivery systems and to utilize these systems to conduct intervention research, should be supplemented to develop, implement, and sustain such partnerships.
  33. NIMH should issue a request for applications (RFA) or PA to encourage secondary analyses of service systems data and to establish accessible formats for such data to maximize the use of databases in practice settings.
  34. NIMH should stimulate new alliances by providing developmental funds to establish shared research resources such as data banks, staff time, consultant time, etc.
  35. NIMH should commit resources to identify, describe, and disseminate models of successful partnerships.
  36. NIMH should consider contracts as a mechanism for supporting clearly articulated research needs and deliverables.
  37. The importance of public health relevance should be specifically added to the review criteria in new PAs, RFAs, and requests for proposals in the relevant areas to emphasize its significance for applicants and reviewers.
  38. NIMH should develop a mechanism for responding to unique, but fleeting research opportunities with great public mental health significance.
  39. NIMH is asked to work with the National Institute on Alcohol Abuse and Alcoholism (NIAAA) and the National Institute on Drug Abuse (NIDA) to encourage timely joint funding decisions for comorbid substance abuse and mental illness research.
  40. Efficacy, effectiveness, practice, and service systems research applications should remain at the Institute for review, and the clinical and services study sections reconstituted, including an appropriate mix of expertise among reviewers.
  41. NIMH staff should attend carefully to planned evaluations of the Center for Scientific Review neurosciences and AIDS review mergers, as well as the upcoming behavioral sciences merger, to enhance future restructurings of NIMH review.
  42. NIMH should explore shared review opportunities with NIAAA and NIDA to enhance the review of applications on comorbid substance abuse and mental illness applications.
  43. Review panels should have patients/consumers, providers, or policymakers as members to ensure expertise on the utility of research findings in intervention research.
  44. NIMH staff should explore how other institutes deal with conflicts of interest and consult with the National Institutes of Health on revising and implementing the conflict-of-interest rules for intervention research.
  45. Scientific Review Administrators (SRAs) should strive to maintain a link between special emphasis panels and the regular review committees through a subset of the regular reviewers and, whenever possible, the special emphasis panels should be arranged to precede or follow immediately the regular review meeting to facilitate the use of committee members and their scoring standards.
  46. SRAs and panel chairs should actively consider when scientific breakthroughs require a substantive update for the review panel. The SRA and chair may convene a workshop to brief the committee.
  47. NIMH should make judicious use of its ability to make funding decisions out of percentile order to achieve its programming objectives.
  48. Grant budgets must be based upon providing essential support as determined by review groups and NIMH program staff for achieving research’s scientific and public health aims.
  49. NIMH should increase its use of administrative and competitive supplements to provide additional funds and time to researchers to test the generalizability of their findings or to disseminate important research findings.

Chapter 1
Treatment and Services Research: An Alliance for Progress

Introduction

Recent decades have brought remarkable advances in our abilities to diagnose and treat mental illnesses. The National Institute of Mental Health (NIMH) has played a pivotal role in supporting the research that has made it possible to enhance, often dramatically, the lives of persons with these illnesses. Research has contributed to the quality and breadth of treatments available and has reduced significantly the stigma attached to mental illnesses. Scientific and clinical advances have helped the public to understand that mental illnesses are treatable medical conditions. Ongoing research is elucidating the complex interplay between brain and behavior and forecasts even more significant developments in treatment and prevention strategies and in public understanding.

The scientific grounds for a sense of accomplishment and optimism are firm. Still, satisfaction with what is known about mental illnesses and their treatments must be tempered. Mental health service settings and systems of care are evolving rapidly, often outpacing the rate at which new knowledge enters clinical practice. Also, the economics of mental health services delivery is being transformed. And, importantly, individuals with mental illnesses and their families have achieved a strong voice in helping to set priorities and processes of care.

Amidst these changes, many people are unable to obtain, for themselves or for one close to them, appropriate, state-of-the-art treatment for a mental illness. All too often, clinical practices and service system innovations that are validated by research are not fully adopted in treatment settings and service systems for individuals with mental illnesses. The substantial disparity between what is known through research and what is actually provided in routine care is not limited to mental illnesses. Significant gaps between what is known and what is practiced have been documented in other areas of medicine as well, including cancer (Cronin et al. 1998), diabetes (McClellan et al. 1997), and cardiac care (Howard and Duncan 1997).

Rapid changes across health care, juxtaposed against continuous scientific advances, raise urgent questions for all parties involved in the Nation’s mental health:

• For patients/consumers and their families, the questions are straightforward. What treatments will help? What treatments are best? How can we be sure that a specific treatment is both appropriate and of high quality? How can we afford to pay for the treatment?
• Clinicians have different, but related questions. What is the best treatment for this patient/consumer? What treatment should be recommended next if the first is unsuccessful? What are the clinical and financial barriers to applying different approaches to care?
• Health care administrators must address questions at a different level. For example, what types and levels of care are appropriate for specific patient/consumer groups? What resources are needed to provide such care, and how should they be delivered and integrated?
• Policymakers, purchasers, and insurers must make important decisions regarding access to and coverage for patient/consumer care. For them, the questions are no less urgent. What will it cost to pay for effective treatment? What will it cost, in terms of protracted disability and related costs, not to pay for treatment? What delivery system and financial incentives will provide optimal cost-effectiveness of care? How do purchasers receive value for their investment?

The NIMH, along with other Federal agencies, is responsible for the research that is generating answers to these questions. In 1991, the Institute issued Caring for People with Severe Mental Disorders: A National Plan of Research to Improve Services. Prepared under the auspices of the National Advisory Mental Health Council (NAMHC), this benchmark report was one of the factors that invigorated the field of mental health services research in recent years. Since 1991, NIMH support for treatment and services research has increased from $82 million to $170 million in fiscal year (FY) 1997. Years of accelerated growth now have made it timely and important to examine the Institute’s progress and, where necessary, to adjust the Institute’s efforts in light of rapidly evolving science, systems of health care, and technologies for information dissemination.

The Clinical Treatment and Services Research Group

In September 1997, Dr. Steven E. Hyman, Director of NIMH and Chair of the NAMHC (see Appendix A for roster), convened a workgroup of researchers, policymakers, and mental health advocates to advise the Council on the array of issues that are the domain of clinical treatment and mental health services research. The director’s charge to the NAMHC Clinical Treatment and Services Research Workgroup (see Appendix B for roster) was to speed the development and routine utilization of research-based treatment and service interventions in daily practice. Dr. Hyman asked the Workgroup to consider factors that influence the development, translation, and implementation of research findings into practice. He urged the Workgroup to consider how to build most effectively on the experiences of other institutes within the National Institutes of Health (NIH). He also encouraged development of strategies to improve the synergy of the NIMH’s relationships with patients/consumers and families, managed care organizations (MCOs), public and private purchasers, and other major “stakeholders” in the mental health system.

Definitions

As the Workgroup began its deliberations, members had to come to terms with the lexicons of many different research fields. Although the Workgroup’s name indicates a focus on treatment and services research, these two terms meant different things to different members of the Workgroup. Other terms used in the separate literatures also were ambiguous, so the Workgroup gradually developed its own working lexicon. These definitions are provided for clarification and have the additional benefit of surveying the many exciting areas that mental health researchers are exploring. These distinct groupings are provided for discussion purposes, because as discussed later in this report, these areas do or should overlap.

Efficacy Research

The purpose of efficacy research is to examine whether a particular intervention has a specific, measurable effect and also to address questions concerning the safety, feasibility, side effects, and appropriate dose levels. As a consequence, the classic efficacy study is a clinical trial in which an experimental treatment is compared to a control treatment that can be a standard treatment and/or a placebo. Because the chief concern is to detect any treatment effect, researchers try to eliminate or hold constant any factors that may obscure this effect. For instance, to reduce the influence of researchers’ and participants’ expectancies on the assessment of effect, clinical efficacy trials are often double-blind. This means that both the research staff and the participant do not know who is getting the experimental treatment and who is getting the control treatment. Also, efficacy trials usually employ highly restrictive inclusion and exclusion criteria for selecting participants. This is because the more similar the participants are, the easier it is to detect the effect of a treatment as well as its side effects. Complicating factors such as co-occurring substance abuse or other illnesses are typical exclusionary criteria. Despite these criteria, not all factors can be anticipated or controlled. For this reason, efficacy trials typically randomly assign participants to the experimental or control condition. The goal is to be sure that condition assignment is based on “the luck of the draw” rather than an undetected confounding factor.

In addition to double-blind, randomized, clinical trials, efficacy research typically has other common features: (1) study settings are highly controlled to ensure optimal treatment delivery (e.g., with clinicians who follow a strict set of procedures in providing the experimental and the control treatment); (2) treatment is provided without cost to participants; (3) the “effect” is usually defined by shorter-term clinical outcomes such as symptom reduction after a month or two rather than longer-term outcomes such as the ability to function at home or work over months to years; and (4) economic variables such as costs or health care utilization are not typically measured.

Effectiveness Research

The principal aim of effectiveness research is to identify whether efficacious treatments can have a measurable, beneficial effect when implemented across broad populations and in other service settings. For instance, any person seeking help with a particular mental illness, regardless of other co-occurring conditions or the duration of the illness, might be eligible. Treatments are administered by clinicians who have not necessarily been specially trained in the research protocol; patients/consumers and clinicians exercise choices over treatments; and the frequency and duration of visits, how and when outcomes are gauged, and the use of adjunctive services are dictated by local practice patterns or administrative policies. Effectiveness studies can be randomized controlled trials, but as in some types of efficacy work, blinding of the participants and/or researchers is not always possible. Traditionally, effectiveness studies focus on more broadly defined outcome measures such as disability and quality of life, and have placed less emphasis on detailed evaluation of clinical status. With the increasing emphasis on cost of care and efficient service delivery, many effectiveness studies now incorporate analyses of the cost-effectiveness of various mental health interventions when compared to care as usual or an alternative treatment. Of course, design elements for effectiveness trials can include random assignment and double-blinding, as discussed under efficacy.

Practice Research

Practice research examines how and which treatments or services are provided to individuals within service systems and evaluates how to improve treatment or service delivery. The aim is not so much to isolate or generalize the effect of an intervention, but to examine variations in care and ways to disseminate and implement research-based treatments. Although some studies may have randomized designs, currently most are observational. An emergent field, practice research is built on and encompasses at least three established areas of research inquiry:

• Clinical epidemiology represents a broad field that addresses what happens to people with illnesses who are seen by providers of clinical care. Studies use traditional epidemiological methods and are conducted in groups that are usually defined by illness or symptom or by a diagnostic procedure or treatment given for the illness or symptom. Thus, it includes studies of the natural history of an illness, studies of diagnostic and screening tests, and observational and experimental studies of interventions delivered to people with illnesses or symptoms.
• Quality of care research is concerned with defining and describing the care received in clinical settings and establishing and testing standards for quality of care. Areas of study include processes and outcomes of care; research on the structure of health organizations as it pertains to processes and outcomes; and the relationship among structures, processes, and outcomes. Components also may include research on treatment guidelines and clinical decision making. Considerations of economic impact in decisions about use of medical technologies and pharmaceuticals are important factors in this process.
• Dissemination research evaluates methods by which appropriate interventions are introduced and adopted in clinical practice, including factors that inhibit or facilitate such adoption. This research includes studying mechanisms to change clinician or patient/consumer behaviors, and to improve the delivery of clinical care. Quality improvement studies and practice guideline dissemination evaluations are currently main components of dissemination research. A relatively new field of research in mental health, dissemination research study designs may be either observational or randomized, depending on the nature of the question asked.

Service Systems Research

Service systems research addresses large-scale organizational, financing, and policy questions. This includes the cost of various care options to an entire system; the use of incentives to promote optimal access to care; the effect of legislation, regulation, and other public policies on the organization and delivery of services; and the effect that changes in a system (e.g., cost-shifting) have on the delivery of services. At the interface of service systems research and other areas of practice and treatment research are studies that integrate these different levels of inquiry (e.g., studies of how variations in service system characteristics affect quality of care). Studies of how treatment effects differ, in the context of different systems, integrate effectiveness and service systems research. Cost-effectiveness analysis has a very important role in evaluating various forms of treatment delivery systems. Service systems research study designs may incorporate elements of efficacy and effectiveness research.

Integration of Research Areas

In an effort to bridge the existing divisions between the domains of efficacy, effectiveness, practice, and service systems research, the Workgroup developed a model that took into account these four types of research and the potential users of research (i.e., patients/consumers and families, clinicians, public and private purchasers, insurers and MCOs, and policymakers). This model (see Figure 1) demonstrates the interconnectedness of these domains, which are commonly, but inappropriately, viewed as discrete or linear.

Figure 1: Relationships Among Research Domains and Users of Findings

Treatments do not occur in isolation from patients/consumers, clinicians, and service setting factors. As research findings are made available to “end-users,” investigators in all four domains should consult with end-users to identify the next generation of research questions. Treatments are always administered in a context, and it is that emphasis on context that marks an important change in the treatment research paradigm. Further, a service system must have a strong method for understanding the clinical nature of an individual’s illness and functional status for the most appropriate provision of care. Accordingly, the four areas must be mutually informative, each building on the understanding provided by the other and addressing pressing public mental health concerns.

The model emphasizes the need for interaction across these domains of study and between research results and users of the research. Further, there are important cross- cutting areas of research such as health economics, epidemiology, and prevention. As a case in point, treatments designed without attention to costs may generate interventions that are only unaffordable and therefore impossible to implement. Economic studies are best focused on questions that arise regarding current interventions or those under consideration for implementation, ideally with these latter interventions being evidenced-based.

Status of the NIMH Extramural Research Program

The Workgroup conducted a comprehensive review of all NIMH-supported treatment and services research. Aims of this review were to identify gaps in the research portfolio, ascertain how to develop needed knowledge more effectively, and how to use available knowledge most efficiently and appropriately (i.e., how to foster awareness, acceptance, and use of new knowledge by practitioners and health care systems).

Within the NIMH, primary responsibility for efficacy, effectiveness, practice, and service systems research is assigned to three branches within the Division of Services and Intervention Research.

The Adult and Geriatric Treatment and Preventive Intervention Research Branch had a treatment research portfolio in FY 1997 that included grants and clinical trials contracts supported for a total of $51.7 million, including research training and career development (see Figure 2). The majority of the grants focus on treatment including mood disorders, principally major depressive disorder ($22.3M), schizophrenia and other psychotic disorders ($11.1M), a broad spectrum of anxiety disorders ($11.1M), and dysregulation disorders (principally eating and sleep disorders) account for the remainder ($4.8M). Pharmacology, psychotherapy, somatic treatments, and combination strategies are all represented. Treatment grants in the Branch portfolio at present are largely short-term efficacy trials with highly selected populations and outcomes focused on symptomatology.

In the Child and Adolescent Treatment and Preventive Intervention Research Branch, the treatment research portfolio for FY 1997 included grants and contracts for a total of $17.7 million, including research training and career development (see Figure 3). This Branch focuses on clinical trials of treatments or preventive interventions for children and adolescents. Slightly more than half of the Branch’s funding is devoted to preventive interventions (not represented in this figure). Treatment areas receiving the most funding in FY 1997 were attention-deficit hyperactivity disorder for treatment interventions ($6.6M), autism treatment ($1M), conduct disorder treatment ($2M), mood disorders treatment ($2.5M), anxiety disorders treatment ($2.2M), and other treatment for $2.8M.

The Services Research and Clinical Epidemiology Branch had a research portfolio in FY 1997, including research training, for a total of $59.9 million (see Figure 4). The focus of this Branch’s funding is on access to care, structure and organization of care, process of care, outcomes, and financing and costs of treatment. Supported research areas included service systems and economics ($11.2M), primary care ($8.8M), children and adolescents ($10.7M), sociocultural ($8.6M), clinical epidemiology ($5.0M), outcomes and quality of care ($2.1M), research training ($6.6M), and the remaining ($6.9M) included a small portfolio of methodological research and other portfolios.

Figure 2: Adult and Geriatric Treatment and Preventive Intervention Research Branch, DSIR Treatment Grants and Contracts Fiscal Year 1997 – Total $51.7 million

Figure 3: Child and Adolescent Treatment and Preventive Intervention Research Branch, DSIR Treatment Grants and Contracts Fiscal Year 1997 – Total $17.7 million

Figure 4: Services Research and Clinical Epidemiology Branch Fiscal Year 1997 Services Grants* – Total $59.9 million

Defining the Issues

In addition to reviewing the Institute’s portfolio, the Workgroup sought input from a wide array of perspectives with which it evaluated the Institute’s activities and future needs. Workgroup members conferred with staff of NIMH and other Federal agencies. Patients/consumers and family groups, providers, public and private purchasers, and policymakers also were consulted (see Appendix C for roster).

The Workgroup took the innovative approach of inviting the participation of all interested parties and encouraging discussion. For example, the Workgroup opened a web site on the NIMH home page to post summaries of its activities and solicit comments from the public. Workgroup members held open discussions at various professional meetings to engage the research field. Further, NIMH staff attended Workgroup meetings and were encouraged to implement, in a timely manner, recommendations endorsed during the public Council updates rather than wait for the Workgroup’s final report. The rapid adoption of the Workgroup’s interim suggestions attests to the speed with which NIMH and NAMHC seek to provide more meaningful research findings. While other ideas require further discussion and a sustained commitment on the part of NIMH, stylistically, the Workgroup hoped to set a new tone for the Institute in its planning process. The need for open dialogue with users of research findings in an ongoing, intensive, priority-setting process at the NIMH is a recurring theme in this report.

Given the breadth of issues to be considered in developing an action plan, the Workgroup agreed upon four domains for developing strategic recommendations. Subgroups were formed to focus on each:

• Increase the usefulness of NIMH research for individuals with mental illnesses, clinicians, purchasers, and policymakers through informed priority setting.
• Selectively expand the NIMH portfolio in the domains of efficacy, effectiveness, practice, and service systems research to foster integration across these fields and to expedite the implementation of research-based practices and policies.
• Identify research innovations in design, methods, and measurement to facilitate the translation of new information from bench to trial to practice.
• Strengthen NIMH’s leadership and administrative activities to provide the infrastructure to achieve the goals stated above in a timely manner.

Each subgroup generated issues and recommendations, which were subsequently debated by the entire Workgroup. The Workgroup then met to shape a report that would reflect the diversity of perspectives and constructive ideas provided by the individuals and groups who had been consulted. The Workgroup believes that achieving these goals would position NIMH to fund high quality research to address urgent public health concerns. The Workgroup’s recommendations to foster each goal are described in the following chapters.

Chapter 2
Enhancing the Public Health Value of Mental Health Research

Introduction

A key measure of the value of research is its beneficial impact on society and, more precisely for NIMH-funded research, on public mental health.

This report defines public mental health as the number of mental health conditions of individual, clinical, and societal significance in the general population. Mental health conditions are diagnosable mental illnesses or symptoms that reflect disturbance in emotional, behavioral, or information processing regulation that causes significant suffering or impairment for affected individuals. These conditions have observable clinical courses and outcomes. The major costs of these conditions are borne by the individuals and their families, employers, and society. These conditions result in the disability, pain, and suffering that also accompany other serious chronic health conditions (e.g., asthma, diabetes, and hypertension).

Mental illness interventions are treatments and service delivery programs, and, in some instances, public policies designed to reduce the pain and suffering, disability, and costs of mental illnesses or to improve the delivery of mental illness care. Interventions occur in the broad context of the affected individual’s life, whether in the work, school, family, or community setting.¹

The products of research-new techniques and diagnostic tools and the promise of new treatments-are inherently exciting and important to researchers and research administrators. As stated above, the act of generating new knowledge is not an end in itself, however. Ultimately, the knowledge must lead to improved public mental health in measurable and meaningful ways.

This chapter reviews various issues that must be addressed to enhance the social and public health value and impact of NIMH’s efficacy, effectiveness, practice, and service systems research. It offers three specific infrastructure goals intended to make NIMH-supported research more relevant to the needs of diverse individuals and organizations-stakeholders in the research enterprise:

These activities are highly interrelated. The success of the planning process will be contingent on the extent to which it incorporates stakeholders’ perspectives and leads to research that positively affects the status of mental health and quality of care. The public mental health status must be tracked to understand what has been accomplished and what remains to be done. The planning process should blend an understanding of public health need with awareness of scientific opportunities that will increase both the efficiency and value of NIMH’s research investment.

Key Perspectives and Priority Setting

There are many contributors to and patients/consumers of research on mental illness. Although the diversity of opinions within these groups cannot be fully captured in a brief paragraph or two, identification of these groups and their general issues are mandatory for understanding the breadth and complexity of this arena.

Individuals with Mental Illnesses and Their Families

Individuals with mental illnesses² and their families have asserted themselves as active participants in developing treatment and rehabilitation plans with their care providers. Grassroots groups such as the National Alliance for the Mentally Ill, the National Mental Health Association, and the National Depressive and Manic Depressive Association have formed a concerted effort to eliminate stigma against mental illness, promote access to care, foster self-help, advocate for research targeted toward the elimination of mental illnesses, enhance the utility of research from a patient/consumer perspective, and influence policymakers. Policy targets include fostering research/treatment/services advances and enacting anti-discrimination laws. Such groups have wielded considerable influence on policymakers by providing accounts of the devastation of mental illness, as well as personal gains and triumphs achieved through research advances.

These individuals want to be active participants in the priority-setting process to enhance the effectiveness and availability of treatments. They seek out, for example, research on viable treatments that both provide symptom relief and improve their functional status. They believe that better treatments with fewer side effects can be developed.

Clinicians

Clinicians are the individuals who directly provide treatment or services to individuals. These may be specialists in mental health care or primary care doctors and nurses. From a clinician’s perspective, there is a concern that many treatment guidelines and procedures issued by the government, professional groups, or health care organizations may not be in agreement with each other and may not be optimal for the individuals they serve. Further, there is substantial uncertainty about the degree to which the evidence base of many guidelines relates to the kinds of patients/consumers treated by community providers. Often only suggestive rather than definitive evidence exists to inform some of the most important clinical decisions arising in daily practice (e.g., what to recommend should the initial treatment fail). Clinicians need treatments that have utility and feasibility in a variety of practice settings given available resources. They have adequate information, training, and support to use the most appropriate treatments most efficiently.

Providers also are concerned with patient/consumer receptivity to mental illness treatments and-especially in primary care settings-availability, access to, and quality of specialty providers. Such conflicts and considerations often create tensions among the goals of high quality care, compliance by providers with contracts and cost- containment goals, autonomy by providers and individuals with mental illnesses in clinical decision making, and compliance with review standards and patient/consumer preferences. In sum, developing standards of care that apply across provider groups and for a range of mental health conditions is challenging.

Purchasers

Treatment of mental illnesses takes place in a two-tiered system (employer and commercially paid insurance, or public system). This complicated system impacts both the continuity and quality of care, as well as to what degree scientifically based knowledge is applied. The continuum of settings for treatment of mental illnesses extends from primary to specialty care and from private to public systems. Within each segment, there are additional discrete subsystems for children and adolescents, the elderly, those with serious and disabling mental illnesses, and other patient/consumer groups. Moreover, patterns of care change frequently as a function of managed care, public sector reforms, availability of new medications, and other scientific, economic, political, and social factors.

Employers and public agencies (i.e., local, State, and Federal programs) often are referred to as purchasers. They finance the services and exercise choices over benefits offered to eligible providers or service locations, and sometimes regulate prices of mental health services.

Employers. Private employers are looking increasingly at both the direct (treatment) and indirect (e.g., lack of productivity while on the job or days missed from work) costs of health care. A major concern of employers is whether employees return to work in a timely way and are productive. All employers, large and small, public and private, cannot survive with mounting medical insurance costs, employee absenteeism, or workers incapacitated by mental or general medical illness.

Additional needs identified by private sector employers include: (1) better health system performance measures to assess the quality of care and the impact of treatment on the individuals functioning; (2) an ongoing and collaborative exchange between employers and health care researchers; (3) the conduct of population-based services research in the public sector; (4) information on comorbid conditions (e.g., 60% of children with serious mental illness have a co-occurring condition within 3 to 5 years); (5) issuance of local or national treatment/performance guidelines to assist in writing specifications for benefit components; and (6) service systems research on organized systems of care, costs of treatment in offsetting other costs such as medical visits, disability, recurrence, etc.

Public Purchasers. During the past several decades, the public mental health system has changed dramatically. Changes initiated via the 1963 Community Mental Health Centers Act (Title II, Public Law 88-164) stimulated creation of new community mental health organizations. The combination of new treatments and civil rights-related reforms led to the policy of deinstitutionalization.

The early years of deinstitutionalization largely involved hospital downsizing, without adequate creation of community care alternatives. Stimulated by the 1977 Government Accounting Office (GAO) audit of deinstitutionalization (U.S. GAO 1977), the report of the President’s Commission on Mental Health (1978), and the Community Support Program initiated by the NIMH (Turner and TenHoor 1978), States made dramatic changes in public mental health programs in the last 15 years.

The number of inpatient residents at year end in State and county mental hospitals has dropped dramatically from approximately 370,000 in 1969 to about 83,000 in 1992 (Redick et al. 1996). Similarly, the number of inpatient beds in State and county mental hospitals decreased significantly from approximately 413,000 in 1970 to about 93,000 in 1992. In 1993, for the first time, State mental health agencies’ expenditures for community programs exceeded spending on State psychiatric hospital inpatient services (National Association of State Mental Health Program Directors Research Institute, Inc. 1996). The expansion of Medicaid to cover “optional” clinic and rehabilitation services helped to stimulate this significant redistribution.

As the publicly funded mental health system continues to evolve, policymakers in State and local mental health need more access to research results and evaluation methodologies. Key questions for public purchasers include: (1) how to incorporate or promote the use of research-based treatments and treatment guidelines and quality improvement processes in decentralized community care systems; (2) how to control cost increases (e.g., in Medicaid) without incurring the worst consequences of badly managed care; (3) how to promote concepts of recovery; (4) how best to engage and impact on mental health care in schools, prisons, and other non-mental health settings; and (5) how to improve the quality of care for the very significant percentage of patients/consumers with co-occurring illnesses.

Insurers and MCOs

Insurers and MCOs either serve as intermediaries between purchasers, providers, and patients/consumers, or they organize and deliver services. Insurers and managed care companies are subject to legislative and court-ordered regulations, as well as market demands for services delivery. Issues of particular contemporary concern include understanding the ways payment systems impact access and the quality of care delivered.

Particular interests of MCOs concern balancing cost and quality implications of treatment and service delivery programs and ensuring that populations served in managed care (or covered through insurance) are represented adequately in intervention. Studies of the effect of markets on delivery of mental health services in the public and private sectors and research that tracks changes in market factors that potentially may affect care also provide needed policy-relevant information.

Effectively engaging the managed care industry in a research agenda, which will require addressing proprietary concerns regarding data systems and management mechanisms, is as important as it is challenging given MCOs’ growing presence in the service system. Common ground may be found in the evaluation activities regarding the impact of treatment programs and policies, such as utilization review, that influence the probability of successful/effective treatment.

Policymakers

Policymakers formulate or interpret the rules, regulations, or laws that affect public health. In this context, mental health often competes for resources with other types of health problems and, at times, with housing, education, social programs, defense programs, etc., particularly at State and Federal levels. Court interpretation of legislation and resolution of conflicts (e.g., suits) is another policymaking forum that affects the availability and delivery of mental health interventions.

Policymakers are concerned that the data on treatment and services all too often are not relevant to the constituent population or are not provided in a timely manner. Policymaking occurs in real time. Research findings may lag considerably behind policy decisions they were initially designed to inform. Thus, it is important both to increase the relevance and timeliness of research endeavors to policy issues and to initiate a process of identifying and evaluating trends in public policy as they are forming.

Information on cost-effectiveness of mental health interventions is needed to formulate a rational mental health benefit plan. Most policy debates are informed by available data on cost implications, but information on the likely impact on public mental health also is needed. However, outcome data are more likely to be available at the efficacy level, rather than at levels that address policy decision issues.

Researchers

Researchers conduct studies designed to answer questions posed by the various constituents. They are concerned with ensuring that their studies provide valid answers to the concerns raised by these groups. To do so, they must be able to formulate questions into a theoretical model that can be studied. Without an existing base of knowledge in a particular area, it may be difficult to develop such models. In addition, researchers must be clear about what to measure so they can answer a question. Even more importantly, they must have measures that are reliable and valid.

Many researchers operate in academic settings and must compete for private or Federal grants to support their research efforts. They also strive to publish their findings in a timely manner. Large, complex studies also can be difficult to conduct within one academic setting. Seeking out new collaborators in other locations can be costly and difficult. Due to these constraints, researchers sometimes must simplify their research questions. Thus, many investigators believe that the funding incentives have been weighted toward conducting more simplistic studies that may not generalize well to larger populations of individuals seeking treatment or services.

Figure 5 reflects the larger context in which these key stakeholders operate. These components and their interconnections are where science and mental health policy converge to shape better care and service for individuals with mental illnesses.

Figure 5: The Context of Public Mental Health

Issue: Integrating Public Health Benefit and Scientific Merit to Establish a Research Agenda

Even though one can define science in value-free ways (e.g., theory building and testing based on replicable results), values are clearly felt in the arena of research. Values determine which research idea, out of the many presented to NIMH, receives taxpayer support. Yet, if the research agenda were to reflect predominantly the needs of only one of the many stakeholders, it would be unbalanced and much could be lost.

Research that may be of great benefit to individuals with mental illnesses may be less compelling to researchers. For instance, a fairly simple treatment comparison trial may have high impact on developing evidence-based practice. Comparison studies clarify which treatment a patient/consumer should try next or inform a State mental health commissioner about whether to adopt a new treatment guideline. If the researchers alone were setting the agenda, however, these trials might be insufficiently valued to recommend funding.

As another example, NIMH must develop channels for input from policymakers regarding the biggest decisions they are facing. For example, as State mental health authorities must decide how to allocate their fixed treatment dollars, the NIMH research portfolio should help inform their decision making about which services should be provided, to whom, by whom, for how long, and how to implement such changes in practice. Similarly, health maintenance organizations (HMOs) and other managed care entities should be able to look to NIMH for research findings on how best to treat those conditions that are most common in the people they serve and how to know whether such care is being delivered. Providing timely, useful information to these purchasers and policymakers should also enhance their participation as research partners. Dr. Steven Shon explains the issues from a policy perspective (see story box).

The priority-setting process must involve all stakeholders (patients/consumers, providers, purchasers, and policymakers) to enhance the ability of NIMH to anticipate the types of questions on the horizon. NIMH can then project trends that influence mental health service delivery, benefits, or public mental health, and anticipate the need for relevant intervention research data. Tracking developments in public policy is necessary to anticipate informational needs and to review existing research and formulate new initiatives to fill critical gaps.

The Workgroup encourages NIMH staff to explore models at other agencies or institutes where priority setting is underway. At the National Institute of Allergy and Infectious Diseases (NIAID), for example, the research priority-setting process is done annually and spans three fiscal years. The process opens with a summer policy retreat designed to address research priorities and public health needs, including emerging scientific opportunities, policy issues, special problems, and consideration of refinements to previously approved initiatives. The broad-based process, which may involve invited participants from outside of the NIAID, integrates priorities for both the extramural and intramural programs. Feedback on issues raised at the summer policy retreat are submitted to the fall meeting of the NIAID Advisory Council. During the winter program review, current and future divisional priority activities are discussed. A general consensus is obtained on the prioritization of these activities. Further communication then takes place among the divisions. These findings are presented to Council and concepts are cleared for initiating program announcements (PAs), requests for applications (RFAs), and requests for proposals (RFPs). In addition, issues of interest are posted on the Institute’s web site to inform the extramural community of future priority areas and to provide an opportunity to comment.

Other institutes and agencies have planful approaches, and NIMH staff should consider these and adapt or adopt attractive elements into their priority-setting process.

Recommendation 1

NIMH should establish an ongoing priority-setting process that integrates the perspectives of patients/consumers, providers, purchasers, researchers, and policymakers in determining long-term initiatives and responding to scientific opportunities to improve the relevance of mental health intervention research.

Issue: Research-Based Consensus Development

Establishing a research agenda that is responsive to the needs and priorities of key stakeholders is likely to increase the usefulness of research results. But such agenda setting requires both a forum and information from which consensus can be achieved. Choices exercised by each of these stakeholders can affect whether treatments for illnesses are available, who receives them, and the costs and quality of care. Not surprisingly, these decisions are difficult because the research findings needed to inform a proper response are not always available. As a result, multiple parties are forced to make unenviable choices:

• Patients/consumers are uncertain as to whether they should continue to take medication to alleviate symptoms but increase risks of significant side effects. Should they pay out-of-pocket for new medication(s) with reportedly fewer side effects or continue with the less expensive but more troublesome drug(s) offered through providers or MCOs? What forms of psychotherapy will help? Would their time be better spent in a supported employment program?
• Providers seek to give high-quality care to individual patients/consumers, but also must comply with various insurer requirements for cost-cutting measures. How should they allocate their time?
• Employers seek to optimize employee satisfaction and productivity as well as profitability. Will more extensive coverage produce better care? Will treatment help productivity?
• Insurers and MCOs must compete for a better market share and for contracts, without sacrificing patient/consumer satisfaction. Will paying for prevention or health activities have short-term payoffs or simply minimize cost for the next insurer?

Given the often conflicting goals within and across stakeholder groups, the field would benefit from having a forum to identify, develop, and chart a shared path to achieve shared goals. Over the last 9 months, the Workgroup began to develop such a forum by devoting significant time to hearing from representatives of these diverse sectors.

The Workgroup urges NIMH to play an active role in establishing, maintaining, and using this critical forum. As an overarching principle, coordination and cooperation with all of the Federal agencies addressing mental health and substance abuse problems are encouraged. The Institute should review past, successful use of its “convening power.” A recent example was that of an NIMH-sponsored meeting of all parties concerned with the lack of research to direct pharmaceutical treatment in children, an urgent and important issue in light of the large number of off-label prescriptions written for children in this Nation (Vitiello and Jensen 1997). Drs. Peter Jensen and Benedetto Vitiello, both of the NIMH, invited representatives from parent groups, the pharmaceutical industry, ethicists, researchers, and clinicians to focus on this issue and define what was needed and what they could contribute to this effort.

The Workgroup encourages the NIMH to participate more actively in other institution’s consensus efforts. The NIMH has had a liaison member on the National Advisory Council for the Center for Mental Health Services (CMHS) since its inception, yet the CMHS Council has expressed concern that closer ties need to be created. The need for collaboration is particularly clear given overlapping responsibilities in public information and evaluation of effective interventions. The Workgroup shared this sentiment and extends it to private foundations and businesses.

Recommendation 2

NIMH should renew its role in providing a forum for focusing the key perspectives in the mental health research enterprise to develop clear and productive lines of bi-directional communication.

Bringing people together is not sufficient by itself. Each group requires easily accessible syntheses of research findings to inform their positions. These syntheses should not be limited to NIMH-supported research. Although the NIMH plays a unique role in sponsoring public research on mental health care delivery and treatments, other governmental agencies [e.g., Agency for Health Care Policy and Research (AHCPR) and CMHS], several large private foundations, pharmaceutical companies, and health care delivery systems, such as large HMOs, sponsor research that has overlapping goals.

The Internet is emerging as a primary source for organizing and manipulating information quickly. With this capacity, the challenge for the mental health field is to synthesize research findings and present them in a way that researchers and other important groups can evaluate the information. The work of the Cochrane Collaboration, an international group established to conduct systematic reviews of the literature, offers an example of how large amounts of research data are being collected and managed. It also provides an historical context to determine if the conduct of a future study would be duplicative. Other synthesis processes exist as well, and can be brought to bear on the Institute’s scientific priorities.

Recommendation 3

NIMH should support the synthesis of available information on mental illnesses, their treatment, and service needs to enhance priority-setting meetings.

Just as NIMH must renew its efforts to synthesize information, it also must take the lead in consensus development. Consensus development conferences are effective in bringing together all stakeholders; thus, as findings emerge from research and large-scale synthesis efforts, NIMH, in collaboration with other Federal agencies such as AHCPR and CMHS, should actively initiate such conferences. The results of these conferences would then shape NIMH’s priorities and be of guidance to the large purchasers of care such as State mental heath administrations and MCOs.

Recommendation 4

NIMH should routinely organize consensus development conferences on specific mental illnesses to synthesize the research findings.

Recommendation 5

NIMH should conduct these activities in conjunction with appropriate partners, such as CMHS and AHCPR, to provide the necessary integration for the Federal effort.

Issue: Creating an Infrastructure for Monitoring Public Mental Health and Assessing the Impact of Research

Mental illnesses exact a tremendous toll on society as a whole. Public systems must reallocate increasingly tighter resources and are being forced to make very difficult decisions regarding what services to provide. Private purchasers are concerned about not only the costs of paying for mental health benefits, but also the costs of lost productivity associated with illness.

As sweeping changes in health care unfold, researchers and policymakers want to examine the effects of these changes on the public’s mental health status. An enduring problem in assessing the value of a mental health intervention is that there is no general index of the Nation’s mental health status. Although short-term epidemiological studies have provided critically important “snapshots” of the country’s mental health status, it is difficult to ascertain whether the Nation’s mental health has improved, since studies tend to be conducted too far apart in time and with sufficiently different

methodologies to allow evaluation of change. Similarly, national health surveys have periodically included mental health items, but not in a consistent manner to allow this level of monitoring.

Lacking an index, it is not possible to assess fully the large-scale changes in the mental health status of a State or the Nation. Each study that addresses this question must do so in a vacuum. At present, the only method to assess changes is to compare results of the few studies conducted at restricted points in time. Measures of mental health status used in these studies must be created or adapted for that specific project, and staff must be trained to implement these measures. Such costly, intensive efforts are prohibitive for an individual researcher, the State, or even a single institution such as NIMH. For this reason, developing methodologies that can be included or incorporated into ongoing surveys in a routinized manner is essential. Similarly, sampling methods have not been used effectively in assessing the patterns or needs for mental health services.

Improvements in public mental health are contingent on improvements in access to care and services delivery. Thus, it is important to measure the status of the care delivered in actual practice so decisions about improving the quality of care can be made. This is potentially a more demanding monitoring task, because quality of care may require studying providers and service delivery systems, not just representative individuals in the household or patients/consumers of care and services.

No national databases of the diverse spectrum of mental health providers or service systems exist to enable either national monitoring efforts or individual studies. Similarly, there are no national lists of health care delivery systems. The Substance Abuse and Mental Health Services Administration regularly surveys larger public mental health providers through its inventory of mental health care organizations, but individual providers, MCOs, and social rehabilitation agencies are not included. Without augmenting the current infrastructure to tap into representative providers or mental health care systems, it is prohibitively expensive for individual studies to improve their sampling at higher levels (systems, providers). Current efforts will be limited and less representative over time, as is already reflected in the empirical literature by a disproportion of case studies.

A limited number of intensive, short-term studies of the Nation’s mental health status assessed continuously over time offer an alternative infrastructure for measures of progress, change, and value. Indicators of particular interest should be those that are potentially changeable through treatment interventions. In addition, researchers conducting local or regional studies could incorporate these indicators, thereby testing their findings against the Nation at-large. As a research agency, NIMH should “define the gold standard” for what should and practicably can be monitored (i.e., establish a process to determine the priority areas).

At the national level, for example, it would be useful to ascertain the number of patients/consumers with particular diagnoses who are and are not receiving treatment, where treatment is being provided, and the nature of treatment received. At the local level, an example is seen in the provision by school districts of information about the percentage of children who require special education services and the percentage evaluated by guidance counselors or other mental health professionals within the school setting.

NIMH should provide leadership in developing the infrastructure to regularly and reliably track mental illnesses and the impact of treatment breakthroughs or policy changes in the organization or financing of systems of care on public mental health. A secondary goal of this monitoring is the provision of the next generation of research findings to inform the priority-setting process. Key activities would include coordination with current Federal efforts to develop an economical, written plan for presentation to the NAMHC. It is difficult to think of such a complex project and use the term “economical.” Nonetheless, embedding the NIMH effort into ongoing studies, effectively using sampling strategies, and collaborating with highly experienced Federal agencies should permit a relatively low-cost effort. This is especially feasible given the expected payoff in the type of information policymakers need and patients/consumers want, as well as stimulating more public health relevant research. Specific recommendations on the methods development required for this effort are presented in Chapter 4.

Recommendation 6

NIMH should collaborate with Federal, State, and private agencies to establish a mechanism to monitor the value (as defined by an equation of quality, access, and cost) of mental health care delivered across the Nation and the impact of policies on practice and service systems.

Summary

NIMH should develop a process that integrates the input of diverse stakeholders regarding research priorities, data from systematic monitoring of public mental health and quality of care, and information derived from research funded by NIMH and other organizations. Each recommendation in this chapter represents one step toward development of an informed priority-setting process for the research agenda. Priority-setting based on public mental health values and real impact data will raise the Institute’s research portfolio to a higher level of scientific and public health excellence. This effort is devoted to meeting the primary goal of research supported by NIMH: improving the condition of people with mental illnesses.

Chapter 3
Making Connections

Introduction

Scientific research has generated information that has enhanced greatly the lives of many individuals with mental illnesses. By way of example, recent developments that carry immense potential for improved clinical care are:

• A virtual explosion in the availability of sophisticated new treatments – Nine new antidepressant agents have become available in the U.S. since 1988; three new atypical antipsychotic agents have been introduced within the last 5 years; and at least three compounds (clozapine, lamotrigine, and gabapentin) currently are being studied for efficacy in bipolar disorder;
• A shift from the primacy of “expert opinion” to that of scientific evidence as the basis for treatment decisions – As mental illness diagnoses have been refined and shown to be as reliable as those used in general medicine (Spitzer, Forman, and Nee 1979; Williams et al. 1992) and as treatment options have multiplied, evidence-based treatment guidelines, specific disease management protocols, and medication algorithms are being formulated and tested (Depression Guideline Panel 1993; Frances et al. 1998; International Psychopharmacology Algorithm Project Participants 1995);
• Numerous research findings on innovative delivery system configurations – For example, findings on the effectiveness of Assertive Community Treatment teams can inform policies as to preferred methods of delivering care for persons with severe mental illnesses (p. 580).³

These and other advances have received high visibility in the professional literature. But, as in all other health fields, publication in a journal is only the first step in the slow and arduous task of moving new information from the research to a treatment or service setting. Dr. Anthony Lehman’s research captures the immensity of this gap for individuals with schizophrenia and why making better connections between research and practice is so important (see story box, Translating Research to Treatment: The Challenge).

The large gap between knowing what works and finding these evidence-based interventions in a service setting points to the acute need for: (1) evidence-based interventions that are easily applied in routine settings, and (2) research on how to introduce these findings into practice and service settings to effect rapid adoption. The question for the Workgroup was how best to generate these two types of research. The answer involved three steps. Step one is to establish the rich priority-setting process that was specified in Chapters 1 and 2. Step two is to ensure that the four research domains of efficacy, effectiveness, practice, and service systems research are adequately supported. The third is to make the necessary connections between and across these four research areas that are critical junctures. For the Workgroup, these connections are a matter of establishing overlapping methods and research questions to achieve a dynamic interplay that speeds the discovery process.

Fostering the Dynamic Nature of Public Health Research

Chapter 1 described four discrete spheres of inquiry: efficacy, effectiveness, practice, and service systems research. While such a parsing of research offers advantages, it also carries risks. The principal risk is one of compartmentalization, of hampering the essential give-and-take among research arenas. Without this interplay, NIMH staff and its researchers cannot see that questions generated in one arena may demand a search for answers in another. The challenge of moving a potential new treatment from the point of development to its delivery by appropriate providers to the right patients/consumers in a suitable context requires answers to a diligently informed cascade of questions. At any point, answers to these public health questions may lead back to a reconsideration of earlier questions and stimulate new, needed research.

The structure of intervention development may be likened to an elaborate, multi-tiered fountain in which water splashes downward from one tier to another, with a recirculation pipe at each level of the cascade. Efficacy research would be in the uppermost basin of the fountain. Here, investigators strive to create research circumstances that will answer fundamental questions as to the safety and efficacy of a given intervention. Experimental treatments that show no significant effect, or are found to yield only a marginal improvement over extant treatments, should be siphoned out of the cascade as an important feedback loop to the treatment development researchers.

The second tier, effectiveness research, seeks to expand the scope of the discussion. Studies may relax exclusion criteria to examine the effects of a treatment on a greater breadth of populations (e.g., children and adolescents, the elderly, or others; on “atypical” presentations of the illness in question; or on the given illness when it co-occurs with another). Yet another aim of effectiveness research may be to examine the effect on outcome of a range of procedures used to administer a treatment. Here, before a proposed treatment cascades into further application, the evidence is weighed. For example, the trial may indicate that the treatment is ineffective in adolescents or the elderly and is rerouted back to the level of efficacy research, where more rigorous controls again may be employed to ascertain what characteristics of treatment, patient/consumer population, or other variables explain the lack of effectiveness. Re-examination may suggest what modifications might be warranted in the indications for the treatment in question, lest a treatment that is efficacious, albeit for a narrow population of patients/consumers, be siphoned prematurely out of the system.

With the imprimatur of the treatment community, the next step for a novel intervention is into the world of clinical practice. At this juncture, new questions arise. Does a treatment produce the desired benefit at the cost predicted as it is being used in routine care? Is it being properly used? Do clinicians need to be trained in its use? Do the inevitable practice variations of independent practitioners increase or reduce the effectiveness of the treatment in representative populations? Careful research monitoring, through practice research networks or other means, not only has the capability of contributing answers to such questions, but also may identify practice patterns (e.g., off-label uses of drugs or innovative combined treatment modalities) that may be suitable candidates for recirculation upward to any of the previously described research arenas.

Service systems research constitutes the fourth and final basin of the fountain. Here, interventions are subjected to administrative, clinical, and policy questions. What does it cost to deliver the treatment? Are there savings and when do they occur? Are the benefits worth the cost? How does the treatment compare to other treatments for a given illness across systems of care?

Although the tiers of the fountain are the visible structure, the vibrancy comes from the methodical recycling of the water. The same can be said of research. It is through the constant feedback loop across the research fields that research can most efficiently bring evidence-based interventions into practice. Opportunities to expedite development and testing are ripe, if researchers in one domain anticipate the needs of the other domains given current findings. Might, for example, an innovative treatment for a condition that occurs commonly in association with another illness be tested early in efficacy trials under those conditions? Or, if efficacy trials strongly support further testing of a new treatment, should outcome criteria be broadened early on to include, for example, its effects on functional outcomes in addition to its effect on symptomatic outcomes?

This chapter recommends actions for NIMH to take to increase the dividends realized from its investment in each of the four discrete research domains. For purposes of clarity, these domains are discussed separately and in sequence. But as our hypothetical fountain illustrates, knowledge transfer is a complex, iterative process. Fortunately, an array of models for synthesizing and disseminating clinically relevant information are available for use in the mental health field.

Issue: Building the Bridges Among Efficacy, Effectiveness, Practice, and Service Systems Research

As presented in Figure 1 and discussed above, the interplay between research domains is an essential part of fostering evidence-based interventions. Establishing such linkages is not a simple matter. As said before, NIMH must continue its support of the four domains of research. In this section, the Workgroup will point to additional areas that require development to provide an empirical link across these domains. Additionally, there will be a special emphasis on expanding the Institute’s effort in practice research, a relatively new and underdeveloped area for mental health research. Subsequent chapters will outline the infrastructure for developing the substantive areas described in this chapter.

Efficacy Research

Efficacy research typically emphasizes the reduction of symptoms, looks at a selected group of individuals with the illness, and incorporates well-defined protocols for interventions. This type of research is essential for detecting what is worth pursuing with a broader and more expensive array of outcomes and participants. NIMH’s investment in efficacy work must be maintained. Although necessary, a robust efficacy trials portfolio alone is not sufficient.

A case in point is the use of highly restrictive exclusion criteria in efficacy work. The need for this methodological device was explained in Chapter 2. The problem with an over reliance on this type of trial is that as more and more individuals are excluded, the applicability of the treatment findings is diminished. For instance, in an efficacy study of a treatment for schizophrenia, typical exclusionary criteria would be substance abuse problems or suicidal thoughts. A common inclusion criterion is that this be the participant’s first episode of schizophrenia because this provides a study population without prior exposure to other treatments. With these three conditions alone, the treatment findings may be applicable to less than 20 percent of individuals with schizophrenia in the community. This narrow band of application or generalizability may be why so many patients/consumers and providers find that research does not address their circumstances.

Researchers and NIMH staff must be constantly watching for opportunities to invest wisely in expanding efficacy trials. This may involve supplementing a promising efficacy trial midway through the grant to permit a broader range of participants or to support a longer follow-up period to understand maintenance issues.

Another opportunity is to selectively expand the domains of outcomes earlier in the development and evaluation of treatments. Efficacy researchers speed the “discovery” of promising treatments by looking for shorter-term signs of improvement. Again, this is an excellent strategy that can be improved by extending or expanding a successful trial to include additional measures of improvement valued by individuals with mental illnesses, their families, and their clinicians. Although all of these individuals value symptom reduction, outcomes in mental illnesses, as well as general medical conditions, are far more complex than symptoms. Many patients/consumers respond to treatments with a reduction in, but not a remission of, symptoms and an improvement in, but not a normalization of, function. These “responders” still suffer greatly and are not fully recovered. This difficulty is captured in author Andrew Solomon’s story, see Depression: A Patient’s Perspective. Like diabetes, arthritis, and hypertension, mental illnesses are chronic and recurring. Long- term strategies are necessary to prolong the period of wellness or symptom control and to reduce the risk of relapse or recurrence.

Due to the chronicity of many mental illnesses, efficacy research should also embrace rehabilitation as an important form of intervention. The rationale for rehabilitation is the same as for physically challenged individuals-illness produces impairments that affect basic life skills and the ability of patients/consumers to function as workers, parents, students, and in other life roles. Rehabilitation provides basic life skills using methods that compensate for the impairments. Research demonstrates rehabilitation is effective in helping individuals with severe mental illnesses to acquire new skills and apply them in their own daily lives.

Given the wide array of potentially useful interventions, a better understanding of how to use them alone, in combination, or in sequence to increase the rates of recovery (both symptoms and function) is necessary. In short, NIMH must be prepared to follow up on promising trials with more generalizable efficacy trials to smooth the transition to effectiveness trials.

Recommendation 7

NIMH should augment its efficacy portfolio with research that assesses the generalizability of interventions across diagnostic complexities (e.g., comorbidity or chronicity), as well as individual, social, and demographic factors.

Recommendation 8

NIMH should enrich its efficacy portfolio with trials that measure change in terms of both symptoms and function over a meaningful period of time.

Recommendation 9

NIMH should incorporate rehabilitation into its intervention portfolio.

Effectiveness Research

The Workgroup recommends a number of approaches to revitalize the effectiveness domain of mental health treatment research, thereby making it a stronger link to both the efficacy and practice research domains.

First, the number of patients in effectiveness studies needs to be sufficient to extend the generalizability of efficacy findings. Such studies should be large enough so that meaningful sub-analyses can be performed on patients often excluded from efficacy trials (e.g., patients/consumers with comorbid conditions, minority patients, and other frequently underrepresented groups). Variations in outcomes by subgroups might be an indication to re-examine the intervention in the context of an efficacy trial for that particular subgroup. There are some excellent examples of large effectiveness trials testing interventions in the cardiovascular and cancer areas. Mental health researchers should look closely at some of these methodologies to inform mental health effectiveness study designs.

Whenever possible, effectiveness research should incorporate multiple study sites, thus expanding organizational, patient/consumer, and provider mix. Again, there should be sufficient numbers and sufficient data collected on these aspects of the study to perform meaningful sub-analyses. Such findings can serve as excellent grist for generating practice research hypotheses. For example, if a certain subgroup has a sub-optimal outcome, it may be possible to disentangle certain patient/consumer, provider, or organizational characteristics that contribute to this finding.

An additional need in the effectiveness area is to collect systematically data on the cost of treatment, so that cost-effectiveness ratios can be determined. For this type of analysis, larger numbers of patients and longer timeframes than are traditionally funded in randomized controlled trials at NIMH are needed.

Other opportunities in effectiveness research are to generate studies of commonly practiced but not commonly studied interventions. Hypotheses for such studies may be generated from practice research findings. Many psychosocial treatments, rehabilitation interventions, and some off-label use of medications fall in this area.

NIMH will need to choose wisely among all of the possible effectiveness trials that one would want to see conducted. This selection will need to be informed by the priority- setting process, a shrewd understanding of what other funding sources will wish to pay for partial or full support, and the scientific opportunities. Another determinant would be the anticipated generalizability to adoption in either practice or service settings. The current research portfolio provides a glimpse at the many settings in which promising mental health interventions should be offered. The service settings shown in Figure 5 include, but are not limited to, the interface between mental health and substance abuse treatment, the criminal justice system, workplace, nursing homes, primary care practices, and schools. Ensuring a series of trials that addresses care in each of these settings and the “revolving door” that too many individuals experience across these settings is vital.

The Workgroup also points out that if effectiveness research is to be funded through NIMH, there must be an emphasis on the importance of the findings for providing better interventions rather than an emphasis on perfection. There really is no perfect design in effectiveness research. There will always be a setting left unexplored, a group of patients/consumers who will not be included, or one more outcome measure that would be important to include. But at some point, the researcher and the NIMH must strike a balance between what is ideal and what is feasible to derive what is important. The researcher must do so in the one study, and NIMH staff must do so across the entire effectiveness portfolio.

Recommendation 10

Research should be encouraged that better characterizes promising interventions, patient/consumer populations, ongoing treatments, and service settings.

Practice Research

The investigation, evaluation, dissemination, and implementation of optimal practices in contemporary treatment settings should be an integral component of future mental health research. Inclusion of practice research into the NIMH portfolio is relatively recent, and the Workgroup believes that a significant expansion in this area is warranted. Although new for NIMH, this emerging field is built upon core areas with long histories in health care that have much to contribute to understanding how mental health care is or can be provided.

Clinical Epidemiology

There is a need for more resources to be placed into this area of inquiry. Data from observational studies on what happens with people over time who are receiving treatment provide valuable information to inform clinical care. For example, it would be helpful to mental health providers and patients/consumers in making treatment choices to have information on the relapse and recurrence of illness in those who are treated in the community over long periods of time. Policymakers are interested in knowing how treatment resources are used and, when a particular intervention is used across large numbers of people, whether there are particular groups who may be at risk for problems. This is a special focus of pharmacoepidemiology studies. Other types of questions that should be addressed include determining who provides care for specific illnesses, where it is being provided, and at what stage in the illness. Attaching cost and quality indicators to these data can be very helpful to those making decisions about allocation of resources.

With the growth in new diagnostic and screening measures, it is important that they be studied to determine their utility in various populations and service settings. For example, what is the likelihood that a given person who tests “positive” on a particular screening instrument for depression will actually have a depressive disorder that requires a treatment intervention? What is the utility of screening in promoting earlier treatment?

In order to improve the outcomes of treatments for patients/consumers with mental illnesses, information is needed concerning how treatments are typically provided in an array of provider settings (e.g., school settings for children and adolescents, nursing homes for the elderly, jails for incarcerated patients/consumers, etc.) and to special populations (e.g., the elderly, children and adolescents, patients/consumers with a comorbid drug or alcohol dependence).

NIMH should promote research that would help public and private sector policymakers (e.g., mental health commissioners and corporate benefit managers) decide where to spend what clinical resources. Little is known about the cost or quality of care, consequences of having the more highly trained clinicians relegated to the consultative, rather than direct care, and roles they increasingly play in public and privately funded mental health care systems.

Recommendation 11

NIMH should expand its effort in clinical epidemiology through workshops and through developing research addressing the issues in the epidemiology of care.

Quality of Care

Research on specifying the quality of mental health care is relatively new, with relatively few researchers now conducting quality-of-care research for mental illnesses. This field has developed to a much greater extent in general medical services research, and there are different conceptual and technical approaches that may be adapted to or implemented in mental health research. Points of emphasis for NIMH are in the areas of treatment guidelines, outcome assessments, and in the effects of service organization and financing on the quality of care.

Treatment guidelines that define appropriate care for a variety of disease conditions and constructing monitoring tools and systems to assess adherence to such guidelines are important for developing a capacity to monitor the quality of routine care. Such monitoring helps identify areas where practice needs to be improved, appropriate questions for efficacy and effectiveness research, and gaps between evidence-based treatment and prevailing practice.

Research on treatment guidelines is needed to ensure that they are scientifically based and are sufficiently specific to the providers for whom they are intended. Research also must assess the impact of treatment guidelines on clinicians’ practices and clinical outcomes. Surveys might help elucidate in what ways and why clinicians and institutions deviate from treatment guidelines for both responsive and non-responsive patients/consumers. Another method for obtaining practice parameter information is either to develop multidisciplinary practice networks or to engage existing practice networks. Once established, these networks can facilitate the development of large practice parameter databases that can be used to suggest areas where change is needed and to track changes in practice over time.

The NIMH and other agencies (e.g., AHCPR) should collaborate in fostering studies of the processes by which to implement and evaluate guidelines and scientifically evaluate the relevance of guidelines to the patients/consumers in a variety of settings, as well as obstacles and incentives to follow guidelines. Also, attention should be paid to the increasing number of guidelines issued for a specific illness. Summaries of guidelines should be provided to help clarify where there are potential contradictions, gaps, and uncertainties in available treatment guidelines. Research is required to understand the significance of these differences.

For intervention research conducted from a public health perspective to be truly useful, investigators must use more than the standard outcome measures of reduction in clinical symptoms. Researchers must consider and ascertain whether a treatment or service helps an individual function better at home, work, or in the community. Evaluating improvement at these levels is challenging. For researchers, a “good” measure is determined by psychometric properties such as reliability and validity. That is, if the measurement is repeated, does one obtain the same result (reliability)? Does it actually measure what it is intended to measure (validity)? A central issue that invariably arises is whether increments in the measure correspond to meaningful changes in an individual’s life and whether some change that may be statistically significant is actually clinically meaningful. Although the clinical utility of an intervention may be documented by clinicians, family members, or friends, the most important arbiter is the individual receiving the treatment. Does he or she actually feel better or see an improvement in fulfilling interpersonal, educational, or vocational goals? Only when the needed measures are available and used to assess the effects of interventions will policymakers and purchasers have a better idea of what services must be provided, which ones would be useful if additional funds were available, and which ones are unnecessary.

Quality of care research, a third key area, is the relationship of the structural, organizational, and financial arrangements on the quality of care. As an example in structure and organization, investigators have studied the effects of inter-organizational services coordination teams on the quality and outcomes of services to children in State custody. Findings suggest that such teams improve access to services and enhance outcomes (Glisson and Hemmelgarn 1998). For the general population of children and adolescents, treatment services often span the mental health, general health, school, social welfare, and juvenile justice sectors, among others. Services are often fragmented, and research is needed to test new strategies for integration and coordination.

Other important questions include whether organizing mental health services in a carve- in versus a carve-out arrangement impacts on the patterns and quality of the services delivered? Do various provider incentives to control costs actually contain costs, and do they impact on the quality of care? Do administrative policies that aim to reduce cost actually result in cost savings?

Also of interest is identifying who should be providing what type of care (i.e., Does it matter if the care is provided by a psychiatrist, nurse, psychologist, primary care physician, social worker, State-certified counselor, BA-level worker, or anyone who can demonstrate competence in delivering Treatment X ?). This question of “Where should we spend what clinical resources?” is tied closely to questions about how to get research into practice. It is one thing to say that problem-oriented family therapy is an effective treatment for schizophrenia; it is another to say who should be trained in such a treatment and how to structure the training and treatment environment such that the intervention actually enters routine practice and produces the desired outcomes.

Recommendation 12

NIMH should expand its effort in generating research to inform and evaluate treatment guidelines in conjunction with other Federal agencies.

Recommendation 13

NIMH should review the current measures of functional status and support tests of which measures or items are reliable, valid, and have clinical utility.

Recommendation 14

NIMH should expand its research portfolio to include the interface between the architecture of services (i.e., the structure, organization, and financing of services) and its effects on the quality of care and clinical outcomes (symptoms and functioning).

Dissemination Research

The Workgroup felt strongly that “How to bring research findings into practice” is in itself a researchable issue and one that warrants additional NIMH support. The reasons for the large gap between what is known and what is actually offered can be numerous:

• Clinicians find that reported advances are difficult to implement within their settings.
• Science rarely has complete answers to common clinical and delivery system issues.
• Emerging research developments have outpaced the capacities of many practitioners to remain abreast of new science.
• Individuals with mental illnesses and their family members do not have easy access to research findings to insist on research-based interventions.
• Health care delivery systems cannot possibly implement each and every change, and the data necessary to determine which innovations can most effectively be implemented at their particular site rarely exist.

There are two thriving literatures that can speed this process. Both dissemination and behavioral science deal with the processes involved in changing individual behavior and systems. Determining the best methods for improving knowledge transfer from research findings to practical applications is a complex task involving a broad array of variables and their interactions. Considerations must include the diverse psychological, social, and economic factors that influence the behavior of individual practitioners and groups, as well as the organizational dynamics of large-scale service delivery systems. Areas of relevance to mental health include:

• Behavior change literature to determine what factors seem to be associated with changes in behavior of individuals, organizations, and systems.
• Patient/consumer and provider behavior change literature to determine similarities and differences in factors between the mental health and general medical literatures.
• Complex decision-making literature to determine what is known about the acceptability and tolerability of a treatment for both patients/consumers and providers.
• Community adoption or adaptation of innovations in treatment.

Furthermore, dissemination must include care settings and care providers who are most commonly today’s treatment providers. This will include not only clinicians, but case managers and rehabilitation workers who often make up the greatest portion of personnel budgets for programs that treat people with the most serious mental illnesses, as well as workers in nursing homes, schools, prisons, and MCOs.

Given that practice is influenced by what is paid for, it will be important to include purchasers in developing such research, in that they, as benefit managers, authorizers of care, and contractors of care, are critical players in creating incentives for guideline-based practice. How do systems promote knowledge acquisition and knowledge application? For example, do the systems provide incentives for staff to increase their knowledge about the effectiveness of established treatments? Similarly, do they provide educational activities such as workshops, grand rounds, journal clubs, etc., all of which are intended to increase knowledge? Do systems facilitate the use of the Internet, so that staff have access to current reviews of treatment literatures and treatment guidelines? What other knowledge bases are available to staff? In addition, information is needed about the way in which systems facilitate the application of knowledge regarding effective treatments. Which methods of knowledge transfer improve outcomes? For instance, data indicate that giving Continuing Medical Education (CME) credits for the first 15 hours of guideline-driven treatment of schizophrenia increases the use of such guidelines by clinicians in need of CME. What other vehicles need to be developed to improve the practice of the wide range of treatment providers for whom CME is irrelevant?

At the system level, information needs to be obtained regarding obstacles to the facilitation of knowledge transfer and application. Managed care and public sector delivery system policies (as well as selected Medicaid/Medicare reimbursement policies) have set procedures by which individuals with mental illness are treated that may affect outcome (e.g., capping the number of outpatient visits; recommending 15-minute medication “visits”; creating obstacles to providing more expensive but often safer, more effective or better tolerated medications). None of these procedural intrusions have undergone scientific evaluation. General medicine indicates that at some point, very infrequent visits result in increased attrition, inadequate patient adherence, and poor outcomes. Similar findings are suggested by a few recent studies with patients/consumers with mental illnesses (Katon et al. 1995).

Recommendation 15

NIMH should support targeted research on how to synthesize and incorporate existing knowledge into clinical practice better, with particular emphasis on understanding factors and mechanisms involved in changing practice and systems of care.

Service Systems Research

Ample research evidence supports the general conclusion that the organization and financing of systems of care have an impact on patients’/consumers’ access to needed services and on the quality of those services. There is a need to continue much of the descriptive work in this domain. For example, it is still important to know the relative cost and distribution of mental health services (e.g., inpatient versus community-based care in State mental health systems) and the changes over time.

However, it is critical to be able to link changes in the organization and financing of service systems to indicators of quality of care and clinical outcomes in terms of both symptoms and function. For example, we need to understand the impact of changes in service patterns and cost on quality of care and outcomes. How does the organization and financing of systems affect treatment delivered in these systems? Do financial constraints limit the availability of known effective treatments, duration of care, and access to specialists? Or, are some systems of care able to allocate resources parsimoniously (and therefore more efficiently) without adversely affecting quality of care? How are scarce resources allocated? What is the affect on outcomes?

Also critical is the study of the impact of changes in one system on another. For example, when inpatient days of care are reduced in the State mental health system, are there more days spent in forensic institutions?

It is important to link service systems research with clinical epidemiology issues. For example, how do policy changes instituted at the system level (such as mandatory screening of all patients/consumers) affect the overall health of the covered population?

Last, studies of procedures to improve access to care for underserved segments of our population (e.g., inner-city women with children, rural, and frontier populations) are needed. Do these procedures improve outcomes?

Recommendation 16

NIMH should encourage and sponsor more research that links service systems changes with quality of care and other clinical indicators of patient/consumer status.

The Workgroup identified several areas in service systems research where new partnerships can open unique research opportunities given their importance to public health and/or their policy significance.

Increasingly, advocacy and accrediting organizations are looking for objective measures to assess the quality of the clinical care provided in various systems of care. Entities now producing report cards or other performance rating scales include the National Council on Quality Assurance, The Joint Commission on Accreditation of Healthcare Organizations, the Veterans Health Administration, CMHS’s Mental Health Statistics Improvement Program, and the provider profiling systems of various MCOs.

Organizations are beginning to devote substantial resources to enhance performance on these report cards; hence, NIMH should promote research on the validity of these measures and on enhancing their utility in improving the quality of care. This information dissemination is occurring absent a substantial research base to support using these tools to inform policy. Often, the scores on such report cards are from relative scales, and the reader is left wondering whether everyone might be passing (or everyone might be failing). Similarly, it is unclear whether performance on one narrow domain can predict performance more broadly. Studies identifying which processes of care are most predictive of good outcomes of care and then examining whether increasing the performance of those processes results in improved outcomes could be particularly informative.

Recommendation 17

NIMH should sponsor validation studies of assessment tools designed to measure the quality of health care across systems of care to assist policymakers and patients/consumers with decision making.

Many private and public purchasers and MCOs are developing sophisticated information systems to pay claims, track patient/consumer care, monitor service utilization, and assist in care management. These systems are the core technology of carve-out MCOs and are being developed in other organizations as well. The Workgroup felt that there should be increased attention to understanding and exploring the quality of the data in these evolving systems, since they have the capacity to affect care substantially. Linking outcomes and diagnoses across such large samples would be significant, as would linkages with type and duration of services provided.

Recommendation 18

NIMH should encourage the use of existing databases for understanding service systems.

Summary

This Chapter advocates blurring the lines across research domains by creating areas of overlap between what are all too often disparate areas of research. But make no mistake, progress cannot be made without the continuing use of smaller-scale trials to efficiently detect promising leads. Equally critical are the service studies that work with existing databases, whether clinical conditions are fully documented or not. What requires a boost are the areas of linkage that include some elements of two or more areas of research, such as sharing diagnosis or outcome measures, that will provide the common element for translating and building across research domains. One can think of these as hybrid designs or as methodological grafts upon sturdy stock. The salient idea is that measures or methods that can function in multiple domains should be developed and used. Specific recommendations on these methodological areas of overlap will appear in Chapter 4.

Chapter 4
Methods Development and Innovation

Introduction

Innovations in intervention research methodologies are essential to improving individual and public mental health through enhanced treatments and service delivery systems. Among the necessary innovations are new research designs and creative analytic approaches. In addition, the NIMH and other funding sources must employ these and other new tools to maximum advantage by supporting sufficiently large studies that are necessary to generate data that can inform public policy. This chapter describes basic methodological needs and offers several recommendations to promote methodological development and innovation. Given the technical nature of many of these research problems, some portions of this chapter require a fundamental understanding of research and evaluation methodology.

Issue: Incorporating Measures of Costs Associated with Interventions

A priority in understanding public health relevance is the cost associated with providing an intervention. Costs can be divided into direct costs (i.e., what it actually costs to deliver the services in a specific setting) and indirect costs (i.e., costs other than direct treatment, such as lost productivity). Research also may seek to calculate the longer-term cost difference resulting from providing an intervention or not providing it. What is the cost to society as a whole, not simply to the treatment system, of leaving a mental illness untreated? The answer to this question affords an economist’s estimate of the value of an intervention.

At present, economic measures (e.g., cost-effectiveness and cost utility) are not widely applied in mental health intervention research. Such measures should be considered for broader use. The NIMH will need to ensure that well-targeted, large-scale opportunities for improved studies of the costs of treatments are developed carefully. In addition, cost-effectiveness analyses should be incorporated as part of all large clinical treatment studies.

There are numerous challenges to implementing good cost-effectiveness studies. Intervention research often examines only direct treatment costs, but specific components of direct and indirect costs should be considered for different types of studies. For example, in treatment studies for individuals with psychotic disorders, costs for lost work may not always be applicable, but information on lost workdays for family members may be an important consideration. Implementing high quality, cost-effectiveness studies necessitates improved cost measures, larger sample sizes, and incorporating economic expertise into mental health intervention research.

Recommendation 19

NIMH should improve measures and analysis of costs in intervention studies, with special attention to successful examples from general medicine.

Recommendation 20

Large intervention studies should include a cost-effectiveness component that uses the best methodology available.

Issue: Tailoring Interventions to Communities

Greater attention is needed to ensure individual and community applicability of treatments that have been developed and tested in more restricted settings and populations. External validity refers to the generalizability of research findings to a broad variety of treatment settings.

Treatments need to be tested for their application in community settings. Investigators may need to modify existing treatment protocols for such studies or to develop new “applied” protocols that are independently tested for effectiveness. The translation of treatment and service system interventions developed in the course of research studies into forms suitable for community application may be a focus in its own right for research.

Research also must examine how treatments and their impact vary with the characteristics of communities and health care systems, and with the behaviors of patients/consumers and clinicians. A related task is to study the characteristics of interventions used in community practice, and to develop classification schemes that permit key elements of these interventions to be reproduced and tested in effectiveness studies. NIMH should encourage development of minimum standards for describing treatment settings and should encourage researchers to report such information on treatment context. Eventually, such information should prove helpful in models of external validity.

Recommendation 21

Approaches to conceptualize and assess key characteristics of intervention settings should be developed, as should models for understanding the effects of these characteristics on outcomes.

Understanding community settings requires incorporating usual care into the intervention protocol or, at least, to have usual care as a control condition. For either purpose, the features of usual care must be defined explicitly. Usual care settings may differ on a variety of characteristics, including types of patients/consumers, providers, financing, organization, and constraints and barriers to care. Which characteristics of usual care are implemented in intervention protocols should be specified. Since usual care can vary across settings, the comparison standard for a specific treatment could vary across settings; this, in turn, imposes an interpretative challenge to the external and internal validity of study results. Techniques to analyze these variations, such as explicit modeling of “process” variations in usual care and their outcome implications, should be developed. Theoretical and empirical work is needed to determine which features of usual care are relevant to the estimate of the true intervention effect.

Recommendation 22

NIMH should encourage the development and evaluation of key research measures for assessing “usual care” and develop analytic methods to adjust for variation in components of usual care.

Issue: Tailoring Interventions to Distinct Subgroups

External validity also refers to how well a finding may generalize to a specific individual with the target illness. This means that the sample for the study should be selected to represent accurately the full range of those individuals suffering with the illness. As previously discussed, intervention studies frequently exclude individuals with comorbidities, yet comorbidities such as substance abuse or other health problems are the rule rather than the exception in community settings.

Some balance of separately studying comorbid populations, purposive sampling of key subgroups, and naturalistic inclusion or representation of a range of comorbidities is needed to develop information on intervention effects in people with comorbid problems. This requirement implies gathering participants from a broader range of treatment facilities, including those which specialize in treating people with dual diagnoses, and expanded or modified protocols that adequately address the treatment needs for people with comorbidities.

Prioritizing among groups for inclusion in studies requires determining which cultural or demographic subgroups affect true treatment responsiveness. Relatedly, it is important that cost-effectiveness research include groups for whom the costs of care differ. This requirement implies need for theoretical and empirical research on the cultural and demographic predictors of compliance and intervention response across a range of relevant outcomes.

Broadly representative samples increase generalizability, but they pose many challenges to intervention studies. Samples become quite large if the study is to be applied to many types of individuals, because a generalization must be based on including a sufficient number of such individuals. When an intervention is to inform insurance coverage decisions, the task becomes even more unwieldy for research findings ideally should be generalized to different types of employers and insurers affected by the policy. One of the challenges for the future is developing representative sampling frames of large systems. Another challenge will be engendering cooperation for recruiting representative systems.

Recommendation 23

NIMH should promote development of innovative sampling strategies for inclusion of underrepresented groups and sufficiently large intervention studies to incorporate representative community populations.

Issue: Considering Treatment Preferences and Decision Making

Clinicians and patients/consumers often tailor treatment decisions (e.g., choice of medication or psychotherapy) on the basis of provider or patient/consumer preferences and clinical characteristics. Outcomes may be monitored through individualized goals and changes in symptoms that are troublesome to the individual patient/consumer. Little research has been conducted on provider and patient/consumer treatment preferences, how to assess them, or how to incorporate them explicitly into treatment protocols. Research is needed on the theory, constructs, and assessment of preferences, how these factors are incorporated into usual clinical decision making, and their impact. It is important to understand, for example, how clinicians utilize data from research findings to make treatment decisions. In addition, research is needed on translating practice results to individual patient/consumer care decisions and on conducting research in a manner that more closely mirrors individualized patient/consumer monitoring and decision making.

As noted in Chapter 3, efficacious treatments are ineffective if they cannot be applied in a particular setting. For instance, cognitive therapies have been shown to be efficacious with 12 to 20 sessions. In a setting where only 6 sessions are permitted, what is a clinician to do? It is important to describe variations in both the clinician’s and the service setting’s ability to adhere to recommended procedures. In addition, it is important to develop models of predictors of such adherence and to develop methods that incorporate information on adherence into estimates of effectiveness. Adherence rates may explain variation in effectiveness across systems or the conflicting results obtained from efficacy and effectiveness studies, respectively. Research on adherence suggests the need for both measurement and theoretical development. There has been much research in general medicine on clinician and patient adherence (e.g., treatment fidelity and integrity). These findings should be extensively studied and applied in mental health intervention research.

Recommendation 24

NIMH should encourage the development of methods to study and incorporate clinician and patient/consumer decision-making processes into intervention research.

Recommendation 25

NIMH should support research to identify common practices believed to be helpful and bring them under research scrutiny, that is, ascertain what is going on in the practice community, and determine how much of that is beneficial.

Recommendation 26

NIMH should encourage the improvement of methods for both evaluating clinician implementation and patient/consumer adherence to treatment recommendations and estimating the consequences of these variations on the effectiveness of treatment.

Issue: Confidence in Drawing Conclusions from Findings

Decision analysis, econometric modeling, and other analytic techniques should be explored as tools to develop models of effectiveness across systems with different characteristics, especially when observational data are used. Studying treatment effects in the context of an observational study requires analytic techniques with reasonable assumptions about measured and unmeasured differences in compared groups. The last two decades have brought considerable advances in methods used in social policy evaluation, econometrics, evaluation research, and biostatistics to support drawing inferences from observational data. The key technical issue is identifying and accounting for bias from unmeasured variables. These techniques are controversial, can be difficult to apply, and are unfamiliar to most mental health researchers. To address these issues adequately, the application of various new methods to mental health treatment research should be explored vigorously.

Since they are much less precise than analyses for experimental data, these methods could result in sample sizes that become quite large. This increases the potential utility of large, naturalistic data sets that include outcomes and quality-of-care data, but such data sets are rare. An alternative is carefully designed, often expensive studies that combine, in innovative ways, experimental and observational features.

The need to ensure confidence in findings also has implications for the types of data needed in a study. In economics, for example, the instrumental variable technique is used to adjust for treatment biases that are due to observed or unobserved factors that could bias the selection of subjects or their attrition from the study. It is well known that such instrumental variables can be difficult to identify. Thus, methods development is needed to explore suitable instrumental variables for mental health intervention research. Other techniques include causal modeling, propensity variables, and regression discontinuity design and analysis. The strengths, limitations, and underlying assumptions of these techniques and their suitability and feasibility for mental health intervention research should be explored.

Recommendation 27

NIMH should explore new methods for analysis of data from studies that incorporate innovative combinations of research designs.

Issue: Assessing External and Internal Validity

Across medicine, numerous reviews of the quality of clinical trials have pointed to difficulties in design and implementation of trials that limit the certainty of causal inferences. Such difficulties may include, for example, excluding dropouts and individuals who refused treatment, which can limit interpretation of intent-to-treat samples. Contamination across treatment groups in their exposure to treatment limits both generalizability and the estimate of the true treatment effect. “As-treated” analyses break randomization and use the experiment as an observational study, which may lead to many of the same problems encountered in correlational observational studies. However, the sample sizes in most mental health clinical trials are often too small to evaluate selection effects adequately or to permit the use of observational analysis techniques to analyze the potential effects of bias from unmeasured variables. Similar or even more troublesome problems arise in experimental health services intervention studies, which typically do not have the advantages of masking and other features to protect internal validity that are common to clinical efficacy trials.

Studies that combine both experimental and observational designs or analytic features to address questions about treatment in a community context will typically require studies of longer duration, larger sample sizes, and broader domains of outcome, which, in turn, may require a reduction in the intensity (or depth) of data collection (e.g., about clinical factors) for the study to be feasible. Thus, such alternative approaches can entail a trade- off between breadth and depth of information collected and analyzed, on the one hand, and the ability of studies to inform either clinical or policy audiences on the other. Some goals of these studies, such as having adequate case-mix adjustment methods to permit outcomes comparisons, may be compromised by requirements for reductions in the depth of measures. Methodologists should develop approaches to model those trade-offs in breadth and depth for given study purposes and to maintain reasonable costs for the study.

When studies involve multiple design decisions that favor external or internal validity, rarely are both improved simultaneously. Decisions to improve community usual-care relevance, for example, probably will increase external and decrease internal validity. Developing approaches to explicitly model the trade-off of internal and external validity, given the research purposes, state of knowledge of the field, and research costs would be useful in informing these design decisions. Such methods work may be feasible using data from early multiplex-designed intervention studies, existing observational data, or from meta-analyses. These complex studies may require development of methods to estimate power and conduct analyses in the context of hierarchical sampling designs and a mixture of experimental and nonexperimental questions and analyses.

Basically, once an intervention study indicates that a treatment works and can be tolerated by participants, the next series of questions is to understand the characteristics of the study, to see how well the treatment can be generalized. Simply put, the questions are who, what, how, and by whom? Who received the intervention and responded or did not respond? What was the exact treatment? How was the treatment provided and by whom? Documenting these characteristics in the efficacy and effectiveness stages is very important for building the next set of questions.

While standard approaches exist to statistically adjust findings for research participants’ demographics or clinical characteristics, no comparable approach is available to adjust for characteristics of the treatment setting. It should also be noted that as the treatment progresses through the cascade of tests, an efficacious intervention may be tried in a group of patients/consumers where there is no significant improvement (e.g., the elderly). Only by knowing the specifics of the previous efficacy and effectiveness work can such a result be interpreted and a plan to search for the cause of this lack of effect established. The trade-offs between generalizability and feasibility are never ones that a researcher wants to make. One simply cannot afford the time or costs that would be involved in recruiting a sufficient number of participants to reflect each diagnostic complexity or age group. Equally difficult choices must be made regarding the number of service settings that can be included in a study to afford generalizability. Because of the needs to make such trade-offs, it is important to develop methods of explicitly considering the trade- offs. How should a researcher consider budget, sampling plans, and public health relevance to determine when an increase in representativeness is worth the associated increase in research costs?

This lack of information limits the ability of clinicians or policymakers to consider whether their settings are sufficiently similar to those in which the research was conducted to apply findings to a local or front-line situation. This is a critical issue, since health care settings have diverse characteristics, such as physical setting, staff configuration, financing characteristics, and treatment style. It is not clear which are salient to implementing a new intervention.

Recommendation 28

NIMH should encourage development of methods to explicitly evaluate trade-offs in alternative design features that differ in their implications for internal and external validity.

Issue: Fostering Innovative Methods

Pursuit of the goals of methodological development requires integration of clinical and social science, particularly incorporation of statistical and econometric expertise. Expertise in qualitative analyses also is likely to be needed to generate new approaches to studying systems, treatments, and adherence with treatment. Exposing mental health services and treatment researchers to methods and methodologists from non-mental health and non-health fields may lead to innovative solutions. The Workgroup considered several strategies to develop rapidly a body of knowledge and alternative methodological strategies for implementation into mental health intervention studies. While emphasizing the need for NIMH to support research studies, the Workgroup also urged the Institute to take some immediate steps to foster development of methodological research.

NIMH should host one or more methods workshops focused on several key aspects outlined in this chapter. These workshops could advise the NIMH on likely alternative methods to develop improved standards for innovative, combined research designs. Study section members should be invited to participate.

Following these workshops, NIMH should host a broader conference on the key methodological problems facing intervention research. This conference should be focused on data-based presentations from existing studies, and may even mirror activities in the statistical genetics community by holding a “competition” for analysis of a particular data set. Teams of researchers could present their approaches and findings, so that the robustness of the findings to the analytic procedures becomes evident.

Recommendation 29

NIMH should convene a methods workshop to identify options for advancing intervention and service systems research. The results of this workshop should be debated broadly and options tested in appropriate follow-up activities.

Chapter 5
NIMH’s Leadership Role in Program Development, Review, and Administrative Activities

Introduction

To generate answers to the questions raised in the areas outlined in the previous sections, NIMH will need new approaches to stimulate, review, and support the specified research, training, and dissemination activities. New approaches to develop this infrastructure are required because of the nature of collaborations that must be developed and maintained, as well as the numerous new service settings where research has not been routinely conducted. For example, collaborative, public health-oriented projects may not fit easily into procedures that work well for basic science initiatives. Areas identified in this report as being in need of emphasis typically require research staff networked across academic and provider settings. Also, opportunities for research in diverse practice settings (e.g., primary care, workplace, schools, or public health delivery systems) may arise and disappear quickly. NIMH must be able to make funding decisions rapidly to take full advantage of these windows of opportunity.

This chapter discusses administrative efforts that NIMH staff can undertake to provide the infrastructure for the Workgroup’s research, training, and dissemination recommendations. These recommendations principally urge tailoring existing procedures and speeding administrative processes to meet emerging program objectives.

Recommendations addressing the NIMH’s role in developing new research areas—from the conceptualization of a research or training initiative, through peer review, into the monitoring of funded projects—are particularly timely due to Vice-President Gore’s Reinvention Initiative. In response to this initiative, institutes across NIH are proposing novel approaches to targeting research development, facilitating new alliances among research teams, and expediting administrative processes. NIMH is encouraged to take advantage of this time of reinvention to build creatively and efficiently the infrastructure for generating research to shape the public mental health.

Program Development: Selecting or Creating an Appropriate Funding Mechanism

The NIMH supports its extramural research activities through either grants or contracts.⁴ These two funding mechanisms can take many different forms to support a surprisingly diverse range of activities responsive to scientific opportunities and public health need. The Workgroup recommends that these same mechanisms be adapted as necessary to accomplish newly identified program objectives: (1) creating research opportunities in nonacademic settings, and (2) training researchers who can work in both academic and provider settings to yield more meaningful treatment and services research.

New areas slated for development, identified both in this report and through the evolving priority-setting process that the Workgroup proposes in Recommendation 1 must be announced publicly. Interest in new grant activity is announced in the NIH Guide,⁵ which prints the full text of all RFAs⁶ and PAs. The Catalog of Federal Domestic Assistance⁷ contains brief synopses of each government agency’s funding interests. Also, each NIH institute maintains a web site that lists all of its PAs and RFAs.⁸ Similar tools are available to publicize RFPs for NIH-led contracts. Indeed, all Federal contracting opportunities are announced in the Commerce Business Daily,⁹ and each institute typically posts these on its web site.

Most announcements issued by the NIMH are calls for research grants. The Institute highlights general areas of research interest and need, but asks independent investigators to develop applications within their area of interest. Many different types of grants are used, with each solicited through an RFA or PA.

Each RFA and PA lists programmatic objectives, a specific funding mechanism, review criteria, a description of the review and award process, and the terms and conditions of the award process. The announcements are powerful tools for shaping a research portfolio. Accordingly, recommendations in this section call for tailoring the specifics of future issuances to begin shaping both the proposals submitted and the reviews that follow. The section discusses: (a) selection of the funding mechanism; (b) project requirements; and (c) review criteria.

In articulating program objectives, NIMH staff often find themselves constrained by existing mechanisms. Research training, for example, typically occurs at single academic sites, supported by institutional training awards. This works well for mature fields. In areas recommended for development in previous chapters, however, multiple academic and non-academic settings may be required to give a well-rounded training experience for an individual. New or revised research training support mechanisms are needed to enable research to be conducted in non-academic treatment and service settings and to expedite the review and award process.

Issue: Mechanisms for Training

New multi-site training models designed to provide developing researchers with well-rounded experiences should include training in research and evaluation offered at academic institutions, along with hands-on experience in provider or policy settings. More researchers also are required who focus on mental health and who are trained in economics. Because few sites offer the required range of expertise and experiences, consortium arrangements between training sites in clinical and services research are needed.

Established investigators may be skilled in one area, but require additional training or experience to move into these new areas of research. The Workgroup found that the new K awards in clinical research could be very helpful in achieving this goal. Research grant supplements and support for faculty scholars programs also offer methods to support established researchers’ training at academic or non-academic areas of research excellence. Training sites might include current NIMH Intervention or Services Research Centers, other academic sites, businesses, State or local governments, or private foundations. Generalizing from successful examples of creative training activities should be encouraged. For example, the Public Academic Fellowship Program at the National Association of State Mental Health Program Directors Research Institute provides interested researchers with research skills in large-scale policy analysis on mental health issues.

Additional possibilities for helping individuals hone their clinical and services research skills include summer institutes (perhaps similar to those sponsored by the American Association of Geriatric Psychiatry), the establishment of a Faculty Scholars Program in partnership with medical schools, and the creation of training programs without walls where an individual would craft an individually tuned training program with components from a variety of relevant institutions (e.g., academic, purchaser, or governmental monitoring settings). These efforts should focus on substantive areas outlined in Chapter 3, but also on the methodological areas in Chapter 4.

Recommendation 30

NIMH should develop additional training and career development programs that offer hands-on experience in diverse research settings in order to provide researchers with an enriched training experience.

Issue: Mechanisms for Creating Collaborative Research at Provider Sites

Working partnerships involving academic researchers and front-line service providers are the crux of efforts to develop the types of research the Workgroup outlined to improve the quality of care and the public health impact of the research. The diverse sources that the Workgroup consulted regarding research partnerships identified several common ingredients for successful alliances: ensuring benefits to all parties; selecting topics of mutual concern; requiring that each participant contributes a needed role or service; designing studies with timeframes appropriate to each partner; maintaining continuity of relationships to promote ongoing access and follow-through; nurturing good relationships at all levels of the partnership; and using partners’ time efficiently. Such ingredients are challenging to assemble yet hold enormous promise.

Creating and maintaining such partnerships will require flexible NIMH support over the long term. Although no single recipe for partnerships exists, NIMH can provide a range of options for identifying and fostering possible new alliances based on developing shared goals, joint problem solving, and resource sharing.

Expanding Previously Successful Approaches

Once established, support is necessary to maintain long-term relationships. A decade ago, the NIMH-sponsored Public-Academic Liaison (PAL) Program created incentives for establishing such research partnerships requiring collaboration with a State mental health administration. This criterion created incentives for academic researchers to work with States and helped ensure the relevance and utility of their work. The program also helped to develop the research capacity in several States.

Recommendation 31

NIMH should revise and renew PAs in the spirit of the PAL Program to maintain and promote existing partnerships between academic researchers and public care systems, health plans, both carve-outs and HMOs, and employers providing health benefits and their representative groups.

In addition, NIMH should make use of its current investments wisely. The Workgroup found that a significant effort should be made to enable the Intervention and Services Research Centers to develop ongoing partnerships with care delivery systems. The States’ public mental health systems are obvious partners, which Centers and other research groups should cultivate more extensively. In addition, managed care entities such as large HMOs, Preferred Provider Organizations, and managed care vendors are potential partners. The NIMH should judge the effectiveness of Centers, in part, by their ability to establish and utilize productive partnerships; at present, however, although these Centers often house needed research expertise, they may be constrained by their narrower missions and the funding levels. Permitting the supplementation of these efforts would be quite valuable. Once adequately supported, the ability to establish and maintain productive partnerships should be a core expectation for Centers.

Recommendation 32

NIMH research centers, which demonstrate great potential to secure partnerships with service delivery systems and to utilize these systems to conduct intervention research, should be supplemented to develop, implement, and sustain such partnerships.

Purchasers—government health plans as well as corporate, HMOs, and other MCOs—are important research partners for many reasons. Researchers increasingly are turning to purchaser plans as loci for research, especially service systems research. These clinical settings typically have large administrative databases useful in describing patterns of care. Research conducted in collaboration with database administrators could improve the availability, scope, and quality of data obtained in the course of routine mental health care, thereby facilitating more accurate case-mix adjustment, quality-of-care analyses, and outcomes analyses.

Treatment research also may benefit from examining large-scale observational data, such as claims files. In presentations to the Workgroup, individuals representing various practice settings described the promise of these data sets, but also the difficulty in accessing and maintaining them. As outlined substantively in Recommendation 15, the Workgroup felt that a more specific recommendation was required since this would be of immediate benefit to both sides of the research partnership with minimal investment. Members pointed to the need for support to enable information specialists and researchers to work on these data sets.

Recommendation 33

NIMH should issue an RFA or PA to encourage secondary analyses of service systems data and to establish accessible formats for such data to maximize the use of databases in practice settings.

New Alliances

In some cases, NIMH will need to foster initial alliances. Likely partners might include established academic sites seeking to build research programs with State mental health administrations, MCOs, group practices, hospitals, or businesses. For new partnerships, there are many aspects of planning and conducting research that must be considered and reconsidered. Taking advantage of what has worked at other sites can be a significant starting point in this trial and error process.

For instance, the New Hampshire-Dartmouth Psychiatric Research Center originated in such an award, as did the Research Division of the Connecticut Department of Mental Health. The story of the New Hampshire-Dartmouth Psychiatric Research Center is provided in the side bar as an example of successful partnership, but also to capture the flavor of the interactions in this partnership.

Recommendation 34

NIMH should stimulate new alliances by providing developmental funds to establish shared research resources such as data banks, staff time, consultant time, etc.

Recommendation 35

NIMH should commit resources to identify, describe, and disseminate models of successful partnerships.

Issue: Requirements for Meeting the Broader Scope of Public Health Research

PAs and RFAs afford NIMH program staff the opportunity to convey to the field the Institute’s plans for augmenting its portfolio. Therefore, it is critical that these announcements clearly and concisely articulate requirements for the research to be supported; issuances must outline general areas of need, provide clear examples, and ensure that the requested research is feasible and within budget allocations. These factors should be presented in the context of broader NIMH program objectives. Specifically, announcements should call for public health-oriented research, enunciating the type of research sought as suggested by this report and future priority setting. That is, make clear statements about the need for longer trials or a focus on rehabilitation outcomes. This tailoring is required to promote such complex undertakings, and clarify what is wanted for both the applicant and the reviewers.

Sometimes, the RFA and PA approach is not specific enough to meet programmatic goals. Contractual support may be preferable when a clearly articulated goal and plan of action is required. Emphasizing the targeted nature of the procured service (in this case conducting the research or providing the infrastructure for the research) in the RFPs is essential. If the goals and deliverables can be clearly specified to encompass priority intervention questions, a focus on contracting-out of large-scale trials through a coordinating center can extend the Institute’s ability to provide meaningful research findings. NIMH should be prepared to support active external advisory groups to aid in the ongoing oversight of such large-scale efforts.

Recommendation 36

NIMH should consider contracts as a mechanism for supporting clearly articulated research needs and deliverables.

Issue: Review Criteria

The NIH Director has established and the Center for Scientific Review (CSR) has implemented five specific criteria with which to assess applications. These criteria include:

• Significance—Does this study address an important problem? If the aims of the application are achieved, how will scientific knowledge be advanced? What will be the effect of these studies on the concepts or methods that drive this field?
• Approach—Are the conceptual framework, design, methods, and analyses adequately developed, well integrated, and appropriate to the aims of the project? Does the applicant acknowledge potential problem areas and consider alternative tactics?
• Innovation—Does the project employ novel concepts, approaches, or methods? Are the aims original and innovative? Does the project challenge existing paradigms or develop new methodologies or technologies?
• Investigator—Is the investigator appropriately trained and well-suited to carry out this work? Is the work proposed appropriate to the experience level of the principal investigator and other researchers (if any)?
• Environment—Does the scientific environment in which the work will be done contribute to the probability of success? Do the proposed experiments take advantage of unique features of the scientific environment or employ useful collaborative arrangements? Is there evidence of institutional support?

The Workgroup encourages NIMH to educate reviewers that the NIH criteria are highly relevant to evaluating scientific and public health significance of grant applications. The point is that innovation need not be limited to research design. The innovation can be in the aims, such that a new treatment evaluated by a standard clinical trials design can be quite innovative. The “new” treatment could also lack innovation if the treatment was actually only a minor modification of a previous treatment or an assessment of a drug that is highly similar chemically to another well-researched drug. Institute staff are urged to insert in future RFAs and RFPs specific criteria that will prompt reviewers to consider public health importance and policy significance when evaluating intervention applications.

Recommendation 37

The importance of public health relevance should be specifically added to the review criteria in new PAs, RFAs, and RFPs in the relevant areas to emphasize its significance for applicants and reviewers.

Issue: Expediting Administrative Decisions

In addition to listing criteria for review and award, RFAs and PAs contain a timeframe for review and award processes. Just as review criteria are important factors in tailoring responsive treatment and services research, the rapidity with which the Institute can make and act on funding decisions is critical to building and shaping the research portfolio in such fast-evolving areas as service systems research.

Opportunities for research collaborations, particularly with purchasers, often come and go before a traditional review and award process can be completed. For example, changes in welfare legislation that require wrap-around mental health services, or the shift by States to Medicaid managed care contracts, offered valuable but fleeting opportunities for research regarding the cost-effectiveness of alternative treatments and/or the impact of changes in payment mechanisms on people’s lives. Unfortunately, such initiatives typically are not fully evaluated because researchers cannot obtain research or evaluation monies from NIMH quickly enough through the traditional review and grant award process.

Developing an application for Federal support can take several months and an additional 9-12 months may elapse pending a funding decision from NIMH. The more complicated and expensive the project, the more likely that a longer timeframe will be required. Businesses or State mental health authorities cannot delay changes they want to or must implement until researchers can obtain support to evaluate such changes. Even if they could, staffing and budgets can change significantly over time. For these and other reasons, promises on the part of non-traditional research co-sponsors cannot be expected to endure without timely follow-through. The same often is true of private foundations that launch research or evaluation projects that would benefit—or require—supplemental funding from NIMH.

The Workgroup hoped that some more timely mechanism could be developed. The members’ shorthand for this was a “911 mechanism” for rapid review and consideration. Indeed, other institutes are exploring methods for shortening the 9-12 month framework.

Recommendation 38

NIMH should develop a mechanism for responding to unique, but fleeting research opportunities with great public mental health significance.

A related issue is the additional time and resources that might be required if an investigator assesses comorbid or other conditions. In practice settings, patients/consumers often have comorbid substance abuse and other health problems that complicate treatment and service plans. Ignoring these complexities reduces the clinical utility of information gained through treatment and services research. Yet, each Institute receives its own allocation from Congress to address these conditions separately [i.e., the National Institute on Alcohol Abuse and Alcoholism (NIAAA), the National Institute on Drug Abuse (NIDA), and NIMH]. Investigators can find themselves negotiating with three NIH institutes for grants or grant supplements, a process which they—not to mention business leaders and State mental health directors—describe as a significant burden. Representatives of these groups urged that monies for cross-cutting research be established and that cross-institute award processes be formalized and encouraged for research on comorbidity.

Recommendation 39

NIMH is asked to work with NIAAA and NIDA to encourage timely joint funding decisions for comorbid substance abuse and mental illness research.

Review Considerations

All research grant applications and contract proposals are subject to scientific peer review, wherein substantive or methodological experts evaluate the merit of the research. This case-by-case evaluation is a primary factor in funding decisions. The Workgroup emphasized the crucial role of peer review in fulfilling the NIMH’s scientific and public health mission.

Achievement of the field development activities proposed in this report will increase the burden on review groups. The number of applications requiring review will increase, as well as demands for a broader mix of expertise among reviewers and review staff to evaluate the scientific merit and practical significance of proposed work. Applications responding to calls for “boundary spanning” research will require reviewers who are able to assess the value of such research to diverse stakeholders including patients/consumers, families, purchasers, governmental agencies, and service providers.

Issue: Review Structure

The Workgroup conducted its work during a period of extensive change in peer review processes at the NIH. Following the 1992 NIMH reunification with NIH, certain distinctions were maintained, such as separate peer review for the three institutes that formerly comprised the Alcohol, Drug Abuse, and Mental Health Administration. In 1996, all NIH institutes with substantial investments in neuroscience initiated efforts to merge the separate peer review systems into one. By the time this Workgroup began its work, NIMH was completing the integration of its neuroscience and AIDS review committees, and had begun the behavioral science review merger process. Recognizing the unusual opportunities for change, the Workgroup requested that NIH reserve the treatment and service interventions for a later phase in the integration process; a delay would afford additional time to examine review issues for the portfolio and generate recommendations. Had these areas been merged, investigator-initiated applications to NIMH would be reviewed in newly created standing panels at the CSR, whereas applications in response to NIMH-issued RFAs or special mechanisms would undergo review by the Institute.

The integration of peer review within the NIH structure has afforded NIMH the opportunity to restructure its Institute-based review groups. Such restructuring is required in all areas, but the Workgroup limited its considerations to the treatment and services applications, where both investigator-initiated and certain types of funding mechanisms will continue to be reviewed by NIMH.

Research that attempts to draw from both the treatment and the services research worlds may find such boundary expansion difficult since the separate pools of reviewers rarely get to work together. Indeed, the Workgroup found that review groups responsible for efficacy, effectiveness, practice, and service systems research applications typically have had limited contact or crosstalk. As NIMH promotes the submission of boundary-spanning proposals for innovative interventions research, it must ensure that reviews be conducted by experts in all relevant domains who have an appreciation of the allied fields.

After examining two rounds of applications, the Workgroup concluded that, in general, applications for efficacy, effectiveness, and most of practice research should be reviewed in one committee. Service systems research and mental health economics research would form a second standing committee. Despite this division, there should be areas of overlap between the two committees in terms of expertise to handle applications at the cusp of this interface. As examples, both groups will require expertise in cost-effectiveness and in quality of care since practice research is likely to fall into both committees. Both groups will require expertise in substance abuse and in clinical epidemiology. Determination of assignment should be based on the major question the proposed research seeks to answer. Both committees should span all disorders and age groups.

The question of merging NIMH treatment and services applications into the CSR remains open at present. NIMH staff are participating in and learning from ongoing evaluations of the earlier mergers with CSR in neuroscience and AIDS. The Workgroup believes that unintended benefits and problems may arise, and that it will be important to examine these before determining what should stay in the NIMH review structure. NIMH should closely watch how such proposals fare within the traditional review structure, making adjustments if necessary to ensure that field development efforts have a chance to flower.

The Workgroup anticipates that should NIAAA and NIDA place their treatment and services applications into centralized CSR review, there could be a highly beneficial co-mingling of substance abuse and mental illnesses research. The Workgroup emphasized the importance of exploring shared review across these institutes. Apart from the benefits of more timely administrative decisions previously discussed, the Workgroup noted that considering shared review options would offer a means of reducing demands on finite reserves of reviewer and staff expertise. Redundant review structures may waste talent and time and hinder the review of applications looking at the complex interrelationships of co-occurring illnesses. NIMH should explore preliminary areas of joint review as opportunities for small tests emerge. Also, lessons from areas such as epidemiology and prevention, where such mergers will take place, may be a helpful predictor.

Recommendation 40

Efficacy, effectiveness, practice, and service systems research applications should remain at the Institute for review, and the clinical and services study sections reconstituted, including an appropriate mix of expertise among reviewers.

Recommendation 41

NIMH staff should attend carefully to planned evaluations of the CSR neurosciences and AIDS review mergers, as well as the upcoming behavioral sciences merger, to enhance future restructurings of NIMH review.

Recommendation 42

NIMH should explore shared review opportunities with NIAAA and NIDA to enhance the review of applications on comorbid substance abuse and mental illness applications.

Issue: Ensuring Expertise in Peer Review

In the competition for research funding common to all medical science, what constitutes expert review? In rapidly developing fields such as treatment and service systems research, generic challenges are exacerbated by special issues: How to ensure the availability of appropriate and sufficient review expertise from within a relatively small pool of experts and, given demands from all sectors of a field for the participation of those experts in multiple research activities, how to apply and adhere to Federal conflict-of-interest policies and procedures. The Workgroup addressed these questions, offering recommendations on issues under its purview and flagging others for the attention of the NIH leadership.

Peer review traditionally has focused on scientific expertise in the substance of the topic under review. In order to ensure appropriate and informed discussion, reasonable effort is made to have sufficient authorities in the subject matter of an application as members of a peer review panel. A more recent enhancement of the process has been the inclusion, when appropriate, of reviewers with expertise in the proposed methodology of a study. These reviewers may or may not have expertise in a given content area; the purpose of their participation is to advise knowledgeably whether a project will be able to address its stated aims using research and analytic techniques proposed by the researcher.

Similarly, review committees for science seeking to affect care may require the expertise of individuals with a clear understanding of the true public health need and the personal impact of mental illnesses. State mental health commissioners, patients/consumers, and family members are particularly well qualified to assess the potential impact of research within the care setting, to determine what would work in a care setting, and to help identify information most useful for planning and policymaking.

Recommendation 43

Review panels should have patients/consumers, providers, or policymakers as members to ensure expertise on the utility of research findings in intervention research.

Issue: Conflicts of Interest

A recurring issue in providing fair and expert review is the application of conflict-of-interest rules. With increasing frequency, reviewers are surprised to learn that they (and often other experts in a particular area) must recuse themselves from evaluating an application due to limited participation (e.g., consultation to a large multi-site study) on a related topic. Such consultation means that the potential reviewer not only has a conflict with the particular multi-site project, but with each of the investigators, consultants, and advisory group members on that project. As NIMH moves to promote large, collaborative treatment trials, one can imagine instances where virtually all experts in a small, cutting-edge area of investigation would be ineligible to serve as peer reviewers due to financial, personal, or scientific conflicts of interest. Actually, there is already evidence that this problem is affecting established areas in mental health research as well. The Workgroup generated no simple recommendation on this issue, but believes that the NIH leadership as a whole must engage this issue that occurs throughout health research.

Recommendation 44

NIMH staff should explore how other institutes deal with conflicts of interest and consult with NIH on revising and implementing the conflict-of-interest rules for intervention research.

Issue: Role of the Scientific Review Administrator (SRA)

The SRA is responsible for overseeing the review of a given application. This accomplished professional must ensure that each application receives fair and expert review and that the review process adheres to all pertinent Federal regulations and policies. Recent changes in review policy have shed light on a process that many applicants once viewed as shrouded in mystery. For example, applicants now receive in their summary statements, compiled critiques rather than a single integrated one; the former providing direct and comprehensive feedback, similar to that of a journal review. The strengths and weaknesses of each application are measured against specific review criteria. If the application is judged to be in the top half of those reviewed by a committee, the reviewers discuss their assessment of the application and this discussion, the weighing of strengths and weaknesses, is summarized and provided to the applicant in a resume.

Such steps to open the process notwithstanding, applicants who do not receive a competitive score often have lingering concerns about the review process. In an effort to further demystify this process, the Workgroup considered it useful to list the checks and balances that NIMH SRAs perform to ensure a fair and expert review. Workgroup members also suggested that these points be presented at professional meetings to enhance all potential applicants’ understanding of the review process.

• SRAs discuss assignments with the chair and members of the review committee.
• SRAs discuss potential committee members with the chair, members of the review committee, and program staff.
• SRAs discuss potential conflicts of interest that may affect an applicant’s review. All SRAs prefer to hear about such conflicts before, rather than after, the review.
• SRAs advise all reviewers that extreme opinions are part of peer review, but must be discussed at the review meeting. In the rare circumstance when a reviewer votes an extreme outlier score, without discussing it with the other review members, the SRA follows uniform guidelines for considering the deletion of the reviewer’s score in calculating the final score.
• In addition to their own judgments regarding the need for augmenting the committee’s expertise with an additional reviewer or an outside opinion, SRAs ask the committee chair and members to advise them on this matter.
• An appeals process exists that provides a re-review of an application, if a flawed review takes place. This policy is posted on the NIH web site.¹⁰

Reflecting upon the anticipated rise in applications at the interface of efficacy, effectiveness, practice, and service systems research, the Workgroup wished to emphasize the importance of two additional options available to SRAs for enhancing the review process.

Recommendation 45

SRAs should strive to maintain a link between special emphasis panels and the regular review committees through a subset of the regular reviewers and, whenever possible, the special emphasis panels should be arranged to precede or follow immediately the regular review meeting to facilitate the use of committee members and their scoring standards.

Recommendation 46

SRAs and panel chairs should actively consider when scientific breakthroughs require a substantive update for the review panel. The SRA and chair may convene a workshop to brief the committee.

Administration

Upon completion of review, NIMH program staff must decide which applications to fund. Principal sources of advice include review committee determinations and the recommendation of the NAMHC. An important consideration in making funding decisions is the balance between what exists in the Institute’s portfolio in a given area and research that is needed to address unanswered questions. For these reasons, program staff are encouraged to make decisions that are linked to, but not limited by, exact review committee ratings. In some circumstances, NIMH may need to support a study on a topic not rated as outstanding by the review committee but deemed useful in promoting field development. Likewise, on rare occasions, NIMH may decline to fund an application that, while highly rated, is judged unlikely to generate significant new information. Another factor requiring staff discretion is maintaining the vibrancy of the portfolio by investing not only in “blue chip,” lower risk research, but also having some “speculative,” higher risk research ventures. Council advice is invaluable when such judgment calls must be made.

Recommendation 47

NIMH should make judicious use of its ability to make funding decisions out of percentile order to achieve its programming objectives.

Once funding decisions are made, budgets need to be finalized and progress monitored. Investigators frequently face budget reductions, regardless of the review committee’s budgetary recommendations. While such reductions historically have affected all facets of the NIMH portfolio, they pose a special challenge to treatment and services research. Unlike research projects in which a few discrete studies can be dropped to meet cuts, treatment and services research offer only a few non-determined cost categories: the length of the trial, the number or types of participants, or the variables assessed. The Workgroup emphasized the need to ensure the integrity of all these elements or the generalizability of the findings will be significantly diminished. This delicate balance was discussed more fully in Chapter 4. The Workgroup urges NIMH staff and applicants to recognize that the importance of research lies in the utility of findings, and advises that it is better to fund one such study adequately than to underfund two studies.

Recommendation 48

Grant budgets must be based upon providing essential support as determined by review groups and NIMH program staff for achieving research’s scientific and public health aims.

After grants are awarded, staff should be vigilant in monitoring each project to determine if there are sudden opportunities that arise and should be fostered through administrative or competitive supplements. Program staff should encourage awareness in grantees of this option and notify investigators on the necessary procedures for requesting supplements.

Recommendation 49

NIMH should increase its use of administrative and competitive supplements to provide additional funds and time to researchers to test the generalizability of their findings or to disseminate important research findings.

References

Attkisson, C., Cook, J., Karno, M., Lehman, A., McGlashan, T.H., Meltzer, H.Y., O’Connor, M., Richardson, D., Rosenblatt, A., Wells, K., Williams, J., and Hohmann, A.A. Clinical services research. Schizophrenia Bulletin, 18(4):561–626, 1992.

Cronin, K.A., Weed, D.L., Connor, R.J., and Prorok, P.C. Case-control studies of cancer screening: Theory and practice. Journal of the National Cancer Institute, 90(7):498–504, 1998.

Depression Guideline Panel. Depression in Primary Care: Volume 1. Detection and Diagnosis. Clinical Practice Guideline, Number 5. Rockville, MD. U.S. Department of Health and Human Services, Public Health Service, Agency for Health Care Policy and Research. AHCPR Publication No. 93-0550, April 1993.

Frances, A.J., Kahn, D.A., Carpenter, D., Docherty, J.P., and Donovan, S.L. The Expert Consensus Guidelines for treating depression in bipolar disorder. Journal of Clinical Psychiatry, 59(Suppl. 4):73–79, 1998.

Glisson, G. and Hemmelgarn, A. The effects of organizational climate and interorganizational coordination on the quality and outcomes of children’s service systems. Child Abuse & Neglect, 22(5):401–421, 1998.

Howard, P.A. and Duncan, P.W. Primary stroke prevention in nonvalvular atrial fibrillation: Implementing the clinical trial findings. Annals of Pharmacotherapy, 31(10):1187–1196, 1997.

International Psychopharmacology Algorithm Project Report Participants. International Psychopharmacology Algorithm Project Report. Psychopharmacology Bulletin, 31(3):457–507, 1995.

Katon, W., Von Korff, M., Lin, E., Walker, E., Simon, G.E., Bush, T., Robinson, P., and Russo, J. Collaborative management to achieve treatment guidelines. Impact on depression in primary care. Journal of the American Medical Association, 273(13):1026–1031, 1995.

Lehman, A.F., Steinwachs, D.M., and the Survey Co-Investigators of the PORT Project. Patterns of Usual Care for Schizophrenia: Initial Results from the Schizophrenia Patient Outcomes Research Team (PORT) Client Survey. Schizophrenia Bulletin, 24(1):11–20, 1998.

McClellan, W.M., Knight, D.F., Karp, H., and Brown, W.W. Early detection and treatment of renal disease in hospitalized diabetic and hypertensive patients: Important differences between practice and published guidelines. American Journal ofKidney Diseases, 29(3): 368–375, 1997.

National Advisory Mental Health Council. Caring for People with Severe Mental Disorders: A National Plan of Research to Improve Services. Washington, DC: Superintendent of Documents, U.S. Government Printing Office, DHHS Publication No. (ADM) 91-1762, 1991.

National Advisory Mental Health Council Workgroup on Mental Disorders Prevention Research. Priorities for Prevention Research at NIMH. Bethesda, MD: National Institutes of Health/National Institute of Mental Health, NIH Publication No. 98-4321, 1998.

National Association of State Mental Health Program Directors Research Institute, Inc. Closing and Reorganizing State Psychiatric Hospitals: 1996. Report of the State Mental Health Agency Profiles System. Alexandria, VA: National Association of State Mental Health Program Directors Research Institute, Inc., 1996.

President’s Commission on Mental Health. Report to the President From the U.S. President’s Commission on Mental Health. Vol. 1. Washington, DC: Superintendent of Documents, U.S. Government Printing Office, 1978.

Redick, R.W., Witkin, M.J., Atay, J.E., and Manderscheid, R.W. Highlights of organized mental health services in 1992 and major national and state trends. In: Manderscheid, R.W. and Sonnenschein, M.A., eds. Mental Health, United States 1996. Center for Mental Health Services, Substance Abuse and Mental Health Services Administration, DHHS Publication No. (SMA)96-3098, 1996. pp. 90–137.

Spitzer, R.L., Forman, J.B.W., and Nee, J. DSM-III field trials: I. Initial interrater diagnostic reliability. American Journal of Psychiatry, 136(6):815–817, 1979.

Turner, J.C. and TenHoor, W.J. The NIMH Community Support Program: Pilot approach to a needed social reform. Schizophrenia Bulletin, 4(3):319–348, 1978.

U.S. General Accounting Office. Returning the Mentally Disabled to the Community: Government Needs to Do More. Washington, DC: Superintendent of Documents, U.S. Government Printing Office, 1977.

Vitiello, B. and Jensen, P. Medication development and testing in children and adolescents. Archives of General Psychiatry, 54:871–876, 1997.

Williams, J.B., Gibbon, M., First, M.B., Spitzer, R.L., Davies, M., Borus, J., Howes, M.J., Kane, J., Pope, H.G., Jr., Rounsaville, B., and Wittchen, H-U. The Structured Clinical Interview for DSM-III-R (SCID). II. Multisite test-retest reliability. Archives of General Psychiatry, 49:630–636, 1992.

Appendix A
National Advisory Mental Health Council

Chairperson

Steven E. Hyman, M.D.
Director
National Institute of Mental Health
Rockville, Maryland

Executive Secretary

Jane A. Steinberg, Ph.D.
Acting Director
Division of Extramural Activities
National Institute of Mental Health
Rockville, Maryland

Members

Thomas J. Coates, Ph.D.
Professor
Director, UCSF AIDS Research Institute and Center for AIDS Prevention Studies
University of California, San Francisco
San Francisco, California

Kathy Cronkite
Mental Health Advocate
Austin, Texas

Mary Jane England, M.D.
President
Washington Business Group on Health
Washington, DC

Ellen Frank, Ph.D.
Professor of Psychiatry and Psychology
Department of Psychiatry
School of Medicine
University of Pittsburgh
Pittsburgh, Pennsylvania

Apostolos Georgopoulos, M.D., Ph.D.
Professor, Department of Physiology, Neurology and Psychiatry
University of Minnesota Medical School
Director, Brain Sciences Center
Veterans Administration Medical Center
Minneapolis, Minnesota

Ann M. Graybiel, Ph.D.
Walter A. Rosenblith Professor
Department of Brain and Cognitive Sciences
Massachusetts Institute of Technology
Cambridge, Massachusetts

Michael F. Hogan, Ph.D.
Director
Ohio Department of Mental Health
Columbus, Ohio

Dale L. Johnson, Ph.D.
Professor
Department of Psychology
University of Houston
Houston, Texas

Robert L. Johnson, M.D.
Director of Adolescent Medicine
Department of Pediatrics
University of Medicine and Dentistry of New Jersey
Newark, New Jersey

Anne C. Petersen, Ph.D.
Senior Vice President for Programs
W. K. Kellogg Foundation
Battle Creek, Michigan

John Rush, M.D.
Professor, Betty Jo Hay Distinguished Chair
Department of Psychiatry
Southwestern Medical Center
University of Texas
Dallas, Texas

Richard H. Scheller, Ph.D.
Investigator,
Howard Hughes Medical Institute
Professor
Department of Molecular and Cellular Physiology
Stanford University
School of Medicine
Stanford, California

G. Richard Smith, Jr., M.D.
Professor
Director of Centers for Mental Healthcare Research
Department of Psychiatry
University of Arkansas for Medical Sciences
Little Rock, Arkansas

José Szapocznik, Ph.D.
Professor and Director
Center for Family Studies
Department of Psychiatry and Behavioral Sciences
University of Miami
School of Medicine
Miami, Florida

Joseph S. Takahashi, Ph.D.
Investigator
Howard Hughes Medical Institute
Walter and Mary E. Glass Professor
Department of Neurobiology and Physiology
Northwestern University
Evanston, Illinois

James G. Townsel, Ph.D.
Professor and Director
Center for Molecular and Behavioral Neuroscience
School of Medicine
Meharry Medical College
Nashville, Tennessee

Myrna M. Weissman, Ph.D.
Professor
Department of Clinical and Genetic Epidemiology
New York State Psychiatric Institute
Columbia University
New York, New York

Ex Officio Members

Office of the Secretary, DHHS

Donna E. Shalala, Ph.D.
Secretary
Department of Health and Human Services
Washington, DC

National Institutes of Health

Harold E. Varmus, M.D.
Director
National Institutes of Health
Bethesda, Maryland

Department of Defense

Robert A. Mays, Jr., Ph.D.
Colonel, U.S. Army
Office of the Inspector General
North Atlantic Regional Medical Command
Walter Reed Army Medical Center
Washington, DC

Department of Veterans Affairs

Thomas B. Horvath, M.D., F.R.A.C.P.
Chief Consultant for Mental Health
Department of Veterans Affairs
Veterans Health Administration
Washington, DC

Liaison Representative

Center for Mental Health Services

Thomas H. Bornemann, Ed.D.
Deputy Director
Center for Mental Health Services
Substance Abuse and Mental Health Services Administration
Rockville, Maryland

Special Consultants

Constance E. Lieber
President
National Alliance for Research on Schizophrenia and Depression
Great Neck, New York

Edward Scolnick, M.D.
President
Merck Research Laboratories
West Point, Pennsylvania

Appendix B
NIMH Clinical Treatment and Services Research Workgroup

Members

A. John Rush, M.D.** (Chairman)
Professor, Betty Jo Hay Distinguished Chair
Department of Psychiatry
Southwestern Medical Center
University of Texas
Dallas, Texas

Michael F. Hogan, Ph.D.**
Director
Ohio Department of Mental Health
Columbus, Ohio

Howard Abikoff, Ph.D.
Professor of Clinical Psychiatry
Director of Research
New York University Child Study Center
New York University School of Medicine
New York, New York

Kathy Cronkite**
Mental Health Advocate
Austin, Texas

Mary Jane England, M.D.**
President
Washington Business Group on Health
Washington, DC

Susan Essock, Ph.D.
Director
Division of Health Services Research and Professor
Department of Psychiatry
Mount Sinai School of Medicine
New York, New York

Ellen Frank, Ph.D.**
Professor of Psychiatry and Psychology
Department of Psychiatry
School of Medicine
University of Pittsburgh
Pittsburgh, Pennsylvania

Junius Gonzales, M.D.
Deputy Chairman
Department of Psychiatry
Georgetown University Medical Center
Washington, DC

Dale L. Johnson, Ph.D.**
Professor
Department of Psychology
University of Houston
Houston, Texas

Robert L. Johnson, M.D.**
Director of Adolescent Medicine
Department of Pediatrics
University of Medicine and Dentistry of New Jersey
Newark, New Jersey

John Kane, M.D.
Chairman
Department of Psychiatry
Long Island Jewish Medical Center
Glen Oaks, New York

Lon S. Schneider, M.D.
Professor
Department of Psychiatry
University of Southern California
School of Medicine
Los Angeles, California

G. Richard Smith, Jr., M.D.** 
Professor
Director of Centers for Mental Healthcare Research
Department of Psychiatry
University of Arkansas for Medical Sciences
Little Rock, Arkansas

Kenneth B. Wells, M.D., M.P.H.
Director
Research Center in Managed Care for Psychiatric Disorders
Professor
Department of Psychiatry
UCLA Neuropsychiatric Institute
Los Angeles, California

NIMH Staff to the Workgroup

Staff Director

Jane A. Steinberg, Ph.D.
Division of Extramural Activities

Associate Staff Director

Susan Matthews
Division of Extramural Activities

Science Writer

Paul J. Sirovatka, M.S.
Office of Science Policy and Program Planning

Art and Graphics

Catherine W. West
Office of Resource Management

Editor

Joan A. Cole
Division of Extramural Activities

Typist

Debra S. Dabney
Division of Extramural Activities

Advisors

Barry D. Lebowitz, Ph.D.
Division of Services and Intervention Research

Kathryn M. Magruder, Ph.D., M.P.H.
Division of Services and Intervention Research

Jean G. Noronha, Ph.D.
Division of Extramural Activities

Grayson S. Norquist, M.D., M.S.P.H.
Division of Services and Intervention Research

Benedetto Vitiello, M.D.
Division of Services and Intervention Research

Appendix C
Roster of Consultants*

Robert Battjes, D.S.W.
Division of Clinical and Services Research
National Institute on Drug Abuse

Soo Borson, M.D.
University of Washington
School of Medicine

James Burris, M.D.
Department of Veterans Affairs

Gerald E. Calderone, Ph.D.
Division of Extramural Activities, NIMH

Sarah Carr
Office of Policy Analysis
National Institute of Allergy and Infectious Diseases

Isabel S. Davidoff
Division of Services and Intervention Research, NIMH

Robert E. Drake, M.D., Ph.D.
New Hampshire-Dartmouth Psychiatric Research Center

Michael J. English, J.D.
Center for Mental Health Services
Substance Abuse and Mental Health Services Administration

Bennett Fletcher, Ph.D.
Division of Clinical and Services Research
National Institute on Drug Abuse

Barbara Geller, M.D.
Washington University
School of Medicine

Paul Gorman, Ed.D.
Division of Behavioral Health
Department of Health and Human Services
Concord, New Hampshire

Henry J. Haigler, Ph.D.
Division of Extramural Activities, NIMH

Ann Hohmann, Ph.D., M.P.H.
Division of Services and Intervention Research, NIMH

Steven E. Hyman, M.D.
Director, NIMH

Douglas B. Kamerow, M.D., M.P.H.
Center for Practice and Technology Assessment
Agency for Health Care Policy and Research

Philip Lavori, Ph.D.
Palo Alto Veterans Affairs Health Care System

Anthony F. Lehman, M.D.
University of Maryland School of Medicine

Bruce Levin, Ph.D.
Columbia School of Public Health

Teresa Levitin, Ph.D.
Office of Extramural Program Review
National Institute on Drug Abuse

Lydia Lewis
National Depressive and Manic Depressive Association

Clarissa Marques, Ph.D.
Green Spring Health Services

Robert A. Mays, Jr., Ph.D.***
Colonel, U.S. Army
Department of Defense

Carolyn Mercer, Ph.D.
Dartmouth Medical School

Richard K. Nakamura, Ph.D.
Office of the Director, NIMH

Harold Perl, Ph.D.
Division of Clinical and Prevention Research
National Institute on Alcohol Abuse and Alcoholism

Anne C. Petersen, Ph.D.**
W.K. Kellogg Foundation

Stanley Rosenberg, Ph.D.
Dartmouth Medical School

Agnes Rupp, Ph.D.
Division of Services and Intervention Research, NIMH

Robert H. Stretch, Ph.D.
Division of Extramural Activities, NIMH

Carolyn Strete, Ph.D.
Division of Mood Disorders,
Behavioral Research and AIDS, NIMH

Ronald H. Weich, J.D.
Zuckerman, Spaeder, Goldstein, Taylor and Kolker, L.L.P

Washington Business Group on Health Research Advisory Group Meeting April 23, 1998

Angela G. Burgess
Washington Business Group on Health

Wayne Burton, M.D.
First Chicago NBD

Isabel S. Davidoff
National Institute of Mental Health

Bruce Davidson, M.P.H.
Digital Equipment Corporation

Lynne DeGrande, M.S.P.H.
General Motors Employee Assistance Program (Senior Consultant)
DeGrande and Associates

Mary Jane England, M.D.**
Washington Business Group on Health

Susan M. Essock, Ph.D.
Mount Sinai School of Medicine

Veronica Goff
Washington Business Group on Health

William Goldman, M.D.
United Behavioral Health

James Harburger, M.D.
Harvard Pilgrim Health Care

Paul Johnson, M.D.
US West

Kathryn M. Magruder, Ph.D., M.P.H.
National Institute of Mental Health

Grayson S. Norquist, M.D., M.S.P.H.
National Institute of Mental Health

Jane A. Steinberg, Ph.D.
National Institute of Mental Health

David Whitehouse, M.D.
MCC Behavioral Health Care

Center for Mental Health Services Staff/National Advisory Council
May 5, 1998 Meeting

Bernard S. Arons, M.D.
Center for Mental Health Services

Thomas Bornemann, Ed.D.
Center for Mental Health Services

Frank G. Burgmann
G. Pierce Wood Memorial Hospital

Michael J. English, J.D.
Center for Mental Health Services

Marshall Forstein, M.D.
Fenway Community Health Center

Ruby J. Martinez, R.N., Ph.D.
University of Colorado Health Sciences Center

Anne Mathews-Younes, Ed.D.
Center for Mental Health Services

Donna Mayeaux
Louisiana Alliance for the Mentally Ill

Andres J. Pumariega, M.D.
East Tennessee State University

Ian A. Shaffer, M.D.
Value Behavioral Health

Steven P. Shon, M.D.
Texas Department of Mental Health and Mental Retardation

David Shore, M.D. (Liaison)
National Institute of Mental Health

David K. Yamakawa, Jr., J.D.
Attorney-at-Law

Footnotes

1. Interventions also refer to preventive strategies, which were addressed in a previous NAMHC Workgroup report entitled Priorities for Prevention Research at NIMH: A Report by the National Advisory Mental Health Council Workgroup onMental Disorders Prevention Research. For purposes of this report, intervention refers to either treatment or services delivery.
2. Individuals with mental illnesses sometimes are referred to as patients or consumers. Each term has its own merits, and to capture these strengths patients/consumers also will be used in this report.
3. Attkisson, C. et al. Clinical services research. Schizophrenia Bulletin, 18(4):561-626, 1992.
4. A grant is a financial assistance mechanism whereby money and/or direct assistance is provided to carry out approved activities, and is to be used whenever the PHS awarding office anticipates no substantial programmatic involvement with the recipient during performance of the financially assisted activities. A research and development contract project is a circumscribed activity identified by the government agency that may involve a single contract or two or more similar, related, and interdependent contracts encompassing the project concept and approach.
5. Web site: http://www.nih.gov/grants/guide/index.html 
6. An RFA is a formal statement that invites grant or cooperative agreement applications in a well-defined scientific area to accomplish specific program objectives, and it indicates the estimated amount of funds set aside for the competition, the estimated number of awards to be made, and the application receipt date(s).
   A PA is a formal statement about a new or re-emphasized, ongoing NIH extramural activity, and, except for a few programs, applications in response to PAs may be submitted for any appropriate receipt date and are reviewed with all other applications received at that time.
7. Mailing address: Superintendent of Documents, Government Printing Office, Washington, DC 20402; web site: https://www.gpo.gov/ 
8. NIMH web site: http://www.nimh.nih.gov
9. Web site: http://cbdnet.access.gpo.gov 
10. Web site: http://www.nih.gov/grants/guide/1997/97.11.21/index.html

* The workgroup appreciates the contribution of the consultants in developing this report; however, inclusion in the listing does not necessarily indicate endorsement of the workgroup’s recommendations.

** Member of the National Advisory Mental Health Council

*** Ex Officio Member of the National Advisory Mental Health Council

Updated: January 14, 1998