The Center for Global Mental Health Research Webinar Series 2023: Human Subjects Protection, Data and Safety Monitoring, and Operational Considerations in NIMH-Funded Clinical Research

Transcript

LEONARDO CUBILLOS: Good Morning, good afternoon and good evening. Thank you for joining us today. Today’s webinar will be on a clinical research overview of clinical trials, human subjects protection, data and safety monitoring and good clinical practice. I am Leonardo Cubillos. I am the Director of the Center for Global Mental Health Research at the National Institute of Mental Health.

Today’s webinar is the second webinar of our 2023 Center for Global Mental Health Webinar Series and, since we started this series, is the ninth webinar. In 2022 we had seven webinars that covered different scientific and administrative matters we think are important, and in 2023 thus far we have had two webinars. You can find all these recordings as well as the transcripts at the NIMH web page.

As I mentioned, we will be discussing clinical research overview and good practices in clinical research as well as other very important aspects.

I want to introduce our three speakers, but before I do that I want to remind you that the third webinar in the 2023 series will be on September 20 and we will be discussing RDoC in global health.

Our first speaker for today is Sabiha Ethridge. Sabiha is a health policy analyst at NIMH’s Office of Clinical Research. She has been with NIMH since 2013 and she has different responsibilities where she focuses on human subjects protection for the NIH clinical trials and clinical studies. Sabiha received her bachelor’s degree in psychology from Duke University and she has a Master of Public Health from Boston University.

Our second speaker is Ashley Kennedy. Ashley also works at the Office of Clinical Research where she coordinates the clinical trial programs in the Clinical Trials Operations Branch within the Office of Clinical Research. She supports different extramural research activities and she currently serves in different positions helping and supporting global as well as domestic research sites.

Our last speaker is Anna Ordonez. Anna is the Director of the Office of Clinical Research at the National Institute of Mental Health. Her background is a psychiatrist and a child and adolescent psychiatrist. She also completed a master’s degree in health administration sciences at UCSF. Prior to coming to NIMH Anna worked as faculty at UCSF where she also was involved in different areas of research.

As you see, our three speakers today come from the Office of Clinical Research and we are delighted to have this office supporting the webinar series, and without further ado I will pass it to Anna who will kick us off in this great journey in learning about clinical research in clinical trials, human subjects protection, data and safety monitoring and good clinical practice.

I wish you a great webinar and I will see you later during the round of questions and answers.

ANNA ORDONEZ: Thank you, Leo, and good morning, good afternoon and good evening wherever you are in the world. It is a pleasure for the Office of Clinical Research staff to be here today with you to present this webinar. I am very excited to have Sabiha Ethridge and Ashley Kennedy joining me in this presentation. The Office of Clinical Research provides consultation and support to our institute in human research protections, in clinical trials operations, data and safety monitoring and many of the topics you’re going to be hearing about today.

We have Sabiha Ethridge joining us from the Human Research Protection Branch. She is a health policy analyst who has been with us for quite some time, since 2013, and has a host of responsibilities including providing a lot of consultation and support to our NIMH staff and, in turn, to investigators in the field related to human research protections.

Ashley Kennedy, who is from our Clinical Trials Operations Branch, is a clinical trial program coordinator and provides a lot of support to investigators who have studies that are coming to our Data Safety Monitoring Board in terms of operational support, so you will be hearing from her related to operations.

We are excited to start today. This presentation has a lot of information in it so I am going to not take anymore time and hand this over to Sabiha to get us started. We look forward to hearing your questions at the end of the session and having some time to discuss.

Thank you all very much. On to you, Sabiha.

SAHIBA ETHRIDGE: Thank you, Anna. Like Anna said, this presentation has a lot of information so bear with us. I also want you to note that while you might not have the slide decks right now you might be able to access the slides later on in a few weeks and you will probably receive correspondence via email on whether or not those slides have been put up.

I would like to start today’s presentation by saying that we are going to primarily focus on the PHS Human Subjects and Clinical Trial Information Form in the grant application, as well as some additional points to consider when you’re filling out the grant application. The next few slides specifically speak to Section 3, Protection and Monitoring Plans, of the PHS Human Subjects and Clinical Trials Information Form.

As an overview, I will primarily be talking about this section in the next few slides. This section is broken down into five subsections and I will go into more detail on each of these sections as we progress through the presentation.

Generally, the PHS Human Subjects and Clinical Trials Information Form is used to collect information on human subjects research, clinical research and clinical trials. You want to include study population characteristics, protection and monitoring plans and a protocol synopsis in this section. Also, this form collects information on human subjects in clinical trials on the study level. All applicants must use this form regardless of whether you answer yes to the question, are human subjects involved, on the R&R Other Project Information Form.

One thing to note is that if you answer no to “are human subjects involved” then you will go from that question and skip through the rest of the PHS Human Subjects and Clinical Trials Information portion and move on with filling out the grant application.

There is a detailed list of instructions on NIH’s website. There’s a document called the General Instructions for NIH and Other PHS Agencies, and this document breaks down each section of the grant application and provides corresponding guidance for you to reference as you fill out the application.

If you answered yes to “are human subjects involved in your proposed research” you will need to add a study record for each one of your proposed studies that will be involving human subjects research, and you do that by clicking on the “add new study” button. If you are conducting human subjects research and you are claiming an exemption, you must justify why your research meets the criteria for that exemption, and you should make sure to not just include the definition of the exemption or verbatim what might be in the regulations. You want to make it specific to why your research meets the criteria for that exemption.

I provided some links that you will get once these slides are posted for some helpful resources to help you determine if you are doing human subjects research or if you are doing exempt human subjects’ research. NIH specifically has an exempt human subjects research infographic on their website that is very helpful.

If you are not proposing to do human subjects research but you plan on using human specimens or data, you have to provide a justification for your claim of no human subjects involvement. It is very important to make sure at that point that you really are truly not doing human subjects research. That can be something where we have seen applicants who think they are not doing human subjects research but are actually doing it. So it’s important to work with your IRB and your regulatory bodies, and if you have an office of human research protection that is also a good resource to use when trying to figure out if you are doing human subjects research or not.

The justification you give for whether your study meets the criteria for exemption will be reviewed during peer review so it’s encouraged that you work with the resource that you have at your institution to make sure you are accurately determining whether your study meets the exemption or not.

The HHS Office for Human Research Protection has some helpful resources that can help you in determining whether or not you are doing human subjects research, or, if you are doing human subjects research, they have many tutorials and videos at their website that are very helpful. NIH also has helpful resources. One of them that I particularly like is the NIH’s decision tool. This decision tool basically presents a series of multiple choice responses that help you determine whether or not you are doing human subjects research or you’re doing exempt human subjects research.

Once you determine that you are doing non-exempt human subjects research, NIH requires that you fill out their Protection of Human Subjects Attachment. The information that you put in this attachment needs to be commensurate with the risks of the study, the size of your study and the complexity of your study. It is also important to keep in mind that this Protection of Human Subjects Attachment is reviewed during peer review.

The Protection of Human Subjects Attachment is broken out or organized in four sections, and it’s very important to make sure that you are providing the information that is relevant to each of these sections. You have the risks to human subjects -- what are the risks that are going to be involved for people participating in your study. What are the strategies you have in place to minimize or protect against these risks? What are the potential benefits of the proposed research to the subjects? And what is the importance of the knowledge to be gained?

One thing I want to note, too, is that in this section there may be relevant in-country laws or regulations or policies that you may need to mention that are country-specific. For example, the age of consent for medical intervention or research participation may differ from country to country and you want to make sure you bring that up in this particular section.

In writing the Protection of Human Subjects Attachment it is very important to identify all the potential risks for the participants that are going to be involved in your study and to consider the likelihood and the seriousness of these risks. Something I want to note is that you need to make sure that you’re considering everyone that would be considered a participant.

We see often where individuals who are receiving the intervention are considered the participants, but some of the other participants may not be described in this section, so it is important to consider who exactly is providing study data. It could be patients at a clinic, it could be clinic staff, it could be guardians or parents. So that is important to consider when you’re considering the risks to human subjects.

In this section, the first thing you want to do is provide a description of the human subjects’ involvement, characteristics and design. Specifically, a brief overview of the study design, a description of the study population, for example, how old are the people that you are going to enroll. Who are the people? What are the inclusion/exclusion criteria for the individuals you plan on enrolling? Will you be enrolling any vulnerable populations?

The second thing you want to consider are the study procedures, the materials and potential risks to subjects associated with the intervention. Specifically, describe your planned research procedures involving human subjects. What research materials, like biospecimens, data, study records, will you be obtaining?

You should also describe any private identifiable information that you might collect in this section as well. And, very importantly, in this section you want to discuss the risks to all potential subjects. These risks could be physical, psychosocial, cultural, financial, legal risks. You want to talk about risk of loss of privacy and confidentiality or any other risks that should be considered.

It is also important to talk about the risks associated with each of the interventions and procedures that you might have. If individuals are undergoing multiple procedures, so if they’re having surveys or interviews or also going through MRI scan, you want to make sure you discuss the specific risks of each one of those.

Once you have discussed the study risks you next want to talk about what strategies you have in place to mitigate these risks or protect against them, and the first part of that is discussing the informed consent process. You want to talk about the informed consent process as well as how you are going to be recruiting individuals. For instance, are you required to obtain consent or assent or parental consent? Do you plan on requesting a waiver of informed consent from your IRB? Who is going to be seeking consent? What methods are you going to use to document consent? Are you going to obtain signatures, is it going to be oral verification, is it going to be a thumbprint?

Another factor to think about in filling out this section is any strategies that you have in place to mitigate the risks you described in the previous section where you listed the risks to study participants. This section is also where you describe incidental findings such as those that may be found during imaging or screening tests or diagnostic tests or paternity tests. And lastly, if you are going to involve a vulnerable population, this is the section where you want to discuss that.

Next, you want to consider the potential benefits of the proposed research to the subjects or to others. The study may have a benefit for individuals like individual participants or it may benefit an entire population or community, but it is important to put that information in this section.

You should also adequately describe how the potential risks to the participants appear reasonable in relation to the anticipated benefits of the study. One thing that I think is really important to note and that we see often is financial compensation should not be listed as a benefit of study participation in research because it can appear coercive in nature. We recommend that it not be included in this part of the Attachment.

Lastly, you want to address the importance of the knowledge to be gained as a result of the proposed research, and you want to discuss why the risks to subjects are reasonable in relation to the importance of the knowledge that can be expected to be gained as a result of the study.

For the next subsection of the PHS Human Subjects and Clinical Trials Information Form, if you answer yes to your study being a multisite study that you will use the same protocol to conduct non-exempt human subjects research at more than one domestic site -- I just want to note that by domestic site I mean US site -- then you will be required to use a single IRB to conduct the ethical review of the protection of human subjects.

For NIH applicants the single IRB plan attachment is no longer required; instead, you have to provide a statement naming the single IRB of record in the “just in time” submission prior to award to make sure you are adhering to the NIH single IRB policy requirement.

As a note, the single IRB policy does not apply to certain types of grants, so, career development grants, research training grants, fellowship awards. Also, it is important to note that foreign sites participating in NIH-funded multisite studies are not expected to follow the single IRB policy.

If you answer yes to all the questions in the clinical trial questionnaire, which is in the PHS Human Subjects in Clinical Trials Information Form, then you are required by NIH grant policy to include a Data and Safety Monitoring Plan, or DSMP. For human subjects research that doesn’t meet criteria for a clinical trial, investigators still have the option to include a DSMP, but it is not required. An example of this would be if you have a study that is not a clinical trial but may have significant risk to participants. Your DSMP is required to be commensurate with the risks of the trial, the size of your trial and the complexity of the trial.

The DSMP should include the following content as appropriate. You want to indicate how many people and what type of entity will be providing monitoring. Is it going to be the PI along with the IRB or will you have an independent safety monitor or DSMB? You also want to provide a description of the overall framework for safety monitoring and what information exactly will be monitored.

You want to talk about the frequency of monitoring including any plans for interim analysis or any stopping rules if they are applicable to your study.

And then you want to talk about how adverse events will be managed and reported in accordance with specific policies and guidance. You may have specific policies and guidance from your IRB, from the individuals or group that are monitoring the study, any other ethical or regulatory bodies that might be involved, and also the NIH awarding institution. NIMH, for instance, has a reportable events policy that requires NIH-funded researchers adhere to.

You may also want to include in your DSMP who will be responsible for monitoring your study. Will it be the PI and IRB? Will it be an ISM, will it be the DSMB? In some cases, it is acceptable for a PI and IRB alone to monitor the study, and these tend to be for lower-risk studies that are blinded. In all cases I want to note that the PI and IRB retain their monitoring and oversight responsibilities regardless of any additional monitoring requirement that might be added. If you were to have a DSMB, a DSMB enhances the monitoring responsibilities of the PI and IRB; they do not replace them.

Independent safety monitors can be used as well. These are independent physicians or other appropriate experts with relevant expertise who can advise the study team or the NIH institution. You can also have independent monitoring committees or safety monitoring committees. These are small groups of independent experts that monitor the study. And then you can have a data and safety monitoring board, or DSMB. A DSMB is a formal independent board of experts that may include investigators or biostatisticians but that are not associated with the study.

One thing I want to note is that sometimes we see the term “independent monitoring committee” or “safety monitoring committee” or “data safety monitoring board” used interchangeably, but these are different monitoring entities so it is important when you’re filling out the DSMP to make it very clear which one you are using. I am going to go into more detail about DSMBs in some of the upcoming slides.

NIH and NIMH have some helpful resources - the links are here on the slide - for filling out the DSMP. I would encourage you to reference these as you are writing your grant application.

Next, if you answered yes to all the questions in the clinical trial questionnaire you are required to answer the question regarding a data safety monitoring board. This is optional for all other human subjects research, but if you answer yes to all the clinical trials questionnaire questions then you are required to do it.

As noted in the previous slide, the monitoring responsibilities of a DSMB enhance those of the PI and IRB; they do not replace them. NIH requires the establishment of DSMBs for certain studies; for instance, multisite clinical trials involving interventions that entail potential risks to participants and also generally for all Phase III clinical trials. As necessary, a DSMB may be appropriate for other studies, for instance, Phase I or Phase II clinical trials or other clinical trials that may be blinded or employ high-risk interventions or involve vulnerable populations.

For NIMH grantees there are two types of DSMBs that you may choose to have in your study or may be assigned to your study. The first one is an external independent DSMB that is usually established by the study team. The second is an NIMH-constituted independent DSMB which is appointed and convened by the NIMH Director. NIMH-sponsored DSMBs can be assigned for studies that have significantly greater than minimal risk or high-risk studies or at the discretion of NIMH leadership.

If a DSMB is used, at the time of your application you want to make sure you describe the composition of the board without naming specific individuals. You may say that you will have a biostatistician, or you'll have another relevant expert in the field of your research.

DSMB members are expected to be independent of any professional or financial conflict-of-interest with the research project or any of the investigators on the study. As a note, once your grant is funded it is expected that your IRB will provide the NIMH with the DSMB members’ specific qualifications and conflict-of-interest statements indicating that they have no conflict-of-interest, no direct involvement with the study or with the investigators. NIH has some helpful links regarding monitoring of NIMH-funded studies as well as DSMBs on the NIMH OCR website.

Lastly, in Section 3, you are going to talk about the overall structure of the study team. This attachment is optional, but I would suggest that you refer to your NOFO, Notice of Funding Opportunity, for specific instructions on the overall structure of the study team attachment because they may have some specifications in there.

Within the attachment you want to provide a brief overview of the organizational structure of the study team. You want to talk about administrative sites, data coordinating sites, enrollment/participating sites and any separate labs or testing centers you might be using.

I want to note that in this section you should not include study members’ individual professional experiences. So the information that you had in your bio sketch, this is not the section where you would put that.

Now that we have covered points to consider when filling out the Protection of Human Subjects Attachment and before moving on to the next segment of our presentation, I want to talk about some of the commonly identified human subject issues that come up during NIMH review of applications, specifically the Human Research Protection Branch of the Office of Clinical Research.

The NIMH program officers will consult with the Human Research Protection Branch and the Office of Clinical Research on matters involving the protection of human subjects so it is not unusual that we would see these, and these come up often. This list is not inclusive, but these are some of the more frequent things we see.

Oftentimes we see missing or inadequate DSMPs. A DSMP might have no mention of how adverse events will be monitored or reported. They may be doing screenings or imaging and there are no plans for how incidental findings will be handled.

There may be insufficient clinical oversight, so there may be a medication washout proposed but there is no indication of whether or not there’s an involvement of a licensed physician or other professional who will be monitoring this washout.

Another thing we see often is, if pregnancy is listed as an exclusion criterion in the study it is important to make sure you have a plan and a method for evaluating pregnancy status as needed.

Now we are going to test your knowledge a little bit by asking you a couple of questions, so I hope it is fun. If you want to, you can put your responses to the questions in the chat box, but I will make sure to indicate which one is the correct response.

The first one is: Who designates if human subjects are involved in the proposed research? Is it (a) the institution’s signing official, (b) the NIH program officer, (c) the applicant, or (d) the study statistician? I’ll give you a second to write your responses in the chat box.

The answer is (c) the applicant. All right.

The next question is: Which of the following is not a section that should be included in the Protection of Human Subjects Attachment? (a) the statistical analysis plan, (b) the adequacy of protections against risk, (c) the potential benefits, (d) the importance of the study team taking holidays. I don’t know if this one was obvious or not. The answer is the importance of the study team taking holidays.

Which of the following is a level of monitoring used in NIH funded grants: (a) a DSMB, (b) principal investigator and IRB, (c) independent safety monitor, or (d) all of the above? The answer is (d) all of the above.

Thank you so much. I am going to hand it over to my colleague, Ashley Kennedy, who is going to take you through the operational aspects to consider when writing your grant application.

ASHLEY KENNEDY: Thanks, Sabiha. Now we are going to segue into the next part of the presentation which is the operational aspects to think about when writing your grant application.

As an overview, we are going to briefly discuss the inclusion policies for research involving human subjects and then launch into the operational component to think about when you’re writing your grant application. These include the study population and procedures, the data confidentiality and quality assurance, regulatory considerations and reportable events.

NIH has two inclusion policies. The first is the policy on the inclusion of women and members of racial and ethnic groups, and this requires that women and members of racial and ethnic groups must be included in NIH-funded clinical research studies unless there is a compelling rationale for their exclusion. The rationale for their exclusion is usually based on the science. For example, if you’re doing a prostate cancer research study and you are not including individuals whose sex at birth was female, that is probably fine because the condition does not usually occur in that group.

However, if maybe you are doing a study and you want to exclude women because it’s too expensive to do pregnancy tests on everyone, that would not be acceptable. Cost is not an acceptable reason to exclude groups.

Our next inclusion policy is inclusion across the lifespan policy, and this policy requires that individuals of all ages must be included in NIH human subjects research unless there are scientific or ethical reasons not to do so. This was an expansion of our inclusion of children policy and the requirement has now been expanded to older adults and to the entire population.

If you are going to exclude people based on age you need to have a reason based on the science or an ethical or safety concern for them not to be included in your research study. Another example is if you are studying pediatric cancer and you don’t include older adults in the study, that is probably okay. The condition does not usually occur in that group.

If you’re developing an application, you are going to need to include plans for inclusion of women and racial and ethnic minorities and plans for individuals across the lifespan. In your inclusion of women and minorities plan you need to include a description of the plan distribution by sex or gender, race and ethnicity, and you also need to provide a rationale for this election. Questions to think about when you’re writing the rationale are why does the population look the way it does? How did you decide what the demographics of your sample should look like?

You are also going to need to justify your exclusions in this section. For example, if you are not including women in your study, you would explain why you are not including women. You should also describe any proposed outreach programs for recruitment. If you’re working with community advisory boards or if you have certain methods of recruitment that you’re using, such as outreach at clinics, you should provide those in your application.

You are also going to include a separate inclusion of participants across the lifespan plan, and in that plan you need to provide similarly a rationale for the age distribution, so describe the age of your participants. In particular, why did you choose participants of that age group? You are also going to need to explain how it contributes to the analysis and again justify any exclusions based on age. So, if your age limits are, say, 16 to 25, you are going to want to say why did you choose those specific age limits, and why is it important to the science that you have participants of those specific age groups.

You are going to also want to include a description of the study team equity and the appropriateness of the facilities for those included age groups.

So the purpose of the policy is not simply to include people and try to put them into a study that is not designed for them, but you also want to make sure that you have the right study team expertise to deal with those populations and make sure that you have the appropriate facilities.

Also in your application you need to provide an inclusion enrollment report. You will include both the plan and the actual data tables for your planned enrollment.

We have touched on the inclusion policy and now we are going to go over the operational components to consider when writing your grant application. Writing your grant application involves spending a considerable amount of time describing how your scientific idea is valid, original, achievable and valuable. In writing your application you should also be thinking about how to turn your ideas into procedures that will ultimately allow you to recruit and enroll your population that you need, ask the participants to complete the interventions and assessments that you are proposing, and allow you to measure the study outcomes.

When identifying your study population, you really want to consider who you’re collecting data from. Is it going to be patients, clinicians, family members? Also, what kind of data will you be collecting from these people? This is going to help you in establishing who may be a research participant in your study.

Let’s go through some of the examples. Parents providing information about their child, such as their mental history. In that case, the parent is likely not going to be considered a research participant.

The next example is a parent providing information about their own mental health such as their own diagnosis in the past. In that case, the parent is likely going to be considered a research participant.

The last example is a provider being trained with a new treatment technique and their level of proficiency and acceptability of the technique is being assessed. In that case, the provider will likely be considered a research participant.

You should also be thinking about the population in your study. Some of the questions you might want to think about are: where are you going to recruit the study participants? What sites are participating in the study? Are the participants patients who go to the hospital or a clinic? Is the sample from the community, school or neighborhood? Will your recruitment goals be affected by other researchers that are recruiting the same population as your study?

Also, how are you going to recruit your participants? For example, can you approach them in the clinic or hospital waiting room? Are you relying primarily on provider referrals for your enrollment? Will you be utilizing various ad placements such as social media, flyers in the community or mail-outs?

When thinking about the operations you also need to spend some time thinking about the inclusion and exclusion criteria for each group of participants, as that criteria may have an impact on recruitment and enrollment and meeting your planned enrollment goals. For example, how likely will you be to find participants who fulfill all of your criteria? Are they too narrow or too broad to where you are going to have difficulty enrolling participants? How specifically will you measure each one of those criteria?

Let’s go through these examples. For major depressive disorder, are you going to be relying on and assessing via SCID, or are you going to rely on medical records for recent diagnosis?

An IQ greater than 80 -- Will you be able to perform an IQ test or are you going to rely on using proxy measures? Which of the two is more feasible for you?

The last one is absence of neurological conditions. How and who is going to be able to evaluate this? Will it be based on parent report or are you going to use a medical screen by a clinician?

Now I am going to hand it over to Anna to go over the other operational components to consider when writing your grant application.

ANNA ORDONEZ: Thank you, Ashley. These are just examples of the different kinds of things that you can think about in terms of the many different ways to measure these inclusion and exclusion criteria.

Similarly, for your study procedures, we want you to be thinking about what are the different procedures for each group. It may be that your study is delivering an intervention to participants who you are going to assess mental health status or outcomes. You may also be including providers who you are going to train in this and who you want to provide you information and feedback about this intervention. So it may be that you have both a provider group of study participants and a patient group. As you’re thinking through all of this you want to think about what the different procedures are for those different groups.

Here are some of the things we’re trying to walk you through to help you think about how to make your studies feasible. It’s one thing to have it all written on paper. It’s a very different thing to start and finish the study and actually be able to recruit everybody you wanted to and have them participate in all of the different activities in your study. So you want to be thinking about how long you expect people to participate. Is this a study that’s longitudinal? Would they be coming back several times?

Think about what types of procedures and what procedures are needed to be completed. Might those procedures lead to incidental findings, for example, which Sabiha mentioned at the beginning? And if there is a potential for incidental findings what is your plan to deal with those?

How many procedures you’re conducting. Sometimes we know that you start with one assessment and then you want to add another one and suddenly you have four hours’ worth of assessments, so really thinking about do you need all of those. Would that impact people’s interest and ability to complete all the study procedures?

The same thing in terms of whether the procedures are invasive. You might want to think of whether participants will really want to participate and what might help them find it something that is valuable.

Who completes the procedures with the participant? Are the study personnel individuals going to be blinded to what type of arm the participant might be in? Are they getting your active new treatment, are they getting the control? If they are blinded you need some sort of plan or system to break the blind if something happens to those study staff. Do they need a particular type of training? Do they need to have some specific proficiency in some health background? And if they do need training, how often do they need to be retrained, for example, or to check in around their training?

You also want to think about where the procedures are going to be performed. Are these things that you do in questionnaires over the phone, are you having people come to a specific location, to a site for your study? That might matter for lots of reasons. One is if you’re doing a phone assessment, is that space where the participant is going to answer the phone private enough? If you’re having people come to your site, is it difficult to get there? Is that going to make it so that people don’t really want to participate in your study even if it sounded really interesting?

And same if you have a longitudinal study or a specific timeframe in which you need to see people. For example, you may be screening them and then you need them to come in for their first appointment within a certain amount of time, so really thinking about what that window needs to be is very helpful. Sometimes the window becomes too narrow or isn’t really realistic or feasible, and you end up with all these issues in following your protocol because the window itself was a little too stringent or not realistic.

So really thinking through, if people are going to have a three-month follow-up can they come one week before three months, can they come two weeks before three months? What is a reasonable amount of time that fits within your study and your research but also within the reality of people’s lives participating in research studies?

If your study has pharmacological procedures, that is also something you need to think through. For example, some studies ask individuals to discontinue medications they’re taking before they start a study. We call that a medication washout. And if that is something you’re doing in your study you really want to think about who is going to be directing that and how they are going to monitor the safety.

For example, if you’re taking someone off a medication their symptoms might get worse before they are even able to start your study procedures, and how would you address those kinds of situations and who is allowed to do that in the countries in which these are occurring? Does it have to be a physician, do they have the appropriate licensing and training they need to be able to do that?

The same for an intervention where you may be giving someone a medication and slowly increasing the dose. How are you going to do that dose increase, that titration? Who is going to direct it and who is going to monitor and make sure that if people are having side effects or any concerns around that those are being addressed in the study>

As I mentioned, it’s very important to know who can write prescriptions for study medications, and those rules may vary depending on the country in addition to some of the rules and regulations, for example, here in the States.

If the study has medication that is being prepared specifically for the study, that’s being compounded for that study, you want to be really clear on where that compounding is occurring and whether that site, if it’s a pharmacy, if it’s a lab, is following all of the regulations in that local country, region, state in order to compound a medication. Sometimes there are very stringent rules around what’s coming in, what is going out, where all these things are being tracked and how they are being tracked. So you want to make sure that you are aware of all of the rules and regulations in the place where your study is taking place, including your institution, to know that you’re following what needs to occur.

If your study has a medication that, for example, is only available for the study, you really want to think about what is going to happen to the participants when the study ends. Are they no longer going to have access to that medication? How are you going to transition their care? Who can continue their treatment once the study ends?

Similarly, you want to be thinking about whether there are stopping rules for your study. What that means is there may be certain things that if they occur would signal to you that the study should stop even if you have not completed recruiting all the participants in your study. What kind of situations might those be?

As you’re thinking about your study procedures you want to, again, think about the risks that are associated with the procedures for each group. For some, the main risk may be loss of confidentiality. You’re asking the provider their opinion on how this intervention went. For others it might be something more. They might be experiencing adverse effects, they might have negative outcomes from the intervention that you’re leading, and you want to think about what those risks might be.

Similarly, you want to think about procedures you can put in place to help address those risks. If you have a lot of assessments you might be thinking about, well, people could become distressed with certain questions so you might have a plan for how you would address that. You might want to structure having sufficient breaks so people can take a break in between different assessments.

If people are endorsing really severe clinical symptoms you might want to have a clear plan in place for how you would refer people to get clinical care, especially if they have something like suicidal ideation or if there are other types of situations that you are obligated to report, for example, elder abuse or child abuse.

We know that there are different rules for all of these in different countries, so you want to be sure of what the rules are in your institution, in your state, in the country in terms of really being in compliance with the rules and regulations and also, most importantly, thinking about the safety of the participants in your study.

Confidentiality and quality assurance. We talked a lot about confidentiality and really thinking about how you are going to ensure that the information you’re collecting from your participants remains private. First, you want to think about where you’re collecting the information and how you’re recording it. Are you recording it on paper, are you making video recordings, are you putting it into some sort of electronic data capture system? And how are those different forms, different systems, protected?

If you’re collecting paper forms in the field, how are you making sure that those paper forms are kept safe until you get yourself to your research facility, and then where are they going to be kept and how are they going to be kept so other people aren’t able to access them? Are you storing them in a locked box, are you storing them in the trunk of your care which can easily be broken into and you might lose the materials?

The same thing for data quality. You want to think about how you are going to check for errors. If you are entering data on paper first and then putting it into an electronic system, how are you making sure that transferring the data from the paper to the system doesn’t include errors? There are lots of different ideas to help you make sure that the data is entered correctly. Different studies have different types of data quality procedures. Some may be entering the data twice if it’s going from paper into a system.

Others may be having algorithms within the system to highlight to you a value that is totally out of range. For example, my blood sugar could be 2,500. Well, that’s a number that my blood sugar could never be and so the system may have parameters where it says there might be an error here; go back and check.

We also want you to think about whether your systems have an audit trail. For example, a lot of people may be familiar with Excel. Excel doesn’t have an audit trail. You can enter data into a little box, you can delete it and the data is gone forever, and you don’t have any way of knowing who entered the data, when they entered it and if it was altered what the changes were.

Whereas, there are other systems that are now being used very widely that allow you to make sure that the data is entered and you know who entered it, when they entered it and if there are changes that you are able to track what the original entry was and what the changes were, and you have an understanding of why that data was changed.

Who has access to your data? You want to be really sure that if you are dealing with protected health information only the people who should be privy to that information have access to it and nobody else does.

Another thing to think about is how long are you expected to store this research data? NIH has rules for retaining research data for at least three years from the time of your final report, but we know that some institutions have even longer terms that they expect people to keep their data, so this is something else for you to check both for your institution, for the country your study is taking place in or any other kind of regulations you need to follow. For example, if your study is FDA-regulated they have other stipulations for how long you need to store your research data after your study is completed.

Regulatory considerations. Sabiha at the beginning talked a little bit about some of the resources. I know we put them in the chat in terms of resources we have here in the US to help think about all of these regulatory considerations. But this is something that we know varies from country to country, and we have seen some global studies where there are several different regulatory bodies that you need to be interacting with at any given moment in time.

Most places have either an institutional review board or an ethics board, something in that realm. Oftentimes we see ministries of health involved. Institutions and universities have other regulations and policies that you need to follow. And you may have a data safety monitoring board or some other oversight body in your study. You want to make sure that you’re tracking all of your regulatory documents so that you are always in approval of all the different regulatory bodies and you know when you need to submit updates in your documents.

We really recommend that you keep track of the versions and the dates of your documents so that you are using the most up-to-date and approved consent form, for example, and that all the forms you’re using are the ones you are supposed to be using in your study.

Again, having a really good system to track approvals, especially if you have a lot of different regulatory bodies you’re dealing with, is really important because otherwise your study may have to stop if one of your approvals lapses.

And then it’s really important to make sure that your study staff is well trained, that they understand the importance of these different oversight bodies and that they are always using the most up-to-date and approved documents in the study. The consent form is a perfect example of that but you may have other documents that you need to make sure you’re using the most recent and approved version.

Consent and assent into studies is really, really important, and we think you should double-check with your own institution and your ethics or institutional review board to make sure that you are aware of all of their policies in addition to the policies of the Office of Human Research Protection in the NIH.

Things to think about. Where you are going to be consenting participants? If you’re going to be consenting them in the field, you want to make sure that you have a space that’s private where people have an opportunity to ask questions and to discuss things with you. You want to think about whether it’s going to be in an office, over the phone, and that your regulatory body, your IRB or ethics committee is aware of how you are actually intending to consent the participants. Is it going to be an oral consent or written consent? And there may be some rules around what kinds of things can be done with an oral consent versus which ones we need to have a written consent.

You want to think about who is qualified to consent participants and make sure that they really understand what the consent process is about so that they are conveying all of the information to the participants so they really are able to consent voluntarily to participate in your study. You want to think about whether these individuals need training. There’s a variety of different types of trainings to help make sure everybody understands how important consent and assent are.

Some studies we know in some countries and some institutions require there to be a witness to the consent, so again, you want to be familiar with what the rules are in your institution and in your country in terms of what needs to be included in the consent form.

In some studies, you need to evaluate capacity to consent, and if that is the case in your study you want to have a plan for how you are going to do that and how you’re going to document that you have done that.

Similarly, assent for minors might vary depending on the age for when you consent versus assent. And there may be rules related to specific health topics in which, even if a minor can’t consent to some things, there is information that you are not allowed to share with their parents or guardians, so being really familiar with what the rules are related to consent and assent and what information can and cannot be shared with parents or guardians is really important for you to be clear about.

We talked a little bit about reportable events, and we had a link in there for the NIMH policy around reportable events. Again, you really want to think about what an adverse event did, what a serious adverse event did, an unanticipated problem that potentially may pose risk to participant. All of these definitions for the US, for the Office of Human Research Protection are posted in these links. I think it’s a similar link to the one we sent earlier.

The really important thing is you want to think about how you are going to track these things. Are you going to be checking in with people periodically and systematically asking them questions around adverse events or serious adverse events? How often are you going to be doing that? And what happens in between those times in which you’re talking to people? Are you going to give them a phone number and say please call me if you have some problem? What is your process going to be to keep track of any adverse events that occur in your study?

You also want to have some thoughts about what is an expected adverse event in your study and how you are going to determine whether that adverse event is related to your study. For example, someone may walk into the clinic, slip on a puddle and fall and break their leg, and that is an adverse even in your study. They were coming to your clinic to see you but it may not be related to the intervention in your study, so you really need to be thinking about how you are going to classify all of these different adverse events and who is in charge of making those determinations.

And then, very importantly, you want to think about when and how and to whom you need to report those events. I mentioned the NIMH has a policy that walks you through certain types of events like serious adverse events that are related to your study may be things you need to report in a very short period of time, whereas adverse events may be something that you report periodically. Different regulatory bodies, different IRBs or ethics committees have different rules about when they expect you to submit these reports, so it’s important to know what the expectations are for the different organizations you’re working with.

Similarly, you want to think about what a protocol violation is or what a protocol deviation is, and some institutions and some IRBs spell this out very clearly. You want to keep track of those as well. You want to know when are you not following your protocol and what happened. Of course, if this is something that needs a correction or needs you to address something or re-train your study team, these are important things for you to track and make sure you are addressing appropriately.

As I have said over and over again, it’s just really important to know what your institutional policies are. If you are getting an NIH grant, NIH has expectations about how you’re going to conduct your study, and that is layered onto whatever your institution’s expectations are and the country in which your study is taking place. So being sure that you know how often you need to report the different adverse events and protocol violations and to whom and within what timeframe is very important.

We thought we would do some questions again just to wake everybody up. Here is a study operations question. There’s a psychologist who’s being trained in a new behavioral therapy. The level of proficiency of the new behavioral therapy and the acceptability of the therapy is being evaluated. The provider is also piloting the new therapy in adolescents ages 12 to 17 and the parents of the adolescents as well. The research assistant will be collecting data from the adolescents and the parents. So who can be considered a research participant in this study, (a) the adolescents, (b) the parent and the adolescents, (c) the parent and the adolescents and psychologist, and (d) the parent, adolescent, the psychologist and the research assistant?

I’m seeing some variability here. The answer is the parents and the adolescents and the psychologist. Again, this is why we are covering these questions because this comes up a lot for people. If your intervention is not only delivering care, let’s say, to this adolescent and their parent but is also training the provider, asking the provider what they think of the training, whether it has been acceptable, then you might want to think about what kind of data you’re collecting from that provider and whether they, too, should be considered a research participant.

Here's a second question. In the same study one of the exclusion criteria for the adolescents is an IQ below 70. Which one would not typically be used to operationalize -- that means to turn it into an action you can track -- this exclusion criteria? Is it (a) to perform a full neuropsychological testing, (b) to obtain previous documented testing results, (c) to use school report cards or grades as a proxy, or (d) having the child chew gum?

This one has a little less variability in the responses. Not just as a teaser but to think, right, there are many different ways to operationalize your different inclusion and exclusion criteria. It’s really about giving some thought to how you are going to take that criteria and turn it into reality for your study. So, having the child chew gum is the answer.

Here’s Question 3. Which of these procedures might make it difficult to recruit and retain study participants: (a) obtaining a one-time cheek swab, (b) obtaining depression ratings every three months for two years, (c) obtaining study questionnaires during regular clinic visits, or (d) offering transportation to and from research sites and flexible scheduling time? Which might make it difficult to recruit and retain study participants?

Mostly people are answering (b), but this is one to think about. Longitudinal studies where you’re asking people to stay engaged with your study and keep coming in and answering questionnaires or providing different study samples over two years might be something a little more difficult to make sure you are able to retain all those participants, and you need to be really mindful of how you are going to try to ensure success in keeping participants enrolled in your study.

Here is Queston 4. Which of the following is an adverse event: (a) a participant slips and breaks his leg during a study visit, (b) a participant has an allergic reaction to the study medication, (c) a participant reports suicidal ideation during a study visit, or (d) all of the above? I’m seeing a lot of consensus on this one, and yes, it is all of the above.

We are wrapping up our time today for the webinar and we are going to now start the Q&A, but we want you to know that the slides will eventually be posted on the website. The slides really have a lot of resources, and we want you to have a way of seeing those resources together. They are slightly organized by topic, and you can access them in the future as well, because we know there was a lot of information in the talk today and it’s hard to keep track of everything.

Leo, I think we are going to open it up for Q&A.

LEONARDO CUBILLOS: Yes, we are. Thank you, Anna, Ashley and Sabiha, for this wealth of knowledge. I was taking notes and there is a lot to learn. Thank you for making sure that the slides had those links. I find them very useful, not only today but for future reference as well.

We have a few written questions. Some of you have received written answers already. I want to read out loud the thematic or content questions.

Marian Brooks -- I’m sorry if I’m mispronouncing your name. Marian asks Sabiha a question. Sabiha started the presentation talking about a clinical trials form which asks for a lot of information. She asked for inclusion criteria, study population, ages, et cetera. Can we just say, quote, “see clinical trials form”, or just focus on the information that isn’t in that form when we do the human subjects attachment?

SABIHA ETHRIDGE: Thank you for the question. I think you are asking whether or not you can reference other sections of your grant application when you’re filling out the Protection of Human Subjects Attachment that’s part of this clinical trials information form.

Because the Protection of Human Subjects Attachment is peer-reviewed I would recommend that you include all relevant information. There are some points that you are required to put in there, and so I would recommend that you do include the information in that section. When you are filling out the Data Safety Monitoring Plan, if it’s relevant for your study or you decide to have one, I think that section is where you can reference other parts of your grant application so that you are not redundant.

But I think because that Protection of Human Subjects section is closely reviewed during peer review, it is probably in your best interest to include the information there as well.

LEONARDO CUBILLOS: We have a question for you as well, Sabiha, from Phoebe Gilbert. Phoebe asks, who and what is responsible for reviewing any submissions? What verification processes are used by these reviewers, and what documentation is being processed?

SABIHA ETHRIDGE: I hope I understand the question. I think this is in regard to submitting your grant application, and your grant application typically goes through a peer review process. There is a group of your peers who review it and they have certain guidelines they use to review it, and they have a discussion about it during a meeting and they come up with any concerns or questions they have, or anything that they think you are doing really well they comment on that.

My colleagues can add to that, if I missed anything.

ANNA ORDONEZ: I think one of the webinars in the webinar series is about the NIMH review process, and I think that talk is already posted on the website. That might be a good place to get some additional information.

The NIH does have very clear steps for how they receive the application, how they refer the applications to the different institutes once they receive them, and then the process that it goes through in terms of how the review is managed, whether it’s managed by the NIH central review process or the different institutional review processes, et cetera.

LEONARDO CUBILLOS: Thank you, Sabiha and Anna. That is correct, Anna. On April 14th of last year the first webinar was entitled “Submission and Peer Review of NIH Grant Applications.” The speaker was Nick Gaiano, who is the Chief of the Extramural Review Branch here at NIMH, and he provided a comprehensive overview of how revise works.

Just one point to complement Anna’s and Sabiha’s answers. We of course do have these review sections and review panels that are external peer reviewers who do that first review. They provide a document that is called a summary statement that all applicants receive. A subgroup of those applications that were reviewed that could be considered for funding go to what is called a second tier review that is done internally by NIMH colleagues including folks from OCR and primarily colleagues from program at NIMH.

Sabiha, Padma asks if warning signs are involved in a multi-site study that originated in the US, what is the rule for single IRB?

SABIHA ETHRIDGE: I think this question is asking, if there is a multi-site non-exempt human subjects study that has sites both in foreign countries as well as in the United States, what do you do in terms of single IRB policy.

According to the policy, foreign sites participating in these NIH-funded multi-site studies do not need to -- they are not expected to follow this policy. So the foreign site is not expected to. But, for instance, if you have multiple US sites in the same grant, those multiple US sites are required to follow the single IRB policy.

LEONARDO CUBILLOS: Thank you. Padma also asked a question for Ashley. In case of providers doing pre and post-knowledge assessment after completing a training developed as part of research, are they considered participants?

ASHLEY KENNEDY: Yes, if you are collecting data on the providers for the pre and post, they would be considered participants.

ANNA ORDONEZ: One thing to think about, Padma, is those participants would have a consent that addresses their risk of participation in the study, so it might not look like the consent form for the participants who got the intervention itself. But this is a group of research participants, and you need to think about how you are going to consent them into the study, what information you’re going to be collecting from them, what kind of risks there are for those participants, for those providers, what risks exist in providing that information to you and how you are going to address those risks.

LEONARDO CUBILLOS: This reminds me of a conversation we had recently with some applicants, and Ashley and Anna and Sabiha, please correct the following statement if need be.

We were discussing that subjects for clinical trials are not only patients or community members themselves; they can also be health system participants. These are providers at a different level and they can be the subjects of research but also subjects of intervention. I think you also alluded to that in your slides.

ANNA ORDONEZ: Yes, that is totally true. For example, we see a lot of implementation studies where you may be conducting focus groups, you may be collecting these types of assessments from a variety of different individuals. And again, if you’re collecting research data that you are going to use to answer research questions, those individuals are participants in your research study even if they are not the patient or the target population for your intervention.

LEONARDO CUBILLOS: We are interested in improving mental health outcomes at the individual and community level, and that entails doing research not only at that level but also upstream in the health system and maybe sometimes outside of the health system.

ANNA ORDONEZ: And I will just say, Leo, it is not always so straightforward, so that is why we’re encouraging you to think about what information you’re collecting from whom, and that might help you think about is there a chance that this person would be considered a research participant. And if you are not sure, your institution review board or your institution’s human research protection program probably has resources and individuals that can help you think about would this person be considered a participant and do I need to prepare a specific consent form for them.

LEONARDO CUBILLOS: This question came in, Anna, while you were presenting. Depending on the type of clinical study, the investigators can ask the IRB to waive the study to conduct in minors or vulnerable groups. Do you have such kind of experience at NIH?

ANNA ORDONEZ: Yes. Sahiba is probably more of an expert in this than I am, but in the US, the Office of Human Research Protections is a regulatory body that regulates the IRBs and they define a lot of these regulations around how you consent, when you can waive consent. There might be instances where IRBs even consult with them to see if that is an instance in which consent can be waived, so certainly I think the NIH has experience with that.

We also know that the rules and regulations around this can vary in different countries and so we have provided links and resources to things we have here in the States. But there may be specific rules and regulations around when consent can be waived in the country that you are conducting your study in, and if that is really different than the rules and regulations here, you want to be prepared to answer additional questions of how that will work. Because people just want to make sure that you are only enrolling participants in your study who appropriately consented. So it is a very important topic to all of us.

Sabiha, I don’t know if you have anything else to add to that.

SABIHA ETHRIDGE: No. I totally agree. I was going to say that here in the US we do have federal regulations that allow IRBs to waive informed consent, but there are very specific criteria that the IRB uses to determine whether or not that is appropriate for the study or for the population.

To answer the question, yes, we do have that here in the US. But as Anna said, it is important to know what is being done in-country as well if you’re doing a study in a foreign site.

LEONARDO CUBILLOS: Marian Brooks asked another question. Thank you, Marian for these questions. Is the Protection of Human Subjects Form supposed to be identical to an IRB protocol?

SABIHA ETHRIDGE: No, it is not. It will include a lot of the same information but it is not meant to be identical. Also, IRB applications differ between IRBs. NIH has very specific guidelines for what your Protection of Human Subjects Attachment should include. I will put a link in the chat that links to that document that I was talking about earlier that outlines every section of the grant and the corresponding guidelines for what you should include.

To answer your question, no, it is not identical. It may have the same information.

LEONARDO CUBILLOS: Thank you. I think that was an excellent question that helps us understand that sometimes we use the same information in different ways and in different parts of this project, but the way we present it and maybe sometimes even the content of it may not be the same. I think it was a terrific question.

Mendoza asks the following question. Can you expand a little on participants’ monetary compensation guidance during enrollment?

SABIHA ETHRIDGE: The only thing I can speak to in terms of monetary compensation is that you need to make sure, one, that the amount isn’t coercive within the population that will be involved in the study; also, that it is allowable. The reason I brought it up in our presentation was that oftentimes we see it listed as a potential benefit of the study and that is not how it should be presented. It should not be presented as a benefit because then it becomes potentially -- gives the appearance of being coercive.

ANNA ORDONEZ: Just to add, I think with all these ethics questions there is a lot of grey sometimes, and there are very specific study populations in which there is debate about what is an appropriate compensation. I can think of an example of a conference a few years ago where there was a lot of conversation around monetary compensation in a population that is addressing substance use and how that might be a good thing and how it might be a bad thing, what are the risks, would it contribute to more substance use.

So I think this is a topic that there is not a clear line that you must always do it this way, or you can’t do it that way. It’s really that you want to be very thoughtful about how that compensation is being decided, what is appropriate. We know that different institutions and different IRBs have guidelines and have very strict rules, and we really want to make sure that what you are doing is compensating for people’s time, for transportation, for things that are appropriate and not leading to something that might be concerning from an ethical perspective.

I think it is just something to give a lot of consideration. And also, like we said before, oftentimes institutional review boards or human research protection programs have people who can help guide you in terms of how to think about this and what is considered appropriate and not coercive.

LEONARDO CUBILLOS: I will say for these topics that are grey in nature, potential applicants can come to program and ask us questions and we can also get some preliminary feedback from OCR before the application is submitted. Is that something that is allowed?

ANNA ORDONEZ: OCR is available to consult for some of these more difficult questions. I would say that my first recommendation if you have questions about ethical issues is that you start with your ethics review body at your institution because they will be the most knowledgeable around the rules and regulations in the country where you’re conducting the study and in the populations that you may be working with.

We are certainly available to provide additional consultation on that, but that would be the place where I would start because it’s really going to give you a lot -- you know, many of these ethics rules, for many of them context really matters in terms of understanding what you’re proposing and what might be different factors to consider.

LEONARDO CUBILLOS: This question asks how do you handle confidentiality issues with pregnant adolescents/adolescent mothers as participants? Should their parents be invited for focus group discussions or interviews?

ANNA ORDONEZ: I think that is a great question and it’s another one where the locality matters. There are different rules. In the US there are state-based rules in addition to federal regulations in terms of how this gets managed. And I’m sure in different countries there are different rules as well.

So, depending on where your study is taking place you have to be very thoughtful about what are the rules and regulations and what do you feel -- how are you going to inform those adolescents about how you would manage certain types of information. You want to be very clear upfront if there is information that you can’t keep private from a teen. You want to be very upfront with them, like this is the type of information that I would have to talk to -- whoever you need to talk to -- about. So instances where you have to break confidentiality.

Here in the US and in many states there are very clear rules around how you manage information related to pregnancy specifically. There are also rules around mental health and a number of other health topics, so it can be very health topic-specific. And you really need to be familiar with what those rules are and have a plan in place to provide the information upfront and then address it if an issue comes up. Sahiba, do you have anything else to add?

SABIHA ETHRIDGE: No, I think you covered it.

ANNA ORDONEZ: I think it is an excellent question. It’s the kind of thing you want to be thinking about for your study way ahead of time as you’re putting together how you are going to conduct it.

LEONARDO CUBILLOS: Great. We have addressed all the written questions that we have received. Ashley, Sabiha, Anna, is there anything else? Do you want to make any additional points the audience should have in mind? Or I should have in mind?

SABIHA ETHRIDGE: I just want to note I put some of the links in the chat box. Those are very helpful links when you’re filling out your grant application. I highly recommend that you take a look at them. The first one is the general instructions for the entire grant application, and the second link is for the Human Subjects in Clinical Trials Information form, and they have a lot of detail in there and there are some additional resources embedded into that. I also recommend you visit the NIMH OCR website, the Office of Clinical Research website, as well as NIH’s Human Research Projection website, and I can try to include those in there as well.

LEONARDO CUBILLOS: Great. Thank you, everyone, for attending this webinar, wherever you are, whether it’s morning, noon or evening. Thank you for taking the time to be with us. A special thanks to Sabiha, Anna and Ashley for sharing all this knowledge, for educating us and for answering the questions. Also thank you to all the staff that have been working back stage and ensuring that this webinar was possible.

Our next webinar is on September 20th and we will be discussing RDoCs in global mental health. If you do not know what RDoC is or you are not entirely familiar with it, that will be a great opportunity to further our knowledge in those domains as well.

Have a great rest of the day. Thank you.