Using Simulation to Evaluate Social Determinants of Health in People with Mental Illness: Potential Use of Findings in Discussions with Policymakers, Community Groups, Consumers and Advocates

Transcript

WEBINAR OPERATOR: Please stand by, your program is about to begin.

Hello, and thank you for joining the National Institute for Mental Health Office for Research on Disparities and Global Mental Health 2018 webinar series. This presentation is entitled “Using Simulation to Evaluate Social Determinants of Health in People with Mental Issues: Potential Use of Findings in Discussions with Policymakers, Community Groups, Consumers and Advocates.” Please note all lines are in a listen only mode. If you’d like to ask a question during today's presentation, you may do so at any time through the Q&A pod located in the lower right-hand corner of your screen. This call is being recorded, and it is now my pleasure to turn the call over to Chief of Mental Health at NIMH, Ms. Andrea Marques.

ANDREA HORVATH MARQUES: Thank you so very much and welcome everybody. I would like to welcome you to our NIMH Global Mental Health and Mental Health

Disparities Webinar Series. This is the last one for this year, but we are very happy to welcome our speakers today. I would like to, just before I present them, I would like to remind you that today’s presentation is about a very important paper that was published last year in the Health Affairs Journal, and we are going to put the link for you all to have that paper also.

Today, as we have already said, the name of the talk is going to be “Using Simulation to Evaluate Social Determinants of Health in People with Mental Illness: Potential Use of Findings in Discussions with Policymakers, Community Groups, Consumers, and Advocates.” We are going to emphasize for ethics about this webinar today and that we are going to be highlighting some important opportunities to address mental health disparities, and to improve the efficacy of treatment of poor people with mental illness by tackling the social determinants of health.

And we also, the speakers are going to be talking about how to use the modern research techniques, like simulation techniques, and how that kind of techniques can help us to provide some education of what might work to address mental health disparities among people with mental illness. We also ‑‑ our speakers are also going to be sharing with us and illustrate the reactions of policy makers, community groups, consumers, and advocates on the research results from this study. And then finally they are going to be talking-- and how we can use the results from the research to inform policy.

{Inaudible} I’m going to just … already to our speakers and the…

Umm, I’m going to just draw up already-- throw some questions to our speakers that they’re going to be emphasizing and so why we are talking about social determinants of health to mental health disparities, and why we are using those computation approaches to address them. And I’m sure that our speakers are going to be able to talk about them. And so, before I introduce them, I just want to remind you that this is just some logistics aspects of this webinar. This webinar is being recorded and will be archived at our website in a while, in a few weeks.

Our speakers are going to have around one hour to talk, and then I would ask the audience that you please write your questions in the Q&A and for the interest of time I would kindly ask you to write your questions as you think about them, and by the end, when the speakers have already finished, I’m going to be presenting the questions for them and they are going to be answering as possible. In case we don’t have time to address all of them, I will be able to share the questions to them and they are going to be happy to send you an email.

And now, I will introduce our speakers: Dr. Margarita Alegria. She’s the Chief of the Service Research Unit at the Massachusetts General Hospital and Professor in the Department of Medicine and Psychiatry at Harvard Medical School. Her work has focused on an improvement of health care services for racial and ethnic populations and she has done extensive work on the area of mental health disparities.

And then we're going to have Dr. Justin Metcalfe, who is a policy researcher at Westat and his research focuses on behavioral health and employment.

And then after, Dr. Justin, Dr. Amanda Moyer is talking to us, she's a postdoctoral fellow in the research unit at the Massachusetts General Hospital and the Department of Health Care Policy at Harvard Medical School. And her work has been focused on juvenile and criminal justice policies and practices; and emphasizing on how to address racial and ethnic disparities in mental health and juvenile justice.

And finally, Dr. Robert Drake who is a professor of Health Policy and Clinical Practice at the Geisel School of Medicine at--- and the Vice President of the Westat Corporation. He has extensive experience and research focusing on psychiatric rehabilitation, including integrated treatment for people with dual disorders evidence-based mental practice and implementation of vocational service.

And so, we are thrilled to have them all here, and we will pass over to you, Dr. Margarita Alegria who is going to start. Thank you.

MARGARITA ALEGRIA: Thank you so much, Andrea. We are very excited to be here. We want to start by saying that this is a great opportunity given that this is a time when people with mental disorders are being asked to do more and more things, and so we wanted to start-- let me start by introducing the group that is going to be talking today just briefly to say that it includes Robert Drake, Justin Metcalfe, ourselves in the disparities research unit here at Massachusetts General Hospital. And then I also want to include two people that are not today in the call but were a part of this group, Hyeon-Ah Kang from the Department of Statistics at Columbia University and Jingchen Liu from the Department of Statistics at Columbia. Hyeon-Ah Kang is a Post-doctoral Research Scientist and Jingchen Liu is an Associate Professor.

Umm, let me start with a brief presentation. I want to tell you an overview of what you would be hearing today. First, we're going to start talking about what are social determinants of health, like Andrea mentioned, why are they so important to include in terms of trying to look at solutions for improving the mental health of our communities. We want to talk a little bit about mental health, life expectancy, how the determinants play a role, and disparities. And then finally, we want to talk about our simulation study. And here, we want to really spend some time talking about what is a simulation study. We want to tell you why we selected a simulation study to really answer the research questions that we had, what was the background about it, and we would also clarify who participated in this study, why we include two samples in this study, what we did to simulate, and then what did we find. And more important is we would tell you what do we think our findings mean? And so, I think this is a great opportunity to show you one of the ways we could approach how we can test potentially different hypotheses of what might work without actually having to do the study itself in terms of doing the actual data collection and actually implementing different interventions and then finally finding out the results. This is a way to at least try to see which are the hypotheses that are worth investing in. Next.

Let me start by social determinants of health. During the last, I would say, 10 years, there's been an escalation of studies, you know, really a plethora of studies, on social determinants because people are finding that, you know, you really need to address social determinants of health if you really want to have a big impact in terms of the well‑being of people.

And what are social determinants? I mean, people typically discuss them as the conditions in which people are born, grow, work, live and age. But I wanted to say that social determinants are different than biological determinants of people, are more socially malleable, and are things that you can actually make a difference and modify. This has to do with employment, housing, social support that people receive, education, income, the opportunities for healthcare. So, this has factors that really have a big impact in your health and well-being, that you can actually intervene to change so people do better.

And there's quite a lot of work that has been done by Elizabeth Bradley and others showing how social determinants have a big impact in terms of health. Next.

Let me give you a few examples to show you why we selected to do this work. The first example I'm going to use is from education. And this actually shows that if you see here, having four years of education in addition to ‑‑ this is in addition to high school ‑‑ reduces a lot of health risks. For example, it lowers the probability of diabetes by 1.3 percent. It lowers the probability of heart disease by 2.2 percent. It lowers the probability of obesity by 5 percent, and smoking by 12 percent, according to the Robert Wood Johnson Foundation. So, if you think about it, this might sound like actually there are not big decreases in population health, but actually, they are, because this is over thousands and thousands of people. So, it means that if you improve education, you have the opportunity to increase the health of people by quite a bit. Next.

The next one I’m going to use is income, and there's been a lot of work now looking at life expectancy and how it changes depending on people's income. This is following the work of Marmot in the UK, but showing that people have higher life expectancy when they have, they are higher income earners.

So, for example, since 1977, there's been proof that people that are in the higher income earners actually gain six more years in life expectancy. This is for male workers that retire at age 65.

Imagine those people at the top of the income distribution have six more years in life expectancy. However, when you look at those in the lower income distribution, you find that the gains in life expectancy for male workers retiring at age 65 are ‑‑ is reduced to only 1.3 years. So, a lot less. So, income also seems to have a big role in terms of health.

The next one I'm going to use is employment. And employment is an important one because unemployment, there has been some very interesting data showing that when people are unemployed, it affects their health in many, many ways.

So, when we compare laid‑off workers, for example, laid‑off workers have a 54 percent more likely to have fair or poor health compared to those that are continuously employed. So, it means that you really have a much bigger opportunity of going into the fair or poor health category when you are a laid off worker, compared to those that are continuously employed. Similarly, when you look at laid off workers, they are more likely to develop stress‑related conditions, like cardiovascular disease, compared to the continuously employed. So again, a big impact of the possibility that employment plays a role people’s health. Next.

I want to follow by saying in the area of mental health, we have been really sort of struggling with trying to figure out, how come people without mental disorders have a likelihood of living 10 more years than people with mental disorders.

There’s a conference that actually the National Institute of Mental Health did, where they brought experts-- this was like 10 years ago-- trying to figure out how come we have a different, such a dramatic difference, in terms of life expectancy when people have mental disorders?

What could be the substantial causes of this, and obviously people are now and now more focusing on what about mental disorders puts you at higher risk of substantial deaths worldwide. Next.

I think tied to this is the area that has to do with mental health disparities, because we now know from a lot of epidemiologic data that both Latinos and African‑Americans tend to show lower prevalence of mental health disorders, both lifetime and 12 months.

But we do know or at least we have evidence that they seem to have, although they have lower prevalence, they’re more likely to have disorders that are persistent compared to non‑Latino whites.

So, this disparity, not so much in prevalence but in the persistence of disorders, seems to be something that we're trying to work around and trying to problem‑solve and see what are interventions that we could do to try to lower the persistence of disorders among ethnic racial minorities.

So, although again, they are less likely than whites to have mental health problems, it seems that the recovery process is more difficult for Latinos and African‑Americans as compared to whites. Next.

I think we did this study-- it was a request for a papers that was done by Health Affairs where they asked about people doing ‑‑ how could people use social determinants in terms of thinking about interventions for social determinants that might make a difference for mental disorders. And so, one of the things we wanted to see is, can we improve the mental and physical well‑being of people with mental disorders if we were to address their social determinants?

So that's a big question that we are trying to address in this paper. And then, we actually tried to explore this across three different levels. The first level is we wanted to see whether whatever social determinants play a role, did they play a role for people with common mental health disorders, as compared to those with severe mental health conditions.

Second level we wanted to see is the effect that we see for these social determinants, similar or different across Latinos, African‑Americans, Asians, and whites.

And thirdly, are there-- if you look at different types of social determinants, will we see a different impact based on those social determinants?

So, we looked and simulated changes in education, employment, and income. Why use a simulation study? Well, ideally, if you had lots of money, time, and lots of people willing to offer these social services, you would really be able to do a lot of the studies, but that’s typically not the case. These experiments are extremely time-consuming, they take a lot of resources, take a lot of effort and time to actually find the answers. So, the idea is that if we can use simulation studies, we could at least try to identify what are programs that are more likely to have a change using data that has already been collected, rather than collecting new data. So, you actually use information from real participants but what you do is you change or simulate the situation to try to see, well, if you to change their education, what would be the impact?

If you change-- if you actually change their employment, what would be the likely impact? And what you do is, you do it through statistical techniques with already-collected data, you can try to explore what is the most likely intervention that is most likely to work, and work for those three different levels we talked about previously. Next.

The methods, I’m going to go very quickly through the methods, but we were very fortunate to use two data sets. One was the National Institute of Mental Health Collaborative Psychiatric Epidemiologic Surveys. And this actually is a group of three surveys that were merged together because they use the same sampling design and were designed to be integrated together and they also use the same measures and instruments to collect the data within a very similar timeframe, and so this with a study of 16,423, and this is conducted from 2001 to 2003. It’s not a new data collection, but it is one of the richest data collections in terms of sample size, diversity, and also it collects very, very good information about psychiatric disorders and all of the social determinants that we were exploring. This group, this sample we are going to use for participants with common psychiatric disorders.

The next one is the Social Security Administration Mental Health Treatment Study that was actually also conducted in the early 2000s, and it has a control group of 1,006 people. So, this is important because when you are trying to do the simulation studies, you actually need very good data and also very good samples that can be representative of the population that you want to target. Next.

I am going to explain that why we did the study in two data sets. Like I said, the first data set included people that had common disorders. It includes a population that is of adults that are 18 or older, and the next data set, that I call group B, like I said, is a mental health treatment study of SSDI recipients with a primary diagnosis of schizophrenia or an affective disorder. So, this are severe people suffering from severe mental illness. Next.

So, if you think about the disorders when you think about which are the disorders that are being included in the common disorders, we're including social phobia, generalized anxiety disorder, PTSD, eating disorder, so it’s really more a general type of disorder in the population.

While for the severe mental health disorders, we're including pretty much people that have schizophrenia or affective disorders, including major depressive disorder, bipolar disorder, and psycho-affective disorder.

So, you see a general idea of the distribution of race ethnicity by groups. The first group A, that included, out of that big sample that we talked about, we only included 3,417 adults that had one, at least one disorder. And this as you can see here is mostly composed of 1,424 whites, 1,039 blacks, 717 Latinos and 237 Asians. On the other hand, we had 1,006 adults, and from those, again, 611 were white, 272 were black, and 123 were Latino and they did not have a sufficient Asian sample to be represented. Next.

I'm going to leave it to Justin who is going to explain the simulation study. Thank you, Justin.

JUSTIN METCALFE: Alright, thanks, Maggie.

Yeah so just a quick aside, we are using the term simulation in a specific way. Sometimes when people hear the term simulation, they think that we're reproducing some sort of mechanism, and so we're making causal assumptions. In this case what we’re doing is, we’re basically reconstituting or redistributing these samples to simulate what that sample would look like if somehow the hypothesized improvements have been made in income, education, or employment.

And so, you know, we have two data sets and we followed the same basic procedure with each data set. So, we have each one stratified by racial or ethnic group and we want to project how altering levels of education, income or employment within each group will impact mental or physical health. As with any other comparison study or comparison groups, we have to control for certain factors.

And in this case, we want to control for the effects of age, sex, and marital status when we are comparing across the groups. These are known predictors of health, and they are legitimate known predictors of health. We’re not trying to change them in this study. So, the problem is that age, sex, and marital status-- excuse me, that age, sex, and marital status, they’re also associated with education, income, and employment. It’s a very tangled web of associations. Therefore, we simply control for age, sex, and marital status.

We will drag education, income and employment along with them, and we will upset ‑‑ we will fundamentally alter the group‑specific relationships between health, which is our outcome, and our modifiable social determinants. In a sense, if we just straight controlled for age, sex, and marital status, we’d be altering the natures of the groups and the disparate relationships between health, education, income and employment to achieve this exchangeability.

So, to preserve the group-specific relationships between health, education, income, and employment, we used a method developed to look at disparities and access to health care, which is a situation in which you want to eliminate the role of legitimate predictors of the need for care, for instance, age and gender, and maintain the distribution of predictors that determine, in part, the group's specific disparities. And this method, instead of blanket just controlling across groups, we're effectively controlling within levels of our modifiable social determinants: education, income, and employment. And this preserves the group‑specific distributions of these factors that we're so interested in.

So, for example, if you have a certain income bracket and people in this income bracket in a comparison group are older than those in the reference group, then what we would do is we would in that income bracket, we lower the ages of people in the comparison group to match the distribution of ages from the reference group. So, we haven’t changed the distribution of income overall, we have just changed the distribution within the income bracket. So obviously we aren’t stratifying in this analysis, per se, because we’re dealing with three factors, and that would constitute far too many strata. So instead we’re making the distributions of age, sex, and marital status conditional on education, income, and employment. And we're making across all the comparison groups, we're making that relationship the same as in the reference groups, just as I described previously with my example. We do this with the propensity score-based weighting technique, used in the disparities research that I previously mentioned. And we re‑weigh every observation so their relative contributions create samples with this conditional control.

Then, the mean health scores within each group control for age, sex, and marital status while preserving the group-specific relationships between health and our modifiable social determinants. And furthermore, if we stratify, for instance, by level of education, the health scores within these levels maintain this level of control. And then we can simply alter the relative weight of the different strata to stimulate how, for instance, how a broad improvement in education would make for a specific group, how big of an improvement it would make.

So, the next three slides illustrate the way we’re simulating the group in age, income and employment.

Here in education, we have--we divide education levels up into categories, you can see the categories here. And in study A and study B, the general idea is the same. We are moving a certain proportion from ‑‑ up from their current category into the next category. So, we’re basically bleeding people away from the lower level and pushing them up. And in study A, we did 30 percent and in study B, we just did the full range from 0 to 100 percent. The difference in methods, the slight difference in methods are just down to the fact that we have two different teams working on these two analyses because of different access to the data sets. So, in income, in study A, they actually increased ‑‑ they simulated increase in household income by a certain amount of money. In study B, we just divided income into quartiles and bumped everybody up from 0 to 100 percent in steps of 10 percent. So, at the beginning it was that baseline same as in the study sample, and in the 100 percent, there was no one in the lowest quartile any longer.

And umm-- yeah so, the only problem there with study B is that study B was a sample SSDI recipients and they had a very narrow range of income, so that would, in part, explain some of the subsequent results. For employment, study A moves 60 percent of the unemployed to the employed category and study B is a range of the category.

So, before we get into the results just to touch quickly on the difference between significant and substantial, a significant effect, simply put, is a real effect, and a substantial effect is a meaningful effect.

We can say something is significant and it doesn't have a meaningful ‑‑ that change doesn’t represent a meaningful improvement in someone's life. And technically a significant effect means that, in this case, since we have – we’re are looking at an alpha of 5 percent, that there was only a 5 percent or less chance of seeing what we saw, assuming that there was no real effect. And meaning that there was less than a 5 percent chance-- that what we measured was a result of sampling error. Okay.

So, here are the results. When we modified education in group A, we saw generally we saw small improvements in physical and mental health, the one exception being Latinos in the physical health measure, but the improvements are small enough that I wouldn't call that a big difference between the no change and small, and the significance levels weren't incredibly high. In group B, the people with severe mental health disorders, from the MHTS study, we saw no significant change.

When we modified income, we bumped group A up $20,000, or group B up a quartile, various proportions of people up a quartile of income. In group A, people with common mental health disorders, we saw small improvements across the board, except for in Asians. And in physical health, we saw moderate improvements in whites and African‑Americans, no change in Latinos and Asians. In a subsequent slide, you will see graphically displayed what that actually means.

And then when we bump people up into the employed category, we saw in group A generally substantial improvements in mental health, except for in the Latino portion.

And in physical health, no real change, but a small improvement among African‑Americans.

And this is the only real improvement we observed in group B, people with severe mental health disorders, there were moderate improvements of both mental and physical health, each at a slightly different level of bumping people up into the – each with improving the employment rate by a slightly different amount and there's a graphic to show that as well.

So, this is Exhibit Two from the paper we are discussing right now. It depicts changes in physical health, after improvements in income, education, and employment. In the sample, with common mental health disorders stratified by a group, and we can see with the large green bars that the changes associated with employment improvements were substantial if not always significant, and you can see that in the Latinos.

And the improvements associated with household income and education were of relatively small magnitude and occasionally significant. But clearly the employment improvements yielded the biggest changes.

This is an exhibit describing changes in self‑reported mental health among group B, which is the SSDI recipients with severe mental health disorders.

Across the bottom, that's the proportion of people we moved from the unemployed into the employed category, from zero to 100 percent, in increments of 10 percent.

And on the left is the change in the SS12 mental health summary score. Umm-- and we can see here the dash lined represent estimates that do not differ significantly from the baseline. When we improved education and household income, there is no significant effect. They’d go down, but that doesn't mean anything in the context. Employment goes up pretty steadily, and right around 30 percent it starts to differ significantly from baseline.

Yeah, that's basically how that turned out.

So, the limitations, there are a number of the limitations in a study like this. I consider this, personally I consider this to be a hypothesis‑generating analysis. We used different populations, outcome measures and study group sizes, it was difficult to compare the groups because they were from two totally different studies so they complement each other, the findings complement each other and you can’t really cross reference them, they’re not redundant. You probably can’t generalize these findings to use because interventions as it says here to improve SES conditions for members and minority groups, maybe most affected at a young age.

Other social determinants of health, we couldn't ‑‑ we couldn't simulate those, because we didn't have comparable measures in the data sets. We used cross‑sectional data, and this gets back to what I was saying earlier about the type of simulation we're doing. It’s essentially a re‑weighting of a cross‑sectional description of a population, and with relatively small samples, which reduced our power, especially when we stratified, we saw in the MHTS, we found we had estimates in the MHTS simulations specific to the group's strata, but we were under powered to see any significance or ‑‑ any significant changes.

And then the precision of the estimates, Number Five, which I already covered, and there's also the issue with self-report bias, and this is all correlational analysis, and this gets back to what we were saying earlier about causation and what is a simulation. We’re re‑weighting the sample and with the idea ‑‑ the assumption being that we can re‑create what the sample would look like if there were different facts on the ground.

So overall, our findings mean, or they imply that mental health symptoms among African‑Americans, Asians, and whites, with common mental health disorders will probably improve with supportive employment programs within mental health care settings. Both mental and physical health systems, among all the groups, with severe mental health disorders will likely improve with supportive employment programs.

And that's it, employment was, by far, the strongest intervention in this simulation.

So, I’ll pass it on to Amanda.

AMANDA NEMOYER: Thanks, Justin.

So, Margarita and Justin sort of took some time to talk about the details of this simulation study that they put together.

But, in addition to further identifying which addressing which social determinants might relieve some of the racial ethnic disparities in mental health, a large part of this project has focused on bringing those kinds of results back to the people who are most affected by those disparities, and those individuals who actually have the ability to do something to do with the results.

And we wanted to do that because, historically, there's been a limited translation of evidence‑based findings from research into practice. So, in other words, empirical findings like these are typically, do not bridge that gap between the researchers who produce relevant information and the stakeholders who can actually use it.

So existing methods of knowledge creation and sharing findings through academic journals, they’re not consistently successful and, even when they are, these kinds of efforts can take 17 years to make any sort of kind of impact. So obviously we want to do things a little bit quicker.

So we thought ‑‑ and there has been some thought about how to do that, because policy makers and practitioners note that their lack of access to research evidence, and also their inability to contribute to the input about how to use that information, how to interpret that information, and how to apply it, those are some of the barriers that limit their ability to use the research information that gets produced. So, to improve upon the existing methods, our research group thought to work directly with stakeholders to interpret the findings of our stimulation study and try to develop practical recommendations for their application.

And so, to do so, we developed partnerships with three organizations that work with persons with lived experiences with behavioral health conditions or PLE’s, community‑based health advocacy groups and state-level policy makers. And so, with the help of these organizations, we hosted three focus groups comprised of these relevant stakeholders. And during each meeting, we presented our simulation results, just like Maggie and Justin did for you all. We also asked them to sort of explore the implications of the findings that we had from a simulation, and then encouraged them to generate ideas for how our findings should be interpreted and applied in the real world.

So, I'm going to talk about what they said.

So, first, across groups, stakeholders had several thoughts and questions about the study and its results. Specifically, state policy makers expressed surprise that our results, which didn’t demonstrate a significant effect from education, appeared to conflict with the research that we presented to them and that you all saw earlier from Maggie that the Robert Wood Johnson foundation had conducted, that suggested that an additional 4 years of education could reduce a range of health risks.

So, they had questions about that. Additionally, some group members identified some concerns they had about the study, including the categorization of common versus severe mental health disorders, and the fact that certain minority groups, such as Asian Americans and American Indians were not well-represented. So, this is all feedback about the study. They also raised several factors that they thought should be further explored. Justin talked about why we couldn't do that, umm--but they thought it would be helpful for researchers to explore these factors in the future, including health literacy, medication, availability, and use, urban versus rural settings, housing status, language, and nativity. And across groups, stakeholders really tended to agree that policies that currently exist should better serve individuals with mental health challenges and facilitate their ability to obtain employment.

So, we will go into more detail about stakeholder feedback and recommendations for such policies in the next several slides.

So, in the persons with lived experience group, participants frequently reflected on their own histories of mental health challenges and the difficulty they face in obtaining employment. They also identified several limitations of existing programs aimed at providing supports in this area. For example, {inaudible} reported that application processes for employment programs and disability benefits in general are often arduous and often require assistance that costs money. So, they need to hire someone to help them fill out paperwork, but that's a pretty limited resource for them.

Additionally, they thought that, they reported that, in their perception, existing programs often target younger adults and older adults feel that they aren’t able to obtain the similar resources through these employment programs.

They also pointed out that programs don’t always recognize that mental health recovery isn’t really a steady process, and that there are plateaus and relapses along the way that sometimes programs don’t really account for that and provide for that. They also noted that there are other areas and other needs that PLEs have and other people with mental health conditions that have that need to be addressed and aren’t addressed through these employment programs, such as food security or housing stability. They also noted that participating in these kinds of programs and even obtaining employment can cause reductions in the supports that they already receive, whether that is from Social Security, disability, or other services that only apply if you are not employed.

And then finally, they mentioned that existing programs, they’re not very common, they’re not very easily accessible in the sense that they may not be easy to get to via public transportation, and then also because there's not so many of them they don’t have enough space for all of them that are participating or interesting in participating.

So those were the concerns that they raised with existing programs. And then they also provided several recommendations for the development of quality, supportive employment programs, including employing certified peer specialists that can help participants obtain employment and also engage in daily routines, they might also help navigate some of the paperwork and systems that individuals with mental health disorders often have to navigate in order to get the supports that they need. They also recommended that programs have better safety nets to account for relapses and assist with specific challenges they might face, such as the lack of transportation to the program, or to potential employment sites, or a lack of professional clothing for going out on interviews and similar things. PLEs also suggested that programs incorporate support groups into the available services they offer so that participants could help support each other as they work to achieve their employment and mental health goals. And finally, they noted that supportive employment programs should really strive to achieve a holistic-- what they call a “whole‑person approach” when working with individuals with mental health challenges. In particular, they thought that programs should include supports for substance use and history of trauma, as these conditions are often co‑occurring, and should also provide resources related to wellness and skills training. Some examples they gave included discounted fitness memberships to YMCA and workshops on skills that they’ll need, including complex problem solving. So, these were the recommendations from our persons with lived experience focus group.

We also had a focus group with a group of community health advocates that work in community‑based organizations across the country. And when this group was interpreting and commenting on simulation results, they often framed the issues that they raised around the needs of-- {Audio disturbance}.

For example, they noted that Latinos often experience different types and conditions for employment, so they--such as higher rates of manual labor and cash employment that might not be reported in a typical way. And so, they noted that these groups would also face additional challenges related to economic and cultural differences that might not be captured just by saying, you know, all they need to do is find a job and they should be all set.

They also had similar thoughts about many African‑American communities, as many men and women are concentrated in urban areas, with safety and access issues that could contribute to poor health and mental health regardless of employment status.

And then they also raised concerns about the lack of Asian American, American Indian, and Alaska Native in these types of surveys. So, if they are not included adequately in the surveys that are used to form these simulations, then we can’t really get a good sense of the needs that they have and their unique needs, likely related to multiple languages that can make inclusion a challenge.

So, members of this focus group really raised circumstances unique to each diverse group to illustrate how our simulation results alone might not paint a completely accurate picture of the needs of underrepresented groups. So, they gave us several recommendations, and many of those recommendations really did focus on the needs of communities of color. So, for example, they noted that cities might establish resident ID cards that could allow for health treatment and supported employment program involvement without creating anxiety about immigration or citizenship status. They also suggested that employment and mental health programs should improve their language accessibility to better support individuals from the Latino and Asian immigrant community.

And that programs should employ the use of peer support; they gave the example of “promotoras”, to educate and disseminate information to communities of color.

And they also recommended strengthen local partnerships between and among community organizations, health clinics, and programs that provide supportive employment services.

They talked about the need for funding. And they encourage programs to seek out state, federal, and foundation funding so they can develop or extend on the services they provide.

And they also really urge Federal agencies, like the Social Security Administration, to reduce and remove barriers for people receiving benefits that also want to work. So, in that way, those individuals are not discouraged from returning to employment by having concerns about losing their benefits, which is something that the PLE group mentioned as well.

And so, our third focus group was a group of state-level health policy makers from around the country. And so, they really described their own experiences with supported employment programs at the systems level in their state.

So, for example one policy maker reported feeling pleased that our simulation results supported their state’s ongoing efforts to expand individual placement and support services in the state. Another policy maker reported that their state actually already offers a number of supportive employment programs, but they have had challenges related to utilization, so people aren’t using the programs that are in place, perhaps because of limited resources and low reimbursement rates. And then others noted that some state agencies have been working with other agencies and community and faith‑based organizations on these issues already, so for example one state policy maker described the interagency council that focused on homelessness that included representatives from mental health, substance use, and Medicaid agencies that are actively working on this. And another state created a dedicated unit under Medicaid that aimed at working with African‑American men and women to address social determinants of health in general. And they have been working with a lot of community and faith‑based organizations to try to do more outreach.

And policy makers really spoke at length about the need for funding to support ongoing work in this area. And as an example, a promising strategy, one state policy maker noted that their state created a health disparities work group that is focused solely on finding funding for programs like supportive employment, so that's very promising.

And this group provided us with several recommendations for states hoping to address these issues. So, first, they encouraged partnerships between state agencies, such as Medicare groups, the Department of Intellectual and Developmental Disabilities and others.

They also recommended teaming up with corporate entities who might be able to assist with job placement for program participants and academic researchers that can help educate agencies about existing strategies and tailor programs to state’s specific needs.

Policy makers also noted that groups within these partnerships must be able to share their data with each other to aid in measuring the impact of new and existing policies.

And like the health advocate group, policy makers encouraged programs to seek state and federal funding, and they also identified potential funding mechanisms, including state grants, ADA grants, and SAMHSA grants related to supportive employment and they listed several options.

And then finally, they discussed potential preventative methods, such as, including education and support for workers that were laid off. For example, through training in new skills and career coaching, to try to prevent long‑term unemployment and reduce reliance on disability. And therefore, reducing the need for supported employment programs in such a large part of the community.

So just in sum--the takeaways from this, we thought this is really essential to bridge different perspectives by including the voice of various stakeholders when disseminating these results. We thought it was essential to implementing effective programs.

So just because we had this result, we don't really know exactly how it will work in practice so having these three groups is really helpful in that. So, in this study eliciting reactions and feedback like I mentioned, from these groups enabled us to obtain a more robust picture of the opportunities and barriers to translating supported employment research practice.

And so whenever possible, we encourage individuals who want to pursue similar research to identify those individuals that might spearhead change to policy and practice, but also those individuals who can most benefit from those changes and then work collaboratively with them to take next steps on the way to progress.

So that's it, I will pass it over to Dr. Drake.

ROBERT DRAKE: Okay thanks very much, Amanda.

I'm a disability and rehabilitation and employment researcher, and I’d like to say more about policies related to employment. We can go to the next slide… okay, thank you. Whoops. There you go. So first, I want to say that employment is so strongly intertwined with health that, in some countries, northern European countries, they’ve begun to define employment as a health indicator. In this country, we insist on separating social services and social determinants from health and medical necessity, but that does not have to be the case.

Many countries have moved ahead to recognize that social determinants really account for the great proportion of variance in how people with all chronic illnesses do over time, and to make sure that they pay attention within health settings to social determinants. So many wealthy countries are way ahead of us in terms in trying to integrate health and employment services.

The second thing I want to say is that we don't really understand the results from the simulation study in terms of Latinos not appearing to benefit as much as the other groups that we study. But we do know from randomized controlled trials on supportive employment that Latinos benefit as much or, in some studies, even more than other folks when they are given supportive employment services.

So, I think we should recognize that people who are relatively disenfranchised, people who are relatively poor, people who are relatively marginalized, people who are somewhat outside of the healthcare system, need employment services as much or more than everyone else does.

And then third point I want to make here is that helping people with mental health concerns to become employed is good for all of us. It is good for people with mental health issues because, you know, overall, they do earn more income, even though people have fears about losing benefits. They end up being wealthier and getting out of poverty situations. It improves their health in all sorts of ways. You know, we talked about physical health, and mental health, but also in terms of substance abuse. You know, we've known for a long time that people that are unemployed tend to slide into substance abuse. And we know from our studies in supportive employment, that once people are employed, it helps to get control of their substance use and, in many cases, to recover from addiction. It also helps them psychologically and improves their self-esteem, it improves their relationships with other people. For their families, there are enormous benefits because people who are employed have something to do every day, they have a structure in their lives, they have more independence, they have more money to be able to live independently so the burden on families is not as great.

For employers, the benefits are-- are robust, you know, especially in a situation like ours now, where employers need good employees and they’re starting to become harder and harder to find. Across all the supportive employment studies, we hear from employers that people who have disabilities make good workers. A typical comment that I hear all the time is that the folks with disabilities are just like all of my other workers, except they have a better attitude. They value -- they really value the job and they stay in the job for a long time, so they had a positive effect on the rest of our employees.

**stopped at 59:40

Employment helps communities because people who are working are not falling into the other social ills that tend to attend unemployment, they are not drinking and using drugs, they are not becoming homeless, and so on.

And in terms of society, employment has robust benefits because people who are working decrease their use of hospitals, emergency services, day treatment programs, and on and on.

We’ve known for a long time that that supportive employment is cost‑effective compared to all these other programs, but it may save a lot of money in terms of health care costs that we haven't realized. People who are working, of course, pay taxes. People who are working get smaller amounts of social welfare benefits. And we believe, although it hasn't been demonstrated for certain yet, if we can help people with mental health problems to become employed, it will make it less likely that they enter into the disabilities system.

So, there are lots of studies that are psychosis and there is the large Social Security study called the supportive employment demonstration going on in 30 cities around the country now to examine this hypothesis, if we can get a good mental health and vocational services to people early on, we can decrease the likelihood that they will end up in disability. Which really means that we are conferring the life of poverty on them. People, once they get into the disability system rarely escape from the disability system and they rarely get out of poverty.

We could go to the next slide.

So, there are now 24 randomized controlled trials of supportive employment for people with mental illness. They all show large benefits somewhere between two and three times as much employment, about 60 percent of the people become successful in competitive jobs. These studies have been done across a wide range of countries, and also in many different settings in the U.S.

You know, we did the first study of supportive employment in New Hampshire, and that was criticized because people said that, well, New Hampshire is all white and they don’t have HIV and crack cocaine and all of these other American traditions. So, we did the next study in southeast DC and we recruited everybody out of homeless shelters, and we got the same results, exactly. People that are homeless and live in shelters and people that have crack cocaine and addiction to mental health problems wanted to work and had a very good chance of being successful.

So, if supportive employment is such a win‑win proposition, one really has to ask, you know, why is it that only two percent of people with serious mental illness in the U.S. have access to supportive employment. You know we have been working for 20 years to try to scale up the supportive employment programs and now, over 600 supportive employment programs across the U.S. but they still serve only a small percentage of people who want to work. You know, across all studies, about 2/3 of people with serious mental illness say they want to work competitively, and we're applying-- providing employment services to only about 2 percent of them. And, of course, that also leaves out many people who are living outside of the mental health system.

There are many barriers to implementing supportive employment. People with mental health problems are worried they are going to lose some of their benefits, or lose their health insurance, and if they lose their jobs, they’ll really be in big trouble.

People who are mental health professionals have the persistent incorrect belief that people with mental illness are unable to work and need to be protected and need to be supervised closely in segregated settings. But the big barrier to supportive employment is funding, you know.

Supportive employment is a relatively inexpensive service, compared to other things we do in health care. It costs about $5,000 in the first year to help somebody with supportive employment and most people don't need any services to speak of after the first year. But there's no consistent stream that mental health centers and other programs can build to pay for that $5,000. I put together a list for a NASMHPD meeting a few weeks ago of 15 different funding mechanisms that can help pay for supportive employment. You know, things like the ticket to work program which is in Social Security, and like Medicaid waivers, which come from CMS, and like vocational funding that comes from the Federal State Voc Rehab System, and state block grants for early psychosis programs, and on and on.

But the issue is that none of those programs pay simply for supportive employment. In order to pay for the service, local programs have to put together or grade the funding from these many, many different sources. And local programs just don't have the, you know, billing capacity to put together all these programs and keep them separate and they are worried about making errors and getting audited by Medicaid and so on and so on. And so, the net effect is that local programs are not able to overcome the fragmentation of our large systems. And the fragmentation starts at the federal level but extends into state programs and then, therefore, you know, extends to the local level. We don't really know how to solve this problem. You know, in other countries, they have real health care systems rather than messy non‑systems that we have in the U.S. They just make decisions at the top that, you know, employment services are going to be part of health care and they are going to be within the health care system, and they will be paid for just like national insurance, health insurance is paid for. But we can't do that in this country.

And I think that one solution would be that people at the state level, and this is what I have heard the NASMHPD state directors talk about, should be able to put all of the different funding streams together and create a single billing system for local programs so that the burden is not on all of these small mental health centers to try to figure this problem out. We could go to the next slide, please.

So, we have a lot of work to do still to try to make sure that people who have disabilities of all kinds have access to the employment services that they need. It will be essential to their really recovering from their health problems.

And we know, for many studies, that when we integrate employment and health services, we get better outcomes than when we try to deliver them in parallel service systems.

But we’ve gotta figure out a simple way to fund these integrated services. And this is true for people who have mental health problems, but it is probably true for everybody else who is unemployed, too. You know, there are now supportive employment studies around the world looking at people who have anxiety and depression and looking at people who have developmental disorders, Autism spectrum disorders, substantial use disorders, neurological disorders, and on and on.

But we have a long way to go to figure out how to integrate employment services into all of these different health care settings.

We also need to figure out how to integrate supported education into these programs. I think somebody said earlier in the—in this talk that ‑‑ people need education also in order to realize their employment goals. We often see, within supportive employment programs, that people will work for a while and, once they start to develop self‑confidence about working, they realize that they need to get some more education or training in order to get better jobs and get promoted and earn more income.

And I think that's a service that is an essential part of supportive employment, especially for young people. You know, most of the young people we've worked with in transition‑age programs, you know, they do some education, they do some employment, sometimes they do both at the same time, and our programs need to be ready to provide all of those services.

Okay. I'm going to stop at this point, and I think we have time for questions of all the presenters. Thank you very much.

ANDREA HORVATH MARQUES: Thank you very much for all of the speakers. It was a very extensive and important information that we have, all the way from the-- the explaining the design of the paper and why we are looking for the social determinants of health and impacts on health and mental health and how can we use that to address mental health disparities. And so, I will have some questions for you, and it was a, umm ‑‑ let's see. I was going to start, I'm going to start looking at some questions here. But I will--let me go back and point--to a little bit ‑‑ talking about the importance of using these kinds of models to use ‑‑ to identify pathways, and different pathways we can use to address mental health disparities.

So, umm-- I wonder if maybe, Dr. Alegria can—or Justin, or anyone of you can tell us how important it is to be using these kinds of datasets that can help us to come with different solutions to help to address mental health disparities?

MAGARITA ALEGRIA: Yeah, so thanks Andrea for the question. I really think the importance of using this computational modeling and data analytics, it is that ‑‑ it really allows us as Justin was saying, to identify potential targets for interventions. I think that one thing that Justin mentioned is that this really raises which hypotheses are worth--worth pursuing. Like we are saying, this is only something that helps which direction you should put your energies and your investments. Doesn't tell you exactly that everything will work. But it tells you, for example, out of this work it would suggest that employment, the social determinant of employment is one that is likely to work out and be pursued.

Umm, like we are not saying that education and income should not be pursued, but it might be that, and this, that we explain in the paper, that education works more for people that are early in their illness trajectory when they’re younger, and maybe doesn't have such a big impact now and it would be worthwhile to try to see whether that is the case. So, these are the only hypotheses, but it does help you target what are the potential interventions if we want to address, you know, both mental health disorders and disparities. The same with the use for disparities, if we wanted to test what are different types of mechanisms or pathways that might be likely options to test in an intervention. For example, increasing access, you might try with different data sets, you start testing the different hypotheses as far as why you are selecting some targets and not others.

ANDREA HORVATH MARQUES: Umm, Dr. Alegria, you also mentioned the importance when you are using those two data sets that you have, umm, because those data sets also have measures that you could be comparing that were collecting some social determinants of health. So, my question to you is (inaudible) how can we, umm, as a federal agency, how can we help to, umm, facilitate data sets, large data sets from different agencies, not only HHS agencies, to be-- for them to be merged and then to be, umm, linking those data sets?

MAGARITA ALEGRIA: Right. I think that there's three things that are needed for people to be able to do studies like this.

One is I think people need a lot of information on what data that is available that could be used, umm, because either it’s that is de-identified, but it has a lot of information of the social determinants and then it has good information about the outcomes you want to try to achieve, so that it can be used.

And then if you’re ‑‑ especially if you’re trying to test things that have to do with disparities that you have diverse representation in your population. So that might be something, that, you know, like here, we're looking for data sets that had big numbers so we can do the simulation that had that diversity, and then that had ‑‑ most of the measures that we needed to actually do the simulation.

That, that--and one thing I think is having a menu of options as the old data sets that are available with explanations and code books of what they include, how are the measures done, and then what are the samples and distribution of those samples by race/ethnicity, and especially with the outcomes that we could explore, I think would be very useful for the population, for--for researchers in general.

Second, I think, umm, having the opportunity to do trainings on how to do simulations, so it is an easy--an easier ‑‑ what Justin was trying to explain was actually computationally, umm, quite complicated. So, it might be very useful to hold some training seminars on how to do these simulations and other different simulations that people have done as a way to--to train a generation of researchers in this.

And then, finally, I think there needs to be some investment done in merging different data sets that might share the same people. For example, let me give you an example. For some of the data sets that we use for this, if we had had ‑‑ if we were able to invest in putting mortality data, it would have made it even more important in terms of trying to see who are the people, especially that might have a bigger chance of dying. If you remember, I mentioned that people with mental disorders are more likely to--to die 10 years earlier. Well, we could try to see, well, what, you know, we could simulate-- are some social determinants that might play a role in early mortality, or premature mortality. So, linking data sets that have different outcomes, especially outcomes that are‑‑ could be more objective would be a way to go.

So those are three things that I would recommend.

ANDREA HORVATH MARQUES: Thank you so much, thank you Umm, umm, I will ‑‑ I'm going to ask also some other questions from our, umm, audience here—umm, somebody is asking us that, umm, if there are any combinations of mental health, social service, and cash transfers, umm probably-- that are being led, umm‑‑ all of the services are being provided simultaneously?

ANDREA HORVATH MARQUES: Bob, do you want to take that question?

ROBERT DRAKE: Well, you know, part of the supportive employment model is that people need to get, umm, good benefits counseling, because people have unrealistic ideas about what’s going to happen to their benefits when they go to work and it’s a very complicated situation. Umm, there are, you know, situa—you know, people are able to earn quite a bit of money without having any impact on their cash transfers, but they don’t realize that.

Umm--within the VA, the system is a little different because people who have service dependents, or service‑related disabilities, umm, don't lose any of their cash transfers when they go to work. They—they’re allowed to work as much as they want and so that has been an interesting area to study.

Umm, there are also different arrangements in other countries for, umm, you know, supporting people financially while they are getting back to work, and there are various arrangements for supporting employers as well to hire people with disabilities.

I think one of the… I’m probably getting off the track here, but I’ll say one more thing, umm, one of the important things that policy makers need to be working on is thinking about how to make sure that people's health insurance is not diminished by going to work. You know, that is really a tragic fault in our system. We need--Policy makers need to be working hard to figure out to align all the incentives so that people can have insurance and also can benefit completely from returning to work.

MARGARITA ALEGRIA: Yeah, I also wanted to say, umm, Bob, that I--I don't think there is an answer to Regina's question. Umm, and I'm sorry for pronouncing your name—your name incorrectly. I don’t think to my knowledge, a study that actually combines all interventions into one. Not to my knowledge, I know one that do, umm,- cash transfer, one that do--does the employment, but not all of them at the same time.

ANDREA HORVATH MARQUES: Thank you. Umm, so Dr. Drake--Drake--Drake, umm, I’m--I want to follow‑up on your-- on what you were saying, that how some countries ahead of us in a sense of, umm, already merging—and-- and measuring those factors as an important factor for health.

Umm, how do you think of that, umm – if, umm, so I'm asking as an agency, for that--from us,…that we can help in a sense of providing support for, umm, research? How do you think we can help with that?

ROBERT DRAKE: …are you talk--when you say we, do you mean the NIMH, specifically?

ANDREA HORVATH MARQUES: I--I mean the NIMH, yes, umm, I mean, umm, the NIH, the research institution.

ROBERT DRAKE: Yeah... well, you know, if you look at OECD countries, umm, in terms of how many of them have much better health outcomes than we have across the board, not just in mental health but in all kinds of areas, they have better health outcomes than we do. And the big difference between those countries and the U.S. is that they spend twice as much on social services, or social determinants, as they spend on health. And in the U.S., we spend twice as much on health as we do on social determinants. And the fundamental truth here, I think, is that medical solutions to social problems are very expensive and incredibly ineffective. Very expensive and incredibly ineffective. We spend all of this money on, umm, you know, keeping homeless people in the hospital and in emergency rooms rather than providing housing. It just doesn't make any sense, and other countries, have, you know, figured this out long ago.

And I think what NIH could be doing to help work on this is try to figure out better how to integrate some of the social aspects of people's lives, helping them with the social aspect lives within the health care system. You know, we have all the money tied up in this gigantic, cyclopean health care system in this country. It’s ‑‑ we're not about to get it out of a system, and so I think that the social determinants have to be addressed within the health care system. And that services researchers have got to be thinking about how do we make the funding more streamlined so that it, umm, you know, really makes sense for providers and they're not forced to spend, you know, a 1/3 of their money on trying to figure out the crazy billing system that we set up with our fragmented services. Umm, It really should be possible for people to come into one center and get the services they need in an integrated fashion, and for the program that is providing health care and mental health care and addiction care and employment services and so on, umm, to be doing so as a package of services and getting paid for the package rather than having to decompose everything and bill for all these different kinds of services in 15‑minute intervals.

Our system just doesn't make any sense.

ANDREA HORVATH MARQUES: We are hoping to get some more solutions on that, no? Thank you for--for your comments and--and all of the information (inaudible)

Let me see, I’m getting more questions here... umm, just a moment— umm, so, umm, in a sense somebody is asking is kind of—what kind of steps that the government policy makers, researchers, and NGOs can take to, umm, to help to cut that cycle between poverty and mental health?

MARGARITA ALEGRIA: I think one of the things that we could do, again, is to address the social determinants, especially early on. I think there are programs that the government is doing, like, umm, through the income tax credit return, improve, you know, getting people and families out of poverty (inaudible), that is one example, and expanding the programs so the more people are eligible early on as a way to break the cycle of poverty and mental health.

And I think the other one has to do with how we implement, you know, safety in communities, so people have more secure environments where the exposure to violence and the exposure to trauma is reduced considerably.

Umm, those are things that we could do by having both the state and federal agencies to support more, and how do we do community healing and providing, you know, secure environments. I think there's a lot that can be done in the school system. I think people are now starting to think more how they could start not only with socioemotional development, but actually doing more public health interventions at the levels of school, especially with various bad school climates that affects children's mental health. The area, for example, expulsions and suspensions of children is an area that we could target early on because we know that area has a lot of biases and a lot of people are really‑‑ a lot of youth early on are targeted for those suspensions and expulsions. And as a result, we build a, you know, a pipeline to the prison system, or to the juvenile justice system. So I think, especially investment in, umm, both, you know, prevention and remediation early on, rather than only clinical care, is--is the way – you know, it is one of the ways that we have suggested that might be a way to break this cycle of poverty to mental health.

AMANDA NEMOYER: And also, I might add, just given my background in--in juvenile justice and criminal justice, that the overlap between people who are justice-involved and who have mental health, umm, ever since the deinstitutionalization is very large, and so the collateral consequences of being justice-involved often it--it kind of lead to poverty in the sense that job options are fewer, benefit options are fewer and so on for being a convicted felon. Umm, so, the overlap between those kinds of groups of people addressing decriminalization for minor offenses, things that can prevent people with mental health challenges from being criminalized, umm-- and, therefore, then sort of suffering those collateral consequences that lead to poverty, is also another way.

ANDREA HORVATH MARQUES: Thank you. Thank you for your answers.

And umm-- I just also – so, I-- getting closer. I just want to, umm, get a little bit about, umm, one of the questions that is related to the--the model, umm, of the (inaudible) ‑‑ sorry, about the, umm, individual placement and support model, umm, that, umm, you’ve been working with Dr. Drake and Dr. Justin, umm, I wonder if you—if you can-- mention because you mentioned also that various people get afraid about, umm, once they get the job, they are going to lose their--their support system, umm, all their support systems, and how—how you’ve been working with this and how you advise to use those models to be addressing those barriers?

ROBERT DRAKE: Well, people have a lot of unrealistic fears about going to work, and they’ve often been told by mental health providers that they can't go to work, or they are not ready to go to work. Our professional societies have lots of incorrect ideas about these things, too. But the reality is, when people go to work, they become more integrated in their communities, and they start to have friends outside of the mental health system, and they just do better in all kinds of unexpected ways.

You know, 30 years ago, I used to think that employment was a peripheral service outside of the mental health system that I didn't know anything about, and I didn't need to worry about it. And--but, you know, over the years of studying all of these treatments, as well as employment, I’ve come to believe that employment is a most effective treatment that we have.

We don't see people really get well on medications, or in psychotherapy in the same way that we see people, umm, you know, get well and start to take charge of their own lives and get out of the mental health system when they succeed in employment.

ANDREA HORVATH MARQUES: Thank you, thank you. Yeah, I think, umm, we had a nice overview here about the importance of, umm, measuring and tackling social determinants of health to-- also to help mental health and reduce mental health disparities. Umm…we--I appreciate, umm, everybody’ feedback, inputs here. Umm, I will let—I will also-I’ll be sharing with you all on the screen, umm, a link for the papers one of the papers that was, umm—umm, the main paper that was, umm—umm, discussed in this webinar. Umm, I welcome people to send us more questions if you, if you--—if you need it, if you think about it. Umm, you will have my email address. Umm you have the email address from our speakers. I would to thank you all, umm and it would has been a very, umm, important, umm, day for us to be – umm, and I want to thank you all for your feedback that I will definitely bring to our, umm, leadership in a sense of, well how we can we also help. Umm we are all working together on that. So, umm any comments from the speakers? I would like a word from you, please.

MARGARITA ALEGRIA: Yeah, I--I want to also thank the people that work with us in trying to put this, umm, you know, feedback from community—umm, from community advocates, people with lived experience, and people from, umm, the state health policy people. I wanted to thank ‑‑ (Reading described section) ‑‑ all who were really a big part of doing—umm, really, umm, giving us feedback about this work. So, I want to thank them, too.

ANDREA HORVATH MARQUES: Any--any other comments? Umm…okay, I just umm--so I just want to finalize and also to—to, umm, bringing back what, umm, Dr., umm—Amanda NeMoyer mentioned about the importance of, umm, addressing also the communal justice and practice policies and—umm, related to the social determinants of health. And I just wanted to--to remind people you can also take a look that last August, there was the NIMH Service Research Conference here, umm, that was supported by one of our divisions. And one of the, umm, the keynotes, umm was from the Judge Steve Leifman who, umm, has been a leader in helping to end the criminalizations—umm, the criminalizations of mental illness (indiscernible). So, he--he gave us an amazing keynote and that is going to be available on our—on our NIH website.

Umm, so I encourage people to also take a look at that.

So, I think, umm, we are--we are done right now, and I would like to one more time, I want to thank everybody, umm, and, umm, have a nice day. Anything, send us a note. we are going to share more information as you—as you need it. Thank you.

ROBERT DRAKE: Thank you.

JUSTIN METCALFE: Thank you.

MARGARITA ALEGRIA: Thank you.

AMANDA NEMOYER: Thanks.

ANDREA HORVATH MARQUES: Thank you, Thank you. Bye-bye.

WEBINAR OPERATOR: That concludes today's call, thank you for your participation, you may disconnect at any time.