Pioneering Suicide Risk Detection Through LLMs: Fine Tuning GPT-3 With N-Short Learning

Author(s): MSVPJ Sathvik

Originally published on Towards AI.

Have you noticed anyone feeling suddenly disconnected who has socially isolated themselves? Has anyone closer to you expressed their innate feelings of lacking a sense of belonging? Has anyone you know said anything similar to “Even in a huge crowd with everyone I know, I still feel alone; no one understands me”?

If so, your friend might be experiencing Perceived Burdensomeness(PBu), a feeling that can be dangerous when combined with Thwarted Belongingness (TBe).

Have they ever revealed signs of believing to be a burden to their loved ones? Have they ever conveyed having a sense of being harmful to their loved ones, too?

If so, they might be undergoing the Thwarted Belongingness (TBe) we discussed. The ones who experience a combination of the two risks, Perceived Burdensomeness(PBu) and Thwarted Belongingness (TBe), are at a high risk of suicidal behavior.

We, as researchers, have been deeply moved by the happenings everywhere, seeing. (What we decided to build)

These two factors are powerful indicators for us to detect mental health crises among the ones around us. Still, unfortunately, the traditional methods of identifying these risk factors, like self-reports and clinical interviews, have limitations where most of us might not show the convenience of expressing our deep, sullen emotions.

As we have been looking for some good way to detect mental health crises around, we’ve seen that in our everyday lives, a lot of us show some mental health signals that might be subtle, fragmented, and dispersed across posts on social media.

We’ve taken this small opportunity from where we get signals and integrated AI, specifically large language models (LLMs), as they are brought together, can filter massive volumes of text data to detect patterns indicative of emotional distress.

Revamping LLMs: Can LLMs be trusted for mental health decision-making?

We all know GPT-3, the most popular LLM like every other LLM, has revolutionized natural language processing (NLP) by generating human-like text based on vast amounts of data. These LLM models are trained on enormous data in diverse language inputs. They can easily understand a language’s context, syntax, and semantics in ways that traditional machine-learning methods struggle to achieve.

However, when we tried to unify the existing LLMs like GPT-3 with sensitive domains like mental health, we realized that a significant drawback with these LLMs was the lack of transparency in their decision-making process. Though the models effectively classify and even predict emotional states, understanding the reasoning behind their predictions has been a considerable concern. In domains like healthcare, where decisions must be explainable and interpretable, this hindrance can limit the potential of AI tools.

We’ve considered this issue and buckled up to refine the existing LLMs so that healthcare professionals can trust AI decisions. We needed the reasoning and explanation behind the predictions to be transparent and interpretable. This is where our InterPrompt method comes into play.

InterPrompt: Our game-changing method to better AI-driven mental health detection

This InterPrompt method addresses the need for interpretability using N-shot learning in large language models to detect and, most importantly, explain Thwarted Belongingness (TBe) and Perceived Burdensomeness (PBu). We brought N-shot learning into the LLM, where this method helps train models using a small number of “shots”(examples) to achieve robust performance even with limited data. Since we’ve chosen social media data to detect the presence of mental health crises, expecting a great amount of data with clarity from a single post might not be very practical. We’ve designed the N-shot learning to take up a small number of examples and still produce good results.

We could successfully fine-tune models like GPT-3, where we could not only classify Reddit posts that exhibited signs of TBe and PBu but also explain why the model has reached a particular conclusion.

This approach is precious in mental health, as subtle text interpretations are critical. For example, say a phrase expresses loneliness and disconnection; it might be classified as a sign of Thwarted Belongingness. Though the classification is helpful, mental health professionals might be hesitant to act on the AI’s recommendation without its valid reasons for concluding.

We’re happy to announce that the interpretability provided by our InterPrompt method can enhance the model’s trustworthiness by generating coherent and understandable explanations, bridging the gap between raw and crude predictions and practical insights. This has made it easier for healthcare professionals to make informed and calculated decisions based on AI outputs.

Fine-tuning and N-shot learning: The perfect combo!

We’ve used a dataset of Reddit posts(primarily the subreddit communities focusing on mental health issues)of almost 3522 divided into training, validation, and test sets; Reddit was an ideal choice because it had millions of stories and points of view where many people have put their personal, innate feelings and struggles down, making it a rich data source for analyzing risk factors.

The biggest challenge we faced was differentiating posts that exhibited TBe or PBu from those that did not. To deal with this, we worked on a merge of N-shot learning and fine-tuning techniques. Fine-tuning the GPT-3 model allowed it to learn keen intricacies of emotional distress related to interpersonal risk factors. Meanwhile, N-shot learning ensured that the model could generalize from just a few examples, making it quite efficient in detecting the risk factors with limited data.

The Big reveal: How did our model perform?

Our results were promising, with four variants of the GPT-3 model(Ada, Babbage, Curie, DaVinci) being fine-tuned using the InterPrompt method. They have outperformed baseline models like BERT, Roberta, and MentalBERT in detecting the two major suicide risk factors. We’ve used several evaluation metrics, including ROUGE-1 scores, BLEU scores, and Exact Match(EM) scores, where our analysis has proved the fine-tuned GPT-3 model’s superior performance.

Furthermore, we’ve also used statistical parameters like t-tests to examine and validate.

Ethical questioning and a look into the future

Blending AI into mental health care has raised several ethical questions, particularly about privacy, bias, and accountability. As the mental health data is quite sensitive, it is crucial to ensure the AI models do not misinterpret or over-generalize emotional expressions.

One of the key features of the InterPrompt method is its ability to focus on explainability, which helps it mitigate some of the ethical concerns by providing reasoning behind the AI predictions, making them more understandable.

Looking forward, we think this method could better be expanded to detect TBe, PBU, and other risk factors. Moreover, this model could be applied to other platforms, including Twitter and Facebook, where many share personal experiences and anecdotes about mental health.

Beyond InterPrompt: A new leap in AI-driven mental health solutions!

The InterPrompt method represents a significant leap in the advancement of AI in detecting interpersonal risk factors in mental health. By bettering the LLMs’ interpretability, we have successfully demonstrated that it is possible to detect emotional distress and also explained why a model has made a particular prediction.

We feel that this crucial step involving innovations like InterPrompt, being taken in creating AI tools that are both effective and trustworthy, especially in domains like mental healthcare, would pave the way for personalized support and save lives by providing help to the ones in need.

Join thousands of data leaders on the AI newsletter. Join over 80,000 subscribers and keep up to date with the latest developments in AI. From research to projects and ideas. If you are building an AI startup, an AI-related product, or a service, we invite you to consider becoming a sponsor.

Published via Towards AI