Assessing Bias in Predictive Models with PROBAST

Author(s): Eera Bhatt

Originally published on Towards AI.

PROBAST. No, this bast is not the tree bark that helps us make ropes. Instead, PROBAST stands for Prediction Model Risk Of Bias ASsessment Tool.

But why do we need it? We live in such a busy time. But more specifically, scientists and clinicians get so swamped that it’s hard for them to keep up with the latest medical research papers. Understanding medical literature is a huge chunk of their job, so how can they do this efficiently?

Instead of manually reading over 15 papers on a specific topic, researchers can use a systematic review to synthesize a load of studies on a topic and their outcomes. On a high level, this helps researchers figure out what strategies — like diagnosis methods — work and which ones don’t work. Yay!

But there’s another problem: when researchers have to skim through papers on a time crunch, it’s hard to account for biases that might have occurred in them. This is why the authors of this paper created PROBAST.

OK, but what does PROBAST actually do?

According to the researchers who created this tool, it assesses the risk of bias in a study and how well its predictive models can be applied to a medical topic.

But what does this actually mean?

Risk of Bias (ROB)

No, it’s not that kind of rob. Bias is when the researchers make a systematic error during a study that affects its results.

For instance, let’s say we create a model to predict the risk of a disease in a person. But the researchers train the model with much more data for white individuals than for South Asian individuals. As a result, when the model is used on South Asians, it has a lower accuracy; the model is biased.

In general, researchers know some important features that can affect bias, but there isn’t much evidence out there that highlights the most significant factors. Here are two examples of factors that affect biases in not just predictive models, but also in other medical research:

Blinding of outcome assessors to other study features. Participants and investigators in the study are purposefully kept unaware of the treatment that patients are receiving in the medical study. When patients are asked questions about their pain and symptoms for the study, they don’t add as much bias to the results.

Consistent definitions for predictors and outcomes. If we aren’t clear about how we define certain predictors and how we label our outcomes from the beginning, this can interfere with our results. So let’s avoid that.

Applicability

This refers to how much a certain piece of medical research actually relates to the clinical question or issue that needs to be addressed.

For example, when a predictive model can’t be applied, it might be because the participants in the study are in a different environment than the participants in the systematic review question.

Think about it. We might train a predictive model using data from participants in a clinic, even though the systematic review question is specifically about patients in hospitals. Clinic patients are probably less sick than those in a hospital. So if we use this predictive model on hospital patients, there are more variables to the environment that can make the patients’ results less accurate.

Development

These researchers developed PROBAST using what is called a Delphi procedure. In this case, they answered questions about bias using the consensus of the 38 experts they asked. Afterward, the PROBAST tool was piloted (tested) and refined thoroughly.

Organization

PROBAST was organized into four areas: participants, predictors, outcomes, and analysis. In total, the researchers included 20 signaling questions to judge the risk of bias of predictive models. Through signaling questions, the PROBAST model learns about features in a study that might cause bias, so PROBAST can measure the ROB with this information.

PROBAST isn’t just used by researchers and clinicians who want evidence-based medicine. It’s also used by organizations who want to consider ROB as part of their decisions, as well as science journal editors and manuscript reviewers who are editing content for accuracy.

Prediction

PROBAST detects the risk of bias of predictive models. In medicine, the prediction often refers to one of the two:

• Diagnosis: the chance that a condition or illness is present in a person, but hasn’t been detected yet.
• Prognosis: the chance of a certain medical outcome developing in the future. Often, this refers to whether a patient will recover from an illness.

Stages of PROBAST

PROBAST was created over four stages: defining scope, reviewing evidence base, Web-based Delphi procedure, and refining the tool through testing.

To start, a group of nine experts in prediction model studies agreed on some broad key features that PROBAST needs. Afterward, 38 more experts refined PROBAST’s scope using the Delphi procedure that we mentioned earlier.

The experts decided that PROBAST is meant for primary studies dealing with multivariable prediction models. Let’s break down some of these big words…

A primary study is original research that — in most cases — ends up published in a peer-reviewed scientific journal. For a primary study to be considered by PROBAST, it has to update, validate, or develop some kind of prediction model. Also, “multivariable” means that the model uses multiple factors to predict an outcome.

Conclusion

Without medical researchers and clinicians, so much of the knowledge saving our lives wouldn’t exist as it does today. Because of the authors who developed PROBAST, it’s so much easier to catch biases in the medical studies that we need to evaluate. Thankfully, researchers can now make more thorough decisions to help our patients for years to come.

Further Reading:

[1] Barrett, D. and Heale, R. (2020) What are Delphi Studies?, Evidence-Based Nursing. Available at: https://ebn.bmj.com/content/23/3/68

[2] Uman, L.S. (2011) Systematic reviews and meta-analyses, Journal of the Canadian Academy of Child and Adolescent Psychiatry = Journal de l’Academie canadienne de psychiatrie de l’enfant et de l’adolescent. Available at: https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3024725/

[3] van der Ende, N.A. et al. (2022) ‘Blinding of outcome assessors and its association with outcome in a randomized open-label stroke trial’, International Journal of Stroke, 18(5), pp. 562–568. doi:10.1177/17474930221131706.

[4] Wolff, R.F. et al. (2019) ‘PROBAST: A tool to assess the risk of bias and applicability of Prediction model studies’, Annals of Internal Medicine, 170(1), p. 51. doi:10.7326/m18–1376.

Join thousands of data leaders on the AI newsletter. Join over 80,000 subscribers and keep up to date with the latest developments in AI. From research to projects and ideas. If you are building an AI startup, an AI-related product, or a service, we invite you to consider becoming a sponsor.

Published via Towards AI