A Novel and Practical Meta‑Booster for Supervised Learning

Author(s): Shenggang Li

Originally published on Towards AI.

A Stacking‑Enhanced Margin‑Space Framework for Dynamic, Loss‑Driven Ensemble Updates in Classification and RegressionPhoto by Thorium on Unsplash

Ensemble methods thrive on diversity, yet most frameworks exploit it sequentially (boosting) or statically (stacking). We introduce Meta‑Booster, a unified system that blends incremental updates — the “deltas” — of several base learners at every boosting step. Built on XGBoost, LightGBM, AdaBoost, and a compact neural network, the method supports both classification and regression.

At each round, we:

Delta extraction: Capture each learner’s one‑step update — margin increments for classifiers or residual deltas for regressors — to isolate its immediate predictive gain.Stacked combination: Solve a constrained regression on the held‑out set to derive a weight vector that best explains the current residuals, allowing contributions from all learners simultaneously.Iterative update: Apply the weighted delta with an optimal learning rate found via line‑search, producing a greedy, loss‑driven ensemble evolution that adapts to the task.

Unlike static stacking, where weights are fixed or full‑model outputs are averaged, Meta‑Booster tweaks the blend a little at every round, always chasing a better validation score. This dynamic scheme not only lifts accuracy (log‑loss, AUC) and precision (MAPE, RMSE) but also shows which learner is pulling its weight at each step. Tests on car‑price and credit‑risk datasets confirm: margin stacking drives classification, residual stacking powers regression…. Read the full blog for free on Medium.

Join thousands of data leaders on the AI newsletter. Join over 80,000 subscribers and keep up to date with the latest developments in AI. From research to projects and ideas. If you are building an AI startup, an AI-related product, or a service, we invite you to consider becoming a sponsor.

Published via Towards AI