Can CatBoost with Cross-Validation Handle Student Engagement Data with Ease?

Author(s): Talha Nazar

Originally published on Towards AI.

Understanding student engagement is essential in the digital age of online education, internships, and competitions. But what if we could predict a student’s engagement level before they begin? This story explores CatBoost, a powerful machine-learning algorithm that handles both categorical and numerical data easily.

Using real data from a company that offers courses, internships, and competitions, we’ll demonstrate how CatBoost can help predict student engagement factors — revealing which students are likely to stay engaged and complete programs successfully.

Here’s what we’ll walk cover:

• What is CatBoost?
• Key Advantages of CatBoost
• How CatBoost Works?
• Real-world applications of CatBoost in predicting student engagement

By the end of this story, you’ll discover the power of CatBoost, both with and without cross-validation, and how it can empower educational platforms to optimize resources and deliver personalized experiences. So, let’s jump in and explore the future of student engagement prediction!

What is CatBoost?

CatBoost is a powerful, gradient-boosting algorithm designed to handle categorical data effectively. Developed by Yandex, CatBoost was built to address two of the most significant challenges in machine learning:

1. Handling categorical variables efficiently.
2. Avoiding overfitting and reducing gradient bias.

CatBoost is part of the gradient boosting family, alongside well-known algorithms like XGBoost and LightGBM. Still, its automatic handling of categorical features particularly distinguishes it without the need for extensive preprocessing or one-hot encoding.

Gradient boosting involves training a series of weak learners (often decision trees) where each subsequent tree corrects the errors of the previous ones, creating a strong predictive model.

Key Advantages of CatBoost

CatBoost has several advantages that set it apart from other gradient-boosting methods:

1. Automatic Categorical Feature Handling: No need to manually encode categorical features. CatBoost automatically transforms them, making it ideal for datasets with many categorical variables.
2. Fast Training Time: With built-in support for GPU processing, CatBoost is optimized for speed, even on large datasets.
3. Superior Accuracy: CatBoost uses a unique way to calculate leaf values, which helps prevent overfitting and leads to better generalization on unseen data.
4. Reduced Hyperparameter Tuning: CatBoost tends to require less tuning than other algorithms, making it easier for beginners and saving time for experienced data scientists.

How CatBoost Works

CatBoost incorporates a few novel approaches to overcome the limitations of traditional gradient boosting:

• Ordered Target Encoding: Traditional methods of encoding categorical variables often lead to data leakage, where information from the validation or test set can “leak” into the model during training. CatBoost addresses this by using an ordered boosting technique, which processes categorical data to respect the sequential nature of data, thus preventing leakage.
• Oblivious Trees: CatBoost uses symmetric (or oblivious) decision trees, where the same condition is checked across all nodes at a given level. This structure speeds up calculations and makes the model more interpretable. It also enhances the model’s stability by reducing variance, which helps combat overfitting.

How to Use CatBoost in Python

Let’s look at how to get started with CatBoost in Python. First, install the library using:

!pip install catboost

Dataset Overview

The heatmap visualizes missing data across various columns in the dataset. Yellow indicates missing values, while purple shows available data. Key columns with missing values include ‘Reward Awarded Date,’ ‘Completion Date,’ ‘Reward Amount,’ and ‘Skill Points Earned,’ potentially due to incomplete course or program engagements by students. This visualization helps in identifying data quality issues and planning imputation or cleanup strategies for meaningful analysis.

Step-by-Step Guide: Predicting Student Engagement with CatBoost and Cross-Validation

1. Data Preprocessing

To start, we preprocess the data by:

• Handling Missing Values: We handle missing values as necessary, either by imputing or ignoring them as appropriate.
• Specifying Categorical Features: CatBoost handles categorical features natively, so we simply define which columns are categorical.

import pandas as pd
from catboost import CatBoostClassifier, Pool

# Load and clean data
data = pd.read_csv("student_engagement_data.csv")
data['Completion Status'] = data['Completion Status'].apply(lambda x: 1 
if x == 'Completed' else 0)

# Define categorical features
categorical_features = ['Profile Id','Opportunity Name', 'Opportunity Category',
 'Gender', 'Country', 'Current Student Status', 'Status Description',
 'Current/Intended Major','Learner SignUp Month','Learner SignUp Day of Week']

2. Splitting the Data

To ensure robust evaluation, we split the data into training and testing sets.

from sklearn.model_selection import train_test_split
# Set target variable `y` and feature set `X`
X = data[['Profile Id','Opportunity Name', 'Opportunity Category', 
'Gender', 'Country', 'Current Student Status', 'Status Description', 
'Current/Intended Major','Age', 'Engagement_Duration',
'Learner SignUp Month','Learner SignUp Day of Week','Learner SignUp Year',
'Learner SignUp Day']]

y = data['Completion Status'] 

# Split data for training and evaluation
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, 
random_state=42)

3. Initialize the CatBoost Model with Parameters

Here we set up the CatBoost model with key parameters and perform cross-validation for robust model evaluation.

# Create Pool for CatBoost
train_pool = Pool(data=X_train, label=y_train, 
cat_features=categorical_features)

# Define model parameters
params = {
 'iterations': 100,
 'depth': 6,
 'learning_rate': 0.1,
 'loss_function': 'Logloss', # Use 'Logloss' for binary classification
 'eval_metric': 'AUC'
}

# Perform cross-validation
cv_data = cv(
 params=params,
 pool=train_pool,
 fold_count=5, # Number of folds for cross-validation
 shuffle=True,
 partition_random_seed=42,
 plot=True
)

# Display the cross-validation results
print(cv_data[['test-AUC-mean', 'test-Logloss-mean']].tail())

Cross-validation is crucial because it provides a more reliable estimate of a model’s performance. Here’s why it’s important:

Reduces Overfitting: By evaluating the model on multiple subsets of data, cross-validation ensures that the model generalizes well across different data distributions.

Hyperparameter Tuning: It helps in fine-tuning model parameters, as it gives insights into how changes in parameters (like iterations, depth, learning_rate) affect performance.

Stable Performance Estimation: Cross-validation provides an averaged metric (like AUC), which is more stable and less biased than a single train-test split.

Understanding CatBoost Performance: Cross-Validation Metrics for Student Engagement Prediction

This chart showcases the performance of a CatBoost model over 100 iterations, using cross-validation on a student engagement dataset. The plot tracks two essential metrics — Test AUC and Test Logloss — to illustrate how well the model learns and stabilizes in its predictions.

Key Metrics Explained

1. Test AUC (Area Under the Curve) — Blue Line
   The AUC starts at a lower value and quickly rises to nearly 1.0 within the first few iterations, maintaining this high level throughout. An AUC close to 1.0 signifies excellent classification capability, meaning the model can effectively distinguish between engaged and disengaged students. The early stabilization of AUC highlights CatBoost’s strength in swiftly achieving high accuracy.
2. Test Logloss — Orange Line
   Logloss, a measure of classification error, begins relatively high and rapidly declines to near-zero levels. Lower Loss values indicate more confident and accurate predictions. The quick descent of Logloss within the initial iterations shows how efficiently CatBoost learns and optimizes its classification for student engagement.

Key Takeaways

• Fast Convergence: Both AUC and Logloss converge rapidly, indicating that CatBoost reaches optimal performance with minimal training, a crucial advantage for real-time applications.
• Consistent Accuracy: After the first few iterations, both metrics stabilize, showcasing the model’s reliable classification power for student engagement prediction.

4. Analyze Feature Importance

Analyzing feature importance helps identify which features contribute the most to predicting student engagement.

# Extracting feature names and their importances
feature_importances = model.get_feature_importance(prettified=True)
features = feature_importances['Feature Id']
importances = feature_importances['Importances']

# Plotting
plt.figure(figsize=(12, 8))
sns.barplot(x=importances, y=features, palette="viridis")
plt.title("Feature Importance from CatBoost Model")
plt.xlabel("Importance")
plt.ylabel("Feature")
plt.tight_layout()
plt.show()

Here’s how ‘Current Student Status’ might be influencing ‘Completion Status’:

In analyzing student engagement, we find that the ‘Current Student Status’ variable plays a crucial role in predicting course completion. By examining the feature importance chart, it’s clear that this factor has a significant impact on whether students complete their courses. Here’s a closer look at how different student statuses might influence completion rates:

Graduate Program Students

Graduate students make up the largest group in this dataset, with 4,389 entries. Despite their high enrollment numbers, many fall into the non-completion category (indicated by a ‘Completion Status’ of 0). This could be attributed to their busy schedules, balancing advanced studies, research, and possibly even professional commitments, which may limit their time to finish courses.

Undergraduate Students

With 1,550 entries, undergraduate students form the second largest group. Like graduate students, they also face academic demands, which may explain the higher non-completion rates. However, they generally have a slightly more flexible schedule, which might provide better opportunities for course completion compared to graduates.

Not in Education

This group, consisting of 591 individuals, includes people who are not currently enrolled in any formal education. While they might not experience academic pressures, work or personal commitments could still impact their ability to complete courses. Their completion behavior may vary depending on their motivation and availability, suggesting they could benefit from flexible, self-paced course options.

High School Students

With only 304 high school students in the dataset, this group has a smaller impact on overall completion rates. High schoolers might face additional academic challenges or lack the commitment to finish online courses. Their completion likelihood could depend on factors like academic workload, parental support, or the relevance of the course content to their current studies.

Key Takeaways: Why ‘Current Student Status’ Matters

The ‘Current Student Status’ feature provides valuable insights into the likelihood of course completion across different educational backgrounds. Here’s what we can infer:

• Graduate and Undergraduate Students: These groups are at higher risk for non-completion due to academic obligations. Support strategies, such as reminders, flexible deadlines, or even small incentives, could help improve their completion rates.
• Individuals Not in Education: For those outside formal education, course completion might depend on personal motivation. Offering flexible schedules or self-paced options could encourage consistent engagement.
• High School Students: While a smaller group, high schoolers could benefit from course structures that are more aligned with their school routines or guidance to help them stay motivated.

Understanding the nuances of ‘Current Student Status’ can help design targeted interventions to boost completion rates. Educational platforms can enhance student engagement and success by tailoring support for each group.

Let's Train the Model on Full Training Data

After cross-validation, train the final model on the entire training set for testing on the hold-out test set.

# Initialize the final CatBoost model
final_model = CatBoostClassifier(**params)

# Initialize the final CatBoost model
final_model = CatBoostClassifier(**params)

# Train the model on the full training data
final_model.fit(train_pool)

# Create test pool
test_pool = Pool(data=X_test, label=y_test, cat_features=cat_features)

# Predict on test data
y_pred = final_model.predict(test_pool)
y_pred_proba = final_model.predict_proba(test_pool)[:, 1]

# Evaluate performance on the test set
from sklearn.metrics import accuracy_score, roc_auc_score

print("Test Set Accuracy:", accuracy_score(y_test, y_pred))
print("Test Set AUC:", roc_auc_score(y_test, y_pred_proba))

0: total: 5.19ms remaining: 0us
Test Set Accuracy: 0.9963423555230432
Test Set AUC: 0.4246696035242291

Model Evaluation

ROC Curve

The ROC curve plots the true positive rate against the false positive rate. An ideal model’s curve would hug the top-left corner, while a random model’s curve lies along the diagonal line.

The ROC curve is close to the diagonal line, further emphasizing the poor predictive ability of the model. The AUC score (area under this curve) of 0.42 reinforces the lack of predictive power.

Confusion Matrix

The confusion matrix shows the counts of true positives, true negatives, false positives, and false negatives. Here, we observe:

• True Negatives (Not Completed, Not Completed): 1362
• False Negatives (Completed, Not Completed): 5
• True Positives (Completed, Completed): 0
• False Positives (Not Completed, Completed): 0

The model predicts all cases as “Not Completed,” with no cases identified as “Completed.” This one-sided prediction suggests that the model is biased towards the majority class, possibly due to class imbalance. Since the model failed to identify any true positives, it is not learning patterns associated with students who complete the program.

AUC Bar Plot

The AUC (Area Under the Curve) score on the test set is approximately 0.42. This is a key metric for evaluating the model’s ability to distinguish between classes (in this case, completed vs. not completed).

An AUC score of 0.5 represents a random model with no predictive power. A score of 0.42 is below the random threshold, indicating that the model is performing poorly on the test set. It is struggling to differentiate between students who will complete the program and those who won’t.

Final Thoughts

The results illustrate a clear difference in model performance with and without cross-validation. When cross-validation is applied, the model achieves a higher Test AUC score, indicating a stronger predictive ability and a more robust model. The ROC Curve and confusion matrix further show that the model without cross-validation struggles, as evidenced by the lower AUC score and imbalanced predictions.

Cross-validation has proven beneficial, ensuring that the model generalizes better on unseen data. For applications like student engagement prediction, where accuracy is essential for providing targeted support, cross-validation enhances reliability and prevents overfitting.

Thank you for taking the time to explore this analysis! I’d love to hear your thoughts — please feel free to share your views in the comments. Do you find CatBoost with cross-validation more effective for similar tasks? Let’s discuss it!

Join thousands of data leaders on the AI newsletter. Join over 80,000 subscribers and keep up to date with the latest developments in AI. From research to projects and ideas. If you are building an AI startup, an AI-related product, or a service, we invite you to consider becoming a sponsor.

Published via Towards AI