Feature Selection Unlocked: Exploring Filter, Wrapper, and Embedding Techniques

Author(s): MD TAHSEEN EQUBAL

Originally published on Towards AI.

Filter Method, Wrapper Method, and Embedding Method

Feature Selection:

Feature Selection is the process of identifying and selecting the most important features (variables, predictors) from your dataset that contribute the most to predicting the target variable. The goal is to improve model performance by reducing overfitting, enhancing accuracy, and speeding up computation.

Choosing the important/relevant Feature for ML Algorithm.

Why Feature Selection is Important:

1. Improves Model Performance: By removing irrelevant or redundant features, the model becomes more accurate and efficient.
2. Reduces Overfitting: Fewer features reduce the risk of the model learning noise instead of actual patterns.
3. Speeds Up Computation: With fewer features, training the model requires less time and computational resources.
4. Simplifies Models: A model with fewer features is easier to interpret and understand.
5. Enhances Generalization: Helps the model perform better on unseen data by focusing on the most informative features.

Supervised Feature Selection Method:

Overview

For each Method, I will follow these steps in the article:

1. Define
2. Visual Representation
3. Advantages
4. Disadvantages
5. Example
6. Python Implementation and Treatment

******************Filter Method*****************

Definition:

The Filter Method selects features based on statistical measures before training the model. It evaluates the relevance of features independently of the model.

Block Diagram:

Advantages:

• Fast and computationally inexpensive.
• Works well with large datasets.
• Model-independent.

Disadvantages:

• Does not consider feature interactions.
• It might miss some useful feature combinations.

In this section, we will look at four common techniques used in the Filter Method:

1. Duplicate Columns

2. Zero or Low Variance Features,

3. Relationship

4. Mutual Information.

1. Duplicate Columns

Definition:

Duplicate columns are features that have the same values across all rows. Removing them helps reduce redundancy in the dataset without losing valuable information.

Visual Representation:

Insert a table showing two identical columns before and after removal.

In this example, Column A and Column B are duplicates.

Advantages:

• Reduces redundancy.
• Simplifies the model without loss of information.
• Speeds up model training.

Disadvantages:

• Only remove exact duplicates; near-duplicates may remain.
• Cannot identify non-obvious redundant features.

Example:

Removing identical columns from the dataset to ensure each feature provides unique information.

Python Implementation and Treatment:

# Example of removing duplicate columns
import pandas as pd

# Sample DataFrame
df = pd.DataFrame({
 'A': [1, 2, 3],
 'B': [1, 2, 3],
 'C': [5, 6, 7]
})

# Remove duplicate columns
df = df.loc[:, ~df.T.duplicated()]
print(df)

2. Zero or Low Variance Feature

Definition:

Features with zero or low variance have little to no variation in their values across the dataset. These features are often irrelevant for model building as they do not provide useful information for prediction.

Visual Representation:

Insert a table showing a column with zero variance (same value in every row).

Here, Column X has zero variance, meaning all values are the same.

Advantages:

• Helps remove irrelevant features.
• Improves model performance by focusing on more meaningful features.

Disadvantages:

• May not detect features that are useful for specific cases despite low variance.
• Works well only for numerical data.

Example:

Identifying and removing columns where values do not change significantly across observations.

Python Implementation and Treatment:

# Example of removing zero or low variance features
from sklearn.feature_selection import VarianceThreshold

# Sample DataFrame
data = [[0, 1, 0], [0, 1, 1], [0, 1, 1]]
selector = VarianceThreshold(threshold=0.1)
X_new = selector.fit_transform(data)
print(X_new)

3. Relationship (Correlation)

Definition:

Correlation measures the relationship between two numerical features. Highly correlated features (positive or negative) provide similar information, and one can often be removed without losing predictive power.

Visual Representation:

Insert a heatmap of correlations between multiple features.

Here, Feature 1 and Feature 2 are highly correlated (0.9), meaning they provide similar information.

Advantages:

• Reduces multicollinearity, which can negatively affect model performance.
• Simplifies the model by removing redundant features.

Disadvantages:

• Only works for numerical features.
• Might overlook complex relationships that aren’t linear.

Example:

Removing one of two features that have a high positive correlation (e.g., > 0.8).

Python Implementation and Treatment:

# Example of removing highly correlated features
import pandas as pd

# Sample DataFrame
df = pd.DataFrame({
 'Feature 1': [1, 2, 3, 4],
 'Feature 2': [1.1, 2.1, 3.1, 4.1],
 'Feature 3': [10, 20, 30, 40]
})
# Correlation matrix
corr_matrix = df.corr().abs()
# Remove highly correlated features
upper = corr_matrix.where(pd.np.triu(pd.np.ones(corr_matrix.shape), k=1).astype(bool))
to_drop = [column for column in upper.columns if any(upper[column] > 0.8)]
df_new = df.drop(df[to_drop], axis=1)
print(df_new)

4. Mutual Information

Definition:

Mutual Information measures the dependency between two variables. It quantifies how much knowing one feature reduces uncertainty about another. This is useful for identifying relevant features even if they have non-linear relationships.

Visual Representation:

Insert a table showing how mutual information evaluates the relationship between target and features.

Mutual information helps determine the feature that provides the most information about the target variable.

Advantages:

• Works with non-linear relationships.
• Can be applied to both categorical and continuous data.

Disadvantages:

• Computationally expensive for large datasets.
• Does not provide insight into the direction of the relationship.

Example:

Identifying features that have high mutual information with the target variable, even if the relationship is complex.

Python Implementation:

# Example of using mutual information to select features
from sklearn.feature_selection import mutual_info_classif
from sklearn.datasets import load_iris

# Load sample data
data = load_iris()
X = data.data
y = data.target
# Calculate mutual information
mi = mutual_info_classif(X, y)
print(mi)

****************Wrapper Method****************

Definition:

The Wrapper Method tries different combinations of features and selects the best subset based on model performance. It uses algorithms like forward selection, backward elimination, or exhaustive search.

Block Diagram:

Advantages:

• Considers feature interaction.
• Tends to give better results since it evaluates combinations.

Disadvantages:

• Computationally expensive.
• Slow with large datasets.

In this section, we will look at four common techniques used in the Wrapper Method:

1. Best Subset.

2. Forward Selection.

3. Backward Elimination.

1. Best Subset:

Definition:

Best Subset Selection is a method where all possible combinations of features are evaluated, and the combination that yields the best performance is selected. This method ensures that the optimal subset is chosen, but it can be very computationally expensive.

Visual Representation:

Insert a diagram showing different subsets of features being evaluated for a model.

In this example, Subset 3 (Feature A, B, C) yields the best performance.

Advantages:

• Guarantees finding the best combination of features.
• Results in a highly optimized model with the best-performing features.

Disadvantages:

• Very computationally expensive, especially for large datasets with many features.
• Not feasible for high-dimensional datasets due to the large number of possible combinations.

Example:

You have a dataset with 5 features and you want to find the combination of features that results in the highest accuracy. Best Subset Selection evaluates all possible subsets and selects the one that maximizes performance.

Python Implementation:

# # Best Subset Selection requires manually testing multiple models
import itertools
from sklearn.linear_model import LinearRegression
from sklearn.model_selection import train_test_split
from sklearn.metrics import mean_squared_error

# Sample DataFrame
import pandas as pd
import numpy as np

df = pd.DataFrame({
 'Feature A': np.random.rand(100),
 'Feature B': np.random.rand(100),
 'Feature C': np.random.rand(100),
 'Feature D': np.random.rand(100),
 'Target': np.random.rand(100)
})

X = df.drop('Target', axis=1)
y = df['Target']
features = list(X.columns)
best_score = float('inf')
best_subset = None

for L in range(1, len(features)+1):
 for subset in itertools.combinations(features, L):
 X_subset = X[list(subset)]
 X_train, X_test, y_train, y_test = train_test_split(X_subset, y, test_size=0.2, random_state=42)
 model = LinearRegression()
 model.fit(X_train, y_train)
 y_pred = model.predict(X_test)
 mse = mean_squared_error(y_test, y_pred)
 
 if mse < best_score:
 best_score = mse
 best_subset = subset

print(f'Best Subset: {best_subset}, Best Score: {best_score}')

2. Forward Selection

Definition:

Forward Selection is a stepwise method where features are added one by one. At each step, the feature that improves the model’s performance the most is added to the subset. This process continues until adding more features does not significantly improve performance.

Visual Representation:

Insert a diagram showing features being added one by one and evaluating the performance after each addition.

In this example, the model performance improves as features are added step-by-step.

Advantages:

• Easier to implement and less computationally expensive than Best Subset Selection.
• You can stop early when adding features no longer improves performance.
• Suitable for smaller datasets.

Disadvantages:

• Might miss the best subset as it only evaluates features added at each step.
• Can be suboptimal compared to Best Subset Selection.

Example:

You start with no features and add features one by one, selecting the feature that results in the highest improvement in model accuracy after each addition.

Python Implementation:

# Example of Forward Selection using a custom function
from sklearn.linear_model import LinearRegression
from sklearn.metrics import mean_squared_error
from sklearn.model_selection import train_test_split

# Forward Selection
def forward_selection(X, y):
 remaining_features = list(X.columns)
 selected_features = []
 best_score = float('inf')
 
 while remaining_features:
 scores = []
 for feature in remaining_features:
 current_features = selected_features + [feature]
 X_train, X_test, y_train, y_test = train_test_split(X[current_features], y, test_size=0.2, random_state=42)
 model = LinearRegression()
 model.fit(X_train, y_train)
 y_pred = model.predict(X_test)
 mse = mean_squared_error(y_test, y_pred)
 scores.append((mse, feature))
 
 scores.sort()
 best_score, best_feature = scores[0]
 selected_features.append(best_feature)
 remaining_features.remove(best_feature)
 print(f'Selected Feature: {best_feature}, Best Score: {best_score}')
 
 return selected_features

X = df.drop('Target', axis=1)
y = df['Target']
best_features = forward_selection(X, y)
print(f'Final Selected Features: {best_features}')

3. Backward Elimination

Definition:

Backward Elimination starts with all features and removes the least significant feature at each step. The feature that has the least contribution to improving model performance is eliminated, and this continues until only the most important features remain.

Visual Representation:

Insert a diagram showing features being removed one by one, starting with all features.

In this example, removing features improves performance, and the process stops when removing more features starts to decrease accuracy.

Advantages:

• Efficient in reducing features from a large set.
• Works well when there are many irrelevant or insignificant features.

Disadvantages:

• May remove features that, when combined with others, are important.
• Computationally expensive for large datasets with many features.

Example:

You start with all available features and iteratively remove the least important ones based on a performance metric, such as p-value or accuracy.

Python Implementation:

# Example of Backward Elimination using statsmodels
import statsmodels.api as sm
import pandas as pd

# Sample DataFrame
X = df[['Feature A', 'Feature B', 'Feature C', 'Feature D']]
y = df['Target']

# Adding a constant column for intercept
X = sm.add_constant(X)

# Fit the model
model = sm.OLS(y, X).fit()

# Backward Elimination process
while True:
 p_values = model.pvalues
 max_p_value = p_values.max()
 if max_p_value > 0.05: # Threshold for significance
 excluded_feature = p_values.idxmax()
 X = X.drop(columns=[excluded_feature])
 model = sm.OLS(y, X).fit()
 print(f'Removed: {excluded_feature}')
 else:
 break

print(model.summary())

**************Embedding Method***************

Definition:

The Embedding Method selects features as part of the model training process. Tree-based methods like Random Forest or Decision Trees are popular for this technique.

Block Diagram:

Advantages:

• Automatically identifies important features during model training.
• Considers interactions between features.

Disadvantages:

• Model-specific, so the selection may not generalize to other models.
• Can be harder to interpret than Filter Methods.

In this section, we will look at four common techniques used in the Embedded Method:

1. Tree Based

2. Linear Based

1. Tree-Based Feature Selection

Definition:

Tree-Based Methods, like decision trees, random forests, and gradient boosting machines, can automatically rank the importance of features. They work by splitting the dataset on the most relevant features, making them highly effective in selecting important features.

Visual Representation:

Insert a tree diagram showing how the dataset is split on the most important features.

In this example, Feature A is the most important, followed by Feature B.

Advantages:

• Automatically selects important features.
• Works well with non-linear data and interactions between features.
• No need for manual feature selection.

Disadvantages:

• May overfit on noisy datasets with irrelevant features.
• Importance scores can vary depending on the model used (e.g., decision trees vs. random forests).

Example:

You have a dataset with multiple features, and you use a Random Forest model to rank the importance of each feature. The model ranks the features based on how often they were used for splits in the decision trees.

Python Implementation:

from sklearn.ensemble import RandomForestRegressor
import pandas as pd
import numpy as np

# Sample DataFrame
df = pd.DataFrame({
 'Feature A': np.random.rand(100),
 'Feature B': np.random.rand(100),
 'Feature C': np.random.rand(100),
 'Feature D': np.random.rand(100),
 'Target': np.random.rand(100)
})

X = df.drop('Target', axis=1)
y = df['Target']

# Train a Random Forest model
model = RandomForestRegressor(n_estimators=100)
model.fit(X, y)

# Get feature importance
feature_importances = model.feature_importances_

# Print feature importance
for feature, importance in zip(X.columns, feature_importances):
 print(f'{feature}: {importance}')

2. Linear Models with Regularization (Lasso, Ridge)

Definition:

Linear Models with Regularization, such as Lasso and Ridge regression, use penalties to shrink the coefficients of less important features toward zero. Lasso (L1 regularization) can completely eliminate some features, while Ridge (L2 regularization) reduces the coefficients of less important features without removing them.

• Lasso (L1 Regularization): Adds a penalty equal to the absolute value of the coefficients, forcing some of them to become exactly zero, thus performing feature selection.
• Ridge (L2 Regularization): Adds a penalty equal to the square of the coefficients, shrinking them but not eliminating any features completely.

Visual Representation:

Insert a bar chart showing Lasso regression shrinking some coefficients to zero.

• Feature A and Feature C have non-zero coefficients, so they are retained in the model.
• Feature B and Feature D have zero coefficients due to Lasso regression, meaning they are effectively removed from the model.

Advantages:

• Helps in avoiding overfitting by reducing model complexity.
• Lasso regression automatically eliminates irrelevant features, leading to a simpler model.
• Ridge regression prevents overfitting by shrinking coefficients but still uses all features.

Disadvantages:

• Lasso can be unstable when the number of features is larger than the number of observations.
• Ridge regression does not perform feature elimination (it only shrinks the coefficients).
• May not capture complex feature interactions like tree-based methods.

Example:

You use Lasso regression to fit a linear model, and it automatically eliminates irrelevant features by shrinking their coefficients to zero. This results in a simpler model with only the most important features retained.

Python Implementation:

from sklearn.linear_model import Lasso, Ridge
from sklearn.model_selection import train_test_split
from sklearn.metrics import mean_squared_error

# Sample DataFrame
X = df.drop('Target', axis=1)
y = df['Target']

# Split the data into training and testing sets
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)

# Lasso Regression
lasso = Lasso(alpha=0.1) # Alpha controls the regularization strength
lasso.fit(X_train, y_train)

# Ridge Regression
ridge = Ridge(alpha=1.0)
ridge.fit(X_train, y_train)

# Coefficients from Lasso and Ridge
print("Lasso Coefficients:", lasso.coef_)
print("Ridge Coefficients:", ridge.coef_)

# Evaluate models
y_pred_lasso = lasso.predict(X_test)
y_pred_ridge = ridge.predict(X_test)
print("Lasso MSE:", mean_squared_error(y_test, y_pred_lasso))
print("Ridge MSE:", mean_squared_error(y_test, y_pred_ridge))

Conclusion:

1 Filter Method

2. Wrapper Method

3. Embedded Method

— — — — — — — — — -Thank You! — — — — — — — — —

Thank you for taking the time to read my article. I hope you found it useful and informative. Your support means a lot, and I appreciate you joining me on this journey of exploration and learning. If you have any questions or feedback, feel free to reach out!

— — — — — — — — — Contact — — — — — — — — — — —

Linkdein –https://www.linkedin.com/in/md-tahseen-equbal-/
Github –https://github.com/Md-Tahseen-Equbal
Kaggle- https://www.kaggle.com/mdtahseenequbal

Join thousands of data leaders on the AI newsletter. Join over 80,000 subscribers and keep up to date with the latest developments in AI. From research to projects and ideas. If you are building an AI startup, an AI-related product, or a service, we invite you to consider becoming a sponsor.

Published via Towards AI