Author(s): Alexandre Warembourg

Originally published on Towards AI.

Let’s move beyond static encoding methods and explore dynamic, meaningful techniques for high-cardinality categorical variables.
source: Image generated by Dall-E from Author prompt

The categorical data type is data divided by several modalities; let’s take the color feature with the value (blue, red, green). This value is the modality of the variable, and we can not feed it to a machine learning algorithm in this state; we need to map this value in a numerical way that the machine learning algorithm can understand.

There are several techniques to map these modalities to numerical values, but among categorical data, there is one type that’s very difficult to deal with: the high cardinality categorical characteristic.

For example, in retail, we often encounter high cardinality categorical variables such as

– Brands (e.g., Nutella, Ferrero, Twix)- Product nomenclatures (product families, segments, etc.)- Product attributes (flavors such as peach, apple, ..etc.)- Territories, departments

The challenge lies in their huge cardinality, often spanning hundreds of different categories. So, how do you map these hundreds of different categories in a meaningful way? And how do we go beyond that and be time-aware when mapping those values? These are the two questions we will answer in this article.

For testing purposes, I will use a subset of the M5 dataset to illustrate the implementation and results of each method… Read the full blog for free on Medium.

Join thousands of data leaders on the AI newsletter. Join over 80,000 subscribers and keep up to date with the latest developments in AI. From research to projects and ideas. If you are building an AI startup, an AI-related product, or a service, we invite you to consider becoming a sponsor.

Published via Towards AI