Data4ML Preparation Guidelines (Beyond The Basics)

Author(s): Houssem Ben Braiek

Originally published on Towards AI.

Data preparation isn’t just a part of the ML engineering process — it’s the heart of it.

This member-only story is on us. Upgrade to access all of Medium.

Photo by Myriam Jessier on Unsplash

To set the stage, let’s examine the nuances between research-phase data and production-phase data.

Table: Research Phase vs Production Phase Datasets

The contrast highlights the “production data” we’ll call “data” in this post. Data is a key differentiator in ML projects (more on this in my blog post below).

We don’t have better algorithms; we just have more data. — Peter Norvig, The Unreasonable Effectiveness of Data.

medium.com

Here, I’ll focus on preparing it to achieve the quality required for success. This post dives into key steps for preparing data to build real-world ML systems. Each phase is loaded with practical tips to keep your process streamlined and effective.

Data ingestion ensures that all relevant data is aggregated, documented, and traceable. It involves the following core operations:

1. Connecting to Data: Data may be scattered across formats, sources, and frequencies.

2. Reading Data: Aggregating all sources into a single combined dataset.

3. Writing Output: Centralizing data into a structure, like a delta table.

This step, often done with data engineers, ensures a reproducible data snapshot from sources like production databases or APIs.

With a pipeline and incremental snapshots, metadata documentation is essential to track:

Data Source… Read the full blog for free on Medium.

Join thousands of data leaders on the AI newsletter. Join over 80,000 subscribers and keep up to date with the latest developments in AI. From research to projects and ideas. If you are building an AI startup, an AI-related product, or a service, we invite you to consider becoming a sponsor.

Published via Towards AI