Author(s): Vladimir Artus

Originally published on Towards AI.

Introduction

In the sterile vacuum of a Jupyter notebook, building AI feels like a clean, linear process. You collect data, you train a model, you wrap it in an API, and you watch the metrics turn green. It’s a beautiful mental model that works perfectly — until your code leaves the data center and hits the mud.

The first time our production model failed, the code was perfect. The metrics were stellar. The deployment was a success. The problem wasn’t technical; it was that someone had cleaned the barn and slightly bumped the camera. That single, unlogged physical nudge quietly dismantled weeks of assumptions we didn’t even know we had.

When your “production environment” involves barns, on-premise servers, and cameras covered in a thick layer of dust, you realize very quickly that applied AI isn’t about machine learning. It’s about designing systems that can survive reality.

And reality is messy.

The SaaS Mental Model (and Why It Breaks)

Most software engineers implicitly think in SaaS terms. We assume stable infrastructure, reliable fiber, predictable inputs, and instant access for debugging. This model assumes that the environment is controlled, or at least cooperative.

Applied AI quietly violates this assumption.

In a lab or a notebook, everything is measurable. Camera positions are сalculated down to the centimeter. Lighting is assumed to be “good enough.” Angles are fixed. Distances are known. Inputs behave.

But production does not live in a lab.

In applied AI, “calculated” almost always becomes “estimated” — and nobody tells you when the estimate stopped being valid.

Your data is generated outside your system. Your sensors live in hostile conditions. Cameras get bumped, covered in dust, or slightly rotated. Lighting changes with seasons, weather, and time of day. Power and connectivity are intermittent. Nothing is perfectly aligned anymore — and no one tells you when it stops being so.

Your users are not clicking buttons — they are part of a physical process that existed long before your code.

The moment your model’s behavior depends on where and how it is deployed, the SaaS mental model starts leaking. Slowly at first. Then catastrophically.

What looked precise on paper becomes probabilistic in reality.
What was “calculated” becomes “estimated.” And what was assumed stable turns out to be fragile. This is not a modeling problem. It is a systems problem.

Applied AI Lives Outside Your Data Center

Our models don’t live in notebooks. They live behind cameras installed in barns. That introduces problems no dataset prepares you for: steam, dust, shadows, and physical obstructions. In our case, a model that passed offline validation started drifting simply because the morning sun began hitting the barn at a lower angle in winter.

Reality is a hostile data generator.

A model can quietly degrade when someone rotates a camera during cleaning, or when a curious cow decides to investigate the new hardware and licks the lens.

This is not an edge case. It is the default.

Data Collection Is a Physical Process

Data collection involves traveling to farms, talking to people who do not speak the language of ML, explaining why camera placement matters, compromising between what is mathematically ideal and what is physically possible.

You don’t just get data in applied AI.
You negotiate it.

And every negotiation introduces technical debt long before a single line of code is written. Every time you hear the sentence, “Let’s just move this camera a bit,” a hidden debt is born.

The Invisible Infrastructure Gap

In the city, we take “connectivity” for granted, like oxygen. We assume there’s always a sysadmin nearby or at least a stable 5G signal.

On a farm, you realize how spoiled we’ve become.

You quickly learn that there is no “IT department” to call when a router freezes. Your “on-site support” is a farmer who is busy with a thousand more urgent tasks than your camera’s IP address. And the internet? It’s not just slow; it’s often anemic. You might find a Wi-Fi signal, but it’s barely enough to send a text, let alone a high-res video stream for inference.

Before this project, I never had to think about how to compress data to the absolute limit just to get a heartbeat from a remote server, or how to design a system that can stay “sane” while being offline for three days straight. You start realizing that your cloud-native architecture is a house of cards when the only thing connecting it to the world is a weathered copper wire or a flaky satellite link. You don’t just optimize for speed anymore; you optimize for isolation.

Infrastructure Has Physical Weight

Infrastructure in the field is a literal machine humming in a corner. It’s a router that reboots for no reason. It’s a network that goes dark because of a thunderstorm. Cloud abstractions vanish when the internet goes down. You stop caring about millisecond latency and start worrying about local buffering. You stop dreaming of real-time streaming and start building for delayed consistency.

In many environments, latency is not your biggest enemy — connectivity is. This is why applied AI systems evolve toward manual recovery paths and local storage. Not because it’s elegant, but because it works when the nearest technician is a three-hour drive away.

In many environments, latency is not your biggest enemy.
Connectivity is.

This is why many applied AI systems quietly evolve toward:
• local buffering instead of streaming
• delayed consistency instead of real-time guarantees
• manual recovery paths instead of fully automated pipelines

Not because it’s elegant — but because it works.

Surviving the “Cleaning Day”

You can train the strongest model in the world, but its performance is a joint function of hardware quality and sensor placement. Chasing state-of-the-art metrics matters less than robustness.

On a farm, “environmental variability” isn’t just a term. It’s the high-pressure hose used to wash the floors, sending a spray of water and grime directly onto your sensors. It’s the curious nose of an animal or the dust kicked up by machinery. A slightly worse model that fails safely beats a brilliant one that collapses the moment reality gets a bit too “real.”

Debugging with Your Own Eyes

Debugging a web service means checking logs and metrics. Debugging applied AI often means receiving a vague message: “Something doesn’t work.” You have no remote access. You realize the only way to understand the issue is to physically go there. Sometimes the most reliable monitoring tool is not Prometheus; it is your own eyes, standing in the mud, seeing exactly what the camera sees.

Sometimes the most reliable monitoring tool is not Prometheus. It is your own eyes.

Applied AI Is a Team Sport (Beyond Engineers)

Applied AI is a team sport played with people who have been doing their jobs for decades. They don’t care about your F1 score; they care if the tool helps them. Sometimes the most important “feature” isn’t a new layer in your neural net, but a simpler way for a technician to reboot the system.

It requires humility to design around existing workflows. I’ve found myself standing at the gates of a closed university, or in a freezing barn, waiting for a domain expert who is late because a cow was calving. Reality doesn’t care about your sprint cycle or your Google Calendar.

Survival as a Design Goal

Once you stop fighting reality and start accepting it, your architecture changes. You move toward offline-first assumptions. You build conservative defaults. You prioritize simple, observable failure modes.

The goal shifts from elegance to survival.

Applied AI isn’t harder because the math is complex. It’s harder because the world is messy, unlogged, and unpredictable. But there’s a secret we’ve learned at Axis Core Tech: once you design a system that can survive the dust, the shadows, and the human chaos of a farm — most of those “impossible” ML problems suddenly start looking very easy.

Join thousands of data leaders on the AI newsletter. Join over 80,000 subscribers and keep up to date with the latest developments in AI. From research to projects and ideas. If you are building an AI startup, an AI-related product, or a service, we invite you to consider becoming a sponsor.

Published via Towards AI