Author(s): Lin Jia

Originally published on Towards AI.

GenAI in Experimentation: Notes from the Panel and the Sidelines at 3rd Booking.com Experimentation Conferene

🎤 The GenAI & Experimentation panel at Booking.com’s 3rd Experimentation Conference covered roughly four questions in 20 minutes minutes — I was invited to join the panel discussion alongside David Gregory (Skyscanner), Marcel Toben (Zalando), and Dima Bordiugov (Delivery Hero), moderated by Gosia Popławska (Netflix).

The article frames GenAI’s role in experimentation around function and trust rather than timing: it can improve hypothesis quality before experiments, speed detection and debugging during them, and help synthesize and communicate results afterward, but should support decisions without becoming the “source of truth.” It argues that the hardest parts of a GenAI copilot roadmap are operational trust, evaluation, and whether the right problems are being solved—not just faster prototyping. It highlights key frictions, such as fluency being mistaken for correctness, and proposes guardrails that ground outputs in trusted sources, evaluate factuality and relevance, and continuously monitor production behavior. It also addresses maintaining experimentation “memory” as a continuously maintained and evaluated system, evolving experimentation KPIs to include both enablement and GenAI-system metrics, and building production evaluation pipelines tailored to specific use cases. Finally, it emphasizes that when LLMs become part of the user experience, teams must adopt ML-systems practices (monitoring, datasets, shipping gates), while trade-off-heavy judgments should remain human—so faster experimentation doesn’t erode rigor, evaluation quality, or accountability.

Read the full blog for free on Medium.

Join thousands of data leaders on the AI newsletter. Join over 80,000 subscribers and keep up to date with the latest developments in AI. From research to projects and ideas. If you are building an AI startup, an AI-related product, or a service, we invite you to consider becoming a sponsor.

Published via Towards AI