Name: Towards AI Legal Name: Towards AI, Inc. Description: Towards AI is the world's leading artificial intelligence (AI) and technology publication. Read by thought-leaders and decision-makers around the world. Phone Number: +1-650-246-9381 Email: [email protected]
228 Park Avenue South New York, NY 10003 United States
Website: Publisher: https://towardsai.net/#publisher Diversity Policy: https://towardsai.net/about Ethics Policy: https://towardsai.net/about Masthead: https://towardsai.net/about
Name: Towards AI Legal Name: Towards AI, Inc. Description: Towards AI is the world's leading artificial intelligence (AI) and technology publication. Founders: Roberto Iriondo, , Job Title: Co-founder and Advisor Works for: Towards AI, Inc. Follow Roberto: X, LinkedIn, GitHub, Google Scholar, Towards AI Profile, Medium, ML@CMU, FreeCodeCamp, Crunchbase, Bloomberg, Roberto Iriondo, Generative AI Lab, Generative AI Lab Denis Piffaretti, Job Title: Co-founder Works for: Towards AI, Inc. Louie Peters, Job Title: Co-founder Works for: Towards AI, Inc. Louis-François Bouchard, Job Title: Co-founder Works for: Towards AI, Inc. Cover:
Towards AI Cover
Logo:
Towards AI Logo
Areas Served: Worldwide Alternate Name: Towards AI, Inc. Alternate Name: Towards AI Co. Alternate Name: towards ai Alternate Name: towardsai Alternate Name: towards.ai Alternate Name: tai Alternate Name: toward ai Alternate Name: toward.ai Alternate Name: Towards AI, Inc. Alternate Name: towardsai.net Alternate Name: pub.towardsai.net
5 stars – based on 497 reviews

Frequently Used, Contextual References

TODO: Remember to copy unique IDs whenever it needs used. i.e., URL: 304b2e42315e

Resources

Take our 85+ lesson From Beginner to Advanced LLM Developer Certification: From choosing a project to deploying a working product this is the most comprehensive and practical LLM course out there!

Publication

Evaluating LLM Summaries using Embedding Distance with LangSmith.
Latest   Machine Learning

Evaluating LLM Summaries using Embedding Distance with LangSmith.

Author(s): Pere Martra

Originally published on Towards AI.

LangSmith is the new tool from LangChain for tracing and evaluating models. In this article, we will explore how to use it to assist in assessing the quality of summaries produced by two open-source models.

This article is part of a free course about Large Language Models available on GitHub.

The metrics that we have been using up to now with more traditional Machine Learning Models and tasks, such as Accuracy, F1 Score or Recall, do not help us to evaluate the results of generative models. It’s quite challenging to assess the quality of generated text because there isn’t a single metric capable of telling us if the outcome meets expectations.

The variables for evaluating a text are multiple: Does it adhere to the truth? Does it exhibit any bias? Is the tone used appropriate? These are just a few aspects to analyze.

There are also other metrics, such as BLEU or ROUGE, that allow for evaluating the quality of a text regarding a specific task. BLEU is typically used for measuring the quality of translations, and ROUGE for measuring the quality of summaries.

However, even these metrics are becoming relatively outdated. In this article, I will use Langsmith, the new tool from LangChain, specifically designed for tracing and evaluating the results of Large Language Models (LLMs), to obtain the embedding distance between the summaries generated and a reference summary.

To calculate the distance between the embeddings, Langsmith uses the… Read the full blog for free on Medium.

Join thousands of data leaders on the AI newsletter. Join over 80,000 subscribers and keep up to date with the latest developments in AI. From research to projects and ideas. If you are building an AI startup, an AI-related product, or a service, we invite you to consider becoming aΒ sponsor.

Published via Towards AI

Feedback ↓