Author(s): Shivang Doshi

Originally published on Towards AI.

Introduction

In the age of AI chatbots and conversational assistants, one question often gets overlooked amid the excitement: How do we evaluate these large language models (LLMs)? You might have a state-of-the-art model powering your chatbot — say, GPT-4 or a fine-tuned LLaMA — but how do you know if it’s actually performing well? Is it giving accurate answers? Is it being helpful and not toxic? Evaluating LLMs, especially in chatbot applications, is both crucial and surprisingly tricky. In this post, we’ll dive into what LLM evaluation means, why it’s necessary (spoiler: LLMs can be unpredictable at times), the key challenges involved, and the tools/frameworks that can make your life easier. We’ll keep it casual but technical, and by the end you should have a solid grasp of how to measure what really matters in LLM-driven chatbots.

What Does “Evaluation” Mean for LLMs (and Chatbots)?

When we talk about evaluating a large language model, we mean systematically measuring its performance on tasks or criteria that we care about. Unlike traditional software where you might write unit tests with clear pass/fail conditions, LLMs deal in the ambiguity of human language. For a chatbot, evaluation could involve checking how often the model’s answers are correct or factual, how fluent and coherent its responses are, how helpful it is to users, and whether it avoids problematic outputs (like offensive or unsafe content). In essence, evaluation is about defining what “good” looks like for your model — and then testing the model to see if it meets those standards.

Notably, “good” can mean different things to different people. Ask ten teams what “best” means in an LLM, and you’ll likely get ten different answers. Some care about accuracy, others about bias; some prioritize speed and efficiency, while others demand interpretability or safety[1]. For a customer service chatbot, helpfulness and factual accuracy might be top metrics. For a creative writing AI, originality and style could matter more. The truth is, no single benchmark captures the full picture of an LLM’s performance[2]. This is why evaluation for LLMs tends to be multi-faceted — we use a mix of tests and metrics to assess various aspects of the model’s behavior.

In practical terms, evaluating an LLM often involves:

• Benchmarking on Tasks: e.g. asking the model a set of questions (trivia, math problems, coding challenges) with known answers to measure accuracy.
• User Simulation or Dialogues: e.g. having the model engage in sample conversations and then rating those for qualities like clarity, correctness, or politeness.
• Human Feedback: e.g. collecting human ratings on model outputs — was this answer good? was it harmful? — to gauge quality on subjective criteria.
• Automated Metrics: e.g. using metrics like BLEU, ROUGE, or newer ones like BERTScore for text quality, or specialized scores for factual accuracy. (Automated metrics are faster but often only loosely correlate with actual quality on complex language tasks.)

For chatbots, evaluation might also include role-playing conversations to test consistency, edge-case prompts to test robustness, and safety tests (making sure the bot refuses or handles disallowed content appropriately). In short, LLM evaluation is about probing the model’s capabilities and weaknesses in a systematic way. It’s part science (quantitative metrics and experiments) and part art (defining what matters and interpreting results).

Why Do We Need to Evaluate LLMs?

Can’t we just trust a big model to be good out-of-the-box? Definitely not! Rigorous evaluation is necessary because LLMs are notoriously unpredictable and context-dependent. Even the most advanced model might dazzle you with one answer and then produce a total blunder for a slightly different prompt. Let’s break down a few reasons why evaluation is essential:

• Unpredictable Outputs: Large language models sometimes generate responses that are wrong, nonsensical, or inappropriate, even if they look fluent. For example, an LLM-based chatbot might usually give correct answers, but once in a while it will confidently sprout misinformation or gibberish. These failures can be rare (say 1% of the time) but they often occur in unpredictable contexts[3]. Especially in high-stakes applications (medical advice, financial info, etc.), that 1% of bad outputs is not acceptable. We need to evaluate models extensively to catch these issues. As one expert noted, LLMs can produce excellent text 99% of the time, but also produce poor (inaccurate or unsafe) text 1% of the time — and finding those worst-case failures is critical for safe deployment[3].
• Complex, Multi-Dimensional Quality: Quality for LLMs isn’t one-dimensional. A model’s output might be factually correct but written in a convoluted way, or it might be extremely eloquent but completely incorrect. There’s also considerations like bias or offensive content. Because “good” output has many facets (accuracy, clarity, relevance, safety, etc.), we have to measure a variety of aspects. It’s not obvious how a model will trade off these factors without testing. For instance, a tweak that improves factual accuracy might accidentally make the model more verbose or even more biased — only evaluation on multiple metrics would reveal that.
• Difficulty of Human Assessment: Humans are the ultimate judges of language quality, but human evaluation is slow, costly, and subjective. If you have a chatbot, you can’t manually read through every response it might ever generate — you need systematic tests. Moreover, what one person finds “helpful,” another might not. Evaluation helps impose some consistency and objectivity. We often use benchmark datasets or standardized tests as proxies for human judgment. Without evaluation frameworks, you’d be flying blind regarding your model’s behavior.
• Model Updates and Regression Risk: LLMs (especially those provided via APIs) can evolve over time. OpenAI, for example, periodically updates models like GPT-4, and those updates can change model behavior. If you don’t continuously evaluate, you might not notice that a new version of the model has started making errors on cases it used to handle well. In fact, one challenge with closed-source models is that they are constantly evolving — an experiment run on GPT-4 in January might yield different results in July after updates[4]. Evaluation is needed to catch regressions and ensure newer model versions don’t break your application. As OpenAI’s own team puts it, “Without evals, it can be very difficult and time intensive to understand how different model versions might affect your use case.”[5]
• Safety and Alignment: There’s a big focus on making sure AI systems behave safely and align with human values. Evaluation is the tool we use to test a model’s safety — e.g. does it refuse to produce hate speech or disallowed content? Is it fair across different user demographics? These qualities aren’t guaranteed just because the model is large; they must be tested. OpenAI and Anthropic, for instance, conduct extensive safety evals (often with red-team prompts) to probe where the model might do something harmful[1][3]. For anyone deploying a chatbot widely, running these kinds of evals is a necessary due diligence step.

In summary, we evaluate LLMs because you can’t fix what you don’t measure. Good evals shine a light on a model’s blind spots and help ensure that improvements are real. As Greg Brockman of OpenAI has emphasized, creating high-quality evals is one of the most impactful things you can do when building with LLMs[5] — it’s how you gain confidence in your model’s behavior before it interacts with real users.

Key Challenges in Evaluating LLMs

Okay, so evaluation is important — but it’s also hard to do right. Why is evaluating LLMs challenging? It turns out there are several thorny issues that engineers and researchers face when trying to assess these models:

• Subjectivity in Human Judgments: A lot of what we care about (like answer helpfulness or conversational tone) is subjective. Different human evaluators might have different standards and interpretations, leading to inconsistent ratings[6]. What one person calls a “polite and useful” answer, another might find lacking. This variability makes it hard to get reliable ground truth for model quality. It’s a challenge to design evaluation criteria that are clear-cut, or to aggregate human opinions in a meaningful way.
• Cost and Scale of Human Evaluation: Relying on people to rate answers or conduct chat conversations doesn’t scale well. Obtaining reliable human feedback is time-consuming and labor-intensive, especially if you need to evaluate hundreds or thousands of prompts[7]. This is a practical bottleneck — if each model update requires a new round of human eval on 1,000 samples, that can become very costly. It pushes us to find more automated or efficient evaluation methods (like having models judge other models, or using smaller scale targeted tests).
• Limitations of Automated Metrics: On the flip side, traditional automated metrics often fall short for LLMs. Metrics like BLEU or ROUGE (borrowed from machine translation and summarization) only check surface-level overlap with reference answers. They fail to capture nuanced aspects of quality, such as whether the model’s answer is logically correct, contextually appropriate, or insightful[8]. A chatbot might word an answer differently from a reference but still be excellent — or it might match the reference closely yet miss the spirit of the question. New metrics (e.g. leveraging embedding similarity or LLM-based evaluators) are being developed, but choosing the right metric is still an art. Automated metrics are useful, but you have to be aware of what they don’t measure.
• Dynamic Model Behavior (Reproducibility): LLM outputs can vary from one run to another, especially if any randomness (temperature setting) is involved. Even more challenging, models get updated or change over time, which can make it hard to reproduce results. A research survey found that many LLM studies suffered because model details and prompt specifics weren’t fully documented, and continuous model updates meant results from a few months ago might not hold later[9][10]. As one analysis noted, closed-source models like ChatGPT can be a moving target — an experiment today might yield different outcomes next week if the provider silently improved the model[4]. This dynamism complicates evaluation: you need to pin down model versions, ensure consistent settings, and possibly re-run evals regularly to track drift.
• Wide Range of Scenarios: LLMs (and chatbots) can be used in many different contexts — from writing code, to answering medical questions, to casual chit-chat. Evaluating across such diverse scenarios is tough. A model might ace one kind of task but flub another. Historically, NLP evaluation was fragmented into narrow benchmarks (one for translation, one for question answering, etc.), which only cover slices of a model’s capabilities[11][2]. For a holistic picture, you’d need to test a broad spectrum of tasks. This is exactly what newer efforts like Stanford’s HELM benchmark emphasize — covering dozens of scenarios and multiple metrics to avoid missing the bigger picture[12]. But of course, running such comprehensive evals is a large undertaking and requires significant coordination and resources.
• Evaluating “Open-Ended” Generations: Unlike a classification model that outputs a label you can easily check, an LLM’s open-ended text output can be evaluated along many axes. There’s often no single “right” answer to compare against. For example, if a user asks a chatbot for travel advice, there are many valid helpful answers. This makes ground truth hard to define. Evaluators sometimes have to resort to relative judgments (is output A or B better for this prompt?) or to check adherence to instructions/policies rather than correctness per se. It’s a challenge to devise automated tests for these without human involvement.
• Metric Gaming and Overfitting: An important (if subtle) challenge is that if you rely on a fixed set of benchmarks, models can effectively overfit or “game” those metrics. We’ve seen cases where a model is tuned to excel on a well-known benchmark but then fails on slightly varied questions. If an LLM is trained or optimized on specific evaluation metrics, it might do well on those tests without truly being general. Models may score high on narrow benchmarks but fail to generalize to real-world scenarios[13]. This means we as evaluators have to constantly update and diversify our evals to stay ahead of the models’ learning — a bit of a cat-and-mouse situation.

In short, evaluating LLMs is harder than it looks. It requires careful thought to design meaningful tests and often a combination of approaches (human + automated) to cover all bases. As the field matures, best practices are emerging (like keeping evaluation data secret to avoid models training on them, using multiple metrics, etc.), but there’s no one-size-fits-all solution. Awareness of these challenges helps us interpret evaluation results with the proper grain of salt.

Tools and Frameworks for LLM Evaluation: OpenAI Evals, HELM, RAGAS, & More

The good news is you don’t have to build your evaluation workflow entirely from scratch. In response to the need for better LLM evaluation, a number of out-of-the-box frameworks and tools have been developed. Here we’ll look at a few prominent ones — OpenAI Evals, HELM, and RAGAS — discussing what they offer and when you might use each. Each tool has a slightly different focus, so they can even complement one another.

OpenAI Evals: Customizable Eval Harness

OpenAI Evals is a framework released by OpenAI for evaluating LLMs and LLM-powered systems. It was originally used internally to benchmark OpenAI’s models (ensuring new versions of GPT-4 didn’t break things, etc.), and now it’s open-sourced for anyone to use[14]. Think of OpenAI Evals as a kind of “pytest for LLMs” — it lets you define tests (called “evals”) that consist of prompts, expected answers or scoring logic, and then run those systematically against a model.

Key features of OpenAI Evals:

• It comes with an existing registry of evals for common benchmarks. These include things like trivia QA, coding problems, math word problems, etc. You can run these out-of-the-box to see how a given model performs on them[15].
• It provides a framework to write your own evals for the use cases you care about. For example, if your chatbot needs to excel at answering questions from a specific knowledge base, you can create a custom eval with representative questions and the correct answers. You can even keep your eval private (useful if it contains proprietary data)[16].
• It supports evaluation on multiple dimensions — you can specify different metrics (accuracy, F1, etc.) or even write custom Python logic to grade an output. If the standard metrics don’t fit, you can code an eval that, say, uses a regex or another model to judge the answer.
• OpenAI Evals integrates with the OpenAI API by default, making it easy to test models like GPT-3.5 or GPT-4. But it also has a concept of “completion functions” that can wrap other models or systems. In fact, you can plug in open-source models or even chain-of-thought reasoning via these completion functions[17]. This means you’re not limited to OpenAI’s models; you could evaluate local models too with some setup.

Using OpenAI Evals at a high level is straightforward. You typically create a YAML specification that defines your evaluation task — for instance, what data to use (questions/expected answers), what metric to compute, and which eval class to use (the framework has some built-in classes for common patterns like exact match checking). This YAML plus a dataset (often in a simple JSONL file) constitutes an “eval”. You can then run it with a one-liner CLI command such as:

oaieval gpt-3.5-turbo my_custom_eval

This would run your my_custom_eval on the GPT-3.5-turbo model and print out a result (e.g. “Accuracy: 85%”) along with a detailed log of prompts and model outputs. The design philosophy is to make it easy to spin up new tests for your specific use case[18]. As the OpenAI Evals README notes, without such evals it’s difficult to understand model changes, but with a good set of evals, you can quickly benchmark different models or versions on the things that matter to you[19].

When to use OpenAI Evals: If you are building an application with LLMs (chatbot or otherwise) and want a bespoke test suite for it, OpenAI Evals is a great choice. It’s most useful for evaluation-as-testing, where you treat your evals like regression tests — run them whenever you update the model or prompt, to ensure performance hasn’t dropped. It’s also useful for comparing models (e.g., is Model A or Model B better on my task?). Since it’s quite flexible, you would use it when you have specific success criteria in mind and possibly your own data to test on. One thing to note: running evals against the OpenAI API will incur costs (it’s essentially calling the model many times), so budget accordingly[20].

HELM: Holistic Evaluation of Language Models

HELM (Holistic Evaluation of Language Models) is more of a benchmarking platform than a tool you integrate into your codebase. Developed by Stanford’s Center for Research on Foundation Models, HELM is an effort to provide a “living benchmark” and a 360° view of LLM performance[21]. The idea behind HELM is that no single metric or scenario suffices, so it evaluates models across a broad range of tasks and metrics to paint a comprehensive picture.

Key characteristics of HELM:

• Broad Scenario Coverage: HELM evaluates language models on 42 different scenarios (as of its latest report) covering things like summarization, dialogue, question answering, coding, and more[22][23]. This wide coverage means it tries to simulate real-world use cases the models might face, from writing an email to answering trivia to generating code.
• Multiple Metrics: For each scenario, HELM measures up to 7 metrics beyond just accuracy — including things like robustness, calibration, fairness, toxicity, and efficiency[24][25]. This is important because a model might be super accurate but have a toxicity problem, for example. HELM’s multi-metric approach makes those trade-offs visible[26].
• Many Models Side-by-Side: HELM has benchmarked around 30 prominent models (both closed API models like GPT-4 and Claude, and open ones like LLaMA, Mistral, etc.) under the exact same conditions[23]. This allows apples-to-apples comparisons. If you’re curious how your model of choice stacks up against others in, say, long-form summarization or math word problems, HELM’s published results can tell you.
• Transparency and Reproducibility: The HELM project emphasizes open access: it releases prompts, completions, and evaluation code so that others can reproduce the results or extend them[27][28]. It’s meant to be a community resource, and indeed it’s continually updated (hence “living benchmark”) as new models and scenarios come on the scene.

You can interact with HELM primarily through their website or papers. For a software engineer, HELM is useful to consult when you need to understand a model’s strengths/weaknesses or to pick a model for your application. For example, if you’re building a chatbot that must be safe and unbiased, HELM’s fairness/toxicity scores could inform your choice of model (some models might have notably lower toxicity rates[24]). Or if you need multi-lingual capability, HELM’s scenarios include multilingual tests.

When to use HELM: You’d “use” HELM in a scenario where you want a broad benchmarking across models or tasks. It’s less about testing your own application (since HELM’s scenarios are predetermined), and more about getting a holistic assessment of model behavior. HELM is great for research and comparison: say you want to compare GPT-4 vs. an open-source model on a variety of dimensions — HELM data can help. If you’re a practitioner reading up, HELM’s reports can highlight pitfalls (like how models might have high accuracy but poor calibration or high variability in certain domains). In summary, turn to HELM when you need a big-picture evaluation or to decide which model to use for a given job. It’s like a leaderboard with context, going beyond just single-number scores to reveal where each model shines or struggles[29][11].

RAGAS: Evaluating Retrieval-Augmented Generation

Many chatbots and QA systems nowadays use a retrieval-augmented generation (RAG) approach — they fetch relevant documents from a knowledge base and have the LLM ground its answer on that retrieved info. Evaluating these systems brings its own special challenges, because you have to assess both the retrieval and the generation components. This is where RAGAS comes in.

RAGAS (Retrieval Augmented Generation Assessment) is a framework specifically designed to evaluate RAG pipelines[30]. It provides a suite of metrics and tools to analyze how well the retrieval+generation combo is working. In RAG setups, you typically care about things like: Did the retriever find the right information? Did the LLM actually use that info correctly (faithfully) in its answer? And is the final answer good and correct?

What RAGAS offers:

• Component-Level Metrics: RAGAS breaks down evaluation into the retriever’s performance and the generator’s performance, as well as the end-to-end result[31][32]. For example, it includes metrics like Context Relevancy/Precision (how relevant are the retrieved documents to the query) and Context Recall (did we retrieve all the information needed to answer the question)[33]. Then for the generation, it measures Faithfulness (did the model’s answer stick to the retrieved facts, or hallucinate?) and Answer Relevancy (did it actually address the user’s question fully)[34]. These metrics help pinpoint if a failure is due to retrieval or the generative part.
• Reference-Free Evaluation: One very interesting aspect of RAGAS is that it started as a reference-free evaluation framework[35]. This means it can evaluate the quality of answers without needing a human-written ground-truth answer for every question. How? RAGAS leverages LLMs themselves to judge things like faithfulness and relevancy. Essentially, the framework can use a GPT-like model under the hood to compare the answer and the retrieved docs and score how well they match. This makes it much easier to get evaluations because you don’t need an extensive labeled dataset (which is often hard to obtain for every domain). It’s a faster, cheaper way to evaluate, suitable for continuous monitoring of a system in production[36]. (Of course, one has to be mindful of potential bias when using an LLM as the judge — but research so far shows promising results in using LLMs to evaluate other LLMs’ answers[37].)
• Ease of Use and Integration: RAGAS comes as a Python library (ragas on pip) with built-in functions to compute these metrics on your data. You input a set of (question, retrieved_docs, model_answer, optional_ground_truth) and it will output scores for each metric. It’s designed to be developer-friendly to plug into your evaluation pipeline — you can integrate it to regularly test your RAG chatbot’s performance on, say, a validation set of Q&A pairs.

When to use RAGAS: If your application uses retrieval-augmented generation (like many chatbots that cite sources or do enterprise Q&A), RAGAS is a highly relevant tool. It directly helps answer: “Is my retrieval component bringing back good info?” and “Is my model properly using that info to answer questions correctly?”. Use RAGAS during development to tune your retriever (maybe you find context_recall is low — indicating you need a broader search) or to adjust your prompt so the model uses the context more faithfully. Also use it in production eval: you can continuously sample some queries and evaluate them to catch if your system’s quality is drifting. RAGAS is less needed if your bot doesn’t do retrieval (if it’s purely generative with no knowledge base, you’d focus more on general metrics), but in this era of tools + LLMs, many systems do incorporate retrieval, making RAGAS quite handy. In sum, RAGAS fills the niche of evaluation tailored to open-book QA systems, ensuring both halves of the pipeline are working in concert[38].

Other Noteworthy Mentions

Beyond the three above, there are of course many other evaluation tools out there:

• Hugging Face’s Evaluation Toolkit: The 🤗 Hugging Face ecosystem provides datasets and evaluate libraries that include a ton of standard NLP metrics (BLEU, ROUGE, F1, etc.) out-of-the-box. These are great for computing classical metrics or using community benchmarks (like SQuAD or CNN/DailyMail for summarization) to test your model. However, they’re more low-level (you have to script your evaluation) and don’t handle the human-judgment aspects; they shine when you have a labeled dataset to evaluate on.
• Langsmith (by LangChain): LangChain, a popular framework for LLM apps, has introduced evaluation capabilities (formerly in beta under “LangChain Evaluators”). These allow you to use LLMs to grade outputs or compare two models’ answers. It’s convenient if you’re already using LangChain to manage prompts and chains.
• LM Evaluation Harness: There’s an open-source tool called lm-eval-harness (by EleutherAI) which provides a framework to evaluate language models on a collection of academic benchmarks (like HELLASWAG, PIQA, etc.). It’s geared toward research and comparing model capabilities on standard tasks — useful if you want to validate an open-source model against known benchmarks.
• HELM Lite / Model Cards: Some platforms or model providers give “model cards” or mini-evaluations for their models (covering basic metrics or known limitations). While not interactive, these are good resources to glean a model’s evaluated properties if you can’t run a full eval yourself.

Each tool or framework has its sweet spot. Often, in practice, teams will use a combination: for example, run some automated metrics via Hugging Face evaluate, use OpenAI Evals for custom scenario testing, and refer to HELM for a sanity check against broader benchmarks.

Further Reading and Resources on LLM Evaluation

If you’re eager to dig deeper into this topic (there’s a lot more to explore!), here are some recommended papers and resources:

• “Holistic Evaluation of Language Models” (Liang et al., 2022) — This is the Stanford HELM paper[26], which lays out the framework for evaluating models across a broad spectrum of scenarios and metrics. It’s a fantastic read to understand why traditional narrow benchmarks fall short and how HELM addresses that. The paper also presents detailed findings comparing many models. (If not the paper, you can also check out the HELM website for interactive results.)
• OpenAI Evals GitHub Repository — The official repo for OpenAI Evals[15] contains documentation and examples. Reading the README and browsing the evals and examples folders can give you a practical sense of how to write evals. OpenAI’s own cookbook has a “Getting Started with OpenAI Evals” guide, which is a great step-by-step tutorial[18].
• RAGAS Paper and Docs — For those specifically interested in retrieval-augmented generation evaluation, check out “RAGAS: Automated Evaluation of Retrieval-Augmented Generation” (Es et al., 2025)[30]. It explains the philosophy and validation of using LLMs for reference-free evaluation. The RAGAS documentation[41] is also useful to see how to apply the library in practice. Leonie Monigatti’s Medium article “Evaluating RAG Applications with RAGAS” provides a nice walkthrough as well.
• Survey on LLM Evaluation (2024) — A recent systematic survey by Jin et al. (2024) discusses the challenges, limitations, and recommendations for evaluating LLMs[42]. It’s an academic but insightful overview of the state-of-the-art in eval methods, covering topics like reproducibility (and lack thereof), reliability of LLM-as-a-judge approaches, and the need for standardization. If you want to understand the pain points researchers are identifying, this is a solid resource.
• Blogs and Guides: There are many online articles that condense best practices. For example, Ehud Reiter’s blog post “Challenges in Evaluating LLMs” is an accessible commentary from an NLG expert, highlighting issues like data contamination and evaluating quality beyond just reference comparisons[43][44]. The Arize AI blog series on LLMops (from which we cited the OpenAI Evals piece) has practical insights into setting up eval pipelines and even using LLMs to evaluate other LLMs (“LLM as a Judge” concept). These can be helpful for getting industry perspectives on the topic.
• Benchmarks and Leaderboards: Beyond HELM, you might explore benchmarks like BIG-bench (a crowd-sourced big benchmark of tasks for LLMs), MMLU (Massive Multitask Language Understanding test[45]), and domain-specific evals (like Medical QA benchmarks, Toxicity tests, etc.). Many of these have associated papers and leaderboard websites. They can serve as sources of evaluation data or inspiration for your own evals.

As the field of AI progresses, evaluation will remain both crucial and challenging. By staying informed and leveraging the right tools, we can better quantify how our chatbots and LLM systems are doing — and more quickly iterate towards making them better (and safer) for everyone. Happy evaluating!

Sources

1. Prajna AI — “Everything You Need to Know About HELM — The Stanford Holistic Evaluation of Language Models” (Medium, 2025)[46][11]
2. OpenAI Evals — Official GitHub README[15][5]
3. Ehud Reiter — “Challenges in Evaluating LLMs” (Blog, 2024)[4][3]
4. Frugal Testing — “Best Practices and Metrics for Evaluating LLMs” (Blog, 2024)[6][7]
5. Arize AI — “Evals from OpenAI: Simplifying and Streamlining LLM Evaluation” (Blog, 2023)[14][39]
6. Shahul Es et al. — “RAGAS: Automated Evaluation of Retrieval Augmented Generation” (arXiv, 2025)[36]
7. Leonie Monigatti — “Evaluating RAG Applications with RAGAS” (Medium, 2023)[35][33]
8. Jin et al. — “A Systematic Survey on Evaluating Large Language Models” (arXiv, 2024)[9][10]

[1] [2] [11] [12] [21] [22] [23] [24] [25] [29] [45] [46] Everything You Need to Know About HELM — The Stanford Holistic Evaluation of Language Models | by PrajnaAI | Medium — https://prajnaaiwisdom.medium.com/everything-you-need-to-know-about-helm-the-stanford-holistic-evaluation-of-language-models-f921b61160f3

[3] [4] [43] [44] Challenges in Evaluating LLMs — Ehud Reiter’s Blog. — https://ehudreiter.com/2024/07/10/challenges-in-evaluating-llms/

[5] [15] [16] [19] [20] GitHub — openai/evals: Evals is a framework for evaluating LLMs and LLM systems, and an open-source registry of benchmarks. — https://github.com/openai/evals

[6] [7] [8] [13] Best Practices and Metrics for Evaluating Large Language Models (LLMs) — https://www.frugaltesting.com/blog/best-practices-and-metrics-for-evaluating-large-language-models-llms

[9] [10] [42] A Systematic Survey and Critical Review on Evaluating Large Language Models: Challenges, Limitations, and Recommendations- https://arxiv.org/html/2407.04069v1

[14] [39] [40] Evals from OpenAI: Simplifying and Streamlining LLM Evaluation — Arize AI — https://arize.com/blog-course/evals-openai-simplifying-llm-evaluation/

[17] Mastering OpenAI’s ‘evals’: A Deep Dive into Evaluating LLMs | by Xinzhe Li, PhD in Language Intelligence | Medium — https://medium.com/@sergioli/evaluating-chatgpt-using-openai-evals-7ca85c0ad139

[18] Getting Started with OpenAI Evals — https://cookbook.openai.com/examples/evaluation/getting_started_with_openai_evals

[26] [27] [28] [2211.09110] Holistic Evaluation of Language Model — https://arxiv.org/abs/2211.09110

[30] [36] [38] [2309.15217] Ragas: Automated Evaluation of Retrieval Augmented Generation — https://arxiv.org/abs/2309.15217

[31] [32] [33] [34] [35] [37] Evaluating RAG Applications with RAGAs | by Leonie Monigatti | TDS Archive | Medium — https://medium.com/data-science/evaluating-rag-applications-with-ragas-81d67b0ee31a

[41] Ragas — https://docs.ragas.io/en/stable/

Join thousands of data leaders on the AI newsletter. Join over 80,000 subscribers and keep up to date with the latest developments in AI. From research to projects and ideas. If you are building an AI startup, an AI-related product, or a service, we invite you to consider becoming a sponsor.

Published via Towards AI