FastEval: Single Click Evaluation of Language Models

Author(s): Dr. Mandar Karhade, MD. PhD.

Originally published on Towards AI.

Evaluation of various benchmarks with a single command

FastEval is a tool designed to accelerate the evaluation process of instruction-following and chat language models. It stands out for its efficiency, providing a way to evaluate models on various benchmarks swiftly and cost-effectively. This article will delve into the features, installation, and usage of FastEval, underlining its significance in the landscape of language model evaluation.

FastEval offers a streamlined and high-performance solution for evaluating language models across different benchmarks. It leverages vLLM (vectorized Large Language Models) for fast inference, significantly reducing evaluation time compared to traditional methods like using huggingface transformers. By storing outputs and intermediate results, FastEval enables detailed performance analysis, allowing users to inspect model performance across various categories and even individual outputs.

Multiple Benchmark Support: FastEval can evaluate language models on benchmarks like MT-Bench, HumanEval+, DS-1000, and others, covering areas from conversational capabilities to Python coding performance and reasoning.High Performance: Utilizing vLLM and optional text-generation inference, FastEval achieves a speed of about 20 times faster than traditional methods.Detailed Performance Insights: It provides a comprehensive view of model performance by saving model outputs and intermediate results.Customizable Evaluation: Supports model-specific prompt templates and integrates with FastChat for extended capabilities.

To install FastEval, one needs to have Python 3.10 installed and then… Read the full blog for free on Medium.

Join thousands of data leaders on the AI newsletter. Join over 80,000 subscribers and keep up to date with the latest developments in AI. From research to projects and ideas. If you are building an AI startup, an AI-related product, or a service, we invite you to consider becoming a sponsor.

Published via Towards AI