The Directory

AI Tools

New launches tracked daily — what they cost, who they're for, how to get the most out of them

All tools

Researchby UK AI Safety Institute

Inspect AI

Name: Inspect AI
Author: UK AI Safety Institute

Free

Open-source framework for rigorous evaluation of large language models

Inspect AI is a comprehensive framework developed by the UK AI Safety Institute for conducting systematic evaluations of large language models. It provides researchers and developers with tools to create custom benchmarks, run evaluations across multiple models, and analyze LLM performance on various tasks. The framework supports complex evaluation scenarios including multi-turn conversations, tool use, agent behavior, and safety assessments. Built with Python, it offers flexible task definitions, integrated scoring metrics, and reproducible evaluation pipelines.

Who it's for

AI researchersML engineersAI safety researchersAcademic institutionsOrganizations evaluating LLMs

Pricing · free

checked Oct 11, 2026

Plan	Price	Includes
Open Source	Free	Completely free and open source · Full access to all framework features · Self-hosted evaluation infrastructure · Community support via GitHub · Python-based with extensive documentation

AI-researched pricing — verify on the official site before subscribing.

Use it for

— Creating custom LLM evaluation benchmarks
— Testing model capabilities across diverse tasks
— Conducting AI safety assessments
— Comparing performance across different models
— Running reproducible evaluation experiments
— Evaluating agent and tool-use behavior
— Multi-turn conversation testing

Get the most out of it

01Start with the built-in example evaluations to understand the framework's structure before creating custom tasks
02Use the model grading feature to have one LLM evaluate another's outputs for complex assessments
03Leverage the logging and visualization tools to track evaluation runs and compare results across experiments
04Design evaluations with clear success criteria and scoring rubrics to ensure reproducible results
05Take advantage of the async evaluation capabilities to run multiple model assessments in parallel and save time

Visit Inspect AI

Was this useful?