# FactBench

[Canonical page](https://www.mohtasham.dev/work/archive/factbench)

A CLI for measuring how consistently language models classify factual claims from the FEVER dataset.

2025. Open source. Role: Independent build.

FactBench turns factuality testing into a repeatable run instead of a handful of cherry-picked questions. It evaluates the same FEVER claims across several models and keeps the raw predictions.

Each answer must resolve to supports, refutes, or not enough info. The runner reports accuracy, invalid responses, latency, and a confusion matrix for every model.

A local cache skips completed examples on later runs. Model lists, sample limits, output paths, and concurrency are all set from the command line.

## Facts

- FEVER labels: 3. 

- Default models: 5. 

- Default concurrency: 2. 

## How it works

1. Load FEVER: Read labelled claims from JSONL.

2. Check cache: Reuse predictions already completed.

3. Ask models: Run the remaining claims through OpenRouter.

4. Score: Compare predictions with the three gold labels.

5. Report: Write Markdown summaries and raw JSONL results.

## Languages

- TypeScript: 37937 bytes

[Public GitHub repository](https://github.com/MohtashamMurshid/checkmate-factbench)