A CLI for measuring how consistently language models classify factual claims from the FEVER dataset.
FactBench turns factuality testing into a repeatable run instead of a handful of cherry-picked questions. It evaluates the same FEVER claims across several models and keeps the raw predictions.
What I made
Each answer must resolve to supports, refutes, or not enough info. The runner reports accuracy, invalid responses, latency, and a confusion matrix for every model.
A local cache skips completed examples on later runs. Model lists, sample limits, output paths, and concurrency are all set from the command line.
How it works
5 stepsRead labelled claims from JSONL.
Reuse predictions already completed.
Run the remaining claims through OpenRouter.
Compare predictions with the three gold labels.
Write Markdown summaries and raw JSONL results.
Repository breakdown
This comes from GitHub's detected language breakdown. It measures source size, not the time or difficulty of the work.
Language mix
GitHub source bytesIndependent build
Open source