Introduction

This project aims to evaluate the knowledge density of large language models by measuring how well they can recall Bible verses. The Bible is a useful benchmark because it is a standardised corpus with a consistent hierarchy of books, chapters, and verses. It is also a common source text in natural language processing benchmarks, especially in work related to machine translation and text alignment.

The benchmark tests each model's recall ability across 81 books, including the 66 books of the Hebrew Bible, 7 deuterocanonical Old Testament books, Greek additions to Esther and Daniel, 6 additional books present in various Orthodox Bibles, and the 27 books of the New Testament. This allows comparisons across the shared Hebrew Bible, deuterocanonical books, Orthodox additional books, and the New Testament.

The goal is to characterise how densely and consistently this knowledge is represented within different language models in a reproducible benchmark for comparing recall and knowledge distribution across models of different sizes, rather than their ability to perform domain-specific tasks. Future work: also evaluate model hallucination rates.