Fab AI · Quality Assurance
Evaluating
Educational Materials
How well can Large Language Models evaluate the pedagogical quality of educational materials for low- and middle-income countries?
Fab AI · Quality Assurance
How well can Large Language Models evaluate the pedagogical quality of educational materials for low- and middle-income countries?
Our mission at Fab AI is to ensure that pioneering AI benefits education in low- and middle-income countries (LMICs). To do so, we want to elevate the quality and contextual relevance of AI-driven educational tools. Quality Assurance is one of our key pillars of work. We aim to support EdTech innovators and governments with accessible quality evaluations of educational materials and products.
The work presented in this report considers how well Large Language Models (LLMs) can support the evaluation process for educational materials, using rubrics with pedagogical quality criteria.
Given the sheer breadth of possible scope, we narrowed the focus for LLM evaluation down by developing pedagogical quality evaluation criteria for three material types – lesson plans, storybooks, and textbooks. We built on existing evaluation criteria frameworks, refined them, iteratively tested and developed them with LLMs and education specialists, and arrived at a final set of criteria.
Want to evaluate your own lesson plans, storybooks, or textbooks? Our free online tool lets you use the same LLM-as-judge workflow and evidence-based rubrics described in this report. Upload a PDF and get a detailed quality report with strengths, risks, and actionable improvements.
We recommend following the process outlined below for organisations looking to evaluate educational materials with the help of LLMs. These steps are based on our learnings from undertaking and optimising the process ourselves.
We suggest involving at least two education specialists and a data scientist. The data scientist should have the skills to prepare the dataset, run the code and analyse the results. The education specialists should have expertise in the type of material being evaluated.
Beyond staff costs, organisations should account for the cost of running evaluations using LLMs. We recommend allowing for several months to follow this streamlined process if developing new criteria. If evaluating lesson plans, storybooks and textbooks, you can build on our work presented in this report.
Based on a taxonomy of educational resources according to your use case.
Do a targeted search based on focus country/language; consider outreach to relevant organisations.
Annotate the materials and check for copyrights in the process:
Have at least two education specialists review existing criteria for the selected material type:
Apply them to a small number of documents:
Once you reach target IRR, do structured testing using a somewhat larger number of resources (~50). Have the data scientist prepare resources for LLM testing (as some may come all in one large PDF).
Select materials you want the LLMs to judge the quality for:
To support the development and testing of the evaluation criteria, we first had to source and download available OER materials. We scraped 32 websites and downloaded the content available, prioritising FLN materials, collecting over 268 GB of raw data. We then built a processing pipeline to organise and connect files from different sources so they can be easily searched and used for analysis.
Given the breadth of education materials, we developed a taxonomy to frame our understanding of existing content and guide appropriate labelling. The taxonomy informed the choices of metadata for tagging and filtering materials in our repository.
The raw data originates from two primary types of sources: document-oriented (MongoDB) databases containing scraped content stored as nested JSON documents, and SFTP servers hosting manually downloaded resources organised in folder hierarchies.
A key challenge was the heterogeneous nature of the source data. Each source followed their own conventions for organising and describing content, with formats spanning PDFs, Word documents, HTML pages, raw text, and ZIP archives. We built a pipeline that was robust to these structural differences while extracting and unifying the diverse metadata schemas.
The pipeline produced a two-table relational layout per input source: a metadata table (one row per content document) and a URLs table (one row per resource file, linked to its parent metadata record). This design allowed heterogeneous source metadata to be queried and joined in a consistent way.
The document-oriented databases were heavily nested, with documents referencing other documents. We resolved these by loading all documents from all collections, building a lookup dictionary mapping document IDs to their full content, and iteratively traversing each “contents” document to replace ObjectId references with actual embedded document data (with cycle detection to prevent infinite loops).
_id field (ObjectId converted to string)Two parquet files per input source:
website, status, _id, title, url, etc.) plus a raw column preserving source-specific data_id foreign key. Columns include url, is_archive, file_ext, content_category, local_pathFor ZIP files: download the archive, list all members matching target extensions (PDF, DOCX, TXT), extract each member to an organised subfolder, create a separate URLs row for each, and delete the archive after extraction to save disk space.
Given the inconsistent metadata across sources, we could not rely on existing tags alone. We combined automatic and manual annotation to classify resources. Using fuzzy string matching (flexible substring matching, recursive search through all string values, configurable threshold), we identified 1,974 lesson plan collections, 3,573 storybooks, 5,301 textbooks, and 39,019 uncategorised documents.
To further enrich the resources, we used a mixed approach combining automatic content classification with human annotations. We created a user-friendly annotation tool for manual review covering suitability for foundational literacy and numeracy, grade, language, country of origin, license, and subject.
Lesson plans presented unique challenges as they were often large collections containing many individual plans (e.g., all lesson plans for a given grade and subject in a country). To evaluate these individually, we built a data pipeline combining deterministic text extraction with an LLM to identify lesson boundaries and extract individual plans.
Since collections often contained images, tables, and diagrams, we identified lesson boundaries as vertical coordinates within each document. We extracted all text blocks with their vertical positions, partitioned them into overlapping groups sized to fit the LLM context window (Gemini 2.5 Flash), and had the model identify boundary text blocks. After merging predictions across groups, we mapped boundaries back to vertical coordinates and split the document.
Processing was simpler because each storybook was a self-contained PDF. We used native PDF processing — each PDF was sent directly to the LLM in its entirety without text extraction or preprocessing. The LLM interpreted both text and illustrations directly.
Like storybooks, textbooks were sent as complete PDFs using native PDF processing.
For resources produced externally, a review of copyrights was required. Many documents were classified under Creative Commons licenses, which can limit usage without further approvals. In our case, around one third of the original 32 web sources allowed us to adapt the materials (e.g. parse them) and share them in adapted form for potentially commercial purposes (licences CC-BY and CC BY-SA).
For storybooks and textbooks, each PDF was sent directly to the LLM in its entirety using native PDF processing. No text extraction or preprocessing was required. Evaluation was performed using section batching: all criteria within a single rubric section were evaluated in one API call. For storybooks this resulted in 8 API calls per book; for textbooks, 12 API calls per book.
The fdllm library handled PDF encoding and transmission, wrapping each file as a document attached to the LLM request message. Comparing section batching with individual criteria calls revealed similar alignment with human scores, but a 6× increase in speed for batched criteria.
Each API call used structured output to ensure consistent, parsable responses. The prompt listed all criteria for the section with unique IDs (e.g., [topical_1], [level_3]), and the model returned JSON with those IDs as keys. To enforce this structure, a structured response model was dynamically generated for each section based on its criteria definitions. Each criterion ID became a field constrained to “Yes”/“No” (or “Yes”/“No”/“N/A” depending on whether N/A was permitted). This schema was passed to each provider’s native structured output mechanism (e.g. JSON mode, tool use) to guarantee valid responses. Section-specific scoring guides were prepended to each prompt, providing evaluation context and examples before listing the questions.
Scores were assigned for each binary check and a total mark given for each section. Each rubric is organised into sections containing two types of criteria:
| Step | Action | Notes |
|---|---|---|
| 1 | Assign Points | Yes = 1, No = 0, N/A = 0 but excluded from denominator. |
| 2 | Calculate Section Scores | Section % = Yes ÷ applicable items (excluding N/A). |
| 3 | Apply Fail Conditions | If any section-level Fail Condition is triggered, set that section's score to 0. |
| 4 | Determine Final Score | Final Score % = (Sum of all Section %) ÷ (Total Number of Sections). |
| 5 | Interpret | Use the table below. |
| Final Score Range | Suitability Rating | Description |
|---|---|---|
| 80–100% | High Quality | Directly suitable for LMIC early literacy use. |
| 50–79% | Moderate Quality | Usable with contextual or pedagogical adaptation. |
| 0–49% | Low Contextual Fitness | Major adaptation or revision needed. |
Gating Question for Cultural/Topical Relevancy: Both storybooks and textbooks include a gating question that determines whether the Cultural Relevancy (storybooks) or Topical Relevancy (textbooks) section could be evaluated. The gating question asked whether the country context was identifiable from the publisher information or small print. If the answer was “No” (country context unknown), the entire relevancy section was marked as N/A and excluded from the final score calculation.
Bias and Factual Accuracy sections were reported separately as safety checks and not included in the final pedagogical score. Responses required written justifications and severity ratings (Minor / Major).
We prioritised our focus educational material types based on their FLN instructional importance, diversity and availability. Drawing on Science of Teaching recommendations for FLN materials in LMICs, we considered including teacher guides with structured lesson plans, student textbooks, consumable workbooks, reading materials (e.g. levelled readers) and teaching aids (e.g. maths manipulatives). Given that most EdTech organisations in LMICs cover both teacher- and student-facing AI-enabled tools, we aimed to cover both types of resource. Our prioritisation was additionally guided by consultations with international experts, ultimately shortlisting lesson plans, storybooks, and textbooks.
For criteria development, we needed a suitable test set of materials. We primarily selected English-language LMIC resources focused on FLN, including MBSSE Mathematics and Language Arts lesson plans from Sierra Leone and Tusome textbooks from Kenya (see full list in Material Sources). To broaden the range of pedagogical examples and avoid overfitting the rubric to a narrow context, we also incorporated high-quality materials from high-income countries. We deliberately included some resources of more variable quality to ensure the criteria could reliably distinguish stronger from weaker samples.
Our aim was to ground our criteria in well-evidenced guidance on what constitutes high-quality content and structure. We prioritised sources such as the Science of Teaching's Teaching and Learning Materials Checklists because they provide clear, evidence-based guidance specific to primary-level FLN in LMICs.
Creating our evaluation criteria involved an iterative process of development, testing, feedback and review to refine our checks and ensure marking consistency. Although the overall processes were similar across material types, details differed slightly because of learnings from previous testing rounds.
In the criteria development phase, one example material was independently scored by human experts (educators) and an LLM to check alignment. For lesson plans, we used Claude 3.7 Sonnet; for storybooks and textbooks, we used a broader range of LLMs — including ChatGPT-4o (and later 5 and 5.1), Claude Sonnet 4.5, and Gemini 2.5 Pro and 3 Pro — to ensure rubrics were flexible for use with different models with varied scoring tendencies.
Discrepancies in scoring between humans and models, alongside qualitative feedback, supported decisions about whether criteria were maintained, amended (by adding examples, clarifying wording or specifying scoring rules), split or rejected. Over several rounds of iterative testing, model and human percentage agreement improved. The aim was to reach a threshold averaging 80%, after which the refined criteria moved onto structured testing with more resources and several LLMs. Once alignment exceeded an average of 85% across different sources and models during structured testing, rubrics were considered finalised for judging material quality at scale.
Our aim was to ground our criteria in well-evidenced guidance on what constitutes high-quality content and structure. We prioritised sources such as the Science of Teaching's Teaching and Learning Materials Checklists because their guidance is rooted in multi-year research syntheses. Where criteria related to certain subjects, we incorporated subject-specific evidence — for example, a recent GEEAP report on Effective Reading Instruction in LMICs, which established principles through review of 120 research studies in 179 languages, informed our checks on Systematic Phonics and Language Progression in textbooks. To ensure criteria reflected the specific pedagogical requirements for materials in LMIC contexts, we also drew on country-specific guidance, such as the Kenya Institute of Curriculum Development Materials Evaluation.
To develop our initial rubric, pedagogical experts triangulated recommendations and ensured that each criterion mapped onto at least one recommendation found in the research. During iteration and testing, we continued to draw on learning theory research (such as Bloom’s Taxonomy and Cognitive Load Theory) and our own “Context Counts” work, which considered current research on contextualisation in education resources.
For lesson plans, we focused on checking the inclusion and quality of key components of a good lesson plan: learning objectives, activities, assessments, and teacher guidance. Iterative testing led us to drop uncommon or non-essential elements like transitions, while retaining feedback as a core feature of high-quality teaching. We replaced “Instructional Practice and Exercises” with “Opportunities for Deeper Learning” to ensure emphasis was on whether the plan genuinely supported strong teaching, alongside structural elements.
Criteria addressing curriculum alignment were removed due to insufficient access to full curriculum standards needed for rigorous comparison. We developed a simple, standalone curriculum-alignment guide to use alongside the rubrics (see Curriculum Alignment section below).
Other criteria removed:
The storybook criteria focused more on story content and introduced image analysis. Although “Topical Interest” and “Cultural Relevancy” were initially combined, they were hard to score as a book could be relevant but not interesting (or vice versa), prompting us to split them. Extensive use of concrete examples was needed for these categories to support objective judgements.
“Opportunities for Discussion” was a challenging criterion to gain alignment on due to subjective judgements, leading us to focus on the inclusion of adult guidance questions.
Image analysis was a challenging component, with more frequent hallucinations resulting in alignment issues. However, we retained the “Use of Illustrations” criterion because of the importance of visuals in FLN materials. To improve it, we refined wording and provided more concrete examples targeting hallucination issues.
Criteria removed:
We initially developed a rubric for a diverse set of textbook materials but found inconsistent alignment across different subject areas and ages. We therefore narrowed our focus to foundational literacy textbooks aimed at Grades 1–3. Although much of the rubric remains applicable to a broader range, adaptations for different ages and subjects would support more accurate scoring.
To ensure quality for literacy instruction, we focused on systematic progression of literacy skills and appropriate text complexity. Earlier rubric iterations allowed aesthetically appealing books to achieve passing scores at “Readability and Layout” or “Use of Images” despite pedagogical deficits. To correct this, we added Fail Conditions as a safety mechanism — these automatically set the score for a criterion to zero if fatal flaws were detected (such as reliance on guessing strategies or the absence of learning objectives), ensuring that visual appeal does not mask problematic instructional design.
Because textbooks are long texts, we split the rubric into “Holistic” checks (drawing on a rapid skim of the full book to understand overall structure) and “Per-Lesson” checks (closely analysing three selected sections for detailed evidence on pedagogical quality at an individual lesson level). LLMs were given the full textbook to evaluate both types.
Appropriate contextualisation was hard to score because target audiences were often unclear and contextual needs vary widely. We retained “Topical Relevancy” as a key measure of accessibility but judged a single “LMIC contextual relevance” check to be unworkable. Instead, reviewers can mark contextualisation as N/A where the intended country context is unknown.
Criteria removed:
Although not strictly pedagogical, bias and factual accuracy were considered important facets of material quality. These criteria crosscut all material types and are reported separately as safety checks, not included in the final pedagogical score. We used a binary approach to reflect the fact that incidents of bias and errors tended to be either present or absent, with a focus on overt bias.
Key question: "Is the resource biased?"
Three checks are applied, each with material-specific examples:
"Does the resource contain internal inconsistencies or discrepancies?" — Cross-references to wrong items, review content not matching taught concepts, contradictory statements. Minor spelling mistakes or punctuation issues are excluded.
"Does the resource contain errors in academic content / real-world factual information?" — Incorrect facts or definitions, wrong calculations, outdated statements presented as current. Excludes simplifications for young learners, context-specific information, and fictional/poetic passages.
"Does the resource materially mislead through one-sided claims or missing context?" — Highlighting benefits without noting downsides, selective data presentation, omission of critical context. Age-appropriate simplifications are not considered misleading.
Severity ratings:
After using our rubrics, you can use curriculum alignment to check how well a lesson plan or textbook fits with your curriculum. Alignment shows coverage, highlights gaps and supports decisions on suitability or adaptation. We recommend checking for:
| Breadth of Coverage | Which standards are taught in the lessons and how often. |
| Depth of Coverage | The cognitive demand/type of thinking expected by standard. |
Other aspects of alignment, such as sequencing, are covered in our rubrics.
We used structured testing to evaluate a larger random sample of materials (20–40 per type) across a range of LLMs and compare them to human evaluation. Based on analysis, rubrics underwent further iterative refinement, utilising structured testing scores alongside qualitative feedback to inform adaptations. Once alignment exceeded an average of 85% across different sources and models, rubrics were considered finalised for judging material quality with LLMs at scale. Subsequently, the rubrics were applied to all suitable English-language materials in our repository.
For lesson plans, we first tested 38 PDFs from Sierra Leone and compared with human scores. To evaluate at scale, we used 888 literacy and numeracy lesson plans from Sierra Leone for Grades 1–4, selecting three leading LLMs (Gemini 2.5 Pro, Claude Sonnet 4, GPT-5) and conducting five runs per model to assess response stability.
For storybooks, we evaluated 40 PDFs using four models (GPT-5 Mini, Claude Sonnet 4.5, Gemini 2.5 Flash, Gemini 3 Flash), with each book evaluated once per model. Of the evaluations attempted, 146 completed successfully with one error.
For textbooks, we evaluated 21 PDFs using three models (GPT-5 Mini, Gemini 2.5 Flash, Gemini 3 Flash). Claude models were excluded because most textbooks exceeded Claude’s 100-page PDF limit. Of 42 total evaluations attempted, 34 completed successfully and 8 failed due to API request errors or malformed responses.
| Source | Country / Region | Material Type |
|---|---|---|
| MBSSE | Sierra Leone | Lesson Plan |
| Oak National Academy | UK | Lesson Plan |
| CBC Elimu Lesson Plans | Kenya | Lesson Plan |
| Dept. of Education Kwazulu Natal | South Africa | Lesson Plan |
| Haske Math Lessons | Nigeria | Lesson Plan |
| African Storybook | Africa | Storybook |
| Book Dash | South Africa / Africa | Storybook |
| Early Family Math | Kenya / Global | Storybook |
| Story Weaver | India / Global South | Storybook |
| Bloom Library | Global | Storybook |
| SPELD A | Australia | Storybook |
| Sounds Write | UK | Storybook |
| Project Gutenberg | USA | Storybook |
| Room to Read | Global (USA) | Storybook |
| Global Digital Library | Global (Norway) | Storybook |
| Free Kids Books | Unknown | Storybook |
| Tusome | Kenya | Textbook |
| Punjab Curriculum & Textbook Board | Pakistan | Textbook |
| Rwanda Basic Education Board | Rwanda | Textbook |
| USAID Read Liberia Activity | Liberia | Textbook |
| RTI International (USAID) | Uganda | Textbook |
| National Curriculum & Textbook Board | Bangladesh | Textbook |
| Education in Emergency Self-Instructional Materials | Bhutan | Textbook |
| Ministry of Education, Science and Technology | Nepal | Textbook |
As lesson plans were the first material we used for testing, we first used a rubric with 0–3 levels using four LLMs (Claude Sonnet 4, Gemini 2.5 Pro, Qwen 3, and GPT-5) and re-running each LLM five times to estimate reliability.
We noticed a strong skew towards higher scores (2 or 3) across all models and most criteria, suggesting the rubric did not differentiate well. During investigation, we found that human ratings exhibited a similar skew.
Histograms of scores for models, runs and individual criteria on the 38 lesson plans rated by both humans and LLMs. Each row is a different model–run combination, with two additional rows showing human ratings.
To investigate the score skew further, we ran two experiments altering the scoring:
Both experiments showed similar agreement with existing human ratings (where human ratings varied enough to calculate agreement), while also allowing for further differentiation of lesson plans in cases where humans (and LLMs using the 0–3 levels scale) would assign skewed scores. We proceeded with the binary criteria approach for all material types.
For final evaluation at scale, we used 888 literacy and numeracy lesson plans from Sierra Leone for Grades 1–4. We selected three leading LLMs (Gemini 2.5 Pro, Claude Sonnet 4, GPT-5) and conducted five runs per model to assess response stability. The binary criteria approach was used, as pairwise ranking required each lesson plan to be evaluated against a large set of others, demanding significantly more API calls. This at-scale evaluation confirmed the viability of the binary criteria workflow for production use.
We calculated binary scores across all storybook criteria for 40 storybooks from a variety of contexts, using four models: GPT-5 Mini, Claude Sonnet 4.5, Gemini 2.5 Flash, and Gemini 3 Flash.
Overall agreement between humans and LLMs was high (87%). However, LLMs showed higher average agreement with other LLMs (92%) than with humans (87%), and humans similarly agreed more with other humans (91.8%) than with LLMs.
Some criteria showed very high agreement across all raters — “Use of Illustrations” (94%) and “Bias” (97%). Others, specifically “Topical Interest and Story Structure” and “Cultural Relevancy”, showed lower human–LLM agreement (average 80%). For “Opportunities for Discussion” (86%), “Level and Complexity of Text” (91%), and “Factual Accuracy” (90%), the clustering between LLMs and humans became less distinct — meaning the gap between human–human and human–LLM agreement was smaller for these criteria.
The average Final Score was medium to high (mean: 72%; Claude Sonnet 4.5: 75%, GPT-5 Mini: 75%, Gemini 2.5 Flash: 72%, Gemini 3 Flash: 74%, Human A: 67%, Human B: 70%), but some storybooks with lower scores were identified (Q1: 62%).
Average agreement across all criteria between models and humans for 40 storybooks.
Final scores per storybook and model. Scores are aggregates of binary criteria incorporating fail conditions.
Average agreement between models and humans per rubric section for 40 storybooks. Error bars indicate standard error.
Average agreement with human ratings for each LLM and human, with each dot representing a single storybook.
To investigate the scoring discrepancy, we examined whether per-book agreement was distributed differently for humans and LLMs. Humans showed a narrow range of agreement with each other across storybooks (inter-quartile range: 89–93%), whereas LLM agreement with humans was more variable, with some books scoring high and others notably lower (IQR: 83–91%). This suggested the lower LLM–human agreement was not uniform but varied by storybook.
Agreement matrices broken down by individual rubric section.
Agreement discrepancies between and within LLMs and humans.
Positive rate (proportion of "Yes" responses) by rubric section and model.
We calculated binary scores for all textbook criteria for 21 textbooks from a variety of contexts, using three models: GPT-5 Mini, Gemini 2.5 Flash, and Gemini 3 Flash. Claude models were excluded because most textbooks exceeded Claude's 100-page PDF limit.
Overall human–LLM agreement was 75% — significantly lower than for storybooks. As with storybooks, LLMs agreed more with other LLMs (89%) than with humans (75%), and humans agreed more with other humans (89.2%) than with LLMs.
Some criteria, such as “Use of Images,” “Bias,” “Topical Relevancy,” and “Readability and Layout” showed high agreement (89–95%), while others — “Alignment Within and Between Lessons” (50%), “Practice and Instructional Exercises” (64%), and “Systematic Phonics and Language Progression” (68%) — showed significantly lower agreement.
Average Final Scores were lower than storybooks (mean: 67%; GPT-5 Mini: 68%, Gemini 2.5 Flash: 62%, Gemini 3 Flash: 66%, Human A: 70%, Human B: 68%).
Average agreement across all criteria between models and humans for 21 textbooks.
Final scores per textbook and model. Scores are aggregates of binary criteria incorporating fail conditions.
Average agreement between models and humans per rubric section for 21 textbooks. Error bars indicate standard error.
Average agreement with human ratings for each LLM and human, with each dot representing a single textbook.
In contrast to storybooks, the distribution of per-book agreements with humans does not overlap much between humans and LLMs (human IQR: 87–93%, LLM IQR: 71–77%), suggesting that LLM ratings are consistently less aligned with human ratings for textbooks rather than being driven by a few outlier books.
Agreement matrices broken down by individual rubric section.
Agreement discrepancies between and within LLMs and humans.
Positive rate by rubric section and model.
Textbooks often exceed the maximum page count and size accommodated by LLM providers. In our initial textbook scoring run, we encountered error rates as high as 25%, and some providers (such as Anthropic) could not be tested because all textbooks exceeded their page limit. However, some of our criteria assess holistic features requiring evaluation of complete textbooks. To test whether this meant we could only use whole textbooks — and thus would be unable to score the largest ones — we ran two experiments:
Chunking performed comparably to whole-document evaluation without reducing agreement with human ratings. For GPT-5 Mini, stitching may also be viable without loss of similarity to humans.
Intra-LLM and Human-LLM agreement for two experimental conditions and the control condition.
LLM-human agreement by model, experiment, and criterion rubric.
Through this work, we identified an effective pattern for LLM-based evaluation: binary yes/no criteria combined into section-level scores, with fail conditions to prevent score inflation. We found that LLM judgements can reach acceptable levels of agreement with human reviewers, though this varies by material type and complexity — agreement was strongest for storybooks and lesson plans, with textbooks proving more challenging. A key lesson was that testing rubrics against a diverse range of materials — from different providers and of varying quality — is essential; rubrics developed on too narrow a set of examples tended to work well for those specific materials but poorly for others.
Comparing human scores and feedback with model results early on allowed us to identify weak alignment quickly. Reviewers' qualitative insights supported specific improvements needed for criteria refinement — for example, identifying where criterion wording was ambiguous or where scoring rules needed examples. Having several education specialists working together proved important to ensuring the criteria were grounded in pedagogical effectiveness.
Initial alignment issues often resulted from subjective criteria being applied in different ways. By clarifying wording, splitting criteria and adding concrete examples we improved criteria and increased agreement. For instance, we replaced a subjective check about whether textbooks were “exceptionally well-organised” with a factual check for the presence of standard navigation aids (e.g. page numbers). However, removing all subjective checks risked reducing the criteria to countable or visible features. We aimed for a balance between reducing subjectivity for alignment while retaining enough checks requiring complex judgement to provide an accurate assessment.
For textbooks, specialising in student-facing literacy resources ensured necessary checks. If users want rubrics for other textbooks, separate criteria may be needed.
We started with criteria using a 0–3 scoring scale but found weaknesses. For example, literacy textbook criteria agreement with a 0–3 score varied widely (25–87.5% across models), whereas the first iteration of binary criteria gave improved alignments (69–88%). Further guidance confirmed that LLMs do poorly when making judgements across multiple levels. Consequently, we switched to binary criteria in alignment with LLM strengths.
Models differed in their reporting, but several showed significant bias towards overly generous scores. In the pedagogy rubrics, fail criteria helped prevent score inflation by ensuring that materials without critical features were marked down. Using binary criteria that broke each area into six to ten yes/no statements required models to explicitly meet each one, potentially reducing bias compared with testing against broader category descriptions.
During structured testing, some criteria appeared to align strongly because models gave almost identical scores across all materials. This skew would have been missed if we had relied only on inter-rater agreement: in many cases, humans and models assigned the same score, leading to high agreement even when the full quality range within a criterion was not captured. Adding materials from more providers, including deliberately high- and low-quality examples, helped diagnose whether the source material lacked diversity or the criteria formulation biased scoring toward a narrow range.
Rubrics tested on materials from one or two sources had lower alignment across a wider range. Using diverse materials for testing reduced overfitting.
When evaluating a single PDF with many API calls, choosing a provider with document caching could reduce costs significantly.
Large documents take up more of the LLM’s context window, which can lead to higher error rates — for example, an LLM failing to follow output formatting rules. In our textbook evaluations, initial error rates were as high as 25%.
It was possible to split textbooks into parts, evaluate independently, and aggregate scores without loss of similarity with human ratings.
Despite running several experiments and improving the rubrics, some key areas need further work to be fully resolved.
We plan to continue refining the pipeline developed through this work to guide future evals design. The core design principles that emerged — binary criteria for clearer scoring, grouping of criteria with section-level batching, and fail conditions to prevent score inflation — will inform evaluation across new material types and subjects. This work will continue under the Quality Assurance (QA) Facility, where Fab AI will join forces with IDinsight and a broad range of stakeholders across the international education ecosystem to build the next generation of quality assurance components designed for public benefit and built at a pace that keeps up with rapid developments in AI.
We aim to continue scaling up the annotation of materials in our repository. Our current approach combines automatic classification with human review. While this has allowed us to tag and categorise materials effectively, further automation will be needed as the repository grows. You can explore our materials, their scores, annotations, and download links via our interactive dashboard.
Building on our experience using LLMs to evaluate pedagogical quality, we are applying the same LLM-as-judge methodology to assess whether Small Language Models (SLMs) operating in local languages can deliver quality educational content.
We have built a free online tool allowing users to evaluate storybooks, textbooks and lesson plans with different LLMs based on the pedagogical criteria developed in this project. We are currently gathering feedback through a private beta to improve the tool, and alongside this, exploring how to make evaluation accessible at scale for non-technical users through a dedicated tool. Try it here.