# How we validated AI title and abstract screening on 19 published systematic reviews

> The method, benchmark and limits of Edgaze's systematic review screening workflow: two independent screeners, 19 SYNERGY reviews, 41,203 records, 97.2% and 97.6% of included studies kept over two runs. Written to be cited in a methods section.

By Arjun Kuttikkat, Founder · Published November 20, 2026 · AI Workflows

Source: https://www.edgaze.ai/blogs/systematic-review-screening-validation

---

Researchers who try the screening workflow on Edgaze ask the same two questions: which models does it use, and has it been validated? If you use an automated tool to screen records, your methods section has to name it and say why you trusted it.

This page is the public record of that validation: how the workflow screens, how it was tested, what it found and missed, and what it does not do. It is written so you can cite it. Every number here comes from Edgaze's benchmark of the workflow on 19 systematic reviews from SYNERGY, an open dataset of published reviews with their final inclusion decisions, run on November 11, 2026.

One thing up front: this is not a peer-reviewed publication. It is a documented benchmark on public data, with the method and the reviews named, so that anyone can check it or repeat it.

## Summary

- **Task:** title and abstract screening for systematic reviews.
- **Workflow:** "Title and Abstract Screening for Systematic Reviews" on Edgaze, with two independent large language model screeners, GPT-6 Luna and GPT-6 Sol.
- **Rule:** a record is excluded only when both screeners exclude it on a criterion that can be judged from the title and abstract. Disagreements are marked Check for a person to decide.
- **Benchmark:** 19 reviews from SYNERGY v1.0, 41,203 records, 1,440 studies the authors finally included. Each review's own published abstract was the protocol; nobody wrote criteria by hand.
- **Result:** 1,399 of 1,440 included studies kept in the first run (97.2%) and 1,406 of 1,440 in the second (97.6%). A median of 67% to 71% of the records the authors excluded were set aside.
- **Cost:** about $1.13 of model use per 1,000 records.

## How the workflow screens

You give the workflow two things: your database search export and your protocol.

**1. Reading the search.** The workflow reads RIS, PubMed .nbib, Web of Science plain text, CSV and Excel files, up to 10 files per run. Duplicates across databases are removed by DOI, PMID, or title and year, and counted for your PRISMA flow diagram. A paid run screens up to 3,000 records after duplicates are removed.

**2. Writing the criteria.** GPT-6 Sol turns your protocol into one criterion per eligibility element: population, condition, intervention or exposure, comparator, outcomes if the protocol requires them, study design, language, date and publication type. It keeps the protocol's own scope and does not add requirements the protocol does not state.

Each criterion is given a stage. Population, condition, intervention or exposure, the kind of study design, language and publication type are judged now, from the title and abstract. Comparator and outcome criteria, dates, and design details that abstracts rarely report completely (allocation ratio, blinding, sample size, follow-up length, setting) are left for full text and never used to exclude a record at this stage. A comparator or outcome criterion is judged now only when the protocol explicitly says to exclude studies with it.

**3. Two independent screeners.** Records are screened in batches of 25. Each batch goes to two screeners that do not see each other's answers:

- Screener 1 is GPT-6 Luna, instructed to screen every record against the criteria.
- Screener 2 is GPT-6 Sol, instructed to protect recall: missing an eligible study is far worse than sending an ineligible one to full-text review.

For every criterion, each screener answers met, not met, or unclear. A criterion is "not met" only when the title or abstract states something that rules the study out. Missing information is unclear, never not met. A named variant or subtype of an eligible intervention or condition is unclear unless the criteria explicitly exclude it. Every exclusion comes with the failed criterion and the words from the record that rule it out.

**4. Combining the decisions.** Each record ends up in one of three piles:

- **Exclude:** both screeners excluded it, each on at least one criterion that can be judged from the title and abstract.
- **Check:** the screeners disagreed, or a screener did not return a decision for it. A record with no abstract is judged on its title alone; if either screener keeps it, it is marked Check rather than Include.
- **Include:** neither screener excluded it. It goes on to full text.

Each batch is checked for missing record ids. A record any screener skipped, or a batch that failed after retries, is marked Check, never excluded.

**5. The report.** The run returns a spreadsheet with a decision for every record (Check first, then Include, then Exclude) and the reason for each, plus the PRISMA 2020 counts, exclusions by criterion, the criteria used, and a methods paragraph to adapt. The paragraph follows the reporting items in the RAISE guidance and names the models and the date of the run.

The listing has model failover turned off, so a screener never switches to a model that was not benchmarked. A passing provider error is retried on the same model.

## Why two screeners, and why these models

The models were chosen on two SYNERGY reviews before the full benchmark: van Dis 2020 (cognitive behavioural therapy for anxiety disorders, 1,000 records, 56 eligible) and Oud 2018 (psychotherapies for borderline personality disorder, 760 records, 17 eligible).

GPT-6 Luna on its own found 47 of 56 eligible studies in van Dis and 14 of 17 in Oud. With GPT-6 Sol as the second screener, the pair found 56 of 56 and 17 of 17. Two other second screeners, Kimi K2.6 and DeepSeek V4 Pro, each missed 2 of 56 in van Dis. The second, recall-focused screener is what keeps the studies.

## The benchmark

### Dataset

[SYNERGY](https://github.com/asreview/synergy-dataset) v1.0 is ASReview's open set of 26 systematic reviews, each with its search results and the authors' final inclusion decisions. "Eligible" in this benchmark means included in the published review after full-text assessment, so these are the studies a screen must never lose.

### Protocol

For each review, the protocol given to the workflow was the review's own published title and abstract, word for word. Nobody wrote eligibility criteria by hand. That is a harder test than real use, where a review team pastes its full eligibility criteria or its PROSPERO record.

### Records

Every included study plus randomly chosen excluded records, at most 3,000 per review, which is the limit of one run. Abstracts came from PubMed when the record had a PMID, and from OpenAlex otherwise. Included studies with no title or abstract in either source cannot be screened by anyone and were left out: 6 in Brouwer 2019, 2 in Hall 2012, 4 in Leenaars 2020 and 1 in Oud 2018.

### How it was run

Exactly as the published workflow runs: its own prompts and models, the platform's system prompt around them, the production batching, one retry on a malformed answer, and the same merge that produces the three piles.

Each review was run twice on the same records, because the criteria step writes slightly different criteria each time. Reporting both runs shows how much that matters.

### Measures

- **Found:** eligible studies kept, as Include or Check. This is recall.
- **Set aside:** the share of the records the authors excluded that the workflow also excluded. This is the screening work saved.

## Results

Found and set aside are shown for both runs, as run 1/run 2.

| Review                        | Records | Eligible | Found   | Set aside, % |
| ----------------------------- | ------- | -------- | ------- | ------------ |
| Appenzeller-Herzog 2019       | 2,860   | 26       | 26/25   | 55/56        |
| Bos 2018                      | 2,993   | 10       | 9/10    | 90.5/86.3    |
| Brouwer 2019                  | 2,945   | 56       | 56/56   | 85.5/83.9    |
| Donners 2021                  | 251     | 15       | 14/15   | 66.1/50.4    |
| Hall 2012                     | 2,968   | 102      | 101/100 | 96.5/96      |
| Leenaars 2019                 | 2,988   | 17       | 16/17   | 83.9/60.7    |
| Leenaars 2020                 | 2,984   | 579      | 565/563 | 63.2/66.7    |
| Meijboom 2021                 | 882     | 37       | 37/37   | 71.1/69.9    |
| Menon 2022                    | 975     | 74       | 73/73   | 58.7/58.9    |
| Moran 2021 (corrected labels) | 3,000   | 139      | 137/135 | 71.7/72.4    |
| Oud 2018                      | 944     | 19       | 19/19   | 78.6/76.3    |
| Radjenovic 2013               | 2,978   | 48       | 48/47   | 74.3/83.1    |
| Sep 2021                      | 271     | 40       | 40/40   | 65.4/64.1    |
| Smid 2020                     | 2,551   | 27       | 26/26   | 57.9/62.8    |
| van de Schoot 2018            | 2,942   | 38       | 22/35   | 68.9/65.9    |
| van der Valk 2021             | 721     | 89       | 89/87   | 49.5/56.2    |
| van der Waal 2022             | 1,958   | 33       | 33/33   | 57.2/64.5    |
| van Dis 2020                  | 2,992   | 72       | 71/71   | 74.1/73.7    |
| Wolters 2018                  | 3,000   | 19       | 17/17   | 88.3/89      |

Across the 19 reviews and 41,203 records:

- **Recall:** 1,399 of 1,440 eligible studies kept in run 1 (97.2%) and 1,406 of 1,440 in run 2 (97.6%). The median per review was 98.7%.
- **Work saved:** a median of 67% to 71% of ineligible records set aside.
- **Check pile:** a median of 20% to 26% of records came back as Check.
- **Cost:** about $1.13 of model use per 1,000 records.

### What it missed

Over both runs, 54 distinct eligible studies were missed at least once, and 21 of them were missed both times. None of the missed studies lacked an abstract. Most of the 21 fall outside the review's own stated scope: Leenaars 2020's labels, for example, include a montelukast trial for otitis media in children, and narrative reviews. The rest are judgement calls.

The clearest weak spot is a protocol that does not say what the search was for. The published abstract of van de Schoot 2018 describes a Bayesian analysis of burn patients, while its search covered PTSD trajectories after any trauma. In one run the criteria step narrowed the population to burn patients and found 22 of 38; in the other it did not and found 35 of 38. A review team's own protocol states its scope; this published abstract did not.

## The seven reviews left out, and why

SYNERGY has 26 reviews. Seven were left out of the numbers above because their labels are not the published review's final inclusions, so they cannot measure whether a screen keeps the studies the review included. They are listed here with what the workflow found against those labels, so nothing is hidden.

- **Jeyaraman 2020:** 96 studies labelled as included, while the review's abstract reports nineteen studies. Found 14 of 96.
- **Muthu 2021:** 336 labelled, while the abstract reports seventy trials. Found 247 of 336.
- **Walker 2018:** 762 labelled, while the abstract reports 257 relevant studies. Found 506 of 762.
- **Wassenaar 2017:** 111 labelled, while the abstract reports 61 articles. It is a rodent-only review whose labels add human studies and narrative reviews. Found 24 of 111.
- **Chou 2003, Chou 2004 and Nelson 2002:** the labels come from the Cohen 2006 collection of drug-class reports, which is broader than the journal papers whose abstracts were used as the protocol. Nelson's labels, for example, include trials for hot flushes that the published abstract never mentions. Found 14 of 15, 5 of 9 and 77 of 80.

Moran 2021 is in the results, but with SYNERGY's corrected labels. The v1.0 labels included binge eating in adolescents and honey bees during an eclipse; SYNERGY later published corrected labels, which are the ones used.

## How the benchmark shaped the workflow

The criteria step's instructions were revised during this benchmark, on the same reviews. That matters when you read the results: they are not from a held-out set.

The first version treated dates as something to screen on, and wrote criteria about the topic even when the review studies other reviews. On the same records it found 0 of 74 in Menon 2022, a study of how environmental-health reviews are done, and 49 of 80 in Nelson 2002, where studies published after the search, added in later updates, were excluded on their year. The current version first decides what one record is meant to be (a primary study, or a review when the review studies reviews), and leaves dates and design details that abstracts rarely report to full text.

Moving comparator and outcome criteria to full text had a similar effect: on the Oud review it took recall from 13 of 17 to 17 of 17.

## Limits

- **Title and abstract only.** The workflow does not read full texts and does not make final eligibility decisions. Everything marked Include or Check goes on to full-text assessment by the review team.
- **A person stays responsible.** Someone on the review team should decide every record marked Check and read a random sample of the excluded records before relying on the exclusions. Cochrane, Campbell, JBI and CEE guidance (RAISE) asks you to report the tool, its version and date, and how you checked it.
- **Recall is high, not perfect.** 2.8% and 2.4% of eligible studies were lost in the two runs. Most of the studies missed in both runs were outside the review's stated scope, but not all.
- **The criteria vary between runs.** The criteria step writes slightly different criteria each time, which is why both runs are reported. In some reviews this moved the share set aside a lot (Leenaars 2019: 83.9% and 60.7%). The criteria a run used are part of its report; keep them with your review records.
- **The protocol does the work.** Results depend on how clearly the protocol states the scope. The benchmark used short published abstracts; a full protocol states more.
- **Not tuned on held-out data.** The criteria step was revised on these same 19 reviews.
- **Run size.** A paid run screens up to 3,000 records after duplicates are removed; a larger export has to be split across runs.
- **Specific models.** These results apply to GPT-6 Luna and GPT-6 Sol with the prompts in use on November 11, 2026. The methods paragraph from each run records the models and the date, so report those.
- **Not peer reviewed.** This benchmark was designed and run by Edgaze, on public data.

## Using it in your methods section

Each run's report includes a methods paragraph filled in with your counts, the models and the run date. A version you can adapt:

> Titles and abstracts were screened with the Edgaze workflow "Title and Abstract Screening for Systematic Reviews" (run on [date]), which uses two independent large language model screeners (GPT-6 Luna and GPT-6 Sol) against eligibility criteria derived from our protocol. A record was excluded only when both screeners independently judged that it clearly failed at least one criterion assessable from the title and abstract; records on which the screeners disagreed, records not screened by both, and records without an abstract that either screener retained were assessed by the review team. The workflow's validation on 19 systematic reviews from the SYNERGY dataset is described in [citation]. [Reviewer] assessed all records marked for checking and a random sample of [n] excluded records; final decisions were made by the review team.

Replace the bracketed parts with your own details, and describe your own checks honestly.

## How to cite this page

Vancouver:

> Kuttikkat A. How we validated AI title and abstract screening on 19 published systematic reviews [Internet]. Edgaze; 2026 Nov 20 [cited YYYY Mon DD]. Available from: https://www.edgaze.ai/blogs/systematic-review-screening-validation

APA:

> Kuttikkat, A. (2026, November 20). How we validated AI title and abstract screening on 19 published systematic reviews. Edgaze. https://www.edgaze.ai/blogs/systematic-review-screening-validation

If you describe the benchmark data, cite SYNERGY as its maintainers ask in the dataset's repository.

Questions about the method, or a review you would like checked against it, are welcome through the [help center](https://www.edgaze.ai/help).

---

Upload your search export and your protocol. The free demo screens up to 5,000 records. [Open the screening workflow](https://www.edgaze.ai/arjun_kuttikkat/systematic-review-screening)
