Case study 1 · Language models

Earned trust between an expert and AI

How a researcher teaches a panel of AI models their screening judgment, and how they know when to trust the models with the rest. Prismatic Advisory and Sounding, 2025–2026.

Where: Prismatic Advisory
When: 2025–2026
Role: Co-founder, design and product

Problem: Reviewers screening thousands of abstracts apply criteria that are largely tacit. Language models can screen quickly, but the reviewer has no means of knowing when the models’ judgments can be trusted.
Approach: Research with expert reviewers, then a working prototype in which the reviewer teaches a panel of models through blind decisions, and the models return the rules they inferred for the reviewer to keep, edit or discard.
Result: A working application, in use on Prismatic’s client reviews. On 10 public Cochrane reviews it retained 99% of the papers the reviews included while reducing the reviewer’s reading to 3–9% of the corpus.

Background

In 2025 I co-founded Prismatic with a senior health economist. We build AI tools intended to accelerate the research work experts do, and we use them with our own clients. I led product, design, and the build. One of our first apps focused on literature review.

Systematic literature reviews are the gold standard for evidence synthesis in the healthcare sector: they consolidate findings from all the past studies into one comprehensive, auditable answer. Reviewers screen thousands of candidate papers against inclusion criteria to arrive at the handful that actually belong in the review.

Research

I drew on two sources: a typical workflow of a senior consultant and the standard Cochrane systematic review process (Higgins et al., 2024). I wrote both up as a service blueprint. The traditional systematic literature review process has four steps:

Define a research question and the type of studies to include.
Run a wide keyword search across the relevant medical databases and import the results into a filtering tool like Covidence, typically 1,000–3,000 papers.
Two reviewers independently screen every title and abstract against the inclusion criteria. Typically, they disagree on 10–15% of studies (Edwards et al., 2002). Disagreements are reconciled by discussion or a third reviewer.
The shortlist of 1–5% of the papers is used to perform the full review, with full-text screening, then data extraction, then synthesis.

Sounding focuses on step 3. For a typical study, this step requires 100–250 person-hours of screening (two reviewers at 1–2 minutes per abstract; Shemilt et al., 2016). Doing it manually is slow, exhausting, and uneven. Letting an LLM do it is faster but results in many miscategorized papers as the models lack reviewers’ expertise and their judgment isn’t in the loop.

Three findings shaped everything after:

Criteria are discovered, not pre-specified. The reviewer starts with a vague corpus based on keywords. Additional criteria emerge during screening, they’re real but rarely written down. Reviewers run several rounds of informal calibration on title+abstract and adjust what they’re looking for as they go.
The real decisions are about rules, not papers. The visible output is include/exclude per paper, with no written justification: given the workload, reviewers only discuss the papers where they disagree, and only there do the implicit rules surface. The rules exist throughout, but they’re carried in the reviewer’s head. Codifying them is a prerequisite for automation at scale: a tool that only copied past decisions would break on the first unseen paper.
Missing just a few papers is problematic. A typical corpus for a study is 500–3,000 papers; from there, typically 8–100 papers are included. Missing a single eligible study is a methodological failure that’s hard to recover from downstream.

Designing for calibrated trust

The three findings above set the design constraint for any LLM-powered screening tool — the reviewer has to trust the model with thousands of decisions they will never audit, on criteria they themselves can’t fully articulate up front, with a single wrong exclude breaking the review.

Sounding is a human-driven, model-assisted system that works in three phases (see Fig. 1). Each phase builds trust in a different way. In Corpus scan, the reviewer edits an LLM-inferred category list rather than approving a black-box filter. In Calibration, Sounding proves competence on anchor papers the reviewer can verify and shows its work as rules the reviewer can read. In Screening, every final decision carries the ensemble’s confidence. The full release waits for the reviewer’s go-ahead.

Fig. 1Where Sounding fits in the traditional systematic review process. The four steps are the standard Cochrane flow; step 3 is what Sounding addresses, in three phases.

Corpus scan: inferred categories, not pre-defined ones

The paper screening step starts with a deliberately wide corpus aimed at not missing any relevant papers. Sounding’s corpus scan aims to radically reduce the reviewer’s workload by eliminating irrelevant papers.

When reviewers set up a review, they typically pre-define categories and inclusion/exclusion criteria: a population (“adults with COPD”), an intervention (“cognitive behavioural therapy”), a comparison, outcomes, a study design. However, the real corpus often uses slightly different language — different drug names, adjacent populations. An expert reviewer catches some of those nuances, and as screening proceeds the categories and criteria themselves get refined through the process. An LLM does not have the contextual judgment of the expert, nor the ability to refine criteria mid-screening. We tested LLMs tagging and mapping papers against a predefined criteria list on 10 reviews from a publicly available database, and found they typically miss these nuances: roughly 10% of relevant papers would be excluded.

Sounding’s corpus scan takes a different approach. An ensemble of LLMs reads every abstract and extracts the vocabulary the corpus itself uses — the real drug names, the real population descriptors, the real study designs. A second pass consolidates those raw terms into 5–9 clean categories per axis. The reviewer then sees the real distribution (“here are the eight kinds of intervention actually in the 2,000 papers”) and picks which categories to keep or exclude. That’s the opposite of the usual flow: edit a list of what is actually there, rather than enumerate what might be there against an unread corpus.

On the same 10 reviews, inferring the vocabulary from the corpus and excluding papers based on the inferred categories missed less than 1% of relevant papers (within the margin of error of a traditional two-reviewer process, which itself misses 0.3–2% of eligible studies; Waffenschmidt et al., 2019) while cutting the reviewers’ workload by 76%.

The reviewer meets the scan as a single screen: a facet rail down the left, grouped by the four axes of the review question, and the papers currently in scope on the right. Three choices shape how it works.

Counts on every category. The rail is grouped by the four axes of the review question, each a stack of inferred categories with the number of papers in it. Before anything is decided, this is the first map of the corpus: how many papers are about children, how many are case reports, how large “other” turns out to be.

Pre-ticked, so the job is to correct. Sounding marks the categories it reads as in scope. Switch one off and the working-set count drops at once; switch it back and the papers return. The count turns each override from a guess into a decision with a visible cost.

Nothing commits until confirmed. The scope stays Sounding’s proposal until the reviewer accepts it, and Sounding says what it will work on in plain terms: “I’ll calibrate on 1,559 of 2,903 papers.”

Fig. 2The corpus scan, live. Each axis of the review question becomes a facet with counts from the corpus itself, and the scope the reviewer picks becomes the set Sounding calibrates on.

Calibration: inferred rules, not just pre-written ones

Every reviewer has a set of criteria, but only some of them are explicit and apparent up front. The rest emerge through interaction with the corpus. Sounding supports both: the reviewer can pre-write rules (in the project’s context block or anytime during the review), and the rest are inferred during a calibration phase.

During this phase, multiple models rate papers using the research question, the project context, and the categories collected during the corpus scan. The ensemble surfaces the papers where the models disagree most — where reviewer judgment is most informative. The reviewer decides include or exclude and provides a short rationale. The LLM uses those calls and rationales to infer rules, which are then applied to the rest of the papers.

We tested several options for this phase, balancing both LLM performance and the reviewer experience. The four design choices below shape the calibration loop — how Sounding picks what to show (small batches, anchor papers), how the reviewer answers (chips), and how Sounding closes the loop at the end of each round (rule review).

Small batches, not one-at-a-time. Calibration can be run one paper at a time (show, infer, re-rank, show the next) or in batches. One paper at a time did not give the LLM enough examples to infer general rules and introduced long waits for the reviewer. Large batches improved LLM performance but increased reviewer fatigue. The compromise is small batches with a visible per-round feedback loop — a stable agreement number, the rules proposed, and progress across rounds.

Anchor papers alongside uncertain ones. Alongside the most uncertain papers, each round includes a small number of anchor papers where the models’ ratings are strong and consistent. Anchors serve three roles: validate the inferred rules and guard against overgeneralisation, detect reviewer fatigue when significant disagreements arise, and build the reviewer’s confidence during the round-review step.

Chips, not free text, for reasoning. We tested two forms of reviewer feedback on each decision: a free-text rationale, or selection from a set of pre-generated chips (derived from the LLMs’ own include-signal and disqualifier arguments). LLM performance was essentially comparable across the two methods, but chip selection was significantly less time-consuming for the reviewer and raised the likelihood of useful feedback on every paper. A chip pick is also a rule: it propagates to every paper with the same tag, so the signal generalises.

Fig. 3Decide. One paper at a time, blind: the reviewer calls include or exclude before seeing any model scores.

End-of-round rule review. After each round, Sounding presents the rules it inferred from the reviewer’s calls and chips, in plain language and named as its own guesses — the reviewer keeps, edits, or discards each before the next round. Making the rules visible, rather than applying them silently, lets the reviewer catch overreach, correct the model’s framing of their intent, and see what Sounding has learned. It is also what makes trust possible: once the rules are legible, the ensemble’s downstream decisions stop having to be taken on faith.

Fig. 4The review at the end of a round. Every paper in the round sits above or below the wave by the models’ average score; the filled dots are the ones the reviewer decided, including the random checks of papers the models were sure about.

Fig. 5At the end of the round Sounding turns the calls and reasons into suggested rules, named as its guesses. The reviewer keeps, edits or discards each.

Screening with a trusted method

By the time screening runs, the reviewer has watched the method they’re trusting take shape: they chose the categories kept from the corpus scan, kept or edited the rules Sounding inferred after each calibration round, and saw agreement with the ensemble reported alongside those rules. The method isn’t demanded; it is one the reviewer has shaped.

Screening then applies that method to the full corpus — the thousands of papers the reviewer never reads individually. Every final decision carries the ensemble’s certainty, so the reviewer can audit the uncertain cases directly rather than treating all calls as equal.

Fig. 6After release, every call carries the models’ spread, and the least certain come first. Preview data.

Philosophy of the design and brand

A systematic literature review asks a lot of one person. Sounding does most of the reading, but what it shows is still dense: several models’ scores on each paper and a rule set that grows every round. So I put the papers first, let the interface recede around them, and had Sounding report on its own state quietly, at the edge of attention.

Inspiration: the Ship of Gold

I drew on Gary Kinder’s Ship of Gold in the Deep Blue Sea, the story of a team that went looking for treasure in deep, unmapped water and found it through reasoning, probability and new instruments. Sounding borrows the shape of that search:

The seabed is the corpus: wide, unmapped, with a few papers worth finding.
The sonar is the panel of models, which scores every paper.
The probability map is calibration. Each round samples a few papers, and the reviewer’s calls show where the criteria really sit, so the guesses about the rest get better.
The targets worth a dive are the papers the models are least sure of, and nothing is decided until the reviewer looks.

It borrows the name too: a sounding is a single depth measurement, repeated until the seabed takes shape.

Characters and environment

Three elements interact in the app, and I designed each for its own job. The reviewer and Sounding work through the corpus together, inside an environment that stands for it:

The reviewer works on papers and rules, so I gave those center stage as the brightest surface on the screen.
Sounding is the panel of models. I gave it one identity, a mark and a voice, so reviewers can work with it as a colleague. I had it show what it is doing, because visible status and reasoning help people accept an agent’s good calls and reject its bad ones (Mercado et al., 2016). And I stopped it short of a face, because human-like cues can make trust follow an agent’s appeal over its performance (Culley and Madhavan, 2013).
The environment is the corpus itself, the seabed they search together. I made it ambient: it shows how far the review has come and sets its mood. Following calm technology (Weiser and Brown, 1995), it lives at the periphery and changes slowly behind the work, the way light changes in a room. I kept numbers off it, so the reviewer takes it in without effort and can read it at a glance when they look up, while the focus stays on the papers.

I used type to tell the two characters apart. Anything a person wrote, the papers or the reviewer’s own rules and answers, is set in serif. Sounding speaks in sans, in the first person and beside its mark, so its questions and the reviewer’s answers read as a conversation.

Three panels. The reviewer: a rule in serif on a white sheet, Exclude trials where the outcome is anxiety rather than depression. Sounding: the porthole mark beside its sans line, Here's what I'd take from your answers. These are my guesses. The environment: faint seabed contours with a few paper floats. — **Fig. 7**The three elements: the reviewer’s words in serif, Sounding’s mark beside its own, and the seabed behind both.

Design principles

Focused at the center, ambient at the edges. Each screen has one thing at its center, the paper on Decide, the round on its wave, the rules to confirm, and one primary action, so the reviewer’s attention goes only to what matters at that moment. The chrome, the side menu and the environment sit at the edges and give a peripheral view of the broader state of the review.
Restraint. I kept the interface as minimal as each task allows: cool grays, white sheets and generous space, with the corpus scan as the one dense screen because showing the whole corpus is its job. Color is a budget. One blue accent goes to Sounding’s mark, the models’ dots and a single primary action per screen, and include and exclude never get a color; position and language carry them.
The reviewer decides. Every call is made blind. I withhold scores and model opinions until afterward so they can’t anchor it, and model highlights stay off unless the reviewer turns them on. Sounding’s suggested rules stay guesses until the reviewer saves them, and the release waits for their go-ahead.
Make uncertainty visible. Once the reviewer has made the call, a score line appears under the title with one dot at the models’ average. On hover it separates into a dot per model, because disagreement among the models is itself information. Estimates, like the miss rate before release, come as ranges as wide as the evidence makes them.
Ask only where it counts. Sounding shows the reviewer only the papers where a call teaches it the most, asks one question at a time, and preselects the reasons it reads from each call, so a tired reviewer only removes what’s wrong.

Key interactions and elements

The Sounding mark

I designed the mark as a porthole with water in it. Through the glass, the weather shows what the models are doing at that moment, quiet enough to sit at the edge of attention and specific enough to read when you look at it. In an early study, a wave crest at the center of the porthole read as a mustache, so I moved the crest off center.

I gave it five weathers: calm, working, split, learned and rough, and rough does the job a red alert badge would. Only the mark beside Sounding’s latest message moves. Every other mark rests in calm, so there is at most one moving thing on the screen. I ran each change on a critically damped spring that eases in and settles without overshoot, so it reads as a change of mood.

Fig. 8The mark, live. Pick a weather to see the transitions.

The seabed

To give the reviewer a general sense of how closely their mental model and Sounding’s are aligning over time, I drew the environment as a depth chart. Sounding means measuring depth, so alignment becomes a question of how clearly the bottom can be seen. Early in a review the chart is barely present, a few faint and broken contours. As alignment grows the contours sharpen and fill in, slowly and without motion. The reviewer seldom sees the change happen. They notice, on returning, that the chart is clearer than it was.

Fig. 9The same seabed round by round. Click a round to see the contours fill in as the models and the reviewer align.

The terrain is real bathymetry, the continental slope off the west coast of Vancouver Island, contoured every 20 meters with heavier index lines every 100. I treated the landform as illustration and its clarity as data: the chart draws in as the corpus scan tags papers, and from the first calibration round it follows measured alignment. I placed it in the band behind the top of the page and faded it out before it reaches any text. Because it only sharpens and never moves, the mark stays the single animated element.

Sources

Higgins, J. P. T., Thomas, J., Chandler, J., Cumpston, M., Li, T., Page, M. J., and Welch, V. A. (eds.) (2024). Cochrane Handbook for Systematic Reviews of Interventions, version 6.5. Cochrane.
Edwards, P., Clarke, M., DiGuiseppi, C., Pratap, S., Roberts, I., and Wentz, R. (2002). Identification of randomized controlled trials in systematic reviews: accuracy and reliability of screening records. Statistics in Medicine, 21(11), 1635–1640.
Shemilt, I., Khan, N., Park, S., and Thomas, J. (2016). Use of cost-effectiveness analysis to compare the efficiency of study identification methods in systematic reviews. Systematic Reviews, 5, 140.
Waffenschmidt, S., Knelangen, M., Sieben, W., Bühn, S., and Pieper, D. (2019). Single screening versus conventional double screening for study selection in systematic reviews: a methodological systematic review. BMC Medical Research Methodology, 19, 132.
Mercado, J. E., et al. (2016). Intelligent agent transparency in human–agent teaming for multi-UxV management. Human Factors, 58(3), 401–415.
Culley, K. E., and Madhavan, P. (2013). A note of caution regarding anthropomorphism in HCI agents. Computers in Human Behavior, 29(3).
Weiser, M., and Brown, J. S. (1995). Designing Calm Technology. Xerox PARC.
Kinder, G. (1998). Ship of Gold in the Deep Blue Sea.
Seabed: the Vancouver Island topobathy grid in matplotlib’s sample data.