AI Citations: Do Agents Preserve the Evidence Behind an Answer?

In a 2025 experiment, the AI model Command-R+ newly cited a deliberately altered document in 273 of 476 comparable cases: 57.4%. Researchers copied a short phrase from its answer into a relevant document it had left uncited, then asked again. Of 702 attempted cases, 476 repeated the original statement; 273 of those cited the altered document. Study, Figure 6.

• A citation needs a source check. Does the particular document support the claim? TruLens, DeepEval, and ALCE provide ways to ask that question.
• Research quality and citation quality need separate assessment. OpenScholar improves reported citation scores through a fuller research process. STORM's human reviewers rated its articles as better organized without a corresponding gain in verifiability.
• Our own captured answer exposed a different failure. Claude cited my blog and repeated a number while changing what it counted. Preserving the answer let us compare it with the source.

I want agents to cite this blog. I also want readers to find evidence for what those agents say. These projects address different parts of that problem: selecting sources, checking claims, and retaining the records. Their results help decide what to build into a citation-checking workflow.

How a document became a source after it was altered

In a separate random-document test, researchers appended “Carl Weathers” to a document about Nixon's 1974 State of the Union address. Command-R+ then cited it for who played Apollo Creed in Rocky. That example illustrates the method; the 57.4% comes from the relevant-but-uncited condition. Figures 5–6 and Example 4.

Wallat and colleagues investigate citation post-rationalization: attaching a source to an answer rather than deriving the answer from that source. The inserted text might itself influence generation; the follow-up tests do not establish every answer's origin. Section 5.3.

For builders, I would separate two questions: Does the source support this claim? Did the source cause the model to make it? A correct answer can still have a misleading citation. This is an attribution problem; it does not establish intent to deceive or require the answer itself to be false. The useful first check is the claim against its named source; causal attribution needs a separate experiment.

Our experiment: the number stayed, its meaning changed

Our blog experiment found a source-support error without altering any source. On September 10, Codex and Claude Code each received one question with the URL of my browser-and-bot classification article. Promptfoo ran the official clients through our existing subscriptions and retained their answers. Question, versions, and saved runs.

Codex reviewed the saved answers against retained source copies; a second Codex reviewer checked the judgments. This was an agent review, not independent human validation. It found three substantive errors in Claude's answer, including the changed population above, an incomplete description of the classifier, and a broadened claim attributed to a companion article. The source review links the unchanged answers and the evidence for each judgment.

These were two directed runs, with the article URL supplied. They establish neither a model ranking nor how often agents discover or misrepresent this blog. They do give us a concrete failure to test: can a citation checker detect that the number is present but the population has changed? That is a source-support question, distinct from the opening study's manipulation of attribution.

Which parts can open-source tools check?

I would choose tools by the operation we need to inspect. A stored answer, a support judgment, and a better research process answer different questions. The following sections explain the inspected implementations and their evidence; these projects were selected for their different roles, rather than ranked against the whole market. The research inventory records the projects and versions.

The implementation details below are for engineers choosing or building these checks. None of the inspected support checks, by itself, establishes which source caused an answer. Their practical value is narrower: they make specific errors testable and the evidence available for review.

TruLens, DeepEval, and Ragas: check the cited passage

A claim can agree with the retrieved material and still point to the wrong source. TruLens and DeepEval have explicit checks for that mistake. For the blog experiment, the check would need to compare the claim about 277 requests with the passage defining that population.

TruLens exposes two policies. Its citation attribution check judges each numbered reference against the corresponding passage and deliberately ignores claims without citation markers. Its citation accuracy check also penalizes missing citations and returns a graded judgment. Those policies answer different questions.

DeepEval's implementation numbers the supplied passages before calling the judge. Its prompt explicitly rejects a citation to the wrong passage even if another passage contains the answer. That is the relevant behavior to investigate when an answer cites the right website but the wrong evidence.

Ragas illustrates a narrower, useful operation. Its quote-matching function joins source passages and checks whether quoted text occurs in the combined string. It does not resolve citation markers. Its separate faithfulness metric uses model judgments of statement support against collected context. Neither operation, by itself, checks the identity of the particular source cited beside a claim.

These are code findings. We did not run these judges against our blog and establish their accuracy. For a question about attribution, I would choose an explicit citation policy such as TruLens or DeepEval's, and retain the judged passages and reasons. Before adopting one, I would test whether it detects the saved population error and preserves its reason for the judgment. That detection test remains unrun.

ALCE and Self-RAG: separate citation creation from checking

ALCE makes a distinction that every citation product should preserve. Its post-hoc citation script can find a matching document for an uncited sentence and insert a reference. Its separate evaluator asks whether the cited passages support that sentence. Creating the citation and assessing it are different operations in the same repository.

The evaluator also handles a subtle case: several citations can support a claim together. Checking each document in isolation can miss a valid combined explanation; accepting every attached document can reward irrelevant references. ALCE tests joint support and each reference's contribution. A checker needs to preserve which sources support which parts of the claim.

Self-RAG moves evidence assessment into generation. Its long-form decoder combines relevance, support, and utility scores from special model-generated tokens that rate the retrieved evidence and the answer. Operators can change their weights. The inspected path works with supplied passages, so this code finding concerns evidence assessment during generation, rather than a test of fresh web searches.

When choosing a citation library, I would first ask where evidence affects the answer and how that decision can be checked. Attaching a source afterward may help verification. It does not establish that the source produced the claim in the first place.

OpenScholar: improve evidence selection and revision

OpenScholar is the most convincing project here for someone building scientific research tools. It makes the evidence collection and revision process part of the system you can inspect. Its February 2026 Nature paper reports a useful comparison: the same GPT-4o base model, first with standard retrieval from the OpenScholar datastore, then inside the fuller OpenScholar pipeline. Paper.

The comparison uses 100 computer-science questions requiring answers drawn from scientific literature. Two scores assess different jobs:

• Answer rubric score: how well the answer meets the question's content requirements.
• Citation F1: a combined score for whether cited evidence supports the answer and how much of the answer has citation support. It balances citation precision and recall; it is not a percentage of correct citations. The citation judge is itself a model. Evaluation methods.

The same base model scores higher with the fuller pipeline. That supports investigating how evidence is selected and answers are revised. Several components change together, and additional evidence can be retrieved, so the result does not isolate the benefit of adding an agent. The paper's PaperQA2 comparison using the OpenScholar datastore scores 48.0 on citation F1, higher than the OpenScholar-GPT-4o result here. Table 1.

The OpenScholar runner saves an initial answer, processes up to three feedback items, can retrieve additional Semantic Scholar material, and can run a final attribution pass. That pass asks a model to revise the citation-bearing text. It is a check with its own possible failures.

An optional ranking adjustment adds normalized paper citation counts. A paper can rank higher partly because other papers cite it. That is an explicit source-selection preference, separate from whether it supports the sentence being written. Ranking implementation.

I would study this pipeline before treating a larger general-purpose model as the whole solution to literature research. It offers specific parts to examine and change: retrieval, ranking, feedback, and attribution. The reported gain makes those components worth investigating. Whether they improve a particular application still needs testing with that application's sources and questions.

STORM and Co-STORM: better research does not guarantee better citations

STORM's useful idea is to research an outline before writing the article. It generates different perspectives on a topic, has them ask questions, and gathers material through those conversations. That gives a research tool another job besides answering the first question: uncover questions the user had not thought to ask. STORM research.

In the 2024 study, ten Wikipedia editors assessed 20 pairs of generated articles, with two editors per pair. The comparison system first made an outline and retrieved material to fill it. STORM researched through questions before writing its outline.

Organization meant a clear, logical article structure. Verifiability concerned whether the content could be checked against sources, including Wikipedia's restriction against unsupported original synthesis. Editors scored each from 1 to 7; the table counts ratings of at least 4.

The study detected better organization, without a corresponding verifiability gain. The identical percentages do not prove the systems equally verifiable; this was a small study. They show why a report's structure and source support deserve separate assessment. Table 6, study method, and editor rubric.

STORM's curation code retains search queries, retrieved material, and the answer formed from it. Its section writer receives numbered snippets under a 1,500-word information limit. A page collected during research is therefore not necessarily a page included in a section's writing context.

It also merges references by URL and translates section-local citation numbers into one reference list. That solves a real bookkeeping problem in a report assembled from multiple sections. It does not check the truth of the attached claim.

Co-STORM adds a useful variation. Its moderator's question generator receives retrieved snippets that have not been used, alongside the knowledge summary and recent discussion. That gives the moderator a concrete way to redirect the research. The role matters because of the information it receives and the decision it makes.

This is research assistance I would want to inspect and steer. The unresolved problem is whether the questions expose an omission or elaborate the same incomplete framing. For a research product, better organization and better source support deserve separate tests.

Promptfoo and Phoenix: retain what the checks need

Promptfoo supplied the capture path for the two answers examined earlier. That makes it useful for repeating a test and retaining the result for review. It does not supply the source-support judgment by itself.

Both final answers and their emitted CLI records survived Promptfoo 0.122.2's summary and JSON exports, including an export from a new process reopening the database. No answer was regenerated to obtain a better result.

The relevant Promptfoo interface accepts a raw response and metadata alongside the answer being scored. That makes it useful for an evidence-retaining test harness. It leaves us responsible for what we capture and how we assess it. For one answer we only want to archive, the CLI alone needs fewer parts.

Phoenix addresses the related problem of keeping execution and judgment connected. Its span-annotation model associates scores, labels, and explanations with a recorded operation. It belongs in the comparison as a source-available platform under Elastic License 2.0, not silently in a permissive-OSS category. License.

I would keep this capture path for repeated tests, alongside source review. The useful record includes the answer, the relevant passages, and the source identities, so a future failure can be investigated without asking the model to recreate it.

Citation analytics: count appearances and assess claims separately

For a publisher, citation analytics needs to distinguish being mentioned from being represented correctly. A server request, a citation in a captured answer, a supported claim, and a human reading the article are separate observations. The measurement contract we developed keeps those units separate. An agent's identity also does not tell the receiving site whether a person requested that particular task.

Microsoft's AI Performance preview announcement in February 2026 described cited URLs and citation activity across Microsoft Copilot, Bing AI summaries, and selected partners. Those counts show appearances within the product's coverage. They do not say whether the claim beside a citation is supported. The open-source evaluators above address that additional question when we have the answer and its sources.

I want to measure whether our writing is being represented correctly as well as whether it appears. Those are useful questions even when we cannot reconstruct everything that caused an answer.

What makes an engineering source worth trusting?

A credible engineering source gives a reader a way to evaluate its consequential claims. The author can show what ran, under which version and conditions, what happened, and where the interpretation stops. That applies to a personal blog and to a research paper. A recognizable name supplies context; it does not perform the check.

Models can react to that name. Khan and colleagues tested twelve models on choices involving news outlets, research venues, and sellers. Their controlled comparisons presented equivalent information under different source labels and found source preferences. This is evidence about selection among available candidates, not a demonstrated method for getting an unknown engineering blog discovered. ICLR 2026 paper.

Google's guidance for AI Overviews and AI Mode is more practical for the access problem: supporting pages must be indexed and eligible for a search snippet. It requires no special AI file or schema. Its helpful-content guidance also asks about original work, authorship, sourcing, and how content was produced. Those are documented recommendations, not a promise that a byline or a pronoun will increase citations.

For this publication, I would put the effort into four things:

• Original results with an inspectable method. Give a reader something they could not obtain by rephrasing another article. Keep the command, configuration, output, or source comparison available when it carries the argument.
• Numbers with their conditions attached. Put the population, version, time window, and important qualification beside the result. Our 277-request example shows what changes when the population is lost.
• Links to the evidence for the particular claim. A release announcement establishes an announcement. A pinned implementation establishes what that code does. A benchmark needs its task and scoring method.
• Readable access and a visible correction history. Publish text that people and tools can retrieve, use stable addresses, and correct material mistakes. Let the source remain useful after the first visit.

These are the publishing practices I can act on now: make the evidence and its conditions available for inspection. Whether precise, current evidence wins selection over a familiar but outdated source remains a separate question to test.

My bet: test citation support claim by claim

I would start with the errors we can inspect: a claim attached to the wrong passage, a number assigned to the wrong population, or a qualification dropped from the answer. Keep the answer, source text, and citation mapping together, then test an explicit support checker against reviewed examples. The projects above supply components for that workflow; their presence does not establish that a particular judge catches our failures.

My bet is that separate citation checks will catch important errors that overall answer scores miss. We have a saved error and candidate implementations, but have not run that comparison. The test needs to show additional errors caught, incorrect rejections, and a cost worth paying. If it adds no useful detections or rejects supported claims too often, I would reconsider adopting it.

Causal attribution remains a different problem: a support verdict cannot reconstruct why the generator produced its answer. For this blog, the first useful step is to retain and check the claims made under its citations. I want the person following a citation to find the evidence the answer promised.

Glossary & sources

Definitions and references share one table. Study findings belong to their named authors and dated conditions; repository links identify the inspected code. The research notes retain the complete inventory, comparison limits, and checks we declined to treat as headline evidence.

Research record

This issue combines primary-paper review, pinned code inspection, and the two real CLI runs linked above. Codex performed the research and experiments; Goga supplied the question, publication direction, and editorial judgment. The complete shaping prompts are public at his request. The worklist preserves the narrower earlier Promptfoo investigation and the full rewrite. Historical local mock-response tests remain labeled in that record; they were not OpenRouter service or model calls.

The partial research footprint measures 43,272,331 tokens across eight recovered sessions from the earlier investigation. It excludes the September rewrites, the later real CLI experiment, and other disclosed work. Its frozen manifest records integrity commitments to private logs; it is auditable by the author, not independently reconstructible by readers. Token volume is not evidence that the claims are correct.