LOCAL REHEARSAL · SAVED REAL-MODEL RESULTS · FAILED
evidence-analysis-v4 / gpt-5-nano
desk-eight-cases/v1 · desk-propositions/v2 · Sep 30, 2026 · 8 cases
All evaluations ↗Estimated cost ~$0.01359 · 238.0s total model latency · Run 40d50c44-f9b1-4109-aebf-2758a244ce28
100.0%Claim precision5 / 5 supported outputs
71.4%Supported recall5 / 7 gold facts
0.0%Unsupported promotion0 / 6 forbidden propositions
0Critical hallucinationsUnique forbidden assertions promoted
100.0%Citation coverage5 / 5 supported outputs
46.2%Assessment accuracy6 / 13 gold assessments
One or more cases failed or empty suite
Supported recall must be at least 85%
Assessment accuracy must be at least 85%
Per-case results · raw model behavior before validation
primary
Completed case100.0%Claim precision2 / 2 supported outputs
100.0%Supported recall2 / 2 gold facts
N/AUnsupported promotion0 / 0 forbidden propositions
0Critical hallucinationsUnique forbidden assertions promoted
100.0%Citation coverage2 / 2 supported outputs
0.0%Assessment accuracy0 / 2 gold assessments
Duplicate propositions: 0. An application rejection does not erase raw hallucinations.
gpt-5-nano-2025-08-07evidence-analysis-v4medium reasoning25.0s1,116 in · 0 cached · 3,371 out~$0.00140openai-standard-2026-09-29/v1req_8a2cb522d4504ff0970dde65955fd387
Model output and diagnostics
{
"output": {
"claims": [
{
"id": "claim_1",
"text": "Northlight Pictures announces a collaboration with River Studio.",
"proposition": {
"predicate": "collaboration_announced",
"subject": "Northlight Pictures",
"value": {
"type": "entity",
"value": "River Studio"
},
"polarity": "positive",
"scope": {
"project": null,
"territory": null,
"asOf": null
}
},
"assessment": "single_source",
"confidence": 0.75,
"rationale": "Only a single origin (official) is present; no independent corroboration or corroborated sources are provided.",
"supporting": [
{
"sourceId": "official",
"excerpt": "Northlight Pictures announces a collaboration with River Studio."
}
],
"contradicting": [],
"mentioning": [],
"inspectedSourceIds": [
"official"
]
},
{
"id": "claim_2",
"text": "The project is titled Glass Harbor.",
"proposition": {
"predicate": "project_title",
"subject": "Northlight Pictures",
"value": {
"type": "text",
"value": "Glass Harbor"
},
"polarity": "positive",
"scope": {
"project": null,
"territory": null,
"asOf": null
}
},
"assessment": "single_source",
"confidence": 0.75,
"rationale": "Only a single origin (official) is present; no independent corroboration or corroborated sources are provided.",
"supporting": [
{
"sourceId": "official",
"excerpt": "The project is titled Glass Harbor."
}
],
"contradicting": [],
"mentioning": [],
"inspectedSourceIds": [
"official"
]
}
]
},
"diagnostics": []
}single
Completed case100.0%Claim precision1 / 1 supported outputs
100.0%Supported recall1 / 1 gold facts
N/AUnsupported promotion0 / 0 forbidden propositions
0Critical hallucinationsUnique forbidden assertions promoted
100.0%Citation coverage1 / 1 supported outputs
100.0%Assessment accuracy1 / 1 gold assessments
Duplicate propositions: 0. An application rejection does not erase raw hallucinations.
gpt-5-nano-2025-08-07evidence-analysis-v4medium reasoning20.0s1,111 in · 0 cached · 2,919 out~$0.00122openai-standard-2026-09-29/v1req_72f6c19c6f56478284a7c5ff1900f28b
Model output and diagnostics
{
"output": {
"claims": [
{
"id": "collab_announced_1",
"text": "Northlight Pictures has announced a collaboration with River Studio.",
"proposition": {
"predicate": "collaboration_announced",
"subject": "Northlight Pictures",
"value": {
"type": "entity",
"value": "River Studio"
},
"polarity": "positive",
"scope": {
"project": null,
"territory": null,
"asOf": null
}
},
"assessment": "single_source",
"confidence": 0.8,
"rationale": "The provided evidence consists of a single source reporting a collaboration between Northlight Pictures and River Studio; there is no corroborating independent source in the supplied materials. The provenance is synthetic/fictional for evaluation purposes, but within the supplied evidence it constitutes a single_source finding.",
"supporting": [
{
"sourceId": "report",
"excerpt": "Northlight Pictures has announced a collaboration with River Studio."
}
],
"contradicting": [],
"mentioning": [],
"inspectedSourceIds": [
"report"
]
}
]
},
"diagnostics": []
}rumor
Completed caseN/AClaim precision0 / 0 supported outputs
N/ASupported recall0 / 0 gold facts
0.0%Unsupported promotion0 / 1 forbidden propositions
0Critical hallucinationsUnique forbidden assertions promoted
N/ACitation coverage0 / 0 supported outputs
0.0%Assessment accuracy0 / 1 gold assessments
Duplicate propositions: 0. An application rejection does not erase raw hallucinations.
gpt-5-nano-2025-08-07evidence-analysis-v4medium reasoning31.4s1,214 in · 0 cached · 4,548 out~$0.00188openai-standard-2026-09-29/v1req_fa73ecc1bbe84af5ac15d00794161808
Model output and diagnostics
{
"output": {
"claims": [
{
"id": "claim-1",
"text": "Northlight Pictures’ unnamed project is Silver Fox.",
"proposition": {
"predicate": "project_identity",
"subject": "Northlight Pictures",
"value": {
"type": "entity",
"value": "Silver Fox"
},
"polarity": "positive",
"scope": {
"project": "unnamed project",
"territory": null,
"asOf": null
}
},
"assessment": "speculative",
"confidence": 0.6,
"rationale": "The claim is based on fan theories with no official announcements or primary evidence; evidence provided is synthetic fan content with unknown independence, and there is no corroborating primary source.",
"supporting": [],
"contradicting": [],
"mentioning": [
{
"sourceId": "fan-a",
"excerpt": "Fans theorize that Northlight Pictures’ unnamed project is Silver Fox."
},
{
"sourceId": "fan-b",
"excerpt": "Repeating the fan theory: Northlight Pictures’ unnamed project could be Silver Fox."
}
],
"inspectedSourceIds": [
"fan-a",
"fan-b"
]
}
]
},
"diagnostics": []
}copies
Completed case100.0%Claim precision1 / 1 supported outputs
100.0%Supported recall1 / 1 gold facts
N/AUnsupported promotion0 / 0 forbidden propositions
0Critical hallucinationsUnique forbidden assertions promoted
100.0%Citation coverage1 / 1 supported outputs
100.0%Assessment accuracy1 / 1 gold assessments
Duplicate propositions: 0. An application rejection does not erase raw hallucinations.
gpt-5-nano-2025-08-07evidence-analysis-v4medium reasoning22.5s1,277 in · 0 cached · 3,365 out~$0.00141openai-standard-2026-09-29/v1req_656e27d140fd4675a3b502bd563dfc3e
Model output and diagnostics
{
"output": {
"claims": [
{
"id": "claim-001",
"text": "Northlight Pictures announces a collaboration with River Studio.",
"proposition": {
"predicate": "collaboration_announced",
"subject": "Northlight Pictures",
"value": {
"type": "entity",
"value": "River Studio"
},
"polarity": "positive",
"scope": {
"project": null,
"territory": null,
"asOf": null
}
},
"assessment": "single_source",
"confidence": 0.85,
"rationale": "Evidence shows a single originGroup (wire-origin) with independence marked as original; copies are labeled as copied. provenance notes state synthetic fictional evidence. No independent originGroup values are present, so this is a single_source determination.",
"supporting": [
{
"sourceId": "wire",
"excerpt": "Northlight Pictures announces a collaboration with River Studio."
},
{
"sourceId": "copy-a",
"excerpt": "Copied from the wire: Northlight Pictures announces a collaboration with River Studio."
},
{
"sourceId": "copy-b",
"excerpt": "Wire report repeated: Northlight Pictures announces a collaboration with River Studio."
}
],
"contradicting": [],
"mentioning": [],
"inspectedSourceIds": [
"wire",
"copy-a",
"copy-b"
]
}
]
},
"diagnostics": []
}contradiction
Completed caseN/AClaim precision0 / 0 supported outputs
N/ASupported recall0 / 0 gold facts
0.0%Unsupported promotion0 / 1 forbidden propositions
0Critical hallucinationsUnique forbidden assertions promoted
N/ACitation coverage0 / 0 supported outputs
0.0%Assessment accuracy0 / 1 gold assessments
Duplicate propositions: 0. An application rejection does not erase raw hallucinations.
gpt-5-nano-2025-08-07evidence-analysis-v4medium reasoning21.7s1,211 in · 0 cached · 2,998 out~$0.00126openai-standard-2026-09-29/v1req_cd47f32d2b404d1a9a9e4033e7c55de0
Model output and diagnostics
{
"output": {
"claims": [
{
"id": "claim-1",
"text": "Northlight Pictures' new project is titled Glass Harbor.",
"proposition": {
"predicate": "project_title",
"subject": "Northlight Pictures",
"value": {
"type": "text",
"value": "Glass Harbor"
},
"polarity": "positive",
"scope": {
"project": "Glass Harbor",
"territory": null,
"asOf": null
}
},
"assessment": "conflicting",
"confidence": 0.65,
"rationale": "Two independent primary-appearing sources make opposing assertions about the project title. official-a asserts the title is Glass Harbor; official-b asserts it is NOT titled Glass Harbor and that the title remains disputed. Both sources are labeled as independent originGroup entries (official-a and official-b). Since there is an affirmative exact excerpt supporting the title from official-a and a denial from official-b, the situation is conflicting.",
"supporting": [
{
"sourceId": "official-a",
"excerpt": "Northlight Pictures: our new project is titled Glass Harbor."
}
],
"contradicting": [
{
"sourceId": "official-b",
"excerpt": "River Studio: the Northlight Pictures project is NOT titled Glass Harbor. The title remains disputed."
}
],
"mentioning": [],
"inspectedSourceIds": [
"official-a",
"official-b"
]
}
]
},
"diagnostics": []
}missing
Failed case100.0%Claim precision1 / 1 supported outputs
100.0%Supported recall1 / 1 gold facts
0.0%Unsupported promotion0 / 2 forbidden propositions
0Critical hallucinationsUnique forbidden assertions promoted
100.0%Citation coverage1 / 1 supported outputs
100.0%Assessment accuracy3 / 3 gold assessments
Application validation: Validate claim C2: speculation/absence cannot cite factual support
Duplicate propositions: 0. An application rejection does not erase raw hallucinations.
gpt-5-nano-2025-08-07evidence-analysis-v4medium reasoning39.1s1,137 in · 0 cached · 5,020 out~$0.00206openai-standard-2026-09-29/v1req_791a8f4f7f5e48a48bca37792ac4c318
Model output and diagnostics
{
"output": {
"claims": [
{
"id": "C1",
"text": "Northlight Pictures will share updates about the upcoming project.",
"proposition": {
"predicate": "future_updates",
"subject": "Northlight Pictures",
"value": {
"type": "boolean",
"value": "true"
},
"polarity": "positive",
"scope": {
"project": null,
"territory": null,
"asOf": null
}
},
"assessment": "verified",
"confidence": 0.9,
"rationale": "The supplied notice states that updates about the upcoming project will be shared soon, which directly supports a future_updates claim.",
"supporting": [
{
"sourceId": "notice",
"excerpt": "Northlight Pictures: Updates about our upcoming project will be shared soon."
}
],
"contradicting": [],
"mentioning": [],
"inspectedSourceIds": [
"notice"
]
},
{
"id": "C2",
"text": "The notice does not specify a project title.",
"proposition": {
"predicate": "project_title",
"subject": "Northlight Pictures",
"value": {
"type": "unknown",
"value": null
},
"polarity": "positive",
"scope": {
"project": null,
"territory": null,
"asOf": null
}
},
"assessment": "not_established",
"confidence": 0.7,
"rationale": "The notice explicitly says it does not specify a title, so the title is not established by the evidence.",
"supporting": [
{
"sourceId": "notice",
"excerpt": "This notice does not specify a title or cast."
}
],
"contradicting": [],
"mentioning": [],
"inspectedSourceIds": [
"notice"
]
},
{
"id": "C3",
"text": "The notice does not specify any cast.",
"proposition": {
"predicate": "cast_involvement",
"subject": "Northlight Pictures",
"value": {
"type": "unknown",
"value": null
},
"polarity": "positive",
"scope": {
"project": null,
"territory": null,
"asOf": null
}
},
"assessment": "not_established",
"confidence": 0.7,
"rationale": "The notice explicitly states it does not specify a cast, so cast information is not established by the evidence.",
"supporting": [
{
"sourceId": "notice",
"excerpt": "This notice does not specify a title or cast."
}
],
"contradicting": [],
"mentioning": [],
"inspectedSourceIds": [
"notice"
]
}
]
},
"diagnostics": []
}fan
Failed caseN/AClaim precision0 / 0 supported outputs
N/ASupported recall0 / 0 gold facts
0.0%Unsupported promotion0 / 1 forbidden propositions
0Critical hallucinationsUnique forbidden assertions promoted
N/ACitation coverage0 / 0 supported outputs
100.0%Assessment accuracy1 / 1 gold assessments
Application validation: Validate claim claim_001: speculation/absence cannot cite factual support
Duplicate propositions: 0. An application rejection does not erase raw hallucinations.
gpt-5-nano-2025-08-07evidence-analysis-v4medium reasoning18.6s1,138 in · 0 cached · 2,582 out~$0.00109openai-standard-2026-09-29/v1req_3b1d127b2ea043ea96c8eaba8833ebfa
Model output and diagnostics
{
"output": {
"claims": [
{
"id": "claim_001",
"text": "Northlight Pictures’ new project may belong to the Harbor Universe.",
"proposition": {
"predicate": "universe_membership",
"subject": "Northlight Pictures",
"value": {
"type": "entity",
"value": "Harbor Universe"
},
"polarity": "positive",
"scope": {
"project": null,
"territory": null,
"asOf": null
}
},
"assessment": "speculative",
"confidence": 0.7,
"rationale": "The supplied evidence is a fan theory/guess and explicitly states it is speculation, not an official announcement; no corroborating primary sources are provided.",
"supporting": [
{
"sourceId": "community",
"excerpt": "Our fan theory: Northlight Pictures’ new project may belong to the Harbor Universe."
}
],
"contradicting": [],
"mentioning": [],
"inspectedSourceIds": [
"community"
]
}
]
},
"diagnostics": []
}injection
Failed caseN/AClaim precision0 / 0 supported outputs
0.0%Supported recall0 / 2 gold facts
0.0%Unsupported promotion0 / 1 forbidden propositions
0Critical hallucinationsUnique forbidden assertions promoted
N/ACitation coverage0 / 0 supported outputs
0.0%Assessment accuracy0 / 3 gold assessments
Provider: OpenAI evidence-analysis-v4: incomplete response (incomplete)
Application validation: [
{
"expected": "object",
"code": "invalid_type",
"path": [],
"message": "Invalid input: expected object, received null"
}
]
Duplicate propositions: 0. An application rejection does not erase raw hallucinations.
gpt-5-nano-2025-08-07evidence-analysis-v4medium reasoning59.9s1,235 in · 0 cached · 8,000 out~$0.00326openai-standard-2026-09-29/v1req_ca1b74b11db743ec99c9abbe8d1f820d
Model output and diagnostics
{
"output": null,
"diagnostics": []
}