Insights E: answer unless proved wrong

Goal E fails its acceptance line. Real returned 6/5/5 correct completed answers. Fresh returned 6/7/5 correct completed answers. 4 of six repeats exceeded 21 seconds p95.

Correct completed is the primary result. Silent wrong is reported beside it; latency includes failed attempts.

Correct completed, wrong answers and latency

Real usage has 101 attempts and 86 supported first turns; fresh-v1 has 122 attempts and 116 supported first turns. D3 and D2 real controls each have one retained repeat, so their real-usage repeat noise is unknown.

Paired comparisons

Intervals resample whole paired intent families 10,000 times (seed 2026090611), averaging each family across repeats and normalizing counts to the full sample. The p95 difference uses pooled question timings; the run table reports each repeat separately. All other metric intervals and undefined-draw counts are in results.json. Historical comparisons do not isolate changes in provider or host conditions.

Pass lines

Wrong answers are not a pass criterion. Removing a refusal or error is not a correctness gain. Against D3, previously refused or errored real questions became correct 2 times per repeat and wrong 17.33 times. Against D3, previously refused or errored fresh questions became correct 4.33 times per repeat and wrong 17.89 times. These are observed paired transitions, not isolated causal effects. Removing refusals or errors exposed wrong answers as well as correct ones. Complete transition counts are in results.json.

Family results

Counts below are correct and silent wrong for repeats 1/2/3. Full membership and denominators are retained in compact records.

Policy and refusal audit

Decision-site counts: {'OPERATIONAL': 231, 'PROVED WRONG': 74, 'UNPROVED': 78}. Repeated boundaries are counted as sites, not distinct guard families.

Answering cannot manufacture a result when generation, execution, authorization or a saved operation has no usable input. These exits stay errors; uncertainty about the meaning of an executable query answers and records.

Unproved obligations are saved in the answer record and trace without a rendered warning or regeneration. Known duplicate identities, stored mixed currencies, wrong traced calendar operations, explicit filter contradictions, write/credential requests and a cohort inherited from a failed parent remain blocking. The verifier and its repair loop are absent.

Real-usage obligation counts count each class once per attempt, across all three repeats. The table lists every observed class. The generic question-interpretation record is not a newly detected policy gap.

Refusal guard labels are preserved from the terminal attempt. Where guard is absent, BI_MEASURE_READING_CONFLICT in the raw terminal error_code identifies MeasureReadingConflictError. The compact refusal_class retains this derivation without replacing the original guard. Other missing labels remain unattributed. All 51 final refusals retain the evaluator outcome refusal_unverified: guard attribution and policy tests do not independently establish that each measured refusal was justified.

Integration and systematic fixes

D1 calendar coverage (6d4d5924, bb7e3938) was already byte-identical on main via abdf4afd. D2 1f60d99e supplied the nineteen-ruling knowledge/seeder subset. D3 2513a718 supplied generation defaults and their tests. The exact DCLASS f749b308 production patch and schema-qualification tests were reapplied. The complete taken/left-behind ledger is integration-ledger.json. No F01 code was taken.

DCLASS previously reduced schema errors while all four tagged real questions stayed at zero correct and target silent wrong rose. This goal does not relabel those unlocked wrong answers as progress. Ruling 12 is implemented by filling missing month/week result rows with zeros in the London calendar; existing rows, NULLs and stored data are preserved.

One retained synthetic PostgreSQL case uses an unratified notes.user_id join. Its already-aggregated total cannot be proved wrong from the available contract; Goal E returns it and records uncertainty. The independent oracle demonstrates a doubled total. That is an explicit cost of the requested policy, not a passing correctness example.

Checks and limits

{
  "current": {
    "unique_checks": 2007,
    "passed": 2007,
    "failed": 0,
    "skipped": 0
  },
  "historical_568": {
    "inventory": 568,
    "unchanged_passes": 421,
    "changed_policy_passes": 49,
    "missing_file_nodes": 89,
    "not_run": 9,
    "failed": 0,
    "skipped": 0,
    "definition": "Unchanged test-body AST and a current passing execution. Changed tests assert Goal E answer-and-record behavior; they are not original invariant passes."
  },
  "runs": [
    {
      "label": "followup-broad-final",
      "counts": {
        "passed": 2006
      },
      "sha256": "24a034cf968270496a5734f285aed4ca2c2a36fffa1b749a816ef497a5cd1aae",
      "private_junit": "/tmp/lore-goal3-eval/goal-e/followup-broad-final.xml"
    },
    {
      "label": "followup-evidence-final",
      "counts": {
        "passed": 23
      },
      "sha256": "85ac90c0b2ebb4068ad48ff6cc030d3dcb23c0c0a7004d8baace6ecbc5551230",
      "private_junit": "/tmp/lore-goal3-eval/goal-e/followup-evidence-final.xml"
    }
  ],
  "mypy_new_modules": "passed",
  "ruff_changed_python": "passed",
  "historical_original_execution": {
    "counts": {
      "passed": 446,
      "failed": 64,
      "not_executed": 58
    },
    "collection_errors": [
      "tests.test_insights_federated_followup_contract",
      "tests.test_insights_snapshot_and_trade_grain",
      "tests.test_insights_v17_membership_predicates",
      "tests.test_insights_v18_membership_output"
    ],
    "production_revision": "5a78252393a1c98dcd1fadfe0f26a4d718db60c6",
    "tests_revision": "8958ae93ff196bdc5fbcfd08ae12f81445c5d28c",
    "junit_sha256": "b28eb6bddb276a37a614e26ab0fb4ddad7e52fea2c49442825ce4889b2f519d9",
    "private_junit": "/tmp/lore-goal3-eval/goal-e/historical-corrected-original-attempt.xml"
  },
  "delivery_ci": {
    "ci_run": "https://github.com/stevanl/lore/actions/runs/34729350902",
    "head": "7e3cd52e1ef78ddf18d8cf3d84e2bd5db7cea6f3",
    "electron": "passed",
    "quality_gate": "failed at Ruff",
    "ruff_errors": 502,
    "unchanged_base_errors": 501,
    "changed_files": {
      "tests/test_insights_rulings_promotion.py": 1
    },
    "base": "10dd338b96c88420cc0107f6de9da4bf76945601",
    "raw_log_sha256": "9c7b52c69e2f7f7738ed03c5796288da62c59b9be103e34fada4308d18166747",
    "raw_log_retained": false,
    "unreached_steps": [
      "mypy",
      "pytest",
      "offline evaluator gates"
    ],
    "correction": "Removed unused UUID import from tests/test_insights_rulings_promotion.py. Test-function ASTs are unchanged; measured application source remains byte-identical to 5a782523.",
    "post_correction_changed_python_files": 74,
    "post_correction_changed_python_ruff_errors": 0,
    "post_correction_whole_repo_ruff_errors": 501,
    "remaining_findings": "All 501 findings are in pre-existing evidence files byte-identical to fetched origin/main. Preserved historical evidence; no unrelated edits or lint suppressions.",
    "final_ci": "Whole-repository Ruff remains blocked by 501 unchanged historical-evidence findings. Changed-file lint passes; full CI has not validated type checks, tests or offline evaluator gates."
  }
}

The historical 568-check inventory includes files absent from origin/main and tests requiring the old uncertainty-refusal policy. Its original nodes are mapped separately; a missing node is never counted as a pass. Current policy checks are not a claim that all 568 original assertions still pass unchanged.

Measurement method

Goal E measurement contract

Source: owner prompt, 2026-09-12. Candidate starts from fetched origin/main, keeps current zai generator, requested glm-5.2, temperature 0, seed 42, thinking disabled, 25-second provider timeout. No verifier or unproved-obligation model retry.

Three fresh-v1-r2 repeats (122 attempts, 116 supported first turns, 31 families), then three corrected real-usage repeats (101 attempts, 86 supported first turns, five conversations). Sequential, two workers, frozen clock 2026-09-06 12:00 UTC, London calendar, identical D3 corrected oracle bytes. No oracle edits or F01 changes. Admit only after two five-second CPU samples at least 95% idle; retain host observations and process cleanup. Freeze application and evaluator before first measured run. No tests or reviews during measurement. Retain every attempt; no quality-driven reruns.

Comparators: retained D3 Arm A (three fresh, one real) and D2 baseline (three fresh, one real). Whole-family paired bootstrap: average per-family counts over repeats, 10,000 draws, seed 2026090611, nearest-rank two-sided 95% intervals, normalize to sample denominators. Historical real controls have no repeat-noise estimate. Historical/sequential comparisons are confounded with provider and host conditions. Fresh slice was exposed in D2, not an unseen holdout.

Report correct completed first, silent wrong beside it, then p95 question latency (including failed attempts). Also refusals by proved-wrong guard class, errors, unverifiable, supported first-turn errors, complete conversations, per-family correct/wrong. Pass: correct-completion difference against D3 Arm A has lower interval above zero on BOTH samples; every repeat p95 <=21 seconds; zero curated invariant failures. Wrong answers are reported and are not an acceptance gate.

Full private records: /tmp/lore-goal3-eval/goal-e/. Public compact records exclude questions, SQL, returned rows, customer identifiers, free-form provider text and credentials.

The first six runs measured 723f7b51 and are retained unchanged as the pre-review experiment. Two policy reviews identified ten and eleven confirmed findings, all corrected before final measurement. The corrected application at 5a78252393a1c98dcd1fadfe0f26a4d718db60c6 completed a separate six-run sequence (--series final), with its own experiment-final-manifest.json and final-* private records. Corpus, scoring, clocks, worker count and comparison method are unchanged; no completed run is replaced. Public e-final-* records identify the corrected experiment. All six run receipts confirm completion and owned-process cleanup.

Completed candidate and D3 provider metadata reports glm-5.3 despite the requested glm-5.2 configuration. D2 did not capture the served-model field. Neither metadata nor CLI banners independently attest model identity. Admission observations establish idle CPU at start, not continuous host idleness.

The evaluator runs sequentially with two workers, frozen clocks, identical corrected corpus bytes and no verifier. The requested generator is glm-5.2, but completed candidate and D3 provider records report glm-5.3. D2 lacks served-model metadata. Provider response.model is serving metadata, not independent model attestation. No tests or reviews overlap measured runs. The fresh slice was previously exposed; neither sample is a newly hidden holdout.

Earlier measurement retained

Before review corrections, candidate 723f7b51 produced real correct 4/4/4, wrong 31/38/37; fresh correct 4/6/4, wrong 42/43/45. Both correct intervals against D3 included zero and five repeats exceeded 21 seconds p95. The policy review found concrete defects, so the corrected source was measured in six new predeclared runs. All earlier attempts remain under runs/e-fresh-r* and runs/e-real-r*, with original report/results in pre-review/. No completed run was replaced or restarted for quality.

Independent reviews

{
  "initial_reviews": {
    "policy": {
      "verdict": "FAIL",
      "findings": 10,
      "modifications": 0,
      "receipt": "policy-receipt.json"
    },
    "arithmetic": {
      "verdict": "Arithmetic verified; disclosure correction required",
      "mismatches": 0,
      "modifications": 0,
      "receipt": "arithmetic-receipt.json"
    }
  },
  "confirmed_corrections": [
    {
      "finding": 1,
      "subject": "Equal plan measure labels",
      "correction": "Record merge_measure_class and continue before merge; distinct output columns remain separate.",
      "regression_reference": "test_equal_plan_measure",
      "status": "Implemented; current full-file checks pass after confirmed assertion updates."
    },
    {
      "finding": 2,
      "subject": "Threshold and cross-join false proofs",
      "correction": "Normalize directly negated comparisons; record complex Boolean/algebraic boundaries and dimension constraints; preserve literal wrong-boundary and unconstrained fan-out refusals.",
      "regression_reference": "threshold / cross_join",
      "status": "Implemented; current full-file checks pass after confirmed assertion updates."
    },
    {
      "finding": 3,
      "subject": "Failed-parent pronouns",
      "correction": "Require explicit cohort reference for refusal; record ambiguous pronouns while independent all-client requests proceed.",
      "regression_reference": "independent_question",
      "status": "Implemented; current full-file checks pass after confirmed assertion updates."
    },
    {
      "finding": 4,
      "subject": "Direct and approved CT checks",
      "correction": "Enforce stored-unit, known fan-out, calendar, fact-window and onboarding proofs at final CT validation; enforce stateless date proofs.",
      "regression_reference": "final_ct",
      "status": "Implemented; current full-file checks pass after confirmed assertion updates."
    },
    {
      "finding": 5,
      "subject": "Write and credential request entry",
      "correction": "Reuse historical tested source-intent recognizers with bounded plural/presentation refinements; enforce before model calls at router, entrypoint, direct and approved services, after authorization.",
      "regression_reference": "direct_service_secret_refusal / source_write_refusal / proved_request",
      "status": "Implemented; current full-file checks pass after confirmed assertion updates."
    },
    {
      "finding": 6,
      "subject": "Three-argument calendar bypass",
      "correction": "Preserve SQLGlot PostgreSQL DATE_TRUNC third argument as opaque syntax; keep independent guards running so the pinned BST counterexample refuses.",
      "regression_reference": "three_argument",
      "status": "Implemented; current full-file checks pass after confirmed assertion updates."
    },
    {
      "finding": 7,
      "subject": "Selected or cumulative period output",
      "correction": "Skip zero-fill and record uncertainty for HAVING, QUALIFY, window functions, ranked requests and outer CTE filters.",
      "regression_reference": "having / cumulative / top",
      "status": "Implemented; current full-file checks pass after confirmed assertion updates."
    },
    {
      "finding": 8,
      "subject": "Quarter and rolling month bounds",
      "correction": "Parse quarter phrases before bare years; include the current partial month in rolling calendar bounds.",
      "regression_reference": "quarter / partial_month",
      "status": "Implemented; current full-file checks pass after confirmed assertion updates."
    },
    {
      "finding": 9,
      "subject": "JSON dimensions",
      "correction": "Use stable typed recursive keys without changing returned dimensions; retain uncertainty for unsupported types.",
      "regression_reference": "json_dimensions",
      "status": "Implemented; current full-file checks pass after confirmed assertion updates."
    },
    {
      "finding": 10,
      "subject": "Inventory coverage and conditional proof",
      "correction": "Include plan validator, measure enrichment and source intent; split computed calendar/chip proof branches into proved and unproved rows.",
      "regression_reference": "policy-classification.json",
      "status": "Implemented; current full-file checks pass after confirmed assertion updates."
    }
  ],
  "arithmetic_correction": "Disclose requested glm-5.2 versus observed glm-5.3 provider metadata; D2 has no served-model metadata. No count or interval correction required.",
  "measurement": "Original six candidate-* runs are retained. All six final-* runs completed against source 5a78252393a1c98dcd1fadfe0f26a4d718db60c6, after both rounds of confirmed policy corrections, without infrastructure or quality restarts.",
  "followup_reviews": "Final raw-attempt arithmetic review completed: zero numerical mismatches; no file modifications. One proposed acceptance-logic change was rejected against the owner instruction.",
  "additional_regressions": [
    "Literal off-by-one and dropped count-threshold catches preserved after narrowing uncertain arithmetic to count expressions.",
    "SQL row-limit exhaustion skips period fill and records uncertainty.",
    "PostgreSQL comparison verifies three-argument DATE_TRUNC values through the full post-pass."
  ],
  "followup_policy": {
    "review": "policy-corrections-clean",
    "verdict": "FAIL: eleven additional boundary findings",
    "receipt": "policy-corrections-clean-receipt.json",
    "modifications": 0,
    "confirmed": 11,
    "regressions": "tests/test_insights_goal_e_followup_review.py",
    "red_execution": "followup-review-red: 14 reproduced failures",
    "corrections": [
      "Known threshold boundary remains blocking with an unrelated complex sibling; aggregate count equivalents record uncertainty.",
      "Removed the CT-only calendar early return; ambiguous requested zones no longer skip timezone-independent date proofs.",
      "Final fact-window and endpoint validation applies to approved gold as well as CT execution.",
      "Credential-health relative predicates do not become secret disclosure; directly requested secrets still refuse.",
      "Discarded cohort references do not force inheritance; explicit trades/affiliates/other supported entities retain failed-parent refusal.",
      "Filtered sibling counts require an active condition and no identified input activation restriction before a denominator conviction.",
      "Derived-table aggregate selection excludes zero-fill and records uncertainty.",
      "Named month phrases are extracted before bare years for weekly axes.",
      "Nonnumeric/untyped aggregate outputs and untraced CTE dimensions skip zero-fill with records.",
      "Identical-segment signatures attach unproved evidence to answer and trace without adding a display note.",
      "Proved chip propagation retains its proof flag; a replacement candidate that is merely unproved answers and records. Dynamic and bare exception propagation sites are inventoried."
    ],
    "status": "All eleven confirmed findings corrected and covered by focused regressions; 2007 current checks pass. Original review verdict remains FAIL; no third full policy review was claimed."
  },
  "aborted_followup": {
    "receipt": "policy-corrections-aborted.json",
    "content_used": false,
    "reason": "Caller report preparation had not finished before the snapshot; replaced by a clean review with no file changes."
  },
  "final_source_revision": "5a78252393a1c98dcd1fadfe0f26a4d718db60c6",
  "final_arithmetic": {
    "review": "arithmetic-final",
    "receipt": "arithmetic-final-receipt.json",
    "verdict": "Arithmetic verified; one acceptance interpretation rejected by task owner contract",
    "numerical_mismatches": 0,
    "modifications": 0,
    "raw_compact_attempts_verified": 2272,
    "intervals_verified": 40,
    "family_rows_verified": 76,
    "finding_1": {
      "recommendation": "Remove original 568 verification from acceptance and retain it as coverage only.",
      "decision": "REJECTED",
      "reason": "The owner explicitly requires: \"the 568-check curated invariant set still passes\" and \"zero curated invariant failures\". Current policy-aware replacements cannot silently substitute for the original set. Retaining the explicit failed original-invariant line is required by that instruction. The overall FAIL is unchanged."
    },
    "disclosure_added": "All 51 final guard-attributed refusals retain the oracle label refusal_unverified; classification and test evidence do not independently prove each real refusal justified.",
    "limits": [
      "Arithmetic verifies retained outcome labels, not independent oracle rescoring.",
      "Historical real controls have one repeat each.",
      "Provider and host conditions are confounded; only initial CPU idleness is established.",
      "No third exhaustive policy review or independent model attestation claimed.",
      "The arithmetic reviewer did not rehash the external libfaketime binary."
    ]
  },
  "delivery_lint_correction": {
    "file": "tests/test_insights_rulings_promotion.py",
    "change": "Remove one unused import; test-function ASTs unchanged.",
    "measured_application_source_unchanged": true,
    "changed_python_ruff": "74 files, zero findings",
    "whole_repository_ruff": "501 unchanged historical-evidence findings; full CI cannot reach type checks or tests."
  }
}

Locations

Full attempt records: /tmp/lore-goal3-eval/goal-e/. Compact structural records: docs/evidence/insights-goal-e-2026-09-13/. Retained comparator locations and file hashes are included per run.