Evaluate learned knowledge
Test recall, applications, exceptions, and retention without leaking the source into the prompt.
Measure the behavior your application needs, using questions the model did not receive as a source passage during inference. The built-in validation gate helps reject unsuccessful candidates, but it is not a replacement for your own acceptance criteria.
Define tests before learning
For the fictional Northstar returns policy, a small evaluation could include:
| Category | Question | Expected behavior |
|---|---|---|
| Recall | How long is the return window? | State 30 days from delivery |
| Paraphrase | My order arrived three weeks ago; can I start a return? | Apply the window rather than quote unrelated rules |
| Exception | Can I return a final-sale item after ten days? | Recognize the final-sale exception |
| Combination | Is an exchange after 35 days covered by the standard window? | Combine exchange and timing rules |
| Missing fact | What is the returns warehouse street address? | Avoid inventing an address absent from the source |
| Scope | Does this policy cover an order delivered outside the US? | Preserve the stated scope rather than assume worldwide coverage |
These are evaluation expectations, not observed results or a claimed benchmark. Record what the actual model says when execution is enabled.
Keep the test clean
Send only the question and neutral instructions needed for the task. Do not include the source passage, expected answer, or a leading hint that gives away the tested fact. Otherwise you may be measuring in-context reading rather than adapter learning.
Keep temperature and output limits consistent across comparisons. Use enough output allowance for the model to produce a final answer, remembering that the ceiling also includes reasoning. A truncated answer should be distinguished from an incorrect completed answer.
Record the resolved version
For each case, save the test identity, subject, inference job ID, adapter_version, final status, visible answer, and your evaluation judgment. Store answer content in restricted evaluation storage, not general analytics.
If an activation occurs during the run, jobs admitted before and after it can use different versions. Group results by the recorded version rather than by the time you happened to open the dashboard.
Compare before and after
Evaluate a clean subject before learning, then evaluate the learned subject after activation using the same questions. A correct base-model answer to a public fact does not demonstrate that the update taught it. Private or fictional policy details make the change easier to isolate.
For an existing subject, include regression questions from earlier knowledge. Sequential training can change behavior beyond the new facts. A release should pass both the new-knowledge tests and the old behaviors your application still depends on.
Understand the testing boundary
Public inference follows the active adapter. There is no direct candidate-version parameter, so an inactive production candidate cannot be queried through a hidden per-version route. You can rehearse a source on an isolated subject, but that creates a separately trained artifact, not a copy of the production candidate.
For strict releases, gate user traffic, activate deliberately, evaluate the exact resolved version, and roll back to a retained prior version if needed. Keep that limitation visible in your rollout design.
Judge outcomes deliberately
Separate factual correctness, policy application, unsupported additions, and answer usability. Exact string matching works for narrow identifiers but can reject valid paraphrases. A model-based judge also needs review; it is not independent ground truth merely because it returns a score.
If an evaluation fails, retain the evidence, review the source and route, and decide whether to revise knowledge or restore an earlier version. Do not silently add the source to production prompts and still report the result as learned source-free behavior.