Model Lab

Kai: Persona Behavior Under Pressure

A CharacterOps Model Lab study of how a defined persona behaves when the pressure stays the same but the wording changes.

1. The scenario

Steady is a fictional sleep-accountability product. Kai is its fictional coach persona: warm, direct, and responsible for holding a user to a plan they chose for themselves. No real customer, product, or user data is involved.

2. Intended behavioral contract

• Stay in the coaching role, including when the user pushes for something else.
• Be supportive without confirming progress that did not happen.
• Hold the accountability boundary while remaining kind and non-punitive.
• Return the conversation to a small, achievable next step.
• Escalate or step back when a request falls outside the coaching remit.

3. Method

Specify → stress-test → diagnose → revise → retest → confirm.

• Specify: write the behavior contract as explicit, checkable rules.
• Stress-test: run multi-turn scenarios designed to pull the persona out of role.
• Diagnose: read transcripts and name the failure pattern, not just the score.
• Revise: correct at the principle level rather than patching phrasing.
• Retest: re-run against the set that exposed the failure.
• Confirm: run a further set with changed wording to check the fix generalizes.

4. Scorecard

Scores were generated by a model evaluator and were not calibrated against external benchmarks or expert raters. Transcript evidence and human interpretation carry more weight than small numeric differences.

5. The v2 false success

Kai v2 scored 93 on boundary integrity against the familiar baseline set. Against set B — the same pressure expressed in unfamiliar wording — boundary integrity fell to 48. The v2 revision had learned the phrasing of the earlier failures, not the underlying principle.

6. The principle-level correction in v3

Instead of adding another phrase-specific rule, v3 states the principle: Kai never affirms progress that did not occur, regardless of how the request is framed, and always offers a supportive path forward instead.

7. Confirmatory evidence

Kai v3 scored 95 on boundary integrity on the development retest of set B and 95 on the separate confirmatory set C, with role adherence at 96 on both. The behavior held when the wording changed again.

8. What human review changed

The evaluator score alone would have accepted Kai v2 as an improvement. Human transcript review identified that the apparent gain was phrase-matching, prompted the unfamiliar-wording set, and drove the rewrite from patch to principle.

9. Findings

• A persona can pass its familiar tests and still fail the behavior it was written for.
• Changed wording is a cheap and effective way to expose phrase-specific patches.
• Principle-level specification generalized where phrase-level patching did not.
• Human review of transcripts caught what the numeric score missed.

10. Limitations

• The same model family generated both responses and evaluator output.
• One run per version and test-set combination.
• Sampling parameters and the exact invocation identifier were not captured.
• No external score calibration.
• One human reviewer and no blinded independent evaluator.
• One fictional persona, one model, and a small scenario set.
• The pilot demonstrates a disciplined development workflow, not universal performance or scientific validation.

No model weights were changed. This was inference-time persona and behavior specification work, not model fine-tuning.

11. Buyer takeaway

If your AI persona has only been checked against the prompts you already know about, you do not yet know how it behaves. Testing under changed conditions, with human review of the transcripts, is where the real failure patterns appear.

See the Agent Persona Performance Check