Define a clear quality bar
Make “good” specific enough that a person, an automated evaluator, and an authoring system can all use the same standard.
The decision
We are building a quality system, not running a model comparison.
Trove already moved from Opus to OpenAI. But recreating the quality of our old Opus-era playthroughs would not solve the real problem: those stories were also inconsistent and often not good enough.
Our direction
Write down an absolute definition of “good enough for now,” then improve the Tangle authoring system and runtime prompts until they produce that quality reliably—regardless of which model is running.
Goals
What this work is meant to achieve
Surpass our best Opus-era playthroughs
Raise the floor and the ceiling. We want the new system to reliably produce stories better than even the strongest playthroughs created by our old Tangles and prompts while Opus was running.
Know what to fix
When an output is weak, identify whether the problem came from the Tangle JSON, the runtime story generation, or the results generation.
Improve Tangle authoring
Learn which JSON structures and prompts produce strong outcomes, then use those rules in the Tangle skill and a future one-click authoring tool.
Remove Helen as the long-term bottleneck
Capture her taste in a bounded review set, prove an automated evaluator can apply it, and use human review only for uncertain cases.
Current state
What we already have
Existing material
150total playthroughs
24different Tangles
68human-reviewed plays
64usable Claude reviews
The usable base is 64 completed Claude reviews across 19 Tangles. The four OpenAI feedback entries are excluded from the primary corpus; one is incomplete and four cases add little evidence for this model-independent goal.
The 64 are useful
There is substantial written evidence.
59 cases have playthrough notes, 46 have results-read notes, 43 have per-block feedback, and there are 141 individual block comments. These comments contain concrete lessons even though they do not contain the new dimension scores.
The structured 20
These are 20 of the 64, reviewed more precisely.
Taste Lab selects a representative subset using the old scores, notes, tags, interaction styles, and failure hints. The 20 add six playthrough dimensions, seven results-read dimensions, hard failures, and evidence.
Already exists
AI evaluator prototype
packages/trove-convex/scripts/taste-judge.ts is a working command-line evaluator. It uses Claude Opus 4.8 as the judge, loads the 64 completed Claude reviews, splits them into 48 development and 16 holdout cases, scores overall playthrough and results-read quality from 1–5, and reports agreement.
Still missing: the new three-rubric output and Tangle JSON input. The planned round-three upgrade has not been implemented.
Already exists
One-line new-tangle workflow
It authors JSON, imports to staging, lints, and runs five personas. It is a starting implementation—not the finished north star—because it does not yet apply all three rubrics or return a rubric-based QA report.
Small tool gap
Taste Lab records the likely source of a playthrough problem, but the Results tab does not record whether a results-read defect came from results generation, a weak playthrough, or authored result-page instructions. Add this attribution before the 20 reviews, or capture it explicitly in notes.
How the system works
Use three fixed names for three different objects.
A review must say which object is weak. “The result was bad” is too vague to tell the team what to fix.
Tangle JSON
The authored blueprint stored before play: block order, fixed prose, per-block generation prompts, choices, result-page prompts, and scoring signals.
Playthrough
One completed run: the exact story blocks a player saw, the options they selected, and any responses they typed.
Results read
The complete set of post-play result cards shown after that playthrough. Taste Lab currently calls this tab “Results.”
How a playthrough is produced
playthrough = f(
Tangle JSON,
model,
runtime prompts,
player choices
)Tangle JSONPer-Tangle structure and per-block prompts.
ModelThe model generating dynamic story blocks.
Runtime promptsShared gameplay instructions in code that apply to every Tangle.
Player choicesSelected options and custom text from this specific run.
The same Tangle JSON can produce a different playthrough whenever the model, shared gameplay prompts, or player inputs change.
How the results read is produced
results read = g(
completed playthrough,
Tangle result instructions,
results model,
shared results prompts,
allowed context
)The results read is a second generation step. It reads the completed playthrough, plus the result-page prompts and scoring signals authored in the Tangle JSON. It is not simply the ending of the playthrough.
What “weak” means
A weak results read means the post-play cards scored poorly.
For example: the cards are generic, contradict what the player did, invent motives or facts, repeat one another, or make an overconfident personality claim.
Playthrough defect: the player types a specific answer; the next story block says “Okay” and continues as if it never happened. Investigate the per-block prompt, shared gameplay prompt, or gameplay model.
Results-read defect: the playthrough shows the player asking thoughtful questions; the cards say they avoided curiosity. Investigate result-page instructions, shared results prompts, or the results model.
Important: attribution tells us where to investigate. It becomes a confirmed cause only when a controlled change improves comparable runs.
The three rubrics
Three things to improve means three different standards.
Only the first two are review forms Helen completes in Taste Lab. The third is derived from evidence.
Playthrough rubric
Was this specific completed run a good interactive story?
Helen scores six dimensions: responsiveness and agency, continuity and causality, choice quality, emotional truth, craft and pacing, and momentum and payoff.
Used by: AI evaluator, gameplay prompt tuning, and QA.Results-read rubric
Were this run’s post-play cards accurate, specific, nuanced, useful, resonant, and coherent?
Helen scores seven dimensions and marks hard failures such as invented evidence, harmful identity claims, or contradiction of the playthrough.
Used by: AI evaluator, results prompt tuning, and QA.Tangle JSON authoring rubric
Is this authored blueprint shaped to produce strong playthroughs and results reads reliably?
We derive it by connecting JSON patterns to the first two rubric scores, then test important rules through controlled playthroughs. Helen does not complete another review queue.
Used by:new-tangle, authoring guidance, lint, and QA.
Why they cannot be combined
A Tangle JSON can pass static authoring checks and still produce a weak playthrough. A strong playthrough can still produce a weak results read. Each rubric points to a different system to improve.
How the evidence is used
The 20 add precision. The other 44 still teach us.
20 structured reviews
These receive every new dimension, hard-failure, evidence, and attribution label. They define and calibrate rubric v1.
The remaining 44 reviews
Mine their overall notes, results notes, and block comments for examples and authoring hypotheses. Later, compare evaluator evidence with these notes.
Tangle JSON authoring rules
Connect all 64 cases to JSON by slug. Use the structured 20 as primary evidence and the other 44 as qualitative evidence. Treat patterns as hypotheses until controlled playtests confirm them.
Important decision
We will not ask Helen to rate Tangle JSON separately. We also will not pretend the 64 old reviews contain dimension scores they never captured. Both choices preserve real human evidence without manufacturing false precision.
Why this approach
Each proposed step solves a specific gap.
We propose
Because
Set an absolute quality bar
The playthroughs produced by our old Tangles and prompts while Opus was running were not consistently good enough, and models will change again. A model-independent target gives us a higher bar that survives those changes.
Relabel 20 of the 64 in detail
The 64 contain rich written feedback, but not the six playthrough and seven results-read scores the new evaluator must reproduce. The 20 provide that missing structure.
Freeze the rubric after review
The team needs a stable target. If the definition of good changes during every test, scores cannot be compared and improvements cannot be proven.
Upgrade the existing evaluator
A prototype already exists, but its transcript-only view missed authored-intent defects. It must see JSON, playthrough, and results read, then reproduce the new rubric on unseen cases.
Attach source JSON by slug
We cannot learn how to author better Tangles unless we can connect each outcome to the blueprint that helped produce it.
Derive, rather than rate, JSON rules
A JSON that looks elegant on paper can still play badly. The useful test is what it produces, so we work backward from rated outcomes.
Simplify prose-generation prompts
The current skill may be over-directing the story. Removing instructions that do not earn their place should give the runtime model more room to write naturally.
Work plan
From human taste to a repeatable system
1
Now · human standard
Complete 20 structured reviews
Before reviewing: add results-source attribution so a weak results read can be traced to results generation, an inherited playthrough problem, authored result instructions, mixed causes, or unclear causes.
Then score the selected 20 cases. The old review stays hidden until the new review is saved. Afterwards, use the legacy note to check whether the rubric missed useful language or an example.
Output: frozen rubric v1, the 20-case calibration export, and an example library mined from the 64 legacy reviews.
Done when: every low score and hard failure has evidence, and no unresolved case requires another rubric definition.
2
Next · upgrade and calibrate
Upgrade the existing AI evaluator
Change the prototype so each case includes Tangle JSON, the completed playthrough, and the results read. It must output the same dimensions, hard failures, evidence, and attribution as Taste Lab.
Does it score the same 20? Yes. Split them before tuning: 14 development cases may be scored repeatedly; six sealed holdout cases are scored once after the evaluator is final.
Initial gate: every holdout hard failure caught; matching playthrough verdict on at least 5/6 cases; matching results-read verdict on at least 5/6; at least 80% of dimension scores within one point; every rationale supported by real evidence.
Then: score the remaining 44 as a secondary audit against their old overall scores and written notes—not as dimension-level ground truth.
3
Analysis
Derive—and test—the Tangle JSON authoring rubric
Attach versioned JSON to all 64 cases by slug. Use the structured 20 as primary evidence and the other 44 comments as supporting evidence. Extract candidate rules for story structure, block prompts, result instructions, and shared runtime prompts.
Do not turn correlations directly into canon. Test important rules by changing one authoring variable while holding the premise, path, model, and shared prompts as stable as possible.
Output: a versioned Tangle JSON authoring rubric and standard with lintable rules, generated-output requirements, anti-patterns, evidence, and confidence.
4
One-input authoring
Build the capability, then choose the interface
Input: one-line premise plus optional constraints.
Automated work: generate JSON → lint → import to staging → run five personas → generate every results read → evaluator scores → bounded repair or stop with evidence.
Output: a versioned staging candidate and QA report containing scores, hard failures, evidence, prompt/model versions, cost, and latency.
Is it a skill? The existing new-tangle skill is the first internal implementation candidate. The north star is the capability above. Add an app or button later only if teammates who do not use agents need it.
5
Validate and release
Run a fixed benchmark, then a real canary
Offline: freeze five test premises, generate five new Tangles, and run the five existing personas on each: 25 playthroughs and 25 results reads.
Proposed gate: zero hard failures; no score below 3; at least 80% “love” verdicts for both playthroughs and results reads; all refuser, hostile, custom-input, and preset paths remain coherent. A small blind founder check confirms the new set is preferred to the strongest structured Opus-era examples.
Canary: QA three staging Tangles, release one approved Tangle, and automatically score its first 25 real completed plays. Roll back on an identity/safety failure, repeated broken generation, or confirmed quality-floor regression. Expand after three consecutive canary Tangles pass.
Version separately: Tangle JSON, gameplay prompts, results prompts, evaluator, model, and settings. Shared prompt changes need a stricter canary because they can affect every Tangle.
Can begin in parallel
Mine lessons from the 64 legacy reviews, fetch Tangle JSON by slug, and audit prompt bloat. These do not need to wait for rubric v1, but no prompt change is considered an improvement until it clears the frozen quality gate.
Success criteria
How we will know this worked
“Good” is written clearly enough that a teammate can understand and apply it without reconstructing this discussion.
The upgraded evaluator passes the sealed holdout gate and its remaining limits are documented.
A weak playthrough or weak results read can be traced to a concrete prompt, JSON instruction, generation stage, or an explicitly unclear/mixed cause.
The team has a documented Tangle JSON authoring rubric and standard derived from real outcomes.
The one-input workflow produces a tested staging candidate plus a complete QA report; clean JSON alone never counts as success.
The fixed 25-run benchmark and three canary Tangles clear quality, safety, reliability, cost, and latency gates.
What we are not doing
- Choosing a winner between Claude and OpenAI.
- Asking Helen to review the entire corpus again.
- Pretending the old 64 reviews contain dimension scores that were never collected.
- Adding a separate Taste Lab queue for rating Tangle JSON.
- Fine-tuning a model or building new Python infrastructure.
- Rewriting every existing Tangle before the evidence identifies a real authored defect.
Immediate handoff
The next action is deliberately small.
First add results-source attribution to Taste Lab, then finish and export the selected 20 structured reviews. In parallel, mine the 64 legacy reviews, attach the 19 current Tangle JSON definitions by slug, and audit prompt bloat. Upgrade the existing evaluator after rubric v1 freezes; do not replace it blindly or call the current skill the finished one-click system.