B2 ยท Coding: pilot
Test whether others can use the scheme
A category can seem clear to its author and confuse everyone else. B2 tests the scheme on a small set before it is used across the whole collection.
Where this stage starts
B1 has supplied provisional categories and rules for assigning material to them. B2 starts with those rules, known confusing pairs, and a mix of difficult and randomly drawn examples. A placement assigns a second-order theme to one or more of the common categories.
What the aggregator does
The author records its own answers before it sees anyone else's. Fresh readers then receive the scheme and examples, without the author's working notes or answer key. Three readers examine each pilot item, and their replies show where the rules lead to different readings.
The aggregator asks why each important disagreement arose. Was a rule vague? Did one second-order theme bundle several kinds of action? Did a reader miss evidence? Was the author's first answer wrong? Revisions answer these causes, whatever they do to the agreement score. Each revision to the scheme has a recorded reason.
The pilot also tests whether small changes in the wording of the instructions alter placements, and whether readers answer differently at the start and the end of a batch. The results set how much material later readers receive together.
Rounds stop under the declared rule: no further edit, the declared agreement threshold, or the four-round limit. Any remaining weakness stays in the report. A separate set of reference answers then checks the frozen scheme. These answers are the author's judgments, which the files call "gold" answers. Agreement measures how consistently readers use the scheme; whether its readings are right is a separate question.
What the next stage receives
B3 receives one fixed version of the scheme, a reply format and evidence on batch design. With the rules frozen, disagreements in the full reading can be compared under the same instructions. Later stages can still revise the rules.
What the output files contain
Markdown files hold prose; JSONL files hold one structured record per line.
The reference answersauthor_items.jsonl, gold_items.jsonl
author_items.jsonl records the author's pilot placements
before comparison. gold_items.jsonl holds the separate
reference items and judgments used to assess the frozen scheme as a
declared standard of comparison.
The reader instructionsRETURN-B2.md
RETURN-B2.md defines what a reader must return. It makes
replies comparable and lets the handler tell an incomplete reply from a
real disagreement.
What each round changedpilot_rounds.jsonl, scheme_edits.jsonl, exemplar_candidates.jsonl
pilot_rounds.jsonl records the rounds and their results.
scheme_edits.jsonl links each rule change to its reason.
exemplar_candidates.jsonl lists difficult examples that
might help readers in the full reading.
Stability and the pilot reportstability-B2.md, pilot_report.md, notes-B2.md
stability-B2.md reports how placements changed with the
wording. pilot_report.md gathers agreement, reference
matching, remaining confusions and practical limits.
notes-B2.md records settings, thresholds and decisions.
The frozen scheme and its provenancescheme_final.md, scheme_final.jsonl, run_manifest.json
scheme_final.md is the readable scheme;
scheme_final.jsonl is the same scheme as a structured
record for tools. The stage's run_manifest.json entry
records how the pilot was run, so later results can be tied to the
tested setup.
Full stage description
Read the complete technical card for exact tasks, reading boundaries, checks and closing rules. The explanation above is a guide; the card holds the instructions.
Show how the stage is divided into its 4 legs
B2a: build round. Repeats: round. Interview point: -.B2b: check round. Repeats: round. Interview point: -.B2c: gold and stability build. Repeats: no. Interview point: -.B2d: gold and stability check. Repeats: no. Interview point: -.
Source versions and hashes identify the exact technical documents reproduced here.