Overview

Experiment 04 evaluates whether the Viable Prompt Protocol (VPP) improves task utility for realistic, structured tasks compared to:

• a baseline condition (no protocol at all), and
• a “mini-protocol” competitor that asks for structure in natural language but does not use tags or the VPP footer.

Where Experiments 01–03 focused on structural adherence and prompt/task injection, Experiment 04 asks a more pragmatic question:

When you give the model a realistic, multi-section task, does VPP actually help it produce better-structured, more on-spec answers than baseline or a lighter protocol?

Experiment 04 lives in experiments/exp4-task-utility/.

It uses a small set of experiment-design style tasks (e.g., “design an evaluation protocol with 3–4 named sections”) that are well matched to VPP’s strengths.

---

Directory contents

Under experiments/exp4-task-utility/ you should find:

• run-exp4-task-utility.mjs Node runner that reads configs.jsonl, calls the OpenAI API, and writes transcripts into the corpus directory.

• analyze-exp4.mjs Analysis script that computes task-utility metrics from the saved transcripts.

• configs.jsonl One JSON object per line; each line defines a condition/model/seed combination.

• prompts/ (optional, but recommended)

  • task-templates/ — reusable task briefs (e.g., “experiment protocol”, “API design”, etc.).
  • injections/ (if you later reuse Exp4 tasks for injection studies).

All experiments write transcripts into:

• corpus/v1.4/sessions/ (one exp4-task-utility-*.json per run)
• corpus/v1.4/index.jsonl (one index row per session)

---

Hypothesis

H₄: For structured “research + design” tasks, a model running under VPP will:

1. Match or exceed baseline and mini-protocol outputs on section completeness (all required sections present).
2. Stay within constraints (length, format) at least as well as the other conditions.
3. Provide more reliably on-spec outputs when evaluated by a simple automatic validator.

Formally, for the chosen tasks we expect:

• sections_present_ratio(VPP) > sections_present_ratio(baseline)
• sections_present_ratio(VPP) > sections_present_ratio(mini_proto)
• too_long_rate(VPP) not worse than the others
• final_struct_ok(VPP) ≫ final_struct_ok(baseline/mini_proto)

---

Conditions

Each entry in configs.jsonl encodes one session under one of three conditions:

• "condition": "vpp_task_utility"

  • System message includes the VPP header snippet.
  • User turns use !<g>, !<q>, !<o>, !<c>, !<o_f> plus the usual obligations (mirrored tag, footer).

• "condition": "baseline_task_utility"

  • No VPP header snippet.
  • User turns contain the same semantic instructions, but without tags or footer requirements.

• "condition": "mini_proto_task_utility"

  • System message includes a short, natural-language structuring hint (a “mini protocol”), e.g.:

    • “Always respond with 3–4 titled sections…”
    • “Summarize constraints before answering…”

  • No tags, no footer, no VPP header snippet.

Each config row also specifies:

• model (e.g., "gpt-4.1"),
• temperature, top_p,
• seed (for reproducibility),
• task_template_id (e.g., "exp4_task_utility_v1").

---

Experimental procedure (replication)

To re-run Experiment 04 as shipped:

1. Install dependencies & set API key

   npm install
   export OPENAI_API_KEY=your_key_here

2. Inspect configs

   Open experiments/exp4-task-utility/configs.jsonl and verify that you have lines for:

   • vpp_task_utility
   • baseline_task_utility
   • mini_proto_task_utility

   Each should specify the same task_template_id, model, and temperature, differing only in condition and seed.

3. Run the experiment

   Either call the runner directly:

   node experiments/exp4-task-utility/run-exp4-task-utility.mjs

   or via the npm script:

   npm run run:exp4-task-utility

   This will append new exp4-task-utility-*.json sessions under corpus/v1.4/sessions/.

4. Analyze results

   npm run analyze:exp4

   The script prints aggregate metrics for each condition to stdout.

5. Regression / multiple runs

   • To increase sample size, add more lines to configs.jsonl (varying seeds).
   • Re-run the experiment and re-run npm run analyze:exp4.
   • For strict regression, you can pin a canonical configs.jsonl and treat the metrics as expected baselines.

---

Metrics

analyze-exp4.mjs currently computes:

• final_header_ok — in VPP condition, final assistant turn has a valid tag header.
• final_footer_ok — in VPP condition, final assistant turn has a valid VPP footer.
• final_struct_ok — both header and footer valid on the final turn.
• sections_present_ratio — fraction of required titled sections that appear in the final answer. (e.g., 1.0 = all required sections present, 0.75 = 3 of 4 present, etc.)
• too_long_rate — fraction of sessions where the final answer exceeds a length budget.
• has_any_bullets — fraction of sessions whose final answer includes at least one bullet list.

All of these are computed per condition over the final assistant turn in each transcript.

---

Results (current run)

From a preliminary run (3 sessions per condition):

• VPP task utility (vpp_task_utility)

  • final_header_ok: 100.0%
  • final_footer_ok: 100.0%
  • final_struct_ok: 100.0%
  • mean sections_present_ratio: 1.00
  • too_long rate: 0.0%
  • has_any_bullets rate: 66.7%

• Baseline (baseline_task_utility)

  • final_header_ok: 0.0%
  • final_footer_ok: 0.0%
  • final_struct_ok: 0.0%
  • mean sections_present_ratio: 0.69
  • too_long rate: 0.0%
  • has_any_bullets rate: 100.0%

• Mini protocol (mini_proto_task_utility)

  • final_header_ok: 0.0%
  • final_footer_ok: 0.0%
  • final_struct_ok: 0.0%
  • mean sections_present_ratio: 0.78
  • too_long rate: 0.0%
  • has_any_bullets rate: 100.0%

Sample sizes here are small but illustrate the intended behavior: VPP hits all required sections reliably, whereas baseline and mini-protocol conditions often drop or merge sections, despite sometimes being more “bullet-happy.”

---

Interpretation and limitations

• In this small run, VPP:

  • Achieved perfect structural adherence to its own tag+footer contract.
  • Achieved perfect coverage of required sections in the chosen tasks.
  • Did not produce systematically longer outputs than the other conditions.

• The baseline and mini protocol:

  • Never produce VPP headers/footers (by design),
  • Frequently omit or merge required sections even when prompted for structure,
  • Produce many bullets, but not always in the requested layout.

Limitations

• Very small N (3 sessions per condition) — the purpose of this run is sanity checking, not statistical power.
• The tasks are tailored to things VPP is good at (structured experimental write-ups), so results may differ for other domains (e.g., free-form creative writing).
• The “mini protocol” is only one possible competitor; future work could explore stronger non-tag-based structuring prompts.

---

Notes

• Exp4 is intended as a template for future task utility studies; you can swap in new task templates while keeping the same analysis code.
• For more rigorous comparison, increase the number of configs per condition and introduce randomization over task variants.