Architecture

Design overview of the create-skill-autoresearch factory.

Pipeline

The factory runs a 5-phase pipeline. Every successful skill build in our case studies followed this exact pattern, regardless of domain.

Agent Roles

The factory orchestrates 4 distinct agent roles. The key architectural constraint is context isolation: the BUILDER and PANEL never share context, preventing bias.

Evaluation Architecture

The evaluation pipeline:

1. evaluate.sh takes a gold standard test case as input
2. Runs the skill on the test case input
3. Uses an LLM-as-judge to compare output to the gold standard reference
4. Scores each rubric dimension independently
5. Emits METRIC <dimension>=<score> lines for the autoresearch skill to parse

Why the factory writes a script instead of judging inline

Committing the measurement to a script — evaluate.sh, plus whatever Python or shell it needs — is a deliberate design choice, not an artifact of the implementation. An agent asked to score its own work inline will score it differently each time, and an improvement loop built on a metric that drifts optimizes noise. A script pins the measurement: same input, same number, every run, and a diff shows exactly when the bar moved. It also makes the deterministic parts genuinely deterministic — line counts, frontmatter validation, link integrity and pattern coverage need no model at all, so the judge is reserved for what actually requires judgment.

The same reasoning drives the METRIC protocol: a scalar on stdout is something the loop can compare across experiments, where prose is not.

What the loop optimizes, and what it does not

overall_score is absolute output quality against your gold standards. It is not a measure of how much the skill helps — that is uplift, measured by a separate benchmark with a bare arm and a with-skill arm. Reading effectiveness off the loop’s metric is a common mistake; see tuning.md.

Consensus Protocol

The verification panel uses a structured protocol inspired by academic research on multi-agent deliberation.

Key design decisions:

• Per-criterion atomic scoring prevents halo effects (Autorubric research)
• Evidence-anchoring requires verbatim quotes for extreme scores (Rulers framework)
• Explicit adversarial assignment achieves 99.2% disagreement detection vs 48.3% for “think critically” (OpenReview research)
• Single synthesis round balances deliberation quality against token cost

Skill Integration

Design Decisions

All 18 locked design decisions are documented in thoughts/07-design-questions.md . Key ones:

• D4: 4-role agent topology (Orchestrator, Researcher, Builder, Panel)
• D5: Structured consensus with synthesis round and escalation
• D7: Two-tier loop budget (per-session + score threshold + plateau detection)
• D8: Shipped skill package vs process artifacts split
• D13: Adaptive data split (70/20/10 for 10+ cases, leave-one-out for fewer)
• D17: Enhance then call autoresearch skill (not embed)
• D18: Incremental build in 5 phases

Case Study Origins

Each factory component traces to a real-world case study: