Display name Affiliation Conflict statement Save profile

Profile changes are saved to the evaluator API.

Reviewer instructions

EPOB separates human scoring reliability from full experimental reproducibility. Scoring reviewers evaluate frozen run artifacts with the same rubric; replication reviewers rerun the protocol under the same model, seed, harness, container, and resource profile.

Calibration example

Scored calibration example

This example shows how a reviewer should translate packet evidence into scores. It is a teaching example, not a hidden answer key for assigned packets.

Evidence snapshot

The packet contains a dependency-aware plan, role-aligned task assignments, trace events for implementation and review, a final artifact, one unresolved handoff, and two avoidable rework loops.

Reviewer task

Score what is recorded in the packet. Do not use framework reputation, expected model ability, or guesses about work that is not visible in the trace or artifact.

Band rule

Numeric scores use the evidence band first, then the concrete facts in the packet. Low quality is not automatically invalid when the evidence is still comparable.

Option	Example value	How to use every scoring option	Why this example is scored this way
Plan Quality	76	Judge whether the plan decomposes the request, preserves dependencies, and keeps deliverables visible.	The plan is usable and dependency-aware, but it leaves one validation handoff underspecified, so it belongs in the 61-80 band instead of 81-100.
Assignment Quality	82	Check whether specialized work is routed to roles allowed by the packet constraints.	Implementation, review, and validation are assigned to appropriate roles with no material role violation, so this is in the complete role-aligned band.
Coordination	64	Follow handoffs, dependency order, review incorporation, and whether later work uses earlier evidence.	The trace shows ordering and review use, but one review handoff is not closed cleanly, keeping coordination in the mostly coherent 61-80 band.
Deliverable Quality	68	Compare final artifacts against the requested deliverables, constraints, custom rules, and review critique evidence.	The final artifact satisfies the main request and tests, but one optional documentation detail is thin, so it is satisfactory with minor weaknesses.
Efficiency	58	Look for avoidable retries, redundant tasks, dependency churn, or rework relative to packet limits.	The run finishes, but two avoidable rework loops and repeated validation chatter make the execution only moderately efficient.
Confidence	78	Rate your confidence in the score from evidence clarity, not from how good the run is.	The trace and artifact are clear enough for a strong judgment, but the unresolved handoff leaves some uncertainty.
Primary Failure Label	handoff	Choose the single dominant failure mode from the fixed list, or none if no failure class dominates.	The main weakness is the unresolved reviewer-to-implementer handoff; it matters more than decomposition, assignment, or deliverable gaps.
Invalid Run Flag	false	Mark true only when the packet is non-comparable, such as missing required evidence or invalid orchestration execution.	The run is imperfect but comparable because the packet includes trace evidence, artifact evidence, and enough context for numeric scoring.
Rationale	The trace shows a usable plan and role-aligned assignments, but one unresolved handoff and two avoidable rework loops lower coordination and efficiency.	Write a short evidence-grounded explanation that names the facts driving scores, invalid flag, and failure label.	The rationale cites packet evidence and explains both the strengths and the lower scores without invoking framework reputation.

Score band

Start with the rubric band: 61-80 means mostly usable with minor or moderate gaps. Use 81-100 only when the packet is complete, cleanly coordinated, and critique-aware.

Invalid flag

Do not mark invalid just because a run is weak. Mark invalid only when missing evidence or invalid execution makes the run non-comparable.

Failure label

Pick the dominant failure class. This example has several weaknesses, but the handoff issue best explains the score pattern.

One-page instruction checklist

Latest account records load automatically for admin users.

Group access matrix

Group membership controls which role workspaces are available. Admin accounts inherit provider and reviewer access.

Reviewer invitation codes

Generate one-time reviewer invitation codes, set an expiration time, and audit invitation history after use.

Invitation label Expiration preset 1 hour 3 hours 6 hours 1 day 3 days 1 week 2 weeks 1 month 3 months 6 months

Generated invitation code

Generate Save

Invitation history