Build lab · Benchmark 1

Step through all six runs, one phase at a time.

The replay uses the same phases for every model and stops exactly where the reviewer’s evidence stops. It is a compact way to see what worked, what failed, and what was never reached.

Deterministic replay of Benchmark 1. Not a live model, not a live ERP, and not a chatbot: the customer, invoices, and payment below are fictional and every step was recorded from the reviewer’s verified results.

Read the full benchmark

Deterministic replay of Benchmark 1. Not a live model, not a live ERP, and not a chatbot: the customer, invoices, and payment below are fictional and every step was recorded from the reviewer’s verified results.

The scenario being replayed

Fictional fixture — invented customer, invoices, and payment.

Operator request

Aarav Components ka kitna pending hai? Pichle 90 din ke bills dikhao. ₹10,000 HDFC mein mila—oldest bills mein adjust karo.

How much is pending for Aarav Components? Show me the bills from the last 90 days. ₹10,000 arrived in HDFC — adjust it against the oldest bills.

Customer Aarav Components · outstanding ₹25,000 · window 90 days as of 2026-08-17 · payment received ₹10,000 in HDFC Bank

Unpaid bills in window, oldest first

  • INV-447₹7,000
  • INV-451₹12,000
  • INV-459₹6,000

Correct allocation, if the run verifies

  • ₹7,000INV-447
  • ₹3,000INV-451
  • Balance still due₹15,000

Outstanding after allocation: INV-447 ₹0, INV-451 ₹9,000, INV-459 ₹6,000.

Replay a run

Use the arrow keys to change run, and the arrow keys inside the replay panel to move one step at a time.

Replaying

Qwen 3.8 Flash Next FP8

FP8 quantization, xhigh reasoning, agent harness

Live chain verified; operator access blocked
  1. Brief, current step
  2. Read
  3. Backend
  4. Frontend
  5. Tests
  6. Artifact
  7. Harness, verified gap
  8. Result

Step 1 of 8Brief

Step 1 of 8. Brief. Context: The shared brief lands.

Context

The shared brief lands

Same feature brief, same baseline state, this run’s configuration. It received 2 extra human turns—all recorded.

  • Operator request: “Aarav Components ka kitna pending hai? Pichle 90 din ke bills dikhao. ₹10,000 HDFC mein mila—oldest bills mein adjust karo.”
  • Deliverables: outstanding balance for Aarav Components, the last 90 days of unpaid bills, and ₹10,000 allocated oldest-first through the existing payment entry path.
  • Constraints: keep ERPNext permission checks and posting rules, read with the requesting user’s permissions, confirm before any write, and show only numbers that come from the system of record.
  • Requirements and constraints were the same for every run; the serving configuration and the access incidents were not, and both are listed on this card and in interventions.json.
  • Session wall span: 2h 05m 03s. Context only—configuration and concurrency differed between runs.
Config
FP8 quantization, xhigh reasoning, agent harness
Extra human turns
2
Intervention categories
Completion nudge ×2

How to read the replay

Three rules keep the replay from turning every run into an animation of success.

  1. Every run follows the same eight phases

    Brief, codebase read, backend, frontend, assistant tests, assistant artifact, real harness, and verified outcome.

  2. A marker means what it says

    Verified progress was reproduced by the reviewer. A verified gap is a limitation the reviewer confirmed. A stop is the exact evidence boundary.

  3. Stopped runs do not finish

    Partial and incomplete runs halt where their proof ends. The allocation result never appears for them because the real implementation never reached it.

The whole field at a glance

The publication order is fixed and is not a ranking. Each row names the outcome and the phase where evidence stopped.

Read the full benchmark