5 Run Tests

5a. Run tests

For verbose output with per-case scores and evaluator reasoning:

pixie test automatically loads the .env file before running tests.

The evaluation harness:

1. Resolves the Runnable class from the dataset’s runnable field
2. Calls Runnable.create() to construct an instance, then setup() once
3. Runs all dataset entries concurrently (up to 4 in parallel): a. Reads input_data and eval_input from the entry b. Populates the wrap input registry with eval_input data c. Initialises the capture registry d. Validates input_data into the Pydantic model and calls Runnable.run(args) e. wrap(purpose="input") calls in the app return registry values instead of calling external services f. wrap(purpose="output"/"state") calls capture data for evaluation g. Builds Evaluable from captured data h. Runs evaluators
4. Calls Runnable.teardown() once

Because entries run concurrently, the Runnable’s run() method must be concurrency-safe. If you see sqlite3.OperationalError, "database is locked", or similar errors, add a Semaphore(1) to your Runnable (see the concurrency section in Step 2 reference).

5b. Fix mechanical issues only

This step is strictly about fixing what you built in previous steps — the dataset, the runnable, and any custom evaluators. You are fixing mechanical problems that prevent the pipeline from running, NOT assessing or improving the application’s output quality.

What counts as a mechanical issue (fix these):

What is NOT a mechanical issue (do NOT fix these here):

• Application produces wrong/low-quality output → that’s the application’s behavior, analyzed in Step 6
• Evaluator scores are low → that’s a quality signal, analyzed in Step 6
• LLM calls fail inside the application → report in Step 6, do not mock or work around
• Evaluator scores fluctuate between runs → normal LLM non-determinism, not a bug

Iterate — fix errors, re-run, fix the next error — until pixie test runs to completion with real evaluator scores for all entries.

Output

After pixie test completes successfully, results are stored in the per-entry directory structure:

The <test_id> is printed in console output. You will reference this directory in Step 6.

If you hit an unexpected error when running tests (wrong parameter names, import failures, API mismatch), read wrap-api.md, evaluators.md, or testing-api.md for the authoritative API reference before guessing at a fix.