Prepare
Copy a fresh fixture into a uniquely named workspace.
Experiment 01 · Python agents
Same model.
Different methods.
Ten controlled Python engineering tasks measure whether skills, agent topologies, and trigger rules can reduce cost without sacrificing correctness.
10
Python cases
9
Conditions
3×
Repetitions
270
Planned runs
How one run works
Every attempt begins from the same clean fixture. The agent sees the task and public tests; an external verifier later applies hidden checks and protects the experiment from accidental contamination.
Configure
Pin model, reasoning effort, condition, permissions, and timeout.
Solve
The agent reads TASK.md, edits Python, and runs visible tests.
Verify
Trusted code runs compilation, public, hidden, and scope checks.
Report
Save success, tokens, estimated cost, elapsed time, and traces.
What the solver receives
- The task specification in
TASK.md - The incomplete
challenge/package - Small public tests demonstrating the contract
What remains hidden
- Hidden edge-case and adversarial tests
- The known-good reference implementation
- Final acceptance and edit-scope enforcement
The ten cases
The suite spans parsing, data pipelines, state, concurrency, API contracts, security, and refactoring. Expand any card to see what makes the case difficult and what successful behavior looks like.
T01
Simple
Localized monetary parser
Convert USD, TRY, and EUR strings into exact integer minor units
while rejecting malformed or ambiguous values.
python-data-normalization
Core requirements
- Recognize locale-specific decimal and grouping separators
- Handle signs, parentheses, symbols, and ISO codes
- Use exact decimal arithmetic—never binary float
Why it catches weak solutions
Naive string replacement accepts malformed grouping and can silently produce the wrong number of minor units.
T02
Medium
Faulty CSV import pipeline
Normalize customer data, preserve record line numbers, and
distinguish identical duplicates from conflicting ones.
python-data-normalization
Core requirements
- Support quoting, embedded commas/newlines, and UTF-8 BOM
- Normalize names, emails, IDs, and three date formats
- Preserve deterministic first-occurrence ordering
Why it catches weak solutions
Line numbering becomes subtle when quoted fields contain newlines, and duplicate policy depends on normalized content.
T03
Simple
TTL cache boundary defect
Repair expiration behavior with an injected clock and precise
handling at the exact TTL boundary.
python-systematic-debugging
Core requirements
- Default to a monotonic clock
- Expire when current time is greater than or equal to deadline
- Remove expired entries from storage
Why it catches weak solutions
A one-character boundary error or a wall-clock dependency can pass ordinary tests while remaining incorrect and flaky.
T04
Complex
Asynchronous job-queue race
Make job claiming atomic without serializing independent handler
work, while preserving ownership and cancellation semantics.
python-systematic-debugging
Core requirements
- Prevent two workers from owning the same job
- Enforce retry and owner-only completion rules
- Abandon claims cleanly when processing is cancelled
Why it catches weak solutions
Holding a global lock during the handler removes the race but also destroys the required concurrency.
T05
Medium
Validated project API endpoint
Implement a mutation-safe endpoint with exact field validation,
stable error messages, and duplicate detection.
python-api-contracts
Core requirements
- Collect every validation error in one response
- Reject unknown fields and malformed identifiers
- Never mutate storage for a rejected request
Why it catches weak solutions
Correctness depends on exact response shape and messages, not only on returning a broadly appropriate HTTP status.
T06
Complex
Stable cursor pagination
Add opaque cursor pagination without breaking legacy behavior or
losing records that share a timestamp.
python-api-contracts
Core requirements
- Sort by UTC timestamp and then ID
- Reject corrupted cursors and naive datetimes
- Keep the no-argument legacy list response
Why it catches weak solutions
Timestamp-only cursors skip or repeat tied events, while careless API changes break existing callers.
T07
Medium
Safe ZIP extraction
Validate every archive entry before writing anything, blocking
traversal, symlinks, absolute paths, and encoding tricks.
python-security-hardening
Core requirements
- Treat slash and backslash as path separators
- Detect percent-encoded traversal and drive paths
- Leave no extracted members if any entry is unsafe
Why it catches weak solutions
Simple prefix checks and extractall() miss
platform-specific and symlink-assisted escape paths.
T08
Complex
SQLite query injection
Secure a search query while treating SQL wildcard characters as
literal input and validating pagination controls.
python-security-hardening
Core requirements
- Parameterize all user values
- Select sort identifiers from explicit allowlists
- Escape LIKE wildcards for literal substring search
Why it catches weak solutions
Parameterizing the search term alone does not make unchecked sort identifiers safe or make percent signs literal.
T09
Medium
Behavior-preserving validator refactor
Extract three duplicated name validators into one shared function
without changing any observable behavior.
python-behavior-preserving-refactor
Core requirements
- Keep existing import paths and public function names
- Preserve normalization and exact exception messages
- Actually remove duplicated implementation logic
Why it catches weak solutions
A refactor can pass happy paths but subtly alter exception types, messages, whitespace rules, or compatibility.
T10
Complex
Bounded concurrent file analyzer
Process large iterables concurrently with a fixed worker pool
while preserving result order and cancellation behavior.
python-behavior-preserving-refactor
Core requirements
- Never exceed the requested concurrency
- Do not create one task for every input item
- Capture ordinary failures but propagate cancellation
Why it catches weak solutions
A semaphore around thousands of pre-created tasks limits active calls but still violates the bounded-worker requirement.
No test cases match this filter.
What each run reports
Cost alone is not enough. A cheap failure is not an optimization, so every configuration is evaluated on resource use and executable correctness together.
Raw tokens
Input plus output tokens, without double-counting cached input or reasoning output.
Estimated API cost
Uncached input, cached input, and output are multiplied by a frozen price snapshot.
Success rate
Accepted runs divided by all attempts. Failed attempts remain in the cost total.
Elapsed time
Wall-clock duration captures the latency tradeoff introduced by skills, retries, and additional agents.
Primary efficiency metric
cost per accepted result = total cost of all attempts ÷ accepted runs
Success is deliberately strict
A run is accepted only when all five gates pass. Human impressions can be reported separately, but they never override an executable failure.
All gates, not an average.
One failed hidden test or one protected-file modification makes the entire run unsuccessful.
- Python source compiles
- All public tests pass
- All hidden tests pass
- Protected files remain unchanged
- No files appear outside allowed edit paths