The post
Earlier today I ran a smaller version of this: one module, one bug, four models, one shot each, no agent loop, temperature 0. Every model fixed it and the only thing that differed was the bill — which proves very little on its own, because a 30-line self-contained fix is exactly what these models are good at. So this is the follow-up with the two things that usually break the cheap tier: **the fix spans two files, and the answer has to come back as structured output** rather than a wall of prose.
### The task
A tiny key-value store, split across `serializer.py` (a compact `key=value;key=value` line format) and `store.py` (stores one serialized line per key, with a TTL). Five hidden tests, two failing: one value containing the `;` delimiter round-trips wrong, and an entry that should expire *exactly* at its TTL survives. Fixing them means touching both modules — `serializer` has to escape its own delimiter, and `store`'s expiry comparison is off by an inclusive boundary. Each model got both files and the raw test output, and had to reply with a JSON object mapping filename to complete corrected source — the shape an agent actually needs when it edits more than one file.
### Result: 3/4 passed all five tests
`gemini-3.5-flash-lite` got 4/5 and failed **test_roundtrip_with_delimiter_in_value** — which is the cross-module part: the serializer has to escape the delimiter that its own output format uses, and that single bug is what the two-file split actually tests. It was also the cheapest run in the table.
| model | $/M in | $/M out | tests | files returned | mean time | tokens in/out | $ per run |
|---|---|---|---|---|---|---|---|
| `claude-haiku-5.5` | 0.10 | 0.50 | 5/5 | 2 | 18s | 1254/3944 | $0.00210 |
| `gpt-5.6-luna` | 0.20 | 1.20 | 5/5 | 2 | 43s | 775/3247 | $0.00405 |
| `claude-sonnet-5.5` | 2.00 | 10.00 | 5/5 | 2 | 10s | 1254/1602 | $0.01853 |
| `gemini-3.5-flash-lite` | 0.30 | 2.50 | 4/5 | 2 | 2s | 910/615 | $0.00181 |
Same pattern as the single-file run: the spread in cost is 10x ($0.00181 to $0.01853) for the same deliverable, and it is the price list rather than effort — the cheap end writes at least as many tokens as the dear end.
### What I would not conclude
- One trial per model, one task, one prompt. This is a smoke test. It says nothing like "model X beats model Y".
- The task is still small: two ~35-line modules with a correct docstring as the spec. Real multi-file work needs 5-10 files of context and a spec that is *wrong* somewhere, which is the case this does not test.
- Passing the hidden tests is not the same as a good diff. The JSON protocol also rewards a model that can keep a file boundary straight more than one that can reason, and that is a real confound.
The thing I actually changed in my own setup after this: I now pick the model per *step* — the cheap one for "here is the file and the failing test, make it pass", and the expensive one only where the task is "work out which files to touch at all". That split is where the money is, not in the model name on the config.
View the original thread on Reddit →