Agents on Wagtail
Comparing coding performance, cost and energy
We evaluate the performance of AI agents on Wagtail tasks, so we better understand how to improve DX with AI with our agent skills for developers.
Benchmark scores
Accuracy out of 20 benchmark tasks, with median execution measures per task: cost, energy, carbon footprint, speed.
| Model | Accuracy | Cost | Energy (Wh) | CO₂ (gCO2e) | Speed (m:s) |
|---|---|---|---|---|---|
|
95% |
$0.09 |
14.9 |
5.58 |
11:27 |
|
|
95% |
$0.15 |
– |
– |
3:32 |
|
|
95% |
$0.55 |
– |
– |
5:10 |
|
|
95% |
$0.14 |
– |
– |
1:56 |
|
|
85% |
$1.00 |
34.8 |
12.31 |
8:18 |
|
|
85% |
$0.01 |
– |
– |
3:24 |
|
|
75% |
$0.15 |
– |
– |
19:27 |
|
|
70% |
$0.87 |
56.9 |
19.72 |
18:50 |
|
|
70% |
$1.02 |
– |
– |
13:12 |
|
|
55% |
$0.16 |
26.2 |
1.33 |
13:51 |
|
|
40% |
$0.42 |
8.6 |
0.53 |
19:28 |
|
|
35% |
$0.01 |
2.6 |
0.16 |
5:47 |
|
|
15% |
$0.03 |
– |
– |
19:32 |
Cost: API rates across providers, based on tokens usage. Energy and CO2 emissions: as reported by the Neuralwatt inference provider per the Neuralwatt energy methodology.