| ground truth | LLM average | LLM std. dev. | DeepSeek-V4-Pro | Out tokens | Time | Kimi K2.6 | Out tokens | Time | MiniMax M2.7 | Out tokens | Time | Qwen 3.6 Plus | Out tokens | Time | GLM 5.1 | Out tokens | Time | GPT-5.5 | Out tokens | Time | GPT-5 nano | Out tokens | Time | Claude Opus 4.6 | Out tokens | Time | Claude Haiku 4.5 | Out tokens | Time | Gemini 3.1 Pro Preview | Out tokens | Time | Gemini 3.1 Flash-Lite Preview | Out tokens | Time | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| ENEL - SPA 2022 | 4.84 | 3.92 | 0.80 | 3.98 | 955 | 76.985 | 4.8 | 8838 | 109.704 | 4.31 | 10432 | 98.435 | 3.84 | 7235 | 143.679 | 4.06 | 1513 | 28.072 | 4.47 | 582 | 9.647 | 2.07 | 19 | 1.568 | 4.63 | 1077 | 25.246 | 3.15 | 1730 | 15.607 | 4.47 | 7178 | 56.629 | 3.38 | 661 | 2.923 |
| ENI S.P.A. 2022 | 0.95 | 1.02 | 0.38 | 1.15 | 3247 | 115.611 | 0.97 | 9098 | 126.136 | 1.29 | 16816 | 130.658 | 0.51 | 3996 | 39.129 | 0.96 | 900 | 19.286 | 0.82 | 567 | 10.005 | 1.85 | 18 | 1.371 | 1.05 | 1102 | 26.282 | 1.3 | 1605 | 16.497 | 0.6 | 13867 | 127.272 | 0.68 | 631 | 3.29 |
| A2A ENERGIA S.P.A. 2022 | 3.47 | 3.42 | 0.58 | 3.92 | 1096 | 43.436 | 3.27 | 5066 | 74.471 | 3.96 | 11668 | 79.672 | 3.03 | 4662 | 59.28 | 3.17 | 1230 | 29.477 | 3.03 | 705 | 14.915 | 2.66 | 18 | 1.294 | 4.74 | 1261 | 32.845 | 3.3 | 1461 | 12.601 | 3.51 | 16486 | 153.474 | 3.04 | 722 | 3.512 |
| "TERNA - RETE ELETTRICA NAZIONALE SOCIETA' PER AZIONI" (IN FORMA ABBREVIATA "TERNA S.P.A.") 2022 | 4.35 | 4.74 | 0.44 | 4.52 | 886 | 64.905 | 4.71 | 5198 | 81.314 | 4.98 | 16262 | 169.171 | 4.59 | 4765 | 100.258 | 4.45 | 921 | 20.25 | 4.45 | 683 | 10.897 | 4.12 | 18 | 1.288 | 5.64 | 1119 | 26.285 | 4.98 | 1169 | 10.78 | 4.46 | 5841 | 47.406 | 5.27 | 683 | 3.679 |
| REDEIA CORPORACION SA 2022 | 3.21 | 3.63 | 0.90 | 4.5 | 3395 | 83.75 | 4.28 | 6498 | 77.671 | 3.55 | 18503 | 117.335 | 4.28 | 2584 | 35.006 | 3.79 | 808 | 15.629 | 3.1 | 641 | 10.189 | 1.36 | 18 | 1.423 | 4.28 | 1196 | 28.645 | 3.26 | 1461 | 13.345 | 3.32 | 14688 | 115.561 | 4.19 | 705 | 3.347 |
| EDP, S.A. 2022 | 4.06 | 3.89 | 0.75 | 4.1 | 2025 | 91.743 | 4.26 | 13000 | 222.821 | 3.53 | 10491 | 71.597 | 3.65 | 5529 | 86.191 | 3.59 | 1712 | 36.26 | 4.29 | 950 | 16.823 | 2.31 | 18 | 1.53 | 4.32 | 1077 | 27.89 | 5.15 | 2509 | 21.904 | 4.37 | 24275 | 185.553 | 3.2 | 933 | 3.783 |
| ENGIE 2022 | 3.25 | 3.50 | 1.53 | 3.84 | 26300 | 529.061 | 3.08 | 10542 | 208.909 | 5.31 | 23516 | 189.136 | 2.63 | 8970 | 175.95 | 2.01 | 1213 | 19.99 | 2.53 | 883 | 14.841 | 7.12 | 18 | 1.348 | 4.04 | 1319 | 33.252 | 2.94 | 1364 | 13.121 | 2.79 | 16262 | 127.922 | 2.24 | 624 | 3.743 |
| IBERDROLA SA 2022 | 3.91 | 4.34 | 0.85 | 5.39 | 1772 | 70.693 | 4.22 | 13457 | 251.056 | 5.23 | 14606 | 134.174 | 4.27 | 6803 | 141.463 | 4.93 | 1643 | 37.156 | 3.5 | 727 | 11.602 | 2.67 | 18 | 1.448 | 4.28 | 1191 | 28.054 | 4.7 | 1392 | 13.112 | 3.5 | 15537 | 154.421 | 5.08 | 672 | 2.953 |
| NATURGY ENERGY GROUP SA 2022 | 2.67 | 2.66 | 0.52 | 2.43 | 1404 | 115.062 | 2.49 | 7032 | 90.409 | 2.11 | 28467 | 235.013 | 2.49 | 2297 | 25.074 | 2.53 | 833 | 14.934 | 2.49 | 539 | 9.781 | 2.25 | 18 | 1.185 | 3.72 | 1175 | 29.93 | 3.64 | 1747 | 16.828 | 2.55 | 8218 | 66.987 | 2.6 | 716 | 3.228 |
| TenneT Holding B.V. 2022 | 83.85 | 30.23 | 28.74 | 86.48 | 819 | 36.853 | 53.79 | 18570 | 275.97 | 8.66 | 40897 | 421.742 | 9.76 | 8875 | 165.954 | 23.15 | 1440 | 33.237 | 51.3 | 717 | 11.222 | 3.12 | 18 | 1.079 | 65.26 | 1192 | 28.738 | 13.1 | 1276 | 11.63 | 9.38 | 17202 | 141.717 | 8.5 | 818 | 3.482 |
| VEOLIA ENVIRONNEMENT 2022 | 3.92 | 3.74 | 0.62 | 3.73 | 1179 | 57.147 | 4.57 | 12639 | 222.95 | 4.09 | 16516 | 142.798 | 3.96 | 4822 | 112.861 | 3.51 | 977 | 23.6 | 4.48 | 996 | 15.534 | 2.47 | 18 | 1.357 | 3.96 | 1877 | 43.268 | 3.4 | 1797 | 15.696 | 4.0 | 17875 | 157.033 | 2.99 | 845 | 3.918 |
| ØRSTED A/S 2022 | 2.5 | 2.72 | 1.30 | 4.79 | 1995 | 67.366 | 2.04 | 11520 | 187.493 | 1.91 | 12847 | 98.873 | 2.79 | 6364 | 92.347 | 2.16 | 1307 | 22.811 | 3.08 | 594 | 11.798 | 4.86 | 18 | 1.348 | 3.75 | 1233 | 29.043 | 0.68 | 1659 | 15.593 | 2.04 | 11461 | 90.39 | 1.84 | 686 | 3.871 |
| ELECTRICITE DE FRANCE 2022 | -8.22 | -43.21 | 60.73 | -150.33 | 1057 | 55.562 | -31.9 | 9431 | 128.32 | 51.38 | 11724 | 104.161 | -25.12 | 5188 | 53.687 | -19.55 | 914 | 23.922 | -35.82 | 616 | 10.421 | 0.62 | 18 | 1.343 | -41.13 | 1176 | 28.824 | -157.68 | 2005 | 20.174 | -24.21 | 23316 | 215.651 | -41.59 | 768 | 3.543 |
| ERG S.P.A. 2022 | not available | 5.04 | 2.45 | 4.38 | 2152 | 108.68 | 4.61 | 15878 | 279.11 | 11.76 | 17733 | 153.281 | 4.35 | 4647 | 64.688 | 4.23 | 1330 | 30.007 | 3.76 | 618 | 10.427 | 2.23 | 18 | 3.141 | 4.23 | 1174 | 25.126 | 6.17 | 1311 | 12.565 | 4.01 | 9548 | 76.165 | 5.7 | 745 | 3.242 |
| Acciona SA 2022 | not available | 2.82 | 0.23 | 2.66 | 1074 | 57.06 | 2.78 | 14482 | 185.079 | 2.41 | 9748 | 77.849 | 2.66 | 7214 | 71.779 | 3.1 | 1017 | 15.983 | 2.68 | 660 | 10.352 | 3.12 | 18 | 1.181 | 2.85 | 1183 | 31.757 | 3.17 | 1459 | 14.16 | 2.79 | 20618 | 158.771 | 2.79 | 718 | 2.971 |
| Fortum Oyj 2022 | 1.84 | 6.03 | 11.67 | 3.16 | 882 | 34.265 | 1.59 | 10750 | 160.731 | 2.54 | 22520 | 179.974 | 1.59 | 4657 | 83.27 | 3.98 | 1365 | 34.194 | 1.59 | 481 | 8.864 | 0.6 | 18 | 1.127 | 1.8 | 1093 | 27.753 | 7.04 | 1744 | 16.89 | 1.58 | 9989 | 92.446 | 40.81 | 714 | 3.971 |
| VERBUND AG 2022 | 1.6 | 1.24 | 0.47 | 0.95 | 1181 | 73.358 | 1.12 | 7164 | 92.256 | 1.04 | 12189 | 79.665 | 1.12 | 4357 | 53.531 | 1.16 | 985 | 23.664 | 1.12 | 649 | 10.14 | 2.63 | 18 | 1.376 | 1.28 | 1220 | 30.07 | 1.18 | 2203 | 19.801 | 1.11 | 11167 | 101.051 | 0.98 | 746 | 3.211 |
| ITALGAS S.P.A. 2022 | not available | 5.13 | 0.92 | 5.93 | 2024 | 67.579 | 5.33 | 11048 | 144.057 | 5.49 | 14845 | 131.42 | 3.83 | 5871 | 121.222 | 5.83 | 1184 | 25.091 | 5.61 | 751 | 10.449 | 3.04 | 18 | 1.309 | 5.5 | 1148 | 27.687 | 4.57 | 1502 | 14.085 | 5.63 | 8020 | 81.012 | 5.63 | 820 | 3.333 |
| ACEA S.P.A. 2022 | not available | 3.55 | 0.50 | 3.82 | 892 | 44.877 | 3.75 | 8391 | 129.337 | 3.18 | 16395 | 166.619 | 3.99 | 3700 | 51.161 | 3.45 | 854 | 18.21 | 3.75 | 708 | 11.877 | 2.33 | 18 | 1.166 | 4.17 | 1102 | 26.571 | 3.33 | 1256 | 11.668 | 3.74 | 11127 | 85.655 | 3.5 | 751 | 3.056 |
| REN - REDES ENERGÉTICAS NACIONAIS, SGPS, S.A. 2022 | 4.37 | 4.41 | 0.75 | 4.54 | 5384 | 115.253 | 4.16 | 8680 | 126.941 | 4.09 | 7162 | 88.466 | 4.49 | 5615 | 106.237 | 4.53 | 1176 | 30.6 | 4.36 | 711 | 11.022 | 2.49 | 18 | 1.214 | 5.05 | 1174 | 26.757 | 5.36 | 1387 | 12.827 | 4.38 | 11246 | 87.022 | 5.11 | 576 | 2.398 |
| HERA S.P.A. 2022 | 4.23 | 4.01 | 1.18 | 3.8 | 1716 | 60.867 | 2.82 | 6115 | 78.343 | 4.5 | 17347 | 145.7 | 3.73 | 4277 | 74.57 | 4.31 | 1037 | 24.107 | 5.07 | 777 | 12.246 | 2.28 | 18 | 1.141 | 6.43 | 1115 | 24.785 | 4.31 | 1521 | 14.358 | 4.28 | 13101 | 108.759 | 2.59 | 759 | 3.046 |
| VINCI 2022 | 1.89 | 2.19 | 0.70 | 2.05 | 781 | 20.053 | 2.33 | 9687 | 153.997 | 1.95 | 18688 | 206.78 | 1.85 | 4905 | 61.848 | 1.77 | 1099 | 25.806 | 1.97 | 541 | 9.247 | 4.23 | 18 | 1.259 | 2.04 | 1161 | 29.143 | 2.1 | 1018 | 9.93 | 1.96 | 13729 | 110.576 | 1.8 | 780 | 3.84 |
| BOUYGUES 2022 | 3.26 | 1.54 | 0.90 | 2.35 | 1397 | 64.841 | 2.21 | 6682 | 87.259 | 0.62 | 17862 | 167.981 | 2.36 | 4225 | 58.475 | 0.55 | 741 | 18.833 | 2.21 | 524 | 8.693 | 0.61 | 18 | 1.406 | 2.58 | 1222 | 29.342 | 0.61 | 1044 | 9.114 | 2.21 | 10342 | 91.185 | 0.68 | 865 | 3.484 |
| FERROVIAL SA 2022 | not available | 8.28 | 2.82 | 10.14 | 1211 | 58.405 | 10.42 | 13922 | 204.105 | 9.38 | 5486 | 56.782 | 9.45 | 5702 | 112.883 | 4.57 | 1333 | 32.208 | 7.26 | 781 | 14.52 | 3.1 | 18 | 1.846 | 12.04 | 1244 | 28.756 | 6.6 | 1725 | 16.685 | 6.92 | 11591 | 106.057 | 11.19 | 847 | 3.858 |
| Failures | Mean Absolute Error (ratio) | Mean Bias Error (ratio) | Root Mean Squared Error (ratio) | Time (total) | Time (average) | Time (std. dev.) | Cost (total) | Cost (average) | Cost (std. dev.) | |
|---|---|---|---|---|---|---|---|---|---|---|
| DeepSeek-V4-Pro | 0 | 1.133 | -0.803 | 3.980 | 2113.41 | 88.06 | 97.43 | 1.52 | 0.06 | 0.02 |
| Kimi K2.6 | 0 | 0.301 | -0.199 | 0.688 | 3698.44 | 154.10 | 65.08 | 1.69 | 0.07 | 0.02 |
| MiniMax M2.7 | 0 | 0.647 | 0.352 | 1.703 | 3447.28 | 143.64 | 75.16 | 0.69 | 0.03 | 0.01 |
| Qwen 3.6 Plus | 0 | 0.294 | -0.227 | 0.546 | 2090.54 | 87.11 | 41.11 | 0.83 | 0.03 | 0.01 |
| GLM 5.1 | 0 | 0.316 | -0.138 | 0.507 | 603.33 | 25.14 | 6.63 | 1.18 | 0.05 | 0.01 |
| GPT-5.5 | 0 | 0.314 | -0.241 | 0.790 | 275.51 | 11.48 | 2.23 | 4.21 | 0.18 | 0.03 |
| GPT-5 nano | 0 | 0.636 | -0.001 | 0.722 | 33.75 | 1.41 | 0.40 | 0.04 | 0.00 | 0.00 |
| Claude Opus 4.6 | 0 | 0.414 | -0.081 | 0.954 | 696.05 | 29.00 | 3.76 | 4.87 | 0.20 | 0.03 |
| Claude Haiku 4.5 | 0 | 1.368 | -0.890 | 4.238 | 348.97 | 14.54 | 3.19 | 1.02 | 0.04 | 0.01 |
| Gemini 3.1 Pro Preview | 0 | 0.250 | -0.227 | 0.513 | 2738.72 | 114.11 | 41.41 | 5.63 | 0.23 | 0.06 |
| Gemini 3.1 Flash-Lite Preview | 0 | 1.605 | 0.728 | 4.960 | 81.68 | 3.40 | 0.39 | 0.25 | 0.01 | 0.00 |
| ground truth | LLM average | LLM std. dev. | DeepSeek-V4-Pro | Out tokens | Time | Kimi K2.6 | Out tokens | Time | MiniMax M2.7 | Out tokens | Time | Qwen 3.6 Plus | Out tokens | Time | GLM 5.1 | Out tokens | Time | GPT-5.5 | Out tokens | Time | GPT-5 nano | Out tokens | Time | Claude Opus 4.6 | Out tokens | Time | Claude Haiku 4.5 | Out tokens | Time | Gemini 3.1 Pro Preview | Out tokens | Time | Gemini 3.1 Flash-Lite Preview | Out tokens | Time | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| ENEL - SPA 2022 | 0.1567 | 0.15 | 0.05 | 0.1402 | 1032 | 67.124 | 0.136 | 12762 | 209.764 | 0.1624 | 19495 | 241.67 | 0.1501 | 9441 | 268.714 | 0.1778 | 1040 | 23.964 | 0.1402 | 718 | 12.606 | 0.0522 | 19 | 1.668 | 0.1542 | 1278 | 29.941 | 0.2429 | 1531 | 13.965 | 0.1668 | 12963 | 103.591 | 0.1779 | 915 | 3.664 |
| ENI S.P.A. 2022 | 0.6593 | 0.67 | 0.19 | 0.7607 | 895 | 87.124 | 0.561 | 15321 | 208.362 | 0.7145 | 18214 | 228.814 | 0.8858 | 7960 | 237.365 | 0.7081 | 1249 | 23.596 | 0.7452 | 669 | 11.092 | 0.1564 | 19 | 1.517 | 0.6083 | 1434 | 36.825 | 0.7759 | 2186 | 19.035 | 0.6116 | 29271 | 267.189 | 0.7882 | 826 | 3.384 |
| A2A ENERGIA S.P.A. 2022 | 0.2317 | 0.23 | 0.05 | 0.2354 | 737 | 21.125 | 0.2699 | 14200 | 224.06 | 0.198 | 11336 | 141.179 | 0.3059 | 5742 | 85.405 | 0.2425 | 1452 | 31.079 | 0.2354 | 526 | 7.561 | 0.1785 | 19 | 2.94 | 0.1733 | 1268 | 30.401 | 0.3012 | 1455 | 13.401 | 0.1601 | 19048 | 158.328 | 0.249 | 697 | 3.029 |
| "TERNA - RETE ELETTRICA NAZIONALE SOCIETA' PER AZIONI" (IN FORMA ABBREVIATA "TERNA S.P.A.") 2022 | 0.1724 | 0.15 | 0.03 | 0.182 | 3723 | 129.794 | 0.1727 | 13846 | 270.397 | 0.1462 | 11963 | 100.884 | 0.1606 | 4766 | 82.341 | 0.1336 | 1007 | 31.06 | 0.1664 | 718 | 12.304 | 0.0638 | 19 | 1.24 | 0.1304 | 1230 | 26.757 | 0.1521 | 1283 | 12.561 | 0.1711 | 18587 | 150.615 | 0.1343 | 845 | 2.999 |
| REDEIA CORPORACION SA 2022 | 0.2063 | 0.16 | 0.04 | 0.1881 | 829 | 44.576 | 0.1579 | 15338 | 263.508 | 0.181 | 21187 | 264.874 | 0.1603 | 2911 | 53.512 | 0.164 | 1298 | 26.928 | 0.1881 | 680 | 10.118 | 0.0443 | 19 | 2.162 | 0.1321 | 1187 | 23.809 | 0.2115 | 1341 | 11.267 | 0.1802 | 10235 | 98.221 | 0.1579 | 669 | 2.917 |
| EDP, S.A. 2022 | 0.1884 | 0.18 | 0.08 | 0.1486 | 1672 | 81.761 | 0.18 | 18491 | 252.437 | 0.1028 | 27065 | 200.3 | 0.1455 | 5046 | 83.792 | 0.231 | 1193 | 28.475 | 0.1744 | 1072 | 17.137 | 0.12 | 19 | 1.289 | 0.1635 | 1543 | 36.682 | 0.2163 | 1391 | 13.577 | 0.1158 | 21013 | 149.086 | 0.3934 | 801 | 3.275 |
| ENGIE 2022 | 0.2483 | 0.33 | 0.13 | 0.474 | 1109 | 56.044 | 0.3776 | 16423 | 326.594 | 0.2608 | 20462 | 254.087 | 0.3816 | 8089 | 101.884 | 0.3342 | 1386 | 27.408 | 0.3152 | 703 | 11.347 | 0.0702 | 19 | 1.816 | 0.207 | 5080 | 114.5 | 0.5137 | 2045 | 20.826 | 0.2834 | 28576 | 277.494 | 0.4235 | 829 | 2.994 |
| IBERDROLA SA 2022 | 0.1965 | 0.19 | 0.05 | 0.2624 | 838 | 54.362 | 0.1804 | 28756 | 578.073 | 0.1722 | 17164 | 207.134 | 0.1743 | 6970 | 122.904 | 0.2096 | 1185 | 32.644 | 0.2185 | 874 | 14.883 | 0.068 | 19 | 1.428 | 0.2252 | 1319 | 34.66 | 0.2122 | 1524 | 14.428 | 0.194 | 28629 | 271.085 | 0.1868 | 976 | 4.29 |
| NATURGY ENERGY GROUP SA 2022 | 0.2847 | 0.27 | 0.07 | 0.3352 | 1292 | 90.51 | 0.2981 | 13708 | 231.776 | 0.3321 | 11222 | 161.586 | 0.2981 | 5983 | 150.177 | 0.2102 | 1274 | 29.68 | 0.2979 | 644 | 11.051 | 0.1332 | 19 | 1.24 | 0.2785 | 1378 | 33.322 | 0.165 | 1447 | 12.73 | 0.3048 | 18538 | 148.398 | 0.2981 | 626 | 2.852 |
| TenneT Holding B.V. 2022 | -0.01 | 0.05 | 0.15 | -0.0105 | 2176 | 52.975 | 0.0567 | 27150 | 376.083 | 0.076 | 19811 | 225.647 | -0.0113 | 9140 | 123.781 | -0.0086 | 1064 | 23.526 | 0.0214 | 1012 | 16.028 | -0.1493 | 19 | 1.223 | 0.0106 | 1287 | 633.845 | 0.4555 | 1961 | 17.024 | 0.076 | 29435 | 240.854 | -0.0141 | 707 | 3.401 |
| VEOLIA ENVIRONNEMENT 2022 | 0.1972 | 0.19 | 0.06 | 0.2021 | 1179 | 80.444 | 0.191 | 19236 | 290.18 | 0.2135 | 22468 | 155.548 | 0.1949 | 5698 | 89.312 | 0.2893 | 1464 | 27.729 | 0.1654 | 1125 | 17.258 | 0.0451 | 19 | 1.401 | 0.1677 | 6431 | 130.498 | 0.2164 | 2162 | 19.071 | 0.1688 | 22205 | 178.886 | 0.2327 | 933 | 3.609 |
| ØRSTED A/S 2022 | 0.3124 | 0.28 | 0.20 | 0.3409 | 1158 | 50.269 | 0.1391 | 12982 | 178.132 | 0.3344 | 23462 | 434.064 | 0.2134 | 8634 | 207.24 | 0.0489 | 1773 | 30.069 | 0.1741 | 672 | 10.628 | 0.0396 | 19 | 1.298 | 0.2941 | 1180 | 28.733 | 0.3821 | 1398 | 13.719 | 0.7479 | 19525 | 154.184 | 0.3721 | 851 | 3.22 |
| ELECTRICITE DE FRANCE 2022 | -0.1579 | 0.01 | 0.18 | -0.0275 | 986 | 56.014 | -0.0157 | 20697 | 315.144 | 0.5349 | 24916 | 233.917 | -0.0854 | 5692 | 97.89 | -0.0302 | 1174 | 29.429 | -0.0501 | 702 | 12.977 | -0.0689 | 19 | 1.482 | 0.0059 | 10228 | 219.446 | -0.0706 | 1222 | 12.063 | -0.0711 | 26451 | 207.937 | -0.0095 | 890 | 3.824 |
| ERG S.P.A. 2022 | not available | 0.15 | 0.07 | 0.1594 | 1041 | 62.311 | 0.1688 | 22142 | 432.758 | -0.0109 | 34987 | 336.558 | 0.1688 | 5547 | 73.184 | 0.1868 | 1332 | 32.432 | 0.1895 | 751 | 12.057 | 0.0473 | 19 | 1.295 | 0.162 | 1262 | 29.34 | 0.202 | 1620 | 14.092 | 0.189 | 16354 | 122.554 | 0.1654 | 785 | 3.132 |
| Acciona SA 2022 | not available | 0.27 | 0.06 | 0.3179 | 1529 | 104.338 | 0.2074 | 14772 | 203.557 | 0.3141 | 22852 | 264.567 | 0.2372 | 8415 | 249.462 | 0.3395 | 1313 | 28.759 | 0.2953 | 736 | 11.766 | 0.1746 | 19 | 1.383 | 0.209 | 1155 | 29.12 | 0.2324 | 1333 | 11.715 | 0.2899 | 19093 | 152.721 | 0.3218 | 887 | 3.4 |
| Fortum Oyj 2022 | 0.4556 | 0.39 | 0.22 | 0.5331 | 766 | 52.503 | 0.5174 | 14470 | 194.797 | 0.182 | 33739 | 404.076 | 0.6213 | 6963 | 116.531 | 0.4499 | 1176 | 25.386 | 0.5331 | 719 | 12.249 | -0.1472 | 19 | 1.321 | 0.4599 | 1112 | 28.792 | 0.3936 | 1660 | 15.428 | 0.5594 | 12088 | 119.613 | 0.2406 | 688 | 2.569 |
| VERBUND AG 2022 | 0.5174 | 0.79 | 0.22 | 1.0054 | 1245 | 55.183 | 0.7847 | 11562 | 243.68 | 0.67 | 15860 | 208.625 | 0.9896 | 7676 | 140.69 | 0.7847 | 928 | 18.619 | 0.7847 | 791 | 12.284 | 0.2324 | 19 | 1.359 | 0.6895 | 1161 | 39.048 | 1.0051 | 1480 | 12.955 | 0.7945 | 24318 | 185.043 | 0.934 | 694 | 3.338 |
| ITALGAS S.P.A. 2022 | not available | 0.15 | 0.02 | 0.1442 | 848 | 28.175 | 0.1485 | 10145 | 142.507 | 0.1426 | 22724 | 193.627 | 0.1471 | 8927 | 165.197 | 0.1445 | 897 | 19.99 | 0.1442 | 656 | 10.062 | 0.1589 | 19 | 3.11 | 0.1383 | 1226 | 28.88 | 0.2213 | 1426 | 12.537 | 0.148 | 10919 | 76.394 | 0.1509 | 912 | 3.521 |
| ACEA S.P.A. 2022 | not available | 0.20 | 0.02 | 0.2128 | 4146 | 86.276 | 0.1773 | 13996 | 204.731 | 0.2024 | 30799 | 1014.628 | 0.1833 | 5601 | 111.654 | 0.1825 | 921 | 20.62 | 0.2117 | 637 | 10.701 | 0.1651 | 19 | 3.737 | 0.2011 | 1122 | 27.772 | 0.2068 | 1186 | 9.59 | 0.1978 | 30303 | 274.05 | 0.2108 | 651 | 3.236 |
| REN - REDES ENERGÉTICAS NACIONAIS, SGPS, S.A. 2022 | 0.1686 | 0.18 | 0.08 | 0.1604 | 3746 | 83.923 | 0.1693 | 11639 | 248.264 | 0.1941 | 9051 | 70.956 | 0.1881 | 1846 | 38.74 | 0.4032 | 1443 | 37.303 | 0.1821 | 758 | 10.297 | 0.0639 | 19 | 1.262 | 0.1436 | 1087 | 26.929 | 0.1503 | 1304 | 11.405 | 0.1735 | 24002 | 247.025 | 0.1495 | 799 | 3.139 |
| HERA S.P.A. 2022 | 0.1733 | 0.19 | 0.04 | 0.1891 | 1183 | 48.753 | 0.2422 | 17295 | 224.264 | 0.1721 | 21351 | 201.873 | 0.211 | 6909 | 94.998 | 0.1816 | 1125 | 24.893 | 0.1986 | 788 | 12.695 | 0.1109 | 19 | 1.163 | 0.1394 | 1079 | 25.52 | 0.2568 | 1417 | 13.315 | 0.1694 | 11329 | 88.904 | 0.203 | 798 | 3.09 |
| VINCI 2022 | 0.4221 | 0.37 | 0.07 | 0.398 | 791 | 50.82 | 0.352 | 16924 | 239.231 | 0.4192 | 26298 | 469.698 | 0.4159 | 5537 | 109.558 | 0.4085 | 994 | 26.212 | 0.3981 | 633 | 9.691 | 0.1839 | 19 | 2.707 | 0.3131 | 1360 | 34.652 | 0.3947 | 1072 | 10.343 | 0.3982 | 19956 | 181.581 | 0.3897 | 936 | 3.253 |
| BOUYGUES 2022 | 0.2446 | 0.64 | 0.53 | 0.3502 | 2464 | 68.668 | 0.377 | 8125 | 117.882 | 0.3754 | 19794 | 222.652 | 0.3334 | 5492 | 97.145 | 1.3709 | 764 | 15.063 | 0.3938 | 653 | 10.171 | 0.2021 | 19 | 1.376 | 0.2454 | 1136 | 26.434 | 1.4638 | 1348 | 12.806 | 0.377 | 7443 | 58.582 | 1.5515 | 860 | 3.674 |
| FERROVIAL SA 2022 | not available | 0.07 | 0.02 | 0.0564 | 1466 | 69.588 | 0.1002 | 17668 | 277.744 | 0.055 | 19103 | 222.74 | 0.0874 | 8199 | 106.594 | 0.1034 | 1021 | 22.109 | 0.0462 | 756 | 12.897 | 0.0796 | 19 | 1.513 | 0.0622 | 1245 | 30.39 | 0.0564 | 1555 | 15.06 | 0.1029 | 11889 | 95.302 | 0.0473 | 761 | 3.157 |
| Failures | Mean Absolute Error (ratio) | Mean Bias Error (ratio) | Root Mean Squared Error (ratio) | Time (total) | Time (average) | Time (std. dev.) | Cost (total) | Cost (average) | Cost (std. dev.) | |
|---|---|---|---|---|---|---|---|---|---|---|
| DeepSeek-V4-Pro | 0 | 0.252 | 0.193 | 0.389 | 1582.66 | 65.94 | 24.04 | 1.43 | 0.06 | 0.01 |
| Kimi K2.6 | 0 | 0.594 | 0.448 | 1.569 | 6253.92 | 260.58 | 96.45 | 2.27 | 0.09 | 0.02 |
| MiniMax M2.7 | 0 | 0.846 | 0.676 | 2.228 | 6459.70 | 269.15 | 184.97 | 0.83 | 0.03 | 0.01 |
| Qwen 3.6 Plus | 0 | 0.254 | 0.133 | 0.334 | 3008.07 | 125.34 | 60.22 | 0.92 | 0.04 | 0.01 |
| GLM 5.1 | 0 | 0.550 | 0.384 | 1.156 | 636.97 | 26.54 | 5.07 | 1.19 | 0.05 | 0.01 |
| GPT-5.5 | 0 | 0.362 | 0.261 | 0.774 | 289.86 | 12.08 | 2.34 | 4.27 | 0.18 | 0.03 |
| GPT-5 nano | 0 | 1.320 | -1.260 | 3.261 | 40.93 | 1.71 | 0.70 | 0.04 | 0.00 | 0.00 |
| Claude Opus 4.6 | 0 | 0.298 | 0.079 | 0.558 | 1706.30 | 71.10 | 128.12 | 5.35 | 0.22 | 0.07 |
| Claude Haiku 4.5 | 0 | 3.002 | 2.912 | 10.748 | 332.91 | 13.87 | 2.76 | 1.01 | 0.04 | 0.01 |
| Gemini 3.1 Pro Preview | 0 | 0.700 | 0.580 | 2.015 | 4007.64 | 166.98 | 67.34 | 7.42 | 0.31 | 0.09 |
| Gemini 3.1 Flash-Lite Preview | 0 | 0.603 | 0.437 | 1.307 | 78.97 | 3.29 | 0.36 | 0.25 | 0.01 | 0.00 |
| ground truth | LLM average | LLM std. dev. | DeepSeek-V4-Pro | Out tokens | Time | Kimi K2.6 | Out tokens | Time | MiniMax M2.7 | Out tokens | Time | Qwen 3.6 Plus | Out tokens | Time | GLM 5.1 | Out tokens | Time | GPT-5.5 | Out tokens | Time | GPT-5 nano | Out tokens | Time | Claude Opus 4.6 | Out tokens | Time | Claude Haiku 4.5 | Out tokens | Time | Gemini 3.1 Pro Preview | Out tokens | Time | Gemini 3.1 Flash-Lite Preview | Out tokens | Time | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| ENEL - SPA 2022 | Improving | -0.55 | 0.69 | Deteriorating | 1246 | 54.156 | Deteriorating | 6038 | 85.018 | Improving | 16298 | 143.165 | Deteriorating | 5670 | 103.644 | Stable | 2178 | 39.072 | Deteriorating | 1199 | 15.687 | Stable | 16 | 1.408 | Deteriorating | 1551 | 31.261 | Deteriorating | 2384 | 20.358 | Stable | 9840 | 70.807 | Deteriorating | 1069 | 3.776 |
| ENI S.P.A. 2022 | Improving | 0.91 | 0.30 | Improving | 2629 | 174.413 | Improving | 13319 | 228.173 | Improving | 12540 | 107.44 | Improving | 3564 | 43.637 | Improving | 1889 | 52.107 | Improving | 1559 | 19.246 | Stable | 16 | 1.213 | Improving | 1684 | 30.93 | Improving | 2037 | 16.3 | Improving | 6438 | 47.67 | Improving | 786 | 3.134 |
| A2A ENERGIA S.P.A. 2022 | Improving | -0.82 | 0.40 | Deteriorating | 1700 | 61.116 | Deteriorating | 11352 | 175.131 | Deteriorating | 21789 | 185.966 | Deteriorating | 3747 | 59.372 | Deteriorating | 1347 | 27.661 | Deteriorating | 1161 | 12.74 | Stable | 16 | 1.149 | Deteriorating | 1330 | 25.779 | Deteriorating | 2599 | 21.247 | Stable | 7987 | 59.816 | Deteriorating | 1191 | 3.843 |
| "TERNA - RETE ELETTRICA NAZIONALE SOCIETA' PER AZIONI" (IN FORMA ABBREVIATA "TERNA S.P.A.") 2022 | Improving | 0.91 | 0.30 | Improving | 1215 | 44.635 | Improving | 3316 | 50.122 | Improving | 8631 | 108.408 | Improving | 3279 | 69.994 | Improving | 1463 | 27.729 | Improving | 1969 | 22.027 | Stable | 16 | 1.084 | Improving | 1914 | 33.376 | Improving | 2365 | 19.099 | Improving | 6717 | 47.407 | Improving | 1257 | 4.04 |
| REDEIA CORPORACION SA 2022 | Improving | 0.27 | 0.65 | Stable | 1531 | 73.324 | Stable | 4398 | 82.049 | Improving | 13443 | 133.097 | Stable | 3403 | 58.724 | Deteriorating | 1303 | 30.924 | Stable | 1624 | 20.056 | Stable | 16 | 1.116 | Improving | 1850 | 34.95 | Improving | 1725 | 13.68 | Improving | 11894 | 89.231 | Stable | 887 | 3.101 |
| EDP, S.A. 2022 | Improving | 0.27 | 0.79 | Improving | 2504 | 92.317 | Improving | 9087 | 112.218 | Stable | 14989 | 98.847 | Stable | 5053 | 79.166 | Stable | 1773 | 38.219 | Improving | 1716 | 18.969 | Stable | 16 | 1.373 | Improving | 4348 | 91.357 | Deteriorating | 3119 | 26.384 | Improving | 8190 | 58.74 | Deteriorating | 1257 | 4.047 |
| ENGIE 2022 | Improving | -0.27 | 0.90 | Improving | 2275 | 64.878 | Deteriorating | 10883 | 128.246 | Deteriorating | 11810 | 94.513 | Deteriorating | 4920 | 60.608 | Improving | 1828 | 32.995 | Deteriorating | 1481 | 18.047 | Stable | 16 | 1.181 | Stable | 8876 | 180.435 | Deteriorating | 3176 | 24.972 | Improving | 7091 | 52.915 | Deteriorating | 863 | 3.269 |
| IBERDROLA SA 2022 | Stable | -0.36 | 0.50 | Deteriorating | 2820 | 65.959 | Stable | 11003 | 144.914 | Stable | 8096 | 79.752 | Deteriorating | 4785 | 80.087 | Stable | 2059 | 43.242 | Stable | 1593 | 16.886 | Stable | 16 | 15.758 | Stable | 2179 | 45.316 | Deteriorating | 2724 | 20.22 | Stable | 10131 | 74.855 | Deteriorating | 807 | 3.61 |
| NATURGY ENERGY GROUP SA 2022 | Improving | 0.91 | 0.30 | Improving | 1074 | 44.55 | Improving | 4684 | 64.078 | Improving | 6624 | 87.995 | Improving | 3858 | 57.747 | Improving | 1601 | 30.599 | Improving | 1547 | 19.143 | Stable | 16 | 1.24 | Improving | 2732 | 54.96 | Improving | 2703 | 22.419 | Improving | 4052 | 30.555 | Improving | 1170 | 4.702 |
| TenneT Holding B.V. 2022 | Deteriorating | -0.91 | 0.30 | Deteriorating | 2707 | 84.255 | Deteriorating | 16411 | 234.749 | Deteriorating | 12769 | 151.027 | Deteriorating | 4710 | 74.051 | Deteriorating | 1703 | 28.386 | Deteriorating | 1530 | 17.297 | Stable | 16 | 1.093 | Deteriorating | 5348 | 124.32 | Deteriorating | 2369 | 17.234 | Deteriorating | 6335 | 50.196 | Deteriorating | 861 | 3.758 |
| VEOLIA ENVIRONNEMENT 2022 | Deteriorating | -0.27 | 0.90 | Deteriorating | 2578 | 133.855 | Stable | 14133 | 167.043 | Improving | 12898 | 85.73 | Deteriorating | 6874 | 105.388 | Deteriorating | 1756 | 37.461 | Deteriorating | 1968 | 23.695 | Stable | 16 | 1.331 | Improving | 7445 | 151.935 | Improving | 2462 | 19.88 | Deteriorating | 18275 | 125.12 | Deteriorating | 888 | 3.346 |
| ØRSTED A/S 2022 | Improving | 0.73 | 0.65 | Improving | 2749 | 82.194 | Improving | 2905 | 42.904 | Improving | 16073 | 144.862 | Improving | 3076 | 38.234 | Improving | 1624 | 32.763 | Improving | 1609 | 19.795 | Deteriorating | 1325 | 10.867 | Stable | 2033 | 42.741 | Improving | 2970 | 22.893 | Improving | 8165 | 70.11 | Improving | 965 | 4.586 |
| ELECTRICITE DE FRANCE 2022 | Deteriorating | -0.73 | 0.65 | Deteriorating | 3515 | 57.595 | Deteriorating | 11260 | 244.548 | Deteriorating | 10747 | 111.594 | Deteriorating | 6756 | 84.989 | Deteriorating | 2040 | 38.573 | Deteriorating | 1791 | 23.397 | Stable | 16 | 6.322 | Deteriorating | 2192 | 49.271 | Improving | 2488 | 19.879 | Deteriorating | 4883 | 39.241 | Deteriorating | 1078 | 4.502 |
| ERG S.P.A. 2022 | NA | 0.91 | 0.30 | Improving | 1830 | 78.944 | Improving | 10706 | 113.982 | Improving | 17515 | 219.485 | Improving | 4629 | 74.29 | Improving | 1855 | 38.53 | Improving | 1427 | 16.561 | Stable | 16 | 1.229 | Improving | 2802 | 57.106 | Improving | 2270 | 17.101 | Improving | 6066 | 42.463 | Improving | 1155 | 3.932 |
| Acciona SA 2022 | NA | 0.45 | 0.82 | Stable | 2079 | 155.307 | Improving | 10594 | 176.585 | Improving | 16772 | 155.079 | Improving | 4364 | 83.547 | Deteriorating | 1763 | 43.683 | Improving | 1727 | 21.461 | Stable | 16 | 1.273 | Improving | 1797 | 32.981 | Improving | 2737 | 23.22 | Improving | 12270 | 91.274 | Deteriorating | 832 | 3.252 |
| Fortum Oyj 2022 | Improving | 0.91 | 0.30 | Improving | 3088 | 108.432 | Improving | 9623 | 222.387 | Improving | 15454 | 106.729 | Improving | 3430 | 55.556 | Improving | 1762 | 32.974 | Improving | 1561 | 16.778 | Stable | 16 | 1.161 | Improving | 1437 | 29.219 | Improving | 2288 | 18.941 | Improving | 7551 | 61.466 | Improving | 978 | 3.813 |
| VERBUND AG 2022 | Deteriorating | 0.09 | 0.94 | Improving | 1780 | 53.206 | Improving | 10694 | 150.586 | Deteriorating | 22518 | 152.183 | Deteriorating | 6160 | 115.646 | Deteriorating | 1855 | 31.258 | Improving | 1465 | 15.028 | Stable | 16 | 1.247 | Improving | 2244 | 50.314 | Stable | 2670 | 22.142 | Improving | 7524 | 56.317 | Deteriorating | 824 | 3.505 |
| ITALGAS S.P.A. 2022 | NA | -0.91 | 0.30 | Deteriorating | 1137 | 57.158 | Deteriorating | 5002 | 82.742 | Deteriorating | 13960 | 178.286 | Deteriorating | 4107 | 73.066 | Deteriorating | 1590 | 34.421 | Deteriorating | 1789 | 18.832 | Stable | 16 | 1.251 | Deteriorating | 1985 | 50.212 | Deteriorating | 2102 | 16.319 | Deteriorating | 6229 | 46.606 | Deteriorating | 848 | 3.228 |
| ACEA S.P.A. 2022 | NA | -0.18 | 0.40 | Stable | 2642 | 78.905 | Stable | 12687 | 186.603 | Stable | 13478 | 121.449 | Deteriorating | 4380 | 58.331 | Stable | 1375 | 23.924 | Stable | 1261 | 13.751 | Stable | 16 | 10.169 | Stable | 1447 | 27.831 | Stable | 2163 | 15.697 | Deteriorating | 18307 | 164.394 | Stable | 906 | 3.387 |
| REN - REDES ENERGÉTICAS NACIONAIS, SGPS, S.A. 2022 | Improving | 0.91 | 0.30 | Improving | 2040 | 79.238 | Improving | 2817 | 46.42 | Improving | 18274 | 177.855 | Improving | 3279 | 36.584 | Improving | 1338 | 26.54 | Improving | 1681 | 17.624 | Stable | 16 | 1.037 | Improving | 1765 | 34.118 | Improving | 1890 | 14.205 | Improving | 5246 | 37.416 | Improving | 846 | 3.315 |
| HERA S.P.A. 2022 | Deteriorating | -0.91 | 0.30 | Deteriorating | 1513 | 57.543 | Deteriorating | 15076 | 219.009 | Deteriorating | 18964 | 233.59 | Deteriorating | 5513 | 104.315 | Deteriorating | 1523 | 31.593 | Deteriorating | 1547 | 16.412 | Stable | 16 | 1.204 | Deteriorating | 1927 | 35.337 | Deteriorating | 2119 | 14.823 | Deteriorating | 6422 | 46.618 | Deteriorating | 935 | 3.737 |
| VINCI 2022 | Improving | 0.91 | 0.30 | Improving | 2182 | 90.575 | Improving | 9972 | 133.37 | Improving | 20052 | 216.119 | Improving | 6064 | 111.582 | Improving | 1795 | 39.174 | Improving | 1310 | 14.813 | Stable | 16 | 1.188 | Improving | 2551 | 55.046 | Improving | 2290 | 16.397 | Improving | 5929 | 42.244 | Improving | 890 | 3.497 |
| BOUYGUES 2022 | Deteriorating | -0.91 | 0.30 | Deteriorating | 3130 | 110.986 | Deteriorating | 11728 | 153.578 | Deteriorating | 19112 | 198.501 | Deteriorating | 5542 | 103.54 | Deteriorating | 1771 | 30.328 | Deteriorating | 1249 | 12.83 | Stable | 16 | 1.033 | Deteriorating | 2006 | 36.183 | Deteriorating | 2296 | 16.944 | Deteriorating | 6549 | 48.055 | Deteriorating | 1029 | 3.738 |
| FERROVIAL SA 2022 | NA | -0.64 | 0.67 | Deteriorating | 7962 | 117.36 | Deteriorating | 20993 | 281.536 | Stable | 16844 | 170.044 | Deteriorating | 6838 | 90.506 | Deteriorating | 1895 | 32.57 | Deteriorating | 1996 | 22.702 | Stable | 16 | 1.187 | Improving | 2699 | 59.127 | Deteriorating | 3563 | 27.24 | Deteriorating | 11056 | 80.463 | Deteriorating | 898 | 3.645 |
| Failures | Mean Absolute Error | Mean Bias Error | Root Mean Squared Error | Time (total) | Time (average) | Time (std. dev.) | Cost (total) | Cost (average) | Cost (std. dev.) | |
|---|---|---|---|---|---|---|---|---|---|---|
| DeepSeek-V4-Pro | 0 | 0.42 | -0.21 | 0.86 | 2020.90 | 84.20 | 34.11 | 1.52 | 0.06 | 0.01 |
| Kimi K2.6 | 0 | 0.53 | -0.21 | 0.97 | 3525.99 | 146.92 | 68.75 | 1.68 | 0.07 | 0.02 |
| MiniMax M2.7 | 0 | 0.37 | -0.16 | 0.83 | 3461.72 | 144.24 | 45.55 | 0.66 | 0.03 | 0.00 |
| Qwen 3.6 Plus | 0 | 0.47 | -0.47 | 0.89 | 1822.60 | 75.94 | 23.18 | 0.80 | 0.03 | 0.01 |
| GLM 5.1 | 0 | 0.32 | -0.32 | 0.73 | 824.73 | 34.36 | 6.42 | 1.26 | 0.05 | 0.01 |
| GPT-5.5 | 0 | 0.47 | -0.26 | 0.95 | 433.78 | 18.07 | 3.15 | 4.92 | 0.21 | 0.03 |
| GPT-5 nano | 0 | 1.00 | -0.37 | 1.05 | 67.11 | 2.80 | 3.91 | 0.04 | 0.00 | 0.00 |
| Claude Opus 4.6 | 0 | 0.53 | -0.11 | 0.97 | 1364.10 | 56.84 | 40.35 | 5.88 | 0.24 | 0.06 |
| Claude Haiku 4.5 | 0 | 0.74 | -0.21 | 1.17 | 467.59 | 19.48 | 3.78 | 1.14 | 0.05 | 0.01 |
| Gemini 3.1 Pro Preview | 0 | 0.21 | 0.00 | 0.56 | 1533.98 | 63.92 | 30.12 | 4.22 | 0.18 | 0.05 |
| Gemini 3.1 Flash-Lite Preview | 0 | 0.53 | -0.53 | 0.97 | 88.76 | 3.70 | 0.44 | 0.26 | 0.01 | 0.00 |
| Failures | Mean Absolute Error | Mean Bias Error | Root Mean Squared Error | Time (total) | Time (average) | Time (std. dev.) | Cost (total) | Cost (average) | Cost (std. dev.) | |
|---|---|---|---|---|---|---|---|---|---|---|
| DeepSeek-V4-Pro | 0 | 0.54 | 0.38 | 0.79 | 1030.65 | 42.94 | 18.80 | 1.40 | 0.06 | 0.01 |
| Kimi K2.6 | 0 | 0.46 | 0.38 | 0.74 | 692.38 | 28.85 | 7.54 | 0.88 | 0.04 | 0.01 |
| MiniMax M2.7 | 0 | 0.62 | 0.62 | 0.94 | 1087.10 | 45.30 | 19.99 | 0.34 | 0.01 | 0.00 |
| Qwen 3.6 Plus | 0 | 0.58 | 0.50 | 0.91 | 665.85 | 27.74 | 15.80 | 0.53 | 0.02 | 0.00 |
| GLM 5.1 | 0 | 0.54 | 0.46 | 0.89 | 387.38 | 16.14 | 3.55 | 1.15 | 0.05 | 0.01 |
| GPT-5.5 | 0 | 0.46 | 0.38 | 0.74 | 323.80 | 13.49 | 1.89 | 4.31 | 0.18 | 0.03 |
| GPT-5 nano | 0 | 0.62 | 0.12 | 0.79 | 112.07 | 4.67 | 1.31 | 0.04 | 0.00 | 0.00 |
| Claude Opus 4.6 | 0 | 0.71 | 0.71 | 1.02 | 856.69 | 35.70 | 4.31 | 5.08 | 0.21 | 0.03 |
| Claude Haiku 4.5 | 0 | 0.71 | -0.12 | 0.89 | 489.09 | 20.38 | 2.26 | 1.07 | 0.04 | 0.01 |
| Gemini 3.1 Pro Preview | 0 | 0.50 | 0.42 | 0.82 | 600.71 | 25.03 | 8.89 | 2.49 | 0.10 | 0.02 |
| Gemini 3.1 Flash-Lite Preview | 0 | 0.58 | 0.50 | 0.91 | 74.74 | 3.11 | 0.33 | 0.24 | 0.01 | 0.00 |
| ground truth | LLM average | LLM std. dev. | DeepSeek-V4-Pro | Out tokens | Time | Kimi K2.6 | Out tokens | Time | MiniMax M2.7 | Out tokens | Time | Qwen 3.6 Plus | Out tokens | Time | GLM 5.1 | Out tokens | Time | GPT-5.5 | Out tokens | Time | GPT-5 nano | Out tokens | Time | Claude Opus 4.6 | Out tokens | Time | Claude Haiku 4.5 | Out tokens | Time | Gemini 3.1 Pro Preview | Out tokens | Time | Gemini 3.1 Flash-Lite Preview | Out tokens | Time | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| ENEL - SPA 2022 | 0.0375 | 0.08 | 0.02 | 7.5% | 1497 | 131.835 | 7.5% | 1611 | 28.711 | 7.5% | 4759 | 55.342 | 7.5% | 1873 | 62.403 | 11.25% | 1036 | 21.206 | 11.25% | 731 | 13.205 | 3.75% | 18 | 2.514 | 11.25% | 1399 | 35.447 | 7.5% | 1566 | 17.256 | 7.5% | 6087 | 55.69 | 7.5% | 681 | 3.493 |
| ENI S.P.A. 2022 | 0.075 | 0.06 | 0.04 | 7.5% | 790 | 43.512 | 3.75% | 2207 | 38.097 | 11.25% | 5072 | 61.399 | 0% | 1570 | 44.593 | 0% | 776 | 20.325 | 7.5% | 785 | 14.471 | 11.25% | 751 | 7.562 | 7.5% | 1310 | 34.977 | 7.5% | 1784 | 19.938 | 7.5% | 9572 | 78.793 | 3.75% | 490 | 3.238 |
| A2A ENERGIA S.P.A. 2022 | 0.075 | 0.06 | 0.03 | 0% | 620 | 34.126 | 7.5% | 1720 | 30.193 | 7.5% | 4524 | 39.131 | 3.75% | 2838 | 52.446 | 11.25% | 879 | 19.958 | 7.5% | 583 | 13.269 | 7.5% | 18 | 1.162 | 7.5% | 1648 | 38.93 | 7.5% | 1655 | 17.314 | 0% | 4848 | 41.697 | 7.5% | 577 | 2.99 |
| "TERNA - RETE ELETTRICA NAZIONALE SOCIETA' PER AZIONI" (IN FORMA ABBREVIATA "TERNA S.P.A.") 2022 | 0.075 | 0.07 | 0.02 | 7.5% | 903 | 35.119 | 7.5% | 2833 | 49.272 | 7.5% | 7968 | 86.875 | 7.5% | 1574 | 37.005 | 3.75% | 904 | 24.285 | 7.5% | 748 | 14.712 | 11.25% | 538 | 6.085 | 7.5% | 1588 | 40.183 | 7.5% | 2200 | 24.38 | 7.5% | 6011 | 59.372 | 7.5% | 523 | 3.532 |
| REDEIA CORPORACION SA 2022 | 0.075 | 0.02 | 0.03 | 0% | 670 | 29.429 | 7.5% | 1571 | 28.135 | 0% | 9204 | 95.239 | 0% | 1504 | 50.086 | 7.5% | 910 | 22.066 | 3.75% | 494 | 9.717 | 3.75% | 18 | 1.675 | 0% | 1374 | 35.581 | 3.75% | 1710 | 18.397 | 0% | 2807 | 24.605 | 0% | 438 | 2.696 |
| EDP, S.A. 2022 | 0.075 | 0.10 | 0.02 | 7.5% | 798 | 47.173 | 11.25% | 1994 | 31.198 | 11.25% | 7434 | 55.762 | 11.25% | 2407 | 39.109 | 7.5% | 860 | 19.973 | 11.25% | 602 | 11.271 | 7.5% | 18 | 1.238 | 11.25% | 1482 | 37.978 | 7.5% | 1943 | 20.352 | 11.25% | 3255 | 32.1 | 11.25% | 543 | 3.669 |
| ENGIE 2022 | 0.0375 | 0.07 | 0.02 | 7.5% | 700 | 47.719 | 7.5% | 1325 | 27.274 | 7.5% | 6062 | 70.207 | 7.5% | 1893 | 44.741 | 7.5% | 1001 | 33.615 | 7.5% | 794 | 13.982 | 7.5% | 18 | 1.396 | 7.5% | 1480 | 38.688 | 3.75% | 1911 | 19.808 | 3.75% | 5312 | 44.57 | 7.5% | 557 | 3.384 |
| IBERDROLA SA 2022 | 0.0375 | 0.07 | 0.02 | 3.75% | 1697 | 80.393 | 11.25% | 2127 | 37.25 | 7.5% | 7700 | 61.979 | 3.75% | 2012 | 49.565 | 7.5% | 829 | 20.06 | 7.5% | 670 | 11.702 | 7.5% | 892 | 7.863 | 11.25% | 1514 | 39.775 | 7.5% | 1988 | 21.201 | 7.5% | 4776 | 40.427 | 7.5% | 697 | 4.504 |
| NATURGY ENERGY GROUP SA 2022 | 0.0375 | 0.05 | 0.03 | 0% | 760 | 78.816 | 7.5% | 2197 | 32.204 | 0% | 11139 | 131.749 | 7.5% | 1750 | 33.767 | 7.5% | 858 | 19.452 | 7.5% | 624 | 11.679 | 7.5% | 656 | 7.41 | 3.75% | 1349 | 34.668 | 7.5% | 2017 | 21.59 | 3.75% | 6129 | 51.619 | 7.5% | 546 | 3.106 |
| TenneT Holding B.V. 2022 | 0.075 | 0.11 | 0.01 | 11.25% | 861 | 37.454 | 11.25% | 1330 | 24.62 | 11.25% | 4595 | 42.234 | 11.25% | 2656 | 45.453 | 11.25% | 784 | 16.223 | 11.25% | 665 | 11.47 | 7.5% | 272 | 3.177 | 11.25% | 1605 | 41.531 | 11.25% | 2052 | 24.042 | 11.25% | 4204 | 33.985 | 11.25% | 505 | 3.209 |
| VEOLIA ENVIRONNEMENT 2022 | 0.0375 | 0.10 | 0.02 | 11.25% | 988 | 51.24 | 11.25% | 2020 | 29.245 | 11.25% | 14523 | 139.462 | 11.25% | 1765 | 32.612 | 7.5% | 909 | 22.879 | 11.25% | 645 | 11.29 | 11.25% | 467 | 26.46 | 11.25% | 1444 | 38.313 | 7.5% | 1357 | 14.891 | 7.5% | 8404 | 68.955 | 11.25% | 662 | 3.552 |
| ØRSTED A/S 2022 | 0.1125 | 0.10 | 0.05 | 3.75% | 940 | 39.251 | 15% | 3209 | 48.379 | 11.25% | 9563 | 70.401 | 11.25% | 1803 | 30.283 | 0% | 765 | 17.171 | 11.25% | 803 | 14.066 | 11.25% | 687 | 6.95 | 15% | 1486 | 38.091 | 3.75% | 2385 | 23.856 | 11.25% | 3272 | 27.582 | 11.25% | 483 | 3.208 |
| ELECTRICITE DE FRANCE 2022 | 0.075 | 0.13 | 0.03 | 11.25% | 980 | 82.264 | 11.25% | 1458 | 24.343 | 11.25% | 8759 | 102.625 | 15% | 3196 | 66.937 | 15% | 831 | 26.03 | 15% | 893 | 14.463 | 11.25% | 409 | 3.994 | 15% | 1618 | 41.134 | 7.5% | 1585 | 18.796 | 15% | 2729 | 24.106 | 15% | 449 | 2.995 |
| ERG S.P.A. 2022 | 0.075 | 0.05 | 0.04 | 7.5% | 1678 | 111.14 | 0% | 2291 | 32.249 | 11.25% | 12717 | 114.854 | 7.5% | 1918 | 29.046 | 7.5% | 872 | 28.88 | 7.5% | 658 | 12.435 | 7.5% | 473 | 5.125 | 3.75% | 1436 | 39.933 | 7.5% | 1745 | 19.548 | 0% | 2049 | 19.754 | 0% | 513 | 3.371 |
| Acciona SA 2022 | 0.0375 | 0.05 | 0.04 | 3.75% | 807 | 35.662 | 11.25% | 1510 | 24.193 | 0% | 6462 | 45.32 | 7.5% | 3756 | 57.088 | 3.75% | 687 | 17.278 | 7.5% | 653 | 12.418 | 3.75% | 18 | 1.213 | 7.5% | 1702 | 41.805 | 7.5% | 2402 | 25.499 | 0% | 7877 | 71.224 | 7.5% | 466 | 2.818 |
| Fortum Oyj 2022 | 0.0 | 0.06 | 0.04 | 3.75% | 1016 | 40.204 | 7.5% | 1088 | 23.771 | 7.5% | 4300 | 51.984 | 7.5% | 1233 | 29.005 | 15% | 748 | 20.13 | 7.5% | 576 | 9.832 | 0% | 16 | 1.52 | 7.5% | 1508 | 37.086 | 7.5% | 1644 | 18.845 | 0% | 4290 | 37.159 | 7.5% | 519 | 3.074 |
| VERBUND AG 2022 | 0.0375 | 0.02 | 0.03 | 0% | 599 | 45.705 | 0% | 1542 | 26.704 | 0% | 5262 | 45.64 | 0% | 1193 | 23.431 | 0% | 797 | 18.167 | 3.75% | 697 | 12.549 | 7.5% | 583 | 6.534 | 3.75% | 1432 | 37.235 | 7.5% | 2096 | 23.525 | 0% | 1893 | 17.252 | 0% | 550 | 7.035 |
| ITALGAS S.P.A. 2022 | 0.0375 | 0.07 | 0.03 | 7.5% | 924 | 41.671 | 11.25% | 1896 | 27.331 | 11.25% | 5290 | 47.346 | 7.5% | 2031 | 38.48 | 7.5% | 675 | 18.899 | 7.5% | 821 | 13.049 | FAIL | 671 | 6.614 | 7.5% | 1609 | 42.253 | 7.5% | 1767 | 19.336 | 0% | 3029 | 28.051 | 7.5% | 544 | 3.275 |
| ACEA S.P.A. 2022 | 0.075 | 0.06 | 0.03 | 7.5% | 674 | 42.507 | 7.5% | 1479 | 33.157 | 11.25% | 7382 | 66.372 | 7.5% | 1615 | 32.806 | 7.5% | 982 | 21.857 | 7.5% | 598 | 10.493 | 0% | 16 | 1.03 | 7.5% | 1504 | 37.951 | 7.5% | 1796 | 18.324 | 0% | 3081 | 32.148 | 7.5% | 553 | 3.344 |
| REN - REDES ENERGÉTICAS NACIONAIS, SGPS, S.A. 2022 | 0.1125 | 0.05 | 0.04 | 0% | 690 | 39.379 | 3.75% | 1353 | 24.646 | 7.5% | 7458 | 59.076 | 0% | 1976 | 34.69 | 7.5% | 829 | 22.258 | 7.5% | 725 | 12.072 | 11.25% | 582 | 5.701 | 7.5% | 1517 | 36.467 | 0% | 1703 | 19.648 | 0% | 2355 | 19.553 | 7.5% | 567 | 3.3 |
| HERA S.P.A. 2022 | 0.0375 | 0.07 | 0.04 | 0% | 839 | 40.698 | 7.5% | 1799 | 47.437 | 11.25% | 8032 | 57.986 | 7.5% | 2416 | 41.765 | 7.5% | 960 | 25.316 | 7.5% | 690 | 24.883 | 11.25% | 579 | 5.293 | 7.5% | 1455 | 35.719 | 7.5% | 1768 | 18.948 | 0% | 3796 | 37.116 | 7.5% | 587 | 3.407 |
| VINCI 2022 | 0.0 | 0.04 | 0.03 | 0% | 1266 | 43.313 | 0% | 1225 | 31.219 | 0% | 4015 | 30.679 | 3.75% | 2587 | 47.435 | 7.5% | 766 | 27.338 | 7.5% | 662 | 11.995 | 7.5% | 232 | 3.052 | 3.75% | 1490 | 38.546 | 3.75% | 1304 | 15.196 | 0% | 2075 | 19.915 | 7.5% | 516 | 3.109 |
| BOUYGUES 2022 | 0.0 | 0.07 | 0.03 | 3.75% | 566 | 30.946 | 7.5% | 1826 | 25.003 | 11.25% | 11941 | 96.065 | 7.5% | 1108 | 20.667 | 7.5% | 748 | 23.345 | 7.5% | 606 | 10.692 | 7.5% | 507 | 5.142 | 11.25% | 1675 | 43.456 | 7.5% | 1744 | 19.676 | 0% | 2908 | 24.281 | 7.5% | 603 | 3.234 |
| FERROVIAL SA 2022 | 0.0 | 0.07 | 0.03 | 0% | 929 | 46.979 | 7.5% | 1094 | 35.832 | 11.25% | 8861 | 123.78 | 7.5% | 1451 | 22.413 | 11.25% | 633 | 15.844 | 7.5% | 753 | 13.781 | 7.5% | 413 | 4.294 | 7.5% | 1365 | 36.606 | 7.5% | 1873 | 22.029 | 3.75% | 4390 | 36.754 | 11.25% | 561 | 3.428 |
| Failures | Mean Absolute Error | Mean Bias Error | Root Mean Squared Error | Time (total) | Time (average) | Time (std. dev.) | Cost (total) | Cost (average) | Cost (std. dev.) | |
|---|---|---|---|---|---|---|---|---|---|---|
| DeepSeek-V4-Pro | 0 | 0.033 | -0.005 | 0.045 | 1256.54 | 52.36 | 25.86 | 1.40 | 0.06 | 0.01 |
| Kimi K2.6 | 0 | 0.044 | 0.025 | 0.052 | 760.46 | 31.69 | 7.63 | 0.90 | 0.04 | 0.01 |
| MiniMax M2.7 | 0 | 0.045 | 0.027 | 0.055 | 1751.51 | 72.98 | 31.02 | 0.44 | 0.02 | 0.00 |
| Qwen 3.6 Plus | 0 | 0.044 | 0.016 | 0.053 | 965.83 | 40.24 | 12.29 | 0.60 | 0.03 | 0.00 |
| GLM 5.1 | 0 | 0.048 | 0.023 | 0.062 | 522.56 | 21.77 | 4.23 | 1.17 | 0.05 | 0.01 |
| GPT-5.5 | 0 | 0.037 | 0.031 | 0.047 | 309.50 | 12.90 | 2.93 | 4.29 | 0.18 | 0.03 |
| GPT-5 nano | 1 | 0.033 | 0.023 | 0.044 | 123.00 | 5.13 | 5.11 | 0.04 | 0.00 | 0.00 |
| Claude Opus 4.6 | 0 | 0.042 | 0.030 | 0.052 | 922.35 | 38.43 | 2.44 | 5.14 | 0.21 | 0.03 |
| Claude Haiku 4.5 | 0 | 0.034 | 0.016 | 0.046 | 482.39 | 20.10 | 2.76 | 1.07 | 0.04 | 0.01 |
| Gemini 3.1 Pro Preview | 0 | 0.036 | -0.011 | 0.048 | 926.71 | 38.61 | 17.47 | 3.06 | 0.13 | 0.03 |
| Gemini 3.1 Flash-Lite Preview | 0 | 0.045 | 0.023 | 0.054 | 82.97 | 3.46 | 0.84 | 0.24 | 0.01 | 0.00 |
| Ground truth priorities | Ground truth order | LLM dist average | LLM dist std. dev. | DeepSeek-V4-Pro | Distance | Out tokens | Time | Kimi K2.6 | Distance | Out tokens | Time | MiniMax M2.7 | Distance | Out tokens | Time | Qwen 3.6 Plus | Distance | Out tokens | Time | GLM 5.1 | Distance | Out tokens | Time | GPT-5.5 | Distance | Out tokens | Time | GPT-5 nano | Distance | Out tokens | Time | Claude Opus 4.6 | Distance | Out tokens | Time | Claude Haiku 4.5 | Distance | Out tokens | Time | Gemini 3.1 Pro Preview | Distance | Out tokens | Time | Gemini 3.1 Flash-Lite Preview | Distance | Out tokens | Time | |
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| ELECTRIC - "TERNA - - IBERDROL 2022 | 11, 8, 3 | C,B,A | 1.82 | 0.98 | A,B,C | 3 | 1124 | 56.978 | A,C,B | 2 | 1411 | 30.837 | A,B,C | 3 | 8335 | 143.533 | B,C,A | 1 | 1290 | 15.608 | B,A,C | 2 | 1251 | 25.662 | B,C,A | 1 | 1162 | 19.158 | A,B,C | 3 | 17 | 16.489 | A,C,B | 2 | 1236 | 32.826 | B,C,A | 1 | 1470 | 16.665 | A,C,B | 2 | 2834 | 25.433 | C,B,A | 0 | 541 | 3.307 |
| ENI S.P. - TenneT H - REDEIA C 2022 | 12, 7, 4 | C,B,A | 1.27 | 0.65 | B,C,A | 1 | 684 | 27.403 | C,A,B | 1 | 1480 | 30.716 | B,C,A | 1 | 4371 | 49.228 | B,C,A | 1 | 1369 | 17.692 | B,A,C | 2 | 690 | 21.989 | B,C,A | 1 | 1397 | 23.493 | A,B,C | 3 | 17 | 2.166 | B,C,A | 1 | 1460 | 40.157 | B,C,A | 1 | 1485 | 16.536 | B,C,A | 1 | 4572 | 38.074 | B,C,A | 1 | 576 | 3.431 |
| VEOLIA E - ELECTRIC - REDEIA C 2022 | 5, 11, 4 | C,A,B | 2.09 | 1.14 | A,C,B | 1 | 936 | 80.943 | B,A,C | 3 | 1502 | 25.329 | A,C,B | 1 | 5146 | 79.074 | A,C,B | 1 | 1293 | 15.674 | B,A,C | 3 | 712 | 22.018 | A,B,C | 2 | 783 | 15.972 | B,A,C | 3 | 17 | 1.762 | B,A,C | 3 | 1462 | 39.949 | C,A,B | 0 | 1726 | 18.814 | B,A,C | 3 | 3891 | 31.315 | B,A,C | 3 | 525 | 3.213 |
| "TERNA - - A2A ENER - REDEIA C 2022 | 8, 9, 4 | C,A,B | 1.09 | 0.70 | A,C,B | 1 | 671 | 20.122 | C,A,B | 0 | 1139 | 21.782 | A,C,B | 1 | 2809 | 35.457 | A,C,B | 1 | 1193 | 13.323 | A,C,B | 1 | 885 | 17.033 | A,C,B | 1 | 591 | 11.294 | A,B,C | 2 | 17 | 1.357 | A,B,C | 2 | 1499 | 44.654 | C,A,B | 0 | 1233 | 13.92 | B,C,A | 2 | 17433 | 144.376 | A,C,B | 1 | 512 | 3.207 |
| A2A ENER - ENEL - S - "TERNA - 2022 | 9, 1, 8 | B,C,A | 1.09 | 1.04 | A,C,B | 3 | 729 | 56.106 | A,B,C | 2 | 1763 | 34.209 | B,C,A | 0 | 4812 | 66.775 | B,C,A | 0 | 1300 | 18.074 | C,B,A | 1 | 833 | 25.915 | B,C,A | 0 | 1043 | 15.437 | B,A,C | 1 | 17 | 1.668 | A,B,C | 2 | 1508 | 42.098 | C,A,B | 2 | 2138 | 22.889 | B,C,A | 0 | 1188 | 12.307 | C,B,A | 1 | 514 | 3.369 |
| EDP, S.A - ELECTRIC - VEOLIA E 2022 | 2, 11, 5 | A,C,B | 1.55 | 0.93 | B,A,C | 2 | 708 | 73.411 | C,A,B | 1 | 1820 | 34.701 | B,A,C | 2 | 5474 | 62.884 | A,B,C | 1 | 2585 | 34.509 | A,C,B | 0 | 854 | 30.015 | C,B,A | 2 | 1171 | 18.025 | B,A,C | 2 | 17 | 2.147 | C,B,A | 2 | 1423 | 44.845 | A,C,B | 0 | 1444 | 14.734 | B,C,A | 3 | 3375 | 31.251 | B,A,C | 2 | 531 | 3.077 |
| EDP, S.A - A2A ENER - TenneT H 2022 | 2, 9, 7 | A,C,B | 1.00 | 0.00 | C,A,B | 1 | 806 | 46.457 | C,A,B | 1 | 1634 | 27.196 | C,A,B | 1 | 7156 | 76.653 | C,A,B | 1 | 1810 | 24.913 | C,A,B | 1 | 732 | 20.383 | C,A,B | 1 | 879 | 14.17 | C,A,B | 1 | 280 | 3.487 | C,A,B | 1 | 1346 | 33.811 | A,B,C | 1 | 1562 | 16.722 | C,A,B | 1 | 3560 | 30.368 | C,A,B | 1 | 495 | 3.629 |
| ØRSTED A - ENEL - S - A2A ENER 2022 | 6, 1, 9 | B,A,C | 0.82 | 0.60 | A,B,C | 1 | 634 | 44.66 | A,B,C | 1 | 1593 | 38.537 | B,A,C | 0 | 8705 | 245.887 | A,B,C | 1 | 1612 | 22.4 | A,B,C | 1 | 859 | 17.819 | A,B,C | 1 | 1164 | 17.703 | C,B,A | 2 | 17 | 1.813 | A,B,C | 1 | 1421 | 33.347 | A,B,C | 1 | 1768 | 17.43 | B,A,C | 0 | 6704 | 54.288 | B,A,C | 0 | 530 | 3.338 |
| REDEIA C - ELECTRIC - "TERNA - 2022 | 4, 11, 8 | A,C,B | 1.55 | 0.82 | C,A,B | 1 | 799 | 52.182 | C,A,B | 1 | 2012 | 27.8 | C,A,B | 1 | 4433 | 69.808 | C,A,B | 1 | 2783 | 33.668 | C,A,B | 1 | 790 | 27.404 | C,A,B | 1 | 820 | 14.212 | C,A,B | 1 | 17 | 1.759 | B,C,A | 3 | 1081 | 26.647 | B,A,C | 2 | 1417 | 15.445 | B,C,A | 3 | 2746 | 26.43 | B,A,C | 2 | 522 | 3.206 |
| ENEL - S - ELECTRIC - VEOLIA E 2022 | 1, 11, 5 | A,C,B | 0.82 | 0.87 | B,A,C | 2 | 852 | 29.602 | A,C,B | 0 | 1392 | 32.527 | B,A,C | 2 | 7214 | 136.586 | A,C,B | 0 | 1245 | 20.351 | A,B,C | 1 | 832 | 28.155 | A,B,C | 1 | 1333 | 24.645 | A,B,C | 1 | 261 | 3.706 | A,C,B | 0 | 1750 | 44.179 | A,C,B | 0 | 1358 | 16.056 | B,A,C | 2 | 3794 | 33.275 | A,C,B | 0 | 497 | 3.552 |
| TenneT H - ENI S.P. - ENEL - S 2022 | 7, 12, 1 | C,A,B | 1.18 | 0.75 | B,A,C | 3 | 752 | 53.371 | A,C,B | 1 | 1379 | 34.049 | A,C,B | 1 | 5651 | 72.049 | A,C,B | 1 | 1823 | 25.446 | A,C,B | 1 | 859 | 25.996 | C,A,B | 0 | 1304 | 19.014 | A,B,C | 2 | 17 | 1.968 | A,C,B | 1 | 1379 | 36.239 | C,B,A | 1 | 1563 | 16.633 | A,C,B | 1 | 6128 | 55.888 | A,C,B | 1 | 622 | 4.02 |
| TenneT H - REDEIA C - ENGIE 2022 | 7, 4, 10 | B,A,C | 1.36 | 0.67 | A,B,C | 1 | 509 | 54.233 | A,C,B | 2 | 1758 | 30.02 | A,B,C | 1 | 7488 | 101.265 | A,B,C | 1 | 1656 | 19.146 | A,B,C | 1 | 933 | 24.134 | A,C,B | 2 | 1045 | 16.099 | A,B,C | 1 | 17 | 1.285 | A,C,B | 2 | 1345 | 36.031 | B,A,C | 0 | 1400 | 15.374 | A,C,B | 2 | 2614 | 26.019 | A,C,B | 2 | 566 | 3.829 |
| Failures | Average Distance | Time (total) | Time (average) | Time (std. dev.) | Cost (total) | Cost (average) | Cost (std. dev.) | |
|---|---|---|---|---|---|---|---|---|
| DeepSeek-V4-Pro | 0 | 1.67 | 595.47 | 49.62 | 17.80 | 1.18 | 0.10 | 0.02 |
| Kimi K2.6 | 0 | 1.25 | 367.70 | 30.64 | 4.61 | 0.71 | 0.06 | 0.01 |
| MiniMax M2.7 | 0 | 1.17 | 1139.20 | 94.93 | 57.30 | 0.28 | 0.02 | 0.00 |
| Qwen 3.6 Plus | 0 | 0.83 | 260.80 | 21.73 | 6.83 | 0.50 | 0.04 | 0.01 |
| GLM 5.1 | 0 | 1.25 | 286.52 | 23.88 | 4.08 | 0.99 | 0.08 | 0.01 |
| GPT-5.5 | 0 | 1.08 | 209.22 | 17.44 | 3.83 | 3.67 | 0.31 | 0.06 |
| GPT-5 nano | 0 | 1.83 | 39.61 | 3.30 | 4.22 | 0.03 | 0.00 | 0.00 |
| Claude Opus 4.6 | 0 | 1.67 | 454.78 | 37.90 | 5.69 | 4.15 | 0.35 | 0.06 |
| Claude Haiku 4.5 | 0 | 0.75 | 201.22 | 16.77 | 2.31 | 0.84 | 0.07 | 0.01 |
| Gemini 3.1 Pro Preview | 0 | 1.67 | 509.02 | 42.42 | 34.29 | 2.42 | 0.20 | 0.05 |
| Gemini 3.1 Flash-Lite Preview | 0 | 1.17 | 41.18 | 3.43 | 0.28 | 0.22 | 0.02 | 0.00 |