There are 25 examples not solved by any model.
Solving some of these can be a good signal that your model is indeed better than leading models if these are good problems.
atcoder.abc373_g, atcoder.abc376_f, atcoder.abc389_g, atcoder.abc397_d, atcoder.arc181_c, atcoder.arc181_d, atcoder.arc182_d, atcoder.arc183_b, atcoder.arc183_d, atcoder.arc184_d, atcoder.arc186_a, atcoder.arc186_c, atcoder.arc186_e, atcoder.arc188_c, atcoder.arc189_a, atcoder.arc190_a, atcoder.arc190_c, atcoder.arc191_c, atcoder.arc192_b, atcoder.arc193_b, atcoder.arc194_c, atcoder.arc196_a, atcoder.arc196_c, leetcode.3527, leetcode.3763
| example_link | model | min_elo |
|---|---|---|
| atcoder.arc184_c | O4-Mini (High) | 1094.870 |
| atcoder.arc183_c | O3 (High) | 1078.192 |
| leetcode.3638 | O3 (High) | 1078.192 |
| atcoder.arc192_e | O3 (High) | 1078.192 |
| leetcode.3762 | O3 (High) | 1078.192 |
| leetcode.3701 | O3 (High) | 1078.192 |
| atcoder.arc191_d | Gemini-2.5-Pro-06-05 | 1069.976 |
| atcoder.arc196_d | DeepSeek-R1-0528 | 1069.158 |
| atcoder.arc191_a | DeepSeek-R1-0528 | 1069.158 |
| atcoder.abc374_d | DeepSeek-R1-0528 | 1069.158 |
| atcoder.abc392_d | DeepSeek-R1-0528 | 1069.158 |
| atcoder.arc188_d | DeepSeek-R1-0528 | 1069.158 |
| atcoder.abc400_g | DeepSeek-R1-0528 | 1069.158 |
| atcoder.arc195_d | Gemini-2.5-Pro-03-25 | 1048.914 |
| atcoder.abc399_e | Grok-3-Mini (High) | 1044.904 |
| atcoder.abc371_f | Grok-3-Mini (High) | 1044.904 |
| atcoder.abc366_g | Grok-3-Mini (High) | 1044.904 |
| atcoder.abc376_g | Grok-3-Mini (High) | 1044.904 |
| atcoder.abc398_g | Grok-3-Mini (High) | 1044.904 |
| atcoder.abc375_b | Grok-3-Mini (High) | 1044.904 |
| atcoder.abc382_g | Grok-3-Mini (High) | 1044.904 |
| atcoder.abc370_g | Grok-3-Mini (High) | 1044.904 |
| atcoder.abc396_e | Grok-3-Mini (High) | 1044.904 |
| atcoder.arc195_c | Grok-3-Mini (High) | 1044.904 |
| atcoder.arc185_c | Grok-3-Mini (High) | 1044.904 |
| atcoder.abc378_g | Grok-3-Mini (High) | 1044.904 |
| leetcode.3613 | Gemini-Flash-2.0-Thinking-12-19 | 979.924 |
These are 10 problems with the lowest correlation with the overall evaluation (i.e. better models tend to do worse on these. )
| example_link | acc | tau |
|---|---|---|
| atcoder.abc384_f | 0.767 | -0.344 |
| atcoder.abc386_f | 0.800 | -0.308 |
| atcoder.abc384_g | 0.600 | -0.251 |
| atcoder.abc375_a | 0.967 | -0.116 |
| leetcode.3613 | 0.033 | -0.098 |
| atcoder.arc193_a | 0.300 | -0.094 |
| atcoder.abc381_c | 0.967 | -0.089 |
| atcoder.abc389_f | 0.533 | -0.071 |
| atcoder.abc382_f | 0.533 | -0.032 |
| leetcode.3659 | 0.933 | -0.026 |
Histogram of problems by the accuracy on each problem.
Histogram of problems by the minimum Elo to solve each problem.